Decoupling Global and Local Representations via Invertible Generative Flows
Authors: Xuezhe Ma
Xiang Kong
Shanghang Zhang
Eduard H Hovy
Venue: ICLR 2021 Poster (2021)
Keywords: Generative Models
Generative Flow
Normalizing Flow
Image Generation
Representation Learning

Abstract: In this work, we propose a new generative model that is capable of automatically decoupling global and local representations of images in an entirely unsupervised setting, by embedding a generative flow in the VAE framework to model the decoder.
Specifically, the proposed model utilizes the variational auto-encoding framework to learn a (low-dimensional) vector of latent variables to capture the global information of an image, which is fed as a conditional input to a flow-based invertible decoder with architecture borrowed from style transfer literature.
Experimental results on standard image benchmarks demonstrate the effectiveness of our model in terms of density estimation, image generation and unsupervised representation learning.
Importantly, this work demonstrates that with only architectural inductive biases, a generative model with a likelihood-based objective is capable of learning decoupled representations, requiring no explicit supervision.
The code for our model is available at \url{https://github.com/XuezheMax/wolf}.

Published as a conference paper at ICLR 2021
DECOUPLING GLOBAL AND LOCAL REPRESENTA- TIONS VIA INVERTIBLE GENERATIVE FLOWS
Xuezhe Ma' Xiang Kong", Shanghang Zhang?, Eduard Hovy?
University Southern California
"Carnegie Mellon University
University of California, Berkeley
xue @ isi, edu, xiangkêes. cmu cma.edu, shz@berkeley.ede
ABSTRACT
In this work, we propose new generative model that is capable of automatically decoupling global and local representations of images in entirely unsupervised setting, by embedding a generative flow in the VAE framework to model the decoder. Specifically, the proposed model utilizes the variational auto-encoding framework to learn a (low-dimensional) vector of latent variables to capture the global information of image, which is fed conditional input to flow-based invertible decoder with architecture borrowed from style transfer literature. Experi- mental results standard image benchmarks demonstrate the effectiveness of model in terms of density estimation, image generation and unsupervised represen- tation learning. Importantly, this work demonstrates that with only architectural inductive biases, generative model with likelihood- based objective is capable of learning decoupled representations, requiring no explicit supervision. The code for our model is available at htttps://ggtte com/ wwwp///eeeeeewweeeeeaaa
INTRODUCTION
Unsupervised learning of probabilistic models and meaningful representation learning are two central yet challenging problems in machine learning. Formally, let X X' be the random variables of the observed data, e.g., X is image. One goal of generative models is to learn the parameter 0 such that the model distribution (X) can best approximate the true distribution P(X). Throughout the paper, uppercase letters represent random variables and lowercase letters their realizations.
Unsupervised (disentangled) representation learning, besides data distribution estimation and data generation, is also principal component in generative models. The goal is identify and disentangle the underlying causal factors, to tease apart the underlying dependencies of the data, so that it becomes easier understand, classify, or to perform other tasks (Bengio al.) (013). Unsupervised representation learning has spawned significant interests and a number of techniques (Chen et al. 2017a Devlin al., 2019 [Hjelm al. 2019)) has emerged over the years to address this challenge. Among these generative models, VAE (Kingma & Welling 2014;) Rezende et al.) 2014)) and Generative (Normalizing) Flows (Dinh al. 2014) have stood out for their simplicity and effectiveness.
1.1 VARIATIONAL AUTO- ANCO-ENS (VAEs)
VAE, member of latent variable models (LVMs), gains popularity for its capability of automat- ically learning meaningful (low- (iw-demensiona) representations from raw data. the framework of VAEs, set of latent variables Z Z are introduced, and the model distribution Px(X) is defined as the marginal of the joint distribution between X and Z:
(x) = po(a, =)d4(==) = po po(r)=))o(2ddp(3) Vr E X,
(1)
where the joint distribution po(x, 2) is factorized the product of a prior po (=) over the latent Z, and the "generative" distribution (z]2). (x2) is the base measure on the latent space Z.
Work done Carnegie Mellon University.

---

Published as a conference paper at ICLR 2021
(b)
(d)
Figure 1: Examples of the switch operation, which switches the global representations of two images from four datasets: (a) CIFAR-IO, (b) ImageNet, (c) LSUN Bedroom and (d) CelebA-HQ.
In general, this marginal likelihood is intractable to compute or differentiate directly, and Variational Inference (Wainwright et al. 2008) provides solution to optimize the evidence lower bound (ELBO),
an alternative objective by introducing parametric inference model (0(2)z))
Ex(X) [log pa (X)) 2 Ex(x) [E x) Ilog pe (X]2)] KL (Z) (Z))
(2)
where ELBO could be seen as an autoencoding loss with ()(=(xr being the encoder and po (z]22)
being the decoder, with the first term in the RHS in as the reconstruction error.
1.2 GENERATIVE FLOWS
Put simply, generative flows (a.k.a,, normalizing flows) work by transforming a simple distribution, P(T) (e.g. a simple Gaussian) into complex (e.g. the complex distribution of data P(X)) through a chain of invertible transformations.
Formally, generative flow defines bijection function f : T (with g f-)), where E T set of latent variables with simple prior distribution pr (v). It provides us with an invertible transformation between X and Y, whereby the generative process over X is defined straightforwardly:
8 Pr()r. then J = (v).
An important insight behind generative flows is that given this bijection function, the change of the variable formula defines the model distribution on X by:
(a( pr (fe(z)) det @f((z)
(4)
dx
where Hft(#) the Jacobian of fo x. A stacked sequence of such invertible transformations is called generative (normalizing) flow (Rezende & Mohamed 2015):
X 11
f2 112
fK Y.
11
91
82
9s
RK
where / = fi f2 ..... fx is flow of K transformations (omitting for brevity).
2

---

Published as a conference paper at ICLR 2021
1.3 PROBLEMS OF VAES AND GENERATIVE FLOWS
Despite their impressive successes, VAΣS and generative flows still suffer their own problems.
Posterior Collapse in VAES As discussed in Bowman al. (203)), without further assumptions, the ELBO objective in may not guide the model towards the intended role for the latent variables Z,
or even learn uninformative Z with the observation that the KL term (g. (Z) (llpe(()
(2)) vanishes to zero. The essential reason of this posterior collapse problem is that, under absolutely unsupervised setting, the marginal likelihood- based objective incorporates no (direct) supervision on the latent space to characterize the latent variable Z with preferred properties w.r.t. representation learning.
Local Dependency in Generative Flows Generative flows suffer from the limitation of expres- siveness and local dependency. Most generative flows tend capture the dependency among features only locally, and are incapable of realistic synthesis of large images compared to GANs (Goodfellow) al. 2014). Unlike latent variable models, e-g. VAEs, which represent the nigh-dimensiooal data as coordinates in a latent low-dimensional space, the long-term dependencies that usually describe the global features of the data can only be propagated through a composition of transformations. Previous studies attempted to enlarge the receptive field by using a special design of parameterization like masked convolutions (Ma et al. 2019a) or attention mechanism (Ho et al. 2019).
In this paper, we propose simple and effective generative model to simultaneously tackle the aforementioned challenges of VAEs and generative flows by leveraging their properties to complement each other. By embedding generative flow in the VAE framework to model the decoder, the proposed model able to learn decoupled representations which capture global and local information of images respectively in entirely unsupervised manner. The key insight is to utilize the inductive biases from the model architecture design leveraging the VAE framework equipped with a compression encoder to extract the global information in now-dimensional representation, and flow-based decoder which favors local dependencies to store the residual information into a local high-dimensional representation ($2). Experimentally,
four benchmark datasets for images, we demonstrate the effectiveness of our model on two aspects: (i) density estimation and image generation, by consistently achieving significant improvements over Glow (Kingma & Dhariwal 2018). (ii) decoupled representation learning, by performing classification on learned representations the switch operation (see examples in Figure [T.. Perhaps most strikingly, we demonstrate the feasibility of decoupled representation learning via the plain likelihood-baal -ased generation, using only architectural inductive biases ($3)
GENERATIVE MODEL FOR DECOUPLED REPRESENTATION LEARNING
We first illustrate the high-level insights of the architecture design of our generative model (shown in Figure D before detailing each component in the following sections.
2.1 GENERATIVE MODEL ARCHITECTURE
In the training process of our generative model, we minimize the negative ELBO in VAE:
Celbo (6, ②) = Ex(x) [ log po [|)] +
(5)
where (0, Φ) is the negative ELBO of RHS in 2 Specifically, we first feed the input image into the encoder qo (z(x) to compute the latent variable 2. The encoder is designed to be a compres- sion network, which compresses the nigh-dimensional image into a low-dimensional vector ($2.2). Through this compression process, the local information of an image is enforced to be discarded, yielding representation that captures the global information. Then we feed as a conditional input to flow-based decoder, which transforms & into the representation v with the same dimension (2..) Since the decoder is invertible, with and v, we can exactly reconstruct the original image It indicates that and u maintain all the information of and the reconstruction process can be regarded as an additional operation - adding z and v to recover æ. In this way, we expect that the local information discarded in the compression process will be restored in
In the generative process, we combine the sampling procedures of VAEs and generative flows: we first sample value of 2 from the prior distribution p(2), and value of u from pr (v);; second, input 2 and u into the invertible function f modeled by the generative flow decoder to generate an image x fo (v, 2) go(u, z).
3

---

Published as a conference paper at ICLR 2021
Flow-based Decoder
pa(x|z)
ff(:z)
fg (v:z)
Local Representation
Same size with
discarding local information
Compression Encoder
Global Representation
gp(2|x)
low-dimensional
Figure 2: Diagram to illustrate the process of decoupling an image æ into the global representation and local representation v. The key insight the architecture design of the compression encoder and the invertible decoder.
2.2 COMPRESSION ENCODER
Following previous work, the variational posterior distribution (o(=|z), a.k.a encoder, models latent variable Z as diagonal Gaussian with learned mean and variance:
(0(=|>) = (22;)(2))
(6)
where vx-) and (-) are neural networks. In the context of 2D images where a tensor of shape [h x w c] with spatial dimensions (h, w) and channel dimension the compression encoder maps each image x to dy -dddimennioona vector. d, is the dimension of the latent space.
In this work, the motivation of the encoder is compress the high-dimensional data low- dimensional latent variable z, i.e. x w e >> dz, to enforce the latent representation = to capture the global features of x. Furthermore, unlike previous studies VAE based generative models for natural images (Kingma et al., 2016 CCenn et al., 2017a; Ma et al., 2019b) that represented latent codes low-resolution feature maps") represent erase the local spatial dependencies. Concretely, we implement the encoder with a similar architecture in ResNet (He et al. 2016). The spatial downsampling is implemented by 2-strided ResNet block with x filters. On top of these ResNet blocks, there is one more fully-connected laver with number of output units equal to de 2 to generate p(x) and log (zx) (details in Appendix
unstructured I-dimensional vector to
Following Ma al. (2019c), initialize the weights of the last fully- connected layer that generates the μ and log a² values with zeros. This ensures that the posterior distribution is initialized simple normal distribution, which has been demonstrated helpful for training very deep neural networks more stably in the framework of VAEs.
Zero initialization.
2.3 INVERTIBLE DECODER BASED ON GENERATIVE FLOW
The flow-based decoder defines a (conditionally) invertible function v = (x; 2). where v follows
a standard normal distribution v ~ (O, I). Conditioned on the latent variable z output from the encoder, we can reconstruct x with the inverse function x = fg (wr; z). The flow-based decoder adopts the main backbone architecture of Glow (Kingma & Dhariwal 2018). where each step of flow consists of the same three types of elementary flows actnorm, invertible x convolution and coupling (details in Appendix
For example, the latent codes of the images from CIFAR-10 10 corpus with size 32 32 are represented by 16 feature maps of size 8 in Kingma al. (016)) Chen 201 7ab.
4

---

Published as a conference paper at ICLR 2021
Coupling layer
Coupling layer (C)
split A
Actnorm
KM
spin
split B
Coupling layer
step
XLL
Coupling layer (A)
split C
squeeee
Invertible conv
split D
Actnorm
(a) One step of our flow (b) Four types of coupling layer splits (c) Fine aeniigr multi-scale architecture
Figure 3: The refined architecture of Glow that used in decoder. (a) The architecture of one re-organized step. (b) The visualization of four split patterns for coupling layers, where the red color denotes va and the blue color denotes 2y. (c) The fine-grained version of multi-scale architecture.
Conditional Inputs in Affine Coupling Layers. To incorporate = as conditional input to the decoder, we modify the neural networks for the scale and bias terms:
Tas Tb
sxlit(x)
Ya
Ta
=
s(Ea, 2) O Ib + b(xa,
y concat yb).
where s() and b() take both and input. Specifically, each coupling layer includes three convolution layers where the first and last convolutions are 3, while the center convolution is 1x 1. ELU (Clevert al. 2015) is used as the activation function throughout the flow architecture:
Convax3 ELU + Convixt @ FC(22) + ELU +
Convax3
where FC() refers to linear full-connected layer and @ is addition operation per channel between 2D image and a ID vector.
Importantly, is fed as conditional input to every coupling layers, unlike previous work (Agrawal Dukkipati| 2016 Morrow & Chiu 2019) where a is only used to learn the mean and variance of the underlying Gaussian of v. This design is inspired by the generator in Style-GAN (Karras et al.) 2019), where the style-vector is added to each block of the generator. We conduct experiments to show the importance of this architectural design (see (.II..
Refined Architecture of Glow. In this work, we refine the organization of these three elementary flows in one step (see Figure Ba) reduce the total number of invertible The reason is that the cost and the numerical stability of computing or differentiating the determinant of the weight matrix becomes the practical bottleneck when the channel dimension c is considerably large for high-resolution images. To reduce the number of invertible 1 x 1 convolution flows while maintaining the permutation effect along the channel dimension, we use four split patterns for the split() function in (see Figure 3bp. The splits perform on the channel dimension with continuous and alternate patterns, respectively. For each pattern of the split, we alternate C. and 4h. Coupling layers with different split types alternate in step of flow, illustrated in Figure 3a We further replace the original multi-scale architecture with the fine-grained multi-scale architecture (Figure 3c)) proposed Ma al. (2019a), with the same value of M 4. Experimental improvements over Glow demonstrate the effectiveness of our refined architecture ($3.
convolution flows.
2.4 TACKLING THE TWO PORBLEMS IN VAES AND GENERATIVE FLOWS
Resolving Local Dependency in Generative Flows with Global Information from z. As dis- cussed in 1. the flow-based decoder suffers the limitation of expressiveness and local dependency.

---

Published as a conference paper at ICLR 2021
Table Density estimation performance on four benchmark datasets. Results are reported in bits/dim.
CIFAR-10 8-bit
ImageNet 8-bit
LSUN-bedroom
CelebA-HQ 5-bit
Model
5-bit
8-bit
8-bit
Autoregressive models
IAF VAE (Kingma et al. 2016
3.11
PixeIRNN (Oord al. 2016
3.00
3.63
MAE (Ma al. 2019b
2.95
PixeICNN+ [Sahmans cl al.
2017
2.92
PixeISNAIL (Chen al. 2017b
2.85
SPN (Menick Kalchbrenner 2019
3.52
0.61
Flow-basee models
Real NVP (Dinh el ul.
2016
3.49
3.98
Glow Kingma & Dhariwal 2018
3.35
3.81
1,20
1.03
Glow: relined
3.33
3.77
1.19
1.98
1.02
1.99
Flow++ (Ho et al.
2019
3.29
Residual Flow (Chen al.
2019
3.28
3.76
0.99
MaCow cl al.
2019a
3.28
3.75
1.16
0.95
Our model
3.27
3.72
1.14
1.92
0.97
1.97
In the VAE framework of our model, the latent codes provides the decoder with the imperative global information, which is essential to resolve the limitation of expressiveness due to local depen- dency. On the other hand, the flow-base decoder favors store local dependencies, encouraging the encoder to extract global information that is complementary to it.
Resolving Posterior Collapse in VAEs with Flow-based Decoders. As discussed in previous possible reason for posterior collapse in VAEs is that the decoder model is sufficiently expressive such that completely ignores latent variables 2, and a solution to posterior collapse is to limit the capacity of the decoder. This suggests generative flow ideal model for the decoder since they are insufficiently powerful to trigger the posterior collapse problem.
work (Chen et al. 2017a),
Architectural Inductive Biases for Decoupled Representation Learning. From the high-level view of our model, we utilize these complementary properties of the architectures of the encoder and decoder as inductive bias to attempt to decouple the global and local information of an image by storing them in separate representations. The (indirect) supervision of learning global latent representation z comes from two sources of architectural inductive bias. First, the compression architecture, which takes high-dimensional image input and outputs -indemensional vector, encourages the encoder to discard local dependencies of the image. Second, the preference of the flow-based decoder for capturing local dependencies reinforces global information modeling of the encoder, since the all the information of the input image æ needs to be preserved by and v.
EXPERIMENTS
To evaluate our generative model, we conduct two groups experiments on four benchmark datasets that are commonly used to evaluate deep generative models: CIFAR-10 (Krizhevsky & Hinton 2009). 64 x 64 downsampled version ImageNet (Oord et al. 2016). the bedroom category in LSUN Yuu et al. 2015) and the CelebA-HO dataset Karras et al. 2018 Unlike previous studies which performed experiments on 5-bit images from the LSUN and CelebA-HQ datasets, all the samples from the four datasets are 8-bit images in experiments. All the models are trained by using affine coupling layers and uniform dequantization (Uria et al.) 2013). Additional details on datasets, model architectures, and results of the conducted experiments are provided in Appendix
3.1 GENERATIVE MODELING
We begin our experiments with an evaluation on the performance of generative modeling, leaving the experiments of evaluating the quality of the decoupled global and local representations $3.2
For LUN datasets, 128 128 downsampled version, and for CelehA-HQ we 256 256 version.

---

Published as a conference paper at ICLR 2021
Model
FID
PixelCNNi
65.93
PixelIQNi
49.46
DCGANE
37.11
WGAN-GPE
29.30
EBM
40.58
NCSN
25.32
Glow
46.90
Glow: refined
46.50
Residual Flow
46.37
Our model
37.52
Table 2: FID scores CIFAR-10.
Figure 4: 8-bit CelebA-HQ samples with temperature 0.7.
The baseline model we compare with is the refined Glow model, which is the exact architecture used in our flow-based decoder, except the conditional input z. Thus, the comparison with this baseline illustrates the effect of the decoupled representations on image generation. For the refined Glow model, we adjust the number of steps each level so that there are similar numbers of coupling layers and parameters with the original Glow model for fair comparison.
Density Estimation. Table 1 provides the negative log-likelihood scores in bits/dim (BPD) on the four benchmark datasets, along with the top-performing autoregressive models and flow-based generative models. For comprehensive comparison, LSUN and CelebA-HQ datasets with additive coupling layers. Our refined Glow model obtains better performance than the original the refined architecture. The proposed generative model achieves soate-of-the-aee BPD on all the four standard benchmarks in the non-autoregressive category, except the 5-bit CelebA-HQ dataset.
report results on 5-bit images from the
in Kingma & Dhariwal] 2018), demonstrating the effectiveness of
Sample Quality For quantitative evaluation of sample quality, report the Fréchet Inception Distance (FID) (Heusel et al. 2017) on CIFAR-10 in Table 2 Results marked with and $ are taken from Ostrovski al. (2018) and Heusel al. (2017), respectively. Table 2 also provides scores of two energy-based models, EBM (Du & Mordatch 20 19) and NCSN (Song & Ermon 2019). We that our model obtains better FID scores than all the other explicit density models. In particular, the improvement over the refined Glow model on FID score demonstrates that learning decoupled representations is also helpful for realistic image synthesis.
Qualitatively, Figure Ssoucesssocasss showcases some random samples for 8-bit CelebA-HQ 256 x 256 at tempera- ture 0.7. More image samples, including samples on other datasets, are provided in Appendix
Effect of feeding z every coupling layer. As mentioned in 22.3 we feed latent codes to every coupling layer in the flow-based decoder. To investigate the importance of this design, we perform experiments CIFAR- 10 to compare our model with the baseline (Agrawal
Table 3: BPD and FID score.
Model
BPD
FID
Baseline
3.31
43.34
model where z is only used in the underlying Gaussian & Dukkipptt] 2016 Morrow & Chiul 2019). Table gives the performance on BPD and FID score. Our model outperforms the baseline both the two metrics, demonstrating the effectiveness of this design in our decoder.
Ours
3.27
37.52
3.2 DECOUPLED REPRESENTATION LEARNING
Image Classification As discussed above, good latent representation need to capture global features that characterize the entire image, and disentangle the underlying causal factors. From this perspective, follow the widely adopted downstream linear evaluation protocol (Oord et al.) 2018 Hjelm et al, 2019) to train linear classifier for image classification on the learned representations using all available training labels. The classification accuracy is measure of the linear separability,

---

Published as a conference paper at ICLR 2021
Model
Acc.
Raw pixel
35.32
AAET
37.76
VAET
39.59
BiGANT
44.90
Deep InfoMax
49.62
Our 59.53
Our (e)
17.16
Table 4: Classification accuracy. Figure 5: 2-dimensional linear interpolation between real images.
which is commonly used as a proxy for disentanglement and mutual information between repre- sentations and class labels. We perform linear classification on CIFAR- 10 using a support vector machine (SVM). Table Table44lish lists the classification accuracy of SVM on the representations of 2 and v. together with AAE (Makhzani al., 2015), VAE (Kingma & Welling 2014). BiGAN (Donahue et al. 2017) and Deep InfoMax (Hjelm et al. 2019). Results marked with are token from Hjelm et al. (2019). Raw pixel is the baseline that directly training classifier on the raw pixels of image. The classification accuracy on the representation 2 is significantly better than that on indicating that captures more global information, while v captures more local dependencies. Moreover, the accuracy of = outperforms Deep InfoMax, which is one of the state-of -frthe unsupervised representation learning methods via mutual information maximization.
Two-dimensional Interpolation Our generative model leads to the two-dimensional interpolation, where we linearly interpolate the two latent spaces and v between two real images:
h(2)) h(v)
(1 (1- 2 + 0.22
8)0 Buz
where a, 8 E [0, 1]. 21, U1 and 22, U/ are the global and local representations of images Tj and æ2, respectively. Figure shows interpolation example from CelebA-HQ, where the images on the left top and right bottom corners are the real image The switch operation two special cases of the vio-dimensional interpolation with (a 1, 3 0) and (a 0, B 1). More examples of interpolation and switch operation are provided in Appendix D
Discussion The results on image classification and wwo-dimensional interpolation empirical demon- strate that model relive the posterior collapse problem in VAEs, by learning meaningful infor- mation the latent space. Due to the entirely unsupervised setting, however, there are no intuitive explanations theoretical guarantees for what information captured by the global and local representations, respectively. It is an interesting direction for future work to explore how to learn more interpretable representations by investigating connections architectural inductive biases, or leveraging weak or distant supervision.
4 RELATED WORK
Combination of VAEs and Generative Flows. In the literature of combining VAEs and generative flows, one direction of research is to use generative flows as an inference machine in variational inference for continuous latent variable models (Kingma et al. 2016 Van Den Berg al. 2018). Another direction is to incorporate generative flows in the VAE framework as trainable component, such the prior (Chen al. 2017ab or the decoder (Agrawal Dukkipati| 2016 Morrow Chiu 2019 Mahajan et al. 2019). Recently, two contemporaneous work (Huang et al.) 2020: Chen et al. 2020 explore the idea constructing an invertible flow-based model on an augmented input space by augmenting the original data with an additional random variable. The main difference between these work and ours is the purpose of introducing the latent variables and using generative flows. In
For each column, ranges in (0.0.0.25, 0.5, 0.75, 1.0]; while for each raw, B ranges in [0.0, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0)
8

---

Published as a conference paper at ICLR 2021
Huang et al. (2020); Chen et al. (022)) the latent variables are utilized to augment the input with extra dimensions to improve the expressivenss of the bijective mapping in generative flows. Our generative model, on the other hand, aims to learn representations with decoupled information, and the design of the latent variables and the flow-based decoder architecture is to accomplish this goal. The proposed generative model is closely related with generative flows with picecewise invertible transformations, such as RAD (Dinh et al.) (2019) and CIFs (Cornish et al. 2020). and can be regarded as infinite mixtures of flows (Papamakarios et 2019).
Disentangled Representation Learning. Disentanglement learning (Bengio et al., 2013 Mathieu al., 2016) recently becomes popular topic in representation learning. Creating representations
where each dimension is independent and corresponds to particular attribute have been explored in several approaches, including VAE variants (Alemi et al. 2017; Higgins et al.) 2017: Kim & Mnih
2018 Chen et al. 2018; Mathieu et al. 2019), adversarial training (Mathieu et al. [2016] Karras et al.
2019) and mutual information muximmzatiionrreuuaazzaua
(Chen et al., 2016 Hjelm et al. [019]]
Sanchez et al. 2019h. Of particular relevance to this work are approaches that explore disentangtemenn
in the context of VAEs, which aim to achieve independence or generalized decomposition (Mathieu al. 2016) of the latent space. Different from these work which attempted to learn factorial
representations for disentanglement, we aim to learn two separate representations to decouple the global and local information.
Neural Style Transfer: From the visualization, the switch operation of our model is also (empiri- cally) related to neural style transfer of natural images (Gatys et al.) 2015 Johnson et al. 2016 Ding et al. 2019) the global and local representations in our model appear to correspond to the style and content representations in neural style transfer models. The main difference is that the global and local representations in model are learned in unsupervised manner, while the content and style representations in neural style transfer models are usually extracted from a pre- teeirrrined classification network (VGG network) (Simonyan & Zisserman 2014). It is interesting direction of future work to further investigate the relation between our learned decoupled representations and the content and style representations in neural style transfer models.
5 CONCLUSION
In this paper, we propose a simple and effective generative model that embeds generative flow as decoder in the VAE framework. Simple as it appears be, our model is capable of automat- ically decoupling global and local representations of images in an entirely unsupervised setting. Experimental results on standard image benchmarks demonstrate the effectiveness our model on generative modeling and representation learning. Importantly, decoupled representation learning via the plain likelihood based generation, using only architectural inductive biases. Morcover, the voodimensional interpolation supported by model, with the switch operation as a special case, is an important step towards controllable image manipulation.
demonstrate the feasibility of
ACKNOWLEDGMENTS
The authors would also like thank Xiangyu Yue and Chunting Zhou for their helpful discussions during drafting this paper.
REFERENCES
Siddharth Agrawal and Ambedkar Dukkipati. Deep variational inference without pixel-wise recon- struction. arXiv preprint /6/ 2016.
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Leurning Representations (ICLR), 2017.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8), 2013.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. arXiv preprint
2015.

---

Published as a conference paper at ICLR 2021
Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation. arXiv preprint arXiv:2002.09741, 2020.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders, In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Tian Qi Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, pp. 9913-9923, 2019.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp- 2172-2180, 2016.
Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. In Proceedings of the Sth International Conference Learning Representations (ICLR-2017), Toulon, France, April 2017a.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autore- gressive generative model. arXiv preprint arXiv: 171.00771 2017b.
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 1eXiv:15110072811 2015.
Rob Cornish, Anthony L Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijectivity constraints with continuously indexed normalising flows. International Conference on Machine Learning, 2020.
Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1xXiv:1605.08800 2016.
Laurent Dinh, Jascha Sohl-Dickstein, Razvan Pascanu, and Hugo Larochelle. A rad approach to deep mixture models. arXiv preprint arXiv:1903.07714, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. In Proceedings of the 5th International Conference on Learning Representations, Toulon, France, April 2017.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Advances in Neural Information Processing Systems, pp. 3603-3613, 2019.
Leon A Gatys, Alexander Ecker, and Matthias Bethge. A neural algorithm of artistic style. arXiv preprint
2015.
Ian Goodfellow, Jean Pbudet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems (NIPS-20144 pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
10

---

Published as a conference paper at ICLR 2021
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. bet:-vae: Learning basic visual concepts with a constrained variational framework. In Proceedings the 5th International Conference Learning Representations (ICI.R-2017), Toulon, France, April 2017.
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations (ICLR), 2019.
Karan Grewal, Phil Bachman, Adam
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference Machine Learning, pp.
2019.
Chin-Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. arXiv preprint arXiv:2002.07101, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Yongcheng Jing, Yezhou Yang, Zunlei Feng, Jingwen Ye, Yizhou Yu, and Mingli Song. Neural style transfer: review. IEEE transactions on visualization and computer graphics, 2019.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. International Conference on Learning Representations (ICLR), 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 4396-4405. IEEE, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412,6900 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. In Proceedings of the 2th International Conference on Learning Representations (ICLR-2014), Banff, Canada, April 2014.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pp. 4743-475 2016.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems, pp. 10236-10245. 2018.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Xuezhe Ma, Xiang Kong, Shanghang Zhang, and Eduard Hovy. Macow: Masked convolutional generative flow. Advances in Neural Information Processing Systems 33. 2019a.
Xuezhe Ma, Chunting Zhou, and Eduard Hovy. MAE: Mutual variational autoencoders. In International Conference on Learning Representations (ICLR), 2019b.
regularization for
Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. Flowseq: Non- autoregressive conditional sequence generation with generative flow. In Proceedings of EMNLP- 2019, Hong Kong, November 2019c.
Shweta Mahajan, Iryna Gurevych, and Stefan Roth. Latent normalizing flows for many-to-man cross-domain mappings. In International Conference Learning Representations, 2019.
11

---

Published as a conference paper at ICLR 2021
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 2015.
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. In International Conference on Machine Learning, pp. 4402-4412, 2019.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann. and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in neural information processing systems, pp. 5040-5048, 2016.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In International Conference on Learning Representations (ICLR), 2019.
Rogan Morrow and Wei-Chen Chiu. Variational autoencoders with normalizing flow decoders. OpenReview, 2019.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of International Conference on Machine Learning (ICML-2016), 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.03748, 2018.
Georg Ostrovski. Will Dabney, and Remi Munos. Autoregressive quantile networks for generative modeling. In International Conference on Machine Learning, pp. 3936-3945, 2018.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02262, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.05770 2015.
Danilo Jimenez. Rezende, Shakir Mohamed, and Daan Wienstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning (ICMI,2004)) pp. 1288-1286, Bejing, China, 22-24 Jun 2014.
Tim Salimans, Andrej Karpathy, Xi Chen, Diederik P Kingma, and Yaroslav Bulatov. Pixelcnn++: A pixelcnn implementation with discretized logistic mixture likelihood and other modifications. In International Conference Learning Representations (ICLR), 2017.
Eduardo Hugo Sanchez, Mathicu Serrurier, and Mathias Ortner. Learning disentangled representations via mutual information estimation. arXiv preprint arXiv: 19,.00999, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895-11907, 2019.
Benigno Uria, lain Murray, and Hugo Larochelle. Rnade: The real-valued neural autoregressive density-estimator. In Advances in Neural Information Processing Systems, pp. 2175 2183, 2013.
Rianne Van Den Berg, Leonard Hasenclever, Jakub M Tomczak, and Max Welling. Sylvester normalizing flows for variational inference. In 34th Conference Uncertainty in Artificial Intelligence 2018, UAI 2018, pp. 393-402, 2018.
Martin Wainwright, Michael I Jordan, al. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2):1-305, 2008
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint /50iv:/506.003 033665 2015.
Chunting Zhou, Xuezhe Ma. Di Wang. and Graham Neubig. Density matching for bilingual word embedding. In Proceedings of NAACL-2019, pp. 1588-1598, Minneapolis, Minnesota, 2019. Association for Computational L Linguistics.
12

---

Published as a conference paper at ICLR 2021
APPENDIX: DECOUPLING GLOBAL AND LOCAL REPRESENTATIONS VIA INVERTIBLE GENERATIVE FLOWS
A PRELIMINARY INTRODUCTION OF GLOW
Flow-based generative models focus on certain types of transformations fe that allow (i) the inverse functions ge and Jacobian determinants to be tractable and efficient compute and (ii) fe to be expressive. Most work within this line of research is dedicated to designing invertible transformations to enhance the expressiveness while maintaining the computational efficiency (Kingma & Dhari wal 2018: Ma et al., 2019a; ZZhuu et al. 2019 Chen et al.] 2019 Ho et al.) 2019), among which Glow (Kingma & Dhariwal 2018) has stood out for its simplicity and effectiveness. The following briefly describes the three types of transformations that comprise Glow, which (in refined version)
is adopted as the backbone architecture of the flow-based decoder in our generative model (detailed in Appendix B
Actnorm. Kingma & Dhariwal (2018) proposed an activation normalization layer (Actnorm) as an alternative for batch normalization (loffe Szegedy 2015) to alleviate the challenges in model training. Similar to batch normalization, Actnorm performs an affine transformation of the activations using a scale and bias parameter per channel for 2D images, such that
Vij + vi..j + b,
where both z and y in shape [A c] with spatial dimensions (h, w) and channel dimension Invertible x 1 convolution. To incorporate permutation along the channel dimension, Glow includes a trainable invertible 1 convolution layer generalize the permutation operation as:
yi, Waijj.
where Wi is the weight matrix with shape x
Affine Coupling Layers. Following Dinh et al. (016)), Glow includes affine coupling layers in its architecture of:
Fa Fb
split(x)
Ya Tg
(3)
Vb
s(x,) O xb + b(za)
y concat (ya: 9b),
where s(xaa) and b(x..) are outputs of two neural networks with va input. The split() and concat () functions perform operations along the channel dimension.
B IMPLEMENTATION DETAILS
B.1 COMPRESSION ENCODER
The encoder first compresses the input image size [h x h x c) to the low-resolution tensor of size 4 x 4 x c'. Then, with a fully-connected layer, the encoder transforms the output tensor to a vector of dimension dz. Concretely, to compress the high-resolution images to low-resolution tensors, the encoder consists of levels of ResNet blocks (He et al. 2016). At each level, there are two ResNet blocks with the same number of hidden units and strides I and 2, respectively, Thus, after each level the input is compressed to half of the spatial dimensions: from h h to h 2. ELU (Clevert et al. 2015) is used as the activation function throughout the encoder architecture.
B.2 SCALE TERM IN AFFINE COUPLING LAYERS
To model the scale term s in 77). a straight-forward way is to take the output of the neural network as the logarithm of 8. Formally, let denote as the output from the neural network described in (8. Then we can compute by taking the exponential function: = exp(u). In practice, however, we found this formulation leads to numerical issues in model training. In our implementation, we calculate in the following way:
tanh "
where the constant a E (0, 1). In this formulation, we restrict & in the range of [1 [1-a, 1 + a]. For ImageNet, we set a 0.5 while for other datasets we used a 1.0. In the experiments, we found this formulation not only improved the numerical stability but also achieved better performance on density estimation and FIID scores.
13

---

Published as a conference paper at ICLR 2021
B.3 PRIOR DISTRIBUTION VAES
In this work, the prior distribution p=(z) in VAE is modeled with a generative flow with architecture similar Glow. The generative flow also consists of three elementary invertible transformations: actnorm, invertible linear layer and affine coupling layer. The actnorm and invertible linear layer similar those in Ma et al.) (201c)), with the difference that we did not use the multi-head mechanism. The affine coupling layer is similar to the one in Glow, which applies the split function across the dimension d2. The neural networks for scale and bias terms in affine coupling layers are implemented with multi-layer perceptrons (MLP).
C EXPERIMENTAL DETAILS
C.I PREPROCESSING
We used random horizontal flipping for CIFARIO, and CelebA-HQ 256. For CIFAR-10, we also used random cropping after reflection padding with 4 pixels. For LSUN 128, we first centre cropped the original image, then downsampled to size 128 x 128.
C.2 OPTIMIZATION
Parameter optimization performed with the Adam optimizer (Kingma & Ba 2014) with (0.9, 0.999) and = Te 8. Warmup training is applied to all the experiments: the learning rate linearly increases to the initial learning rate Ie - Then we use exponential decay to decrease the learning rate with decay rate is 0.999997.
C.3 HYPER-PARAMETER
Table 5: Hyper-paramette parameters in our experiments.
Dataset
batch size
latent dim d₂
weight decay
# updates of warmup
CIFAR-10, 10, 32 x 32
512
64
le
50
ImageNet, 64 x 64
256
128
5e
200
LSUN, 128 x 128
256
256
5e
200
HelebA-HQ. 256 256
40
256
5e
200
C.4 COMPARISON OF MODEL. SIZE AND TRAINING SPEED
Table 6 provides the number of parameters of different models on CIFAR-10, 10. together with the corresponding training time over one epoch (measured on four Tesla V100 GPUs). With similar model size, our refined Glow model obtains better performance and faster speed, demonstrating the effectiveness and efficiency of the refined architecture. Due to introducing the encoder, our VAE-based model contains little bit more parameters and the training speed is little slower than refined Glow. Same as training a standard VAE model, ELBO is used as the training objective.
Table Model size and training speed on CIFAR10.
# params
time/epoch (s)
bits/dim
Glow
46.2 M
210.2
3.35
Glow: refined
46.5 M
120.9
3.33
Ours
51.8 M
144.8
3.27
Similar to VAE-based models. exact inference is not preserved and negative log-likelihood (NLL) is approximated with ELBO in model evaluation. However, since EBLO provides way to estimate the upper bound of NLI., the scores Table are comparable to previous work.
14

---

Published as a conference paper at ICLR 2021
D EXAMPLES FOR TWO-DIMENSIONAL INTERPOLATION
Figure 6: Interpolation operation between samples from bit, 256x256 CelcbA-HQ.
15

---

Published as a conference paper at ICLR 2021
E
MORE SAMPLES FOR SWITCH OPERATION
E.1 CELERA-HQ
Ce
N
Figure Switch operation between samples from 8-bit, 256x256 CelebA-HQ.
16

---

Published as a conference paper at ICLR 2021
E.2 CIFAR- 10 & IMAGENET
Figure 8: Switch operation between samples within the same class from 8-bit, 32x32 CIFAR-10.
17

---

Published as a conference paper at ICLR 2021
Figure 9: Switch operation between samples across different classes from 8-bit, 32x32 CIFAR-10.
Figure 10: Switch operation between samples from 8-bit, 64x66 64 imagenet.
18

---

Published as a conference paper at ICLR 2021
E.3 SSUN-BEDROOM
Figure 11: Switch operation between samples from 8-bit, 128 128 LSUN bedroom.
19

---

Published as a conference paper at ICLR 2021
F MORE IMAGE SAMPLES
F.1 CELEBA-HQ
Figure 12: Samples from 8-bit, 256x256 CelebA-HQ with temperature 0.7.
20

---

Published as a conference paper at ICLR 2021
Figure 13: Samples from 8-bit, 256x256 CelebA-HQ with temperature 1.0.
21

---

Published as a conference paper at ICLR 2021
F.2 -SUN-BEDROOM
Figure 14: Samples from 8-bit, 128x 128 LSUN bedrooms.
22

---

Published as a conference paper at ICLR 2021
F.3 CIFAR-10 10 & IMAGENET
Figure 15: Samples from 8-bit, 32x32 CIFAR-10.
Figure 16: Samples from b-ti 64x64 imagenet.
23


Reviews:
[{"review_id": "yGVRNhSZWC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Summary\n\nThis paper aims to improve a Normalizing Flow generative model, in particular\nGlow, by conditioning the flow on global information of the image in the form\nof a latent vector learned with the VAE framework. This latent vector is\ninjected at the scale and bias terms of the affine coupling layers of the flow\n(inspired by what Style-GAN does at the batchnorm layers). \n\nUnfortunately, critical aspects of the method remain unclear or unspecified. To\nthe best of my understanding, the paper lacks a clear explanation of the\ncomplete pipeline used for training the method and the final objective\nfunction. For evaluation, sampling and likelihood computation, procedures are\nnot completely specified either.\n\n##### Pros\n- The general ideas of the paper are well motivated. Combining the advantages of\n  explicit likelihood and latent variable generative models is in my opinion an\n  extremely interesting research direction.\n- The authors propose architectural improvements to Glow and craft a conditional\n  version that can effectively incorporate additional information to the flow.\n- The authors model achieves improved or competitive density estimation,\n  sampling, and downstream performance across various datasets, compared to the\n  state-of-the-art.\n- Some degree of local and global properties disentanglement is demonstrated in\n  the qualitative results, showing the proposed direction is a promising one in\n  that regard.\n\n#####  Cons\nThe main drawback is in my view the presentation of the method. The method part\nof the paper (Section 2) mixes theoretical justification with architecture\ndetails and fails to clearly explain the full pipeline and training\nobjective. This made it very difficult if not impossible to analyze it and draw\nconclusions. The second half of the paper is dedicated to experimental results,\nyet the sampling procedure and likelihood computation are not clearly explained\neither.\n\nI think the submission would have been much stronger if the authors clearly\nexplained the whole method and dedicated more of the paper to a careful analysis\nand justification of the design decisions and of some of the claims\n(e.g. avoiding posterior collapse, global/local disentanglement).\n\n \n\nGoing into detail, by reading the abstract I get the impression that the VAE\nframework is used, and the normalizing flow is used to model the generative\ndistribution $p(x|z)$. Yet the abstract also claims to only use a plain\nlog-likelihood objective as in explicit likelihood models, instead of the VAE\nELBO. Alternatively, I thought the latent code was learned with a separate VAE,\nbut the introduction states that the generative flow is \"[embedded] in the VAE\nframework to model the decoder\".\n\nAssuming that the VAE framework is used with a (conditional) normalizing flow\nfor decoder, Section 2 introduces more confusion when the authors state \"we feed\n$z$ as a conditional input to a flow-based decoder, which transforms $x$ into\nthe representation $\\nu$ with the same dimension.\" This is confusing since\ntypically the decoder input should be a low-dimensional representation, but here\nit seems to be the image as well, so the concept of decoding seems ill\nplaced. Moreover, if this is the case, what would be the point of the\nreconstruction term in the ELBO if invertibility guarantees perfect\nreconstruction? Shouldn't the flow output $\\nu$ be a stochastic latent code as\nin VAEs? Is a prior density regularization imposed on $\\nu$ as well?\n\nFinally, another option would be that everything is trained with the negative\nlog-likelihood cost of the normalizing flow. This would be consistent with the\nclaim in the abstract that only a plain log-likelihood is utilized. But in that\ncase, what is the justification for using a stochastic encoder if it is not\nregularized? How can it be guaranteed that $z$ will not be ignored? What is the\nrole of $z$ during sampling? Does the likelihood computation involve $z$ or just\n$\\nu$?\n\nI apologize for writing my internal thought process but I also wanted to convey\nthat even if the method was clarified to be some of the options I described, or\nanother one, many of the design decisions would still require extra justification\nand analysis.\n\n\nOther comments:\n- Although Figure 1 shows capturing of some global color properties, looking at\n  the interpolations in Figure 5, there seems to be little variability\n  w.r.t. $z$, so maybe the claims of learning \"decoupled\" and \"disentangled\"\n  representations require more justification.\n\n- About the initialization of the weights of the last linear layer with zeroes\n  (Section 2.1). Wouldn't this create a null output at thus a null backward\n  gradient in the first iteration? Even if a non-zero bias was used, wouldn't an\n  unbreakable symmetry condition be produced?\n\n\n************\nAfter Rebuttal:\n\nI thank the authors for their multiple clarifications, and apologize for my initial misunderstanding.\nI understand now that the flow model is used to compute $p(x|z)$ as a function of $x$ and $z$. Maybe the \"decoder\" terminology is a bit confusing here, but this is quite a nice idea overall. It would have been nice to see multiple samples from  $p(x|z)$ for a fixed $z$, to evaluate the expressiveness of the model.\n\nI'm raising my score to acceptance.\n\nPS: Some typos remain in the revised version, eg: \"varnishes\", \"tne\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mk2fxx0XJCv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n-> I like the idea of conditional generative flows, where a low-dimensional embedding captures high-level features and a larger embedding latent space captures local representations. \n-> Use a compression encoder to enforce informative low-dimensional embeddings\n-> Good experimental results\n\nCons:\n\n-> Can you really say that the low-dimensional embedding z is capturing GLOBAL variables? To me, global variables in a generative model drives non iid samples, hence correlating samples with each other. For instance, see the paper \n\nDiane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Thirty-Second AAAI Con- ference on Artificial Intelligence, 2018.\n\n-> The author should clearly explain the training cost function. At the end they are combining a flow generative model (typically trained using ML) with an encoder VAE like posterior approximation. I guess they optimize some soft of ELBO, but it is not clear to me yet.\n\nOverall, I like the paper and would like to see it accepted.", "extra_scores": {}, "extra_text": {}}, {"review_id": "g61Lb5RzyKq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose a novel combination of VAEs and Flow models, where the decoder is modelled through a conditional flow taking as input a “local” representation of the size of the input image and a “global” representation output by the encoder. The authors evaluate the proposed method on density estimation, quality of generations and linear probing on a variety of datasets and show improvement over state of the art.\n\nGreat:\n* Conceptually simple method that seems to work quite well in practice, for this class of models. \n* The linear probing experiment is quite convincing in justifying the use of “global” and “local” characterizations of the learned representations. So are the interpolations.\n\nCould be improved:\n* It’s not clear to what extent each of the proposed refinements to Glow (reorganization, different splits, fine-grained multi-scale architecture) improves Glow’s performance. \n\nThe authors propose a novel combination of known methods, evaluate it extensively and show considerable improvements over current state of the art. A clear accept.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9D_EvlFkqq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a mixture of flows with the specific intent to disentangle global and local representations, to improve visual quality of samples. Architectures are borrowed from style-gan literature. The contributions are mainly architectural and empirical. The demonstrates improved visual quality over other normalizing flow methods. \n\nStrengths\nThe paper introduces a straightforward latent variable model for flows which is designed in such a way that training on NLL gives good sample quality, which is measured in FID. Even further, the model also performs quite well on the NLL objective itself. Since the focus of normalizing flow literature has been mainly on NLL, I think this paper is a nice complement to existing literature. \n\nWeaknesses:\n- The paper never puts the equations from the background section together in the final objective. It would be helpful to have an equation representing the final objective with some of the relevant variables (i.e. log p(x) >= ... with variables x, z, and v). \n- The paper does not introduce many novel theory or methods. This is not really a problem, but the paper can be better connected to existing work and clarified in this respect. The model the authors propose is very reminiscent of \"infinite mixtures of flows\" as outlined by (Papamakarios et al. \"Normalizing Flows for Probabilistic Modeling and Inference\" page 32). An example would be \"Continuously Index Flows\" by Cornish et al.. Note their method was introduced with a different intend in a different manner, so I think it would only make the paper better by citing these methods. \n- Conditioning on a context variable in flow layers is not new (see Kingma et al., \"Improved variational inference with inverse autoregressive flow.\"  and Lu et al. \"Structured Output Learning with Conditional Generative Flows.\"). This is not really a problem, but again this should be clarified. \n- How much added computation is required by the encoder model plus FCnet in eq. 7 compared to the Glow-refined model on which the proposed method is based? \n- Perhaps the naming \"compressing encoder\" is not particularly useful. It implies a direct connection to actual image compression, which is as far as I understand not the case. Other than that, this seems like a fairly standard VAE encoder other than the size difference between x and z.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a hybrid VAE-normalizing-flow for extracting local and global representations of images.  While the reviewers found the model itself to be "conceptually simple" and "straightforward", all were convinced by the empirical evaluation that, indeed, interesting representation learning is going on, resulting in a unanimous vote to accept.
Decision: Accept (Poster)