CZ-GEM:  A  FRAMEWORK  FOR DISENTANGLED REPRESENTATION LEARNING
Authors: Akash Srivastava
Yamini Bansal
Yukun Ding
Bernhard Egger
Prasanna Sattigeri
Josh Tenenbaum
David D. Cox
Dan Gutfreund
Venue:  (2020)
Keywords: disentangled representation learning
gan
generative model
simulator

Abstract: Learning disentangled representations of  data is one of the central themes in unsupervised learning in general and generative modelling in particular.  In this work,  we tackle a slightly more intricate scenario where the observations are generated from a conditional distribution of some known control variate and some latent noise variate.  To this end, we present a hierarchical model and a training method (CZ-GEM) that leverages some of the recent developments in likelihood-based and likelihood-free generative models.  We show that by formulation, CZ-GEM introduces the right inductive biases that ensure the disentanglement of the control from the noise variables, while also keeping the components of the control variate disentangled. This is achieved without compromising on the quality of the generated samples. Our approach is simple, general, and can be applied both in supervised and unsupervised settings.

Under review as a conference paper at ICLR 2020
CZ-GEM: A FRAMEWORK FOR DISENTANGLED REP- RESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Learning disentangled representations of data is one of the central themes in unsupervised learning in general and generative modeling in particular. this work, we tackle slightly more intricate scenario where the observations are generated from a conditional distribution of some known control variate and some latent noise variate. To this end, present hierarchical model and a training method (CZ-GEM") that leverages some of the recent developments in likelihood- based and likelihood- frree generative models. We show that CZ-GEM introduces the right inductive biases that ensure the disentanglement of the control from the noise variables, while also keeping the components of the control variable disentangled. This is achieved without compromising on the quality of the generated samples. Our approach is simple, general, and can be applied both in the supervised and unsupervised settings.
INTRODUCTION
Consider the following scenario: a hunter-gatherer walking in the African Savannah some 50,000 years ago notices lioness sprinting out of the bush towards her. In a split second, billions of photons reaching her retinas carrying an enormous amount of information: the shade of the lioness' fur, the angle of its tail, the appearance of every bush in her field of view, the mountains in the background and the clouds in the sky. Yet this point there a very small number of attributes which are of importance: the type of the charging animal, its approximate velocity and its location. The rest are just details. The significance of the concept that the world, despite its complexity, can be described by few explanatory factors of variation, while ignoring the small details, cannot be overestimated. In machine learning there is a large body of work aiming to extract low low-dimensional, interpretable representations of complex, often visual, data. Interestingly, many of the works in this area are associated with developing generative models. The intuition is that if a model can generate a good approximation of the data then it must have learned something about its underlying representation. This representation can then be extracted either by directly inverting the generative process (Srivastava al. 2019b) or by extracting intermediate representations of the model itself (Kingma & Welling 20144] Higgins et al. 2017). Clearly, just learning representation, even if it is low-dimeniional, not enough. The reason that while there could be many ways compress the information captured in the data, allowing good enough approximations, there is no reason to a priori assume that such a representation is interpretable and disentangled the sense that by manipulating certain dimensions of the representation one can control attributes of choice, say the pose of face, while keeping other attributes unchanged. The large body of work on learning disentangled representations tackles this problem in several settings, fully supervised, weakly supervised and unsupervised, depending on the available data (Tran et al. 2018 Reed et al. 2014 [ha et al. 2018 Mathieu et al.) 2016 Higgins al. [017)] Chen et al. 2018 Kim & Mnih 2018)
al. 2016
Nguyen- Phuuce et al. 22019 Warayanaswane an unsupervised generative model that can generate samples which approximate the data a high level of accuracy while also giving rise to a disentangled and interpretable representation. In the last decade two main approaches have captured most of the attention; Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs). In their original versions, both GANs (Goodfellow et al.
et al.) (017)) Ideally, we would like to come up with
2014 and VAEs (Kingma & Welling. 2014) were trained in an unsupervised manner and
CCZ-GEM: C7-Generative Model

---

Under review as a conference paper at ICLR 2020
(a) Chair rotation generated by CGAN
(b) Chair rotation generated by cZ- NZ-GEM
Figure 1: Changing the azimuth of chairs in CGAN and CZ-GEM while holding Z constant. Unlike CZ-GEM, C and Z are clearly entangled in CGAN as changing also changes the type chair even though Z is held constant.
gave rise to entangled representations. Over the years, many methods to improve the quality of the generated data as well as the disentanglement of the representations have been suggested (Brock) et al. 2018 Kingma & Dhariwal] 2018 Nguyen-Phuoc et al., 2019 [Jeon et al. 2018). By and large, GANs are better than VAEs the quality of the generated data while VAEs learn better disentangled representations, in particular in the unsupervised setting.
In this paper, we present a framework for disentangling small number of control variables from the rest of the latent space which accounts for all the additional details, while maintaining a high quality of the generated data. We do that by combining VAE and GAN approaches thus enjoying the best of both worlds. The framework is general and works in both the supervised and unsupervised settings. Let us start with the supervised case. We are provided with paired examples (ar, e) where the observation and is control variate. Crucially, there exists a one-to-many map from to the space of observations, and there are other unobserved attributes (or noise) that together completely define For instance, if z were image of single object, controls the orientation of the object relative to the camera and 2 could represent object identity, texture or background.
Our goal is to learn generative model pe (2/cc, 2) that fulfills two criteria:
1. fx Pe (x c, (())(((())) matches the joint distribution fs p(x|c. z)p(c)p(z)dz: If were learning models of images, we would like the generated images to look realistic and match the true conditional distribution p(x(c).
2. The posterior is factorized p(c, x; 0) p(c|x; ()p x; 0): We would like the control variate to be disentangled from the noise. For example, changing the orientation of the object should not change the identity under model.
This problem setup can occur under many situations such as learning approximate models of simula- tors, 3D reconstructions, speaker recognition (from speech), and even real-world data processing in the human brain in the gatherr example above.
We argue that naive implementation of graphical model as shown in Figure (left), (left), e.g. by conditional GAN (Mirza & Osindero 2014). does not satisfy Criterion In this model, when we condition on x, due to d-separation, and could become dependent, unless additional constraints are posed on the model. This effect is demonstrated in Figure a)). To overcome this we split the generative process into two stages by replacing C' with subgraph (C + Y) shown in Figure (center) First, we generate a crude approximation of the data which only takes c into account. The result is blurry average of the data points conditioned on c, Figure (right). We then feed this crude approximation into a GAN-based generative model which adds the rest of the details We call this framework CZ-GEM. The conditioning must be done carefully to make sure that it does not get entangled with y. To that end we rely on architectural choices and normalization techniques from the style transfer literature (Huang & Belongie 2017 The result is a model which generates images of high quality while disentangling e and as can clearly seen in Figure [b). Additionally, the unsupervised setting, when the labels are not available, (C Y) can be realized by 8-VAR a regularized version of VAE which has been shown learn a disentangled representation of its latent variables (Higgins et al.) [2017] 7] Burgess et al.) 20188. In Section 3 we provide implementation details for both the supervised and unsupervised versions.
conditioned
2 in the second stage
We summarize our two main contributions:
Indeed the mapping from Y to X conditioned on can he viewed general form of style transfer.
2

---

Under review as a conference paper at ICLR 2020
c
(Pose)
2
Figure 2: On the left, conditional GAN (CGAN) model. CZ-GEM in the middle replaces node C with subgraph (C + Y) that is trained independently of the rest of the model. This subgraph learns only partially render the observation. As such, Z comes a later stage the rendering pipeline to add details to Y. As an example, consider the rightmost graph where the observation is made up of different types of chairs different poses. Let the pose be controlled by C' and the type (Identity) be explained by Z. Then in step one of Cz-GEM we learn the pose relationship between C and X via the subgraph, giving rise to blurry chair in the correct pose. Once the pose learned, in the second step, the approximate rendering Y is transformed into X by allowing Z to add identity related details to the blurry image.
We break down the architecture to model an intermediate representa- tion that lends itself to interpretability and disentanglement, and then (carefully) use a GAN based approach to add the rest of the details, thus enjoying a superior image generation quality compared VAEs.
Architectural biases:
2. Unsupervised discovery: We show that our model be combined easily with common methods for discovering disentangled representations such as 3 3-VAE to extract and treat them labels generate images that do not compromise on generative quality.
2 PRELIMINARIES
2.1 GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial networks (GAN) (Goddfellow et al. 2014) represent the current state of the art likelihood-free generative modeling. In GANs, a generator network Go is trained to produce samples that can fool discriminator network D., that is in turn trained distinguish samples from the true data distribution p(x) and the generated samples Gø(z) iz (2). Here, Ps is usually a low dimensional easy-to-samply distribution like standard Gaussian. A variety of tricks and techniques need to be employed to solve this min-max optimization problem. For our models, we employ architectural constraints proposed by DC-GAN (Radford al. (0155) that have been widely successful in ensuring training stability and improving generated image quality.
Conditional GANs (CGAN) (Mirza Osindero 2014) adapt the GAN framework for generating class conditional samples by jointly modeling the observations with their class labels. In CGAN, the generator network Gg is fed class labels e to produce fake conditional samples and the discriminator D., is trained to discriminate between the samples from the joint distribution of true conditional and true labels p(zlc)p(c) and the fake conditional and true labels pe (x)c))(c).
While not the main focus of this paper, we present novel information theoretic perspective CGANs. Specifically, we show that CGAN is trained to maximize a lower-bound to the mutual information between the observation and its label while simultaneously minimizing upper-bound to it. We state this formally:
Lemma (Information-theoree interpretation CGAN). Given (æ,c) ~ p(z, c), CGAN learns the distribution po (1)) Go(x: by training a discriminator D.,, approximate the log-ratio the
3

---

Under review as a conference paper at ICLR 2020
true and generated data densities i.e. Du = logp(z, c)/po in num minimizing the following min E po (x)cc)p(())
[ Dw] = min (z, c) E. yee (
max (Ig,0(2, pep( (L (lat)
min 0,0 (x.c) - -Ig (r,c).
where Ig c) is the generative mutual information and q<cl,r, 0) is the posterior under the learned model.
The detailed derivation is provided in Appendix A.1 Notice that at the limit, the model learns exactly
the marginal distribution of and the posterior (((() and the KI. terms vanish.
2.2 VARIATIONAL. AUTOENCODER
Variational autoencoders (VAE) represent class of likelihood-bodd based deep generative models that
have recently been extensively studied and used in representation learning tasks (Higgins et al. 2017) Burgess et al., 2018 Chen et al., 2018). Consider latent variable model where observation X is assumed to be generated from some underlying low-dimensional latent feature space Z. VAE models learn the conditional distribution p(r) 2) using deep neural network (parameterized by 0)
called decoder network. It uses another deep neural network (parameterized by Φ), called encoder model the posterior distribution p(2[z). The encoder and decoder networks are trained using amortized variational inference (Kingma & Welling 2014) to maximizes a variational lower-bound to the evidence likelihood (ELBO). Recently, Higgins et al. (2017) showed that by regularizing the variational posterior approximation of p(=(z) to be close to the prior distribution p(z) in KI-- divergence, the model encouraged learn disentangled representations, I.e. the model learns posterior distribution that is factorized over the dimensions. They call their model 3- We note that information bottleneck based methods for disentangled representation learning, such as 3-VAE, severely compromise the generative quality.
2.3 NORMALIZATION
Naich-Normalization (BN) (Iffe & Szegedy 2015) plays a crucial role in ensuring the stability of
GAN training Radford et (2015). However, as discuss in Section 3 it suitable for our
purposes. Recently, it has been shown that Instance Normalization (IN) Ulyanov et al. (006)) and its variant Adaptive Instance Normalization (Adal))Huanp useful for image generation and stylization. IN normalizes each convolutional channel per training sample, while AdaIN modifies this normalization to function of an additional variable z (usually style in style transfer). The final transformation applied by AdalN is:
Belongie (2017) can be particularly
AdaIN (()) o(x) (- m(()) + +(2)
where ((x) = HW Σn, Tnhwe and (x)) V HW En, (Jntroc - u(x))² + e. 7(2) and ((=) are learned functions of that could be parameterized by neural network, usually fully connected layer.
CZ-GEM
In Section I we provided high level description of our approach. We will now provide detailed description of how the two components of C7-GEM, subgraph C - Y and the conditional generative models (Y, Z) > are implemented and trained in practice. Figure provides an implementation schematic of our proposed framework.
3.1 SUB-GRAPH LEARNING
If C' is known priori then learning the subgraph C + reduces to the regression problem that minimizes | 12 In practice, since observations are images, this subgraph is realized using
4

---

Under review as a conference paper at ICLR 2020
deep ciisspseedcoovoluuton based decoder network and is trained to learn the map between C' and Y. This is similar to the recent work of SSivastavaa et al. (2019b). We emphasize that this network is trained independently of the rest of the model.
Subgraph - Y not only improves disentanglement and inter- pretability in CZ-GEM but also allows for unsupervised discovery of generative factors when C' is not available in the dataset. In such cases, subgraph C + Y is realized
variational autoencoder. This allows us to use any state-of-the-ar unsupervised disentangled representation learning method such as [Higgins et al. (017): Burgess al. (2018); Chen et al. (2018): (Kim & (mih((2118) Chen et al. (2016) to discover disentangled generative control factors. our implementation we use 3- (see Section 2.2). One drawback of these information bottle- neck based methods is that they compromise on the generative quality. This is where the GAN likelihood- free approach in the second stage comes into play. In fact, even if the output of the first stage (i.e. the intermediate image Y in Figure 2) is of very low generative quality, the final image is of high quality since the second stage explicitly adds details using state-of-the-art GAN method. In Section 5 we show how a simple VAE with a very narrow information bottleneck
Convolutional
Networs
ACUIN
Figure 3: Schematic for CZ- GEM
(2-6 dimensions) can be used within CZ-GEM to discover C' in an unsupervised fashion without compromising on generation quality.
3.2 ADDING DETAILS
Vanilla GANs can only model the marginal data distribution i.c. they learn Pa to match ps and in doing so they use the input to the generator (Go) only as source of stochasticity. Therefore we start with a conditional GAN model instead, to preserve the correspondence between Y and X. As shown in section 2.1 this framework trains Gg such that the observation X is maximally explained by the conditioning variable Y. One major deviation from the original model is that the conditioning variable in our case is the same type and dimensionality as the observation. That is, is an image, albeit blurry one. This setup has previously been used by Isola et al. (2017) in the context of image-to to- image translation.
Incorporating Z requires careful implementation due to two challenges. First, trivially adding Z to the input along with Y invokes d-separation and as a result Y and Z can get entangled. Intuitively, Z is adding high level details to the intermediate representation Y. We leverage this insight as an inductive bias, by incorporating Z at higher layers of the network rather than just feeding it as an input to the bottom layer. straightforward implementation of this idea does not work tough. The reason is that BatchNorm uses batch-level statistics to normalize the incoming activations of the previous layer to speed up learning. In practice, mini-batch statistics is used to approximate batch statistics. This adds internal stochasticity to the generator causing it to ignore any externally added noise, such as Z. An elegant solution to resolve this second challenge comes in the form of adaptive instance normalization (see Section 2.3 It not only removes any dependency on the batch-statistics Z in the normalization process itself. For this reason, it has
but also allows for the incorporation previously been used in style transfer tasks (Huang & Belongie (017). We replace all instances of BatchNorm in the generator with Adaptive InstanceNorm. We then introduce to the generative process using equation I. 7(2) and (2) are parameterized as a simple feed-forward network and are applied to each layer of AdaIN in the generator.
4 RELATED WORK
Disentangled representation learning has been widely studied in recent years, both in the supervised and unsupervised settings. In supervised cases, works such as Tran et al. (2018); Reed et al. (2014); Bao et al. (2018); Jha et al. (201 8); Mathieu et al. (2016): Szabó et al. (2017 Isola et al. (017)): Kulkarni et al. 2015): Narayanaswamy et al. (2017) 7) have used the provided labels in the dataset or other form of weak supervision
promote disentanglement in the learned representation. In
201g):Burgess unsupervised learning most methods such as Higgins et al. (2017); Burgess et al. (2018): Chen et al.]
(2018. 2016); Esmaeili et al. (2018): Jeon al. (2018) rely creating an information bottleneck
5

---

Under review as a conference paper at ICLR 2020
to squeeze out representations that statistically independent in its components. Such models often have to sacrifice on the generative capacity of the method in order to learn a factorized latent representation.
Recently, Locatello et al. (2018) has emphasized the use of inductive biases and weak supervision instead of fully unsupervised methods for disentangled representation learning. Nguyen-Phuoo et al. 2019) and Sitzmann et al. (2019) have successfully shown that including inductive biases, respectively an explicit 3D representation, leads to better performance. Their inductive bias comes in the form of learned 3D transformation pipeline. In comparison, CZ-GEM is much simpler and smaller in design and applies to the general setting where the data is determined by control and noise variables, In addition, it can be used in both supervised and unsupervised setting and does not rely on the knowledge of 3D transformations.
Manually disentangled generative models like the 3D morphable model (Blanz & Vetter 1999) have been built for faces. They are powerful in terms of generalization but there a big gap between those synthetic images and real-world face images. In addition, those models are built highly supervised from 3D scans and the approach is limited by the correspondence assumption which does not scale to more complex objects like chairs (Egger et al. (2019). We use a 3D morphable model to generate our synthetic face dataset and show that we can disentangle pose variation from synthetic and real 2D images.
5 EXPERIMENTS
In this section, we provide comprchensive set of quantitative and qualitative results to demonstrate how CZ-GEM is clearly able to not only disentangle from Z in both supervised and unsupervised settings but also ensure that independent components of C stay disentangled after training. Addi- tionally, we show how in unsupervised settings C7-GEM can be used to discover disentangled latent factors when C' is not explicitly provided.
We evaluate CZ-GEM on a variety of image generation tasks which naturally involve observed attributes C and unobserved attributes Z. To that end, we generate three 3D image datasets of faces, chairs, and cars with explicit control variables. Chairs and cars datasets are derived from ShapeNet (Chang et al.) 2015). We sample 100k images from the full yaw variation and a pitch variation of 90 degrees. We used the straight chair subcategory with 1968 different chairs and the sedan subcategory with 559 different cars. We used Blender render the ShapeNet meshes scripted with the Stanford ShapeNet renderer. For faces, we generated 100k images from the Basel Face Model 2017 (Gerig (ct al.) 2018). We sample shape and color (first 50 coefficients), expressions (first 5 coefficients), pose (yaw -90 to 90 degrees uniformly, pitch and roll according to a Gaussian with variance of 5 degrees) and the illumination from the Basel Illumination Prior (Egger et al. 2018). For the generation of the faces dataset, have complete access to C, but we also include unsupervised results on celebA (Liu et al. (015)) with unconstrained real images. All our datasets are built from publicty available data and tools.
the software provided by Kortylewski et al. (2019). For the stated datasets
We use the DCGAN architecture (Radford et al., 2015) for all neural networks involved in all the experiments in this work and provide reference implementation with exact architecture and hyperparameter settings at
5.1 SUPERVISED SETTING
In the supervised setting we compare CZ-GEM to CGAN. We quantitatively compare the two methods to ensure that independent components of C stay disentangled post learning. Furthermore, we qualitatively compare their abilities to disentangle C and Z. And finally, we compare the quality of the samples that the models generate. For chairs and cars, C' contains only the pose variables and all other variations are explained by Z. For faces, C' contains in addition to pose the first 4 principal directions of shape variations.
We start by quantitatively evaluating how well the information about the control variable maintained in the generated image. To that end train deep regression model f(x) to predict the control variate C from the dataset X. We then compute the mean-squared error (MSE) of this regressor on the generated data ||c f(Go I) In Table 1 we report these numbers for both models all

---

Under review as a conference paper at ICLR 2020
Table 1: Disentangled representation learning performance. Lower is better. We report MSE for all the models and the data for reference.
Table 2: Inception score. Higher is better. Incep- tion score of real data is reported for reference.
Dataset
Real data
CGAN
CZ-GEM
Dataset
Real data
CGAN
CZ-GEM
Face Chair
0.010
0.010
0.010
Face 2.49
1.90
2.28
0.027
0.067
0.060
Chair
3.55
2.49
3.50
Car
0.016
0.032
0.037
Car 3.18
2.78
3.41
three datasets. We also include the training error (i.e. the MSE of the regressor the real data) for comparison. The results show that CGAN and c7-GEM are comparable in preserving the label information in the generated data, but as we show below, only CZ-GEM does that while ensuring that C' and Z remain disentangled.
To qualitatively evaluate the level of disentanglement between C' and Z, we vary each individual dimension C over its range while holding Z constant. We plot the generated images for both models on car and chair datasets in Figure variates without changing the identity of the object, whereas CGAN does not. In addition, we find that for CGAN, the noise Z provides little to no control over the identity of the chairs. This is potentially due to the internal stochasticity introduced by the BatchNorm. The last rows for the C7-GEN figures provide the visualization Y. It can be seen how Y is clearly preserving (pose information) but averaging the identity related details.
Notice that cz-GEM allows to vary the control
CZ- 22-GEM car rotation
(b) CZ-GEM car elevation
(c) CZ- CZ-GEM chair rotation
(d) CZ-GEM chair elevation
(c) CGAN car rotation
(f) CGAN car elevation
CGAN chair rotation
(h) CGAN chair elevation
Figure 4: Latent traversal on cars and chairs. Third rows in CZ-GEM results show Y.
We also qualitatively evaluate cZ- CZ-GEM the more challenging faces dataset that includes 10 control variates. As shown Figure in the appendix, Cz-GEM is not only able to model the common pose factors such rotation and azimuth but also accurately captures the principal shape component of
7

---

Under review as a conference paper at ICLR 2020
Basel face model that approximates the width of the forehead, the width of jaw etc. Compared to CGAN, CZ- EZ-GEM does qualitatively better job at keeping the identity constant.
Finally, in order to ensure that our method does compromise the generative quality, evaluate the Inception score (Salimans et al. 2016) on all three datasets. Inception score has been widely used to measure the diversity and the generative quality of GANs. As shown in Table 2 unlike CGAN, cz. Tz-GEMM does not degrade the image quality.
5.2 UNSUPERVISED SETTING
We now test the performance of CZ-GEM in the unsupervised setting, where disentangled components of C' needs to be discovered, using 3-VAE, as part of learning the mapping C' - Y. For our purpose, use simple version of the original 3-VAE method with very narrow bottleneck (6D for faces and 2D for cars and chairs) to extract C.
The latent traversals for the faces dataset presented in Figure 5 Unsupervised discovery is able to recover rotation as well as translation variation present in the dataset. For comparison, we evaluate InfoGAN (Chen et 2016) and present the results in Figure T where it is evident that CZ-GEM clearly outperforms InfoGAN on both disentanglement and generative quality. More traversal results are provided in the appendix. We further test our method on the CelebA dataset (Liu al. 2013). where pose information is not available. This traversal plot is shown in Figure 7 Traversal plots for
cars and chairs dataset are provided in the Appendix Figure 12 and Figure 13
Figure 5: Latent traversal on faces (unsupervised CZ- CE-G The three latent variables capture the rotation, azimuth, and distance respectively.
Figure 6: Latent traversal of InfoGAN on faces. The latent variables are able to capture some pose changes but the pose changes are highly entangled with other pose factors well as the face shape.
6 CONCLUSIONS
We present simple yet effective method of learning representations in deep generative models in the setting where the observation is determined by control variate and noise variate Z. Our method ensures that in the learned representation both C and 7 disentangled as well as the components of C' themselves. This is done without compromising the quality of the generated samples. In future work, would like to explore how this method can be applied to input with multiple objects.
REFERENCES
Mathieu Aubry, Daniel Maturana, Alexei A Efros, Bryan C Russell, and Josef Sivic. Seeing 3d chairs: exemplar part- 2d-3d alignment using a large dataset of cad models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3762-3769, 2014.
8

---

Under review as a conference paper at ICLR 2020
7
T
U
L R
S
Figure 7: Latent traversal CelebA (unsupervised cz GEM). The latent variables consistently capture the azimuth, hair-style, gender and hair color respectively while maintaining good image quality.
Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua. Towards open-set identity preserving face synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6713-6722. 2018.
Voler Blanz and Thomas Vetter. A morphable model for the synthesis of 3D faces. In ACMTOGSIC- GRAPH, pp. 187-194, 1999.
Andrew Brock, Jeff Donahue, and Karen Simonyan, Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, 2018.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 8-vae. arXiv preprint arXiv: (004. 2018.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 1r2iv:151..03012, 2015.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Πlya Sutskever, and Picter Abbeel, Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp- 2172-2180, 2016.
Bernhard Egger, Sandro Schönborn, Andreas Schneider, Adam Kortylewski, Andreas Morel-Forster, Clemens Blumer, and Thomas Vetter. Occlusion-awand aware 3d morphable models and an illumination prior for face image analysis. IJCV, 2018.
Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al. 3d morphable face models-past, present and future. arXiv preprint wwt.ivv:999.0111 2019.
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana Brooks, Jennifer Dy, and Jan Willem van de Meent. Structured disentangled representa- tions. arXiv preprint arXiv:
2018.
Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger, Marcel Lüthi, Sandro Schönborn, and Thomas Vetter. Morphable face models open framework. In FG, pp. 75-82, 2018.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu. David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems, 2014.

---

Under review as a conference paper at ICLR 2020
Michael Gutmann and Aapo Hyvärinen. contt-conte estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. pp. 297-304, 2010.
Irina Higgins. Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR, 2(5):6, 2017.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501-1510, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167 2015.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Insu Jeon, Wonkwang Lee, and Gunhee Kim. Ib-gan: Disentangled representation learning with information bottleneck gan. 2018.
Ananya Harsh Jha, Saket Anand, Maneesh Singh, and VSR Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In European Conference on Computer Vision, pp. 229-845. Springer, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. arXiv preprint 100.03380...5983. 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Adam Kortylewski, Bernhard Egger, Andreas Schneider, Thomas Gerig, Andreas Morel-Forster, and Thomas Vetter. Analyzing and reducing the damage of dataset bias to face recognition with synthetic data. In CVPRW, 2019.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. In Advances in neural information processing systems, pp. 2539-2547, 2015.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. arXiv preprint arXiv:1811.12359) 2018.
Michael F Mathieu, Junbo Jake Zhao, Junbo Zhao, Aditya Ramesh, Pablo Sprechmann, and Yann LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in Neural Information Processing Systems, pp. 5040-5048, 2016.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:1411.1784 2014.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint 1610::6100.03400 2016.
Siddharth Narayanaswamy, Brooks Paige, Jan- Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with semi-supervised deep generative models. In Advances in Neural Information Processing Systems, pp. 5925-5935, 2017.
10

---

Under review as a conference paper at ICLR 2020
Thu Nguyen-Phuose Chuan Li, Lucas Theis, Christian Richardt, and Yong- Yang. Hologan: Unsupervised learning of 3d representations from natural images. arXiv preprint 3XXiv:1904.01220 2019.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: using variational divergence minimization. In Advances in neural information processing systems, pp. 271-279, 2016.
Training generative neural samplers
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 0rXiv:1511.064333 2015.
Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee. Learning disentangle factors of variation with manifold interaction. In International Conference on Machine Learning, pp. 1431- 1439, 2014.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-struuctur-aware neural scene representations. In Advances in Neural Information Processing Systems, 2019.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Information Processing Systems, pp. 3308-3318, 2017.
Akash Srivastava, Kai Xu, Michael U Gutmann, and Charles Sutton. Ratio matching mmd nets: Low dimensional projections for effective deep generative models. arXiv preprint arXiv: 1806.00101, 2018.
Akash Srivastava, Kristjan Greenewald, and Farzaneh Mirzazadeh. Bregmn: scaled- -eeeedbnbemman genera- tive modeling networks. arXiv preprint arXiv: 000///966000333 2019a.
Akash Srivastava, Jessie Rosenberg, Dan Gutfreund, and David D Cox. Simvae: Simulator-assisted training for interpretable generative models. 2019b.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Attila Szabó, Qiyang Hu, Tiziano Portenier, Matthias Zwicker, and Paolo Favaro. Challenges in disentangling independent factors of variation. arXiv preprint 1rX12:771..02245, 2017.
Luan Quoe Tran, Xi Yin, and Xiaoming Liu. Representation learning by rotating your faces. IEEE transactions on pattern analysis and machine intelligence, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arxiv preprint arXi::1607.08022 2016.
A APPENDIX
A.1
CGAN MUTUAL. INFORMATION DERIVATION
Following Sugiyama et al.
Mohamed & Lakshminarayanan (2016) Srivastava et al. (2017) know that optima the logits (D..) of trained discriminator
2012): Gutmann & Hyvärinen 2010
approximate the log-ratio the true data and generated data densities, i.e. Da, log po(z]cpp()
P= (=(cc)p(()
Following Nowozin et al. (016)) Srivastava et al. (2017) 2018 2019a) the generator Go can therefore be trained minimize the following f-divergence between the two sets of densities.
min E Du] min / po(xlc)p(c))
llog p(xlc)p(c) (o(2)c)p(c) d(x, c).
(2)
11

---

Under review as a conference paper at ICLR 2020
The RHS of equation can be re-arranged into terms containing the upper and the lower bounds to the generative mutual information between X and C, i..,
log (x(c) p(z)
log p(c|x)) p(c)
min E
Dw]
min
+ max
min log po (x(x)
+ log po(x))
+
Po (x)
p(x)
log <(c/2, 0)
+ log p(c|x)
max I
P(C)
g(c/20 8)
log po(x]c)
po(x)
min E
+ log
+
(0
p(x)
max 18 log ((c2,, 0) g(c/2, 0)
log
p(c)
p(c|x)
min Z (I,, c) + [e (L
max I, (x, c) - Ep (ax ( 0)1p(((()
min TUB 9.0 [z. IB (z,c).
(3)
A.2 QUANTITATIVE EVALUATION
Apart from the MSE-based estimator reported [ we report and additional evaluation measure. We use the same regressor f(x) trained Ior but we report the Pearson correlation co-efficient (r) between the predicted label and the true label r(e, [(Ga 2))) for each dimension C.
Chairs
CGAN
0.89
0.88
CZ-GEM
0.82
0.89
Cars
CGAN
0.88
0.97
CZ-GEM
0.86
0.96
Faces
CGAN
0.46
0.55
0.41
0.40
0.88
0.91
0.77
0.93
0.87
0.97
CZ-GEM
0.27
0.32
0.31
0.24
0.88
0.87
0.72
0.89
0.78
0.96
A.3 ADDITIONAL EXPERIMENT RESULTS
Comparison of cz- GEM and CGAN on face dataset is shown in Figure 9 CGAN not only produces blurry faces but also shows more undesired identity changes. order to show the shape variation clearly, we provide zoomed-in view in Figure 10
We provide additional results for supervised and unsupervised results on the chair dataset from Aubry al. (2014) in Figure II and Figure 12 respectively. The observation is the same with the previous one. CZ-G ZE-G varies the control variables without changing the shape of chairs. In the first row in visually indistinguishable showing excellent disentanglement
Figure I1 the leg of the chairs between C' and Z. For the results unsupervised setting showing Figure 12 CZ- CZ-GEM is able to disentangle the rotation of chairs without any label.
Additional results of latent traversal of cz- 7Z-GEM in the unsupervised setting is provided in Figure The model is able capture the rotation but the translation is not very smooth.
12

---

Under review as a conference paper at ICLR 2020
(a) CZ-GEM face pose
(b) cz CZ-GEM face shape
Figure 8: Latent traversal of CZ-GEM on faces. The pose variations azimuth, horizontal translation. vertical translation, distance, rotation, and elevation from top to bottom. The shape variations show the difference in face height, forehead, jaw, and from top bottom.
Figure 14 provides the InfoGAN result on the face dataset. Compared with unsupervised CZ-Gre GE-G result in Figure [5] clearly InfoGAN discovers some control variables but the effect is highly entangled.
13

---

Under review as a conference paper at ICLR 2020
(a) CGAN face pose
(b) CGAN face shape
Figure 9: Latent traversal of CGAN on faces. The pose variations are azimuth, horizontal translation, vertical translation, distance, rotation, and elevation from top to bottom. The shape variations show difference in face height, forehead, jaw, and ear from top to bottom.
Figure 10: Zoomed-in comparison face shape. Row I: CGAN forehead variation; Row 2: CGAN jaw variation; Row 3: CZ-GEM forehead variation; Row cZ-GEM jaw variation. Row and Row 3 should have bigger forehead from left to right while Row and Row 4 should have consistent forehead. CGAN and Cz-GEM shows good forehead variation in Row and Row respectively but CZ-GEM (Row does better than CGAN (Row 2) in keeping the forehead the same while another factor is changing.
14

---

Under review as a conference paper at ICLR 2020
Figure 11: Latent traversal on chairs. The first three rows show the effect of the variable in C' that controls the rotation chairs. The last row visualize the corresponding Y.
Figure 12: Latent traversal on chairs (unsupervised CZ-GEM).
Figure 13: Latent traversal of unsupervised CZ-GEM cars.
15

---

Under review as a conference paper at ICLR 2020
Figure 14: Latent traversal InfoGAN on faces dataset.
Figure 15: Latent traversal of unsupervised CZ- CZ-GEM on face dataset.
16


Reviews:
[{"review_id": "ryeEcSeaYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a hybrid technique for rendering “control-variate” and class-conditional image in two steps, first by generating an approximate rendering of the image (“Y”) conditional on the control variate and then filling in the details with a conditional GAN dependent on a latent noise variable Z (although I note that the caption of Figure 2 which identifies “Z” as the identity makes this rather confusing).\n\nTo ensure that Z is used to explain aspects of the model that are separate from the controlled variation, Z is combined in the refinement model at later steps (since otherwise the posterior over Z and Y conditional on X could induce entanglement between the variables).\n\nIn the “supervised” setting where the control variates are observed, Y can be learned as a simple regression problem independent of the other parts of the model, and this two-stage refinement process is demonstrated (using inception scores) to generate convincing samples, including when C consists of up to 10 control variates. In the unsupervised setting, a beta-VAE is used to learn a disentangled representation of X as a proxy for C, but then the data is regenerated using a two step process.\n\nReadability suggestion: the paper starts with a very nice motivating example, but when the setup is provided, i.e., that (x,c) pairs are the input to the learner, the intended content of c is not immediately clear- control variates could assume anything from general context information to privileged information. A similarly informative example would be great!\n\nClarification regarding lemma 1: it seems that if the true posterior cannot be expressed by q, a gap will necessarily remain, even in the “limit” of perfect learning. Is this correct?\n\nOverall: this paper makes a convincing case that it can be used to generate higher quality images, but not that this improves the quality of the disentangled representations. In fact, the separate training seems to make this unlikely.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rJez-cJNcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for learning disentangled representations.  The approach is used on both supervised (where the factors to be disentangled are known) and unsupervised settings. The authors demonstrate the efficacy of their approach in both settings on several datasets with both quantitative and qualitative results.\n\nThis task is an important one. However, I found that the contribution of this paper is fairly small. The proposed approach seems reasonable but it is mostly a work of engineering and provides little insights into the problem nor the proposed model.\n\nThe setup where labeled data (c) also seems a bit unnatural (this also seems to be confirmed by the fact that the authors had to build datasets for the problem). Perhaps the authors could give examples of situations where this would naturally arise. In practice, it seems difficult to obtain these data for all required variables to be disentangled.\n\nThe unsupervised results are more interesting but not very much explored (a single set of sampled faces). I was also curious as to why the learned Y's are blurry. This sort of two-stage generation is also potentially interesting, I was wondering if the authors had ideas to generalize this idea. \n\nI also was not convinced by the experiments which are mostly qualitative. I did not find that this set of experiments provide enough support to the proposed method.\n\n\nDetailed comments:\n- It is a bit unclear to me how the authors propose to obtain independent posteriors over z and c. Is it purely empirical or is there a formal reason that guarantees it?\n\n- Some of the figures your report are compelling but it is a bit unclear to the reader if the results are general (e.g., the examples could have been hand-picked). Are there any quantitative measures you could provide (in addition to Tables 1 and 2 which don't measure the quality of the approach)?\n\n- Comparing to CGAN seems reasonable but given the task at hand, it seems like other methods could have been tried (although I do realize that no one may have done this before for deep generative models).\n\n\n\nOther comments:\n- In Figure 3, it would be good to label the upper trapezoid.\n\n- Some paragraphs are very long and the manuscript may benefit from segmenting them into multiple paragraphs.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkgXCTz-cB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes the use of a hierarchical model for a generative modeling task. They propose a framework of introducing an intermediate latent variable to enforce the independence of the control and noise variable. \nThe paper report extensive experimental results to validate the proposed hierarchical model. \nThe authors also provide the anonymized code to observe the exact implementation in TensorFlow to visualize the latent variable traversals.\n\nComments:\nThe paper proposes the use of a hierarchical model for a generative modeling task by introducing an intermediate latent variable to enforce the independence of the control and noise variable. \nThe paper report extensive experimental results to validate the proposed hierarchical model. \nThis type of framework of crude to fine hierarchical generative model has already been successfully introduced by StackGAN and it's recent variants.\nOn the unsupervised disentangled feature learning, the framework provides incremental advancement by using beta-VAE in conjunction with GAN to use the best of both the worlds. \nEven though the proposed approach is similar to StackGAN, the experiments and the results mentioned in the paper are noteworthy.\n\nQuestions to Authors:\nThere are 2 main claims of novelty made in the paper.\n1. Architectural Biases: \nHow is the approach different in comparison to the StackGAN and it's variable which also use multiple levels of crude to fine image generation?\n2. Unsupervised control variable discovery:\nThis part is just the use of existing disentanglement VAEs to extract the control variables. So how does the paper try to make contributions to improve the disentangled features with the proposed method?\nApart from combining these to existing ideas, what can be considered as an added novelty to improve the quality of the disentangled features?\n\nIn summary, I find there is no novelty involved apart from combining the already existing SOTA model in disentangled feature learning (beta-VAE) and image generation (StackGAN).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of learning disentangled representations in supervised and unsupervised settings. 

In general, the problem of representation learning in of course a core problem in ICLR. However, in the set-up described by the authors, R2 commented on the the set-up for supervised being a bit unnatural in as detailed labels need to be given (somewhat confusingly, the labels are called control variates in the paper).

Several reviewers commented on the novelty of the paper being on the low side, with R2 commenting the contribution being fairly small, and R3 noting similarities to stackgan.

There were also some comments on quality, and clarity. On the topic of technical quality, R2 did note that the authors present extensive results, but R3 mentions that the case for the disentanglement improving is not sufficiently supported. In terms of clarity, there was some initial confusing about e.g. the inference procedure, though the authors addressed these issues in the discussion. 
Decision: Reject