Title: GUIDED VARIATIONAL AUTOENCODER FOR DISENTAN-GLEMENT LEARNING

Abstract: We propose an algorithm, guided variational autoencoder (Guided-VAE), that is able to learn a controllable generative model by performing latent representation disentanglement learning. The learning objective is achieved by providing signal to the latent encoding/embedding in VAE without changing its main backbone architecture, hence retaining the desirable properties of the VAE. We design an unsupervised and a supervised strategy in Guided-VAE and observe enhanced modeling and controlling capability over the vanilla VAE. In the unsupervised strategy, we guide the VAE learning by introducing a lightweight decoder that learns latent geometric transformation and principal components; in the supervised strategy, we use an adversarial excitation and inhibition mechanism to encourage the disentanglement of the latent variables. Guided-VAE enjoys its transparency and simplicity for the general representation learning task, as well as disentanglement learning. On a number of experiments for representation learning, improved synthesis/sampling, better disentanglement for classification, and reduced classification errors in meta learning have been observed.

1. INTRODUCTION:

The resurgence of autoencoders (AE) (LeCun, 1987; Bourlard & Kamp, 1988; Hinton & Zemel, 1994) is an important component in the rapid development of modern deep learning (Goodfellow et al., 2016) . Autoencoders have been widely adopted for modeling signals and images (Poultney et al., 2007; Vincent et al., 2010) . Its statistical counterpart, the variational autoencoder (VAE) (Kingma & Welling, 2014) , has led to a recent wave of development in generative modeling due to its two-in-one capability, both representation and statistical learning in a single framework. Another exploding direction in generative modeling includes generative adversarial networks (GAN) Goodfellow et al. (2014) , but GANs focus on the generation process and are not aimed at representation learning (without an encoder at least in its vanilla version).

Compared with classical dimensionality reduction methods like principal component analysis (PCA) (Hotelling, 1933; Jolliffe, 2011) and Laplacian eigenmaps (Belkin & Niyogi, 2003) , VAEs have demonstrated their unprecedented power in modeling high dimensional data of real-world complexity. However, there is still a large room to improve for VAEs to achieve a high quality reconstruction/synthesis. Additionally, it is desirable to make the VAE representation learning more transparent, interpretable, and controllable.

In this paper, we attempt to learn a transparent representation by introducing guidance to the latent variables in a VAE. We design two strategies for our Guided-VAE, an unsupervised version (Fig. 1 .a) and a supervised version (Fig. 1 .b). The main motivation behind Guided-VAE is to encourage the latent representation to be semantically interpretable, while maintaining the integrity of the basic VAE architecture. Guided-VAE is learned in a multi-task learning fashion. The objective is achieved by taking advantage of the modeling flexibility and the large solution space of the VAE under a lightweight target. Thus the two tasks, learning a good VAE and making the latent variables controllable, become companions rather than conflicts.

In unsupervised Guided-VAE, in addition to the standard VAE backbone, we also explicitly force the latent variables to go through a lightweight encoder that learns a deformable PCA. As seen in Fig. 1 .a, two decoders exist, both trying to reconstruct the input data x: Dec main . The main decoder, denoted as Dec main , functions regularly as in the standard VAE (Kingma & Welling, 2014) ; the secondary decoder, denoted as Dec sub , explicitly learns a geometric deformation together with a linear sub-space. In supervised Guided-VAE, we introduce a subtask for the VAE by forcing one latent variable to be discriminative (minimizing the classification error) while making the rest of the latent variable to be adversarially discriminative (maximizing the minimal classification error). This subtask is achieved using an adversarial excitation and inhibition formulation. Similar to the unsupervised Guided-VAE, the training process is carried out in an end-to-end multi-task learning manner. The result is a regular generative model that keeps the original VAE properties intact, while having the specified latent variable semantically meaningful and capable of controlling/synthesizing a specific attribute. We apply Guided-VAE to the data modeling and few-shot learning problems and show favorable results on the MNIST, CelebA, and Omniglot datasets.

The contributions of our work can be summarized as follows:

• We propose a new generative model disentanglement learning method by introducing latent variable guidance to variational autoencoders (VAE). Both unsupervised and supervised versions of Guided-VAE have been developed.

• In unsupervised Guided-VAE, we introduce deformable PCA as a subtask to guide the general VAE learning process, making the latent variables interpretable and controllable.

• In supervised Guided-VAE, we use an adversarial excitation and inhibition mechanism to encourage the disentanglement, informativeness, and controllability of the latent variables.

Guided-VAE is able to keep the attractive properties of the VAE and it is easy to implement. It can be trained in an end-to-end fashion. It significantly improves the controllability of the vanilla VAE and is applicable to a range of problems for generative modeling and representation learning.

2. RELATED WORK:

Related work can be discussed along several directions.

Generative model families such as generative adversarial networks (GAN) (Goodfellow et al., 2014; Arjovsky et al., 2017) and variational autoencoder (VAE) (Kingma & Welling, 2014) have received a tremendous amount of attention lately. Although GAN produces higher quality synthesis than VAE, GAN is missing the encoder part and hence is not directly suited for representation learning. Here, we focus on disentanglement learning by making VAE more controllable and transparent.

Disentanglement learning (Mathieu et al., 2016; Szabó et al., 2018; Hu et al., 2018; Achille & Soatto, 2018; Gonzalez-Garcia et al., 2018; Jha et al., 2018) recently becomes a popular topic in representation learning. Adversarial training has been adopted in approaches such as (Mathieu et al., 2016; Szabó et al., 2018) . Various methods (Peng et al., 2017; Kim & Mnih, 2018; Lin et al., 2019) have imposed constraints/regularizations/supervisions to the latent variables but these existing approaches often involve an architectural change to the VAE backbone and the additional components in these approaches are not provided as secondary decoder for guiding the main encoder. A closely related work is the β-VAE (Higgins et al., 2017) approach in which a balancing term β is introduced to control the capacity and the independence prior. β-TCVAE (Chen et al., 2018) further extends β-VAE by introducing a total correlation term.

From a different angle, principal component analysis (PCA) family (Hotelling, 1933; Jolliffe, 2011; Candès et al., 2011) can also be viewed as representation learning. Connections between robust PCA (Candès et al., 2011) and VAE (Kingma & Welling, 2014) have been observed (Dai et al., 2018) .

Although being a widely adopted method, PCA nevertheless has limited modeling capability due to its linear subspace assumption. To alleviate the strong requirement for the input data being prealigned, RASL (Peng et al., 2012) deals with unaligned data by estimating a hidden transformation to each input. Here, we take the advantage of the transparency of PCA and the modeling power of VAE by developing a sub-encoder (see Fig. 1 .a), deformable PCA, that guides the VAE training process in an integrated end-to-end manner. After training, the sub-encoder can be removed by keeping the main VAE backbone only.

To achieve disentanglement learning in supervised Guided-VAE, we encourage one latent variable to directly correspond to an attribute while making the rest of the variables uncorrelated. This is analogous to the excitation-inhibition mechanism (Yizhar et al., 2011) or the explaining-away (Wellman & Henrion, 1993) phenomena. Existing approaches (Liu et al., 2018; Lin et al., 2019) impose supervision as a conditional model for an image translation task, whereas our supervised Guided-VAE model targets the generic generative modeling task by using an adversarial excitation and inhibition formulation. This is achieved by minimizing the discriminative loss for the desired latent variable while maximizing the minimal classification error for the rest of the variables. Our formulation has connection to the domain-adversarial neural networks (DANN) (Ganin et al., 2016) but the two methods differ in purpose and classification formulation. Supervised Guided-VAE is also related to the adversarial autoencoder approach Makhzani et al. (2016) but the two methods differ in objective, formulation, network structure, and task domain. In (Ilse et al., 2019) , the domain invariant variational autoencoders method (DIVA) differs from ours by enforcing disjoint sectors to explain certain attributes.

Our model also has connections to the deeply-supervised nets (DSN) (Lee et al., 2015) where intermediate supervision is added to a standard CNN classifier. There are also approaches (Engel et al., 2018; Bojanowski et al., 2018) in which latent variables constraints are added but they have different formulations and objectives than Guided-VAE. Recent efforts in fairness disentanglement learning (Creager et al., 2019; Song et al., 2018 ) also bear some similarity but there is still with a large difference in formulation.

3. GUIDED-VAE MODEL:

In this section, we present the main formulations of our Guided-VAE models. The unsupervised Guided-VAE version is presented first, followed by introduction of the supervised version. 

EQUATION

The first term in eq. ( 1) corresponds to a reconstruction loss q φ (z|x) × ||x -f θ (z)|| 2 dz (the first term is the negative of reconstruction loss between input x and reconstruction x) under Gaussian parameterization of the output. The second term in eq. ( 1) refers to the KL divergence between the variational distribution q φ (z|x) and the prior distribution p(z). The training process thus tries to find the optimal (θ, φ) * such that:

(θ,φ) * =arg max θ,φ n i=1 ELBO(θ,φ;xi)=arg max θ,φ n i=1 E q φ (z|x i ) [log(p θ (xi|z))]-DKL(q φ (z|xi)||p(z))

(2)

3.2. UNSUPERVISED GUIDED-VAE:

In our unsupervised Guided-VAE, we introduce a deformable PCA as a secondary decoder to guide the VAE training. An illustration can be seen in Fig. 1 .a. This secondary decoder is called Dec sub . Without loss of generality, we let z = (z def , z cont ). z def decides a deformation/transformation field, e.g. an affine transformation denoted as τ (z def ). z cont determines the content of a sample image for transformation. The PCA model consists of K basis B = (b 1 , ..., b K ). We define a deformable PCA loss as:

L DP CA (φ, B) = n i=1 E q φ (z def ,zcont|xi) [||x i -τ (z def ) • (z cont B T )|| 2 + k,j =k (b T k b j ) 2 , (3)

where • defines a transformation (affine in our experiments) operator decided by τ (z def ) and

k,j =k (b T k b j ) 2 is regarded as the orthogonal loss. A normalization term k (b T k b k -1)

2 can be optionally added to force the basis to be unit vectors. We follow the spirit of the PCA optimization and a general formulation for learning PCA, which can be found in (Candès et al., 2011) . To keep the simplicity of the method we learn a fixed basis function B and one can also adopt a probabilistic PCA model (Tipping & Bishop, 1999) . Thus, learning unsupervised Guided-VAE becomes:

(θ, φ, B) * = arg max θ,φ,B n i=1 ELBO(θ, φ; x i ) -L DP CA (φ, B) .

(4)

3.3. SUPERVISED GUIDED-VAE:

For training data X = (x 1 , ..., x n ), suppose there exists a total of T attributes with ground-truth labels. The t-th attribute, let z = (z t , z rst t ) where z t defines a scalar variable deciding to decide the t-th attribute and z rst t represents remaining latent variables. Let y t (x i ) be the ground-truth label for the t-th attribute of sample x i ; y t (x i ) ∈ {-1, +1}. For each attribute, we use an adversarial excitation and inhibition method with term:

EQUATION

which is a hinge term. This is an excitation process since we want latent variable z t to directly correspond to the attribute label. Notice thesign before the summation since this term will be combined with eq. (1) for maximization.

EQUATION

where C t (z rst t ) refers to classifier making a prediction for the t-th attribute using the remaining latent variables z rst t .

-log p Ct (y = y(x)|z rst t ) is a cross-entropy term for minimizing the classification error in eq. ( 6). This is an inhibition process since we want the remaining variables z rst t as independent as possible to the attribute label.

(θ, φ) * = arg max θ,φ n i=1 ELBO(θ, φ; x i ) + T t=1 [L Excitation (φ, t) + L Inhibition (φ, t)] . (7)

Note that the term L Inhibition (φ, t) within eq. ( 7) for maximization is an adversarial term to make z rst t as uninformative to attribute t as possible, by making the best possible classifier C t to be undiscriminative. The formulation of eq. ( 7) bears certain similarity to that in domain-adversarial neural networks (Ganin et al., 2016) in which the label classification is minimized with the domain classifier being adversarially maximized. Here, however, we respectively encourage and discourage different parts of the features to make the same type of classification.

4. EXPERIMENTS:

In this section, we first present qualitative results demonstrating our proposed unsupervised Guided-VAE (Figure 1a ) capable of disentangling latent embedding in a more favourable way than VAE and previous disentangle methods (Higgins et al., 2017; Dupont, 2018) on MNIST dataset (LeCun et al., 2010) . We also show that our learned latent representation can be later used to improve classification performance. Next, we extend this idea to a supervised guidance approach in an adversarial excitation and inhibition fashion, where a discriminative objective for certain image properties is given (Figure 1b ) on the CelebA dataset (Yang et al., 2015) . Further, we show that our method can be applied to the few-shot classification tasks, which achieves competitive performance on Omniglot dataset proposed by Vinyals et al. (2016) . 

4.1.2. QUANTITATIVE EVALUATION:

For a quantitative evaluation, we first compare the reconstruction error among different models on the MNIST dataset. In this experiment, we set the bottleneck size to 8 in Guided-VAE and use three settings for the deformation/transformation: Rotation, scaling, and both. In Guided-VAE (Rotation) or Guided-VAE (Scaling), we take the first latent variable z 1 to represent the rotation or the scaling information. In Guided-VAE (Rotation and Scaling), we use the first and second latent variables (z 1 and z 2 ) to represent rotation and scaling respectively. As Table 1 shows, our reconstruction loss is on par with vanilla VAE, whereas the previous disentangling method (β-VAE) has higher loss. Our proposed method is able to achieve added disentanglement while not sacrificing reconstruction capability over vanilla VAE.

In addition, we perform classification tasks on latent embeddings of different models. Specifically, for each data point (x, y), we use the pre-trained VAE model to obtain the value of latent variable Moreover, we attempt to validate the effectiveness of disentanglement in Guided-VAE. We follow the same classification tasks above but use different parts of latent variables as input features for the classifier f (•): We may choose the deformation variables z def , the content variables z cont , or the whole latent variables z as the input feature vector. To reach a fair comparison, we keep the same dimensions for the deformation variables z def and the content variables z cont . Table 3 shows that the classification errors on z cont are significantly lower than the ones on z def , which indicates the success of disentanglement since the content variables should determine the class of digits while the deformation variables should be invariant to the class. In addition, when the dimensions of latent variables z are higher, the classification errors on z def increase while the ones on z cont decrease, indicating a better disentanglement between deformation and content. We first present qualitative results on the CelebA dataset by traversing latent variables of attributes. We select three labeled attributes (emotion, gender and color) in the CelebA dataset as supervised guidance objectives. The bottleneck size is set to 16. We use the first three latent variables z 1 , z 2 , z 3 to represent the attribute information and the rest z 4:16 to represent the content information. During evaluation, we choose z t ∈ {z 1 , z 2 , z 3 } while keeping the remaining latent variables z rst t fixed. Then we obtain a set of images through traversing from the image with t-th attribute to the image without t-th attribute (e.g. smiling to non-smiling) and compare them over methods.

Figure 4 shows the traversal results for β-VAE and our Guided-VAE. β-VAE performs decently for the controlled attribute change, but the individual z in β-VAE is not fully entangled or disentangled with the attribute. Guided-VAE has a better disentanglement for latent variables and is able to better isolate the attributes w.r.t. the corresponding latent variables.

In supervised Guided-VAE, we train a classifier to predict the attributes by using the disentangled attribute latent variable z t or the rest of latent variables z rst t as input features. We perform adversarial excitation and inhibition by encouraging the target latent variable to best predict the corresponding t-th attribute and discouraging the rest of the variables for the prediction of that attribute. (left) shows that the classification errors on z t is significantly lower than the ones on z rst t , which indicates the effectiveness of disentanglement during the training procedure. Furthermore, we attempt to validate that the generated images from the supervised Guided-VAE can be actually controlled by the disentangled attribute variables. Thus, we pre-train an external binary classifier for t-th attribute on the CelebA training set and then use this classifier to test the generated images from Guided-VAE. Each test includes 10, 000 generated images randomly sampled on all latent variables except for the particular latent variable z t we decide to control. As Figure 5 (right) shows, we can draw the confidence-z curves of the t-th attribute where z = z t ∈ [-3.0, 3.0]. For the gender and the smile attributes, it can be seen that the corresponding z t is able to enable (z t < -1) and disable (z t > 1) the attribute of the generated image. Besides, for all the attributes, the probability monotonically decreases when z t increases, which shows the controlling ability of the t-th attribute by tuning the corresponding latent variable z t . 4.3 FEW-SHOT LEARNING Previously, we have shown that Guided-VAE can generate images and be used as representation to perform classification task. In this section, we will apply the proposed method to few-shot classification problem. Specifically, we use our adversarial excitation and inhibition method in the Neural Statistician (Edwards & Storkey, 2017) by adding a supervised guidance network after the statistic network. The supervised guidance signal is the label of each input. We also apply the Mixup method (Zhang et al., 2018) in the supervised guidance network. However, we couldn't reproduce exact reported results in the Neural Statistician, which is also indicated in Korshunova et al. (2018) . For comparison, we mainly consider the Matching Nets (Vinyals et al., 2016) and Bruno (Korshunova et al., 2018 ). Yet it cannot outperform Matching Nets, our proposed Guided-VAE reaches equivalent performance as Bruno (discriminative), where a discriminative objective is fine-tuned to maximize the likelihood of correct labels. 

5. ABLATION STUDY:

We conduct a series of ablation experiments to validate our proposed Guided-VAE model.

5.1. GEOMETRIC TRANSFORMATIONS:

In this part, we conduct an experiment by excluding the geometry-guided part from the unsupervised Guided-VAE. In this way, the nudging decoder is just a PCA-like decoder but not a deformable PCA.

The setting of this experiment is exactly same as described in the unsupervised Guided-VAE section.

The bottleneck size of our model is set to 10 of which the first two latent variables z 1 , z 2 represent the rotation and scaling information separately. In the ablation part, we drop off the geometry-guided part so all 10 latent variables are controlled by the PCA-like light decoder. 

5.2. ADVERSARIAL EXCITATION AND INHIBITION:

In this part, we conduct an experiment of using the adversarial excitation method. We design the experiment using the exact same setting described in the supervised Guided-VAE part.

As Figure 7 shows, though the traversal results still show the traversed results on some latent variables. The results from the adversarial excitation method outperforms the results from the discriminative method. While traversing the latent variable controlling the smiling information, the left part (a) also changes in the smiling status but it's controlled by another latent variable. 

6. CONCLUSION:

In this paper we have presented a new representation learning method, guided variational autoencoder (Guided-VAE), for disentanglement learning. Both versions of Guided-VAE utilize lightweight guidance to the latent variables to achieve better controllability and transparency. Improvements on disentanglement, image traversal, and meta-learning over the competing methods are observed. Guided-VAE maintains the backbone of VAE and can be applied to other generative modeling applications.

:



Figures and Tables:

Figure 1: Model architecture for the proposed Guided-VAE algorithms.

results on MNIST dataset by traversing latent variables received affine transformation guiding signal. Here, we applied the Guided-VAE with the bottleneck size of 10 (i.e. the latent variables z ∈ R 10 ). The first latent variable z 1 represents the rotation information and the second latent variable z 2 represents the scaling information. The rest of the latent variables z 3:10 represent the content information. Thus, the latent variables z ∈ R 10 are represented by z = (z def , z cont ) = (z 1:2 , z 3:10 ).

Figure 2: Latent Variables Traversal on MNIST: Comparison of traversal results from vanilla VAE (Kingma & Welling, 2014), β-VAE (Higgins et al., 2017), β-VAE with controlled capacity increase (CCβ-VAE), Joint-VAE (Dupont, 2018) and our Guided-VAE on the MNIST dataset. z1 and z2 in Guided-VAE are controlled.

Figure 3: PCA basis learned by the secondary decoder in unsupervised Guided-VAE.In Figure2, we show traversal results of all latent variables on MNIST dataset for vanilla VAE (Kingma & Welling, 2014), β-VAE(Higgins et al., 2017), JointVAE(Dupont, 2018) and our guided VAE (β-VAE, JointVAE results are adopted from(Dupont, 2018)). While β-VAE cannot generate meaningful disentangled representations, even with controlled capacity increased, JointVAE is able to disentangle class type from continuous factors. Different from previous methods, our Guided-VAE disentangles geometry properties (z 1 and z 2 ) like rotation angle and stroke thickness from the rest content information z 3:10 .In Figure3, we visualize the basis B = (b 1 , ..., b 8 ) in the PCA part of Dec sub . The basis primarily capture the content information.

Figure 4: Traversal of Latent Factors learned on CelebA: Column 1 shows the traversed images from nosmiling to smiling. Column 2 shows the traversed images from male to female. Column 3 transits the color.The first row is from(Higgins et al., 2017) and we follow its figure generation procedure.

Figure 5: (left) Classification error on CelebA training set. (right) Experts (high-performance external classi-fiers for attribute classification) prediction for being negatives on the generated images. We traverse z1 (gender) and z2 (smile) separately to generate images for the classification test. Each latent z is traversed from -3.0 to 3.0 with 0.1 as the stride length.

Figure 6: Ablation study on geometry transformation: The results here are traversed on z1 and z2. Compared to (b), (a) presents little about the rotation and scaling information.

Figure 7: Ablation study on the adversarial excitation and inhibition method for gender: The left part shows the traversed images from the supervised Guided-VAE without adversarial inhibition. The right part shows the traversed images from the supervised Guided-VAE using adversarial excitation and inhibition. Both images are traversed on the latent variable that is supposed to control the gender information.

Reconstruction loss on MNIST digits data. Here z is a d z -dim vector. We then train a linear classifier f (•) on the embedding-label pairs {(z, y)} in order to predict the class of digits. For the Guided-VAE, we disentangle the latent variables z into deformation variables z def and content variables z cont with same dimensions (i.e. d z def = d zcont ) and use affine transformation as τ (z def ). We compare the classification errors of different models under multiple choices of dimensions of the latent variables in Table2. It shows that generally higher dimensional latent variables result in lower classification errors. Our Guided-VAE method compares favourably over vanilla VAE and β-VAE.

Table content: <table><tr><td>Model</td><td>Reconstruction Loss</td></tr><tr><td>VANILLA VAE</td><td>84.4</td></tr><tr><td>β-VAE (BETA=2)</td><td>89.2</td></tr><tr><td>β-VAE (BETA=4)</td><td>100.1</td></tr><tr><td>GUIDED-VAE (ROTATION)</td><td>86.6</td></tr><tr><td>GUIDED-VAE (SCALING)</td><td>85.8</td></tr><tr><td>GUIDED-VAE (ROTATION, SCALING)</td><td>84.3</td></tr><tr><td>z given input image x.</td><td/></tr></table>

Classification error over different methods.

Table content: <table><tr><td>Model</td><td colspan="3">d z = 16 d z = 32 d z = 64</td></tr><tr><td>VANILLA VAE</td><td>2.85%</td><td>2.63%</td><td>2.87%</td></tr><tr><td>β-VAE (β=2)</td><td>4.70%</td><td>5.10%</td><td>5.23%</td></tr><tr><td>GUIDED-VAE (OURS)</td><td>2.17%</td><td>1.51%</td><td>1.42%</td></tr></table>

Classification error on different latent variables. [↑ means higher is better, ↓ means lower is better] Model d zdef d zcont d z z def Error ↑ z cont Error ↓ z Error ↓

Table content: <table><tr><td>GUIDED-VAE</td><td>8</td><td>8</td><td>16</td><td>27.1%</td><td>3.69%</td><td>2.17%</td></tr><tr><td/><td>16</td><td>16</td><td>32</td><td>42.07%</td><td>1.79%</td><td>1.51%</td></tr><tr><td/><td>32</td><td>32</td><td>64</td><td>62.94%</td><td>1.55%</td><td>1.42%</td></tr><tr><td colspan="2">4.2 SUPERVISED GUIDED-VAE</td><td/><td/><td/><td/><td/></tr><tr><td colspan="3">4.2.1 QUALITATIVE EVALUATION</td><td/><td/><td/><td/></tr></table>

Classification accuracy for a few-shot learning task on the Omniglot dataset.

Table content: <table><tr><td>Omniglot</td><td colspan="4">5-way 1-shot 5-way 5-shot 20-way 1-shot 20-way 5-shot</td></tr><tr><td>PIXELS</td><td>41.7%</td><td>63.2%</td><td>26.7%</td><td>42.6%</td></tr><tr><td>BASELINE CLASSIFIER</td><td>80.0%</td><td>95.0%</td><td>69.5%</td><td>89.1%</td></tr><tr><td>MATCHING NETS</td><td>98.1%</td><td>98.9%</td><td>93.8%</td><td>98.5%</td></tr><tr><td>BRUNO</td><td>86.3%</td><td>95.6%</td><td>69.2%</td><td>87.7%</td></tr><tr><td>BRUNO (DISCRIMINATIVE)</td><td>97.1%</td><td>99.4%</td><td>91.3%</td><td>97.8%</td></tr><tr><td>BASELINE</td><td>97.7%</td><td>99.4%</td><td>91.4%</td><td>96.4%</td></tr><tr><td>OURS (DISCRIMINATIVE)</td><td>97.8%</td><td>99.4%</td><td>92.1%</td><td>96.6%</td></tr></table>

Peer review (iclr2020):
# High-level assessment The manuscript proposes two different extensions of variational autoencoders for learning disentangled representations of image data. The first is an unsupervised approach to disentangle factors that deform the generated images from any other factors such as content whereas the second is a fully supervised approach for disentangling a subset of observed binary factors from any other latent factors. From a purely methodological perspective, these could constitute a fair contribution, albeit leaning on the incremental side (e.g. [8-14], also see major point 1 below) and lacking a sound theoretical motivation as presented by the manuscript. Unfortunately, while further pursuing these contributions might be worthwhile, I believe the submission in its current form has important shortcomings regarding: 1. Lack of clarity/reproducibility (see major point 2 below). 2. Insufficiently compelling experimental results (see major point 3 below). Therefore, while I cannot recommend the current version of the manuscript for publication, I would encourage the authors to address or clarify these issues in their rebuttal. # Detailed summary ### Method summary In this paper, the authors propose two extensions of the variational autoencoder (VAE) framework [1] to learn disentangled representations of images. These two extensions operate on different regimes (unsupervised vs supervised) and are presented and evaluated independently of each other in the manuscript. ###### Proposed extension (A) The first extension aims to disentangle generative factors into two groups: (i) those which modify the generated image via affine transformations and (ii) any other factors, including content. Factors within each of those groups might nonetheless be mutually entangled. In a nutshell, the authors accomplish this by incorporating prior knowledge into the model as follows: 1. The latent space is partitioned a priori into two disjoint subsets: (i) a set of “deformation” latent variables and (ii) a set of “content” latent variables. 2. A secondary decoder is introduced to complement the VAE decoder. This decoder generates an image by applying an affine transformation, parametrized as an unspecified function of the “deformation” latent variables, to a linear combination of basis vectors, with weights given by the “content” latent variables. 3. The VAE evidence lower bound (ELBO) is augmented with an additional mean squared error (MSE) reconstruction loss for the output of the secondary decoder described above, as well as a regularizer to encourage its basis vectors, which are learnt alongside the rest of model parameters, to be pairwise orthogonal. ###### Proposed extension (B) The second extension is a fully supervised approach for disentangling a subset of binary factors, whose ground-truth value is assumed known for all training samples, while the remainder latent variables might remain jointly entangled. To this end, the authors: 1. Assign one latent variable to each of the observed factors. 2. Augment the ELBO with additional losses that simultaneously encourage each observed binary factor to be predictable from its corresponding latent variable while being unpredictable from the rest of latent variables. More precisely, each of these losses is itself composed of (i) an “excitation” loss, given by the hinge loss between the ground-truth factor and its corresponding latent variable and (ii) an “inhibition” loss, given by the log-likelihood of an unspecified classifier that is to be trained adversarially alongside the rest of the model to predict the ground-truth factor from the remainder latent variables following an unspecified procedure. ### Experimental results summary ###### Proposed extension (A) The performance of proposed extension (A) is assessed on the MNIST dataset: 1. Qualitatively, by visualizing a small number of latent variable traversals, in comparison with the original VAE [1], the -VAE [2], the CC -VAE [3] and the JointVAE [3]. 2. Quantitatively, by (i) reporting the reconstruction error and (ii) evaluating the classification error to predict digit class with an unspecified linear model using the resulting representations as input, in comparison with the original VAE and the -VAE. 3. The disentanglement of the representations obtained by the proposed approach is measured by comparing the classification error of a model trained on the “deformation” latent variables with that of a model trained on the “content” latent variables, corroborating that class identity is mostly encoded on the latter. 4. An ablation study is performed by comparing two latent space traversals of the original proposed approach with a variant that does not apply an affine transformation in the secondary decoder. ###### Proposed extension (B) The performance of proposed extension (B) is assessed on the CelebA dataset, modelling *gender*, “smile” and “color” as fully observed factors. 1. Qualitative evaluation is again based on visualization of a small number of latent variable traversals, in this case compared to the (unsupervised) -VAE alone. 2. Quantitative evaluation is performed by comparing the classification error of models trained to predict the ground-truth factors *gender* and *smile* from (i) their corresponding latent variable and (ii) the remainder latent variables. 3. Results for few-shot classification using the learned representations on the Omniglot dataset [4] are presented, where results are inconclusive relative to the performance of baseline approaches such as Matching Networks [5]. 4. An ablation study is performed by comparing one latent space traversal of the original proposed approach with a variant that does not use the *inhibition* losses on the *smile* factor. # Major points ### 1) Significance and novelty of the contribution **The manuscript motivates its two main contributions primarily in comparison to fully unsupervised methods for learning disentangled representations, such as the -VAE. However, I do not believe this comparison is adequate for neither of the two proposed extensions, as they differ in scope and more closely related work and relevant baselines have been missed.** The first proposed extension only aims to disentangle a group of factors accounting for deformations that can be represented as an affine transformation from any other generative factors. In particular, this means that: (i) the model uses prior knowledge about the way in which the generative factors affect the data and (ii) the model does not lead to fully disentangled representations, as it only aims to separate latent variables into two groups that might remain themselves entangled. In contrast, unsupervised disentangled representation methods such as the -VAE and its more recent variants (i) aim to discover the factors of variation in the data with no additional implicit biases than those induced by the model architecture and (ii) try to disentangle all generative factors. In this way, I believe this contribution might be more closely related to work that aims to disentangle certain generative factors in specific applications, such as disentangling content from dynamics in temporal data [6, 7] or, more relevantly, disentangling content from style [8] or translation/rotation [9] in image data. Perhaps most importantly, the second proposed extension, which is fully supervised, ought to be related and compared to other methods for learning disentangled representations which also make use of known ground-truth factors when available, such as [10-12], to cite a few, rather than to fully unsupervised approaches such as the -VAE. Moreover, I believe this contribution should be discussed in relation to prior work which: (i) has already studied the effect of making use of ground-truth factors by incorporating supervision during training with an additional loss, in this case closely related to the proposed “excitation” losses but using cross-entropy rather than a hinge term [13] and (ii) has already proposed measuring disentanglement in terms of the difference in predictability of the ground-truth factor from the assigned latent and other latents, in this case relative to a model using the second most predictive latent rather than all other latents jointly [14]. ### 2) Lack of clarity/reproducibility **The manuscript, in its current form, is severely lacking in clarity and would be, in my opinion, extremely difficult to reproduce.** A non-exhaustive list of relevant information that is entirely missing from the manuscript includes: 1. The specific form and parametrization of the affine transformation . 2. What type of classifier is used for the “inhibition” losses, and how exactly is it trained alongside the rest of the model. 3. How are the model hyperparameters (such as ) to be selected, specially in the unsupervised case. 4. What are the model architectures and training procedures for all variants of the proposed approach and all baselines used to obtain the experimental results. 5. How were hyperparameters selected for all variants of the proposed approach and all baselines to obtain the experimental results. 6. What was the effect of random variability due to changes in the seed and other factors. 7. How exactly were the results of Tables 2, 3 and Figure 5 (left) obtained, including what classifier was used, how was it trained and its hyperparameters (if any) selected, on which data was it trained and tested and what was the input to the classifier in the case of GuideVAE (when relevant). 8. Likewise, virtually all necessary details to reproduce the results of Table 4 are missing, including a description of which data splits were used to learn the representations and which were used to train, validate and test the few-shot learning models. ### 3) Insufficiently compelling experimental results **The experimental results do not clearly convey the benefits of the two proposed approaches, due to a combination of insufficient/inappropriate baselines, limited number of benchmark datasets and metrics, inconsistencies/lack of detail in the way results are reported and not accounting for random variability.** 1. The experimental results should be extended to include datasets and metrics commonly used in the relevant literature. In particular, some datasets with known ground-truth factors such as dSprites, Cars3D, SmallNORB or Shapes3D and disentanglement metrics such as MIG, DCI or SAP should be included as part of the experiments (see [15] for a comprehensive example and relevant citations for each of the datasets and metrics). Likewise, inclusion of other commonly used datasets such as 3DChairs or 3DFaces would also be helpful. 2. Important recent variations of the -VAE, such as FactorVAE [16] or -TCVAE [17] should be included in all relevant experiments and results for CC -VAE [3] and JointVAE [3] should not be provided only for Figure 2. 3. Regarding Figure 2, it appears to me that the latent traversals for -VAE, CC -VAE and JointVAE are identical to those in [3, Figure 3]. If this is the case, it should be clearly stated and be done only with permission from the author, otherwise it could constitute an instance of plagiarism. 4. As discussed above, comparing the fully supervised proposed approach to a purely unsupervised approach like -VAE is, in my opinion, inadequate. Instead, I believe more relevant baselines could include some of the approaches in [10-13], among others. Also, tangentially related to the inadequacy of the unsupervised baselines for Section 4.2, I am of the opinion that the latent traversals in Figure 4 are presented in a somewhat misleading manner. At the moment, it creates the impression that the traversals for -VAE ought to be aligned with “gender”, “smile” and “color”, respectively. However, once again, these traversals seem to have been reproduced directly from [2], which reports them instead as “age/gender”, “image saturation” and “skin colour”. Since -VAE arrived at those factors in an unsupervised manner, interpretation of their underlying meaning is necessarily inexact, and Figure 4 should ideally be modified to reflect that and include other fully supervised baselines. 5. The results on Figure 5 (left) seem to have been obtained by training and testing the classifiers on the same data used to learn the (disentangled) representations. Unfortunately, the lack of detail in the description of the experimental setup makes it difficult to assess the correctness of this evaluation procedure. However, I would argue that these classifiers should instead be trained on held-out data, not used for representation learning, in order to assess generalization. The same consideration would apply to the results presented in Tables 2 and 3, which again are hard to assess due to lack of clarity. Also, results for “color” should be included in Figure 5 to avoid potential “cherry-picking”. 6. Additional latent space traversals should be provided in an Appendix, as qualitative evaluation in the current version of the manuscript relies on a very small number of samples. 7. The ablation studies in Section 5 should not be limited to latent space traversals, but include a quantitative evaluation as well. 8. The results in Table 4 should be reported alongside relevant error bars to allow the reader to assess their statistical significance. # Minor points 1. The proposed contributions are motivated in a largely heuristic manner. While in my opinion not being strictly necessary if strong and comprehensive experimental results were provided instead, the manuscript could benefit from accompanying the proposed method(s) by a sound theoretical motivation. 2. The second, fully supervised contribution is somewhat limited in that (i) it can only deal with binary attributes and (ii) it requires the full dataset to be labeled, both of which might be unrealistic requirements in practice. Given that prior work exists that partly circumvents those limitations (e.g. [10-13]) it could be worthwhile to study extensions of the proposed approach in that direction. 3. In its current form, I believe that the manuscript reads almost as two individual smaller papers, corresponding to the unsupervised and supervised parts respectively, rather than one cohesive contribution. This could perhaps be improved by presenting and evaluating both contributions towards a common goal, and exploring variations of the approach using both contributions simultaneously. 4. The submission would benefit from being proof-read for English style and grammar issues. 5. Additional ablation experiments could be interesting, such as showing the performance of the secondary decoder in comparison to that of the primary decoder. # Proposed improvements I would encourage the authors to address the major points listed above by: 1. Including additional related work as context and clarifying the differences in scope with other unsupervised representation learning approaches throughout Sections 1-3. 2. Revising the manuscript to clarify all aspects of the proposed approach and the experimental setup to guarantee reproducibility. 3. Enhance the depth and breadth of the experimental results, making them more directly comparable to prior work and including more relevant/adequate comparison methods. # References [1] Kingma, Diederik P., and Max Welling. *Auto-encoding variational bayes.* In *International Conference on Learning Representations*. 2014. [2] Higgins, Irina, et al. *beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework.* In *International Conference on Learning Representations*. 2017. [3] Dupont, Emilien. *Learning disentangled joint continuous and discrete representations.* In *Advances in Neural Information Processing Systems*. 2018. [4] Lake, Brenden M., Ruslan Salakhutdinov, and Joshua B. Tenenbaum. *Human-level concept learning through probabilistic program induction.* *Science* 350.6266 (2015): 1332-1338. [5] Vinyals, Oriol, et al. *Matching networks for one shot learning.* In *Advances in neural information processing systems*. 2016. [6] Li, Yingzhen, and Stephan Mandt. *Disentangled sequential autoencoder.* In *International Conference on Machine Learning*. 2018. [7] Hsu, Wei-Ning, Yu Zhang, and James Glass. *Unsupervised learning of disentangled and interpretable representations from sequential data.* In *Advances in neural information processing systems*. 2017. [8] Wu, Wayne, et al. *Disentangling Content and Style via Unsupervised Geometry Distillation.* In *International Conference on Learning Representations Workshop (Deep Generative Models for Highly Structured Data)*. 2019. [9] Bepler, Tristan, et al. *Explicitly disentangling image content from translation and rotation with spatial-VAE.* *In Advances in Neural Information Processing Systems*. 2019. [10] Kulkarni, Tejas D., et al. *Deep convolutional inverse graphics network.* In *Advances in Neural Information Processing Systems*. 2015. [11] Narayanaswamy, Siddharth, et al. *Learning disentangled representations with semi-supervised deep generative models.* In *Advances in Neural Information Processing Systems*. 2017. [12] Klys, Jack, Jake Snell, and Richard Zemel. *Learning latent subspaces in variational autoencoders.* In *Advances in Neural Information Processing Systems*. 2018. [13] Locatello, Francesco, et al. *Disentangling factors of variation using few labels.* In *International Conference on Learning Representations Workshop (Learning with Limited Labeled Data)*. 2019. [14] Kumar, Abhishek, Prasanna Sattigeri, and Avinash Balakrishnan. *Variational inference of disentangled latent concepts from unlabeled observations.* In *International Conference on Learning Representations*. 2017. [15] Locatello, Francesco, et al. *Challenging common assumptions in the unsupervised learning of disentangled representations.* In *International Conference on Machine Learning*. 2019. [16] Kim, Hyunjik, and Andriy Mnih. *Disentangling by factorising.* In *International Conference on Machine Learning*. 2018. [17] Chen, Tian Qi, et al. *Isolating sources of disentanglement in variational autoencoders.* In *Advances in Neural Information Processing Systems*. 2018.