Title: BASISVAE: ORTHOGONAL LATENT SPACE FOR DEEP DISENTANGLED REPRESENTATION

Abstract: The variational autoencoder, one of the generative models, defines the latent space for the data representation, and uses variational inference to infer the posterior probability. Several methods have been devised to disentangle the latent space for controlling the generative model easily. However, due to the excessive constraints, the more disentangled the latent space is, the lower quality the generative model has. A disentangled generative model would allocate a single feature of the generated data to the only single latent variable. In this paper, we propose a method to decompose the latent space into basis, and reconstruct it by linear combination of the latent bases. The proposed model called BasisVAE consists of the encoder that extracts the features of data and estimates the coefficients for linear combination of the latent bases, and the decoder that reconstructs the data with the combined latent bases. In this method, a single latent basis is subject to change in a single generative factor, and relatively invariant to the changes in other factors. It maintains the performance while relaxing the constraint for disentanglement on a basis, as we no longer need to decompose latent space on a standard basis. Experiments on the well-known benchmark datasets of MNIST, 3DFaces and CelebA demonstrate the efficacy of the proposed method, compared to other state-of-the-art methods. The proposed model not only defines the latent space to be separated by the generative factors, but also shows the better quality of the generated and reconstructed images. The disentangled representation is verified with the generated images and the simple classifier trained on the output of the encoder.

1. INTRODUCTION:

The proper choice of data representation is highly correlated with the difficulty of task learning for a given machine learning approach (Higgins et al., 2017; Kim et al., 2018; Kim & Cho, 2018; 2019) . Using a representation appropriate to specific task and data domain can significantly improve the robustness and successful learning of the model (Kim et al., 2018; Kim & Cho, 2018; 2019; Bengio et al., 2013) . In particular, disentangled representation is useful when dealing with data with various features, and can be effective for a large variety of domains and tasks (Bengio et al., 2013; Ridgeway, 2016) . A latent space is disentangled if single latent units are subject to changes in single generative factors, and relatively invariant to changes in other factors (Bengio et al., 2013) . For example, a generative model trained on a dataset of facial images learns independent latent units subject to single independent generative factors such as hair color, gender, and emotion. We define the disentangled representation using equation (1). A change of single generative factor is consistent to the change of single coefficient c i , but not to c j for i = j. z = Σ i c i e i (1) where z ∈ R Z is a latent variable, e i ∈ R Z is a standard unit vector, and c i ∈ R is a coefficient. Disentangled representation can be useful in several machine learning tasks including transfer learning and zero-shot learning (Lake et al., 2017) . Moreover, unlike most representation learning algorithms, disentangled representation can be interpreted because they are consistent with the variability of the data (Dupont, 2018) . The variational autoencoder (VAE) is used to define the latent space by approximating the posterior distribution with approximation as follows (Kingma & Welling, 2013) . log p θ (x) ≥ -D KL [q φ (z|x) p θ (z)] + E q φ (z|x) [log p θ (x|z)]

(2) where D K L is Kullback-Leibler divergence, q φ (z|x) is a posterior distribution inferred by encoder, p θ (z) is a prior distribution, and p(x|z) is a likelihood or decoder. Since the VAEs are powerful to define the latent space, it is often used for disentangled representation learning (Higgins et al., 2017; Dupont, 2018) . However, in most cases, the quality of the generated data is relatively low because of the added constraints to the loss function (Kim & Cho, 2019) . This is because the scale of the latent variable to represent the generative factor drops from R Z to R 1 when the generative model is f :

Z ⊂ R Z → X ⊂ R X ,

where Z and X are the dimensions of the latent space and data, respectively. Several researchers have studied for disentanglement, but the trade-off with performance has not been considered (Higgins et al., 2017; Dupont, 2018; Chen et al., 2016) . In a vanilla VAE, one generative factor changes in the direction of element in a non-standard basis as shown in Fig. 1 . With this result, if the latent space can be decomposed with basis

B = {b 1 , • • • , b n } to denote latent variable z as Σ i c i b i ,

the single generative factor is associated with the single coefficient. Therefore, disentangled representation learning is achieved by the following two constrains:

1. Each coefficient is subject to change in single generative factor, and relatively invariant to the changes in other factors. 2. The generative model is trained to make the basis of the latent space as a standard basis

B 0 = {e 1 , • • • , e n }.

In this paper, we focus on the first constraint to formulate disentangled representation without the second constraint. The rest of this paper is organized as follows. In Section 2, we introduce the research for learning disentangled representation. The work we have done in this paper and the proposed model are presented in Section 3 and the evaluation is discussed in Section 4. Section 5 presents a summary and some future works.

2. RELATED WORKS:

Many studies have been conducted to learn a data representation. It is used on various applications from feature extraction to dimension reduction. Approaches are divided into two categories: conventional methods and deep learning models. Principal component analysis (PCA) or independent component analysis (ICA) are well-known methods to extract features and reduce the size of data (Smith, 2002; Hyvärinen & Oja, 2000) . Dictionary learning develops a set (dictionary) of representative elements from the data such that each datum can be expressed as a weighted sum of the atoms. The elements and weights can be found by minimizing the error with L1 regularization on the weights to enable sparsity (Mairal et al., 2009; Lee et al., 2007; Aharon et al., 2006) . They adopted the methods such as basis on linear algebra that defines the materials and mixes them appropriately to represent the data. In another approach, Kingma and Welling proposed auto-encoding variational Bayes to approximate the posterior distribution (Kingma & Welling, 2013) . Radford et al. showed that the walking in the latent space resulted in semantic changes (Radford et al., 2015) . resentation by using mutual information (Chen et al., 2016) . Higgins et al. introduced an adjustable hyperparameter β that balanced latent channel capacity and independence constraints with reconstruction accuracy (betaVAE) (Higgins et al., 2017) . Dupont improved betaVAE by using a joint distribution of continuous and discrete latent variables (Dupont, 2018) . Deep learning frameworks showed promise in disentangling factors of variation, but there was a degrade in the quality of the generated data due to the trade-off. In this paper, we propose a method to learn disentangled representation while maintaining the quality of the generated data by learning materials and weights for data representation like dictionary learning and disentangling factors like deep learning approach.

3. THE PROPOSED METHOD:

The architecture of a proposed model that constructs disentangled representation (i.e., the association of a single basis element with a single generative factor) with a coefficient of basis element rather than a latent unit is shown in Fig. 2 . Unlike the conventional VAE that outputs the mean and variance of the latent space expressed as a normal distribution, the encoder of BasisVAE outputs the coefficient f (x) = c associated with elements of the basis B. The latent variable z is sampled from the Gaussian distribution

N (M B • f (x), Σ f (x) )

, where operator • means matrix multiplication, Σ f (x) is a variance computed by encoder, and

M B = [b 1 | • • • |b n ]

is a matrix form of bases. The theoretical background, loss function, and algorithms of the proposed model are discussed in detail in the following sections.

3.1. LATENT SPACE DECOMPOSITION:

For the first constraint mentioned in the introduction, it is proved in Theorem 1 that the latent space can be decomposed as a set of single basis elements that are subject to a single generative factor. It is enough to show that the latent variable z in the equation ( 2) can be decomposed into latent variables z 1 , • • • , z n , called latent basis, associated with a single generative factor, not into latent units, and the evidence lower bound (ELBO) is maintained. Let n x be the number of features that data x has and z 1 , • • • , z nx be the corresponding independent latent variables. Theorem 1. Let the latent variable z in ELBO be decomposed into independent latent variables z 1 , • • • , z nx associated with a single generative factor such that p(z) = Π i p(z i ), then the ELBO with respect to z is equal to the average of values of the ELBO with respect to z i . The q φ (z|x) which the expectation value in equation ( 1) with respect to should be modified as the form of q φ (z i |x) i . We prove Lemma 1 in order to prove Theorem 1. Proof of Theorem 1. Since the latent variable z can be decomposed as independent latent variable

EQUATION

EQUATION

As

z 1 , • • • , z nx are independent, log p θ (x) ≥ -D KL [q φ (z 1 |z 2 , • • • , z nx , x) • • • q φ (z nx |x) p θ (z 1 ) • • • p θ (z nx )] (5) + E q φ (z1|z2,••• ,zn x ,x)•••q φ (zn x |x) [log p θ (x|z 1 , • • • , z nx )] log p θ (x) ≥ -D KL [Π nx i q φ (z i |x) Π nx i p θ (z i )] (6) + E Π nx i q φ (zi|x) [log[Π nx i p θ (x|z i )/p nx-1 (x)]] log p θ (x) ≥ -D KL [Π nx i q φ (z i |x) Π nx i p θ (z i )] (7) + E Π nx i q φ (zi|x) [log Π nx i p θ (x|z i )] -(n x -1)p(x)

By Lemma 1, we can separate the expectation as follows:

EQUATION

As a result of equation ( 8), we can say that the first term of RHS is the reconstruction error, and the second term associates the latent space with the data which has i-feature represented as z i .

In the next section, BasisVAE is proposed to maximize the lower bound shown in equation ( 8), with z 1 , • • • , z nx becoming independent. The proof on the case on n >= 3 in Lemma 1 and the derivations from equation ( 5) to ( 6) are more discussed in Appendix C.

3.2. BASISVAE:

We set n x as the number of features existing in the set X of data and latent variable z as linear combination of 3, but the proposed method changes from (a) to (c). It maintains the area responsible for a single generative factor but achieves disentangled representation using coefficient c i . In this method, the model can learn a disentangled representation with coefficients (constraint 1). Besides, it does not have to define the basis of latent space as standard basis (without constraint 2). The direction of the latent basis b i is not limited to two (the latent unit becomes larger or smaller), but is set in all directions in R Z , thus representing the information in various ways. 1 We train the encoder so that c i = 1 and c j = 0 if the input data has i-feature and no j-feature. The latent variable z is sampled from the normal distribution N (M B • f (x), Σ f (x) ) having the linear combination Σ i c i z i as mean, and Σ f (x) as variance, where

z 1 , • • • , z nx . By the assumption, z 1 , • • • , z nx are independent,

M B = [b 1 | • • • |b n ]and f (x) = (c 1 , • • • , c n ).

The decoder is trained to reconstruct the data x with z. Algorithm 1 describes the process of defining the latent space through the encoder and reconstructing the data through the decoder. Three losses are defined to train the Algorithm 1 The process to define the latent space and reconstruct the data 1: Input: Data {x i } N i=1 , encoder q φ , decoder p θ , and basis matrix M B 2: Output: trained encoder q φ , trained decoder p θ , and trained basis matrix M B 3: Initialize q φ , p θ 4: for epochs do 5:

for batches do 6: Sample x from the dataset 7:

c ← q φ (x) 8: Sample z from N (M B • c T , Σ f (x) ) 9:

x ← p θ (z) 10:

Update q φ , p θ , M B with equation ( 12) 11:

end for 12: end for 13: return q φ , p θ , M B latent space in the proposed process: 1) reconstruction loss L recon , 2) inference loss L KL , and 3) basis loss L B as follows.

EQUATION

EQUATION

)

where l is the binary function for measuring the reconstruction error, f (x) is the output of the encoder, and

M B = [b 1 | • • • |b nx ] is the basis matrix. Since the elements in M B have to be indepen- dent, i.e., b i • b j = 0 if i = j, and b i • b i = 1, M T B M

B should be identity matrix I during training. The total loss of the proposed model is as follows.

EQUATION

)

where α, β, and γ are the hyperparameters for balancing between the losses.

4. EXPERIMENTS:



4.1. DATASET AND EXPERIMENTAL SETTINGS:

To verify the performance of the proposed model, we use the MNIST, 3DFaces and CelebA datasets (LeCun et al., 1998; Liu et al., 2015; Paysan et al., 2009) . The CelebA is a dataset with largescale face attributes. We crop the initial 178×218 size to 138×138 and resize them as 128×128.

There are total 202,599 face images and we use 162,769 images as training data and the rest as test data. The pixel values are normalized between 0 and 1. The weights of the model are initialized with the method proposed by Glorot and Bengio (Glorot & Bengio, 2010) . The encoder consists of eight convolutional layers whose filter size is 5×5 with LeakyReLU activation function followed by dropout and batch normalization layer (Maas et al., 2013; Srivastava et al., 2014; Ioffe & Szegedy, 2015) . The decoder is composed of four convolutional layers and 4 deconvolutional layers with ReLU activation function followed by several layers like encoder (Nair & Hinton, 2010) . α, β, and γ are set as 0.0004, 1, and 0.1, respectively. The binary function for measuring the reconstruction error is set as Bojanowski et al. did (Bojanowski et al., 2017) . BasisVAE is trained for 100 epochs with 100 batch size. The optimizer used to train the model is Adam proposed by Kingma and Ba (Kingma & Ba, 2014) .

4.2. GENERATED IMAGES:

To verify the performance of the proposed model, the performance of BasisVAE is compared with the performance of vanilla VAE, betaVAE, and VQ-VAE (van den Oord & Vinyals, 2017), which have the same structure, but different output of encoder to the proposed model, in three aspects: Reconstruction, random generation, and disentanglement. 

4.2.1. RECONSTRUCTION:

We evaluate the reconstruction performance of BasisVAE with MNIST and CelebA datasets. Fig. 4 shows the reconstructed images for the original images. In Table 1 , we show the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) values together with the comparison model for the quantitative evaluation of the performance. The experiment is repeated 10 times to compute the SSIM and PSNR values between the actual images and the generated images by the model trained on CelebA dataset. The results of the t-test show that the performance of the BasisVAE is superior to that of the other models statistically.

4.2.2. RANDOM GENERATION:

The generated data by BasisVAE learned with MNIST and CelebA are illustrated in Fig. 5 . Frechet inception distance is used to evaluate the quality of the generated images (Heusel et al., 2017) as shown in Table 2 . The p-value obtained from the t-test was less than 0.05, indicating a statistically significant difference in performance.

4.2.3. GENERATION FROM BASIS:

We conduct an experiment to verify that the basis learned through BasisVAE has actually influenced the construction of the disentangled representation. BasisVAE generates the images with basis b i by setting the coefficients as c i = 1 and c j for i = j. The feature corresponding to each basis b i is shown on the Figs. 6 and 7 . We also use a 3DFaces dataset as well as CelebA dataset to identify the 

5. CONCLUSION:

In this paper, we have formulated the disentangled representation learning with two constraints. By proving the Theorem, it is shown that the latent space can be decomposed as independent latent variables associated with single generative factor. We have shown that the proposed BasisVAE constructs disentangled representation without the second constraint by constructing the basis of the latent space. Furthermore, BasisVAE outperforms the vanilla VAE and βVAE in both performance and disentanglement. Since our method can be applied to other VAEs by changing the output of the encoder as coefficients for basis element and adding loss L B , we will verify the versatility and validity by applying it to other models. The performance of the proposed model will be evaluated with other well-known benchmark datasets such as CIFAR10, 3DFaces, and ImageNet. In addition, we will achieve the higher quality of the generated data and interpretability of the latent space by constructing disentangled latent space in generative adversarial network. 

C APPENDIX C: MORE DETAILS IN THE PROOF OF LEMMA 1 AND THEOREM 1:

In the Section 3.1, we show the proof of Lemma 1 in the case of n = 2. For more generality, it is necessary to be proved that the Lemma 1 holds in the case of n >= 3. In that case, Let z 2 = (z 3 , z 4 , . . . , z n ), repeat the same verification process, then we can separate the z 3 from the (z 4 , • • • , z n ). If we process it recursively, the Lemma 1 in the case of n >= 3 holds too. As in the proof of Lemma 1, it is enough to prove the Theorem 1 that Theorem 1 holds when n = 2.

The flow of the equations ( 4) to ( 8) except for the equations ( 5) to ( 6) is natural. Equation ( 6) is derived from the (5) with p(x | z 1 , z 2 ) = p(x | z 1 )p(x | z 2 )/p(x), which is equivalent to equation (13). p(x, z 1 , z 2 ) p(z 1 , z 2 ) = p(x, z 1 )p(x, z 2 ) p(x)p(z 1 )p(z 2 )

Since z 1 , • • • , z n are independent, it is drived to (14).

p(x, z 1 , z 2 ) = p(x, z 1 )p(x, z 2 ) p(x)

The LHS of equation ( 14) can be modified as (15) to (18) p(x, z 1 , z 2 ) = p(x, z 1 , z 2 ) p(x, z 1 ) p(x, z 1 ) = p(z 2 | x, z 1 )p(x, z 1 ) = p(z 2 | x)p(x, z 1 ) = p(x, z 1 )p(x, z 2 ) p(x)

Since the LHS of equation ( 14) can be derived to the RHS of equation ( 15), the equation ( 13) is correct, resulting in the correctness of Theorem 1.

D APPENDIX D: THE DISTRIBUTION OF COEFFICIENTS c i:

We show the distribution of c i that the model learns for each data with "female with blond hair", "female with black hair", and "male with black hair" in the Fig. 13 . With comparison of distributions in Fig. 13 (a) and (b), coefficients c 9 and c 10 represent "black hair" and "blond hair", respectively. Furthermore, we can confirm that coefficients c 19 , c 21 , and c 37 represent the generative factors related to the characteristics on gender, comparing Fig. 13 (a) and (c).

:

In R n , as n increases, the number of direction of z ∈ R Z becomes

n .

Figures and Tables:

Figure 1: In the vanilla VAE model, an interpolation experiment results in only changing the single generative factor only for (a) gender and (b) skin color.

Figure 2: The architecture of the proposed model, BasisVAE.

Figure 3: The visualization of (a) the general latent space, (b) the disentangled latent space, and (c) latent space of the proposed model with two coordinates.

and for any z, z = Σ i c i z i so that the set B = {z 1 , • • • , z nx }is the basis of the latent space. For the sake of convenience, let the elements of B be denoted as b1 , • • • , b nx . The output of the encoder is coefficients c 1 , • • • , c nx because,otherwise, the model is not different with vanilla VAE and cannot achieve the disentangled representation. The goal of the previous research is to change the latent space from (a) to (b) in Fig.

Figure 4: (Top) The original images and (bottom) the reconstructed images on (a) MNIST and (b) CelebA datasets. Appendix A shows more generated images.

Figure 5: The generated images of (a) MNIST and (b) CelebA. In the MNIST dataset, the generated data is organized in each row by class.

Figure 6: The images generated from a single basis. The corresponding feature is shown above the image.

Figure 7: The images generated from a single basis. The value of the coefficient is linearly changed along the row. The corresponding characteristics are shown in the left of the images.

Figure 8: The value of the coefficient is linearly changed along the row. The corresponding characteristics are shown in the side of the images. Appendix A shows more generated images.

Figure 9: Randomly generated images with the coefficient c i for b i fixed as 1. According to the basis element, the images reflecting the corresponding feature are generated. Appendix A shows more generated images.

Figure 10: The generated CelebA images. Image blur is less than conventional VAE.

Figure 11: The value of the coefficient is linearly changed along the row. The corresponding characteristics are shown in the side of the images.

Figure 12: Randomly generated images with the coefficient c i for b i fixed as 1. According to the basis element, images reflecting the corresponding feature are generated.

Figure 13: The distribution of c i (1 <= i <= 40) that model learns for each data with (a) "female with blond hair", (b) "female with black hair", and (c) "male with black hair". We have red boxes for coefficients where the difference between distributions in (a) and distributions in (b) and c is apparent.

The results of evaluating the reconstruction performance with SSIM and PSNR.

Table content: <table><tr><td/><td/><td>VAE</td><td>β VAE</td><td>VQ-VAE</td><td>BasisVAE</td></tr><tr><td/><td>Average</td><td>0.7071</td><td>0.6142</td><td>0.7564</td><td>0.7965</td></tr><tr><td>SSIM</td><td colspan="2">Std. dev. 6.0×10 -6</td><td>6.9×10 -6</td><td colspan="2">8.6×10 -6 4.6×10 -6</td></tr><tr><td/><td colspan="4">p-value 2.4×10 -25 1.3×10 -30 2.6×10 -18</td><td>-</td></tr><tr><td/><td>Average</td><td>64.989</td><td>61.512</td><td>66.38</td><td>67.882</td></tr><tr><td>PSNR</td><td>Std. dev.</td><td>0.004</td><td>0.004</td><td>0.001</td><td>0.004</td></tr><tr><td/><td>p-value</td><td>2.1×10 2 6</td><td colspan="2">1.1×10 -32 7.8×10 -23</td><td>-</td></tr></table>

Comparison of image generation quality by FID score. 49×10 -21 1.85×10 -28 5.31×10 -6 -

Table content: <table><tr><td/><td>VAE</td><td>βVAE</td><td>VQ-VAE</td><td>BasisVAE</td></tr><tr><td>Average</td><td>112.883</td><td>168.239</td><td>84.59</td><td>78.449</td></tr><tr><td>Std. dev.</td><td>1.309</td><td>1.964</td><td>7.513</td><td>2.877</td></tr><tr><td colspan="2">p-value 1.</td><td/><td/><td/></tr></table>

Results of classification using the output of the encoder. The logistic regression model for each class is trained to classify the one class. Appendix B shows more details in the numerical results for each attributes.

Table content: <table><tr><td/><td>VAE</td><td>βVAE</td><td colspan="2">VQ-VAE BasisVAE</td></tr><tr><td colspan="3">Average 0.8190 0.8444</td><td>0.8225</td><td>0.8982</td></tr><tr><td colspan="2">Std. dev. 0.015</td><td>0.009</td><td>0.014</td><td>0.005</td></tr><tr><td>p-value</td><td>0.001</td><td colspan="2">0.004 4.1×10 -4</td><td>-</td></tr></table>

Results of classification using the output of the encoder. The logistic regression model for each class is trained to classify the one class.

Table content: <table><tr><td>Class</td><td>VAE</td><td colspan="3">betaVAE VQ-VAE BasisVAE</td></tr><tr><td>5 o clock shadow</td><td>0.8820</td><td>0.8816</td><td>0.8820</td><td>0.9220</td></tr><tr><td>Arched eyebrows</td><td>0.7414</td><td>0.7450</td><td>0.7423</td><td>0.8231</td></tr><tr><td>Attractive</td><td>0.6579</td><td>0.6945</td><td>0.6468</td><td>0.7842</td></tr><tr><td>Bags under eyes</td><td>0.7926</td><td>0.7920</td><td>0.7926</td><td>0.8243</td></tr><tr><td>Bald</td><td>0.9798</td><td>0.9798</td><td>0.9792</td><td>0.9808</td></tr><tr><td>Bangs</td><td>0.8532</td><td>0.8890</td><td>0.8588</td><td>0.9376</td></tr><tr><td>Big lips</td><td>0.8468</td><td>0.8466</td><td>0.8468</td><td>0.8419</td></tr><tr><td>Big nose</td><td>0.7514</td><td>0.7568</td><td>0.7514</td><td>0.8115</td></tr><tr><td>Black hair</td><td>0.7899</td><td>0.8335</td><td>0.7876</td><td>0.8773</td></tr><tr><td>Blond hair</td><td>0.8470</td><td>0.8983</td><td>0.8591</td><td>0.9377</td></tr><tr><td>Blurry</td><td>0.9526</td><td>0.9526</td><td>0.9526</td><td>0.9526</td></tr><tr><td>Brown hair</td><td>0.7586</td><td>0.7614</td><td>0.7586</td><td>0.8304</td></tr><tr><td>Bushy eyebrows</td><td>0.8575</td><td>0.8603</td><td>0.8575</td><td>0.9105</td></tr><tr><td>Chubby</td><td>0.9389</td><td>0.9389</td><td>0.9389</td><td>0.9440</td></tr><tr><td>Double chin</td><td>0.9510</td><td>0.9510</td><td>0.9510</td><td>0.9578</td></tr><tr><td>Eyeglasses</td><td>0.9305</td><td>0.9312</td><td>0.9304</td><td>0.9837</td></tr><tr><td>Goatee</td><td>0.9265</td><td>0.9265</td><td>0.92646</td><td>0.9542</td></tr><tr><td>Gray hair</td><td>0.9513</td><td>0.9517</td><td>0.95131</td><td>0.9702</td></tr><tr><td>Heavy makeup</td><td>0.6430</td><td>0.7586</td><td>0.6796</td><td>0.9037</td></tr><tr><td>High cheeknones</td><td>0.5672</td><td>0.7116</td><td>0.5907</td><td>0.8596</td></tr><tr><td>Male</td><td>0.6522</td><td>0.7720</td><td>0.6689</td><td>0.9780</td></tr><tr><td colspan="2">Mouth slightly open 0.5480</td><td>0.6752</td><td>0.5542</td><td>0.9262</td></tr><tr><td>Mustache</td><td>0.9496</td><td>0.9496</td><td>0.9496</td><td>0.9519</td></tr><tr><td>Narrow eyes</td><td>0.9250</td><td>0.9251</td><td>0.9250</td><td>0.9222</td></tr><tr><td>No beard</td><td>0.8225</td><td>0.8288</td><td>0.8225</td><td>0.9347</td></tr><tr><td>Oval face</td><td>0.7196</td><td>0.7189</td><td>0.7196</td><td>0.7393</td></tr><tr><td>Pale skin</td><td>0.9569</td><td>0.9594</td><td>0.9569</td><td>0.9625</td></tr><tr><td>Pointy nose</td><td>0.7151</td><td>0.7151</td><td>0.7151</td><td>0.7459</td></tr><tr><td>Receding hairline</td><td>0.9281</td><td>0.9277</td><td>0.9281</td><td>0.9339</td></tr><tr><td>Rosy cheeks</td><td>0.9317</td><td>0.9316</td><td>0.9317</td><td>0.9431</td></tr><tr><td>Sideburns</td><td>0.9313</td><td>0.9314</td><td>0.9313</td><td>0.9565</td></tr><tr><td>Smiling</td><td>0.5730</td><td>0.7330</td><td>0.5957</td><td>0.9165</td></tr><tr><td>Straight hair</td><td>0.7942</td><td>0.7947</td><td>0.7942</td><td>0.7989</td></tr><tr><td>Wavy hair</td><td>0.7262</td><td>0.7516</td><td>0.7288</td><td>0.7917</td></tr><tr><td>Wearing earrings</td><td>0.8092</td><td>0.8103</td><td>0.8092</td><td>0.8597</td></tr><tr><td>Wearing hat</td><td>0.9528</td><td>0.9617</td><td>0.9528</td><td>0.9798</td></tr><tr><td>Wearing lipstick</td><td>0.6542</td><td>0.7668</td><td>0.67945</td><td>0.9135</td></tr><tr><td>Wearing necklace</td><td>0.8794</td><td>0.8794</td><td>0.8794</td><td>0.8794</td></tr><tr><td>Wearing necktie</td><td>0.9273</td><td>0.9271</td><td>0.9273</td><td>0.9277</td></tr><tr><td>Young</td><td>0.7465</td><td>0.7541</td><td>0.7464</td><td>0.8588</td></tr><tr><td colspan="5">B APPENDIX B: DETAILED RESULTS TO MEASURE DISENTANGLEMENT</td></tr><tr><td colspan="5">The more detailed classification accuracy, summarized in Table 3 on average, are shown in Table 4</td></tr><tr><td>by generative factors.</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
[updated rating due to supervision of , which was not made clear enough and would require other baseline models] This paper proposes a modification of the usual parameterization of the encoder in VAEs, to more allow representing an embedding through an explicit basis , which will be pushed to be orthogonal (and hence could correspond to a fully factorised disentangled representation). It is however possible for different samples to use different dimensions in the basis if that is beneficial (i.e. x is mapped to , where f(x) = (c_1, ... , c_n) which sums to 1.). This stretches the usual definition of what a “disentangled representation” means, as this disentanglement is usually assumed to be globally consistent, but this is a fair extension. They show that this formulation can be expressed as a different ELBO which can be maximized as for usual VAEs. I found this paper interesting, but I have one clarification that may modify my assessment quite strongly (hence I am tentatively putting it on the accept side). Some implementation details seem missing as well. Otherwise the presentation is fair, there are several results on different datasets which demonstrate the model*s behaviour appropriately. 1. The main question I have, which may be rather trivial, is “are the c_i supervised in any way?”.?When I first read the paper, and looking at the losses in equations 9-11, I thought that this wasn’t the case (also considering this paper is about unsupervised representation learning), but some sentences and figures make this quite unclear: a. In Section 3.2, you say “We train the encoder so that c_i = 1 and c_j = 0 if the input data has i-feature and no j-feature”. Do you? b. How are the features in Figure 6 attached to each b_i??I.e. how was “5_o_clock_shadow” attached to that particular image at the top-left? If the c_i are supervised, this paper is about a completely different type of generative modeling than what it compares against (it would be more comparable to VQ-VAE or other nearest-neighbor conditional density models). 2. There is not enough details about the architecture, hyperparameter and baselines in the current version of the paper. a. What n_x (i.e. dimensionality of the basis) do you use? How does this affect the results? b. How exactly are f(x), Sigma_f(x) parametrized? They mention the architecture of the “encoder” in Section 4.1, but this could be much clearer. c. How do you train M_B? I assume they are just a fixed set of embeddings that are back-propagated through? d. What are the details about the architecture of the baselines, and their hyperparameters? E.g. what is the beta you used for Beta-VAE? 3. The reconstructions seem only partially related to their target inputs (e.g. see Figure 4). This seems to indicate that instead of really reconstructing x, the model chooses to reconstruct “a close-by related 	ilde{x}”, or even perhaps a b_i. This would make it behave closer to VQ-VAE, which explicitly does that. How related are reconstructions/samples to the b_i? 4. Could you show the distribution of c_i that the model learns, and how much they vary for several example images? ?How “peaky” is this distribution for a given image (this feeds into to the previous question as well)??The promise of the proposed model is that different images pick and choose different combinations of b_i, which hopefully one should see reflected in the distributions of c_i per sample, across clusters, or across the whole dataset. 5. What happens when L_B is removed? I.e. what is the effect of removing the constraint on M_B being a basis, and instead allow it to be anything? This seems to make it closer to a continuous approximation to VQ-VAE? 6. Is Equation 10 correct? Should the KL use N(f(x) cdot M_B, Sigma_f(x)), as in equation 9 above? 7. Similarly, in Section 4.2.3, did you mean “c_i = 1 and c_j = 0 for i != j”? If the model happens to be fully unsupervised, I think that these results are quite interesting, and provide a good modification to the usual VAE framework, I find that having access to the M_B basis explicitly could be very valuable. There is still an interesting philosophical discussion to be had about when one would like to obtain a “global basis” for the latent space (i.e. Figure 3 (b)), or when one would prefer more local ones. I can see clear advantages for a non-local basis, in terms of generalisation and compositionality, which your choice (i.e. Figure 3 (c) ) would prohibit. References: [1] VQ-VAE: Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, “Neural Discrete Representation Learning”, https://arxiv.org/abs/1711.00937