BasisVAE: Orthogonal Latent Space for Deep Disentangled Representation
Authors: Jin-Young  Kim
Sung-Bae Cho
Venue:  (2020)
Keywords: variational autoencoder
latent space
basis
disentangled representation

Abstract: The variational autoencoder, one of the generative models, defines the latent space for the data representation, and uses variational inference to infer the posterior probability. Several methods have been devised to disentangle the latent space for controlling the generative model easily. However, due to the excessive constraints, the more disentangled the latent space is, the lower quality the generative model has. A disentangled generative model would allocate a single feature of the generated data to the only single latent variable. In this paper, we propose a method to decompose the latent space into basis, and reconstruct it by linear combination of the latent bases. The proposed model called BasisVAE consists of the encoder that extracts the features of data and estimates the coefficients for linear combination of the latent bases, and the decoder that reconstructs the data with the combined latent bases. In this method, a single latent basis is subject to change in a single generative factor, and relatively invariant to the changes in other factors. It maintains the performance while relaxing the constraint for disentanglement on a basis, as we no longer need to decompose latent space on a standard basis. Experiments on the well-known benchmark datasets of MNIST, 3DFaces and CelebA demonstrate the efficacy of the proposed method, compared to other state-of-the-art methods. The proposed model not only defines the latent space to be separated by the generative factors, but also shows the better quality of the generated and reconstructed images. The disentangled representation is verified with the generated images and the simple classifier trained on the output of the encoder.

Under review as a conference paper at ICLR 2020
BASIS VAE: ORTHOGONAL LATENT SPACE FOR DEEP DISENTANGLED REPRESENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
The variational autoencoder, of the generative models, defines the latent space for the data representation, and uses variational inference to infer the posterior probability. Several methods have been devised to disentangle the latent space for controlling the generative model easily. However, due to the excessive con- straints, the more disentangled the latent space is, the lower quality the generative model has. A disentangled generative model would allocate single feature of the generated data the only single latent variable. In this paper, we propose a method to decompose the latent space into basis, and reconstruct it by linear combination of the latent bases. The proposed model called Basis VAE consists of the encoder that extracts the features of data and estimates the coefficients for linear combination of the latent bases, and the decoder that reconstructs the data with the combined latent bases. In this method, single latent basis is subject to change in a single generative factor, and relatively invariant to the changes in other factors. It maintains the performance while relaxing the constraint for disentan- glement on a basis, as we no longer need to decompose latent space on a standard basis. Experiments and CelebA demonstrate the efficacy of the proposed method, compared to other state-of-the-ard methods. The proposed model not only defines the latent space to be separated by the generative factors, but also shows the better quality of the generated and reconstructed images. The disentangled representation is verified with the generated images and the simple classifier trained on the output of the encoder.
the well-known benchmark datasets of MNIST, 3DFaces
INTRODUCTION
The proper choice of data representation is highly correlated with the difficulty of task learning for given machine learning approach (Higgins et al. 2017: Kim al., 2018 Kim & Cho 2018
2019). Using a representation appropriate to specific task and data domain can significantly improve the robustness and successful learning of the model (Kim et al., 2018 Kim & Cho 2018 2019
Bengio et al., 2013). In particular, disentangled representation is useful when dealing with data with various features, and can be effective for a large variety of domains and tasks (Bengio et al, 2013 Ridgeway 2016). A latent space is disentangled if single latent units are subject to changes in single generative factors, and relatively invariant to changes in other factors (Bengio al.) 2013). For example, a generative model trained on dataset of facial images learns independent latent units subject to single independent generative factors such as hair color, gender, and emotion. We define the disentangled representation using equation (1). A change of single generative factor is consistent to the change of single coefficient c, but not to Cj for + E,c,e;
(1)
where z E 7 is a latent variable, e; E R.7 is a standard unit vector, and C2 E RR is a coeffi- cient. Disentangled representation be useful in several machine learning tasks including transfer learning and zero-shot learning (Lake et al., 2017). Moreover, unlike most representation learn- ing algorithms, disentangled representation can be interpreted because they are consistent with the variability of the data (Dupont) 2018). The variational autoencoder (VAE) is used to define the la- tent space by approximating the posterior distribution with approximation as follows (Kingma & Welling 2013).
log (or)) 2 -Dkrlga + Ega(sta) (log (log po(x]=)]

---

Under review as a conference paper at ICLR 2020
(a)
(b)
Figure 1: In the vanilla VAE model, an interpolation experiment results in only changing the single generative factor only for (a) gender and (b) skin color.
where Dx L is Kullback-Leible divergence, (2(2))) ) is a posterior distribution inferred by encoder, p=(z) prior distribution, and p(x|z) likelihood or decoder. Since the VAEs are powerful to define the latent space, it is often used for disentangled representation learning (Higgins al., 2017; Dupont, 2018). However, in most cases, the quality of the generated data is relatively low because of the added constraints to the loss function (Kim & Cho 2019). This is because the scale of the latent variable to represent the generative factor drops from R² to IR when the generative model ʃ 2 C IR22 - X' C IR-X, where Z and X are the dimensions of the latent space and data, respectively. Several researchers have studied for disentanglement, but the trade-off with performance has not been considered (Higgins et al., 2017]] Dupont 2018: Chen al.] 2016). In vanilla VAE, one generative factor changes in the direction of element in a non-standard basis as shown in Fig. 1. With this result, if the latent space can be decomposed with basis B {b1,". bn) denote latent variable = as Die; bj the single generative factor is associated with the single coefficient. Therefore, disentangled representation learning is achieved by the following two constrains:
1. Each coefficient is subject to change in single generative factor, and relatively invariant to the changes in other factors.
2. The generative model is trained to make the basis of the latent space as standard basis Bu = [c1, ,e.,,..,,,... enn)
In this paper, we focus on the first constraint to formulate disentangled representation without second constraint. The rest of this paper is organized follows. In Section 2, we introduce the research for learning disentangled representation. The work we have done in this paper and the proposed model are presented in Section 3 and the evaluation is discussed in Section 4. Section presents a summary and some future works.
RELATED WORKS
Many studies have been conducted to learn a data representation. It is used on various applications from feature extraction to dimension reduction. Approaches are divided into two categories: ventional methods and deep learning models. Principal component analysis (PCA) or independent component analysis (ICA) are well-known methods to extract features and reduce the size of data (Smith) 2002; Hyvärinen & Oja 2000). Dictionary learning develops a set (dictionary) of repre- sentative elements from the data such that each datum can be expressed as a weighted sum of the atoms. The elements and weights can be found by minimizing the error with Ll regularization on the weights enable sparsity (Mairal et al. 2009 Lee et al. 2007 Aharon et al. 2006). They adopted the methods such as basis on linear algebra that defines the materials and mixes them appropriately to represent the data. another approach, Kingma and Welling proposed auto-encoding variational Bayes approximate the posterior distribution (Kingma & Welling 2013). Radford et al. showed that the walking in the latent space resulted in semantic changes (Radford et al.] 2015). Oord et al. proposed quaat-qua VAE to learn discrete latent representation (van den Oord & Vinyals) 2017]. It is not disentangled, but somewhat with general representation to prevent posterior collapse (i.e., violation of the first constraint). Chen et al. presented InfoGAN that learned interpretable rep-
2

---

Under review as a conference paper at ICLR 2020
E: encoder
MF
decoder
f() RR;;
coefficients for basis
E
D
x'
Σ5c R variance
Eyrr
*
M3 E RZ basis matrix
(0.1)
Figure The architecture of the proposed model, Basis
resentation by using mutual information (Chen al. 2016). Higgins et al. introduced an adjustable hyperparameter / that balanced latent channel capacity and independence constraints with recon- struction accuracy (betaVAE) (Higgins et al. 2017). Dupont improved betaVAE by using a joint distribution of continuous and discrete latent variables (Dupont 2018). Deep learning frameworks showed promise in disentangling factors of variation, but there was a degrade in the quality of the generated data due to the trade-off. In this paper, propose method to learn disentangled repre- sentation while maintaining the quality of the generated data by learning materials and weights for data representation like dictionary learning and disentangling factors like deep learning approach.
3 THE PROPOSED METHOD
The architecture of proposed model that constructs disentangled representation (i.e., the associ- ation of single basis element with a single generative factor) with a coefficient of basis element rather than latent unit is shown in Fig. 2. Unlike the conventional VAE that outputs the mean and variance of the latent space expressed as a normal distribution, the encoder of Basis VAE outputs sampled
the coefficient fx) c associated with elements of the basis B. The latent variable from the Gaussian distribution N(M fx), E((x)), where operator means matrix multiplication, E (x) is a variance computed by encoder, and MM = b, b [b,] is matrix form of bases. The theoretical background, loss function, and algorithms the proposed model are discussed in detail in the following sections.
3.1 LATENT SPACE DECOMPOSITION
For the first constraint mentioned in the introduction, is proved in Theorem that the latent space can be decomposed as set of single basis elements that are subject to single generative factor. enough to show that the latent variable variables Z1, 2n. called latent basis, associated with a single generative factor, not into latent units, and the evidence lower bound (ELBO) is maintained. Let ny be the number of features that data has and ... Znz be the corresponding independent latent variables. Theorem 1. Let the latent variable z in FLBBO be decomposed into independent latent variables Z1, ... Zn, associated with single generative factor such that p(z) = ]T;((;), then the ELBO with respect to is equal to the average of values of the ELBO with respect to 2₁. The (z(2)) which the expectation value in equation (1) with respect should be modified as the form of (2; We prove Lemma in order to prove Theorem 1. Lemma 1. If ",, in are independent and L is a linear operator, where a; coefficient of 2; in L. Proof. We just show
the equation (2) can be decomposed into latent
E.,....... [L(2). 2,)] (E,,E,, (z,])
in the case of n
E Es,,, [L(2)222]
22) + d22d2
(3)
p(21)a22122
p(22)azzgd22
(212)(2)) a2E2g (22)
L(Ez,121 Eey
3

---

We set nu the number of features existing in the set X' of data and latent variable linear com- bination of Z1, .... Zny. By the assumption, Z1. ... Zn, are independent, and for any z, z = Σ2az so that the set B [zı,... Zn jis the basis of the latent space. For the sake of convenience, let Under review as a conference paper at ICLR 2020
(8)
(b)
The visualization of (a) the general latent space, (b) the disentangled latent space, and (c) latent space of the proposed model with two coordinates.
Figure 3:
Proof of Theorem 1. Since the latent variable be decomposed as independent latent variable 21, ... ang equation (4) is derived from equation (2).
Po 2 -DxL ... ,2n, (=), 2n,
(4)
As 21, . 2n, are independent,
log po (x) 2 -DKLlyo(z11e; .. #)) x). .. (o ( (2)))1)(211 po(
(5)
+ +E, ( Fny go (z) (log (orlpo Rna))
log (x) 2 -D [11 (= pa (2:)
(6)
+ En (a [log[ po (I) 2; pm p===1(xx))
log po (x) 2 kD [I, go (2 >> pe (2:)]
+ us log II; po (x)22)) 2:)] (nx 1)p(z)
By Lemma 1, we can separate the expectation as follows:
1 Logpe(z)) 2 - S> (essst
D (xLlav(2/=))loo<2=))))
(8)
Tlx
As a result of equation (8), we can say that the first term of RHS is the reconstruction error, and the second term associates the latent space with the data which has i-feature represented Zí- In the next section, BasisVAE is proposed to maximize the lower bound shown in equation (8), with ... Zng becoming independent. The proof the case on >= 3 in Lemma I and the derivations from equation (5) to (6) are more discussed in Appendix C.
3.2 BASISVAE
the elements of B be denoted as b1, ... bn.- The output of the encoder is coefficients c,. Cn, because, otherwise, the model is not different with vanilla VAE and cannot achieve the disentangled representation. The goal of the previous research is to change the latent space from (a) to (b) in Fig. but the proposed method changes from (a) (c). It maintains the area responsible for a single generative factor but achieves disentangled representation using coefficient model can learn a disentangled representation with coefficients (constraint 1). Besides, does not have to define the basis of latent space as standard basis (without constraint 2). The direction of the latent basis b; not limited to two (the latent unit becomes larger or smaller), but set in direc- tions in IR thus representing the information in various ways We train the encoder that c; and c = if the input data has i-feature and no j j-feature. The latent variable z is sampled from the normal distribution M(M8 fxx), Ex(x) having the linear combination Σ5e;2; mean, and Ex) variance, where Mg [b, ... [b,,ann jand f(x) = (CI, Cn), The decoder is trained to reconstruct the data x with z. Algorithm describes the process defining the latent space through the encoder and reconstructing the data through the decoder. Three losses defined to train the
In this method, the
Cn).
In I increases, the number of direction of R becomes 2
4

---

Under review as a conference paper at ICLR 2020
Algorithm The process define the latent space and reconstruct the data
1: Input: Data { { encoder decoder ps, and basis matrix Ma
2: Output: trained encoder 9s. trained decoder pe, and trained basis matrix Mg
3: Initialize 46-
4: for epochs do
5: for batches do
6:
7: Sample x from the dataset
(z)
8: Sample z from ((Ms of {())
9:
+ (2
Update qo Po, My with equation (12)
11: end for
end for
13:
return qo. po. Ma
latent space in the proposed process: 1) reconstruction loss Creem 2) inference loss LKL, and 3) basis loss LB as follows.
Drccom 1(x, ((el N(Ms f(t), LKL DKLLM Era)lp(()
(9)
(10)
Cg = M E
(11)
where I is the binary function for measuring the reconstruction error, f(x) is the output of the encoder, and MM [b, ... |b,., is the basis matrix. Since the elements in Ms have be indepen- dent, i.e. b by = 0 if + j. and b; b, Mg.M should identity matrix during training. The total loss of the proposed model is follows.
aCrcco + 3LKL 7LB
(12)
where a, B, and are the hyperparameters for balancing between the losses.
4 EXPERIMENTS
4.1 DATASET AND EXPERIMENTAL SETTINGS
To verify the performance of the proposed model,
the MNIST, 3DFaces and CelebA datasets (LeCun et al., 1998; Liu et al. 2015) Paysan et al. 2009). The CelebA is a dataset with large- scale face attributes. We crop the initial 178x2I5 size 138x 138 and resize them as 128 128. There are total 202,599 face images and we use 162,766 769 images as training data and the rest as test data. The pixel values are normalized between 0 and 1. The weights of the model
initialized with the method proposed by Glorot and Bengio (Glorot & Bengio 2010). The encoder consists of eight convolutional layers whose filter size is 5 with LeakyReLU activation function followed by dropout and batch normalization layer (Maas et al. 2013 Srivastava et al.) 20144) loffe & Szegedy) 2015). The decoder composed of four convolutional layers and 4 deconvolutional layers with ReLU activation function followed by several layers like encoder (Nair & Hinton 2010). a, 3, and y are set as 0,0004, 1, and 0. respectively. The binary function for measuring the reconstruction error is set Bojanowski et al. did (Bojanowski et al. 2017). Basis is trained for 100 epochs with 100 batch size. The optimizer used to train the model is Adam proposed by Kingma and Ba (Kingma & Ba 2014).
4.2 GENERATED IMAGES
To verify the performance of the proposed model, the performance of Basis VAE is compared with the performance of vanilla VAE, betaVAE, and VQ-VAE (van den Oord & Vinyals 2017), which have the same structure, but different output of encoder to the proposed model, in three aspects: Reconstruction, random generation, and disentanglement.

---

Under review as a conference paper at ICLR 2020
Orig.
/
7
2
Recon
2
(a)
Orig.
Recon
Figure 4: (Top) The original images and (bottom) the reconstructed images on (a) MNIST and (b)
CelebA datasets.
Appendix A shows more generated images.
3
4
S
S
G
Figure 5: The generated images of (a) MNIST and (b) CelebA. In the MNIST dataset, the generated data is organized in each row by class.
4.2.1 RECONSTRUCTION
We evaluate the reconstruction performance of VAE with MNIST and CelebA datasets. Fig. shows the reconstructed images for the original images. In Table 1, we show the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) values together with the comparison model for the quantitative evaluation of the performance. The experiment is repeated 10 times to compute the SSIM and PSNR values between the actual images and the generated images by the model trained on CelebA dataset. The results of the t-test show that the performance of the Basis VAE is superior that of the other models statistically.
4.2.2 RANDOM GENERATION
The generated data by BasisVAE learned with MNIST and CelebA illustrated in Fig. 5. Frechet inception distance is used to evaluate the quality of the generated images (Heusel et al.] (017) as shown in Table 2. The p-value obtained from the t-test was less than 0.05, indicating statistically significant difference in performance.
4.2.3 GENERATION FROM BASIS
We conduct an experiment to verify that the basis learned through Basis has actually influenced the construction of the disentangled representation. BasisVAE generates the images with basis b; by setting the coefficients as c; = and cj for i + j. The feature corresponding to each basis b, is shown on the Figs. and We also use 3DFaces dataset as well as CelebA dataset identify the
Table 1: The results of evaluating the reconstruction performance with SSIM and PSNR.
VAE
B VAE
VQ-VAE
Basis VAE
Average
0.7071
0.6142
0.7564
0.7965
SSIM
Std. dev.
6.0x 10 6
6.9x 10
8.6x 10 6
4.6 10
p-value
2.4x 10 25
1.3x 30
2.6x 10 18
Average
64.989
61.512
66.38
67.882
PSNR
Std. dev.
0.004
0.004
0.001
0.004
p-value
2
1.1 10
32
7.8x 10 23

---

Under review as a conference paper at ICLR 2020
Table Comparison of image generation quality by FID score.
VAE
3VAE
VQ-VAE
Basis VAE
Average
112.883
168.239
84.59
78.449
Std. dev.
1.309
1.964
7.513
2.877
p-value
1.49x
1.85 10 28
5.31 6
5_ clock
Arched_eg _eye
Bags under
shadow
Banes
Bllond hair
Busky eyebrows
Deuble chin
Heavy makeup
Blurry
Brown hair
Chubby
Eyeglasses
Gnatze
Gray hair
High cheek
Mouth slip Iuly
Narow eves
Pointy no58
Receding hairtine
Rosy checks
Male
Musrache
No beard
Oval face
Pale skin
Strniga1 hair
Weating earzings
Weating hat
Wearing Lipstick
Wearing necklace
Wearing nocktie
Sidebarns
Smile
Wavy hair
Young
Figure 6: The images generated from a single basis. The corresponding feature is shown above the
image.
Azimuth
Lighting
Figure 7: The images generated from a single basis. The value of the coefficient is linearly changed along the row. The corresponding characteristics shown in the left of the images.

---

Under review as a conference paper at ICLR 2020
Hair color
Bald
Hair
color
Pale
Bags
D
under-
Eyeglasses
eyes
Figure 8: The value of the coefficient is linearly changed along the row. The corresponding charac- teristics are shown in the side of the images. Appendix A shows more generated images.
Black hair...
Bags under eyes
Wearing hat
Bald
Pale skin
Bengs
Eyeg loses
Mustache
Figure 9: Randomly generated images with the coefficient e; for b; fixed 1. According to the basis element, the images reflecting the corresponding feature are generated. Appendix A shows more generated images.
characteristic change with coefficient size. As shown in Figs. 8 and 9, we can see the basis element corresponding to azimuth and lighting in 3DFaces dataset and to hair color, bags under eyes, bald, etc. in CelebA dataset. To show that single basis element is subject to single generative factor, randomly generate an image with the coefficient G; for b; fixed 1, shown in Fig. 9. To quantitatively evaluate the disentangled representation, logistic regression is trained to classify the features by inputting the output of the encoder into itself. The more disentangled the latent space is, the higher accuracy the model achieves. We train about 40 binary classifiers for 40 classes of CelebA dataset, and the average accuracy shown in Table 3. Appendix A shows the more generated images and Appendix C shows the distribution of coefficients that model learns with examples.
5 CONCLUSION
In this paper, we have formulated the disentangled representation learning with two constraints. By proving the Theorem, it is shown that the latent space can be decomposed as independent latent variables associated with single generative factor. We have shown that the proposed BasisVAR constructs disentangled representation without the second constraint by constructing the basis of the latent space. Furthermore, Basis outperforms the vanilla VAE and 3VAE in both performance and disentanglement. Since our method can be applied to other VAEs by changing the output of the encoder coefficients for basis element and adding loss Lg. will verify the versatility and validity by applying to other models. The performance of the proposed model will be evaluated with other well-known benchmark datasets such CIFAR10, 3DFaces, and ImageNet. In addition, we will achieve the higher quality of the generated data and interpretability of the latent space by constructing disentangled latent space in generative adversarial network.
Table 3: Results of classification using the output of the encoder. The logistic regression model for each class is trained to classify the one class. Appendix shows more details the numerical results for each attributes.
VAE
3VAE
VQ-VAE
BasisVAE
Average
0.8190
0,8444
0.8225
0.8982
Std. dev.
0.015
0.009
0.014
0.005
p-value
0.001
0.004
4.1 10
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
M. Aharon, M. Elad, and A. Bruckstein. K-svd: An algorithm for designing overcomplete dictio- naries for sparse representation. IEEE Trans. on Signal Processing, 54(11):431-4422 2006.
Y. Bengio, A. Courville, and P. Vincent. Representation learning: review and new perspectives. IEEE Trans. on Puttern Analysis and Machine Intelligence, 35(8):1788-1828 2013.
P. Bojanowski, A. Joulin, D. Lopez-Paz, and A. Szlam. Optimizing the latent space of generative networks, arXiv preprini arXiv: 1707.0576, 2017.
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2172-2180, 2016.
E. Dupont. Learning disentangled joint continuous and discrete representations. In Advances in Neural Information Processing Systems, pp. 710-720, 2018.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Int. Conf. on Artificial Intelligence and Statistics, pp. 249-256, 2010.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. Beta-vae: Learning basic visual concepts with a constrained variational framework. Ini. Conf. on Learning Representation, 2(5):6, 2017.
A Hyvärinen and E Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411 1-430, 2000.
S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 152.0:1502.03167, 2015.
J.Y. Kim and S.B. Cho. Detecting intrusive malware with a hybrid generative deep learning model. In Int. Conf. Intelligent Data Engineering and Automated Learning, pp. 499-507. Springer, 2018.
J.Y. Kim and S.B. Cho. Electric energy consumption prediction by deep learning with state explain- able autoencoder. Energies, 12(4):739, 2019.
J.Y. Kim, S.J. Bu, and S.B. Cho. Zero-day malware detection using transferred generative adversarial networks based on deep autoencoders. Information Sciences, 460:83-100 102, 2018.
D.P. Kingma and J. Ba. 14i::1412.66800 2014.
Adam: A method for stochastic optimization.
arXiv preprint
D.P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
B.M. Lake, T.D. Ullman, J.B. Tenenbaum, and S.J. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.
Y LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recog- nition. Proc. of the IEEE, 86(1 1):2278-225 22244, 1998.
Lee, A Battle, R Raina, and A. Y. Ng. Efficient sparse coding algorithms. In Advances Neural Information Processing Systems, pp. 801-808, 2007.
Z. Liu, P. Luo, X. Wang. and X. Tang, Deep learning face attributes in the wild. In IEEE Int. Conf. on Computer Vision, pp. 3730-3738, 2015.
A.L. Maas, A.Y. Hannun, and A.Y. Ng. Rectifier nonlinearities improve neural network acoustic models. Int. Conf. Machine Learning, 30(1):3, 2013.

---

Under review as a conference paper at ICLR 2020
J Mairal, J Ponce, G Sapiro, A Zisserman, and F.R. Bach. Supervised dictionary learning. In Advances in Neural Information Processing Systems, pp. 1033-1040, 2009.
V. Nair and G. Hinton. Rectified linear units improve restricted boltzmann machines. In Int. Conf. on Machine Learning, pp. 807-814, 14, 2010.
Paysan, Knothe, Amberg, S. Romdhani, and Vetter. A 3d face model for pose and illumi- nation invariant face recognition. In Int. Conf. on Advanced Video and Signal Based Surveillance, pp. 296 3011. Ieee, 2009.
Radford, L Metz, and Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:/5/ 2015.
K. Ridgeway. A survey of inductive biases for factorial representation-leariing arXiv preprint arXiv:1612.05299, 2016.
L.I. Smith. A tutorial on principal components analysis. Univ. of Otago. Technical Report, 2002.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple
way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15 (1):1929- 1958, 2014.
A van den Oord and O Vinyals. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDIX A: MORE IMAGES GENERATED
Figure 10: The generated CelebA images. Image blur is less than conventional VAE.
11

---

Under review as a conference paper at ICLR 2020
Hair color B
Hair color A
Bags under-
eyes
+
G
U
D
D
W
W
W
W
W
W
W
F
L
W
+
Eyeglasses
Pale skin
Bald
Figure 11: The value of the coefficient is linearly changed along the row. The corresponding char- acteristics are shown in the side of the images.
12

---

Under review as a conference paper at ICLR 2020
Black hair
Bags under eyes
Wearing hat
Bald
Bangs
Eyeglasses
Pale skin
Mustache
Smile
Figure 12: Randomly generated images with the coefficient for b; fixed 1. According to the basis element, images reflecting the corresponding feature are generated.
13

---

Under review as a conference paper at ICLR 2020
Table 4: Results of classification using the output of the encoder. The logistic regression model for
each class is trained to classify the one class.
Class
VAE
beta
VQ-VAE
BasisVAE
5_0. wwwcckloocchhllce
0.8820
0.8816
0.8820
0.9220
Arched_eyebre
0.7414
0.7450
0.7423
0.8231
Attractive
0.6579
0.6945
0.6468
0.7842
Bags-under.eyy _eyes
0.7926
0.7920
0.7926
0.8243
Bald 0.9798
0.9798
0,9792
0.9808
Bangs
0.8532
0.8890
0.8588
0.9376
Big_lips
0.8468
0.8466
0.8468
0.8419
Big Big_nose
0.7514
0.7568
0.7514
0,8115
Black_hair
0.7899
0.8335
0.7876
0.8773
Blond hair
0.8470
0.8983
0.8591
0.9377
Blurry
0.9526
0.9526
0.9526
0.9526
Brown hair
0.7586
0.7614
0.7586
0.8304
Bushy eyebrows
0,8575
0.8603
0,8575
0.9105
Chubby
0.9389
0.9389
0.9389
0.9440
Double_chin
0,9510
0.9510
0,9510
0.9578
Eyeglasses
0,9305
0.9312
0.9304
0.9837
Goatee
0.9265
0.9265
0.92646
0.9542
Gray_hair
0.9513
0.9517
0.95131
0.9702
Heavy makeuu
0.6430
0.7586
0.6796
0.9037
High_cheeken cheeknones
0.5672
0.7116
0.5907
0.8596
Male
0.6522
0.7720
0.6689
0,9780
Mouth Noouthssighhll..om
0.5480
0.6752
0.5542
0.9262
Mustache
0.9496
0.9496
0.9496
0.9519
Narrow eyes
0.9250
0.9251
0.9250
0.9222
No_beard
0.8225
0.8288
0.8225
0.9347
Oval_face
0.7196
0.7189
0.7196
0.7393
Pale_skin
0.9569
0.9594
0.9569
0.9625
Pointy_nose
0.7151
0.7151
0.7151
0.7459
greeding-hairline
0.9281
0.9277
0,9281
0,9339
Rosy cheeks
0,9317
0.9316
0,9317
0.9431
Sideburns
0.9313
0.9314
0.9313
0.9565
Smiling
0.5730
0.7330
0.5957
0.9165
Straight_ hair
0.7942
0.7947
0.7942
0.7989
Wavy hairr
0.7262
0,7516
0.7288
0.7917
Wearing earrings
0,8092
0.8103
0.8092
0.8597
Wearing hat
0.9528
0.9617
0.9528
0.9798
Wearing lipstick
0.6542
0.7668
0.67945
0.9135
Wearing necklace
0.8794
0.8794
0.8794
0.8794
Wearing -eectin nnektiee
0.9273
0.9271
0.9273
0.9277
Young 0.7465
0.7541
0.7464
0.8588
B APPENDIX B: DETAILED RESULTS TO MEASURE DISENTANGLEMENT
The more detailed classification accuracy, summarized in Table 3 average, are shown in Table by generative factors.
C APPENDIX C: MORE DETAILS IN THE PROOF OF LEMMA I AND THEOREM
1
In the Section 3.1, we show the proof of Lemma 1 in the case of n = 2. For more generality, it is necessary to be proved that the Lemma 1 holds in the Za = (2g, 2n), repeat the same verification process, then we can separate the 23 from the (24, ... 2.). If process recursively, the Lemma I in the case of >= 3 holds too.
of >= 3. In that case, Let
14

---

Under review as a conference paper at ICLR 2020
Figure 13: The distribution of c1 <= 40) that model learns each data with (a) "female with blond hair". (b) "female with black hair", and (c) "male with black hair". We have red boxes for coefficients where the difference between distributions (a) and distributions in (b) and is apparent.
As in the proof of Lemma 1, is enough to prove the Theorem that Theorem holds when n = 2. The flow of the equations (4) to (8) except for the equations (5) to (6) is natural. Equation (6) is derived from the (5) with p(æ 21, 22) = p(x 21)p(x 22)/p(x), which is equivalent to equation (13).
plx, 21, 21,
plx, z1 (x(,
(13)
p(Z1,
(()))(=))p()))))
Since 21, ... 2n are independent, it is drived to (14).
p(u, 21, x2) p(x, 21 )((r, 22)
(14)
(())
The LHS of equation (14) can be modified (15) to (18)
p(,,21,20 22) 1(x, 21, 22)
p( 21)
plx, 21)
=p(22 x. 21 (2()))))))))))) p(22 xx)p(r,21)
(15)
p(z, )1 22)
p(x)
Since the LHS of equation (14) can be derived to the RHS of equation (15), the equation (13) is correct, resulting in the correctness of Theorem
D APPENDIX D: THE DISTRIBUTION OF COEFFICIENTS C;
We show the distribution of c; that the model learns for each data with "female with blond hair". "female with black hair", mmae with black hair" in the Fig. 13. With comparison of distributions in Fig. 13(a) and (b), coefficients Og and 01 represent "black hair" and "blond hair", respectively. Furthermore, can confirm that coefficients C19. C21. and c37 represent the generative factors related to the characteristics on gender, comparing Fig. 1 13(a) and (c).
15


Reviews:
[{"review_id": "HkgnZixS9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper claims to achieve disentanglement by encouraging an orthogonal latent space.\n\nDecision: Reject. I found the paper difficult to read and the theoretical claims problematic. \n\nIssue 1: The Theorem\nCan the authors explain how they got from Eq 5 to Eq 6? It seems that the authors claim that:\np(x | z1 z2 … zn) = p(x | z1) … p(x | zn) / p(x)**(n - 1)\nI have difficulty understanding why this is true. It would suggest that\np(x | a b) = p(x | a) p(x | b) / p(x). \nSuppose a and b are fair coin flips and x = a XOR b. Then\np(x=1 | a=1 b=1) = 0\np(x=1 | a=1) = 0.5\np(x=1 | b=1) = 0.5\np(x=1) = 0.5\nCan the authors please address this issue?\n\nEven if Equation 8 is somehow correct, can the authors explain why BasisVAE provably maximizes the RHS expression in Eq 8? In particular the object p(x | z_i) is the integral of p(x, z_not_i | z_i) d z_not_i, which is quite non-trivial. \n\nIssue 2: The Model\nThe notation is a bit confusing, but it looks like the proposed model is basically a standard VAE, but where the last layer of the mean-encoder is an orthogonal matrix. I do not think the authors provided a sufficient justification for how this model relates back to Theorem 1. \n\nFurthermore, it is unclear to me why an orthogonal last-layer is of any significance theoretically. Suppose f is a highly expressive encoder. Let f(x) = M.T g(x) where g is itself a highly expressive neural network. Then M f(x) = g(x), which reduces to training a beta-VAE (if using Eq 12). From a theoretical standpoint, it is difficult to assess what last-layer orthogonality is really contributing.\n\nIssue 3: The Experiments\nExperimentally, the main question is whether the authors convincingly demonstrate that BasisVAE achieves better disentanglement (independent of whether BasisVAE is theoretically well-understood). \n\nThe only experiment that explicitly compares BasisVAE with previous models is Table 3. What strikes me as curious about the table is the standard deviation results. They are surprisingly small. Did the authors do multiple runs for each model? Furthermore, the classification result is not equivalent to measuring disentanglement. There exists examples of perfectly entangled representation spaces can still achieve perfect performance on the classification task (any rotation applied to the space is enough to break disentanglement if disentanglement is defined as each dimension corresponding to a single factor of variation).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gCgZOrKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes BasisVAE for acquiring a disentangled representation of VAE. \nThough the topic is much of interest for this conference, I cannot support its acceptance because the paper leaves many aspects unexplained in the model design. \n\nIn particular, the following points need justified and clarified.\n1) Theorem 1 is difficult to follow. \nThe claim of the theorem is unclear. \nI suppose it says ELBO can be written as a sum with respect to z_i given p(z)=\\prod_i p(z_i), but the statement is not clear enough from the text. \nProof of Lemma 1 is logically incomplete. Discuss the cases n>2.\nDerivation of equation (6) from (5) seems erroneous: p(x|z_1, ..., z_n) = \\prod_{i=1}^n p(x|z_i) / p^{n-1}(x) does not hold in general even if z_i's are independent p(z_1, ..., z_n)=\\prod_{i=1}^n p(z_i).\n\n2) Connection between the objective function and Theorem 1 is unclear. \nBasisVAE uses a linear combination of Eqs. (9,10,11) as its objective function. \nHow Theorem 1 motivates this formulation?\n\n3) Reconstruction error (9). \nThe text says \\ell of Eq. (9) is the binary function and configured as in (Bojanowski et al. 2017). \nHowever, Bojanowski et al. used a weighted l1 error Laplacian Pyramid representation. \nFurthermore, the original VAE formulation uses a conditional log-likelihood log p(x|z) for the reconstruction term. \nHow is binary function \\ell related the likelihood?\n\n4) KL regularization term (10).\nFor computing this term, the output of encoder c=f(x) should be converted into z. \nNotation of N(f(x), \\Sigma) is confusing. \n\n5) Figure 6 shows diversity in many factors. \nFigure 6 is not as impressive for disentangled images since many factors change by varying a single basis. \nIs this an expected result?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1xdl-vCFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[updated rating due to supervision of $c_i$, which was not made clear enough and would require other baseline models]\n\nThis paper proposes a modification of the usual parameterization of the encoder in VAEs, to more allow representing an embedding $z$ through an explicit basis $M_B$, which will be pushed to be orthogonal (and hence could correspond to a fully factorised disentangled representation). It is however possible for different samples $x$ to use different dimensions in the basis if that is beneficial (i.e. x is mapped to $z = f(x) \\cdot M_B$, where f(x) = (c_1, ... , c_n) which sums to 1.). This stretches the usual definition of what a “disentangled representation” means, as this disentanglement is usually assumed to be globally consistent, but this is a fair extension.\nThey show that this formulation can be expressed as a different ELBO which can be maximized as for usual VAEs.\n\nI found this paper interesting, but I have one clarification that may modify my assessment quite strongly (hence I am tentatively putting it on the accept side). Some implementation details seem missing as well. Otherwise the presentation is fair, there are several results on different datasets which demonstrate the model's behaviour appropriately.\n\n1.\tThe main question I have, which may be rather trivial, is “are the c_i supervised in any way?”. When I first read the paper, and looking at the losses in equations 9-11, I thought that this wasn’t the case (also considering this paper is about unsupervised representation learning), but some sentences and figures make this quite unclear:\n\ta.\tIn Section 3.2, you say “We train the encoder so that c_i = 1 and c_j = 0 if the input data has i-feature and no j-feature”. Do you?\n\tb.\tHow are the features in Figure 6 attached to each b_i? I.e. how was “5_o_clock_shadow” attached to that particular image at the top-left?\n\tIf the c_i are supervised, this paper is about a completely different type of generative modeling than what it compares against (it would be more comparable to VQ-VAE or other nearest-neighbor conditional density models).\n2.\tThere is not enough details about the architecture, hyperparameter and baselines in the current version of the paper.\n\ta.\tWhat n_x (i.e. dimensionality of the basis) do you use? How does this affect the results?\n\tb.\tHow exactly are f(x), \\Sigma_f(x) parametrized? They mention the architecture of the “encoder” in Section 4.1, but this could be much clearer.\n\tc.\tHow do you train M_B? I assume they are just a fixed set of embeddings that are back-propagated through?\n\td.\tWhat are the details about the architecture of the baselines, and their hyperparameters? E.g. what is the beta you used for Beta-VAE?\n3.\tThe reconstructions seem only partially related to their target inputs (e.g. see Figure 4). This seems to indicate that instead of really reconstructing x, the model chooses to reconstruct “a close-by related \\tilde{x}”, or even perhaps a b_i. This would make it behave closer to VQ-VAE, which explicitly does that. How related are reconstructions/samples to the b_i?\n4.\tCould you show the distribution of c_i that the model learns, and how much they vary for several example images?  How “peaky” is this distribution for a given image (this feeds into to the previous question as well)? The promise of the proposed model is that different images pick and choose different combinations of b_i, which hopefully one should see reflected in the distributions of c_i per sample, across clusters, or across the whole dataset.\n5.\tWhat happens when L_B is removed? I.e. what is the effect of removing the constraint on M_B being a basis, and instead allow it to be anything? This seems to make it closer to a continuous approximation to VQ-VAE?\n6.\tIs Equation 10 correct? Should the KL use N(f(x) \\cdot M_B, \\Sigma_f(x)), as in equation 9 above?\n7.\tSimilarly, in Section 4.2.3, did you mean “c_i = 1 and c_j = 0 for i != j”?\n\nIf the model happens to be fully unsupervised, I think that these results are quite interesting, and provide a good modification to the usual VAE framework, I find that having access to the M_B basis explicitly could be very valuable.\n\nThere is still an interesting philosophical discussion to be had about when one would like to obtain a “global basis” for the latent space (i.e. Figure 3 (b)), or when one would prefer more local ones. I can see clear advantages for a non-local basis, in terms of generalisation and compositionality, which your choice (i.e. Figure 3 (c) ) would prohibit.\n\nReferences:\n[1] VQ-VAE: Aaron van den Oord, Oriol Vinyals, Koray Kavukcuoglu, “Neural Discrete Representation Learning”, https://arxiv.org/abs/1711.00937", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new way to learn a disentangled representation by embedding the latent representation z into an explicit learnt orthogonal basis M. While the paper proposes an interesting new approach to disentangling, the reviewers agreed that it would benefit from further work in order to be accepted. In particular, after an extensive discussion it was still not clear whether the assumptions of Theorem 1 applied to VAEs, and whether Theorem 1 was necessary at all. In terms of experimental results, the discussions revealed that the method used supervision during training, while the baselines in the paper are all unsupervised. The authors are encouraged to add supervised baselines in the next iteration of the manuscript. For these reasons I recommend rejection.
Decision: Reject