Disentangled Representation Learning with Sequential Residual Variational Autoencoder
Authors: Nanxiang Li
Shabnam Ghaffarzadegan
Liu Ren
Venue:  (2020)
Keywords: Disentangled Representation Learning
Variational Autoencoder
Residual Learning

Abstract: Recent advancements in unsupervised disentangled representation learning focus on extending the variational autoencoder (VAE) with an augmented objective function to balance the trade-off between disentanglement and reconstruction. We propose Sequential Residual Variational Autoencoder (SR-VAE) that defines a "Residual learning" mechanism as the training regime instead of the augmented objective function. Our proposed solution deploys two important ideas in a single framework: (1) learning from the residual between the input data and the accumulated reconstruction of sequentially added latent variables; (2) decomposing the reconstruction into decoder output and a residual term. This formulation encourages the disentanglement in the latent space by inducing explicit dependency structure, and reduces the bottleneck of VAE by adding the residual term to facilitate reconstruction. More importantly, SR-VAE eliminates the hyperparameter tuning, a crucial step for the prior state-of-the-art  performance using the objective function augmentation approach.  We demonstrate both qualitatively and quantitatively that SR-VAE improves the state-of-the-art  unsupervised disentangled representation learning on a variety of complex datasets.

Under review as a conference paper at ICLR 2020
DISENTANGLED REPRESENTATION LEARNING WITH SEQUENTIAL RESIDUAL VARIATIONAL AUTOEN- CODER
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advancements in unsupervised disentangled representation learning focus on extending the variational autoencoder (VAB) with an augmented objective func- tion balance the trade-off between disentanglement and reconstruction. We propose Sequential Residual Variational Autoencoder (SR-VAL VAE) that defines a "Residual learning" mechanism as the training regime instead of the augmented objective function. Our proposed solution deploys two important ideas in single framework: (1) learning from the residual between the input data and the accu- mulated reconstruction of sequentially added latent variables; (2) decomposing the reconstruction into decoder output and a residual term. This formulation en- courages the disentanglement in the latent space by inducing explicit dependency structure, and reduces the bottleneck of VAE by adding the residual term to facil- itate reconstruction. More importantly, VAEE eliminates the hyperparameter tuning, a crucial step for the prior state-of-the-art performance using the objective function augmentation approach. We demonstrate both qualitatively and quan- titatively that SR-VA VRE improves the statc-of-the-art unsupervised disentangled representation learning on variety of complex datasets.
INTRODUCTION
Learning sparse and interpretable representation of data is critical component of generalized, robust and explanatory intelligent system. This concept is inspired by human's 's ability to generalize the knowledge with abstract concepts and use them reason the unseen environments Guttt et al. (2018). Despite recent advances on representation learning, it was shown that deep convolutional neural networks (CNN's) have tendency to learn superficial statisties of data associated with given tasks, rather than important generative factors embedded in the physical world Jo & Bengio (001);;
Goodfellow et al.) (2014). One way towards this goal is disentangled representation learning which aims to capture the independent and interpretable generative factors of the data. [Beniio et al. (2013)
defines the disentangled representation intuitively as representation where changes in one dimension correspond to changes in only one generative factor of the data, while being relatively invariant to changes other factors. Recently, HHggins et al. (2018) assigned a principled definition by connecting symmetry transformations to vector representations using the group and representation theory.
Based on these definitions, disentangled representation can be learned in a supervised fashion where explicit and/or implicit prior knowledge on the generative factors of data available. However, is ideal to achieve this in an unsupervised learning setting to take advantage of the large amount of available unlabeled data. Along with the recent development of the generative models, many unsupervised disentangled learning approaches have been proposed based on either the generative adversarial networks (GAN) (proposed as InfoGAN in Chen et al. (2016)) or the variational autoen- coders (VAE) (proposed as 8- BAE in Higgins et al. (2017)). While 3-VAE achieves better results and does not suffer from the training stability issue of InfoGAN, it faces trade-off between the disentanglement and reconstruction due to its information bottleneck. The current state-of-the-e approaches extend the VAE with augmented objective function to reduce this trade-off Burgess et al. (2017): Kim & Mnih 2018) Chen et al. (2018); Kumar et al. (017)) A recent study by Locatello
Codes available at: http://wwwwweppppoo.m///////sshe

---

Under review as a conference paper at ICLR 2020
et al. (2018) carefully compared these approaches based on extensive experiments. They found that the performance of these approaches is very sensitive to the hyperparameter tuning associated with the augmented objective function and the initial random seed during training. More importantly, they proved that unsupervised learning of disentangled representation is impossible without introducing inductive bias either the model the data. We believe the trade-off between disentanglement and reconstruction in VAE-based approaches can be addressed by a different training approach. The idea of relying on modified training approaches, instead of augmented objective function, to encourage network behavior is commonly used for different problems. Take model over-fitting prevention for example, one way to address this is to augment the objective function with regularization terms, such as L or I,2 regularization. An alternative solution is to apply special operations during training to enforce the generalization of the network representations, such as Dropout Srivastava et al. (2014) or Batch Normalization loffe & Szegedy (2015).
Our main contribution in this work four-fold: 1) We propose Sequential Residual Variational Autoencoder (SR-VAE) that uses novel "Residual learning" mechanism to learn disentangled representation with the original VAE objective. This is different from previous VAE-based approaches that merely focus objective function design where hyperparameter tuming is crucial. 2) We show the proposed "Residual learning" mechanism defines an explicit dependency structure among the latent variables via sequential latent variable update. This encourages learning the disentangled representation. 3) We highlight that SR-VAE decomposes the reconstruction into residual and network decoder output via skip connection. This relaxation of reconstruction reduces the trade-off between disentanglement and reconstruction of VAE. 4) We demonstrate both qualitatively and quantitatively that SR-VAE improves the current state-of-the-art disentangiement representation learning performance on variety of complex datasets.
2 CHALLENGES OF DISENTANGLING WITH AUGMENTED VAE OBJECTIVE
In this section, we first briefly review the VAE framework, followed by the 3-VAE and its extensions for disentangled representation learning. We highlight the challenges of using an augmented objective function to balance the VAE's trade-off off between the disentanglement and reconstruction. From these discussions, we then motivate the proposed SR-VAE framework.
VAE is a deep directed graphical model consisting of an encoder and a decoder Kingma & Welling (2013). The encoder maps the input data x to latent representation go (2(ar) and the decoder maps the latent representation back to the data space 4s (z](²), where @ and á represent model parameters. The loss function of the VAE is defined as following:
LV = (x log 9o (=]=)] (L((())))) p(=)).
(1)
where KL( || stands for the Kullback-Leibler divergence. By regularizing the posterior qa (=(x)) with prior over the latent representation ((zitiioonp(p))))))))))) ~ (N(00,)) I). where I is identity matrix, VAE learns a latent representation (=(2x) that contains the variations in the data. The goal of disentangled representation learning to identify the latent representation corresponds to one of the generative factors for given data x. To achieve this, 3- augments VAE objective with an adjustable hyperparameter /3 as:
Rd where each latent variable only
L3- AAE = (= [log go - p())).
(2)
The addition of encourages the posterior (2(x)) match the factorized unit Gaussian prior p(z). It enhances the independence among the latent variables thus disentangling the representation. On the other hand, it reduces the amount of information about x stored in 2, which can lead to a poor reconstruction especially for high values of B3. This trade-off is further discussed from the rate-distortion theory perspective in Burgess et al. (011)).
To reduce the trade-off, different augmentations of 3- objective are proposed Burgess et al. (017)); Kim Mnih (2018); Chen et al. (2018); Kumar et al. (017). Locatello et al. (2018) categorized these methods into three main categories of bottleneck capacity, penalizing the total correlation and disentangled priors. Burgess et al. (017)) focused on bottleneck capacity and proposed to gradually increase the average KL divergence from zero for each generative factor. This method relaxes the information bottleneck during training via increasing the encoding capacity through a parameter C that is linearly dependent on the training iteration. Kim & Mnih (018)) aimed
2

---

Under review as a conference paper at ICLR 2020
Input,
Inpat Iayer
Input lnyer
Impur layer
Input layer
72
m
7:
der
dec
Outpur layendee
Dusput layer
Oulpal layer
Output layer
%1
Step; update
Step2 update
Step, update
Step, update
Latent variables with fixed values from previous updates.
Latent variable to updated.
Output,
Rd
variables with fixed values.
Figure 1: "Residual learning" mechanism consists of d steps in single forward pass with the same encoder
qo (z) (x) and decoder (z(lz) Latent variables are sequentially sampled from the encoder. step i, only the ith latent variable x, follows the distribution learned from the current residual. Previous latent variables follow the same distribution learned from their corresponding residuals. The latent variables Z:41 to za have fixed 0 value.
The final output x' consists of the decoder output using all the latent variables 2t and the skip connection Td-
to solve the disentangled representation learning by minimizing the total correlation (TC) term. They proposed Factor where the objective is augmented with a TC term controlled by hyperparameter y. Minimizing the TC term forces the hidden representation to be factorial and hence independent. Chen et al. (2018) looked into an alternative way minimize the TC term in the augmented objective. named 3- 3-TCVAE. They used mini-batch based alternative instead of a vensity-ratio-trte
based method from FactorVAF. However, the results from both methods are sensitive to hyperparameter associated with the TC term. Kumar et al.] (2017) studied the disentangled prior and introduced regularizer the objective that is associated with hyperparamter This regularizer encourage the covariance of (2) to match the identity matrix. While all aforementioned approaches have shown promising results, they rely on careful tuning of the hyperparamater introduced in the augmented objective functions such as B in Higgins al. (2017). ) C' in Burggss al. (2017). y in Kim & Mnih (2018) and Chen et al. (2018). and À in Kumar et al. (017). Finding the optimal hyperparamater setting can be challenging especially in an unsupervised learning setting where the evaluation of the results mainly relies on visualization and human inspection. More importantly, [Locteelo et al.) (2018) found that hyperparameter tuning is more important for state-of-the-ar performance than the choice of augmented objective functions.
In this work, we propose SR-VAE to address the aforementioned challenge. Instead of the forward pass in the original VAE, SR-VAE uses a "Residual learning" forward pass illustrated in Fig. as an inductive bias on the model. SR-VAE consists of two components: 1) Explicit dependency in the latent space via a multi-step sequential forward pass where one latent variable is updated at each step; 2) Decomposition of the reconstruction via skip connetion between the input and network output to relax the network reconstruction constraint. Together, these two components enable SR- VAE to address the trade-off between the reconstruction and disentanglement in VAE using the original objective in Eq. I In the next section, we first describe the details of the "Residual learning" forward pass and SR-V VAE training. We then discuss the two aforementioned components in detail. In Section 5) we demonstrate the effectiveness of SR- RR-VAE and investigate the effect of each component separately. The experimental results show our approach achieves better disentanglement and reconstruction results compared to the current state-of-the-art approaches, including 3-VAE Higgins et al. (017)) and Factor VAE Kim & Mnih (2018).
3 SEQUENTIAL RESIDUAL VARIATIONAL AUTOENCODER SR-VAE
Same as the original VAE, SR- VAE consists of an encoder network noted as qo (z]x)) and a decoder network noted as go (x)²). 2). Here and stand for input data and latent representation vector; and represent encoder and decoder network parameters. Let the dimension of the latent representation be d. SR- SR-VAE learns ह = [2 ................................ =d] E gd as the latent representation of the data. Its forward pass follows "Residual learning" mechanism that consists of d steps. Each step only updates one latent variable. In the first step, the input data ø passes through the encoder to calculate the parameterized posterior, noted 17 and oJ. Instead of drawing samples for all latent variables I ~ NV Ø1), only sample the first latent variable 21 ~ N [1], o [1]) and set the remaining latent variables to O. The modified latent variables ī = [21, 0, .,...,00 then passes the decoder to generate the output noted
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 SR-VAE Forward Pass
Algorithm 2 SR-VAE Learning
Input: observation x, latent dimenssin d, VAE encoder (o(2/z), VAE decoder, qo (z(2))
Input: Dataset X, batch size m, latent dimension d, Initialize VAE parameters 0,0
Output: reconstruction latent representation pl
1: repeat
1: 7
2:
Randomly select batch
(1(1)
size m,
2: f = [0. 0] E RRd
EEB
3:
(x. の)} }}{ +- Forward [orwarddpass(x)
3: o' = (0..... 0] E R
Lrecon MSE 1oss(xx
4: for i == / d do
5:
[pi, di)
Encoder: go(7))
5:
fxe + - -
6:
ø [i]
-
j=1i=1 [1 + 1ogoo"0 (n' - (o)))
7:
σ [ [.
6:
Exc + Lrecon
8:
Reparameterizetti, ,d)
7: [0, Buckward(C)
9:
8 + Decoder: (x(
until convergence of objective
10:
if < d then
11:
76+1 < 7 2z
12:
end if
13:
end for
14:
Rd + 7/
as $1. We subtract the decoder output from the skip connection (defined as an Identity function) as the input for the second pass, noted as 722 = = 21. In the second pass, 72 passes the same encoder to generates a new parameterized posterior (F2 and σ2). This time, we sample only the second latent variable with this parameterized posterior as 22 (N 2). (2(2)). We ra-sample the first latent variable with 21 ~ WV on [1]) while setting the remaining latent variables to O. The modified latent variable ह [2], 22, .......0 0] is then used generate the new reconstruction $2. We then .......))
calculate the corresponding residual 733 -2 .22 as the input for the third pass. In the ith pass, the ith latent variable sampled from the encoder thus 2; ~ [i] o. [i]). ). The previous updated latent variables follow their corresponding residual encoding and the remaining latent variables are set to zeros, 7 = [21, .2..... 2z, 0, ..... 0j. The process repeats d times such that all the latent variables are sampled. In step d. the final output of VAR,AE, x'' consists of the decoder output &a and the residual term "dd as x' = dd + "ia- In the case where d = 1, SR- VAE follows the last step and the input is connected to the output through the skip connection. Algorithm [ shows the pseudo code of the "Residual learning" forward pass in SR-VAE.
We train the SR-VAE with the original VAE objective defined in Eq. [ The parameters updated using the standard poo-ppropagation demonstrated in Algorithm 2. The prior p(z) is set to the isotropic unit Gaussian ( I) and posterior qa (2)z) 2) is parameterized as Gaussians with a diagonal covariance matrix. The "reparametrization trick" is used to transform each random variable 2; ~ go(2 (z) as differentiable transformation of noise variable ~ (O, 1) with 2, = #4 o.C.
Due to the sequential update process, SR-VAE can generate sequence of images during the forward pass. As we shall see in Section these images reflect image transformations corresponding to the disentangled factors at different steps. Comparing to other VAE based approach that directly generates a single reconstruction output by sampling from the joint distribution variables, this step-by-step visual inspection allows for better understanding of the learned generative factor. As a result, SR- VAE provides a new way to understand the disentangled representation results.
(3(x))) for all latent
Explicit Dependency in the Latent Space: The SR-VAE forward pass defines sequential update of latent variables: the added latent variable 2; at step learns from the residual between the input data and the previously updated latent variables 2j. Vj E f1,..., i 1}. This procedure defines explicit dependency among the latent variables in the posterior that can be written as qo (2) 22,... Za 2a(z) = ga(2) ( (2)=21,))...0p (2. x.) The KL loss term of the original VAE objective in Eq. [ encourages the posterior (0 (2,1,,,,.-0) 22,... zal (r) to match the factorized unit Gaussian prior p(#).
Adding the explicit dependency by the "Residual learning" mechanism, the SR- ER-VAE objective can be seen as a modified VAE objective:
maximize 0,0
LSR-VAE = (g (o })] - KL(g0 p())),
(3)
subject to
p(z1) ~ g0(2)1z), ~ ~ g0(za|21,.... p(Z))
2d-1,
4

---

Under review as a conference paper at ICLR 2020
These constraints encourage the newly added latent variable to be independent of the ones already added, thus enhance the disentanglement of the latent representation. Moreover, the solution space of Eq. | is subset of the original VAE. The constrained objective limits the optimization search space to regions where a better local optimal solution exists. We empirically verify this result in terms of both the performance and stability to random initialization in Section
Decomposition of the Reconstruction: The final output of SR-VAE, x' consists of the decoder output and the residual term x' 7dd. This formulation relaxes the reconstruction constraint on the network's 's decoder output when comparing with other VAE-based approaches. More importantly. it creates balancing measure between the data generation and reconstruction. In one extreme case, the input directly passes through the first d - 1 steps and reaches step d as the imput. In this case, BRE becomes the original VAE model with added skip connection between input and output (see the last step in Fig I. This architecture relaxes the VAE reconstruction hence reduces VAE reconstruction and disentanglement orf We will show in Section [.T] this architecture alone can reach similar performance to Factor VAE. On the other extreme case, if the first d - 1 steps have learned a perfect disentangled representation of the data, the input for step d would be O. In this case, the reconstruction loss encourages RR- to generate the input data from learned latent representation vectors. Combining these two extreme cases, SR- VAE can be understood as a training mechanism to balance between VAE model with emphasis on the reconstruction quality (as the first case) and the data generation model given the learned latent variables (as the latter case).
Notice that each of the aforementioned components can be separately added to VAE model. To add the explicit dependency in the latent space component, we can apply the sequential forward pass of SR-VAE with the output x' .d.- We refer this model as Seq VAE. For the Decomposition of the Reconstruction component, as mentioned earlier, it is equivalent to adding a skip connection to the original VAE between the input and output. We refer to this model as ResVAE. Using these two models, we perform the ablation study to understand the effectiveness of each individual component in Section 5.1
modified
Computational Complexity: SR- VAE replaces the standard forward pass of VAE with d forward passes, thus increases the computational complexity. However, in addition to the improved state- of-the-art performance, it eliminates the hyperparameter tuning associated with prior works. As mentioned earlier, the hyperparameter tuning was shown to be critical for state-of-the-arr performance. It is difficult and time-consuming process especially for unlabeled data due to: 1) The large hyperparameter search space of continuous values; 2) The lack of evaluation metric. As a result, we believe that the increased computational complexity by SR-VAE is reasonable. Moreover, we will show that each of the d forward passes in SR-VAE correspond to a disentangled generative factor. Visualization of these intermediate steps provides new way to understand the result.
4 RELATED WORK
Connection to Other VAE-based Approaches: We highlight the similarity and advantages of SR-VAE over the VAE-based approaches introduced in Section 2 The sequential update of latent variables in VAE is similar the idea of gradually increasing the KI. divergence in Burgess et al. (2017). Instead of introducing an augmented objective, SR- VAE directly achieves this by learning one latent variable at time. When comparing to the work in Kumar al. (0017). SR-VAE encourages the independence among the latent variables by defining an explicit latent variable dependency rather than emphasizing on individual statistics (the covariance between the latent representations in Kumaa et al. (2017). Finally, the explicit latent variable dependency defined by SR-VA VAE also encourages the factorial latent representation, serving the same purpose as lowering the TC term in Kim Mnih (2018); Chen et al. (2018). It is worth noticing that the low TC term is necessary but not sufficient for disentangled representation.
Connection to Residual Deep Neural Network: ResNet He et al. (2016) introduces the idea of learning from residuals by adding the skip connections between layers such that input can propagate through layers. The key idea of ResNets is to replace learning the direct mapping between input and output, H(x) = x - y. with learning a residual formulation, (x)) ) = F F(x) + I + y. where F F(x) represents stacked non-liie minelin layers. This formulation reduces the loss of important information while propagating through the network. In addition, it was suggested that learning the residual mapping is easier compared to learning the direct mapping He et al. (2016). The proposed SR-VAY shares similar
5

---

Under review as a conference paper at ICLR 2020
skip connection structure ResNets. Here F(x) represents VAE, in SR-VAE. As in ResNets, F(x) can learn useful abstraction of data while the skip connection V/2 allows for circumventing difficulties in reconstructing the input data.
Connection to Deep Recurrent Attentive Writer (DRAW): DRAW Gregor et al. (2015) uses a sequential variational auto-encoding framework to achieve iterative construction of image. DRAW deploys the recurrent neural network with the attention mechanism to dynamically determine where and what to generate. The attention mechanism serves a similar purpose to the skip connection in the "Residual learning" mechanism. Moreover, the idea of successively adding the decoder output for image generation in DRAW is similar to the reconstruction decomposition in ER-VAE. One main difference between the two approaches is that DRAW relies on the recurrent network framework to model the iterative image generation in the image space. SR-VAF uses the latent dependency to emphasize iterative generation of image in the latent space.
EXPERIMENTS
We compare SR- VAE with 3-VAE and FactorVAE on four different datasets both quantitatively and qualitatively. The datasets used in this study includes: 2D Shape Higgins et al. (2017). Teapots Eastwood & Williams (0188) CelebA Liu al. (2014) and Chairs Aubry et al.] (2014). Appendix A introduces the details of these datasets. For all datasets, we use visualization for qualitative evaluation by observing the changes in the reconstruction while altering only one latent dimension, known as the traversal of the latent variable. A good disentangled representation reveals interpretable changes in the reconstruction image corresponding to one generative factor. Morcover, 2D Shape and Teapots datasets contain ground truth generative factors that are used to synthesize the dataset, which allow us to conduct guantitative evaluations. To compare with the previous studies, we use the metric proposed in Kim & Mnih (2018) (noted Factor metric) for 2D Shape, and the metric proposed metric)for Teapots. These two metrics found to cover similar notions other disentanglement metries in Locatello et al.] (2018). We implemented our approach using Pytorch Paszke et al. (2017). with the experiments run on several machines each with 4 GTXI0SO Ti GPUs. See Appendix Ci details on model architecture.
in Eastwood & Williams (2018) (noted as Disentanglement- hatigg/eemen-nfformtteeeeess wetamaaiiveees-ooopleeeee
5.1 QUANTITATIVE EVALUATION
Metrics: The FactorVAE metric in Kim & Mnih (2018) is calculated as follows: 1) select latent factor k, 2) generate new data y with factor l: fixed and other factors varying randomly, 3) calculate the mean of go (2)x)))) 4) normalize each dimension by its empirical standard deviation over all the data or large enough subset, 5) build a majority-vott classifier with the input of index of the dimension with the lowest variance and the output of factor k. The classifier accuracy is used as the evaluation metric. Eastwood & Williams (2018) defines three criteria of disentangled representation, namely disentanglement. completeness and informativeness. Disentanglement is the degree to which the learned representation disentangles the underlying generative factors; completeness is the degree to which the generative factors captured by one latent representation; and finally the informativeness is the amount of information of the generative factors that is captured by the latent representation. Disentanglement and completeness can be perceived by visualizing rows and columns of the Hinton diagram; and informativeness is calculated based on the mapping error between the learned latent representation and the ground truth factors.
Comparison to 3-VAE and Factor VAE: Similar to previous studies, we set d = 10 for all datasets except for CelebA where d 32 due to its complexity. We use the optimal parameter setting from the original studies in Higgins et al. (2017); Eastwood & Williams (2018) for 3- 3-VAE and FactorVAE. Fig. a and b)) show that SR- VAE outperforms B V-V and Factor in terms of both the metric in Kim & Mnih (2018) on 2D Shape. The best mean disentanglement measurement of SR-VAE is around 0.86, significantly higher than the one for 3-VAE at 0.72 and FactorVAE at 0.81. For reconstruction error, B-VAE and FactorVAE converge to similar results while SR- VAE achieves better performance. In Fig. (e)) we compare SR-VAE with V-E and Factor using metric proposed in Eastwood & Williams 2018) on Teapots. Three criteria used in this metric are disentanglement, completeness and informativeness. Note that in the informativeness term in this metric is highly dependent to the regressor type. In Eastwood &
reconstruction eтror and the disentanglemen

---

Under review as a conference paper at ICLR 2020
(b)
(c)
(d)
MMA SRM
Completencee
5
300
12
TectoWAL
04
heston VAE
Fettrr VAE
51 VAE
20-VAL
9.4
#Eooch
#Epec7a
ktpochs
#Epocrs
(1)
(c)
(b)
Factor-VAt SA-WNE
PestorkAg Meanic
Partonunt Maricc
0.7
250
Nk Vins, 8-14
Sn p VAA. P--20
044
KCpochs
Epochs
dEpects
(i
(k)
146
165
140-
SR-VAE
50--1-WW65 21-10
EeCOMVALE Meetc
200
110
106-
sEpocts
aEpocho
Figure 2: Quantitative evaluations. Similar to previous studies, all results are reported on the hest 10 of 30 runs with random seeds except for () and (k). The line and shaded area correspond to the mean and confidence intervals. (a) and (b): the metric in Kim & Mnih (2018) and the reconstruction error for 2D Shape; (c), (d) and (e): the three metrics in Eastwood & Williams (2018) for Teapots: (I) and (g): ablation study comparing the Seq VAE, Res VAE and SR- VAE using the Factor VAE metric and reconstruction error on 2D Shape: (h) and (i): comparison between SR- VAE and SR-3-VAE, using the FactorVAE metric and reconstruction error on
Shape: (j) and (k): Sensitivity to initialization study that compares the worst 10 of 30 runs with random seeds using the FactorVAE metric and reconstruction error on 2D Shape.
Lasso and Random Forest regressors are used which resulted in a different ordering of methods in informativeness score. Random Forest is used in our experiments to be comparable with the original paper. The results show BR-VAE achieves higher disentanglement and completeness scores compare to 8-VAE and FactorVAE, while Factor VAE achieves the best informativeness. However, we believe the informativeness can be different if more complex regressor was used.
Williams 2018
To investigate the individual effect of the two main components in SR-VA SR-VAE as discussed in Section 3 we compare the performance among Seq VAE, ResVAE and SR- VAE on 2D Shape dataset. Similar as before, the top 10 of 30 runs with random initialization of all models are reported in Fig. f and gg). The results show that both Res VAE and SeqV VAE perform worse than VREV in terms of Factor VAE metric. When comparing with B- BAE and FactorVAE, Res VAS performs similar to FactorVAE while SeQ performs similar to 3-VAE. One interesting result we noticed is that the reconstruction error from Res is similar to if not better than SR-VAE. These results verify our analysis in Section 3 that the decomposition of reconstruction relaxes the reconstruction constraint of the network, and adding the explicit dependency in the latent space improves disentangled representation learning. While both components are important for the superior performance of SR- VAE, relaxing the construction constraint on the network with skip connection is more important as it directly addresses the bottleneck of VAE.
Ablation Study:
SR-VAE with 3-VAE objective: We also examined if using the 8- VAE objective in Eq. with the "Residual learning" mechanism would improve the performance, referred to as 1R-3-VVE If so, the proposed "Residual learning" mechanism would benefit from the augmented objective to achieve better performance. Figures h) and show that best disentanglement score is obtained by SR- VAE and higher / values do not help with improving the performance. These results further verify the effectiveness of SR-V VAE in solving the trade-off between disentanglement and reconstruction in VAE-based approaches.
Sensitivity to initialization: The study in Locatello et al. (2018) showed that existing approaches sensitive to the initialization in addition to the hyperparameter tuning. One advantage of EREVAE is that it reduces the solution space and improve training stability. To verify this, compare the worst
7

---

Under review as a conference paper at ICLR 2020
X position
X1-10
Y-pmitiinn
V1-10
Rotation
2g-10
Shape
SR Iraversal
Y-position
1--10
Sioe
F1-10
Input imagi to the methods
Rotatinc-shhpe
Shape
R1-10
(b) D P-VA traversal
Y1-10
Y postiioon shape
8
possition size
Rouation-shag shape
V1-10
(c) VAE
VAE, middle steps representation 2;
Figure (a)-(c) Latent traversals with same input image across each latent dimension with d= Shape dataset, using SR- VAE, 3-VAE and FactorVAE respectively (d) Decomposition of decoder output and skip connection at each step in SR-VA RRE during latent traversal towards the last column of (a).
10 runs out of for 2D Shape 2 and Fig kk). We consistently observe better performance and smaller variances by SR- RA-,,, suggesting its robustness against random initialization.
5.2 QUALITATIVE EVALUATION
Figure 3 (c) show the latent traversal of SR- VAE, 3-VAE and FactorVAE for fixed input image of 2D Shape. z; values chosen from the range of -3 as shown in the figure. We that while all three models are capable of finding data generative factors X- X-positiin, Y-position, Shape, Rotation and Scale 3- VAE and FactorVAE struggle to disentangle these factors completely. As mentioned in Kim & Mnih (2018) shape is discrete variation factor in 2D Shape. Ideally, this factor should be modeled with a discrete rather than Gaussian latent variable. Despite this mismatched assumption, SR- VAE still captures the shape in the fifth latent variable. However, it also mixes the size with shape between oval and square in the third latent variable. We also experiment the latent traversal with the Teapots dataset and observe superior performance shown in Appendix D
For datasets without ground truth generative factors, such as CelebA and Chairs, inspecting latent traversals is the only evaluation method. Similar before, used the optimal parameter setting for 3-VAE and FactorVAE from the original studies in Higgins et al. (2017); Eastwood & Williams (2018). As seen in Figure 4 for CelebA dataset, SR-VAE is able learn interpretable factors of variation such as background, face and hair characteristics, skin color, etc. Compared to 3- BAE and FactorVAE, we observe some common factors as well some unique ones. Note that only the most obvious factors are presented in this figure. Moreover, the interpretation of each latent dimension is based our best judgment. We also observe better reconstruction quality with more details using SR- VAE method. Reconstruction losses also confirm this observation with the converged values of 300, 252 and 158 for VAE,, FactorVAE and SR- RAE,, respectively. Admittedly, careful tuning of parameters in 3-VAE and FactorVAF could potentially reveal more latent variables. However, finding the optimal value difficult task especially when there no prior information about the generative factors of data and quantitative metric is not available.
Visualizing the "Residual learning" Mechanism: To gain a better understanding of the internal process of the "Residual learning" mechanism, show the decoder output, the residual mapping, of each internal step (#) ..., f10) and their skip connections (11, ... "/1) for 2D Shape Fig. d). Each row presents the internal steps when setting different latent variables (21 to 2;) to value 3. The final outputs this process correspond to the last column of Fig. a). this figure, we observe the step by step transition to the final transformed image during the latent traversal. The result shows that
8

---

Under review as a conference paper at ICLR 2020
Hor volure
Geske
Badkgwand
Näsz4
Cáocngja
Friage
C.R.R.R
of wetwess
P.R.R.A. U.S.S.RRR
See gleves
R.S.R.A.
Sio
Eycupennco
VR. mobie mposesitio R
Bii logh
Figure 4: Left: Latent traversals across each latent dimension with d= 64 for CelebA using RA-V FactorVAE and 3-V,E, respectively. Right: Decomposition of decoder output and skip connection each step SR-VAE during latent traversal towards the first column the corresponding row.
the two terms are working together to capture the learned disentangled factor at each step. Based on Fig. B),, we know the learned factors in each step are: X-position, Y-position, Size, Rotation+ssappe and Shape, respectively. In Fig. (d)) we observe that X-position the reconstructed image are generated during the first step. In step two, both X-position and Y-position generated. This process continues and at each step the decoder output and the residual transform the image according the learned latent encoding.
Similarly, we show the step-by-step visualization for CelebA dataset along with its latent traversal result in Fig We highlight few factors due to the space limit. Although CelebA presents challenge in the complexity of real-world image, we observe similar results as the 2D Shape dataset. The step-by-step visualization shows how the latent factors are related to the transformed face image during the latent traversal. For example, the gender factor can be identified as the fifth latent factor as we observe major changes in the eyes and hair style from step five. Another example is the background contrast factor where major changes can be observed in step eight. These step-by-step visualizations provide an alternative way to understand and interpret the learned disentangled factors and can be interesting for data generation tasks.
6 CONCLUSIONS
In this work, we propose SR-VAE VAE for disentangled representation learning in an unsupervised setting. The proposed solution defines the "Residual learning" mechanism in the training regime, instead of augmented objective, to solve the trade-off between disentanglement and reconstruction of the VAE-based approaches. SR- VAE defines explicit dependency structure between latent variables and decomposes the reconstruction via skip connection. We showed that SR-VAE achieves state-of-the art results compared to previous approaches including 3- VAE and FactorVAE. Moreover, SR- EREV can be directly applied to any VAE architecture without additional hyperparameter tuning. The step-by-step process of the SR-VAE provides novel ways to visualize the results and understand the internal process of learning disentangled factors. We believe this open direction for future research towards disentangled representation learning.
REFERENCES
Mathieu Aubry, Daniel Maturana, Alexei Efros, Bryan Russell, and Josef Sivic. Seeing 3d chairs: exemplar part-based 2d-3d alignment using large dataset of cad models. In CVPR, 2014.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3588):1798-1888, 2013.

---

Under review as a conference paper at ICLR 2020
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in / -VAE. INPPS),011)
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating Sources of Disentanglement in Variational Autoencoders. (ICLR):I-13, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets. (NIPS), 2016.
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. ICLR, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems. pp. 2672-2680, 2014.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: recurrent neural network for image generation. arXiv preprint arXiv: 1502.00623 2015.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. arXiv preprint 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. B-Vee: Learning Basic Visual Concepts With Constrained Variational Framework. ICLR 2017, (July):1-13, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loie Matthey, Danilo Rezende, and Alexander Lerchner. Towards a Definition of Disentangled Representations. arXiv, pp. 1-29, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03166 2015.
Jason and Yoshua Bengio. Measuring the tendency CNNs to learn surface statistical regularities. 2017.
arXiv preprint arXiv:
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. Proceedings of Machine Learning Research, 80:2649- 2658, 10-15 Jul 2018.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013,
Abhishek Kumar, Prasanma Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. CORR, HiN/1711100844, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. CoRR, 2014.
Francesco Locatello, Stefan Bauer, Mario Lucie, Sylvain Gelly, Bernhard Schölkopf, and L G Dec. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. arXiv, pp. 1-33, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Detito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014.
10

---

Under review as a conference paper at ICLR 2020
A DATASETS
2D Shape Higgins et al. (2017) a synthetically generated dataset with ground truth generative factors. It contains 737, 280 binary 2D Shapes of heart, oval and square of size 64 64 images. There are five independent data generative factors includeing shape, position X, position Y, scale and rotation.
(2018) is another synthetically rendered data with ground truth generative factors. It consists of 200, 000 images of a teapot object of size 64 64 with different colors and poses. The camera is centered on the object and the background is removed. Five different generative factors are independently sampled from their corresponding uniform distribution: azimuth U0, 2. elevation ~ [0, я /2) red ~ U0, I], green ~ U0, 1] and blue U[0, 1].
Teapots Eastwood & Williams
CelebA Liu et al. (2014) is a real-image dataset without ground truth generative factors. It contains ten thousand different celebrity identities, each with twenty images. Each image is annotated with forty face attributes such as: young, pale skin, bangs, straight hair, no beards, etc. The cropped 64 X 64 x face images are used in this study used in several earlier works.
Chairs Aubry et al. (2014) includes 86, 366 3D chair images rendered of 3D CAD models down- loaded from Google/Trimble 3D Warehouse. This dataset also comes without ground truth generative factors.
B MODEL. ARCHITECTURE:
Table I shows the encoder and the decoder architecture used in the experimental section, which is the same as in original 3-VAE. For a fair comparison, similar network architecture is used in all the methods and no parameter tuning is done as for number of layers, nodes, non-linearity and etc.
C TRAINING DETAILS
Table shows the training details which are used throughout the experiments. No tuning is done on these parameters. Table also presents the hyperparameters used in the experiments for Teapots data and metric Eastwood & Williams (2018). No tuning is done on the main network and regressor parameters. Regressor parameters are similar the ones used in Eastwood & Williams (018). Parameters that are not listed in this table are similar to T
D MORE LATENT TRAVERSALS RESULTS
Figures 5 and 6 show the latent traversals of Chairs and Teapots datasets. As seen in Figure SR-VAE can find more disentangled factors compared to 3-VAE and Feator- VAE methods. Three azimuth factors in Figure cover different angles in the space and so they are disentangled. Figure also shows the superior performance of our method disentangling factors for Teapots dataset. interestingly, both SR-VAE and Factor-VAL methods disentangle color factor in 3 latent representations. Each these latent representations cover different range color spectrum and hence they are disentangled.
Table 1: Encoder and Decoder architecture, z-dim: dimension of the latent representation; number of input image channel.
Table 2: Hyperparameters setting.
Parameter
value
Encoder
Decoder
Batch size
64
Input 64 binary/RGB image 4x4 conv, 32 ReL.,, stride 2, pad
Input 5--d
Latent dimension
10
FC dimm 256, ReLa 4 4x upconv, 64 ReLu, stride cony, 64 Rel.u, stride 2, pad 4x4 conv, ReLu, stride 2, pad 4x cons, 32 ReLu, stride 2, pad 4x4 cony, ne stride 2, pad
Optimizer
Adam
4x4 conv. 32 ReLu, stride 2, pad
Adam: hetal
0.9
4x4 cony, 64 Rel.u, stride 2, pad
Adam: beta2
0.999
4x4 conv, 64 ReLu,
2, pad 4x4 conv, 256 ReLu, stride
Learning rate
le-4
FC 256 (2x2 dim)
Decoder type
Bernoulli

---

Under review as a conference paper at ICLR 2020
Table 3: Hyperparameters setting for metric Eastwood & Williams 2018
experiments Teapots.
Parameter
value
Lasso:
0.02
Random Forest: #extimators
10
Random Forest: max-depth for 5 ground truth factors Learning rate
[12, 10, 3, 3] le-5
Size
Leg style
Back length
Azimuth
Leg length
Leg length
(a) SR- VRAV
Size
Back
length
Leg style
Azimuth
(b) 3-VAE
Size
Back
length
Leg style
Azimuth
(c) Factor-VAl
Figure 5: Latent traversals across each latent dimension where set 10 for Chairs dataset.
12

---

Under review as a conference paper at ICLR 2020
Rotation
Y-axis
Color
Rotation X-axis
Rotation X-axis
Color
Color
(a) SR- VAE
Rotation X-axis
Rotation X-axis
Rotation X-axis
Color
Rotation Y-axis
(b) 8-VAE
Rotation X-axis
Color
Rotation Y-axis
Color
Color
Rotation
X-axis
(c) Factor- VAE
Figure 6: Latent traversals across each latent dimension where d is to 10 for Teapots dataset.
13


Reviews:
[{"review_id": "rylCNq2htS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this paper present a novel that for unsupervised disentangled representation learning. The model, named sequential residual VAE (SR-VAE), gradually activates individual latent variables to reconstruct residuals. Quantitative and qualitative experiments show that the proposed model outperforms beta-VAE and Factor-VAE.  Since the training involves a sequence of model training,  SR-VAE certainly consumes more time than other VAEs. Minors: citations in the main text should be put in brackets. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SyxsGxuG5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors applied residual learning machenism in VAE learning, which I have seen such methods in Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (https://arxiv.org/abs/1506.05751), which basically applied the residual learning method in GAN. But the authors fail to discuss the relationship and difference with this paper.\nAlso, the best paper from ICML 2019 claimed that unsupervised learning method can not really disentangle the features. They claim \\beta-VAE, factor VAE is not good. The authors shall all discuess this point. Otherwise, it is not convincing to the readers.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgdDxAatS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nAuthors introduce a new VAE-based method for learning disentangled representations.\nThe main idea is to apply a “residual learning mechanism”, which resembles an autoregressive model, but here conditioning between sequential steps is done in both latent and input spaces. Namely, for each input, the method involves making several (one per *each* embedding dimension) sequential passes, where each pass takes the residual between the input and the current accumulated output (i.e. pixel-wise difference between two images) as well as the values of the sampled embeddings so far. \nAuthors report quantitative and qualitative evaluation with a set of reasonable baselines (FactorVAE, beta-VAE), and the results seem to be slightly better.\n\nDecision:\nThe writing quality is not great: there are frequent typos and not-so-precise formulations, and is at times hard to follow (see list below).\nThe method itself is not really well-motivated: there seems to be no formal justification provided, and the informal one is not very clearly explained in the paper. \nScalability of the method is clearly an issue. Many applications might require the size of embeddings to have hundreds of dimensions, which would mean that the given method cannot really be applied for non-toy problems. \nWith all those considerations in mind, I cannot recommend to accept this paper as is, thus the final rating, “reject”.\n\nAdditional comments / typos:\n* It would be worth seeing how this approach relates to auto-regressive models.\n* p2. “encoder maps … x to a latent representation q” - this sentence is not very strict. \n* p2. “.. is defined as following”: as follows?\n* p2. “.. in addition of b”\n* p3. “.. this reguralizer encourage”\n* p7: “… it reduces the solution space and improve training stability“: is there any sort of theoretical reasoning that could support this?\n* The reference format is not very readable, probably better to change to (Names, Year).\n\n<update>\nAuthors addressed some of my concerns (thanks!), thus I increased my rating slightly.\nThere is still a large concern wrt to the computational complexity. I kind of understand the argument about the hyperparameter tuning, but it seems not really fair to compare to a bruteforce parameter search (one can probably do some sort of bayesian optimization).\n</update>\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper that defines a “Residual learning” mechanism as the training regime for variational autoencoder. The method gradually activates individual latent variables to reconstruct residuals.

There are two main concerns from the reviewers. First, residual learning is a common trick now, hence authors should provide insights on why residual learning works for VAE. The other problem is computational complexity. Currently, reviews argue that it seems not really fair to compare to a bruteforce parameter search. The authors’ rebuttal partially addresses these problems but meet the standard of the reviewers.

Based on the reviewers’ comments, I choose to reject the paper.

Decision: Reject