Improving the Unsupervised Disentangled Representation Learning with VAE Ensemble
Authors: Nanxiang Li
Shabnam Ghaffarzadegan
Liu Ren
Venue:  (2021)
Keywords: Unsupervised disentangled representation learning
network ensemble
variational auto encoder

Abstract: Variational Autoencoder (VAE) based frameworks have achieved the state-of-the-art performance on the unsupervised disentangled representation learning. A recent theoretical analysis shows that such success is mainly due to the VAE implementation choices that encourage a PCA-like behavior locally on data samples. Despite this implied model identifiability, the VAE based disentanglement frameworks still face the trade-off between the local orthogonality and data reconstruction. As a result, models with the same architecture and hyperparameter setting can sometime learn entangled representations. To address this challenge, we propose a simple yet effective VAE ensemble framework consisting of multiple VAEs. It is based on the assumption that entangled representations are unique in their own ways, and the disentangled representations are "alike" (similar up to a signed permutation transformation). In the proposed VAE ensemble, each model not only maintains its original objective, but also encodes to and decodes from other models through pair-wise linear transformations between the latent representations. We show both theoretically and experimentally, the VAE ensemble objective encourages the linear transformations connecting the VAEs to be trivial transformations, aligning the latent representations of different models to be "alike". We compare our approach with the state-of-the-art unsupervised disentangled representation learning approaches and show the improved performance.

Under review as a conference paper at ICLR 2021
IMPROVING THE UNSUPERVISED DISENTANGLED REPRESENTATION LEARNING WITH VAE ENSEMBLE
Anonymous authors Paper under double-blind review
ABSTRACT
Variational Autoencoder (VAE) based frameworks have achieved the state-of-the- art performance on the unsupervised disentangled representation learning. A re- cent theoretical analysis shows that such success mainly due to the VAE im- plementation choices that encourage a PCA--ike behavior locally on data sam- ples. Despite this implied model identifiability, the VAE based disentanglement frameworks still face the trade-off between the local orthogonality and data re- construction. As a result, models with the same architecture and hyperparameter setting can sometimes learn entangled representations. To address this challenge, we propose simple yet effective VAE ensemble framework consisting of multi- ple VAEs. It is based the assumption that entangled representations in their own ways, and the disentangled representations are "alike" (similar up to a signed permutation transformation). In the proposed VAE ensemble, each model not only maintains its original objective, but also encodes to and decodes from other models through pair-wise linear transformations between the latent repre- sentations. We show both theoretically and experimentally, the VAE ensemble objective encourages the linear transformations connecting the VAEs to be triv- ial transformations, aligning the latent representations of different models to be "alike". We compare our approach with the state-of of-the-art unsupervised disen- tangled representation learning approaches and show the improved performance.
3
5
9
unique
11
12
13
14
15
16
17
18
INTRODUCTION
20
Disentangled representation learning aims to capture the semantically meaningful compositional representation of data (Higgins et al., 2018 Mathieu et al., 2018). and is shown to improve the
22
efficiency and generalization of supervised learning (Locatello llo et al. 2019), reinforcement learning
(Watters al., 2019). and reasoning tasks (van Steenkiste et al. (019). The current state-of-the- art unsupervised disentangled representation learning deploy the Variational Autoencoder (VAE) (Kingma & Welling 2013 Rezende et al., 2014). The main challenge to reduce the trade-off between learning disentangled representation and reconstructing input data. Most of the recent works extend the original VAE objective with carefully designed augmented objective to address this trade-off (Higgins et al. 2017 Burgeess al.. 2017, Kim & Mnih, 2018, Chen al., 2018 Kumar et al.] 3017). A recent study in (Locatello et al.) 2018) compared these methods and showed that their performance is sensitive to initialization and hyperparameter setting of the augmented objective function.
26
28
30
32
Recently, Duan el al. (Duan 2019) developed unsupervised model selection method named Unsupervised Disentanglement Ranking (UDR) to address the challenge of hyperparameter search
34
and model selection. UDR leverages the finding in (Rolinek et al., 2019) that the implementation choices of VAE encourage a local PCA-like behavior locally on data samples. As a result, disen- tangled representations by VAEs are "alike" as they are similar up to signed permutation transfor- mations. On the contrary, the entangled representations by VAES are "unique" as they are similar at least up to non-degenerate rotation matrices. UDR uses multiple models trained with different initializations and hyperparameter settings, and builds similarity matrix measuring the pair-wise similarity between the latent variables from different models. A higher score is given to the model that match its representations to many others models. The results show close match between UDR and commonly used supervised metrics, as well as the performance of downstream tasks using the latent representations.
35
36
57
39
41
43

---

Under review as a conference paper at ICLR 2021
45
Inspired by the findings from these studies, propose simple yet effective VAE ensemble frame- work to improve the disentangled representation by VAE. The proposed VAE ensemble consists of multiple VAEs. The latent variables in every pair of these models are connected through linear lay- 48 ers to force the latent representations in the ensemble to be similar up to a linear transformation. We show that the VAE ensemble objective encourages these pair-wise linear transformations to con- verge to trivial transformations, making latent representations of different VAEs in the ensemble to be "alike", thus disentangled. In this paper, we make the following contributions: (I) We introduce
46
49
50
51
52 simple yet effective VAE ensemble framework to improve the disentangled representation learning
53 using the original VAE. (2) We show in theoretical analysis that the linear transformations connect-
54 ing the latent representations of the individual models in the ensemble tend to converge to trivial transformations thus encourage disentangled representation, and verify this result with experiments. 56 (3) We evaluate our approach using the original VAE model, and show the improved state-of-the-art
55
57
performance across different datasets.
2
RELATED WORK
58
59
Variatioanl Autoencoder is a deep directed probabilistic graphical model consisting of an encoder 80 and a decoder (Kingma & Welling 2013 Rezende et al. 2014). The encoder (o(z|z)) maps the input data x R to probabilistic distribution the latent representation = E RRd and the decoder qo (2|z) maps the latent representation to the data space noted as go [x]=2, where & and 0 represent model parameters. The VAE objective is to maximize the marginalized log- likeliield of data. Direct optimization of this objective is not tractable and it is approximated by the evidence lower bound (ELBO) as:
61
62
63
64
65
(I) In practice. the first term is estimated by reconstruction error. The second term is the Kullback- Leibler divergence between the posterior (0(2(x) and the prior p(z) commonly chosen as an isotropic unit Gaussian p(z) ~ I).
LV (x|==) [log go [z][=]] (L(gx<z(x)) || p(=),
66
67
68
69
Disentangled representation by VAE has achieved the of-te-of-the-ar performance (Higgins et al. 2017 Burgess et al.) 2017; Kim & Mnih 2018 Chen et al.) 2018 Kumar et al. (017). despite the fact that the VAE objective only models the marginal distribution of the data instead of the desired joint distribution over data and latent variables. The reason for this success is the implementation choices of the VAE framework (Rolinek et al.J 2019). In practice, the latent variables in VAE often work in "polarized" modes. The "passive" mode is defined by (x) << and (x ss while the
71
72
73
74
75 "active" mode defined by (x) < 1. The "passive" latent variables closely approximate the prior and have little effect on the decoder. The "active" latent variables, on the other hand, are closely related to both the per sample KL loss and the decoder output. The "polarized regime" enables a reformulated VAE objective showing that VAES optimize trade-off between data reconstruction
76
77
78
79 and orthogonality of the linear approximation of decoder Jacobian locally around a data sample. identifiable disentangled latent space by 81 VAF. Furthermore, it was suggested that finding an appropriate "polarized regime" is dependent
This PCA-like behavior near data points encourages
82 on the initialization and the hyperparameter tuning of the state-of-the-an approaches. In this study,
83 we show that the proposed VAE ensemble aligns the "polarized regime" of individual VAE models towards the disentangled representation.
84
85
Model selection In practice, we often observe neural networks achieve similar performance with different internal representations when trained with the same hyperparameters (Raghu et al., 2017 Wang et al., 20188] Morcos et al. 2018). For the unsupervised disentanlged representation, as dis- cussed in (Locatello et al.) 2018 Duan et al. 2019). we often observe high variance in the perfor- mance from the model trained with the same architecture and hyperparameter setting. This poses a challenge for choosing the model in practice. [Duan et al.) (2019) proposed Unsupervised Disen- tanglement Ranking (UDR) to address this challenge. The extensive empirical evaluations on UDR using both the supervised metric measurement and the performance of downstream tasks validates 98 its effectiveness. They also confirm that disentangled representations are "alike" and entangled rep- resentations are unique in their own ways. The proposed VAE ensemble leverages this finding.
86
87
86
89
90
91
92
94
Identifiable VAE Built the recent breakthroughs in nonlinear Independent Component Analy- 96 sis (ICA) literature (Hyvarinen & Monikaa 2016; 2017;; Hyvarinen et al., 2019), Khemakhem et 97 al. show that the identification of the true joint distribution over observed and latent variables is
2

---

Under review as a conference paper at ICLR 2021
Encoder Flow
Expoora
X1
Diocutangee
Withi-sookl
resostaction
Linene Layer
Entingged aquoceteiio
212
T
Original VAES
Cross-mad
trusformation
Sapoots
Layer
Eig.
Decoder Flow
Unlabeled
Cros-atedel
Eapoota
21
75
training data
Linesr Loyes
X
Linear Layer
VAE Ensemble
Opyorg
Linear Layer
VAE
BeaVAE
FacEeVVE
VAT Ensemble
Figure 1: The proposed VAE ensemble consists of multiple original VAE models. The encoders of the VAEs in the ensemble generate imput encoding that can be linearly transformed among each other. The decoders of the VAEs in the ensemble reconstruct the input data from both their corre- sponding encoder and the linearly transformed encodings from other encoders. The x and axis of the circles the left hand side of the plot represent generative factors example. The aligned arrows with and axis show a model with disentangled representation and unaligned ones show model with entangled representation.
possible up very simple transformations (Khemakhem et al. 2019). They proposed identifiable VAE (iVAF) that requires a factorized prior distribution over the latent variables conditioned on an additionally observed variable, such class label almost any other observation. We believe the proposed VAE ensemble is related to such framework where the latent representation from one VAE model can be regarded as the auxiliary observations for another.
89
101
102
Ensemble learning The idea of ensemble learning is to combine multiple learning models (poten-
103
tially weak learners) to improve the task performance or robustness over a single model (Schapire 1990p. It achieves the improved performance by averaging the bias, reducing the variance thus pre- venting the over-fitting (Drucker al.) [994] [Breiman| 1996). Early works in neural networks have used the ensemble learning to achieve top performance in the related competition (Krizhevsky et al., 2012 Simonyyn & Zisserman 2014). In this work, we apply the ensemble learning to enforce the alignment among the latent representations of different models. This results in latent representations that are similar among each other the ensemble up to a trivial transformation.
104
105
106
107
108
110
3 THE VAE ENSEMBLE FRAMEWORK
As illustrated in Figure I the proposed VAE ensemble consists of n original VAE models with the same architecture but different initializations. also consists of x (n-1) linear layers connecting
112
113
the latent representations of every two VAE models. Each model in the ensemble maintains its original VAE objective Eq. [] In addition, loss used force mapping between latent representations via pair-wise linear layers (cross-model linear transformation). The decoder of each VAE model generates the input reconstruction from not only their corresponding encoder (within- model reconstruction), but also the linearly transformed encodings from other encoders (cross-model reconstruction). Overall, the VAE ensemble trained with the following objective:
114
115
116
117
118
119
((0.
E Bg (zy) go (x)2)))))
KL(gon(zzlla |l p(zn))
(2)
E S
||

---

Under review as a conference paper at ICLR 2021
where n is the number of models in the ensemble; Φ := (0ty) is the encoders parameters where dij represents the encoder of VAE, and its associated linear layer mapping the latent representation from VAE, to VAE, (notice that da represents the encoder parameters of VAE, only and its associated linear transformation can be be regarded as an identity transformation); 0 == (0:) represents the de- coder parameters of VAE;; and 2jj represents the latent representation of VAE, while 2ij represents the linearly transformed latent representation from VAE, to VAEj. is a hyperparameter to balance the effect of the estimation error between the latent representaitons. p(z;) ~ NN I) assume to be the prior as defined in the original VAE objective.
121
122
123
125
128
127
128
Comparing to the original VAE objective in Eq. the ensemble, L' V'V. can be written as:
[ the objective of each individual VAE model in
129
() = =oVAE()))) Φ)
Bussi (zs (x) Dlg (0(2))))) -
- Fl 1|2 - =11²]
Ensemble Regularization
where n is the number of models in the ensemble, oj stands for the parameters of the encoder and its linear transformation layers from other VAEs, and 2j represents the linear transformed latent representation of the encoding from other encoders.
131
132
133
In this form, the VAE ensemble regularizes each VAE model with additional terms on the encoder as 7 (=, ||ا - =||², and on the decoder as Σj=1 Bggs(z, 2) log (<<]))))] These regulariza- tions directly constrain the latent representations among different VAE models the ensemble to
134
135
be similar. In particular, for given input data, ||ا =|| encourages the encoders to generate similar encodings up to the linear transformation:: and Σÿ=1 (x)) [log (xr22)))] emphasizes the similar effect the data reconstruction from the latent variables such that the decoders can reconstruct the input data with both the original encoding z and the linearly transformed encoding 2j. As we shall discuss in the next section, together these regularizations encourage similar latent representation up trivial transformation by different models in the ensemble.
136
137
138
139
140
142
We also introduce the hyperparameter y to balance the trade-off between these two regularizations: higher value forces closer mapping between the encoders and reduce the cross-model reconstruction error of the decoders; lower value relaxes the mapping between the encoders and increases the cross- model reconstruction error of the decoders. As we show in Section both components are important and balancing the trade-off between them is important as the ensemble size increases.
143
144
145
146
Computational complexity It is common practice to train number of seeds per hyperparameter setting for the current state-of-the-art unsupervised disentanglement VAE models (Locatello et al.) Duan et al. 2019). Comparing to training original VAEs, the proposed VAE ensemble requires additional n x (n- - 1) linear layers. While this addition does not increase the size of the model much, the estimation of the linear transformations loss and the cross cross-model reconstruction losses grow with n x (n- - 1). which may be computationally expensive especially when n is large. That being said, the results in Section 5 show that the VAE ensemble achieves more stable results comparing to the current state-of the-rr- models. Also, its computation is highly parallelisable.
148
2018
150
152
153
154
4 THEORETICAL JUSTIFICATION
156
In this section, we present the theoretical analysis on why the proposed VAE ensemble can improve the disentangled representation. We start with analysing the l2 objective in Eq. 2 of the pair-wise liner transformations in the VAE ensemble, and show that: (1) the pair-wise linear transformations encourage similar "polarized" regime (see Sec. ( among the VAEs the ensemble; (2) the linear transformations are close to the orthogonal transformations. Based on these two properties, we then discuss how the cross-model reconstructions by the VAE ensemble encourage learning a disentan- gled representation over its entangled counterpart.
157
158
159
180
161
162
163
4.1
THE EFFECT OF LINEAR TRANSFORMATION BETWEEN LATENT REPRESENTATIONS
184
Let VAE, and VAE, be two different VAE models in the ensemble, and Mjt be the linear trans- formation that maps the latent representation of given input x by VAE, to the one by VAE, as
185
4

---

Under review as a conference paper at ICLR 2021
(z) ~ (z), ((j)g))iiaag( (
(z) to (z) ~ ((,((2), (z))(or(()))). In the following we
move the input notation from the VAE latent representations for simplicity (i.e. zz(z) is simplified
167
as zj). while keeping in mind that the analysis is based on the local latent representation of a given input x.
169
For VAE,, the l2 term of the VAE ensemble loss in Eq. 2 aims to find Mjs and 2j that minimize E 24 Mjl 1 where the expectation over the stochasticity of VAEz. We can write zi and 2j as 2 = f4 + E, and وj 24 + Ej. where E, ~ (0, diag(o and Ej ~ AO, (()g(()))) Hence using bias-variance decomposition, the 12 term can be written as:
171
172
173
min My. (Pp
E 1 M3
| - MaiMe,l B MiiMi 2 +E +E - 2ill²}
min My (i
(4)
| - M || M3 +c
min
(p || - M + [ + +
=
min
where the constant C₁ arises from the fact E3 sists of deterministic component of || M 2 and stochastic component of E 6j
p zi does not depend on M,s and zj. Eq. 4 con-
174
xi
||M l|
175
The deterministic component can be minimized by adjusting the parameters in VAE, such that its mean encoding 14j is optimized for any given Mj+- This simplifies analysis focus the stochastic component. Between My; and E, in this stochastic component, we separately optimize one while having the other fixed.
176
178
179
We start with fixed Mj and optimizing for Ej ~ NO, diag(or))) Notice that of is associated with VAE, objective. In (Rolinek al. 2019), the VAE objective is reformulated into the deterministic reconstruction, the stochastic reconstruction and the KL loss. The last two components define the stochastic loss of VAE. It is formulated as:
180
181
162
183
min V.
Σ X =
|DE
s.f.
ILLL= = C1, X
(5)
where X represents the dataset, D represents the local linear approximation of the Jacobian of the decoder with singular value decomposition as = UEVT. Furthermore, the KL loss LKL =
184
185
( + 0f log of - 1) can be simplified as L2K = Ex (n - log oj j - based the "polarized" regime of VAE. (Rolinek et al. 2019) shows that of act the precision control allowed for each latent variable where more intiuential ones receive more precision. Com-
186
168
bining the stochastic loss of the linear transformation in Eq. and the stochastic loss of the original VAE in Eq. 5
the overall stochastic loss on of can be formulated as:
190
min of
E c(
||l +
11 [|
s.t. E 1logozkk = C2.
(6)
where 03 is the kth element of az. Here we further simplify LKL with the L up to additive constants C2 when μj is fixed. In addition to the precision control of of on VAE,, this objective also aims find an optimal distribution of a3 that aligns the "polarized regime" among different VAEs. To see why, let ck be the kth column of Mít we then have E B|,M,,,,l1' Ex ||cl1 oîx- The Arithmetic-
191
183
194
Geometric (AM/GM) inequality sug-
gests that Ea 112 03 2 n [II |ck 11 03,, 1/n =n (IIe [Icall") 1/mz
exp(-C), where the equal-
196
ity is achieved when ||c...1² 3m = ||c,11² o0 for any m * n. This suggests that latent variables
197
with high ||cx|| mapping from Zj to z; will have smaller variance. Hence, these latent variables in Zj encouraged to stay in the "active" mode. On contrary, the latent variables that do not share sim-
198
199
ilar generative factors between zj and z; will be assigned larger variance, and being pushed towards the "passive" mode.
200
201
5

---

Under review as a conference paper at ICLR 2021
Now fix the optimal distribution of oz. and optimize for Mjt. Since E; ~ [0, diag(a))), this objective can be understood as optimally rotating the latent space of VAE, such that the stochastic component in Eq. iis minimized. Specifically, we have the following objective:
203
min | min R
|M
Md
where R is an orthogonal transformation. Let dg. be the kth column of MsgR. Similar as before, the 205
AM/GM inequality suggests ||M, e Ex ||ck|] II Jf 2 II |l exp(-C3). Hadamard's
208
incquality suggests that II || 2 [det(M); and the equality is satisfied when dgs pair- wise orthogonal. This can be understood from the geometric perspective where Ih ||c lI gives an (M, x x E 1)
208
209
) As result, the optimization of My. will lead to an orthogonal transformation. Together the optimization of Eq. [ and Eq. 7 encourages the align- ment of the "polarized regime" among different models under orthogonal linear transformations. They force different models in the ensemble to capture the same mixture of the generative factors. In the next section, we discuss the effect of the cross-moded reconstruction in the VAE ensemble that encourages the disentangled representation over the entangled ones.
upper bound on Volume M,
210
211
213
4.2
THE EFFECT OF MODSS- RECONSTRUCTION
216
In an entangled representation, each latent variable captures a mixture of generative factors in its unique way. Since different generative factors typically have different effects on data variations Duan et al. 2019). the orthogonal transformation from one entangled representation Zj to another one 2i introduces different encoding variance. Some of the transformed latent variables in M, carry larger variance comparing to the corresponding ones in Zi. This discrepancy leads to larger cross-model reconstruction of VAE, than the within model reconstruction. This error forces both VAE, and VAE, to adjust their representations until the effect the data reconstruction by indi- vidual latent variables matches between Mi and Z₁. The process applies to all models in the ensemble and eventually leads to one-to-one mapping of latent variables between different mod- els, where My becomes a trivial transformation (signed permutation matrix). In particular, one of the models in the VAE ensemble learns disentangled representation, other models in the ensemble will converge to it. This is because the orthogonal transformation from an entangled representation disentangled representation introduces larger cross-mode model encoding variance due the mixture of different generative factors in the former, thus larger cross-model reconstruction by the disen- tangled model. On contrary, the orthogonal transformation from disentangled representation to an entangled representation would not introduce larger cross-mode encoding variance than the within model encoding, thus similar cross-model reconstruction as within model reconstruction by the en- tangled model. Such a gap encourages the entangled representations to align with the disentangled representation. We illustrate the geometric interpretation of such a case in Appendix C
217
218
219
221
223
225
226
227
228
229
230
232
234
From these discussions, we conclude that the VAE ensemble encourages different individual mod- els to capture similar generative factors, thus learn representations that are "alike" up to a trivial transformation. In the next section, we verify these analytic results with experiments.
236
238
5
EXPERIMENTS
239
Our experiments are designed to confirm the discussions in the previous sections. Particularly we ask the following questions: Q1: Do the linear transformations in the ensemble converge trivial transformation? Q2: Do the VAEs in the ensemble work in similar "polarized" regime? Q3: Does VAE ensemble improve the unsupervised disentangled representation learning, and what is the ef- fect of ensemble size? Q4: What are the effects of the cross-model reconstruction loss, the linear transformation loss and the hyperparameter y in the VAE ensemble objective?
240
241
242
244
245 We analyze the inner working of the proposed VAE ensemble using the benchmark dSprite dataset Matthey et al. 2017) with fully known generative processes, and the real-world CelebA dataset (Liu)
246
et al. 2015) with unknown generative process. Furthermore, for dSprites dataset, we compare
248
results with the original VAE model and the state-of-the-aat disentanglement VAE models including
3-EAE (Higiins et al.) 2017), Factor VAE (Kim & Mnih 2018). TC-VAE (Chen et al. 2018) and
250
DIP- VAE (Kumar et al. 2017). We use two widely used supervised metrics including Factor VAE
metric (Kim & Mnih 2018) and DCI Disentanglement scores (Eastwood & Williams, 2018) as the

---

Under review as a conference paper at ICLR 2021
log(OrO)
VAE E VAE
VAE E
Es
VAE
B-VAE
FactorVAE
DIP-VAE4
DIR-AE-I
TC-VEE
Figure 2: Comparing the DiO of linear transformations in the VAE ensemble (y=10) with the one between well-trained individual VAEs, as well as the well-trained individual state-of-the VAE models. The latent dimension for all models is set to 10 and evaluated on the dSprite dataset.
quantitative measurements. They are shown to correlate with other common supervised metrics (Locatello et al. 2018). For example. FactorVAE metric and 3-VAE metric (Higgins et al. 2017) çapture similar notions, while DCI Disentanglement and Mutual Information Gap (MIG) (Chen
263
et al., 2018) capture similar notions. In addition, DCI Disentanglement is closely related to the
255
unsupervised model selection method UDR (Duan al., 2019). For CelebA dataset, we show the latent traversal visulization as a qualitative measurement in Appendix D the experiments in Appendix D
256
We provide the details of
257
258
Q1: We use the Distance to Orthogonality (DtO) (Rolinek et al. 2019) to check if the linear trans-
259
formations in the ensemble converge signed permutation matrix during training. DiO is the Frobenius norm of the difference between a matrix M and its closest signed permutation matrix
260
261
P(M). It is solved with -itteer linear programming (MILP) formulation. The details on DIO can be found in Appendix B In Figure [ we show the DtO estimation of the linear transfor-
262
263
mations in the VAE ensemble of different ensemble size for the dSprite dataset. We show the mean
264
and standard deviation of DrO across all linear transformations over 10 different runs. Furthermore, we compare these results with a baseline measurement where DtO is estimated for the linear trans- formations between the mean latent representations of well-trained individual models. Specifically, we use ten well-trained individual models and report the mean and standard deviation the DiO estimations. As seen in the figure, the VAE ensemble models with different ensemble size all ap- proach to trivial transformations between the individual models, while other VAE models do not have such property. In Fig. 6 show that during training, the VAE ensemble remains maintains low DIO while the original VAEs do not have such property. A similar result for models trained on the CelebA dataset with different latent dimensions shown in Fig. Further discussion on these results are provided in Appendix [E]
265
266
268
269
270
271
272
273
274
Q2: To check the models in the VAE ensemble work in similar "polarized" regime, we estimate
275
the relative error between LKL L and L=KE as A LKI_L for each latent variable. Smaller A indicates closer matching between LRT. and L=KI of latent variable, thus more "active". Figure
T.ic
276
277
((a) and B) show log(A) of the 10 latent variables of individual models in VAE_E, and VAE_E, with different y settings trained on the dSprite dataset. The results show that individual models in the ensemble do work in similar "polarized regime'. In Figure ((a,) we also compare the VAE ensemble with the 3- 3-V where 4. This setting was found previously to be the optimal setting for the dSprite data for B-V (Higgins et al. 2017]. We see that the VAE ensemble encourages
278
279
260
281
262
more "active" latent variables than 3-VAE, When we compare Fig. ( and Bb), we see that as the ensemble size increases, individual models are forced to have more "active" latent variables by decomposing the generative factors. This can be observed in the latent traversals shown in Appendix
263
264
265
E The dSprites dataset contains five ground truth generative factors. The VAE.E, models can
have up to eight "active" latent variables depending on input, and these representations capture a decomposition of the ground truth generative factors.
267
288
7

---

Under review as a conference paper at ICLR 2021
12
Ensembtee 1 1
inverble pe Dosemble
trsendle y-17
Ersembl
Drcemble y
Ensembte 5
agiiti -t-ella)
Ensemete 10
1m10
JIVAE
-10-
-10
Latent Dimensiors
Latent Dimanstors
(a)
(b)
Figure 3: The "polarized regime" comparison between models in the VAE ensemble. The latent dimension is set to 10 and the results are over 10 runs of training on the dSprite dataset. (a) The "polarized regime" comparison by log( Lkr_- of each latent variable of the two models in LKL VAE_Ez as well as 3-VAE model. (b) Similar to (a) but for the three models VAE_E3.
Factor VAE Metric
Individual Model
VAE Ensemble (VAE_E)
VAE.
0..03
2=1
7=5
2=10
B-VAE (3=4)
0.665-10.089
VAE Eg
0.71110.106
0.736:10.085
0.741
VAE (0=40)
0.764.10.075
VAE. Ea
0.794110030
0.792-10.075
0.82110.066
DIP- VAE-I (S
0.638=0.1008
VAE E4
0.833=00037
0.790
0.800#0.007
DIP- VAE II (5o==)
0.676=1.122
VAE.Es
0.82810.0166
0.786=40.051
0.739:00008
VAE (B=4)
0..08
DCI-Disentanglement Metric
Individual Model
VAB Ensemble (VAE_E)
VAE.
0.143+0..00
7=1
7=5
7=10
B-VAE (3=4)
0.0067
VAE Eg
0.1761:0.03
0.243:0.029
0.201
FactorVAB VAE (0=40)
0.253:10002
VAE. Ea
0.214410.64
0.236:10.051
0.31110.060
DIP- VAE-I (S)=5)
0.049-10.017
VAE E4
0.240:10.059
0.223+0.045
0.251
DIP VAE- II (Aod=))
0.106:0.002
VAE.E,
0.242:10.032
0.244+0.039
0.196::0.050
VAE (B=4)
0.303=0.052
Table 1: Comparison between the proposed VAE ensemble, the original VAE, and the current state- of-the-art disentangled VAE models. We report the mean and standard deviation of the FactorVAE metric and and DCI Disentanglement scores over 10 runs trained on the dSprite data.
Q3: In Table I we compare the disentangled representation performance between the proposed VAE ensemble and the state-of-the-art models. For the VAE ensemble, we report the performance of the first model in the ensemble. We also report the results for the VAE ensemble with different ensemble size and ry values. As shown in the table, the VAE ensemble significantly improves the
290
292
293
performance over the original VAE model. In many settings, the VAE ensemble achieves similar or better performance over the ttate-of-the-a models. In Table 2 we evaluate the consistency among the models in the ensemble by reporting the standard deviation of the evaluation metries using different models in the same ensemble. The small values confirm that different models in the ensemble learn similar latent representations. Furthermore, Table | shows the joint effect of ensemble size and " setting. When 7 = 1, the performance of VAE ensemble increases as the ensemble size increases, indicated by the higher mean and smaller variance both the Factor
295
296
297
289
and DCI Disentanglement metrics. This behavior is consistent with the characteristic of ensemble learning where the increase in performance becomes smaller as the size of ensemble increases. However, as y increases, having larger ensemble size reduce the performance. We believe this
301
8

---

Under review as a conference paper at ICLR 2021
VAE_E2
VAE_ES H3
VAE_E,
VAE_ES
(-=1)
0.0019
0.0090
0.0048
0.0081
FactorVAE Metric
(7=5)
0.0058
0.0064
0.0089
0.0163
(7=10)
0.0060
0.0046
0.0147
0.0139
(7=1)
0.0024
0.0026
0.0028
0.0036
DCI-Disent Metric
(7=5)
0.0037
0.0054
0.0049
0.0040
(7=10)
0.00T3
0.0024
0.0041
0.0042
Table 2: The comparison between individual models in the same ensemble. We report the average of the standard deviation of the metries by individual models in the ensemble across 10 runs.
0.8
FactorVAE Metric
80
<1g Y-
CTD 5
Y
0.5
W.O.
V2C_C2
VAC_C3
VAC_Ca
VAC_ Cs
Figure 4: Ablation study to understand the effect of cross- moosl reconstruction and linear transfor- mation in the VAE ensemble objective using the FactorVAE metric. (w.o. CR without cross-model reconstruction loss; w.o. LT without linear transformation loss; org - original VAE ensemble loss)
is due to the increased difficulty of balancing between the cross-model and within model objectives of VAE ensemble for larger ensembles. The reduced alignment of the latent representations among different models can also be seen in Table 2 where difference in the performance among individual models in the ensemble increases as ensemble size increases.
303
304
306
Q4: We conduct the ablation study to further understand the effect of the linear transformation loss
307
and the cross- cross-model reconstruction loss in the VAE ensemble objective. As shown in Fig. remov-
308
ing either component leads lower FactorVAE metric for the VAE ensemble. Without the linear
309
transformation loss, the performance of VAE ensemble decreases significantly across different en- semble sizes. Without the cross-model reconstruction loss, the performance of VAE ensemble also decreases but the gap becomes smaller as increases. This matches the discussion in Section that higher 7 forces closer mapping between the encoders and reduce the cross-model reconstruction er- ror of the decoders. However, this also reduces the effect of cross- moosl reconstruction as discussed in Section 4.2 similar result is also found for the DCI Disentanglement metric as shown in Fig- g in Appendix Overall, the results from the ablation study confirms the importance of both the linear transformation loss and the cross-model reconstruction loss in the VAE ensemble objective.
310
311
312
313
314
316
316
317
CONCLUSION
318
In this study, we propose simple yet effective VAE ensemble framework consisting of multiple original VAEs to learn disentangled representation. The individual models in the ensemble are connected through linear layers that regularize both encoders and decoders to align the latent repre-
319
320
321
sentations to be similar up to linear transformation. We show in theory that the regularization by the VAE ensemble forces the linear transformations to be trivial transformations and show improved performance the unsupervised disentangled representation learning. The theoretical discussion in Section iis based on the original VAE objective, and our experiments also focus on the ensemble with original VAE. We believe such framework can be extended to other disentangled VAE models,
322
323
324
325
or even a mixture of different VAE models, as long as the regularization by the ensemble does not conflict with the augmented objective of these models.
327
328

---

Under review as a conference paper at ICLR 2021
REFERENCES
330
Leo Breiman. Bagging predictors. Machine learning, 24(2): -100 1996.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in / -VAE. 2017
332
333
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating Sources of Disentangle- ment in Variational Autoencoders. (ICLR):1-13, 2018.
334
335
Harris Drucker, Corinna Cortes, Lawrence Jackel, Yann LeCun, and Vladimir Vapnik. Boosting and other ensemble methods. Neural Computation, ((6):1299-13111 1994.
337
Sunny Duan, Loie Matthey, Andre Saraiva, Nick Watters, Chris Burgess, Alexander Lerchner, and Irina Higgins. Unsupervised model selection for variational disentangled representation learning. In International Conference on Learning Representations, 2019.
338
339
340
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. ICLR, 2018.
342
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 3-Vae: Learning Basic Visual Concepts With Con- strained Variational Framework. ICLR 2017, (July):1-13, 2017.
343
344
345
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loie Matthey, Danilo Rezende, and Alexander Lerchner. Towards Definition of Disentangled Representations. arXiv, pp. 1-29, 2018.
347
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Advances in Neural Information Processing Systems, pp. 3765-3773, 2016.
349
350
App Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 859-868, 2019.
352
AJ Hyvarinen and Hiroshi Morioka. Nonlinear ica of temporally dependent stationary sources. Proceedings of Machine Learning Research, 2017.
354
355
lyes Khemakhem, Diederik P Kingma, and Aapo Hyvärinen. Variational autoencoders and nonlin- ear ica: A unifying framework. arXiv preprint
356
2019.
357
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. Proceedings of Machine Learning Research, 80:2649-2658, 10-15 Jul 2018.
359
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
380
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
362
384
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. CoRR, 0bb//1111.00843 2017.
385
Ziwei Liu, Ping Luo. Xiaogang Wang. and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
367
368
Francesco Locatello, Stefan Bauer, Mario Lucic. Sylvain Gelly, Bernhard Schölkopf, and L G Dec. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representa- tions. arXiv, pp. 1-33, 2018.
389
370
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf. and Olivier Bachem. On the fairness of disentangled representations. In Advances in Neural Informa- fion Processing Systems, pp. 14584-14597, 2019.
372
373
10

---

Under review as a conference paper at ICLR 2021
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. arXiv preprint arXiv:1812.02833 2018.
376
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http:///ihhbb..coommmmmmm///ssss 2017.
378
377
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, pp- 5727-5736, 2018.
378
379
380
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pp. 6076-6085, 2017.
381
383
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint arXiv:1401.4400 2014.
384
385
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea direc- tions (by accident). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12406-12415, 2019.
366
368
Robert E Schapire. The strength weak learnability. Machine learning, 5(2):197-227, 1990.
389
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.115: 1409.156,, 2014.
390
391
Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? In Advances in Neural Information Processing Systems, pp. 14222-14235, 2019.
392
353
394
Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. In Advances in Neural Information Processing Systems, pp. 9584-9593, 2018.
395
397
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P Burgess, and Alexander Lerchner. Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-driven exploration, arXiv preprint arXiv: 0905. 09275, 2019.
399
400
401

---

Under review as a conference paper at ICLR 2021
A
TECHNICAL LEMMAS
403
In this section, we give the lemmas used in the theoretical discussion in Section
404
Lemma 1. (Jensen's s inequality) If g(x) is a convex transformation on æ, then this convex transfor- mation of a mean g(E(x)] is less than equal to the mean of the convex transformed value Big(z)]; is a simple corollary that the opposite is true of concave transformations.
406
Lemma 2. (AM-GM inequality) As extension Jensen's inequality, given a list of non-negative real numbers 02,....... Ens the arithmetic mean of this list 1 -i== 3i is greater than or equal
408
the geometric mean of the same list ([ ==)=;; and further, the equality holds if and only if Enn
410
11
Lemma (Hadanard's inequality). if M is the matrix having columns Co then (det(M)) S II ||all; and the equality in Hadamard's inequality is achieved if and only if the vectors are 413
orthogonal.
B DISTANCE TO ORTHOGONALITY (DTO)
414
415 In this section, we introduce the detail of Distance to Orthogonality (DiO) that used in our exper- iment to check if the linear transformations in the VAE ensemble approach trivial transformations. 417 This measurement also used in [Rolinek et al. 2019) for similar purpose. DIO the Frobe- nius norm of the difference between square matrix M and its closest signed permutation matrix
419 P(M). Finding P(M) can be formulated as a mixed sxeedinteger linear programming (MILP) problem following:
Σ MM P(M)
min P
s.t.
P(M)s E {-1,0,1), v(i.))
Σ |Pagl 11
(8)
Σ Pa,l-= =1,
Vi
By introducing new variables Piy,, E {0, 1} and Dij- [Mij P(M)ish can reformulate 422 above optimization problem as:
min Σ Das
8.l.
(P. Pr.j) Mi. S Digr V(i,j)
Mijj (Pit. PT.)) 1 Digi viij)
(9)
[(P -PTj)) 1,
Vj
Σ (Pt PT))=1 =1
Vi
Using this optimization formulation, DoT of a given matrix M E Raxn is defined as:
Dol
Σ
Ma. P(M)ial
(10)
n
12

---

Under review as a conference paper at ICLR 2021
Transtonmed to Entangled Representation Space
Transformed to Disentangled
Disentangled Representation
Entangled Representation
Representation Space
Figure 5:
Geometric interpretation of the cross-model reconstruction between a disentangled repre- sentation space and an entangled representation space.
C GEOMETRIC INTERPRETATION OF THE EFFECT OF CROSS-MODEL RECONSTRUCTION
424
Given a disentangled and entangled latent representation space, Fig. illustrates the effect of the cross-model reconstruction by VAE ensemble. The left part shows the orthogonal transformation from disentangled representation to an entangled space, and the right part shows the transfor- mation in the opposite direction. As shown the figure, the orthogonal transformation from the disentangled representation to the entangled space does not introduce larger variance than the en- tangled representation. Hence, we can expect similar cross model reconstruction and within model
426
427
428
429
431
reconstruction. However, the transformation from the entangled representation to the disentangled space introduces larger variance (yellow shaded area over blue area on the right) than the disentan- gled representation. This leads to larger cross-model reconstruction by the disentangled model.
433
D MODEL ARCHITECTURE AND TRAINING DETAILS
435
We conducted our experiments, including training and evaluating the current state-of-the- art disentanglement models as well as evaluating the proposed VAE ensembles, using the d slissetttgleeee ement lib oppen-ouucee library (Locatello al. 2018).
436
437
438
Table 3 shows the encoder and the decoder architecture of the VAE model used in our experiments. This architecture is the same as the one used in the original 3 3-VAB Higgins al. (2017).
439
440
Encoder
Decoder Input IRd
Input 64 binary/RGB image 4x4 conv, 32 ReLu, stride 2, pad
FC 256.,, ReLu
4x4 conv, 32 ReLu, stride 2, pad
4x4 upconv, 64 ReLu, stride 1 4x4 conv, 64 ReLu, stride 2, pad 4x4 conv, 32 ReLu, stride 2, pad
4x4 conv, 64 ReLu, stride 2, pad
4x4 conv, 64 ReLu, stride 2, pad 4x4 conv, 256 ReLu, stride FC 256 x (2xd)
4x4 conv, 32 ReLu, stride 2, pad
4x4 conv, ne, stride 2, pad
Table 3: Encoder and Decoder architecture, dimension of the latent representation; ne: number of input image channel (For dSprites dataset nc 1. for CelebA dataset = 3).
Table 4 shows the hyperparameters setting used throughout the experiments. These parameters are fixed for all the experiments.
442
E ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present the additional results including the DtO and "polarized regime" analysis the models trained on the CelebA dataset similar to the ones conducted dSprite dataset in Section 5 the ablation results with DCI-Disentanglemeet metric and the DtO estimation; and the 446
444
https://gi http://gttttutoccc reseaagreeeeeeerrheee sentangl ement_1in

---

Under review as a conference paper at ICLR 2021
Parameter
value
Batch size
64
Latent dimension
10
Optimizer
Adam
Adam: betal
0.9
Adam: beta2
0.999
Learning rate
le-4
Table Hyperparameters setting.
VAE VAE_E;
+ VAE Es
VAE
+ WAE
Distance to Orshogonelliyy (DoT)
E,
+
-13
VAE
+ WAE_Es
VAE_F,
-20
logiDoTI
3.
-15
10
15
20
25
55
10
15
20
Epochs
Epochs
(b)
Figure 6: Characteristics of the linear transform between latent representations. The latent dimen- sion is set 10 and the results are over 10 runs of training on the dSprite dataset. (a) Comparing the DtO of linear transformations in the VAE ensemble (7=10) and the one between original VAEs. (b) VAE ensemble (-=10) with different ensemble size all achieve small DtO of the linear transfor- mations between the models.
447 latent traversal of the trained model on both dSprite and CelebA dataset along with further discuss the effect of VAE ensemble the latent representation.
449 E. CHARACTERIZATTIN THE LINEAR TRANSFORMATION VAE ENSEMBLE
In Figure 6 we show the DiO estimation of the linear transformations in the ensemble during train- ing for the dSprite dataset. transformations over 10 different runs. Furthermore, where DtO estimated for the linear transformations between original VAEs. Specifically, we train different VAES separately and estimate the DtO of the pairwise linear transformation among these models during training. Similarly we report the mean and standard deviation of these DtO es- timations. As seen in the figure, the VAE ensemble models with different ensemble size all approach to trivial transformations between the individual models, while the original VAEs do not have such property. similar result also found in models trained for CelebA dataset. Similar to the results in Figure 6 we observe decreased DtO of the linear transformations in the VAE ensemble during training.
451
report mean and standard deviation of DIO across all linear
compare these results with VAE baseline
453
455
456
457
458
459
460
461
We also compare models trained with different latent dimension size. We observe decreased DtO as the latent dimension of the model increases Figure 7 This is because, as discussed in the main paper, the VAE ensemble encourages more "active" latent variables. Models with higher latent dimension likely to learn a decomposition of generative factors. As a result, the alignment of the latent variable between different models easier thus the linear transformations between the latent representations is closer to the trivial transformation. On the contrary when there are less latent variables in the model than the generative factors, some of the latent variables will capture more than a single generative factor. As result, the one-to-one mapping between the latent variables of different models will not lead trivial transformation.
462
464
486
488
14

---

Under review as a conference paper at ICLR 2021
0.30
y-10
y-10
y-10
VAE 32d ym5
VAE 16d ym5
VAE_EI 10d y
Distance to Orthogonality (DoT)
VAE, E2 32d y-1
VAE E2 16d y-1
VAE, F2 10d y-1
0.25
0.20
0.15
0.10
0.05
0
5
10
15
20
25
30
35
40 45 Epochs
50
55
60
65
70
75
Figure 7: Distance to Orthogonality (DtO) measurement of the linear transforms between latent representations in VAE E, during training on the CelebA dataset. We also compare models with different latent dimensions 10, 16 and 32 and the results are averaged over 5 runs. In the figure legend, we use "VAE_E, nd g" to represent VAE Ensemble (VAE_E model with i individual VAE models, n latent dimensions and 7 value equal to g.
In Figure 2 and Figure 10 we show the "polarized regime" estimation for models in VAE_Ez and
VAE trained for CelebA datasets, respectively. Similar to the results Figure 3 individual models in the VAE ensemble tend to have similar 'polarized regime", and higher enforces the "po-
471
larized regime" by separating "passive" latent variables from the "active" ones. When we compare
473
between VAE Eg and VAE Eg. we observe increased "active" latent variables similar to the result
on dSprite dataset in Section 5 More importantly, as discussed earlier, latent variables in a model than single generative factor, especially for a complicated real-world dataset such as CelebA. This makes the linear transformation between the latent representations less trivial. As the latent dimension size grows, such constraint relaxed and the linear transformations are closer to trivial.
475
with limited latent dimensions need capture
477
478
479
These additional results confirm the conclusion in Section (1) as the ensemble size increases, DtO increases due to the difficulty of aligning the latent representations among different models; (2) as the model latent dimension increases, DtO decreases due the increased model capacity, and encourages the one-to-one mapping between latent variables in different models; (3) hyperparameter y does not affect DtO significantly, but plays an important role on separating "active" and "passive" latent variables, especially when the latent dimension is large enough.
480
461
462
483
484
Furthermore,
we believe the DIO measurement of the linear transformation in VAE ensemble could
466
be useful indicator for latent dimension size. As shown in Figure and Figure 9 when the latent dimension is sufficient for a given dataset, the DIO the linear transformation is small and some latent variables pushed to "passive" mode.
468
E.2 ABLATION STUDY
Similar to the ablation result shown in Section here we show the same ablation study using the DCI
Disentanglement metric Fig. a(a) as well as the DtO measurement b(.). Similar as the results of
482
the VactorVAB metric in Fig. 4 removing either component leads to lower DCI Disentanglement
metric for the VAE ensemble. Without the linear transformation loss, the performance of VAE
494
ensemble decreases significantly across different ensemble sizes. Fig. ( shows that for VAE
15

---

Under review as a conference paper at ICLR 2021
0.35
0.30
DCI Disentanglemen
0.25
0.20
0.15
yrg-y
y
0.10
0gg Y=
0.05
W.O. CR y
W.O. LT
0.00
VAE_Ep Ez
VAE_Ea Ea
VAE_E+ E4
VAE_ Es
(a) Comparison on the DCI- OCI-Disentanglement score
bogiDal
01g Y=
0rg y 10
y=
W.O. CR Y
W.O. CR y-
LT
VAE E2
VAE_
VAE_Ea
VAE_Ey
(b) Comparison on the Distance to Orthogonality (DiO)
Figure 8: Ablation study to understand the effect of cross-model reconstruction and linear transfor- mation in the VAE ensemble objective using the DCI Disentanglement metric and DtO. (w.o. CR without cross-model reconstruction loss; w.o. LT without linear transformation loss; org original VAE ensemble loss)
ensemble without the cross-model reconstruction, the linear transformations among models are close to trivial transformation (signed permutation). This implies the orthonormal transformation the linear transformations. This result further supports intuitive justification in Appendix that the cross-model objective encourages entangled models to align to disentangled models. Indeed, we see that adding the cross-model reconstruction further reduce the DıO of the linear transformations among the models in the ensemble.
437
499
501
E.3 LATENT TRAVERSAL
In this section show the latent traversal of models trained on both dSprites and CelebA datasets. For fixed input image, to extract the latent traversal we change the value a single latent variable 2; in the corresponding encoding, and observe the generated output image to understand the effect 506 of 2, The range of the value are usually chosen to be from -3 to due to the standard Gaussian prior.
504
508 In Figure II we show the latent traversal for both VAE E, and single VAE model with 10 latent dimensions trained dSprites dataset. Three images shown in the last column of each block
16

---

Under review as a conference paper at ICLR 2021
used input. Both models are able to capture certain generative factors of the data including position, shape, rotation and scale. In Section we argue that the representation by VAE ensemble encourages more "active" latent variables, thus capture decomposition of the ground truth gen- erative factors. Especially from the "polarized regime" estimation in Figure 3 we observe that some latent variables in the VAE ensemble in-between "active" and "passive" modes. This suggests that the VAE ensemble model generates input-dependent factors based on the input complexity. In Figure II we observe this behavior highlighted with color boxes. The traversal the second la- tent variable 22 shows that an ellipse shape does not lead to an "active" latent variable. However, both heart and square shape lead to an "active" latent variable that changes the output. In contrast, the single VAE model does not have such behavior where the "active" modes are consistent across different input data.
511
513
515
516
517
518
519
520
In Figure 12 and Figure 1 we show the latent traversal for both VAE_B, and single VAE model with 16 latent dimensions trained on CelebA dataset, respectively. In this real-world dataset, the generative factors are unknown. We observe different factors including background, azimuth, gen- der, hair style being captured by both models. Similar as before, the single VAE model maintains similar "active" mode for all latent variables where similar traversal patterns observed for both input images. However, VAE E2 shows semantically consistent but input-dependent "active" mode. This is translated into different traversal effects and realistic and sharper images by VAE Eg, especially for the first input image that is less common in the dataset. We believe this is important towards a meaningful compositional latent representation learning.
521
522
524
526
528
529
Overall, the latent traversal results in this section confirm the findings on the inner working of the VAE ensemble shown in the previous section well the discussion in Section 5
530
531
17

---

Under review as a conference paper at ICLR 2021
Nolla "Lob.a
(a) VAE.E, with latent dimension of 32, CelebA data.
Teeenth e
Seeenbe 2
Sweer e yo5
Deerb 2-5
Doamb a y=20
Creamb 2 y020
Valle woll
Litel.
(b) VAE_E, with latent dimension 16, CelebA data.
Deent
2y-00
Willee -/
(c) VAE.E, with latent dimension of 10, CelebA data.
Figure 9: The "polarized regime" comparison between models in VAE.E. The results are over runs of training the CelebA dataset.
18

---

Under review as a conference paper at ICLR 2021
20133
(a) VAE.E, with latent dimension of 32, CelebA data.
(b) VAE_F3 with latent dimension 16, CelebA data.
(c) VAE.E; with latent dimension of 10, CelebA data.
Figure 10: The "polarized regime" comparison between models in the VAE on the CelebA dataset.
19

---

Under review as a conference paper at ICLR 2021
Traversal
230
21 22
21
24
28 29
Z10
21
Z10
(a) VAE_Ez
Treversal
210
(b) Single VAE
Traversal
22
B- VAE
Traversal
24
25
27
22
210
27
21
ZI
25
28
20
2M
21 22
23
(d) VAE
Figure 1I: Latent traversal on three different input images using VAE_E,, a single VAE and state-of-the--rt VAE models with 10 dimensional latent representation. The three input images ellipse, heart and square shapes as shown the last column.
20

---

Under review as a conference paper at ICLR 2021
Traversal
(c) DIP-V
Traversal
DIP-VAE-II
Traversal
25
Z10
21 2I
D
24
22
22
22
23
24
25
(g) TC-VAE
Figure 11: (cont.) Latent traversal on three different input images using E2, single VAE and the state-of-the-art VAE models with 10 dimensional latent representation. The three input images are ellipse, heart and square shapes as shown in the last column.
21

---

Under review as a conference paper at ICLR 2021
Traversal
Z1
Z2
Z3
Z4
Z5
Z6
Z7
Z8
Z9
Z10
Z11
Z12
Z13
Z4
Z15
Z16
Traversal
Z1
Z2
Z3
Z4
Z5
Z6
Z7
Z8
Z9
Z10
Z11
Z12
Z13
Z14 Z15 Z16
Figure 12: Latent traversal on two different input images of CelebA dataset using VAE_ with latent dimension of 16.
22

---

Under review as a conference paper at ICLR 2021
Traversal
5
9
3
8
9
Z1
Z2
Z3
Z4
Z5
Z6
Z7
Z8
Z9
Z10
Z11
Z12
Z13
Z14
Z15
Z16
Traversal
Z1
Z2
Z3
Z4
Z5
Z6
Z7
Z8
Z9
Z00
Z11
Z12
Z1
Z14 Z15 Z16
Figure 13: Latent traversal on two different input images of CelebA dataset using a single VAE with latent dimension of 16.
23


Reviews:
[{"review_id": "nRK0FwFceL9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a simple and effective technique to improve disentanglement by coupling the latent spaces of different VAE models. It builds on Duan et al. (2019)’s proposed method to rank the representations of different models. By learning a VAE ensemble with linear transformations between the latent spaces and an additional “cross-model” reconstruction loss, the authors show that they can achieve significantly better disentangling.\n\nStrengths:\n- The theoretical justification seems reasonable and builds on previous work.\n- The experiments are organized to answer three meaningful questions. The results do suggest the VAE ensemble learns better latent representations which can be converted between models with simple, orthogonal linear transformations.\n\nQuestions:\n- Regarding the last term of the loss in equation (2): for a fixed i and j, the loss is E_{q(z_ij|x)} ||z_jj - z_ij|| = E_{q(z_ij|x)}||z_jj - M_ji z_ii||. This loss term can be optimized by tuning the parameters of VAE i, VAE j, and M_ji. Do you backprop through all these? Or is there a stopgradient on z_ii when used in computing this loss term (i.e. no gradients through VAE i from this loss term)?\n- What would be the effect of training the VAE models in two stages: independently first and then jointly in the ensemble? Would it help or hurt disentangling?\n- How would you express the total information cost of representing an image across the VAEs in the ensemble (say if you wanted to to compare the information rate to a single VAE)? It doesn't make sense to add up the KL costs linearly.\n\nSuggestions:\n- It would help enormously to strengthen the findings and assertions regarding the effect of ensemble size and the gamma hyperparameter.\n- Consider adding another disentanglement metric e.g. MIG.\n- Figure 5 in the Appendix shows a larger effect on DtO of the number of dims than the gamma hyperparameter. This result (and other results on CelebA) are perhaps worth describing in the main paper. \n\nMinor:\n- In Figure 2(a) I assume the curves are overlapping? Does it help to use a log scale for the y-axis?\n- How are the latent dimensions sorted in Figure 3?\n- Are the scores in Table 2 across different training runs?", "extra_scores": {}, "extra_text": {}}, {"review_id": "bO9wmlOyAF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary:\nThis submission proposes an ensemble framework to improve learning disentangled representations with Variational Autoencoders (VAEs). The approach builds on the assumption that entangled latent representations learned by VAEs show some “uniqueness” in their latent space structure, while disentangled representations exhibit some “similarity”; an assumption corroborated by recent studies. On that basis, a VAE ensemble approach is proposed where several VAEs are connected through linear mappings between the individual latent spaces to encourage alignment of latent representations and thus disentanglement. A formal derivation of the framework is provided and the formal validity of the underlying assumption demonstrated. Furthermore, empirical evaluation of the proposed approach in comparison to the standard VAE, beta-VAE and FactorVAE on the datasets dSprites (main results, main text) and CelebA (appendix) is performed, yielding improved results on the FactorVAE disentanglement metric (all baseline methods considered) as well as the Distance to Orthogonality (DtO) metric (only standard VAE considered).\n\n### Strengths:\n- Significance / Novelty: The proposed approach builds on recent work by Rolinek et al. and Duan et al., which show PCA-like behaviour in VAEs and leverage these results to develop disentanglement scores for model selection. This submission uses these insights for training an ensemble of VAEs in order to improve learning of disentangled representations. The claim is validated both formally as well as empirically on a benchmark dataset (dSprites) and state-of-the-art methods like FactorVAE, where the proposed framework performs favourably. To my knowledge the proposed idea is novel and simple yet potentially quite powerful. This approach could be relevant for other disentanglement methods and a wider audience employing VAE approaches.\n- Technical Quality: An important contribution of this paper is the thorough formal derivation and theoretical justification of the approach which to me appears sound. The experimental evaluation is well-designed and mostly succeeds in justifying the claims, with some exceptions outlined below. I believe that all the relevant details to reproduce the results are provided.\n- In particular, the results that the DtO comes close to 0 (fig. 2) for the ensemble approach illustrate that the latent representations of the different VAE in the ensemble converge (question 1), i.e. the linear transformations between latent space converge to (signed) permutations. This means that it should not matter which latent representations in the ensemble is studied (in the paper the first model in the ensemble is chosen; lines 274-275). However, I am curious whether the authors considered the results (“polarisation” and FactorVAE scores) for other latent representations (i.e. not the first model) and how much the results agreed?\n- Clarity: I consider this paper well-written and well-structured. Relevant details and formal justifications are provided in an appropriate manner resulting in a self-contained paper.\n\n### Weaknesses:\n- The ensemble approach comes at a cost which is probably the reason why only up to 5 parallel models were used. Can the authors comment on the running time and memory requirements compared to the competing methods? I think the quality of the paper could be improved if these details and the restrictions of the ensemble approach were provided.\n- The results in table 1 (comparison of baseline methods and ensemble approach w.r.t. FactorVAE metric) show that an ensemble of size >=3 can outperform state-of-the-art methods like FactorVAE on the considered FactorVAE metric. However, they also show that it might not always be beneficial to put more weight onto enforcing aligned latent representations for the same ensemble size (gamma > 1). This is a bit at odds with the premise of the paper. As the discussion points out (question 3, lines 285-289), this could be due to balancing different contributions in the more extensive objective function. However, this could also hint at potential optimisation problems for more challenging tasks. \n- The examples for the latent traversal (in the appendix) are slightly less convincing and a comparison is only done w.r.t. a standard VAE. However, it would be much more insightful to compare the ensemble approach to beta-VAE and FactorVAE latent traversal results.\n- Similar to the last point, in figure 2, it would be quite insightful to see the DtO results for the beta-VAE and especially the FactorVAE. In my opinion, this is a crucial aspect which so far is missing and could justify the approach even more. Isn’t the whole motivation that beta-VAE and FactorVAE should perform slightly worse w.r.t DtO?\n\n### Additional Feedback:\n- Figure 1: I like the illustration, however I do not understand the bar plot (“VAE, BetaVAE, FactorVAE, VAE Ensemble”). Maybe an additional annotation could help?\n- Line 8: *“sometime”* -> *”sometimes”*\n- Line 24: *”state-of-the-arts”* -> *”state-of-the-art”*\n- Line 25: *”[…] deploy Variational Autoencoder […]”* -> *”[…] deploy the Variational Autoencoder […]”* or *”[…] deploy Variational Autoencoders […]”*\n- Line 37, line 190, line 221 : *”On contrary, […]”* -> *”On the contrary, […]”*\n- Line 74: *”[…] closely approximate prior […]”* -> *”[…] closely approximate the prior […]”*\n- Line 127: *”[…] models […]”* -> *”[…] model […]”*\n- Line 164: *”[…] decomposition L2 term […]”* -> *”[…] decomposition, the L2 term […]”*\n- Line 224: *”Such gap […]”* -> *”Such a gap […]”*\n- Line 225: *”[…] such case […]”* -> *”[…] such a case […]”*\n- Line 233: *”Does VAE ensemble improves […]”* -> *”Does the VAE ensemble improve […]”*\n\n### Recommendation:\nThis submission was an enjoyable read, it provides some new insights and I believe this paper can pose an important contribution in areas which are concerned with learning disentangled representations and VAE models. In my opinion, the claims of the paper are justified both theoretically and empirically. However, there are certain aspects and concerns outlined above which need to be addressed adequately to warrant a publication. At the moment, I am inclined to accept the paper, but I would like the authors to clarify the concerns and questions above.\n\n### Post-Rebuttal:\nI would like to thank the authors for the insightful rebuttal! The authors were able to address my concerns adequately and I believe that the revision improved the quality of the paper quite a bit. Therefore, I stand with my initial recommendation and due to the reasons stated above, I endorse accepting this paper. \n\n\n### References: \n- Rolinek et al., “Variational autoencoders pursue pca directions (by accident)”, CVPR 2019.\n- Duan et al., “Unsupervised model selection for variational disentangled representation learning”, ICLR 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IkYZyU1zin", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe authors introduce a novel VAE-based approach for unsupervised learning of disentangled representations of image data.  The approach trains an ensemble of VAEs along with pair-wise linear transformations between their latent spaces.  The objective includes the ELBO objectives for each VAE as well as two additional pressures:  (i) An L2 similarity objective that pressures samples from each VAE latent space to match under linear transformations samples from the other VAE latent spaces, and (ii) A cross-model decoding objective that encourages decoding accuracy of the linearly transformed latent samples.  The authors provide a theoretical argument that the linear transformations should learn to be orthogonal, and show some experimental results indicating that their model performs well compared to baselines when evaluated with an established disentangling metric.\n\n# Pros\n\n* The theoretical analysis in section 4.1 is clear and provides good mathematical intuition for the authors’ results.\n* The introduction and related work sections are clear and include a thorough set of references.\n\n\n# Cons:\n\n* The authors’ baseline results give unexpectedly low metric scores.  The authors report FactorVAE metric values of 0.665 for beta-VAE and 0.764 for FactorVAE on the dSprites dataset.  However, the values reported in the FactorVAE paper (and corroborated by others) on the same dataset are significantly higher.  This makes me suspicious that something went wrong with the authors’ training --- perhaps they didn’t train those baseline models to completion or something else went wrong.  Having baseline results that are inconsistent with the existing literature makes me uneasy.\n* The traversals in Figure 8 from the authors’ model are much less disentangled than other models in the literature.  For example, they are much less disentangled than the traversals shown in the beta-VAE paper and the FactorVAE paper on the same dataset.  Thus from these traversals, it seems that the authors’ model is performing worse than existing models in the literature (the authors’ metrics indicate the opposite, but as mentioned above I’m uncertain about the validity of those metric results).  Figure 3-A also suggests that the authors’ model is using too many informative latents, i.e. not disentangling well.\n* I am not convinced by the authors’ intuitive justification in lines 216-225 (and appendix C) that the cross-model objective encourages entangled models to align to disentangled models.  Specifically, in that argument the authors seem to assume that orthogonal linear transformations are orthonormal.  However, there is nothing to enforce normality of the transformations in the model, hence the cross-model encoding variance from an entangled to a disentangled model could be quite small.\n* The purpose of the cross-model reconstructions is not clear, particularly given that I’m not convinced by the authors’ intuitive justification of them.  The L2 regularization between the transformed encodings should pressure the cross-model reconstructions to be good, so I do not see the reason to include them in the model objective.  It would be good if the authors could do an ablation study without the cross-model reconstructions.\n* The authors do not mention the computational complexity of their model, yet computational complexity seems to be a significant drawback of it.  Ensemble training is very computationally expensive, so the authors should include some discussion about it as well as runtimes and memory requirements for their model.  Furthermore, with the cross-model reconstructions the computational complexity of the authors’ model scales with the square of the number of ensemble elements, which is quite a steep scaling.\n* The authors only compare to a couple (relatively old) baselines, betaVAE and FactorVAE, which are no longer state-of-the-art.  However, more recently a number of other VAE models have been published that perform better.  In order to support their claims about state-of-the-art performance, the authors should compare to newer baselines.  Here are a few examples:\nDIP-VAE  (Variational inference of disentangled latent concepts from unlabeled observations.  Kumar et al., 2017)\nTCVAE (Isolating sources of disentanglement in variational autoencoders. Chen et al., 2018)\nSpatial Broadcast VAE  (Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs.  Watters et al., 2019)\n* The authors also don’t include many metrics or datasets.  dSprites and CelebA were used in the original betaVAE paper, but more recently it has become the norm to test on a larger set of datasets and with a number of different metrics to convincingly show disentangling.  By the way, a number of models, datasets, and metrics have been open-sourced in DistLib (https://github.com/google-research/disentanglement_lib), which may be useful for comparing to more models with more metrics on more datasets.\n\n# Summary\n\nI do not recommend accepting this paper.  Baseline results are inconsistent with prior work, the model seems to disentangle less well than existing methods, and the authors don’t do ablation experiments to justify the high computational complexity of the model.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use an ensemble of VAEs to learn better disentangled representations by aligning their representations through additional losses. This training method is based on recent work by Rolinek et al (2019) and Duan et al (2020), which suggests that VAEs tend to approximate PCA-like behaviour when they are trained to disentangle. The method is well justified from the theoretical perspective, and the quantitative results are good. Saying this, the reviewers raised concerns about the qualitative nature of the learnt representations, which do not look as disentangled as the quantitative measures might suggest. There was a large range of scores given to this paper by the reviewers, which has generated a long discussion. I have also personally looked at the paper. Unfortunately I have to agree that the latent traversal plots do not look as disentangled as the metric scores would suggest, and as one might hope to see on such toy datasets as dSprites. The traversals are certainly subpar to even the most basic approaches to disentanglement, like beta-VAE. For this reason, and given the reviewer scores, I unfortunately have to recommend to reject the paper this time around, however I hope that the authors are able to address the reviewers' concerns and find the source of disagreement between their qualitative and quantitative results for the future revisions of this work.
Decision: Reject