Recursive Disentanglement Network
Authors: Yixuan Chen
Yubin Shi
Dongsheng Li
Yujiang Wang
Mingzhi Dong
Yingying Zhao
Robert P. Dick
Qin Lv
Fan Yang
Li Shang
Venue: ICLR 2022 Poster (2022)
Keywords: disentanglement
representation learning
compositional

Abstract: Disentangled feature representation is essential for data-efficient learning. The feature space of deep models is inherently compositional. Existing $\beta$-VAE-based methods, which only apply disentanglement regularization to the resulting embedding space of deep models, cannot effectively regularize such compositional feature space, resulting in unsatisfactory disentangled results. In this paper, we formulate the compositional disentanglement learning problem from an information-theoretic perspective and propose a recursive disentanglement network (RecurD) that propagates regulatory inductive bias recursively across the compositional feature space during disentangled representation learning. 
Experimental studies demonstrate that RecurD outperforms $\beta$-VAE and several of its state-of-the-art variants on disentangled representation learning and enables more data-efficient downstream machine learning tasks.

Published as a conference paper at ICLR 2022
RECURSIVE DISENTANGLEMENT NETWORK
Yixuan Chen, Yubin Shi; Dongsheng Li
(yixuanchen20, ybsh 2 e wuuuaneddu... dongs @immmmmirras crosoft.com
Yujiang Wang! Mingzhi Dong"
yujiang. wang1 1ewp:l144mmmmmeaaaaammmmm ac, uk, mingahicong@mmii..coom
Yingying Zhao; Robert Dick
yingyingzzhao@fudee edu. en, dickrpeumich.ed
Qin Lv,Fan Yang. Shang"
gin.iv@cloorado. edu, (yangfan, lishang) fudan edu. cn
ABSTRACT
Disentangled feature representation is essential for data-efficient learning. The feature space of deep models is inherently compositional. Existing 3-VAE- methods, which only apply disentanglement regularization to the resulting embed- ding space of deep models, cannot effectively regularize such compositional fea- ture space, resulting in unsatisfactory disentangled results. In this paper, we for- mulate the compositional disentanglement learning problem from information- theoretic perspective and propose a recursive disentanglement network (RecurD) that propagates regulatory inductive bias recursively across the compositional feature space during disentangled representation learning. Experimental studies demonstrate that RecurD outperforms BAE and several of its state-of-the-arr variants on disentangled representation learning and enables more data-efficient downstream machine learning tasks.
INTRODUCTION
Recent progress in machine learning demonstrates the ability to learn disentangled representations is essential for data-efficient learning, such as controllable image generation, image manipulation. and domain adaptation (Suter al., 2019 Zhu et al. 2018 Peng et al.) 2019 Gabbay & Hoshen 2021 2019). 3-VAE (Higgins et al. (017)) and its variants are the most investigated approaches for disentangled representation learning. Recent works on 8 B-Vse-based methods introduce various inductive biases as regularization terms and directly apply them on the resulting embedding space of deep models, such as the bottleneck capacity constraint (Higgins et al.) 2017] Burgess et al.) 2018)) total correlation among variables (Kim & Mnih 2018 Chen et al. 20188 8), and the mismatch between aggregated posterior and prior (Kumar et al.) 2017). 7), aiming to balance among representation capac- ity, independence constraints, and reconstruction accuracy. Indeed, as demonstrated by Locatello ct al. (2020] 2019), unsupervised disentanglement is fundamentally impossible without explicit in- ductive biases models and data sets.
However, our study shows that existing 3- 3-VA--based methods may not able to learn satisfac- tory disentangled representations even for fairly trivial cases. This is due the fact that the feature spaces of deep models have inherently compositional structures, i.e., each complex feature is a com- position of primitive features. However, existing methods with regularization terms solely applied to the resulting embedding space cannot effectively propagate disentangled regularization across such compositional feature space. As shown in Figure I applying the standard 3-VAE to the widely
China and Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University, Shanghai, China, Microsoft Research Asia, Shanghai, China, Department of Computing, Imperial College London, London, United Kingdom, Department of Electrical Engineering and Computer Science, University of Michigan, Michigan, United States, Department of Computer Science University of Colorado Boulder, Boulder, United States, School of Microelectronics, Fudan University, Shanghai, China, The corresponding author.

---

Published as a conference paper at ICLR 2022
Enoockr
Decodlr
(a)
(h)
Figure I: Illustration of the negative impact on ignoring the compositional structure of the repre- sentation space using dSprites dataset: (a) Illustration
(from embedding space) and m (from intermediate layers). (b) z is not disentangled sufficiently when is not disentangled sufficiently. (c) The disentanglement of z improves, as that m improves.
used dataset dSprites (Matheyy al. 2017). we visualize the resulting representation as well as its compositional low-level representations m extracted from the previous layer (as shown in Fig- ure [(a)), and evaluate the independence between each pair of and each pair of z, respectively [ Figure (b) and Figure [ho] that the disentanglement quality of low-level features m may im-
pact the resulting representation in terms of disentanglement quality. This study demonstrates the potential benefit to regularize the compositional feature space of deep models during disentangled representation learning.
This work aims to tackle the compositional disentanglement learning problem. First, we formu- late disentangled representation learning from an information-theoeeei perspective, and introduce new learning objective covering three essential properties for learning disentangled representa- tions: sufficiency, minimal sufficiency, and disentanglement. Theoretical analysis shows that the proposed learning objective is a general form of 3-VAE and several of its state-of-the-art variants. Next, we extend the proposed learning objective to cover the disentanglement representation learn- ing problem the compositional feature space. Governed by the proposed learning objective, present Recursive Disentanglement Network (RecurD), compositional disentanglement learning method, which directs the disentanglement learning process across the compositional feature space by applying regulatory inductive bias recursively through the feed-forward network. We argue that the recursive propagation of inductive bias through the feed-forward network imposes sufficient condition of disentangled representation learning. Empirical studies demonstrate that RecurD out- performs 3-VAE (Higgins et al. (017)) and several other variants of VAE (Burgess et al. 2018 Kim & Mnih 2018; Chen et al., [0188] Kumar et al., (017)) on disentangled representation learning and achieves more data-efficient learning in downstream machine learning tasks.
COMPOSITIONAL DISENTANGLEMENT LEARNING
In this section, we first formulate disentanglement learning from the information -theoretit perspec- tive by introducing three key properties and show such formulation is a general form of the opti- mization objectives of 3- VAE and several of its variants. Next, we extend the principled objective the compositional feature space tackle the compositional disentanglement learning problem.
2.1 DISENTANGLEMENT LEARNING OBJECTIVE
The challenge of representation learning be formulated finding distribution p (z|x) that maps original data x E into a representation z with fixed amount variables z = {z)..... 2,, [Bengio al. 2013). The key intuition of is capture minimal sufficient information in disentangled manner, given the reconstruction task x Rv &. We denote the representation learning process as Markov Chain for which & + X Z. which means z depends on only through x, i.c. p (z|x)
(z) x, X) (see also, (Cover 1999 Achille & Soatto 2018)). The principled properties of are
defined as follows:
Definition 1. Sufficiency: representation of for sufficient if (x, x) (z, x).
TTee independence between two components e; and ej is measured by the normalized mutual informa- tion (Chen et al. 2018). Whenever NMT((c,;,,) = T(c,;))/ 0, c; and c₁ are independent (disen- tangled).
2

---

Published as a conference paper at ICLR 2022
For the reconstruction task, z is sufficient if it can successfully reconstruct by x. The difference between I (x: and I (z; is computed as follows: I (x: 8) I (z: x) = / (x; X|z) ) = H (x|z)
H ($|xx). Given the reconstruction task x &, H (x(x) is constant and independent to z, so the sufficient property can be optimized by minimizing H ($|z) (Federici et al. 2020: Dubois et al. 2020).
Definition 2. Minimal Sufficiency: representation z of x is minimal sufficiemt if (x:2) = / (z; x).
A minimal sufficient encodes the minimum amount of information about required to reconstruct
& (Cover 1999; Achille & Soatto| 2018). Since / (z; 8) equals to I (x; &) when z is sufficient, the difference computed as / (x; Z) - (z; x) T (x; z) 7 (x: Given the reconstruction task x R &, / (x; x) is constant and independent to z, so the minimal sufficiency property can be optimized by minimizing I (x:z).
Definition 3. Disentanglement: representation denoted as z {zi..... Zn} is disentangled if
ΣnSt (zzz,) O.
(z.) H (zi(z)) denotes the reduction of uncertainty in 2; when 2j is observed (Cover 1999). If any two components and 2, are disentangled, changes to z; have no influence on Zjz which means / (z,; z,) 0.
From the definition of mutual information, / (z,; z
A representation satisfying all these properties can be found by introducing two Lagrange multipliers A and X2 for two constrained expected properties with respect to the fundamental sufficiency prop- erty. The principled objective of disentanglement learning is to minimize the following objective:
L = H (x/z) + \1 (x:
A2 Σ (Z,, 2z).
The above objective can be interpreted as the reconstruction error, plus two regularizers that yield an optimally disentangled representation. The principled objective also helps us analyze and un- derstand the success of recently developed 3-VAEEbased methods. These methods operate with an encoder with parameter & and decoder with parameter 8, to induce the joint distributions (x.z) = 9s (z(x) g (x) and p (x, z) Pa (xlz) p (z). respectively, where p (z) is a fixed prior distribution. The learning objective of 8-VAE contains the reconstruction error and KL divergence between the variational posterior and prior. To understand the relationship of learning objectives
between Equation and 8-VAE- methods. we decompose (x; z) (Kim & Mnih 2018) and
estimate upper bound for Ej4s I (z;,z) 2,) as follows:
(Te Sun
1980 1975). then we assign different weights
A1 (x:2) + A2 Σ I (2))))
S Ex (g (2(x) [IP + X (() (19(2))
+ Ae Σ KL (2;) (p,)))
As shown in Table D the learning objectives of 3-VAE and its four variants can be regarded as specific cases of Equation [ i.c.. they assign different weights to our regularization terms, which can balance among latent variables capacity, independence constraints, and reconstruction accuracy, leading to successful disentangled representation learning (Zhao et al.) 2017: Li et al., 2020). More details can be found in Appendix [ However, in these works, the inductive bias toward disentan- glement is only applied to the embedding space of ignoring the need of disentanglement during feature composition in feed-forward networks.
Table 1: Deriving the learning objectives of 3 3-VAE and its four variants as specific cases of Equa-
tion 1 Method
3-VAE
FactorVAE
3-TCVAE Aa
DIP-VAE Aa 1,
VAE
Weight Relation
Aa B.
Ra
A a,
Ab 7.
\ 3.
A A A.
Ab + X = a + 1
2.2
COMPOSITIONAL OBJECTIVE
Considering encoder with L layers to encode original data x into disentangled representation Let us denote m' as the input features of the l-layer, which are divided into groups of features, i.c., m' U,m', where m is the j-th feature subset.
3

---

Published as a conference paper at ICLR 2022
We formulate the compositional relation between features from two consecutive layers follows: [+1 Layer (m' x w)) Layer can be applicable to commonly used neural network layers, e.g., the convolution layer in computer vision tasks. The compositional relation is achieved by composition matrix w' E gdixd+1,, so that features in m' are divided into d++1 groups after passing through all compositional vectors (w's). Note that m 1+1 is only related to the subset of features from m' selected by w/.-
m
Similar to Section 2.1 we assume that the learning process of m I+1 depends on m' through original data x, denoted as the Markov chain: m' x + m can be written as m (+1 + m' according to the conditional independence implied by the Markovity (Cover, 1999). Consider two output features m 1+1 m E m²+1 and their corresponding input feature subsets m m) E m', we define key notions follows:
Definition 4. Compositional Disentanglement: m, and m', are disentangled if (m) m)) = O.
A disentangled representation of m and m' may improve the disentanglement quality between mž*1 and m/- Similar to Definition we can achieve compositional disentanglement by mini- mizing 7 (m/mm)
Definition 5. Compositional Minimal Sufficiency: Assume that the learning process of m is denoted by the Markov chain: mf+1 -x (m), m/). Given the original data x, an inpus feature set m'j for the output feature m 1+1 is minimal sufficient if (x; mx:t))))) = im):m/++)
For the output feature m) m/+1, the input feature set m is sufficient and another input feature set m is superfluous when m' is able to capture all information of m 1+1 as well as the original data x.
Furthermore, according to Pata-Processing Inequality (DPI) (Cover 1999 Achille & Soatto 2018) in the Markov chain, there exists an inequality that:
I (x; (x;m,")) ") 2 I (m/+ m')) - (m [+1, m!) - (m's m²)
(3)
where the difference between I (x: +1) and I (m) (+1, m²) is equivalent to the difference be- tween 7 (m m))) and (m/) m²). Therefore, matching I (m/+); mm! to / (mi) m²) can yield minimal sufficient representation m', for mft1 Based on the definition of compositional disen- tanglement, we can optimize the minimal sufficiency by forcing 7 (m/+t, m to be 0. More details can be found in Appendix B
To learn disentangled representation via effectively regularizing the compositional feature space, we augment the principled learning objective (Equation T with compositional regularizers. Therefore, the compositional learning objective for disentangled representation is defined as follows:
L
dea1
de I (m)) m|) ΣΣ Jiti
L H (x/m (+1 +i1
Σ
(m/:
+
(4)
1=2
sufficient
minimal sufficient
disentangled
where m 1++1 denotes the final disentangled representation Our intuition is that disentangled learning for compositional feature space could benefit the disentanglement learning for high-level
representations.
3 RECURSIVE DISENTANGLEMENT NETWORK
We now describe a learning method with the goal of optimizing the compositional disentanglement learning objective. This method, called Recursive Disentanglement Network (RecurD), propagates inductive bias (disentanglement) recursively across the compositional feature space.
3.1 MODEL ARCHITECTURE
As shown in Figure 2 RecurD contains encoder and decoder to lear the disentangled repre- sentation z of data x and to reconstruct &, where the encoder contains multiple Recursive Modules and the decoder is Deconvolutional Neural Network (Zeiler & Fergus 2014).
4

---

Published as a conference paper at ICLR 2022
Encoder
mm
Eno
m
Recursive Module
ml
m
Einq
m
m
Decoder
(m/+))
Roubor)
Encoders
Figure 2: Recursive Disentanglement Network
The first Recursive Module of the encoder is implemented by a multi-channel convolutional net- work (Lawrence et al., 1997) to encode the original image X. Following the notation in Section 2.2 the output the 1-st Recursive Module denoted m which is also the imput of 2-nd Re- cursive Module. As for the I-th (1 2 2) Recursive Module, it contains (1) Router R to learn composition matrix w/ from the input features m' decompose m' into subsets; and (2) Group-
of-Encoders (GoE) layer consisting of encoders to induce the output feature m
Inspired by the Gate Mixture-of-Experte (Shazeer et al., 2017: Fedus et al. 2021) on parameter selection for each input, we present Router with 7 Topk to learn the compositional relation. In detail, the Router takes m' as input and compute composition matrix w' via learning similarity:
w' softmax (TopK (R
(m') = softmax ([m*) [mm) [m']]
o v, and
R [mm),,,, Topk (R (m') k) = -00,
{
ifR (mm'), is in the top elements of (m'',, otherwise.
(5)
Here, R (m) denotes the similarity matrix and v is learning matrix learned by linear layer. denotes Hadamard product. Topk is the compositional strategy to determine whether input feature m'y belongs to the input feature set of m +1 and & is a hyperparameter. The GoE layer consists of di+1 parallel encoders { Enc,,..., Enca, } to generate the output features m where each encoder implemented by convolutional neural network with specific parameters:
m = Ency (m' w/).
(6)
Then, m [1 m'+1 is obtained by concatenating the outputs from each encoder, which converted directly to the input of the / + 1-th Recursive Module. Note that the output of L-th Recursive Module is the learned disentangled representation Finally, the Decoder Dec takes z as input to obtain the reconstructed input & & = Dec (z).
3.2 LEARNING OF RECURD
As mentioned in Equation 4 the learning objective of RecurD governs the recursive disentanglement learning across the compositional feature space. As shown by related works, precise estimation of mutual information in the high dimensional space is important for accurately estimating the loss. According the recent progresses, the mutual information between two representations can be maximized by using any sample-based differentiable mutual information lower bound. Similar to the work of Federici et al. (2020); [Hjelm et al.) (2019); Wen al. (2020), we utilize MINE es- timators (Belghazi et al.) (2018) to estimate the mutual information. This approach introduces an auxiliary parametric model Estimator Net which jointly optimized during the training procedure using re-parametrized samples from the posterior distribution.
4 RELATED WORK
3-VAE-based methods introduce various regularization terms and directly apply them on the result- ing embedding space. Higgins et al. (017)) proposed 3-VAE method by introducing a constraint over the K divergence between the inferred distribution (z|zx) x and its prior-an isotropic unit Gaussian. Burgess et al. 2018) found that the network specializes in the factor that contributes most to small reconstruction error, with limited channel capacity. Thus, they proposed the Annealed- VAE, an extension 3- VAE, which gradually adds more latent encoding capacity by enforcing the KT. divergence be at controllable value C. Kim & Mnih (2018) analyzed the disentanglement
5

---

Published as a conference paper at ICLR 2022
performance of B- B-V by breaking down the regularization term into components. One is Total Correlation (TC) (aatanabe| 1960), which encourages the marginal distribution of representations to be factorial. The other is mutual information [(x; z). which reduces the amount of information about x stored in z. Based on the decomposition, they proposed FactorVAF to relax the regular- ization of 2xx; while directly penalizing the TC term. Similarly, Chen et al. (2018) proposed 3-TCVAE by decomposing the TC penalty as the dependence among variables and the distance be- tween each variable's posterior and prior. Since TC is intractable, both FactorVAE and 3-TCVAE use approximation methods. Another TC-related approach is DIP-VAE (Kumar et al. 2017), whose designers argued that the estimation TC requires additional parameters and suffers from vanishing gradients. Therefore, DIP-VAE optimizes the moments distance between the aggregated posterior and a factorized prior instead of estimating TC. The above methods improved over / 3-VAE by ap- plying inductive biases directly to the high-level latent variable space. However, the feature space of deep models is compositional in nature. Existing variants of 3-VAE cannot effectively apply disentanglement regularization across such compositional feature space, and yield inferior disentan- glement in representation learning. Our approach diverges from prior works by taking a principled cormmattoo-thoorette approach to formulate and analyze the compositional disentanglement feature structure.
Recent works on hierarchical VAžs introduce layer-wise disentanglement regularization learn conditioning structures across multi-layer latent variables, such as VampPrior (Tomezak & Welling, 2018), Ladder VAEs (Sønderby et al.) 2016). and NVAE (Vahdat & Kautz 2020). In these hicrar- chical model structures, c.g., the cross-layer residual connection structure in NVAE and VampPrior, inter-layer regularization is less of a focus. The latent variables of preceding layer serve as shared inputs to the next layer, which introduces information redundancy and hence impairs representa- tion disentanglement. In contrast, the proposed compositional objective optimizes the statistical independence of inter-layer and intra-layer latent variables simultaneously, thereby minimizing the information redundancy of inter-layer information sharing and improving disentanglement quality.
EXPERIMENTS
This section presents both quantitative and qualitative experiments to evaluate RecurD in terms of disentanglement quality and data efficiency downstream tasks.
5.1 PERFORMANCE OF DISENTANGLEMENT LEARNING
We compare the disentanglement learning performance of RecurD with 3-VAE and its state-of-the- art variants, including 3- VAE (Higgins et al. 2017). canealed-VAE (uurgess et al. 2018). Factor-
VAE (Kim & Mnih, 2018), 8- 3-TCVAE (Chen et al. 2018), DIP-VAE (Kumar et al. 2017). More experimental results can be found in Appendix
Datasets: We consider two datasets in which each image is obtained by a deterministic function of ground-truth factors: dSprites (Matthey et al., (017) and 3DShapes (Burgess & Kim 2018 dSprites contains 737, 280 binary 64 x 64 images of 2D shapes with 5 ground truth factors, I.c 3 shapes, scales, 40 orientations, 32 x-positions, 32 y-position. 3DShapes contains 480, 000 RGB 64 x 64 x 3 images of 3D shapes with 6 ground truth factors, i.e., 4 shapes, 8 scales, 15 orientations, 10 floor hucs, 10 wall hues, 10 object hues. For all experiments, we use a 9:1 training to testing data ratio, following earlier work (Kumar et al.) 2017 Locatello et al. 2019). More details on network architecture and hyperparameter settings are included in Appendix D
Evaluation Metrics for Disentanglement: There is standard metric for evaluating disentangle- ment (Zhou et al. 2021]] Ridgeway& & Mozer 2018). and most existing metrics involve the estimation of variable-factor matrix relating the factors of variation to the learned representations. In the ex- periments, we consider three widely used metrics: Separated Attribute Predictability (SAP) (Kumar) et al.) 2017). Mutual Information Gap (MIG) (Chen et al., 2018) and Disentanglement, Complete- ness, and Informativeness (DCI) (Eastwood & Williams, 2018 DCI contains three metrics for disentangiement (DCI-D), Completeness (DCI-C) and Informativeness (DCI-I), respectively. Due to the space limitation, we present the results of DCI-D and include the results on DCI-C and DCI- I in the Appendix E Overall, these metrics can comprehensively evaluate RecurD from different disentanglement measurements.

---

Published as a conference paper at ICLR 2022
5.1.1 QUANTITATIVE RESULTS
For quantitative analysis, we conduct three sets of experiments: 1) evaluating the average perfor- mance of disentanglement score via three evaluation metrics; 2) measuring the trade-off among reconstruction, minimal sufficiency and disentanglement with mini-batch samples; and 3) analyzing the influence of compositional objective for disentanglement learning via the three properties.
cooprres the reconstruction error and three widely-used disentanglement metrics of RecurD and five other methods on the dSprites and 3DShapes datasets. Compared with all the baselines, RecurD achieves much lower reconstruction error as well as higher SAP, MIG, and DCI scores in most of the cases. It is important point out that the reconstruction error of B-VAE increases as 3 increases (stronger disentanglement regularization), which indicates that f-VAE comprises reconstruction error for disentanglement. However, RecurD achieves better in both reconstruction and disentanglement, demonstrating the superiority of the proposed compositional learning objective and the recursive disentanglement network.
Figure 3 shows scatter plots relating Minimal Sufficinccy and Disentanglement to reconstruction er- ror (Sufficiency), where each point represents a mini-batch of data. Here, we compute the minimal sufficiency score using I [x; z) and present it by the area of each point. In all baseline methods, we observe that smaller scatters are correlated with higher reconstruction errors and higher disentangle- ment scores. The reason is that B-VAE and its variants only place disentanglement regularization on the embedding space z which imposes a limit on the capacity of the information channel (Higgins) et al., (017). ) that features which are important for reconstruction but harmful for disentanglement may lose during training, leading to compromised reconstruction quality. However, RecurD reveals the necessity of disentanglement during feature composition in feed-forward networks and induces effective information compression. Therefore, compared to all the baselines, the representations from RecurD obtain better disentanglement and minimal sufficiency without degrading reconstruc- tion quality.
Figure 4 shows the impact of the number of Recursive Modules in disentanglement learning. We evaluate three RecurD variants on the principled properties, including RecurD 0, RecurD D 1 and Recur with 1, 2 and 3 Recursive Modules, respectively. And we report the values correspond-
Table 2: Three widely used evaluation scores and reconstruction error on the test sets for dSprites and 3Dshapes. Boldface indicates the best results, i.c., reconstruction error or disentanglement scores.
dataset
dSprites
3DShapes
Reconst. Error
SAP
MIG
DCI-D
Reconst error
SAP
MIG
DCI-D
B-VAE(3
0.00166
0.0284
0.2617
0.1191
0.0216
0.1463
0.2519
0.4682
3-VAE(3 16)
0.0094
0.0242
0.2241
0.1820
0.0544
0.1488
0.2629
0.4528
8-VAH(3 = 60)
0.0127
0.0445
0.1432
0.1221
0.0624
0.1041
0.2402
0.4317
Annealed- VAE
0.0171
0.0311
0.1177
0.1449
0.0811
0.0730
0.2217
0.4279
Factor- VAE
0.0228
0.0436
0,2594
0.1955
0,0800
0.1331
0.2630
0.4491
8-TCVAB
0.0162
0.0352
0,1585
0.1774
0,0312
0.0364
0.2070
0,4487
DIP-VAE
0.0213
0.0261
0.0731
0.1038
0,0213
0,2013
0.3108
0,4853
RecurD
0.0047
0.0502
0.2707
0.3841
0.0083
0.1979
0.3
0.5804
Syeiio
dSgrites
a5
0.00
20004
82
0.0
1/
Rocorol, Ease
10
Recodl Emn 23
Recubl Enor
SDSuppss
SDShppss
3DStppes
puacs
0.0
Exice
008
3:30
8:22
0.6
0.06
8.55
0.0
Figure 3: Reconstruction сттог vs. disentanglement performance. Scatters located at the left top indicate better performance. The area of each scatter represents the minimal sufficiency score esti- mated by I(x; z), and smaller area indicates better performance.
7

---

Published as a conference paper at ICLR 2022
ing to the three terms in compositional learning objective the training sets. We observe that Recur D 1 and RecurD 2 perform much better than RecurD 0 on the optimization of minimal sufficiency and disentanglement. In addition, during the early stage of training, the performance of Recur D 2 improves much faster than Recur 1. This experiment demonstrates that the recursive propagation of inductive bias through the feed-forward network improves disentangled representa- tion learning.
5.1.2 QUALITATIVE RESULTS
For qualitative analysis, we present the latent traversals of RecurD two datasets in Figure in which we vary single variable learned by an en- coder in GoE while keeping all oth- ers fixed. For images in 3Dhhapes, the latent traversals show that Re- curl is able successfully capture all the six factors of variation. For images in dSprites, we observe that RecurD is able to discover X-position, y-position and scale (continuous vari- ables). More importantly, RecurD can, to some extent, discover shape and orientation (discrete variables), which have been proved to be strug- gling for many other methods (Ku- mar et al. 2017 Kim & Mnih 2018]
Scadies
Figure 4: The performance of RecurD with varying number of recursive modules on the principled properties.
Locatello et al., 2019). The reason is that 8- B-VAE and its variants encourage independence among variables via controlling the information capacity of and matching the posterior p(z[x) to an isotropic unit Gaussian. However, learning discrete variables would require using a discrete prior instead of Gaussian (Kim & Mnih] (0018). On the other hand, RecurD is able to model both dis- crete and continuous factors by directly disentangling based on the three principled properties on the entire compositional feature space, leading stronger representation capability.
5.1.3 ABLATION AND PARAMETER DEPENDENCE STUDY
In this section, we evaluate the impacts of hyperparameters in both learning objective and model architecture. First, we study the impact of compositional learning objective with varying regular- ization coefficients (A1 and Aς) of minimal sufficiency and disentanglement. Then, evaluate the influence of hyperparameter &t - the group size in the GoE of Recursive Module. Figure 6 (Re- curD with varying Å1, A2 and ke) shows the scatter plots of disentanglement score (SAP) along with reconstruction error on the test sets of dSprites and 3DShapes.
As shown in Figure 6 larger penalties on both minimal sufficiency and disentanglement yield higher disentanglement score and lower reconstruction error, demonstrating the importance of the minimal sufficiency term and the disentanglement term in Equation T
Note that RecurD with X2 is
Hiriies
soslapa
disprrtes
SUNShtpee
reigl
IODES
4ap:
Dax buc
A210
wull luc
alj hee
100
2000
Rocoral,
Resvral,
Figure 5: First row: original images. Second Figure 6: Ablation study on \1. Xg and group reconstructions. Remaining rows: recon- structions of latent traversals.
size k. Note that RecurD with X2 = 0 reduces to
3- 3-V with the compositional architecture.
8

---

Published as a conference paper at ICLR 2022
reduced to standard 3-VAE with the compositional architecture (as [xx; z) is an lower bound of K ([_(pz//))lp(zx)x which underperforms RecurD, indicating that an explicit penalty on disentan- glement is important for disentangled representation learning. As for the group size in the GoE k, it is not surprising that when & increases, RecurD has lower reconstruction errors, with a slightly inferior disentanglement performance. The reason is that a dense composition of feature space can help the model maintain sufficient information but make it difficult to disentangle latent variables. RecurD with k results in poor performance on both reconstruction and disentanglement, in- dicating that overeemphasizing decomposition in the feature space may fail to preserve sufficient information. More results are presented the Appendix
5.2 PERFORMANCE OF DOWNSTREAM TASKS
In this section, we compare the performance of RecurD and five baseline methods by mea- suring their data efficiency two downstream tasks: a standard classification task on the MNIST dataset and domain generalization task on the MNIST-Rotation dataset (Ghifary al. 2015). MNIST-Rotation is synthetie dataset consisting of 6 domains, each contain- ing 1, 000 images the 10 digits randomly se- lected from the training set of MNIST, with 6 rotation degrees: 0%, 15, 30, 45,, 60 and 75%. For both tasks, we set the training sets of different proportions: 20%, 40%, 60%, 80%, and 100%, to evaluate the classification perfor- mance of RecurD and the baselines with differ- ent amounts of training data. For the domain generalization problem. we follow the previous works (Li et al., 2017 BBlaji al. 2018 Du et al. 2020) with the same train-test split strategy and the leave-one-domain-o strategy, i.e., we take the samples from one domain as the target domain for testing, and the samples from the remaining domains the source domain for training.
MNNS
34 07 ns E3 Ratio Triining
Rotio of Taaiinng
Figure 7: Performance comparison of RecurD and baselines on the standard classification task (left) and the domain generalization task (right). Each model is trained by varying ratios of training data. The dotted line represents the performance of single-layer neural network.
Figure reports the average accuracy of different methods on the same test set. We can observe that all methods achieve decent data efficiency on both tasks, i.e., without much performance degradation even with 20% of training data, suggesting that the learning process of disentangled representation can more effectively capture information from the inputs. However, VAV and its variants do not achieve satisfactory classification accuracy for either task compared to a single-layer neural network. The reason that (AE and its variants obtain the disentangled representation by limiting the ca- pacity of information channels, thus they tend to only maintain features that contribute more to dis- entanglement, which may lose informative features that contribute more to classification. Compared to the baselines, RecurD achieves consistently better performance in both tasks, especially on the harder domain generalization task. The reason that RecurD can learn disentangled representations without sacrificing the reconstruction performance, confirming the hypothesis that compositional disentanglement learning yields better generalization and more data-efficient representations. We believe that the informative disentangled representations emerge when the right balance is achieved between sufficient information preservation and minimal sufficient information learned in a disen- tangled manner.
CONCLUSION
This paper has described a solution to the compositional disentangled representation learning prob- lem. We first presented general information-thhoreee formulation of disentanglement representa- tion learning. and then extended it to the compositional feature space. We then described RecurD, recursive disentanglement network, which propagates regulatory inductive bias recursively across the compositional feature space. RecurD outperforms 3-VAE and its state-of-the-an variants on disentangled representation learning and achieves more data-efficient leaming in downstream tasks.

---

Published as a conference paper at ICLR 2022
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. The Journal of Machine Learning Research, (9(1):194-1188, 2018.
Yogesh Balaji, Swami Sankaranarayanan and Rama Chellappa. Metareg: Towards domain gen- eralization using meta-regularriztion.. Advances Neural Information Processing Systems, 31: 998-1008, 2018.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International Conference on Machine Learning, pp. 531-540. PMLR, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 158 35(8):1798-1828, 2013.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://ghhhhbooomddeemmm/////dss 2018.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-vae. arXiv preprint arXiv: 03599, 2018.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. arXiv preprint arXiv:1802.044 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 2180-2188, 2016.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snoek, and Ling Shao. Learning learn with variational information bottleneck for domain generalization. In European Conference on Computer Vision, pp. 200-216. Springer, 2020.
Yann Dubois, Douwe Kiela, David Schwab, and Ramakrishna Vedantam. Learning optimal repre- sentations with the decodable information bottleneck. arXiv preprint urXiv:2009.1278) 2020.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. In International Conference on Learning Representations, 2018.
Marco Federici. Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017, 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2 2021.
Sanja Fidler, Sven Dickinson, and Raquel Urtasun. 3d object detection and viewpoint estimation with deformable 3d cuboid model. Advances in neural information processing systems, 25: 611-619, 2012.
Aviv Gabbay and Yedid Hoshen. Demystifying inter-class disentanglement. arXiv preprint 0rXiv:1906.11796 2019.
Aviv Gabbay and Yedid Hoshen. Scaling-up disentanglement for image translation. arXiv preprint 1.Xiv:2103.14011 2021.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generaliza- tion for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2015.
10

---

Published as a conference paper at ICLR 2022
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICIR 2017, 2017.
Devon Hjelm, Alex Fedorov, Samuel Maaooie-arccildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. ICLR, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658. PMLR, 2018.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentan- gled latent concepts from unlabeled observations. arXiv preprint 17Xvv/711.000855 2017.
Steve Lawrence, C Lee Giles, Ah Chung Tsoi, and Andrew Back. Face recognition: A convolu- tional
approach. IEEE transactions on neural networks, 8(1):98-1133 1997.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Yanjun Li, Shujian Yu, Jose C Principe, Xiaolin Li, and Dapeng Wu. Pri-vae: principle-of-relevan- information variational autoencoders. arXiv preprint arXiv:2007.06503 2020.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 41 4114-4124. PMLR, 2019.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätseh, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look the unsupervised learning of disentangled repre- sentations and their evaluation. arXiv preprint arXiv:2010. 14766, 2020.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement 2017.
testing sprites dataset.
William McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93-111, 1954.
Xingchao Peng, Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with dis- entangled representations. In International Conference on Machine Learning, pp. 5102-5112. PMLR, 2019.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making Advances in neural information processing systems, 28: 1252-1260, 2015.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. arXiv preprint orXiv:1802.05333 2018.
Huajie Shao, Shuochao Yao, Dachun Sun, Aston Zhang, Shengzhong Liu, Dongxin Liu, Jun Wang, and Tarek Abdelzaher. Controlvae: Controllable variational autoencoder. In International Con- ference on Machine Learning, pp. 8655-8666 PMLR, 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 1701.0638.. 2017.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29:3738-3746, 2016.
Raphael Suter. Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In Interna- fional Conference on Machine Learning, pp. 6056-6065. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2022
Han Te Sun. Linear dependence structure of the entropy space. Inf Control, 29(4):337-0) 1975.
Han Te Sun. Multiple mutual informations and multiple interactions in frequency data. Information and Control, 46:26-45, 1980.
Jakub Tomczak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statisties, pp. 2314-223. PMLR, 2018.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03888, 2020.
Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of Re- search and Development, 4(1):66-82, 1960.
Liangjian Wen, Yiji Zhou, Lirong He, Mingyuan Zhou, and Zenglin Xu. Mutual information gradi- ent estimation for representation learning. arXiv preprint arXiv:2005.01 2020.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 518-833. Springer, 2014.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational
autoencoders. arXiv preprint arXiv: 170.::2706002222.. 2017.
Sharon Zhou, Eric Zelikman, Fred Lu, Andrew Y. Ng. Gunnar E. Carlsson, and Stefano Ermon. Evaluating the disentanglement of deep generative models through manifold topology. In Inter- national Conference on Learning Representations, 2021.
Jun-Ya Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Joshua B Tenen- baum, and William Freeman. Visual object networks: Image generation with disentangled 3d representation. arXiv preprint wrXiv:1812.02725. 2018.
12

---

Published as a conference paper at ICLR 2022
A PROPERTY OF MARKOV CHAIN AND MUTUAL INFORMATION
This section lists the related properties of Markov chain and mutual information used in this work.
Markov Chain
Consider three random variables a, b and coming from joint distribution p(a, b, c). if the con- ditional distribution of с depends only on and is conditionally independent of b, the variables can form a Markov chain in the order denoted as: + a + c. Specially, the joint distribution can be written as (Cover 1999):
paa c) pla))
Markovity implies conditional independence between and b when a is observed, the reason is:
p(c. bla) = p(a, b) p(c, a)p(bla) p(cla))(bbaa.
(8)
p(a)
p(a)
By rewriting the conditional independence, the Markov chain of b - - c clso implies - - as:
p(b. p(a,,, b. a)bca))( p(cla))(bla).
(9)
p(a) p(a)
Mutual Information
1. Positivity: ((a b) 2 0: (a; blc) 2 0.
Chain Rule: [(a; b, c) = ((a b) + [(;
3. [(a; b) (aa) (aa(b) = 0 if and only if a and b are independent.
I(a: b) (aa)
H
(a
b)
1
E log
E
log
p(a)
p
(aa bb)
log p(a[b)
p(b)
E
p(a)
p(b)
(10)
p(a,
E
log
p(a)b
D (p(a, b)]lp(a)) x p(b))
Processroosss Inequality (DPI): If three random variables a. b and coming from joint distribution p(a. b, c) can form a Markov chain in the order denoted as: b - - then T(b;a) 2 T(b; c). (Theorem 2.8.1 Cover 1999)
B LEARNING OBJECTIVES DECOMPOSITION
In this section, we decompose the proposed learning objective to analyze the relationship of learning objectives between Equation [ and 3-VAE-based methods. Let p(x) denote the true distribution of the data, and Po (z|x) and pe (x|z) denote the unknown distributions that we need estimate, parametrized by encoder with and decoder with 6.
B. DECOMPOSITION OF MINIMAL SUFFICIENCY
Minimal Sufficiency is defined as z can encode the minimum amount information of x required to reconstruct &, optimized by minimize T(x;2). Inspired from the work of Chen al. (2018).
13

---

Published as a conference paper at ICLR 2022
decompose the minimal sufficiency by assuming the prior p(z) factorized Gaussian
T(x;2) (x((x [KL (q (x, |4 (2) p (x))
9 (x,2) z)
(x(x.2) log q (z)x))p(x) (x)
[B((x2)
log q pz)
y (z)
-p((
Feg(zxx) (2)x) log (z) + log (z) - log (z) + log II (2z)) log II (2g)
log q (z(xx)
Exg(()) Σ log 9 (z,)
(z)
(x,0))
Ex((x) log
(z)
p (2z))
II, (z,)
Eptx [KL (q (z(x) 1p (z)))
KL Σ KL (q (2z) (I
q (z) II (2,) 5
(11)
The first term is KL divergence between inferred posterior and prior, usually penalty term of 3-VAEs for disentangling. The second term is the dimension-wise KL divergence, which represents the distance between each latent dimension the prior. The third term the Total Correlation referring to the independence among variables.
B.2 ESTIMATION OF DISENTANGLEMENT
Disentanglement is defined as the independence between any two variables, optimized by minimiz- ing Eizy Z,), which is an lower bound of Total Correlation.
The concept of Total Correlation (TC) was described by McGill (9544) and formally formulated by Watanabe (1960) to evaluate the mutual independence of imulti-vrian variables. Assume set of of random variables Zi,..., Zn. The TC z is expressed as:
C (Z)..... Zn)
H (z;) H (Z1, Z2...., zn)
(12)
where H (Zz, 22,..., 2,) denotes the joint entropy. Furthermore, according to the work of Te Sun 1980 1975), the relation between TC and mutual information can be described as:
C (z) = (Z) =
Σ AH (z) n22
(13)
The general AHl (z) is defined as Fano's multiple mutual information among variables. In the case of n. 2, reduces Shannon's 's mutual information and in the case of it coincides with McGill's mutual information. The formulation of TC can be simplified as:
C (ZI,...,,Zn)
kn)
Σ 1 ((2zz)) + Σ AHl (2;;2;i8)) + + AH (Zii...
(14)
3
In VAEs, to measure dependence for multiple latent variables, TC is computed as Del (q (z) II, ((z)) Therefore, the proposed disentanglement Eisy 1 (2;22) a lower bound of TC.
14

---

Published as a conference paper at ICLR 2022
B.3 RELATION WITH OTHER WORKS
After decomposing the minimal sufficiency and estimating the disentanglement by TC, we can com- bine two regularizers as:
A (x: z) + X2
(zz:2) zz))
j
(g(())][])) ((
$ xxxx) KL (q (z/x) lpp (z))] X12 E KL (1(zz)) A13 KL
+ A2 Σ KL (g (z,) 110 (z,))
xxx) KT. (q (2|x) Mp (x)) (A2 )12) Σ KT. (2,))))) A13 KT.
((()) (((z)) I[(2s)
(( (
-x,) (g (z) A Σ KL (I(z)) Ae KL
(15)
Therefore, by assigning different weights the decomposed term, we can establish the relationship between the proposed information-theonn objective and existing 3- VAE variants, which will pro- vide insights regarding the capabilities and limitations of existing methods, and further motivate the proposed work.
Specially, the objective function of 8-TCVAE is 2x,2) KL (1(x)) + E, KL (q (zz) (2;)). we decompose the I (x; z) 3-TCVAE as: KL (g (z)x) (2)) KL (g I1, (25)) E, KL (zz) |p (2,)), the regularizer term of 3-TCVAE can be writ- ten as: a (q (z/x) [|p (z)) + (3 a) KL q (z) II, 9 (2,) +(7 (()--))2 a, KL (g (zj) ||p (z,)). Since a = = in 3CCVAE, therefore when A 0, the last line of Equation 15 is equivalent to 8-TCVAE.
C PROOF OF COMPOSITIONAL MINIMAL SUFFICIENCY
In this section, we prove the statements reported the section 2.2 of the paper.
Property:
(Pi): Data- prtceprocsssing inequality in the markov chain: considering the markov chain - - c, then T (b.a) a) 2 7 (b; c). (Theorem 2.8.1 (Cover) 1999b.)
(P2): Chain rule for mutual information: (a: c) I (a; b) (a; c/b). (Theorem 2.5.1 (Covrr 1999).
(P&): Decompositon of conditional mutual information. (Proposition B.1. (Federici et 2020
Conditional independence assumptions:
The Markov Chain m and (m/,, m!) when is observed.
1+1 + x + (m/, m²), implies the conditional independence between m)
/+1
Proof:
(Pt)
T (xx;
2 7 (m [+1 m| m!)
(P)) (m (+1 m|) + (m (+1 1 m| m!)
(16)
(P) (m m!) (m/+); m!) (m/:
15

---

Published as a conference paper at ICLR 2022
D EXPERIMENTAL DETAILS
For all baselines, we use Convolutional Neural Network for the encoder, and Deconvolutional Neural Network for the decoder. Specially, for Factor- VAE, we use 6-layer Multi-Layer Perceptron for the discriminator, with the leaky ReLU as the activation on per layer, and we set 7 as 6.4. For 3 {4, 16, 60). Annealed- VAE uses 1000 with a linearly increasing C' from 0.5 nats to 25.0 nats). 3-TCVAE uses = ? and B = 4. DIP-VAE is implemented by the parameters And 100 and A 10.
V-VAE, we use a
For RecurD, we implement Recur D 0 as the same encoder/decoder architecture with all baselines, as shown Table
for Recur D 1 and Recur 2, we implement the same decoder architec- ture with all baselines, and the details about the encoder of RecurD and Recur D 2 are shown Table 4 and Table 5 Following the previous work, we use negative cross-entropy to compute reconstruction error, which represents sufficiency in our work. As for the computation of minimal sufficiency and disentanglement, we implement the MINE estimator by 4-layer Multi-Layer Percep- tron, similar with the work of et al. (2020). As for the hyperparametter in our model, vary a and A in the set (..1 0.5, 1, 2,5, 5, 10, 50} while fixing λ = and A = 2 for both datasets. During training,
Adam optimiser with learning rate 1e 4, 91 0.9, B2 0.999 for pa- rameter updates. Specially, we utilize RecurD on dSprites, 3DCars and 3DShapes, and RecurD 2 on CelebA.
Table 3: Encoder and Decoder architecture for all baselines and Recur O. Input
dSprites (3DShapes): 64 x 64x 1(3) Images
Conv: k=4, s=2, p=1, channel=32, ReLU
Conv: k=4, s=2, p= channel=32, ReLU
Conv: k=2, p= channel=64, ReLU
Encoder
Conv: k=4, s=2, p= channel=64, RelUU ReLU
FC: 128 (256)
FC: 2 x 8
Input: R
FC: 128, ReLU
FC: 4 x x 64, ReLU
Decoder
Deconv: k=4, s=2, p=
channel=64, ReLU
Deconv: k=4, s=2, p=
channel=32, ReLU
Deconv: k=4, s=2, p=
channel=32, ReLU
Deconv: k=4, s=2, p- channel=1(3), ReLU
Output:
64 x 64x (3) Reconst. Images
Table 4: Encoder architecture of Recur 1. 64 x 64 x 1 (3) Image
Input
Router(v)
0-th Recursive Module
4x [Conv: k=4, s=2, p=l, channel=32, ReLU]
GoE
Conv: k=4, s=2, p=l, channel=64, ReLU
Router(v)
FC, ReLU
1-th Recursive Module
Conv k=4, s=2, p=1, channel:1, ReLU
GoE
x
FC, 16; FC, 2 x 1
16

---

Published as a conference paper at ICLR 2022
Table 5: Encoder architecture of Recur D 2.
Input
64 x 64x (3) Image
Router(v)
0-th Recursive Module
GoE
2x [Conv: k=4, s=2, p=1, channel=32, ReLU ]
Router(v)
FC, 4, ReLU
1-th Recursive Module
GoE
4 [Conv: k=4, s=2, p=l, channe)=16, ReLU]
Router(v)
FC, 16, ReLU
2-th Recursive Module
Conv k=4, s=2, p=1, channel=16. ReLU
GoE
16x
FC, FC,
E
ADDITIONAL EXPERIMENTS
This section reports additional experiments, which draw similar conclusions that of Section
E.1
QUANTITATIVE RESULTS
The first set of quantitative results is the supplementary results of disentanglement score DCI-C and DCI-I dSprites and 3DShapes. Table Figure and Figure show the results of disentanglement score with reconstruction error, which also show RecurD can achieve a better trade-off between reconstruction and disentanglement.
dSprites
dSprites
Pacurt
CROSS
0.5
0.4
0.25
DCL-C
0.3
0.2
0.15
0.1
0.9
05
1.0
3.5
2.0
25
0.5
1.0
1.5
2.0
Reconst. Error
Reconst. Error
Figure 8: Reconstruction error vs. disentanglement performance on dSprites. Scatters located at the left top indicate better performance.
DDShppe
3DShupe
ATONE
BAKE
Racart oom wo/
ARROW
0.60
Anarira
0.30
DCI-C
DC1-1
0.25
0.50
0.20
0.45
0.15
Reconst. Error
Reconst. Error
Figure 9: Reconstruction error vs. disentanglement performance on 3DShapes. Scatters located at the left top indicate better performance.
The second set is the performance comparison results with three additional baselines, i.e, Info- GAN (Chen al., 2016). Control- VAE and NVAE. InfoGAN (Chen al. 2016) maximizes the mutual information between the small subset of the latent variables and the observations to increase the interpretability of the latent representation. Control-VAE (Shao al. 2020) dynamically tunes
17

---

Published as a conference paper at ICLR 2022
Table 6: DCI-C, DCI-I scores and reconstruction the test for dSprites and 3DShapes..
Bold face indicates the best results, i.c., reconstruction error and disentanglement scores. dataset
dSprites
3DShapes
Reconst error
DCI-C
DCI-I
Reconst error
DCI-C
DCI-I
3-VAE(B = 4)
0.0066
0.1148
0.2181
0.0216
0.4463
0.1982
3-VAF(B = 16)
0.0094
0.1575
0.2087
0.0544
0.4691
0.1828
3-VAE(3 60)
0.0127
0.1261
0.1769
0.0624
0.4215
0.1641
Annealed- VAE
0.0171
0.1793
0.1663
0.0811
0.4419
0.1682
Factor- VAE
0.0228
0.1375
0.1699
0.0800
0.4419
0.1707
3-TCVAE
0.0162
0.1988
0.1660
0.0312
0.4516
0.1774
DIP-VAL -ARE
0.0213
0.0968
0.1496
0.0213
0.4621
0.1961
RecurD
0.0047
0.3835
0.2222
0.0083
0.5668
0.2014
the weight 3 the KL term to achieve good trade-off between disentanglement and reconstruc- tion quality. NVAE (Vahdat Kautz, 2020) optimizes high-quality image generation via global correlation capturing across multi-layer latent variables.
We compare with Info-GAN Control-VAL and NVAE on three datasets, including dSprites, 3Dshapes and 3Dcars. The 3DCars (Reed et al.) 2015) exhibits different car models from Fi- aler al. (012) under different camera viewpoints. The evaluation method the disentanglement score MIG. Table 7 show that NVAE and VampPrior achieve comparable reconstruction error compared to RecurD. However, their disentanglement qualities are not as good as RecurD, because Recurl additionally regularizes the inter-layer information sharing and alleviates the information dundancy of multiple layers. This experiment demonstrates the superiority of the proposed Recurl) method in disentanglement compared to existing hierarchical VAEs.
Table MIG score and reconstruction error on the test sets for dSprites, 3DShapes and 3DCars.
Bold face indicates the best results, i.c., reconstruction error and disentanglement scores.
dataset
dSprites
3DShapes
3DCars
Reconst. Error
MIG
Reconst. Error
MIG
Reconst, Error
MIG
3-VAF(3 = 4)
0.0066
0.2617
0.0216
0,2519
0.0376
0.1015
InfoGAN
0.1598
0.1874
0.1083
Control- VAE
0.0102
0,2455
0.0357
0,2630
0.0257
0.1583
NVAE
0.0041
0.0043
0.0078
0.0081
0.0118
0.0034
RecurD
0.0047
0.2707
0.0083
0.3105
0.0132
0.1762
E.2 ABLATION STUDY
In this section, we report the supplementary results of ablation study. The first is to study the impact of compositional learning objective with varying regularization coefficients (A1 and X2) of minimal sufficiency and disentanglement. Figure E.2 and Figure E.2 show the scatter plots of disentan- glement scores along with reconstruction eтror on the test sets of dSprites, with varying Åi, X2- Figure 12 and Figure 13 show the results on the 3DShapes.
The second set is to evaluate the influence of hyperparameter k - the group size in the GoE of Recursive Module. Figure E.2 and Figure E.2 show the influence of varying k for disentanglement scores and reconstruction enror the test sets of dSprites. Figure E.2 and Figure E.2 show the results on the 3DShapes.
The third set is to evaluate the disentanglement metrics the preceding layers. We compute MIG on 1-1 and m !-? of RecurD on 3DShapes. As shown in Table g high-level representations have higher MIG than that of low-level representations (z m L-11
m m These results confirm that
18

---

Published as a conference paper at ICLR 2022
the recursive propagation of inductive bias through the feed-forwart network improves disentangled representation learning.
The last set is to evaluate the impact of architecture of Gate-of- (GoE), conduct three variant GoEs, including Linear: GoE is implemented as a linear layer with softmax; Fix: GoE is implemented fix assignment, i.e., we split my in d 1 equal slices; and Att: GoE is implemented as a multi-head attention layer (the head is fixed as 8). As shown in Table this study demonstrates that GoE indeed fits well with the disentanglement process.
dSprites
dSprites
A A.0,Amm
A 8.Ag
SAP
0.04
0.30
0.03
0.20
0.02
Au 8,
0.01
0.00
0.005
0.007
0.004
0,005
0.006
0.007
0.008
Reconst. Error
Reconst, Error
Figure 10: Ablation study on d1 and X2 on dSprites.
dSprites
dSprites
A
8
0.50
0.50
DCI-C
0.40
DCI-D
0.40
0.30
0.30
0.20
0.20
2, 8.Ay
Ag-8.Ag=0
0.006
0.004
0.005 Reconst, Error
0.006
0.007
0.008
Reconst. Error
Figure 11: Ablation study A and X dSprites.
Table 8: Comparison of MIG on 2, m and m 4-2 Reconst. Error
m4-2
m
=(m²)
MIG
0.1081
0.2892
0.3105
0.0083
19

---

Published as a conference paper at ICLR 2022
3DShape
3DShape
0.16
0.0.A,
0.7g
0
Aq 0,4g =
0.28
0.12
SAP
MIG
0.26
0.24
0.08
0.22
0.06
0.20
0.006
0.007
0.008
0.009
0.010
0.011
0.006
0.007
0.008
0.009
0.010
Reconst. Error
Reconst. Error
Figure 12: Ablation study on A and A2 on 3DShapes.
3DShape
3DShape
-=000
A A,
-
0.56
A -
0.56
0.54
DCI-C
DCI-D
0.52
0.48
0.50
0.44
0.40
0.44
0.006
0.007
0.008
0.009
0.010
0.011
0.006
0.007
0.008
0.009
0.010
0.011
Reconst, Error
Reconst, Error
Figure 13: Ablation study on x and X2 on 3DShapes.
dSprites
dSprites
0.090
0.60
0.075
0.50
0.060
MIG
0.10
0.045
20.30
0.030
0.20
0.015
0.10
0.000
0,400
0.5UD 0.4/
1e-2
0.375 0,400 0.425 0.450 0.475 0.500 0.525 0.550 Reconst. Error
Reconst. Error
Figure 14: Ablation study on k on dSprites.
20

---

Published as a conference paper at ICLR 2022
dSprites
dSprites
0.70
0.60
0.50
DCL-C
DCI-D
0.50
0.40
0.40
0.30
0.30
0.20
0.20
0.10
0.3750.400 0.425 0.450 0.475 0,500 0.525 0.550 1e-2
0,3750.000 0.425 0.450 0,475 0.500 0.525 0.550 1e-2
Reconst. Error
Reconst. Error
Figure 15: Ablation study on on dSprites.
3DShapes
3DShapes
0.225
0.200
5
0.150
0.30
0.125
0.100
0.28
0.075
0.26
0.050
0.025
0.006
0.007
0.008
0.009
0.010
0.011
0.012
0.006 0.007
0.008
0.009
0.010
0.011 0.012
Reconst. Error
Reconst. Error
Figure 16: Ablation study on & on 3DShapes.
3DShapes
3DShapes
0.54
0.54
k=5 X=7
0.52
0.52
DCLC
0.48
10.4
DC
0.44
0.006
0.007
0.008
0.009
0.010
0.011
0.012
0.006 0.007 0.008 0.009 0.010 0.011 0.012
Reconst, Error
Reconst, Error
Figure 17: Ablation study on & on 3DShapes.
21

---

Published as a conference paper at ICLR 2022
Table 9: Ablation study the GoE architecture.
Linear
Fix
Att
RecurD
MIG
0.2539
0.2617
0.2604
0.3105
Reconst. Error
0.0176
0.0116
0.0095
0.0083
E.3 QUALITATIVE RESULTS
In this section, report the qualitative samples of traversal images three datasets, including dSprites(Figure 18 3DShapes(Figure 19 and CelehA(from Figure 20 to Figure
Specifically, on CelebA, tentatively increase the dimensionality of latent variables of Recurl) (from 16 to 32) by doubling the output of the last layer of each encoder from a single latent variable 2-dimensional variable.
Figure 18: Traversal samples on dSprites.
Figure 19: Traversal samples on 3DShapes.
Figure 20: Traversal samples on CelebA (Azimuth).
Figure 21: Traversal samples on CelebA (Background Color).
22

---

Published as a conference paper at ICLR 2022
2
2
Figure 22: Traversal samples CelebA
Figure 23: Traversal samples CelebA (Face Width).
(Face Color).
Figure 24: Traversal samples on CelebA (Gender).
Figure 25: (Smilc).
Traversal samples on CelebA
E.4 EVALUATION OF COMPUTATIONAL COSTS
In this section, we evaluate the computational complexity of our model on 3DShapes and CelebA. The evaluation metrics consist of multiply-accumular operation (MACs), the model parameters (Params), evaluation time (Eva. time) and converge time (all models converge to the same recon- struction error as 3-VAE, denoted as Con. time).
In terms of parameter efficiency, as shown in Table 10 the recursive disentanglement network itself only contains 0.826 million parameters (RecurD w/o MINEs), which are comparable Beta- EAE. Most of the parameters of RecurD 2 are contributed by using MINE estimator. Specifically, in the initial implementation (Recurl) 2/specific MINEs), each pair of outputs of encoders is equipped with specific MINE model, which contributes 7.214 million parameters. We further optimize the design by using shared MINE within the same feature category, which can effectively reduce the total number of parameters down to 1.325 million (RecurD 2/shared MINEs).
Table 10: Complexity comparison of three models on 3DShapes and CelebA.
Dataset
Method
MACs(G)
Params(M)
Eva. time(s)
Con. time(s)
RecurD 1
3.466
3.672
0.005124
27.1212
RecurD 2
3.469
3.694
0.005354
23.3197
3DShapes
Beta-VAB
3.144
0.769
0.003283
32.5251
Factor- VAE
3.401
4.779
0.003389
40.6231
Recurl 1
3.944
7.935
0.01065
29.7421
RecurD 2
3.957
8,040
0.01087
36.6408
CelebA
RecurD 2 w/o MINES
2,960
0.826
RecurD 2/shared MINEs
3,654
1.325
0.01072
36.6402
beta-VAK
3.145
0.769
0.01030
45.2067
Factor- VAE
3.402
4.792
0.01005
48,9939
23


Reviews:
[{"review_id": "mzNr-A_6wb0", "reviewer": "Reviewer_VLG5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the idea is technically sound and the results look promising. I have some questions and suggestions and hope the authors could clarify them during the rebuttal.\n\n* Section 2.1: what is the meaning of defining Markov Chain as \\hat{x} -> x -> z, given that the generative process is actually x -> z -> \\hat{x}? I looked at the cited work (Achille & Soatto, 2018), and they seem to discuss a different setting, where they have a dataset of data points x and associated labels y, and in that case, y -> x -> z makes sense to me.\n\n* Definition 5: Similarly, what is the practical meaning of m_j^{l+1} -> x -> (m_i^l,m_j^l)?\n\n* Figure 6 left: do you use compositional objective (Section 2.2) and recursive disentanglement network (Section 3) in this experiment? If so, lambda_2=0 is not equivalent to beta-VAE as the vanilla beta-VAE does not have the compositional objective.\n\n* Figure 2: the 'w_j^l' in green should be 'w_{d_{l+1}}^l'?\n\n* I would suggest adding GAN-based approaches into the comparison tables. This would be very helpful for readers who want to pick techniques for their downstream applications. \n\n* The decomposition and discussion of the losses around Eq. 2 and Table 1 have been partially discussed in prior work (e.g., https://arxiv.org/abs/1706.02262), and I believe it is not your key contribution. I would suggest highlighting this fact better as it sounds like these are your discoveries from the current writing.", "summary_of_the_paper": "The paper proposes a new approach for learning disentangled variational autoencoders. In addition to pushing the sufficiency, minimal sufficiency, and disentanglement of the latent representation, the paper proposes to also regularize those on earlier features in the network. Experiments demonstrate promising results.", "summary_of_the_review": "In summary, the idea is interesting and the results look promising. I hope the readers could clarify these questions and I will adjust the score accordingly.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eXr9dxh4CLI", "reviewer": "Reviewer_1v1o", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ The paper presents an approach that is theoretically justified and implemented in a convincing manner.\n+ The results appear convincing and beating relevant baselines (though with the disclaimer that mostly these baselines seem rather old by now; some of the relevant comparisons from the last 2 years might be missing, but I cannot name any).\n+ Empirical evaluations are sufficient (though only barely), except for the ablation (*):\n\nWeaknesses\n- I suspect that including layer-wise disentanglement has occurred to many people before, but it has not been attempted due to the computational burden. (That said, I have not seen anyone actually try it.)\n- It is unclear whether this approach *solves* the computational burden and scales to more complex datasets and larger image resolution\n- I am not convinced that the loss formulation (Eq. 4) is that significant wrt. InfoVAE, for example. What would the authors say against the criticism that Eq. 4 is basically just reshuffling the InfoVAE loss? (I'm not saying it is, but it would be helpful if the authors could shoot down this potential concern.)\n- (*) It is unclear to what extent does the performance originate from the loss (Equation 4), and to what extent does it come from the switch-based architecture. Could the authors clarify this? It would seem that one could implement Eq. 4 loss without the switch architecture?", "summary_of_the_paper": "The paper presents a VAE variant that disentangles the features of the inference network at every layer, with disentanglement defined in terms of mutual information between features. The approach is implemented as \"recursive disentanglement network\" based on a switch network (aka Mixture-of-Experts gate, introduced in Shazeer 2017 and used in the switching transformers). The results in dSprites and 3DShapes dataset suggest this variant performs better than well-known disentanglement VAE networks (from few years back) in dSprites and well in 3DShapes (though not the best in all measures). In terms of VAE loss, the approach is presented as a generalization of various other disentangling VAEs.", "summary_of_the_review": "The paper appears to simultaneously give a clear slightly novel generalization to disentanglement losses of prior VAE variants, and to provide an architectural approach to implement their approach.\n\nI have concerns about the scalability, and I am suspicious about whether such a heavy-handed disentanglement can be maintained in larger models.\n\nIt also was not clear to me whether the results originate from the loss or the architecture advances (which build on existing switching architectures).\n\nHowever, either way, I think this is a novel approach and potentially a significant addition to the VAE disentanglement research, and I lean towards acceptance in this case, though I urge the authors to address my questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dG6q7sXtnsw", "reviewer": "Reviewer_yTV3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I am pretty happy with the paper. It's mostly well written and organised.\n\n1. Positives\n\n* Session 2 (Compositional disentanglement learning) is well organised and sets up the scene well for the method in session 3.\n* Good level of implementation detail is available, such as architectures, estimator used, etc. The experiments are well conducted and common mistakes were avoided AFAICT.\n* Use of standard datasets and metrics well stablished in the field.\n\n\n2. For improvement:\n\n* There has been some progress in the use of hierarchical VAEs, which can be interpreted as applying disentanglement regularisation to other layers and making it compositional in a similar fashion to this work, e.g. NVAE (A Vahdat, J Kautz 2020). \n* I would be a bit more careful with the tone on claims about the requirement of compositionality for disentanglement. Figure 1 is only an evidence in a toy example, not an actual demonstration. So statements as \" fig 1 shows when ... is not effectively disentangled.\" (session 1) and \"To achieve better disentanglement between ... their input feature sets .. are expected to be disentangled as demonstrated in our case study\" (session 2.2) could be watered down a little.\n* It's unclear if there are benefits from using MINE and the architecture in some of the experiments. For example, if I understood correctly, the beta-VAE objective yields better metrics on figure 6 than in figure 3.\n* The paper left me wondering what are the disentanglement metrics on the preceding layers, looking into MIG/SAP/DCI-D on m^L-1 and m^L-2 seems like a straightforward analysis that should be in the appendix or even in the main paper.\n* Some ablations on the architecture itself also seem to be missing (ablations on the loss are good). My intuition is that Gate of Mixture-of-Experts fits quite well with the disentanglement that we want, because of the top . For example:\n  * just learning a linear+softmax instead of the Router\n  * no routing at all, just some fixed assignment: e.g. split m_l in d+1 equal slices and pass through the encoders.\n  * using a transformer instead of the Recursive Modules, perhaps over fixed slices as well.", "summary_of_the_paper": "This paper formulates the disentanglement problem from an information theoretic perspective, but focusing on an objective that encourages a compositional disentangled feature space on the layers that precede the final latents.\n\nWith objective, the authors describe a new method using Gate of Mixture-of-Experts to implement the compositional disentangled reconstruction objective. Some of the terms require mutual information estimation, for which they use MINE estimators.\n\nThey run experiments across dSprites and 3DShapes and look into reconstruction error and different disentanglement metrics, observing that they method outperform existing beta-VAE-like baselines, without any compositional incentives. They also analyse the loss components with different architectures and observe that degrees of compositionally in the architecture yields better disentanglement.\n\nFinally, they look into some ablations of the regularisation pressure and into data efficiency in downstream tasks.\n\n ", "summary_of_the_review": "My main concern is the discussion of related hierarchical models missing from related work and the emphasis on this being the only work to apply some disentanglement pressure outside the main z latents. This should be an easy fix for this paper.\n\nThe compositional objective is interesting and novel and the implementation method is clean. The experiments were well conducted and the well analysed. Overall, I am confident that the authors will be able to address the main issue above and that this paper will award acceptance in this venue.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WG2yryZanIN", "reviewer": "Reviewer_p3Zz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Developed a compositional disentanglement learning, called RecurD, that directs the disentanglement learning process across the compositional feature space.\n2. Provide some theoretical analysis based on information theory. \n\nCons:\n1. Optimizing the lower bound of Eq.2 does not mean obtaining the optimal objective function of  $\\beta$-TCVAE paper [1] on the right-hand side of Eq 2. As far as I know, the right-hand side of Eq 2 is the objective of $\\beta$-TCVAE. If we optimize the objective function on the left-hand side, it does not hold for optimizing $\\beta$-TCVAE. Thus, I am afraid that the proposed objective function in Eq 1 fails to be generalized to the existing $\\beta$-TCVAE and FactorVAE. In contrast, optimizing the objective of $\\beta$-TCVAE is approximately equivalent to optimizing the proposed objective function in this paper. What is more, in Table 1, $\\lambda_c =1$ for the original $\\beta$-TCVAE in their paper. \n2. Do not specify the number of Gate of Encoders (GoE) for different datasets. It is hard to know how many GoE should be used for a new dataset, like CelbeA, that contains 40 latent variables. Also, I am curious about the complexity of the proposed network?\n3. The upper bound and lower bound are confounding. On page 9, the author said: mutual information I(x,z) is the upper bound of KL divergence. In fact, based on the proof in prior work [2], I(x,z) is the lower bound of KL divergence. \n4. The Markov Chain in Eq 7 is incorrect. Based on my understanding, the next state $X_{t+1}$ of Markov chain is only related to the current state $X_t$. Hence the joint probability of p(a,b,c) = p(a)p(b|a)p(c|a) rather than p(a)p(b|a)p(c|b). This is because c is conditionally independent on b as you mentioned, b->a->c.\n5. Missing baselines in recent years. The author only discussed and compare the proposed method with the baselines before the year of 2019. There are some recent works [3] [4] [5] on improving the disentanglement and reconstruction error. For instance, ControlVAE [5] dynamically tunes the weight \\beta on the KL term to achieve a good trade-off between disentanglement and reconstruction error.\n6. Did not conduct experiments on complex datasets. The authors should do experiments on 3D chairs or CelebA to demonstrate the good performance of the proposed method.\n7. The result in Fig 5 does not perform well. We can observe that for orientation and scale factors, they are slightly entangled. Besides, the reconstruction quality is not as good as ControlVAE and FactorVAE in the paper. In particular, ControlVAE and FactorVAE  can disentangle both 5 latent factors which are better than those in this work. \n8. There are some typos in this paper, please proofread this manuscript. For instance, priopr work --> prior work on Page 5.\n\nReference:\n[1] Chen, R. T., Li, X., Grosse, R., & Duvenaud, D. (2018). Isolating sources of disentanglement in variational autoencoders. arXiv preprint arXiv:1802.04942.\n[2] Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, Sergey Levine: Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow. ICLR 2019.\n[3] Patrick Esser, Johannes Haux, Bj rn Ommer: Unsupervised Robust Disentangling of Latent Characteristics for Image Synthesis. ICCV 2019.\n[4] Srivastava, Akash, Yamini Bansal, Yukun Ding, Cole Hurwitz, Kai Xu, Bernhard Egger, Prasanna Sattigeri, Josh Tenenbaum, David D. Cox, and Dan Gutfreund. \"Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modelling.\" arXiv preprint arXiv:2010.13187 (2020).\n[5] Shao, H., Yao, S., Sun, D., Zhang, A., Liu, S., Liu, D., ... & Abdelzaher, T. (2020, November). Controlvae: Controllable variational autoencoder. In International Conference on Machine Learning (pp. 8655-8664). PMLR.\n\n", "summary_of_the_paper": "This paper proposed a recursive disentanglement network (RecurD) for the learning of disentangled representations from information theoretic perspective. The experimental results show RecurD outperforms some existing baselines on two benchmark datasets.", "summary_of_the_review": "I think the authors have addressed most of my concerns. I will increase the final rate.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an algorithm for achieving disentangled representations by encouraging low mutual information between features at each layer, rather than only at the encoder output, and proposes a neural architecture for learning. Empirically, the proposed method achieves good disentanglement metric and likelihood (reconstruction error) in comparison to prior methods. The reviewers think that the methodology is natural and novel to their knowledge, and are happy with the detailed execution. The authors are encouraged to improve the presentation of the paper, by providing rigorous formulation of the "Markov chains" to avoid confusions, justification of the independence assumptions behind them, and more in-depth discussions of the learning objectives.
Decision: Accept (Poster)