Variational Autoencoders with Decremental Information Bottleneck for Disentanglement
Authors: Jiantao Wu
Shentong Mo
Muhammad Awais
Sara Atito
xingshen zhang
Lin Wang
Xiang Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: disentanglement
variational autoencoders

Abstract: One major challenge of disentanglement learning with variational autoencoders is the trade-off between disentanglement and reconstruction fidelity. Previous methods, spreading the conflict of disentanglement and reconstruction in time, will lose the constraint of disentanglement when expanding the information bottleneck, which causes the information diffusion problem. To tackle this issue, we present a novel decremental variational autoencoder with disentanglement-invariant transformations to spread the conflict on multiple latent spaces, termed DeVAE, for balancing disentanglement and reconstruction fidelity by decreasing the information bottleneck of diverse latent spaces gradually. Benefiting from the multiple latent spaces, DeVAE allows simultaneous optimization of multiple objectives to optimize reconstruction while keeping the constraint of disentanglement, avoiding information diffusion. DeVAE is also compatible with large models with high-dimension latent space. Experimental results on dSprites and Shapes3D that DeVAE achieves the best performance on both disentanglement and reconstruction.

Under review as a conference paper at ICLR 2023
VARIATIONAL AUTOENCODERS WITH DECREMENTAL INFORMATION BOTTLENECK FOR DISENTANGLEMENT
Anonymous authors Paper under double-blind review
ABSTRACT
One major challenge of disentanglement learning with variational autoencoders is the trade-off between disentanglement and reconstruction fidelity. Previous in- cremental methods with only on latent space cannot optimize these two targets simultaneously, they expand the Information Bottleneck while training to op- timize from disentanglemmnt to reconstruction. However, large bottleneck will lose the constraint of disentanglement, causing the information diffusion problem. To tackle this issue, we present a novel decremental variational autoencoder with transformations to optimize multiple objectives in dif- ferent layers, termed VAE, for balancing disentanglement and reconstruction fidelity by decreasing the information bottleneck of diverse latent spaces grad- ually. Benefiting from the multiple latent spaces, DeVAE allows simultancous optimization of multiple objectives to optimize reconstruction while keeping the constraint of disentanglement, avoiding information diffusion. DeVAE is also compatible with large models with high-dimension latent space. Experimental results on dSprites and Shapes3D that DeVAE achieves good balance between disentanglement and reconstruction. Ve shows high tolerant of hyperparam- eters and on nigh-dimensional latent spaces.
disentanglement--ivariaa
INTRODUCTION
Unsupervised learning for sensing the properties of objects is crucial to reduce the gap between hu- mans and machines intelligence. Inline with human intelligence disentanglement learning (Bengio et al., 2013) is considered to be promising direction to obtain explanatory representations from observations to understand and reason objects without any supervision.
In the recent years, various approaches (Higgins et al., 2017; Chen et al., 2018; Kim & Mnih, 2018; Burgess et al., 2018; Chen et al., 2016) have been proposed to successfully extract basic prop- erties of objects, such as position, color, orientation, and scale (Burgess & Kim, 2018; Matthey et al., 2017). The commonly-used methods are based on variational autoencoder (VAE) (Kingma & Welling, 2014). In particular, 3-VAE (Higgins et al., 2017) introduced extra parameter 3 on the Kullback-Leibler (KI.) divergence to promote disentanglement. However, there a trade-off between disentanglement and reconstruction fidelity on 3-VAE, which problem be solved in the following works.
One common direction for dealing with the trade-off is to penalize the Total Correlation (TC) be- tween latent variables, avoiding reducing the mutual information, such as FactorVAE (Kim & Mnih, 2018), 3-TCVAE (Chen al., 2018), and DIPVAE (Kumar al., 2018), As pointed out in (Träuble et al.. 2020; Dittadi et al., 2020), TC-based VAES have a strong prior assumption that the factors are statistically independent. Beyond that, when comes high-dimension latent space, the esti- mation of TC becomes inaccurate due to the curse of dimensionality, as our experiments observed in Section 3.2. The realistic problems usually have numerous factors, therefore would need a large model with high latent space to extract representations. For example, the popular deep model ResNet50 (He et al., 2016) has 2048 dimensional feature space. However, the current TC esti- mations are not scaled high dimensional problems, causing the low performance of BC-based methods in practice. In this work, instead of calculating TC, we leverage the information bottleneck (IB) (Tishby al., 1999; Burgess al., 2018) to promotes disentanglement.

---

Under review as a conference paper at ICLR 2023
In the meanwhile, previous information bottleneck (IB)-based methods (Burgess et al., 2018; Shao et al., 2022; Wu et al., 2022) have tried to solve the obstacle of trade-off between disentanglement and reconstruction fidelity. A narrow IB enforces the model to find efficient codes for representing the data, which encourages disentanglement. Therefore, they first set a high pressure with a narrow IB and then expand the IB gradually promote disentanglement reconstruction fidelity, termed incremental methods. For example, Dynamic (Shao et al., 2022) initiated 3 with a large value at the beginning of training for disentangiement and stably increase the KL divergence for reconstruc- tion by non-linear PI controller (Äström & Hägglund, 2006). However, they lost the constraint of disentanglement when expanding the IB, which causes the information diffusion problem (Wu et al., 2022). In this work, to avoid information diffusion, we aim to optimize reconstruction while keeping the constraint of disentanglement.
Different from IB-Incremental based approaches listed above, our key motivation is to optimize disentanglement and reconstruction simultaneously. revious methods only have one latent space and unable to optimize disentanglement and reconstruction the same time, which causes them to have to change the target from disentanglement to reconstruction during training. Instead, our work proposes novel multi-layer framework with its latent spaces and objectives in each layer, allowing optimizing multiple targets at a time. In this way, the first layer a vanilla VAE to rebuild high-quality images, and the subsequent layers will distill some important variables by narrow IBs to promote disentanglement. To inherit disentanglement from the subsequent layers, we introduce saannglement-invarian transformations to connect the layers one by one. These extra layers can be seen as regularizations for disentanglement to constrain the representation.
To achieve this, we propose a simple yet effective VAE framework composed of multiple continuous latent sub-spaces with a novel IB-Decremental strategy and eeeentanglementiinvaiiin transform operators, which we call DeV.E. Specifically, we decrease the information bottleneck of each latent space layer by layer, where we constrain the first space for informativeness to recover the input image, and other disentangled spaces for learning factors of the image by narrow IBs. Furthermore, we introduce the disentaagleeenntinvaarie invariant transform operator to ensure simultaneous optimization of disentanglement across continuous latent sub-spaces, which avoids the information diffusion. Our decremental model avoids ID by keeping the constraints of disentanglement while optimizing reconstruction. We also conducted comprehensive comparisons with popular methods quantitatively and qualitatively. Experimental results demonstrate that VAB is robust in hyperparameters and the size of latent spaces.
Our contributions can be summarized as follows:
We introduce several latent spaces sharing disentanglement by disentanglemenntinveee invariian transformations.
We propose a novel diagram for disentanglement learning by decreasing IB, termed decre- mental VAE (DeVAE). Our decremental model can handle large- acale problems and show robustness on several datasets.
METHODOLOGY
2.1 PRELIMINARIES
Problem Setup & Notations. Disentanglement learning aims to learn the factors of variation which raises the change of observations. Given a set of samples o E X, they can be uniquely described by set of ground-truth factors C. Generally, the generation process g(-) invisible æ = g(c). We say a representation for factor e; is disentangled if it is invariant for the samples with Cj- We variational inference to learn the disentangled representation for given problem. p(z| denotes the probability of z f(x), p(=[z) denotes the probability representation function is a conditional Bayesian network of the form go (2(()) estimate p(z(æ=). The generative model is another network of the form pa( xel=))((2) 2). ø, 0 are trainable parameters.
x = g(z). The
Revisit VAE & 3-VAE. The VAE framework (Kingma & Welling, 2014) computes the repre- sentation function by introducing {s(2|ar) ar) and optimizing the variational lower bound (ELBO).
2

---

Under review as a conference paper at ICLR 2023
D
Layer
10
Encoder
Decoder
Layer
Figure I: Illustration of our Decremental Variational Autoencoder (DeVAE). The solid lines denote the information flow of the encoding process. The dash lines denote the decoding process which randomly selects layer's representation and concatenates the corresponding embedding vector. v denotes a layer embedding. т; denotes ddsentanggeeenn--ivvriaa transformation. μ4, a; denote the parameters of latent variables 2;. N denotes a random noise. Each layer has pressure 3, to control the capacity of IB.
3-VAE (Higgins et al., 2017) introduces the hyperparameter 3 to control the IB:
C(0, Φ) (=)m) (log - (+
(I)
Consider using 3-VAE to leam representation of the data; the representation will be disentan- gled but lose information when 3 is large (Burgess et al., 2018). We can set large / to learn a disentangled representation and small 33 to learn an informative representation.
However, previous disentanglement methods (Higgins et al., 2017; 7; Chen et al., 2018; Burgess et al., 2018) are limited in low-dimension latent space and poorly deal with the trade-off between disen- tanglement and reconstruction. Current state-of-the-an approach (Shao et al., 2022) with an an- nealing manner from high pressure to low pressure will loosen the constraint of disentanglement when reducing the pressure. To address this issue, we propose novel decremental variational autoencoder with hierarchical latent spaces, namely DeVAE, optimize disentanglement and construction fidelity simultaneously, which can handle high-dimensional latent spaces, shown Figure 1. Our DeVAE applies a hierarchical structure with a decremental information bottleneck and iissenttnglementinnaariiin transformation to produce latent variables layer by layer. The decoder part randomly selects one layer's 's latents concatenating an embedding vector to generate images.
2.2 HIERARCHICAL LATENT SPACES WITH DECREMENTAL INFORMATION BOTTLENECK
In order to retain the disentanglement constraint while optimizing the reconstruction fidelity, we introduce a Hierarchical Latent Space (HiS) with K layers and assign pressure for the ith layer Z, to promote disentanglement.
The first layer aims rebuild the dataset and the ELBO as objective. The subsequent layers will promote disentanglement by reducing the IB. Therefore, the objective of the ith layer is
C, Φ) (e. (x z,, v.)] - (xx)
where v, E R D denotes the learnable layer embedding for the ith layer, Pe (z/
Pa(x z, v.) is the de- coder network shared with all layers, qa(2; (æ) the encoder network dependent previous ones, R.K is set of coefficients penalize the IB, particularly 3o 1. The parameters of each layer

---

Under review as a conference paper at ICLR 2023
parameterized bottom up process:
=(=])=) (()
= (2)((())) -
where the first layer go( ()) conditional Bayesian network, Ti denotes a transformation to modify the poster distribution of the previous layer to fit the layer objective ())
According information theory, information can only decrease while processing, therefore we gradually decrease the IB in the sequential layers, i.e., 88+1 > 8, In this way, the last layer with a narrow IB can promote disentanglement only, and the reconstruction fidelity will become better and better from the bottom to the top.
2.3 DISENTANGLEMMNT--NAAIII
TRANSFORMATION
Though we create multiple latent spaces, these objectives only encourage the local representations be disentangled informative. We need mechanism to connect these objectives for balancing disentangiement and reconstruction in one layer. In order to make sure disentanglement across all latent layers, we propose a disentanglement-invaan invaaiaaan transformation (DiT) denoted as т.
Theorem w- disentangled if z is disentangled, w is a diagonal matrix.
Proof in Appendix A.2.
According to Theorem 1, we can scale the latent space to keep disentanglement. Scaling the poste- rior z; violates the generation process which wants the marginal distribution g(z) Σ (z (=(x))((()))) to be close to standard normal distribution. Besides, most downstream tasks use the mean repre- sentation instead of sampled representation. Therefore, we only need the mean representations disentangleeennt-nnaattee -nvvaraan.....
The disentanglement-invaaan invariaat transformation scales the parameters of the ith layer:
Tw' ww σ h(w" (u,,h("))))) h(w")o
(4)
where w², w are learnable diagonal matrices belonging the ith layer, h(w) e is the power function to make sure the scaling values greater than Therefore, we get the parameters ith latent variables
μ1 h
Mo:
a: = h Σ 1=0
wf) i>0 0.
(5)
and the KL divergence
i-1 1x(2
i-1 1/2
DKL.- (1 + Σ m² + + log((o)
w
2 w)) 1=
(6)
1=0
1=0
2.4 OPTIMIZATION ALGORITHM
In this section, we combine the above components and introduce the optimization algorithm for the multiple objectives. We use random process to optimize one layer's objective from K latent spaces:
[(0, [c. (0,
where p(z:) denotes the probability of optimizing the ith latent space 2,, which is defined as:
for =
R
p(z,)=
(8)
(1 (1-s)
1++ for
where s denotes the power annealing of scale hyper- parameter for each p(z,). In experiments, we empirically find that achieves better performance, observed in Section 3.3. Note that do
4

---

Under review as a conference paper at ICLR 2023
def Joss_fne (x, encoder, decoder, embeddings, betas,
iax np. random. randint (K)
mu, logvar encoder (x)
for i in range (idx):
wl, w2 [[i]
torch. exp (w1 [illil)
logvar logvar + w2[i]
Z = sample (mu, logvar) # re-paranster trick
recon - decoder (torch cat ([2, embeddings (idx) 1, 1))
loss mse (recon, x) betas [idx]
kid (mu, logvar)
11
return loss
Algorithm 1: PyTorch-like implementation of De VAE loss.
not aggregate the objectives into loss, instead, DeVAE only rebuilds the images and optimizes the objective of one layer in one mini-batch.
In model, (z(x)z) and decoder p=(x|z) are modelled by two neural networks, K-size quence 'betas' denotes the penalties on the KL divergences of corresponding layers, w w² stores the learnable parameters for transforming latent spaces. First, we randomly sample mini-bater and choose target layer to optimize. Then use the algorithm introduced in Section 2.3 to obtain the representation of the target layer and reconstruct the corresponding images. Instead of using K separated decoders to rebuild images, we apply shared decoder with layer embeddings to reduce the parameter size. The only extra computational cost comes from w w² and layer embeddings. Therefore, the parameter size and overhead similar to the vanilla VAE. The Py PyTorch-like algo- rithm is shown Algorithm 1.
3 EXPERIMENTS
3.1 EXPERIMENTAL SETUP
widely-used datasets (dSprites, Shapes3D). dSprites (Matthey et al., 2017) has 737,280 binary 64 x 64 x I images generated from five fac- tors: shape (5), orientation (40), scale (6), position X (32), and position Y (32). Shapes3D (Burgess & Kim, 2018) has 480,000 RGB 64 x 64 x 3 images of 3D shapes generated from six factors: floor color (10), wall color (10), object color (10), object size (8), object shape (4), and azimuth (15).
Datasets. We evaluate method
Evaluation Metrics. We apply the following metrics to evaluate the performance of disentan- glement and reconstruction. MIG (Chen et al., 2018): the mutual information gap between two variables with the highest and the second-highest mutual information. FactorVAB metric (Kim & Mnih, 2018): the error rate of the classifier, which predicts the latent variable with the lowest vari- ance. DCI Dis.: abbreviation for DCI Disentanglement (Eastwood & Williams, 2018), matrix of relative importance by regression. Recon.: abbreviation for Reconstruction Error, a measure of the distance between images; for binary images.
Mean Squared Error for RGB images and Binary Cross Entropy
Implementation. We use a convolutional neural network as the encoder and a deconvolutional neural network as the decoder. Detailed architecture can be found in Appendix A. The activation function is ReLU, The optimizer is Adam (Kingma & Ba, 2015) with a learning rate of lc-4, 31 = 0.9, 32 0.999. The batch size is 256, which accelerates the training process. All experiments train 300, 000 iterations by default. For the hyperparameters, we set B = 12 for 3-TCVAE, 3 for 3-VAE, and K, 0.001, K, 0.01 for DynamicVAE. According to the information freezing points (IFP) (Wu et al., 2022), beta=40 can filter factors orientation and shape, beta=10 can only filter factor orientation,
set (3.) [1, 10, 40 s 1 for De
5

---

Under review as a conference paper at ICLR 2023
FactorVAE Metric
DCI Dis.
Recon.
80
0.3
dSprites
60
0.4
0.2
0.7
0.3
0.1
0.6
20
0.2
0.6
shapes3D
0.75
60
0.9
0.4
0.70
40
0.2
0.65
0.8
DynamicVAE
B-TCVAE
B-VAE
DeVAE
Figure 2: Box plots of quantitative benchmarks MIG, FactorVAF, Disentanglement, and reconstruc- tion error on dSprites and Shapes3D.
3.2 COMPARISON PRIOR WORK
To demonstrate the effectiveness of the proposed DeVAE we compare it to previous all types of baselines: 1) 3-VAE (Higgins et al., 2017): the popular method for disentanglement and also the baseline model for DeVAE when there is only one latent layer; 2) 3-TCVAE (Chen et al., 2018); the TC-based method with good balance of simplicity and effectiveness; 3) Dynamic-Van VAE (Shao et al., 2022): the latest method with incremental information bottleneck.
Disentanglement & Reconstruction. We conducted experiments on dSprites and Shapes3D to compare the above methods. Each trail repeated 10 times with different random seeds. We draw the distributions of three disentanglement scores and reconstruction errors in Figure 2. The visualization of sampling from the best models is shown in Appendix A.5. Experimental results reveal that Ve achieves an average improvement of 8% comparing to 3-TCVAE and 47% to 3-VAE on dSprites for disentanglement. DeVAE has a lower average reconstruction error on two datasets by 2% for 3-TCVAE and by 30% for 3-VAE, Though the improvement is not significant, f- TCVAE and 3-VAE are not robust to one hyperparameter setting. Though DynamicVAB achieves the best overall results, it still suffers from ID problems and is incapable of dealing with high- dimensional space, see Figure and Appendix A.7.
Qualitative Visualization. We also conducted a qualitative analysis to assess disentanglement. R3Q4We show the selected latent traversals whose KL divergence is larger than 0.5 in Figure 3. We can see that DeVAE disentangles position and position Y perfectly. Shape, scale, and orientation are hard to be disentangled. We show the latent traversals of the best models with the highest MIG in Appendix A.4.
Preventing Information Diffusion. Information diffusion is a phenomenon of disentangling that one factor's information diffuses into other latent variables while training, causing the disentangle- ment scores to fluctuate during training (Wu al., 2022). We hypothesize that losing the constraint of disentanglement is the reason for ID.
To prove it, we monitored the changes in mutual information during training. From Figure 4, we see that Dynamic VAE has significant trend of losing information on iteration 3e5. It means that the learned structure representation was destroyed when expanding the IB. In contrast, DeVAE shows relatively steady trend of increasing information for consistent regularizing. De over- comes the drawback of traditional information bottleneck-basee methods by keeping the constraint of disentanglement.

---

Under review as a conference paper at ICLR 2023
Latent Index
2
5
Figure 3: Latent traversal of De with the best MIG score dSprites. Each block shows the generated images of traversing the latent variable from -2 to 2. Each group shows the traversal images sampling from different random noise. The title above each group denotes the index of latent variable.
DynamicVAE
B-TCVAE
B-VAE
DeVAE
0.6
0.6
0.4
1.0
0.4
0.4
NMI
0.2
0.2
0.2
0.5
0.0
0.0
0.0
0.0
2
lteration
le5
lteration
le5
lteration le5
Iteration le5
shape
scale
orient.
posX
pasY
Figure 4:
Comparison results of information diffusion. Each colored curve denotes the learned information that belongs to one factor over training iterations.
Scaling to High-dimensional Latent Space. Most disentanglement methods evaluate their per- formance on simple scenes with only one object and few factors. It is challenge to extend these methods to complex scenes. However, whether these methods adapt to a large latent space to fit more factors is questionable. In particular, the dimension of latent space affects the estimation accuracy of MI for the TC-based methods.
To study the effect of high-dimensional latent space estimating TC, first generate samples from a D-dimensional multi-variable normal distribution æ which divided into two groups x and 2² with D/2 dimensions. The variables in group are independent Cav( anth) 0, + j: the variables between groups correlative Cov(a:" xx p, n; each variable is standard normal distribution. So, the TC of æ is
D TC(x) = 4 log(1 p²)-
We trained a discriminator for 2000 iterations estimate the TC introduced in FactorVAE (Kim & Mnih, 2018). We compared the estimated TC and the real TC over dimensions and p. Each trail repeated 10 times, and report the average results shown in Table 1. One increasing or dimension diminishes the accuracy of estimation, and the estimators always have low errors when the dimension is 10. However, the estimation becomes extremely inaccurate when
that

---

Under review as a conference paper at ICLR 2023
DynamicVAE
60
B-VAE
Recon.
40
DeVAE
B-TCVAE x x
20
0.1
0.2
0.3
MIG
Figure 5: Distributions of MIG scores and reconstruction errors for low-dimensional space (blue) and high-dimensional space (green). The points in the bottom right have a better balance of disen- tanglement and reconstruction.
Table 1: The estimated MI of FactorVAE and the real MI on high dimensional spaces. The cases having large error are bold. P denotes the correlation of two random variables.
Estimated TC
TC
Error
Dim
p
10
0.3
0.23
0.24
0.03
0.6
1.08
1.12
0.03
0.9
4.14
4.15
0.00
100
0.3
2.17
2.36
0.08
0.6
10.27
11.16
0.08
0.9
23.39
41.52
0.44
1000
0.3
8.62
23.58
0.63
0.6
17.40
111.57
0.84
0.9
22.47
415.18
0.95
the dimension raises to 1000, which means such estimation will fail to penalize the TC for large models.
We further conduct experiments on dSprites to validate the above conclusion. The experimental settings are the same except for increasing the dimension latent space 1024. From Figure can see that 3-TCVAE and Dynamic VAE have significant performance decay. Higher dimensional space increases the complexity of calculating the TC and leads significant estimation errors and also increases the chance of the ID problem for DynamicVAB see in Appendix A.7. 3-VAE and DeVAE show robustness in high-dimensional latent spaces, which necessary to train a large model on large data.
3.3 EXPERIMENTAL ANALYSIS
In this section, we performed ablation studies on the benefit of the proposed Hierarchical Latent Spaces (HiS) and iisettanglementiivvrrinn Transformation (DiT). We also conducted extensive experiments to explore the effect of 3 and s on disentanglement and reconstruction performance.
Hierarchical Latent Spaces Dissntanglemenntinvaaiaa Transformation. To demonstrate the effectiveness of the introduced Hierarchy Latent Spaces (HiS) and Disentanglementtinvaraian Transformation (DiT), we performed ablation experiments on the following situations: 1) The model has one single latent space; 2) The model applies a parallel structure instead of the hierarchy that latent spaces are independent; 3) We replace DiT with Linear Transformation (Tx (2)) wzz), w is an arbitrary matrix; 4) The proposed model VeE.E.
We report the MIG and Recon. for each layer in Table 2. MS and HiS can optimize multiple objectives for these layers separately. DIT enforces all layers to share disentanglement. In this
8

---

Under review as a conference paper at ICLR 2023
Table 2: Ablation Study on Multiple Spaces (MS). Hierarchical Structure (HiS) and Disentanglementtinvaiie invariant Transformation (DiT). The reconstruction fidelity becomes better and better from the bottom layer the top layer, but the disentanglement gets worse. DiT can prevent disentanglement to degenerate.
MIG Layer1
Recon. Layerl
MS
HiS
DiT
LayerO
Layer2
LayerO
Layer2
X
X
X
0.19
47.49
X
X
0.24
0.32
0.35
22.21
40.79
62.40
X
0.24
0.29
0.30
38.82
45.48
63.78
/
0.35
0.35
0.35
43.29
75.11
175.99
Table 3: Exploration study of betas on disentangle- ment (MIG) and reconstruction (Recon.). 11, 12, 13 de- note 40, [1.10,40], [1,10] respectively. is fixed to 1.
Table 4: Comparison of scale. We report the meanisst of MIG and reconstruction for 5 trails on dSprites and Shapes3D, The sequence of betas is fixed to [1,10,40].
MIG
Recon.
MIG
Recon.
Dataset
No. betas
scale
dSprites
0.30:10.03
79.65-116.06
0.3
0.21
16.01 01.11
12
0.35
51.99---6.99
0.5
0.29-0.09
22.40+00.77
0.16+0.11
38.19:102.35
1.0
0.35=
51.99=26.99
Shapes3I)
0.54+:0.06
65.01+-25.37
0.3
0.55-0.02
+
12
0.57++0.01
43.24-+11.41
0.5
0.57
28.48+00.31
13
0.55+00.04
39.31-106.96
1.0
0.57=0.01
43.24
way, the first layer aims to optimize the ELBO, and the subsequent layers optimize disentanglement jointly by DiT which works like constraint of disentanglement. Therefore, the key of De is to connect the multiple latent spaces by DiT to form a hierarchical structure with a decremental IB.
Effect of 3. More latent layers mean more chance to explore disentanglement solutions but need more time converge. Though Wu. etc. (Wu al., 2022) proposes the Annealing Test determine the value of 3, requires labels to learn the information freezing point (IFP). Choosing a suitable for each layer is difficult without knowing the information of factors. Fortunately, DeVAE is insensitive to the choice of 3, which means we can create redundant latent layers to cover all suitable Bs. In Table 3, compared tree cases: redundant betas (01, 40,8 just betas (01,)0,40)) insufficient betas ([1,10].. Redundant betas slightly diminish the performance of disentanglement and reconstruction. is an advantage increase the parameter size and training iterations without rebooting
Effeet of Scale Increasing will add the weights of higher beta, encouraging disentanglement more than reconstruction fidelity. It is a crucial hyperparameter to balance the objectives of latent layers. Note that model equals the vanilla VAE when 0. In Table 4, we compared the effeets of choosing s and reported the mean scores MIG (Chen et al., 2018) and reconstruction. For most cases, is good choice.
3.4 LIMITATION
Since our model creates several diverse latent spaces, it is a challenge to optimize multiple objec- tives. Though there are numerous combinations for setting pressures and weighting these objectives, we only search a limited range of mance the benchmarks. Though validated that model is adequate for high- nigh-dimensional space, we did not test on real problems. is challenging to train a disentanglement model on large-scale problems, such ImageNet.
Even so. DeVAE shows compatible perfor-

---

Under review as a conference paper at ICLR 2023
4 RELATED WORK
Disentanglement Learning. Disentanglement learning aims at learning generative factors existing in the dataset, that is, disentangled representation learning. Though the definition of disentanglement is still an open topic (Kumar et al., 2018; Do & Tran, 2020; Abdi al., 2019; Duan et al., 2020), it is widely accepted that the redundancy between latent variables diminishes disentanglement. Pe- nalizing the Total Correlation (TC) (Watanabe, 1960) is important direction in disentanglement learning, and many SOTA methods are based on it (Chen et al., 2018: Kim & Mnih, 2018; Esmaeili al., 2019; Kumar al., 2018; Wei et al., 2021). PM algorithm promotes factorial codes but only works for binary codes (Schmidhuber, 1992); Though ICA (Comon, 1994) and PCA (Wold et al., 1987) ensure independence theoretically, they extract linear representations. Until recently, deep learning has made it workable. FactorVAE (Kim & Mnih, 2018) applies an adversarial training method to approximate and penalize the TC term. 3-TCVAE (Chen et al., 2018) decomposed the KL term into three parts: mutual information (MI), total correlation (TC), and ciinensional-wise KL (DWKL). However, these methods rely on the estimation of TC, which is extremely hard for high-dimensiooal spaces.
Information Bottleneck, Information bottleneck theory (Tishby et al., 1999; Shannon, 1948) plays vital role in interpreting neural networks. Some methods encourage disentanglement by increas- ing the information bottleneck while training (Jeong & Song, 2019; Burgess et al., 2018; 8; Shao et al., 2022; Dupont, 2018; Wu et al., 2022). These methods vary in the way of expanding the IB. Cascade- VAE (Jeong & Song, 2019) sequentially relieves one latent variable at one stage to increase the IB. Dyn (Shao al., 2022) designs non-linear PI controller for manipulating / to control IB steadily increasing. DEFT (Wu et al., 2022) applies a multi-stage training strategy with sepa- rated encoders extract one factor at one stage according to its information freezing point (IFP). However, the above incremental models, increasing the IB while training, suffer from the informa- tion diffusion (ID) problem (Wu et al., 2022) that the disentangled representation may diffuse the learned information into other variables. This work presents novel framework with a decremental information bottleneck to solve the ID problem.
Hierarchical Latent Variables. Normalizing Flow (Rezende & Mohamed, 2015; Kingma et al., 2016) also uses hierarchical latent layers to generate an arbitrary distribution. Unlike Normalizing Flow, each layer aims to encourage disentanglement gradually increases the complexity of the output distribution after entering a new layer. In contrast, model reduces the complexity layer by layer. LadderVAE (Sønderby al., 2016) also applies hierarchical latent variables in the encoder, but it using a symmetry structure decodes these latent variables in hierarchy. Therefore, the information among the i-th layer will increase comparing to the last layer.
reconstruction. Besides, Normalizing Flow
CONCLUSION
We propose novel framework with decremental information bottleneck for disentanglement. Hi- erarchical latent spaces with iisentanglement-invarraaa transformation are the key to overcoming the problem of losing disentanglement constraint while expanding the information bottleneck. The decremental method is compatible with high- -igh-dimensional problems and reduces the information diffusion problem. In practice, the typical disentanglement methods have to refine suitable hyperpa- rameters for a dataset without labels by trail and error. In contrast, VAE is tolerant to redundant layers such that we can set a large parameter set to fit kinds of datasets.
Broader Impact Unlike previous works that spread the conflict of the trade-off over time, our work demonstrates a novel direction to solve the trade-off by spreading the conflict in spaces. Our work provides insights on balancing disentanglement and reconstruction.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Amir H Abdi, Purang Abolmaesumi, and Sidney Fels. A preliminary study of disentanglement with insights on the inadequacy of metrics. arXiv preprint arXiv: 2019. 10
Karl Áström and Tore Hägglund. Pid control. IEEE Control Systems Magazine, 1066, 2006. 2
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine (788))77888 2013.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gihikccom//eemmmi//h/hphs 2018. 1,5
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in f3vae. In International Conference Machine Learning (ICML), 2018. 1, 3,
Jinkun Cao, Ruiqian Nai, Qing Yang, Jialei Huang, and Yang Gao. An empirical study on disentan- glement of negative-free contrastive learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https: wtwpe/eeeeeeeeeeeeeeeeeeeeeeeeep net forum? f= 15
Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David Duvenaud. Isolating sources of disentan- glement in variational autoencoders. Neural Information Processing Systems (NeurIPS), 2018.
3, 6, 9, 10
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Neural Information Processing Systems (NeurIPS), pp. 2172-2180, 2016.
Pierre Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314,
1994. 10
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wüthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. arXiv preprint arXiv:2010.14407. 2020.
Kien Do and Truyen Tran. Theory and evaluation metries for learning disentangled representations.
In International Conference on Learning Representations (ICLR), 2020. 10
Sunny Duan, Loic Matthey, Andre Saraiva, Nick Watters, Christopher Burgess, Alexander Lerch-
ner, and Irina Higgins. Unsupervised model selection for variational disentangled representation learning, In International Conference on Learning Representations (ICLR), 2020. 10
Emilien Dupont. Learning disentangled joint continuous and discrete representations. In Neural
Information Processing Systems (NeurIPS), pp. 708-718, 2018. 10
Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations (ICLR), 2018. 5
Babak Esmacili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and Jan- Willem Meent. Structured disentangled representations. In The 22nd International Conference Artificial Intelligence and Statisfics, pp. 2525-2534. PMLR, 2019. 10
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. International Conference on Learning Representations (ICLR), 2017. 3,

---

Under review as a conference paper at ICLR 2023
Yeonwoo Jeong and Hyun Oh Song. Learning discrete and continuous factors of data via alternating
disentanglement. In International Conference on Machine Learning (ICML), 2019. 10, 20
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In International Conference on
Machine Learning (ICML), 2018. 5, 7, 10, 20
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In 3rd Interna- tional Conference on Learning Representations (ICLR), 2015. 5
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014. 1,2
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29, 2016. 10
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations (ICLR), 2018. 10
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http::/hhwhh..oooo/e///////////s 2017. 1,5
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna- tional conference on machine learning, pp. 1530-1538. PMLR, 2015. 10
Jürgen Schmidhuber. Learning factorial codes by predictability minimization. Neural Computation, 4(6))863-87) 887, 1992. ISSN 08997667. 10
Claude Elwood Shannon. A mathematical theory of communication. Bell system technical journal, 27(3):379 423, 1948.
Huajie Shao, Yifei Yang, Haohong Lin, Longzhong Lin, Yizhuo Chen, Qinmin Yang, and Han Zhao Rethinking controllable variational autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19250-19259, 2022. 2, 3, 6, 10
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. NIPS, 2016. 10
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proceedings of the 37-th Annual Allerton Conference Communication, Control and Comput-
ing. 1999. 1, 10
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. arXiv preprint 2xi::20006008886 2020.
Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of re- search and development, 4:66-82, 1960. 10
Yuxiang Wei, Yupeng Shi, Xiao Liu, Zhilong Ji, Yuan Gao, Zhonggin Wu. and Wangmeng Zuo. Orthogonal jacobian regularization for unsupervised disentanglement in image generation. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), pp. 6721 6730, October 2021. 10
Svante Wold. Kim Esbensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987. 10
Jiantao Wu, Lin Wang, Bo Yang, Fanqi Li, Chunxiuzi Liu, and Jin Zhou. DEFT: dis- tilling entangled factors by preventing information diffusion. Mach. Learn., 111(6))2275- 2295, 2022. doi: 10. 10.100776119994-22--0644444 012994-022-06134-2222 2, 6,
URL https:/ :ttps://dccioooo http:///ooo.rg//.10000000
12

---

Under review as a conference paper at ICLR 2023
Table The architecture details. "FC.." denotes fully connected layer, "conv." denotes convolutional layer, "deconv" denotes transposed convolution layer. e is the dimension of color channel.
Encoder
Decoder
x 4 conv. 32 stride 2
FC.256
x 4 conv. 32 stride 2
FC. 4 x x 64
4 conv. 64 stride 2
4 deconv. 64 stride 2
4 conv. 64 stride
deconv. 32 stride
FC. 256
4 deconv. 32 stride
FC. 10
x 4 deconv. e stride
A APPENDIX
A.1 ARCHITECTURE
The details of architectures are listed in Table 5.
A.2 DISENTANGLEMENT-NNAARRR
REPRESENTATIONS
In this section, we prove the proposed disentagglement-invvaian transformation. Consider that we have new representation by multiplying diagonal matrix: z' =wz, w. We can calculate the Covariance between any two latent variables:
Cov(w; w,zz) = [[[w; - E[w; (w E[ 2,
w; wy (E[zy] B[z]B[zz)]
(10)
w Cov Zz),
where the subscript denotes the index of latent variables. Note that use a different notion in this section to simplify the formula.
Then we can get the correlation coefficient by
(,1 = Cov w,zz) /Varlw, Var [ar[w,z)]
(II)
Therefore, the correlation matrix will not change by multiplying a diagonal matrix w, w * We could create disentanglement innaannggemmntnnnviiiann
invariant representation by multiplying a diagonal matrix.
A.3 ESTIMATION OF ((zy:c;)
Given an inference network y(2(z), we use the Markov chain Monte Carlo (MCMC) method to get samples from q=x) by the formula (=)) (<=(x)p(z). We use 10, 000 points estimate q22). Then, we discretize these latent variables by a histogram with 20 bins. After discretizing one latent variable, we call discrete mutual information estimation algorithm calculate 2y; c:) by 2D histogram.
A.4 LATENT TRAVERSALS
We compare VeE to others with latent traversals on Shapes3D and dSprites. Each column denotes the generated images by traversing one latent variable from -2 to 2. We also interpret the extracted factor at the bottom. From Figure and Figure 7, we can see that DeVAE has lower entanglement level. Note that only DeVAE disentangles object size isolated Shapes3ID.
13

---

Under review as a conference paper at ICLR 2023
DynamicVAE
8-TCVAE
INg Vil
B-VAE
Faxe
Eyen
calor
KARKLEI 594
03 cauo
Figure 6: Latent traversal on Shapes3D. "back." denotes background color, "floor" denotes floor color, "obj." denotes object, and "wall" denotes wall color.
A.5 RANDOM SAMPLING
We show the visualization of random sampling from the best models with the highest MIG trained on dSprites and Shapes3D in Figure 9 10.
A.6 DECREMENTAL DIAGRAM
Figure 11 shows how the mutual information between factors and latent variables decreases layers on dSprites. One can see the mutual information decreases along the layers, and information of shape, scale, and orientation is totally disappeared in layer2. The last layer more likely to
14

---

Under review as a conference paper at ICLR 2023
DynamicVAE
8-TCVAE
YOd
FRNX
PAVY
PINT
OPEEEE
PaX
B-VAE
poat
Figure 7: Latent traversal on dSprites.
disentangle them, and that property will be preserved and passed the first layer for constraint of disentangiement.
A.7 DIMENNIINNNSI LATENT SAPCE
DeVAE has significant advantages for handling high- tigh-dimensional latent spaces. Though Dynamic- VAE outperforms low low-dimensional latent spaces, there is a gap in the high-dimensional latent spaces. We trained DynamicVAF VAE and DeVAE with 1024-dimensional latent spaces on dSprites to investi- gate what causes the difference. (Cao et al., 2022) found that existing disentanglement metrics fail to make meaningful measurements for high- dimensional representation models, therefore we apply the proposed metric by them in this experiment, Active variables denote the latent variables containing
15

---

Under review as a conference paper at ICLR 2023
B-TCVAE
LayerO
Layer1
Layer2
DeVAE
Figure 8: Generated images of 8-TCVAE and DeVAE from sampling random noise on dSprites.
16

---

Under review as a conference paper at ICLR 2023
B-TCVAE
Lavero
Layer1
Layerz
DeVAE
Figure 9: Generated images of 3-TCVAE and DeVAE from sampling random noise on Shapes3
17

---

Under review as a conference paper at ICLR 2023
DynamicVAE
B-VAE
DynamicVAE
B-VAE
Figure 10: Generated images of DynamicVAE. and 3-VAE from sampling random noise on dSprites
and Shapes3D.
18

---

Under review as a conference paper at ICLR 2023
1
2
4
7
9
0.21
0.01
0
0.51
0.17
0.07
0.26
0.22
0.2
0.01
-0
0.26
0.23
0
0.03
0.01
-0
1.3
0.91
0.41
0.13
0.07
0.03
0
-0
0.13
0.07
0.07
0.01
0
0.05
0.82
0.54
0.08
0.01
0.52
0.01
0
0
0.02
0
0
-0
2.8
2.5
1.6
0
2.8
2.5
1.6
0.01
2
2
Figure 11: The mutual information between factors and latent variables over layers on dSprites. We select five informative variables, and the title denotes the index of the latent variable. The rows denote the factors, shape, scale, orientation, posX, and pos respectively. The column denotes the layer where the latent variable (title) is.
800
DeVAE DynamicVAE
0.8
600
MED
DeVAE
400
DynamicVAE
0.2
50000
100000
150000
200000
250000
50000
100000
150000
200000
250000
(a) The number of active variables (std >> 0.1)
(b) The MED score
Figure 12: Comparison active variables and MED scores on 1024-dimensional models on dSprites. DynamicVAF. VAE is unstable while expanding the information bottleneck.
19

---

Under review as a conference paper at ICLR 2023
Table 6: Comparison reconstruction error (Recon.). MIG score, and ELBO for six disentangle- ment methods.
dataset
model
Recon.
MIG
ELBO
FactorVAR
21.55
0.34+10.0
46.05++2.24
Cascade VAE
12.04+1.23
0.20++0.07
-32.14 + 1.29
Dynamic
19.81=1.19
0.35+0001
-37.83=1.17
dSprites
beta-TCVAE
73.04=3.41
0.29-1010
-82.29:3.771
beta- VAE
48.75=2.84
0.17=0.05
-61.17+3.13
DeVAE
36.02+220.02
0.32+0.11
-51=22.26
Factor VAE
18.48+2.28
0.38
1,87
Cascade VAE
14.84=1.98
0.4610.11
-32.5442.10
Dynamic
29.70=4.115
0.54
47.68
Shapes3I)
beta- beta-TCVAE
44.53+5.69
0.49+0.11
-60.01+6.29
beta- VAE
34.95+2.33
0.42+10.18
29.09
DeVAE
46.80:11.97
0.52+0.10
-74.734311.6
information, and those containing no information will collapse into one point, the active variables will have large variances. From Figure 12, Dynamic VAE has more active variables and performs worse than De VAE. Dynamic VAE expanding the IB smoothly could have good performance low-dimensional spaces, but the increment of dimensions raises the chance leaking information others. As result, the number of active variables increases quickly when expanding the IB, iteration 20000 to 100000.
A.8 COMPARISON WITH FACTORVAE AND CASCADEVAE
FactorVAE (Kim Mnih, 2018) and Cascade VAE (Jeong Song, 2019) are two relevant methods for disentanglement, We compare six disentanglement methods on dSprites in Table 6. Similar to 3- TCVAE, FactorVAF can not consistently outperform one hyperparameter on two datasets. Though Cascade VAE has good reconstruction fidelity, it cannot disentangle all factors properly.
20


Reviews:
[{"review_id": "VE2JM7Pfoig", "reviewer": "Reviewer_8f7N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method called DeVAE for learning disentangled representations based on beta-VAE. Specifically, the proposed method builds a hierarchical latent space with disentanglement-invariant transformation between them and decreases information bottleneck layer-by-layer to balance disentanglement and reconstruction fidelity. Experimental results show that DeVAE performs comparably as previous works. DeVAE seems to perform better than other methods when the latent dimension is high.", "summary_of_the_review": "The paper focuses on the important problem of the balance between disentanglement and reconstruction fidelity in disentangled representation learning research. The proposed method is technically sound. However, there are significant concerns with the experiments in the paper and the paper clarity is low. Hence, I would recommend the paper for rejection in its current form.", "strength_and_weaknesses": "Strengths\n- The paper focuses on the balance between disentanglement and reconstruction fidelity, an important problem in the field of disentangled representation learning.\n- The proposed method is technically sound.\n- The proposed method performs better than beta-VAE and beta-TCVAE on dSprites and shapes3D datasets.\n- Ablation studies show the importance of each proposed component and selection of hyperparameters.\n\n\nWeaknesses:\n\n- The paper has low readability. A lot of the issues are certainly fixable but in its current form, it is confusing enough to distract from evaluating the technical contributions of the paper. Certain examples are:\n    - “DeVAE surpasses 2% for β-TCVAE and 9% for β-VAE.” In what terms exactly?\n    - Many instances in the introduction talking about spreading the conflict of disentanglement and reconstruction over time and space is not easy to follow and understand.\n    - “However, in this work, we get rid of calculating TC by leveraging the narrow information bottleneck (Tishby et al., 1999; Burgess et al., 2018) to find efficient codes for representing the data, which promotes disentanglement.” This is pretty confusing right where it is in the introduction and only becomes somewhat clear after reading the method section.\n- The authors claim that DynamicVAE suffers from Information Diffusion problems. If that is the case, wouldn’t that result in low disentanglement scores or at least high variance across different seeds for DynamicVAE? But, that’s not the case in Figure 2.\n- The experiment for high-dimensional latent space is weak. 1024 dimensional latent space for dSprites seems unrealistic, it ideally should be for a dataset that requires high-dimensional latent space. And, there are no quantitative numbers. Why is DeVAE worse for low dimensions? A more high-level question would here is — what exactly in DeVAE makes it more compatible for handling high-dimensional latent space?\n- A lot of design decisions are unexplained. How are the hyperparameters for other methods chosen? Are the beta values chosen for DeVAE selected based on test performance or on a validation set? \n- What are the layer embeddings? They seem to be an important component of the method but are never explained.\n- How much is the computational overhead because of the hierarchical latent space and how does it compare to other methods that are compared within the paper?\n- Why is DeVAE not compared with FactorVAE and Cascade-VAE, they both seem highly relevant as well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GZl-A8x8og", "reviewer": "Reviewer_Z9wf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel hierarchical latent space structure and regularisation strategy (beta-pressure) to improve the balance of representation and information retention, or reconstruction quality, in VAEs. The latent space is built to have a Markov chain-like structure, where, in the encoder, the first latent space is conditioned on inputs q(z_0|x), just like standard VAEs, and all subsequent encodings are conditioned on the previous one, i.e., q(z_i+1|q_z_i). The decoder reconstructs the input from each latent space in parallel. Although I am not entirely clear on the whole mechanism (see below).", "summary_of_the_review": "The novelty of the paper is sufficient in my opinion, but the clarity and quality do not meet the bar. The technical section is not sufficiently clear and the experiments are difficult to read and not sufficiently conclusive.", "strength_and_weaknesses": "Strengths: \n\nThe idea to have parallel latent spaces with different betas and using them all to obtain \"the best of both worlds\" is interesting and, to the best of my knowledge, sufficiently novel in this form.\n\n\nWeaknesses:\n\nI have several concerns regarding the technical description and experiments.\n\nTechnical Section:\n- The description lacks clarity in some places. For example, it is not clear what is the role of the vectors(?) v_i shown in figure 1, as they are never mentioned in the text and do not appear in the ELBO of equation 2 or the transformation tau_i of equations 3-6. Therefore, I could not understand how the decoder works; does it individually decode each z_i with the same weights and add up the reconstruction costs?\n- According to theorem 1, the disentanglement of z_0 is preserved in all subsequent latent spaces. This is inconsistent with the aim of the method, where, as phrased in the paper \"the sequential layers aim to disentangle factors of variation by setting narrow bottlenecks\". The representations' disentanglement cannot be changed by scaling. I may have got this wrong somewhere, but if this is the case, this is a critical flaw.\n\nExperiments:\nThe experimental section is rich, but in most cases hard to interpret. My main concerns are the following:\n-  The results of figure 2 have several issues (at least in their presentation): 1) the plots show 3 disentanglement metrics and 1 reconstruction metrics. As per the objective of the method, what counts is their tradeoff, so a much better way to show performance would be to do something more similar to figure 5 and plot each of the 3 disentanglement metrics vs. reconstruction error. 2) It is not clear where better is higher and where better is lower. Clearly lower rec. error is better, but do we want high or low MIG for example? From the descriptions below it is not clear. 3) In the text it is stated that the DeVAE is the best, but in this plots it is often somewhere in the middle, so it is not clear how the reader is meant to interpret these plots as proving superior performance of the proposed method.\n- Figure 3 is not interpretable at all. I think the columns (x-axis) are the 3 random samplings, but what is the 2-6 numbers above? Where is the -2 to +2 positions? what is the y-axis? There is no comparison with any other method, so the \"disentangled perfectly\" mentioned in the text is not very meaningful.\n- Figure 4 has a similar problem to figure 2; The information diffusion shows how well the model is disentangling, but we need also to see the reconstruction. As stated also in this paper, it is their relationship that counts.\n- Figure 5 shows, in my opinion, the relevant information; reconstruction vs. a measure of disentanglement. However, these results raise some doubts. Assuming MIG is better when higher, for high dimensions the proposed method is the best, but for low dimensions, it is outperformed significantly in both MIG and reconstruction error by the DynamicVAE. Is this right?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-immOahdh4", "reviewer": "Reviewer_nZJo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a hierarchy of latent representations weighted by a series of monotonically increasing hyper-parameters, which compose an information bottleneck. The authors empirically demonstrate their idea on dSprites and Shapes3D to show the proposed method is able to learn disentangled representations while preserving reconstruction fidelity.", "summary_of_the_review": "The disentanglement of representation has been an important topic in the study of representation learning and has been studied for a long time along the development of information bottleneck and variational autoencoders. The proposed method has its contributions in improving the disentanglement of the latent variables while preserving the reconstruction quality. Both the quality and clarity of the paper still need improvement. I am inclined to consider this paper below the acceptance threshold given the current version.  ", "strength_and_weaknesses": "**Strength**\n\nThe proposed method is not complex and easy to understand. The experiments are comprehensive and sufficient ablation studies are provided to support the claims.\n\n**Weakness**\n\nThe method part is not properly stated; some mathematical expressions might be wrong (at least they cannot correspond to either Fig 1. or Alg. 1). I am confused after reading it:\n\n- According to Fig 1. and Alg. 1, the latent variables depend on previous ones, i.e., $z_1$ is dependent to $z_0$, $z_2$ depends on $z_1$. However, Equation 2 seems to show $z_i$ is only dependent to $x$ or $z_0$, and $z_1, z_2, ..., z_K$ are independent like K VAEs. Equation 3, also seems to have $z_i$ marginalized out, while the code in Alg. 1 does not have such an operation. The authors need to clarify this part.\n\n- The notation v is in Fig. 1 is not used in the mathematical definition.\n\n- What is j in Equation 5-6? \n\nThe paper lacks theoretical analysis. The choice of $\\beta$ does not have clear guidance. The authors do not show whether the modified ELBO is still valid and whether it is a tight bound.\n\nAccording to the results in Table 2, by adding HiS and DiT, the reconstruction fidelity seems to get worse, which does not support the claim of the paper: \"optimize reconstruction and keep disentangled representation\". \n\nIn the case when the latent space is stacked with redundant variables, a non-trivial study could be whether there are some variables that are more important while some variables can be distilled. \n\nThe granularity of the disentanglement of the proposed method is correlated to the parameter size and training efficiency. In other words, if we would like to refine the granularity of the information bottleneck (have more choices of $\\beta_i$ like done in Table 3), the parameter size and required training iterations also increase. \n\nFrom the perspective of practical use, the improvement is a little bit marginal, and the experiments are conducted on small-scale data. It is still unclear whether this method could be useful for larger data and models. \n\n**Minors**\n\nEquation 4: some 'i's are bold while some are not.\nEquation 5: same issue.\n\nBelow Equation 7: $p\\left(\\boldsymbol{z}_i\\right)=(1-s) s^i /\\left(1-s^{i+1}\\right), s \\neq 1 ; \\frac{1}{K}, s=1$ is not a rigorous math expression. \n\nSection 4. Paragraph1-Line7: one citation is missing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject