$\alpha$TC-VAE: On the relationship between Disentanglement and Diversity
Authors: Cristian Meo
Louis Mahon
Anirudh Goyal
Justin Dauwels
Venue: ICLR 2024 poster (2024)
Keywords: Disentanglement
Diversity
Generative Modeling
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Understanding and developing optimal representations has long been foundational in machine learning (ML). While disentangled representations have shown promise in generative modeling and representation learning, their downstream usefulness remains debated. Recent studies re-defined disentanglement through a formal connection to symmetries, emphasizing the ability to reduce latent domains (i.e., ML problem spaces) and consequently enhance data efficiency and generative capabilities. However, from an information theory viewpoint, assigning a complex attribute (i.e., features) to a specific latent variable may be infeasible, limiting the applicability of disentangled representations to simple datasets. In this work, we introduce $\alpha$-TCVAE, a variational autoencoder optimized using a novel total correlation (TC) lower bound that maximizes disentanglement and latent variables informativeness. The proposed TC bound is grounded in information theory constructs, generalizes the $\beta$-VAE lower bound, and can be reduced to a convex combination of the known variational information bottleneck (VIB) and conditional entropy bottleneck (CEB) terms. Moreover, we present quantitative analyses and correlation studies that support the idea that smaller latent domains (i.e., disentangled representations) lead to better generative capabilities and diversity. Additionally, we perform downstream task experiments from both representation and RL domains to assess our questions from a broader ML perspective. Our results demonstrate that $\alpha$-TCVAE consistently learns more disentangled representations than baselines and generates more diverse observations without sacrificing visual fidelity. Notably, $\alpha$-TCVAE exhibits marked improvements on MPI3D-Real, the most realistic disentangled dataset in our study,  confirming its ability to represent complex datasets when maximizing the informativeness of individual variables. Finally, testing the proposed model off-the-shelf on a state-of-the-art model-based RL agent, Director, significantly shows $\alpha$-TCVAE downstream usefulness on the loconav Ant Maze task. Implementation available at https://github.com/Cmeo97/Alpha-TCVAE

Published as a conference paper at ICLR 2024
Q - TCCAE: ON THE RELATIONSHIP BETWEEN DISEN- TANGLEMENT AND DIVERSITY
Cristian Meo
Louis Mahon
Anirudh Goyal Google DeepMind, UK.
Justin Dauwels TUDelft, NI..
TUDelft, NI..
University of Edinburgh, UK.
ABSTRACT
Understanding and developing optimal representations has long been founda- tional in machine learning (ML). While disentangled representations have shown promise in generative modeling and representation learning, their downstream usefulness remains debated. Recent studies re-defined disentanglement through formal connection symmetries, emphasizing the ability to reduce latent do- mains (i.e., MI problem spaces) and consequently enhance data efficiency and generative capabilities. However, from an information theory viewpoint, assign- ing a complex attribute (i.c., features) to a specific latent variable may be infea- sible, limiting the applicability of disentangled representations to simple datasets. In this work, we introduce a-TCVAE, a variational autoencoder optimized using novel total correlation (TC) lower bound that maximizes disentanglement and latent variables informativeness. The proposed TC bound is grounded in informa- tion theory constructs, generalizes the 3-VAE lower bound, and can be reduced to convex combination of the known variational information bottleneck (VIB) and conditional entropy bottleneck (CEB) terms. Moreover, we present quan- titative analyses and correlation studies that support the idea that smaller latent domains (i.e., disentangled representations) lead to better generative capabilities and diversity. Additionally, we perform downstream task experiments from both representation and RL domains to assess our questions from a broader ML per- spective. Our results demonstrate that a-TCVAE consistently learns more dis- entangled representations than baselines and generates more diverse observations without sacrificing visual fidelity. Notably, a-TCVAE exhibits marked improve- ments on MPI3D-Real, the most realistic disentangled dataset in our study, con- firming its ability to represent complex datasets when maximizing the informative- ness of individual variables. Finally, testing the proposed model off-the-shelf on state-of-the-at model-based RL agent, Director, significantly shows a-TCVAE downstream usefulness on the loconav Ant Maze task. Implementation available at ltttss:/ com cttp://m/T////AApaa-TTAAAOO
INTRODUCTION
The efficacy of machine learning (MI. algorithms is intrinsically tied to the quality of data repre- sentation (Bengio et al. 20013.. Such representations are useful not only for standard downstream tasks such as supervised learning (Alemi et al.) 2017)) and reinforcement learning (RL) (Li) 2017), but also for tasks such as transfer learning (Zhungg et al. 2020) and zero-shot learning (Sun et al. 2021). Unsupervised representation learning aims identify semantically meaningful represen- tations data without supervision, by capturing the generative factors of variations that describe the structure of the data (Radford et al. 2016 Locatello al. 2019b). According [Beggio et al. (203)), disentanglement learning holds the key to understanding the world from observations, gen- eralizing knowledge across different tasks and domains while learning and generating compositional representations (Higgins et al. 22016; Kim & Mnih 2018).
The goal of disentanglement learning is identify of independent generative factors z that give rise to the observations æ via p(x]2). However, from an information
Problem Formulation.
Work done while doing research internship Mila, Quebec AI Institute.

---

Published as a conference paper at ICLR 2024
theory perspective, the amount of information retained by every latent variable may be insufficient
to represent realistic generative factors (Kirsch et al. 2021; Do Tran 2020), limiting the applica- bility of disentangled representations to simple problems. W What more, Friedman & Dieng (2022) recently introduced the Vendi score, a new metric for gauging generative diversity, showing that en- tangled generative models, such the Very Deep VAE (Child 2020), consistently produce samples with less diversity compared to ground truth. This is indicative of their limited representational and generative prowess. In contrast, Higgins al. (2019 2022) re-defined disentangled representations through the lens of symmetries, linking disentanglement to computational problem spaces (e.g.. dis- entangled representations inherently reduce the problem space (Arora & Barak (009)). suggesting that disentangled models should be able to explore and traverse the latent space more efficiently, leading to enhanced generative diversity.
Previous Work.
Most existing disentangled models optimize lower bounds that only impose an information bottleneck on the latent variables, and while this can result in factorized representa- tions (Higgins et al., al., (016). it does not directly optimize latent variable informativeness (Do & Tran 2020). As a result, while several approaches have been proposed to learn disentangled represen- tations by optimizing different bounds (Chen al.) 2018: Kim & Mnih 2018), imposing sparsity priors (Mathieu et al. 2019), or isolating source of variance (Rolinck et al. 2201)), none of the pro- posed models successfully learned disentangled representations of realistic datasets. Moreover, to the best of our knowledge, no systematic and quantitative analyses have been proposed to assess to what extent disentanglement and generative diversity (Friedman & Dieng 2022) are correlated.
Proposed method. In this work, we propose a- -CTC,,,, a VAE optimized using novel convex lower bound of the joint total correlation (TC) between the learned latent representation and the input data. The proposed bound, through convex combination of the variational information bot- tleneck (VIB) Alemi et al. (2017) and the conditional entropy bottleneck (CEB) Fischer & Alemi 22200) maximizes the average latent variable informativeness, improving both representational and generative capabilities. The effectiveness of a-TCVAE is especially prominent in the MPI3D-Real Dataset (Gondal et al.] 2019), the most realistic dataset in our study that is compositionally built upon distinct generative factors. Figure Taarreeilltttttn iillusttraees a comparison of the latent traversals between a-TCVAE, Factor- VAE and 3-VAE, showing that a-TCVAE leads to the best visual fidelity and generative diversity (i.e.. higher Vendi Score). Interestingly. the proposed TC bound is grounded in information theory constructs, generalizes the B-VAE (Higgins et al. 2016) lower bound, and can be reduced to convex combination of the known variational information bottleneck (VIB) (Alemi ct al. 2017) and conditional entropy bottleneck (CEB) (Fischer & Alemi 2020) terms.
Experimental Evaluation In order to determine the effectiveness of a- a-TCVAE and the down- stream usefulness of the learned representations, we measure the diversity and quality of generated images and disentanglement of its latent representations. Then, we perform correlation study between the considered downstream scores across all models, analyzing how generative diversity and disentanglement are related across different datasets. This analysis substantiates our claim that disentangiement leads to improved diversity. Finally, we conduct experiments to assess the down- stream usefulness of the proposed method from broader ML perspective. Notably, the proposed method consistently outperforms the related baselines, showing a significant improvement in the RL Ant Maze task when applied off-the-shelf in Director, a hierarchical model-based RL. agent (Hafner et al. 2022).
2 RELATED WORK
Generative Modelling and Disentanglement Recently Locatello et al. (2019b) demonstrated that unsupervised disentangled representation learning theoretically impossible, nonetheless disentan- gled VAES, acting as both representational and generative models, Kiings & Welling (2013): Hig-
gins et al. (0066): Chen et al. (2018) Kim & Mnih| (2018) achieve practical results by leveraging implicit biases within the data and learning dynamics Burgess & Kim (2018); Higgins et al. (2019).
Mathieu et al. (2019). On the generation side, they have been widely used to generate data such as images (Chen et al. 2019), text (Shi al. 2019), speech (Sun et al. 2020 Li et al. (203)) and music (Wang et al., 2020). Various extensions to the base VAE model have been presented to improve
generation quality in terms of visual fidelity (Peng et al. 2021
Vahdat & Kautz, 2020) Razavi al.
2

---

Published as a conference paper at ICLR 2024
a-TCVAE
Factor-VAE
B-VAE
Vendi Score 12.04
Vendi Score 10.69
Vendi Score 9.93
Figure 1: Ground truth (first row), reconstructions (second row) and latent traversals comparison of a-TCVAE, Factor-VARE and 3-VAE on the MPI3D-Res Dataset. Notably, a-TCVAE showcases superior visual fidelity and generative diversity, as indicated by a higher Vendi Score.
2019). On the representational side, aiming for explainable and factorized representations, Hig- gins et al. (2016) proposed B- VAE, which inspired number of following disentangled VAE-based models, such Factor- VAE (Kim & Mnih 2018), 3-TCVAE Chen et al. (2018), and A-Annealed VAE (Burgess et al. 2018). Both 3 VAE and Factor- VAF aim to learn disentangled representations by imposing bottleneck the information flowing through the latent space. While 3-VAE does this by introducing a 3 hyperparameter that increases the strength of the information bottleneck, Factor-VAE introduces TC regularization term. Chen et al. (2018) proposed 3-TCVAE, which minimizes the total correlation of the latent variables using Monte-Carlo and importance sampling. Roth al. 2023) proposed the Hausdorff Factorized Support (HFS) criterion, a relaxed disentangle- ment criterronn that encourages only pairwise factorized support, rather than factorial distribution, by minimizing a Hausdorff distance. This allows for arbitrary distributions the factors over their support, including correlations between them. Our model, namely a- a-TCVAE is optimized by TC: lower bound as well, however we do not make use of any trick or expensive sampling strategy. In contrast, we derive TC lower bound that does not require any extra network sampling strategy and is theoretically grounded in the Deep Information Bottleneck framework Alemi et al. (2017).
Disentanglement and Deep Information Bottleneck In the last few years, link between the la- tent space capacity and disentanglement of the learned variables (Bengio et al. 2013 Shwartz-Ziv & Tishby 2017 Goyal et al. 2021) has been identified, showing that decreasing the capacity of a net- work induces disentang lement eement on the learned representations. This relationship has been explained by the information bottleneck (IB) principle, introduced by Tishby et al.) (2001) as a regularization method obtain minimal sufficient encoding by constraining the amount of information captured by the latent variables from the observed variable. Variational IB (VIB) (Alemi et al., 2017) has extended the framework by applying neural networks, which results in a simple yet effec- tive method to learn representations that generalize well and are robust against adversarial attacks. Furthermore, (Alemi al. 2017 Kirsch al. 202T) outlined the relationship between VIB, VAE Kingma & Welling 2013) and 8- VAE (Higgins et al.) 2016), providing an information theoretical interpretation of the Kullback-Leibe (KL) divergence term used in these models as a regularizer. Despite the advantages introduced by the VIB framework, imposing independence between every latent variable may be too strong an assumption (Roth et al. (202)). For this reason, Fischer Alemi 2020) introduced the conditional entropy bottleneck (CEB), which assumes conditional indepen- dence between the learned latent variables, providing the ability to learn more expressive and robust representations (Kirsch et al. 2021). Recently, generalization of the mutual Information (MI),
3

---

Published as a conference paper at ICLR 2024
namely total correlation (TC), has been used to learn disentangled representations well (Kim & Mnih 2018). Following Hwang et al. (2021). who propose a similar TC bound for a multi-view setting, derive novel TC lower bound for the unsupervised representational learning setting. As
a result, the proposed bound is able to learn expressive and disentangled representations.
Disentanglement and Diversity The ideal generative model learns distribution that well ex- plains the observed data, which can then be used to draw a diverse set of samples. Diversity is thus an important desirable property of generative models (Friedman & Dieng 2022). We desire the ability to produce samples that are different from each other and from the samples we already have at train time, while still coming from the same underlying distribution. The benefits of diversity have been advocated in a number of different contexts, such as image synthesis (Mao et al. 2019).
molccular design (Lin et al. 2021; Wu et al. 2021). natural language text (McCoy et al. 2023). and drug discovery (Kim & Mnih 2018). Motivated by the benefits of generative diversity, sev- eral VAE-based models have aimed to show increased diversity in their generated samples (Razavi let al. 2019). Some works have also noted improvements in diversity due to disentanglement. Lee et al. 2018) adversarially disentangle style from content and show enhanced diversity of image-to- image translations. Kazemi et al. (2019) also perform style-content disentanglement, this time in the context of text generation, and again observe an increase in diversity. [Li et al. (2020) shows that disentangling posc, shape, and texture leads to greater diversity in generated images. Collec- tively, these studies emphasize that diversity is often a valuable indicator of effectiveness in various applications, and suggest that diversity and disentanglement are intertwined aspects of generative models. Yet, to the best of our knowledge, no quantitative analyses that support this claim have been presented. In this work, we present correlation study, showing how downstream metrics of disen- tanglement (e.g., DCI (Eastwood & Williams| (2018) and diversity (e.g., Vendi Score (Friedman & Dieng 2022 are correlated across several models and datasets.
3 a- TCTE FRAMEWORK DERIVATION
Motivation. In contrast to most existing methods, which only impose an information bottleneck to learn disentangled representations, we seek to maximize the informativeness of individual latent variables as well. The total joint correlation (TC) can be explicitly expressed in terms of mutual information between the observed data and the latent generative factors, as shown in equation allowing us to link disentanglement to latent variables informativeness. As result, leveraging the TC formulation, we can derive a lower bound that not only promotes disentanglement but also maximizes the information retained by individual latent variables.
Derivation. In this section, we formally derive the novel TC bound. Let (X, V} be the ground-truth set that consists of images x E R N and a set of conditionally independent ground- RA77 where log p())=)) = Σn p(y) The goal is to develop
truth data generative factors an unsupervised deep generative model that can learn the joint distribution of the data while uncovering set of generative latent factors E R.K K 2 M, such that z can fully describe the data structure of æ and generate data samples that follow the underlying ground-truth generative factors v. Since directly optimizing the joint TC is intractable, we are going to maximize a lower bound of the joint total correlation TC(z, æ) between the learned latent representations z and the input data æ. following the approach proposed by |Hwang et al.] (2021). The total correlation is defined as the KL divergence between the joint distribution and the factored marginals. In our case:
(x) e DKL / (=(
K
II go k=1
where the joint distribution is (z)=)= = go ()(2)=))DD(()dd PD the data distribution, qo (zx) = qo yd2z and zyt indicates that the k-th component of not considered. Since we aim to find the encoder qo (=(=))) that disentangles the learned representations z, we can formulate the following objective:
TCo(z, A TCp(=) TCC((z)))
where the conditional TC(z|a:) can be expressed as:
K
(2(() و Eno(*)
DKL xx)2 II
go (z)
(3)
4

---

Published as a conference paper at ICLR 2024
which is the expected KL divergence of the joint conditional from the factored conditionals. Intu- itively, we can see that minimizing TCa(2 æ)) TCp(z, x) will be maximized, enhancing the disen- tanglement of the learned representation Moreover, decomposing equation we can express the TC in terms of MI (Gao et al. 2019):
K TCo(z, = Σ Tolzk, x) I0(2, x)),
(4)
1=
where To(z, x) is the mutual information between and and is known the VIB term Alemi et al. 2017). Additionally, we can also express it in terms of Conditional MI:
K [K - 10)o Io( mx
1
TCo(=, x) K
(5)
where (zytk, x) Zi.) is known as the CEB term (Fischer Alemi 2020). Equation[4annd equation illustrate the link of the designed objective to both VIB and CEB Trameworks. A complete derivation of them can be found in Appendices [ and |A.2] respectively. While the VIB term promotes compression of the latent representation, the CEB term promotes balance between the information contained in each latent dimension. Since we want to promote both disentanglement and individual variable informativeness of the learned latent representation we propose lower bound that convexly combines the found VIB and CEB terms. We define the bound as follows:
Ka DKr((1lzzza) (r
(1
TC(z, x)) 2 (x [[og]((((2)]
VIB
CEB
where a a hyperparameter that trades off VIB and CEB terms. Following Hwang et al.
2021
we define (2|xx) N(pp T) and r(z) = N(0, I), respectively, where op -
-k=1 of
and Hp o Σ jt while μ4 and 0 are the mean and standard deviation used to compute of
z; using the reparametrization trick as in Kingma & Welling (2013). A full derivation of the bound defined in equation [ can be found in Appendix A
Practical Implications. Disentangled models with M generative factors and K latent dimensions usually have (K - MM) noisy latent dimensions Do & Tran (2020), but our CEB term induces an inductive bias on the information flowing through every individual latent variable, pushing otherwise noisy dimensions to be informative. The derived TC lower bound generalizes the structure of the widely used 8- B-V (Higgins al. (016) bound. Indeed, for a 0, the TC bound reduces to 3-VAE one. A comparison of a-TCVAE, 3-VAE, 3-TCVAE, HFS and Factor-VAr lower bounds can be found in Tab. A.3
4 EXPERIMENTS
In this section, we design empirical experiments to understand the performance of a- C-TCVAE and its potential limitations by exploring the following questions: (1) Does maximising the informativeness
of latent variables consistently lead to an increase in representational power and generative diversity? (2) Do disentangled representations inherently present higher diversity than entangled ones? (3) How are they correlated with other downstream metrics (i.e., FID (Heusel et al. 2017) and unfairness (Locatello al.) 2019a) (1)))) ? (4) To what extent does maximising the latent variables' informativeness in disentangled representations improve their downstream usefulness?
Experimental Setup. In order to assess the performance of both proposed and baseline models, we validate the considered models on the following datasets. Teapots (Moreno et al.) (016) con- tains 200, 000 images teapots with features: azimuth and elevation, and object colour. 3DShapes (Burgess & Kim 2018) contains 480, 000 images, with features: object shape and colour, floor colour, wall colour, and horizontal orientation. MPI3D-Real (Gondal et al.) 2019) contains 103, 680 images of objects at the end of robot arm, with features: object colour, size, shape, camera height, azimuth, and robot arm altitude. Cars3D (Reed et al. 2015) contains 16, 185 images with features: car-type, elevation, and azimuth. CelebA (Liu et al. 2015) contains 200, 000 images of faces
5

---

Published as a conference paper at ICLR 2024
under broad range of poses, facial expressions, and lighting conditions, totalling 40 different fac- tors. All datasets under consideration consist RGB images with dimensions 64 x 64. Among them, CclebA stands out the most realistic and complex dataset. On the other hand, MPI3D-Real considered the most realistic among factorized datasets, which we define as those compositionally generated using independent factors. To assess the generated images, we use the FID score (Heusel iet al., (017)) to measure the distance between the distributions of generated and real images, and the Vendi score (Friedman Dieng (2022) measure the diversity of sampled images. Both Vendi and FID use the inception Network (Szegedy et al.) (2017) to extract image features and compute the related similarity metrics. Since DCI (Eastwood & Williams 2018) scores can produce unreliable
results in certain cases, (Mahon et al. 2023: Cao et al. [022]] Do & Tran 2020). we measure dis-
entanglement using also single neuron classification SNC (Mahon al. 2023). Further details on used datasets and metrics are given in Appendix
Baseline Methods. We compare a- a-TCVAE to four other VAE models: 3-VAE (Higgins et al.) 2016), 3-TCVAE (Chen et al. 2018), 3-VAE+HFS (Roth et al. 2023) and FactorVAE (Kim & Mnih 2013), all of which are described in Section 2 as well as a vanilla VAE (Kingma & Welling 2013). To assess diversity and visual fidelity beyond VAE-based models, also compare
generative adversarial network model, StyleGAN (Karras et al. 2019).
Generation Faithfulness and Diversity Analyses. We present image generation results from our model alongside baseline models, evaluating performance the FID and Vendi metrics across datasets. For image generation using VAE-models, we adopt two strategies: (1) Sampling a noise vector from multivariate standard normal and decoding it. (2) Encoding an actual image, then selecting a latent dimension. The value of this chosen dimension is adjusted by shifts of +/- 1, 2, 4, 6, 8, or 10 standard deviations. Subsequently, we decode the adjusted representation. In Figures 2 lad the the two sampling strategies are labeled as 'Sampled from Noise' and 'Sampled from Traversals' respectively. Figures 2 and show that a-TCVAE consistently generates more diverse (higher Vendi) and more faithful (Tower FID) images than baseline VAE models.
MPI3D-Real
Cars3D
Teapots
Caleba
16
Vendi
Sampled from Noise
Traversals
TCVAA
B-VAE
B-TOVAE
Factor
B-VAE+HFS
VAE
Style GAN
R-TOVA B-VAE
B-TOVAE
FactorvAE
B-VAE+ HFS
VAE
Style-GAN
a-TOVAE
B-VAE
B-TOVAE
Factor B-VAE+HFS
VAE
Style- GAN
T-TCVAL
B-VAE
B-TCVAE
Pactor-VAE
B-VAE+ VAE
Style GAN
G-TCVAE
3-VAE
B-TCVEE
Factor
B-VAE+
VAE
Style-GAN
Figure 2: Diversity of generated images, as measured by Vendi score. Two different sampling strate- gies are considered: sampled from white noise and from traversals. The diversity of the images of our model, a-TCVAE, is consistently higher than baseline VAE models, and on par with StyleGAN. The green dashed line represents ground truth dataset diversity. Traversals produce significantly more diverse images than samples.
The Vendi score of a-TCVAE is comparable to that of Style-GAN, and its FID score is only slightly worse. Moreover, Style-GAN takes 15x the training time (~> 2hrs vs. 30hrs on a single Nvidia Titan XP) and learns only a generative model, whereas VAEs learn both a generative model and a representational model. Noticeably, all VAE-based models perform poorly terms of both diversity and reconstruction quality when sampling from white noise, highlighting the benefit of structured sampling strategy when using VAE-based models for generative tasks. Another finding is that wwweersl--genertee images are superior to those obtained from the prior, i.e. sampling from a standard normal and decoding. This is keeping with prior work showing that drawing latent samples from distribution other than the standard normal, e.g. GMM, often leads to higher quality generated images Chadebec et al. (2022). and supports the claim that disentangled models allow more systematic cxploration of the latent space leading more diverse images. This claim is also supported by noting that all disentangled VAEs give higher diversity than the vanilla VAE.

---

Published as a conference paper at ICLR 2024
3DShapes
MelaD-Real
Cars3D
Teapots
Celeba
400
400
400
400
300
300
300
300
Sampled from Noise
200
200
200
200
Traversals
100
100
100
100
Style-GA GAN
TCVAV
B-TOVAE
FactorsiA
VAE
Style GAN
@-TCVAE
B-TCVAE
Factor-VAE
B-VAE HFS
VAE
Style GAN
T-TCAAE
8-VAE
8-TCVAE
Factor-
B-VAE +
VAE
Style GAN
T-TCVAE
B-VAE
B-TCVAE
Factor-VAE
NAE++FS
VAE
Style-GAN
--TCVAE
8-VAE
B-TCVAE
Factor
B-VAE+HFF
VAE
B-VA++HiP
Figure 3: Faithfulness of generated images to the data distribution, as measured by FID score. Two different sampling strategies considered: sampled from white noise and from traversals. The scores for the images of our model, a-TCVAE, are consistently better than baseline VAE models (lower FID is better), and only slightly worse than StyleGAN. Traversals produce significantly more faithful images than samples.
Disentanglement Analyses and Downstream Metrics Correlation Study In this section we ex- amine the disentanglement capabilities a-TCVAE and the related VAE baselines, and how relates, statistically, with the diversity and quality of generated images, as measured in Section 4 Figures and show that a- a-TCAAE consistently achieves comparable or better DCI, SNC and unfairness scores. The improvement of a-TCVAE over the baselines is most significant on the most realistic factorized dataset, namely MPI3D-Rea Interestingly, while there is a significant gap be- tween the DCI scores of disentangled and entangled models across every factorized dataset, SNC shows that in terms of single neuron factorization, for both Cars3D and MPI3D-Real, a-TCVAE the only model that significantly improves over the entangled VAE. This is perhaps due to the tendency of DCI to sometimes overestimate disentanglement Mahon et al. (2033). Cao et al. 2022
Furthermore, as illustrated in Figure [ no model has been successful in learning dis- entangled representations from the CelebA dataset. To meaningfully encode CelebA im- ages, we used high-dimensional latent repre- sentations (e.g.. 48 dimensions). However, as highlighted by Do & Tran (2020), dis- entangling and measuring disentanglement in high-dimensional representations are notori- ously challenging tasks. Indeed, while DCI and unfairness present unrealistic results, SNC gave all models a score of zero, and do not display the figures here. Figure [0 il- lustrates significant correlation between the Vendi, unfairness, and DCI metrics. There is compelling correlation between Vendi and DCI scores, underscoring that diversity and disen- tanglement are statistically related. This res- onates with the understanding that disentangled latent spaces naturally exhibit superior genera- tive diversity (Higgins al. 2019). Addition- ally, Vendi and DCI both exhibit a negative cor- relation with unfairness. This observation is consistent with Locatello et al. (2019a)'s 's find- ings, implying that the fairness of downstream prediction tasks is deeply associated with the diversity and disentanglement of the represen- tations being learned. Further correlations re- sults are given in Appendix [] along with ex- amples of latent traversals.
3DShapes
Ca
MP13D-Res
Teapots
BAVE
Figure 4: Comparison of DCI scores of our model with those of baseline models.
3DShapes
CarsaD
Teapots
Moda's
e-TEVAE E-MAE E-TCVAE
03.
MAME-HFS
6.1
Figure 5: Comparison of SNC scores our model with those of bascline models.
3DShapes
Cars3p
Calebo
NPBD Rea
Teapots
03
Unfaimess
Models 3 VAE
S-TOWN
3-MA HPP
Figure 6: Comparison of unfairness scores of our model with those of baseline models.
Attribute Classification Task In this exper- iment, train multilayer perceptron (MIP)

---

Published as a conference paper at ICLR 2024
3hapess
MPI3D-Real
Cars3D
Te Te-spots
Celeba
Vendi
00:49
10.29
0.85
0.58
0.26
10.48
0.5
0.55
0.96
00.49
10.61
0.53
Unfaimess
10.49
0.58
0.39
0.55
10:48
0.72
00.5
10.5
0.5
0.44
00.49
0.36
0.79
FID
00.29
0.47
0.11
0.26
0.39
0.93
0.56
0.61
0.84
0.55
00.64
0.69
0.53
0.72
0.96
0.44
0.53
0.78
0.15
Vendi
Unfaiiness
FID
DCI
Vendi
Unfairness
FID
DCF
Vendi
Unfairness:
FID
DCI
Vendi
Unfairness
FID
DCI
Vendi
Unfairnass
FID
DC1
Figure 7: Correlations between diversity (Vendi score), generation faithfulness (FID score), unfair- ness and DCI. Correlations are computed using the results from all models across 5 different seeds.
classify sample attributes using the models' encoded latent representations. Figure & reveals that a-TCVAE either matches or surpasses the baseline models in terms of attribute classification ac- curacy. The improvement is minor on 3DShapes and Teapots, but more significant Cars3D and MIP3D-Real. Interestingly, the only dataset where all VAEs exhibit commendable performance is CelebA, where high-dimensiooal representations are used. This suggests that, for this particular downstream task, the dimensionality of the representation may be the main constraining factor. In fact, this downstream task inherently favours high-dimensional attributes, considering that MIP employed for the attribute classification.
Loconav Ant Maze Reinforcement Learning Task. In this experiment, model-based RL agent has to learn its proprioceptive dynami- cal system while escaping from maze. Re- cently, Hafner et al. (2022) introduced Direc- tor, hierarchical model-based RL agent. Di- rector employs a hierarchical strategy with
3DShapes
Cars3D
Celeba
MPI3D-Real
Teapots
Classtfication Accuracy
0.6.
0.4
Goal VAE that learns and generates sub-goals, simplifying the planning task. The first hierar- chy level represents the agent's internal states, while in the second one, the Goal VAE en- codes the agent's state and infers sub-goals goals. result, the Goal VAE generates sub-goals to guide the agent through the environment. Given the enhanced generative diversity of a- e-TCVAE, we postulated that integrating our proposed TC bound could improve Director's exploration. In this experiment, we replaced the beta-VAE objective, used to train Director's Goal VAE, with our TC-bound, expecting richer diversity in sub-goals, thus expediting environment exploration and enhancing overall learning behaviour. Figure compares the performance of Director and Alpha-
Figure 8: Comparison of a-TCVAE and baseline models the Downstream Attribute Classifica- tion Task. Our proposed model either matches or surpasses the baseline models terms of attribute classification accuracy
2000
Exploration Reward
M
Mean Retum
Vendi
1000
Aipha Director Director
500
1006000
2000000
(a)
(b)
(c)
Figure 9: Performance of Director, a model-based hierarchical RL agent, and Alpha-Director on the Antmaze task. While director samples sub-goals using the original V-VAE, Alpha-Director samples sub-goal -oalo using the proposed a-TCVAE. Sampling using a-TCVAE gives more diverse goals (b), better exploration (c) and significantly higher mean return (a).
8

---

Published as a conference paper at ICLR 2024
Director, which replaces 3- VAE objective with the proposed TC-bound instead, the results eraged across three seeds. Figure (a) presents the mean return, which scores the performances of the agent on the given task (i.c., finding the exit of the maze while learning proprioceptive dynam- ies), showing that Alpha-Director significantly outperforms Director, learning faster and to a higher final high mean reward. Figure (bb()) illustrates the Vendi score of sampled goals across batch and sequence length, showing that OCTEA generates sub-goals with higher diversity score. As result, Alpha-Director has better exploration, as shown in Figure (c), leading to faster learning. Collectively, these findings highlight that a- a-TCVAE enables the agent to sample broader range of sub- uub-goals, fostering efficient exploration and ultimately enhancing task performance.
5 DISCUSSION AND FUTURE WORK
Through comprchensive quantitative analyses, we answer the defined research questions while de- lineating the advantages and limitations of the proposed model relative the evaluated baselines. Our findings resonate with the hypothesis posited by [Higgins et al. (2019), emphasizing strong correlation between disentanglement and generative diversity. Notably, disentangled representa- tions consistently showcase enhanced visual fidelity and diversity compared to the entangled ones. This correlation persists across all datasets rendered using disentangled representations. Intrigu- ingly, traversal analyses of a-TCVAE, illustrated in Figures [ and 16 in Appendix C reveal that is able to discover novel generative factors, such as object positioning and vertical perspectives, which are absent from the training dataset. We hypothesize that the CEB term is responsible for this phenomenon. Most existing models optimize only the information bottleneck, and while this result in factorized representations, does not directly optimize latent variable informativeness. Our proposed bound also includes a CEB term, and so maximizes the average informativeness as well, which may push otherwise noisy variables to learn new generative factors. Future research will delve deeper into comprehending this phenomenon and exploring its potential applications.
In accordance with the literature, the main limitation of a- TCVAE is that, akin to other disentangled VAEs, it is difficult to scale efficiently. This scaling challenge permeates the entire disentanglement paradigm. In high-dimensional spaces, not only do disentangled VAE-based models struggle to pro- duce disentangled representations, but also the metrics used to measure disentanglement tend not to be useful. (c.g., DCI and (NC(Eastwood Williams 2018 Mahon et al., ) On the other hand, disentangled representations have number of desirable properties, already showcased in the literature (Higgins et al., 2022). In particular, their impact undeniable in the Ant Maze RL experiment from Figure 9 Reinforcing this observation, our correlation study underscores the rela- tionship between disentanglement and diversity, leading to the following question: can we leverage diversity surrogate for measuring disentanglement complex and high-dimensional scenarios? We leave the answer to this question as a future work.
CONCLUSION
We introduce a-TCVAE, VAE optimized through convex lower bound the joint total cor- relation (TC) between the latent representation and the input data. This proposed bound naturally reduces a convex combination of the known variational information bottleneck (VIB) (Alemi et al. 2017) and the conditional entropy bottleneck (CEB) (Fischer & Alemi 2020). Moreover, it generalizes the widely adopted 3-VAE bound. By maximizing disentanglement and average infor- mativeness of the latent variables, our approach enhances both representational and generative ca- pabilities. comprehensive quantitative evaluation indicates that a-TCVAE consistently produces superior representations. This is evident from its performance across key downstream metrics: dis- entanglement (i.c., DCI and SNC), generative diversity (i.c., Vendi score), visual fidelity (i.c., FID), and its demonstrated downstream usefulness. In particular, our a-TCVAE showcases significant improvements on the MPI3D-Res dataset, the most realistic factorized dataset in our evaluation, and in downstream reinforcement learning task. This highlights the strength of maximizing the average informativeness of latent variables, offering a pathway to address the inherent challenges of disentangled VAE-based models.
9

---

Published as a conference paper at ICLR 2024
7 ETHIC STATEMENT AND REPRODUCIBILITY
To the best of the authors' knowledge, this study does not involve any ethical issues. The authors aim maximize the reproducibility of the study. The codes of this project will be released in the camera-ready version. In the methods section, notions align with existing literature.
8 ACKNOWLEDGEMEENT
We thank Prof. Yoshua Bengio for the useful feedback provided along the project. We thank Mila Quebec Al institute, TUDelft, and Compute Canada for providing all the resources to make the project possible.
REFERENCES
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. International Conference on Learning Representations, 2017.
Sanjeev Arora and Boaz Barak. Computational Complexity: A Modern Approuch. Cambridge University Press, 2009. doi: 10. N0.1017//B0978051 18040900
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE Transactions on Pattern Analysis and Machine Intelligence, 1798)11798- 2013.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http://gihhb.coomdeppmmm////aaas 2018.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 3-VAE. arXiv preprint ArXiv:/804.03599 2018
Jinkun Cao, Ruiqian Nai, Qing Yang, Jialei Huang, and Yang Gao. An empirical study dis- entanglement of negative-free contrastive learning. Advances in Neural Information Processing Systems, 35:1210-1221 2022.
Clément Chadebec, Louis Vincent, and Stéphanie Allassonnière. Pythae: Unifying generative au- toencoders in python-a benchmarking case. Advances in Neural Information Processing Sys- tems, 35:21575 25522999-2188, 2022.
Ricky TQ Chen, Xuechen Li, Roger B. Grosse, and David K. Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. Advances in Neural Information Processing Systems, 31, 2018.
Wenxiao Chen, Haowen Xu, Zeyan Li, Dan Pei, Jie Chen, Honglin Qiao, Yang Feng, and Zhaogang Wang. Unsupervised anomaly detection for intricate KPIs via adversarial training of VAE. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications, pp. 1891-1899 IEEE, 2019.
Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them im- ages. In International Conference on Learning Representations, 2020.
Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled representations. In International Conference on Learning Representations, 2020.
Cian Eastwood and Christopher KI Williams. A framework for the quantitative evaluation of disen- tangled representations. In International Conference Learning Representations, 2018.
Ian Fischer and Alexander A Alemi. CEB improves model robustness. Entropy, 22(10):1081, 2020.
Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning. arXiv preprint ArXiv:22100024100, 2022.
10

---

Published as a conference paper at ICLR 2024
Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. Auto-encoding total corrla-- tion explanation. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1157-1166. PMLR, 2019.
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov,Joc Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Anirudh Goyal, Aniket Didolkar, Nan Rosemary Kc, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael C. Mozer, and Yoshua Bengio. Neural production systems. Advances in Neural Information Processing Systems, 34:25673 0-22677--26877, 2021.
Danijar Hafner, Kuang-Huse Lee, lan Fischer, and Pieter Abbeel. Deep hicrarchical planning from pixels. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 26091-26104 Curran Associates, Inc., 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta- VAE: learning basic visual concepts with constrained variational framework. In International Conference Learning Representations, 2016.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations. In Theoretical Physics for Deep Learning Workshop. ICML, 2019.
Irina Higgins, Sébastien Racanière, and Danilo Rezende. Symmetry- based representations for arti- ficial and biological general intelligence. Frontiers in Computational Neuroscience, 16:836498, 2022.
HyeongJoo Hwang, Geon- Ceennny Kim, Seunghoon Hong, and Kee-Eung Kim. Multi nulti-view repre- sentation learning via total correlation objective. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401 44100 2019.
Hadi Kazemi, Seyed Mehdi Iranmanesh, and Nasser Nasrabadi. Style and content disentanglement in generative adversarial networks. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 848-856. IERE, 2019.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning. pp. 2649-2658. PMLR, 2018
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes.
arXiv preprint
ArXiv:1312.6114 2013.
Andreas Kirsch, Clare Lyle, and Yarin Gal. Unpacking information bottlenecks: Surro- gate objectives for deep learning, 2021. URL
net wwt.:e/foreerremm forumiidurir
5rc0KDezhqT
Hsin Ying Lee, Hung- Yu Tseng, Jia- Huang, Maneesh Singh, and Ming Hsuan Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European
Conference Computer Vision (ECCV), pp. 35-51, 2018

---

Published as a conference paper at ICLR 2024
Yang Li, Cheng Yu, Guangzhi Sun, Weiqin Zu, Zheng Tian, Ying Wen, Wei Pan, Chao Zhang. Jun Wang, Yang Yang, et al. Cross-utteranes conditioned VAE for speech generation. arXiv preprint ArXi:22309.04566 2023.
Yuheng Li, Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Mixnmatch: multifactor dis- entanglement and encoding for conditional image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint ArXiv:1701.07274 2017.
Zeming Lin, Tom Sercu, Yann LeCun, and Alexander Rives. Deep generative models create new and diverse protein structures. In Machine Learning for Structural Biology Workshop, NeurIPS, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the faimess of disentangled representations. Advances in Neural Information Processing Systems, 32, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. International Conference on Muchine Learning, pp. 4114 4124. PMLR. 2019b.
Louis Mahon, Lei Shah, and Thomas Lukasiewicz. Correcting flaws in common disentanglement metrics. arXiv preprini ArXiv:2304.00333: 2023.
Qi Mao, Hsin- Ying Lee, Hung-Yu Yu Tseng, Siwei Ma, and Ming-Hsuan Yang. Mode seeking gener- ative adversarial networks for diverse image synthesis. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp.
2019.
Emile Mathieu, Tom Rainforth, Nana Siddharth, and Yee Whye Teh. Disentangling disentanglement in variational autoencoders. International Conference on Machine Learning, pp. 4402-4412. PMLR, 2019,
R. Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics, 1 11:652-670, 2023.
Pol Moreno, Christopher KI Williams, Charlie Nash, and Pushmeet Kohli. Overcoming occlusion with inverse graphics. In European Conference on Computer Vision, pp. 170-185. Springer, 2016.
Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse structure for image inpainting with hierarchical vq- VAE. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10775-10784, 2021.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint ArX1v:1511.06434, 2016.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-VAE-2. Advances in Neural Information Processing Systems, 32, 2019.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making In Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015.
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea direc- tions (by accident). Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12406-1241 2019.
Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentan- glement correlated factors via hausdorff factorized support. In International Conference on Learning Representations (ICLR), 2023.
12

---

Published as a conference paper at ICLR 2024
Yuge Shi, Brooks Paige, Philip Torr, al. Variational mixture-of of-experts autoencoders for multi- modal deep generative models. Advances in Neural Information Processing Systems. 32, 2019.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via informa- 2017.
tion. arXiv preprini
Guangzhi Sun, Yu Zhang, Ron J. Weiss, Yuan Cao, Heiga Zen, Andrew Rosenberg, Bhuvana Ram- abhadran, and Yonghui Wu. Generating diverse and natural text-to-speedl samples using quan- tized fine-grained VAE and autoregressive prosody prior. In ICASSP 2020-202222222 0220-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6699-6703, IEEE, 2020.
Xiaohong Sun, Jinan Gu, and Hongying Sun. Research progress of zero-shot learning. Applied Intelligence, 51(6):3600-30 3614, 2021.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception- resnet and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Naftali Tishby, Fernando C. Pereira, and William Bialek. ceedings of the 37th Allerton Conference Communication, Control and Computation, 2001.
The information bottleneck method. Pro-
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 13666767-9677, 2020.
Ziyu Wang, Yiyi Zhang, Yixiao Zhang, Junyan Jiang, Ruihan Yang, Junbo Zhao, and Gus Xia. Pianotree VAE: Structured representation learning for polyphonic music. arXiv preprint ArXiv:2008.071 18, 2020.
Zachary Wu, Kadina E. Johnston, Frances H. Arnold, and Kevin K. Yang. Protein sequence design with deep generative models. Current Opinion in Chemical Biology, 65:18-27, 2021.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey transfer learning. Proceedings of the IEEE, 109(1): 43-76, 2020.
13

---

Published as a conference paper at ICLR 2024
A TOTAL CORRELATION LOWER BOUND DERIVATION
In this section are going derive the TC lower bound defined in equation Since it is defined as a convex combination marginal log-likelihood, VIB, and CEB terms, we are going to split the derivation into subsections. First, will derive first TC bound that introduces the VIB term. Then, we will derive another TC bound, which explicitly shows the CEB term. Finally, we will define the TC bound shown in equation as convex combination of the two bounds.
A.1 TC BOUND AND THE VARIATIONAL INFORMATION BOTTLENECK
Unfortunately, direct optimization of mutual information terms is intractable Alemi al. [2017 Therefore, we first need to find lower bound of equation Following the approach used in Hwang et al. 20221) expand as:
(2,
To (zk, xx) To(2,
k=1
K
log 20(x2|zx)
log g)(x)|2)
Beu
Byo(e
PD (ar)
pD(x))
K
log g0(x|(z) Pp(xx(z))) 11 (D P)(x)zz)
log [0(2|00) r(z)
E (x)
(z(z) r(z)
Let's expand these two terms:
Equ log g0(2|zx) Po 2x)
/
(Zk, x)) log 20 (0(x)2x)) pa (a(a 2x)
dz
(8)
pD(a)) (a(
pD( Po Po(ar (arl =x)
= =/ʃ (0(2z)))) log qa (a(a)=z)) zx)
+ log (og log dz
11 (x) + (zx) (e))) (x) + E Bu Plo
(x,() log go (x(2(()) r(z)
20(2) r(z)
=/ 2o (x,, 2og
(2(())z) r(z)
dzdæ,
g0(z) r(z)
/ 9o (=(2(=))D(()) (log 00(21x)
xx) + log
r(z)
dzdæ,
r(z)
(0(2))
Epp( DKL (2K Bgn(m/=) (ool0)))DK((000)))(((()))))
As result, write:
14

---

Published as a conference paper at ICLR 2024
K Σ
TC)(z.
[(x)) + (oo Dog
k=1
(10)
(xx (x ao -Kp
K
(x) + (fgelz,z)dez4) logpula/2)) - (p (e0c)))))
(xx) + (2 (ogp EpD( (xpeeen)Die((
K K (z) + (x loog ps (2)(2))) Ppn (oro0(0(DDD(((((()))))))))) [r
KH ( + E (z. (x) + (og (p) DxL0[(())))](((
(=|==) pa ( (a) D (z)=)) =: x(z,0).
VIB
Maximizing L(z, a)) not only maximizes the original objective TC(2 but at the same time minimize the gap produced by upper bounding equation I0 As result,
+ (0a Eg (ew.csallllxxx(((())))))))))))) (1I)
will be minimized, leading to: r(z) ~ (=(z) and (a(=)))) go (z]
Moreover, since (xx) and Po) do not depend on O, we can drop them from L(2, a)). Finally, to avoid using heavy notation, we will denote the VIB term as (xxx(o()/=>))(((()))
leading to the first TC bound which introduces the VIB term:
TCo(z, 2 (log (k=((o(((=)]I((())
(12)
VIB
A.2 TC BOUND AND THE CONDITIONAL VARIATIONAL INFORMATION BOTTLENECK
Expanding Eq. equation 2 we can reformulate TC(z, x) as follow:
(2,(2,
T0(2k, x.) Tz(2,
(13)
K
K
1
10 16(2,, 2)) + 1s 1z(2
K K I0(2, xx)
K
K -
1
Σ
18(25,
+ K
(Io( - [0(2,2))
K
(14)
15

---

Published as a conference paper at ICLR 2024
Interestingly, can write the last term of Eq. equation 14 as:
log Pe (x(
E (x0,)) log Po
T [k(z x) - Iø(2, x²) = Bps(o)))))
(15)
po(x)
po(x)
- ( (o((0))))(())dz) (d)zzz
log (o(20 dz
pD(x)
f P=(a))))(()) log
P (2c x
dzdæ,
PD (a))
/ po(a))))(()))))))))) log
Pol (z) kkk dzdæ,
p( z))
p(x 2)
po(2|2 po(z|z)p(=)1og log
dædx,
Pol ( z/p
/ pa(2) p=(z|z)p(()) log
p( 2y 2k. dzdx, Ps (,)2))
1s (Z 2/20)
We can now write equation 5
K
TC)(2,
1 K
Σ [K -o(() m/2z]]
Interestingly, the second IB term in Eq. (8) can now be expressed as multiple conditional MIs between the observation and K- other latent variables given the k-th latent representation variable, penalizing the extra information of the observation not inferable from the given latent representation variable. Moreover, we can further expand the TC as:
TCo(z, = K
Σ [(K 1)TT x) To(= 20 m0|2)]]
(16)
1=1
log 20(x|zz) (a(x)
(K 1) ()
K k=
(x( ( zz
log go (x) rx(2(())]
[x,,(
(x,() [log go
go (z (x) rp (2(2)()))))
log (0(2) p,
K 1) (epo
(D( (z(x)(z))
rp (p(z (r)r prpppp (zzeh xx)
Epp (a: (g ( (m,().))
log
go z | J (z(
K K
K
[x)) + plo
(pp((e) (er.eeeeee
K
K
k=1
K
By (zus ( ([+(=]]
K
K
(17)
+ | DEL (x
K
K
H(w) + E 2n) (ox
Epp (w) (x
K
CEB
(18)
16

---

Published as a conference paper at ICLR 2024
Maximizing Eq. 18 not only maximizes the original objective (C(2, æ)) but at the same time min- imizes the gap produced by upper bounding Eq. equation 17 leading to: rp (2(21/z) xx) s3 (0 (x), qo (z T2(22 x)) and ga( (()) pp(( x.) Moreover, since (x) does not depend on can drop from Eq. equation 18] Finally, to avoid using a heavy notation, we will denote the CEB
term DEL (20 [|([(=]))) leading to the second TC bound which introduces the CEB term:
K 1-1
(2, 2
E (2)(m)) [Z (x=(90(=/=)11pp/]
(19)
K
A.3 FINAL TC BOUND
In order to obtain the final expression of the derived TC bound, compute convex combina- tion of the two bounds defined in Eq. equation 12 and equation T9
K
= -a)
Σ
10( x)- - (0(2
(20)
K 1
1 [lo(2,
+
(z x))
10
(21)
K
K
k=1
K(1 a)) a(K 1)
K
Io(zk;
(z(( (1 (11-))))(
K
K
K
E (2) [log [[og)(<<|2]] a KL ( ( -11--))kx(o((tll)liir
K
Ka D D a:) (x
(=|(m)
[logp<(<|2)]
CER
VIB
where a is a hyperparameter that balances the effects of VIB and CEB terms. Table A.3 illustrates the lower bounds defined for 3-VAE Higgins al. (2016), FactorVAE Kim & Mnih (2018). HFS Roth et al. (2023) and 3-TCVAE Chen et al. (2018) comparing them to the derived TC bound. We see that the three bounds present similar structure, presenting marginal log-likelihood term
and either one or two KI. regularizers that impose some kind of information bottleneck.
17

---

Published as a conference paper at ICLR 2024
Model
Lower Bound
5-VAE
(=|== flog pal x - (0xl(g0(2))))lpf
FactorVAE
E (ootsla)(logPp(( - (/t
IT (o(2)))
3-TCVAE (ltein)lpooologggn (2; - 3D (D E, D (((2)))l1((3)
HES
Eg (2o (x|z) - - Σ www
xZ., min, y d(a,z')
a-TCVAE
(x logg ( z)] K-a
Ka DKL (
[1-1 DxL 100 (1(())
Table I: This table compares the lower bound objective functions of B-V, 3-TCVAE, Factor- VAE and UFS. The lower bound objective function of 3-VAE is composed of the expected log- ogglikklihood of the data given the latent variables and the KL divergence between the approximate posterior and the prior of the latent variables (i.e., VIB term). The FactorVAB model further adds KI. divergence term between the approximate posterior and the factorized prior of the latent vari- ables, which approximates the total correlation of the latent variables, and HES--VE further adds Monte-Carie approximation of Hausdorff distance. a-TCVAE, vex combination of VIB term and KL. divergence between the approximate posterior and the prior of the latent variables conditioned on the k-th latent variable (i.e., CEB term). K represents the dimensionality of the latent variables, while B, y and a are hyperparameters of the models.
the other hand, uses
B ARCHITECTURES AND HYPERPARAMETERS DETAILS
The hyperparameters used for the different experiments are shown in Table
Table 2: Comparison of the different hyperparameters used across the Datasets
Dataset
B
latent dim K
Training Epochs
Teapots
2
10
0,25
10
50
3DShapes
3
10
0.25
10
50
Cars3D
4
10
0.25
10
50
MP13D-Real
5
10
0.25
10
50
Celeba
5
10
0.25
48
50
All encoder, decoder and discriminator architectures are taken from Roth et al. (2023)
C FURTHER DETAILS ON DATASETS AND METRICS
C.1 DATASETS
We test five datasets. Teapots (Moreno et al. 2016) contains 200, 000 images of size 64 x 64. Each image features a rendered, camera-centered teapot with 5 uniformly distributed generative factors of variation: azimuth and elevation (sampled between 0 and 2xr), along with three RGB colour channels (each sampled between 0 and 1). 3DShapes (Burgess & Kim 2018) consists of 480, 000 images of size 64 64. Every image displays rendered, camera-centered object with uniformly distributed generative factors of variation: shape (sampled from [cylinder, tube, sphere, cube]), object colour, object hue, floor colour. wall colour, and horizontal orientation, all determined using linearly spaced values. MPI3D-Real (Gondal et al.) 2019) comprises 103, 680 images of size 64 64. Each image captures objects a robot arm's 's end, characterized by 6 factors: object colour, size, shape, camera height, azimuth, and robot arm altitude. Cars3D (Reed et al.) (015)) is made up of 16, 185 images of size 64 x 64. Each image portrays a rendered, camera-centered car, categorized by 3 factors: car-type, elevation, and azimuth. CelebA (Liu et al. 2015) encompasses over 200, 000 images of size 64 x 64. Every image presents a celebrity, highlighted by broad range of poses, facial expressions, and lighting conditions, which sum up 40 different factors. Every model is trained using a subset containing the 80% the selected dataset images in a fully unsupervised way. The models evaluated on the remaining images using the following downstream scores. While
18

---

Published as a conference paper at ICLR 2024
CelebA is the most complex dataset, MPI3D-Real is the most realistic among the ones usually used in the disentanglement community.
C.2 METRICS
When using the FID score assess image quality, we compare the distribution of generated images to that of the real images. Specifically, FID (Heusel et al. 2017) measures the distance between two distributions of images, and we apply measure the distance between the generated images and the real ones. A lower distance is better, indicating that the generated images belong to the distribution of ground truth images.
The Vendi score (Friedman & Dieng 2022 which we use to measure the diversity of the generated images, is computed with respect to a similarity measure. Specifically, it is calculated as the expo- nential of the entropy of the eigenvalues of the similarity matrix, i.e. the matrix whose (ž, (i,j))t entry the similarity between the ith and jth data points. It can be interpreted as the effective number of distinct elements in the set.
To assess the quality of encoded latent representations, we use DCI, SNC/NK (Mahon et al. 2023) and the unfairness measure of Locatello et al. (019a)).
DCI, the first disentanglement metric we compute, first trains a regressor to predict the generative factors from the latent representation, and from this regressor extracts matrix of feature impor- tances, where the (i. entry is the import of the ith latent dimension to predicting the jth gen- erative factor. It then takes normalized version of) the entropy of rows and columns to compute 'disentanglement and 'completeness' respectively. The accuracy of the regressor is taken as the 'informativeness' score. The average of these three scores, across all factors and neurons, is the final DCI score.
SNC/NK, the second disentanglement metric we compute, works by first aligning neurons to latent factors using the Munn-MMukres algorithm to enforce uniqueness. Then each aligned neuron is used as a classifier for the corresponding factor, by binning its values. A higher accuracy of this single-neuron classifier (SNC) is better, indicating that the factor is well-represented by a single unique neuron. Neuron knockout (NK) caleulated as the difference between an MI.P classifier that predicts the generative factor from all neurons, and one that predicts using all neurons but the one that factor was aligned to. A high NK also better, indicating that no ncurons, other than the one it was aligned to, contain information about the given factor. SNC/NK measures a slightly different and stronger notion of disentanglement than DCI, as it explicitly assumes an inductive bias that enforces each factor to be represented by single latent variable.
MIG, is a disentanglement metric that quantifies the degree of separation between the latent vari- ables and the generative factors in dataset. It calculates the mutual information between each latent variable and each generative factor, identifying the variable that shares the most information with each factor. The gap, or difference, in mutual information between the top two variables for each generative factor is then computed. A larger gap indicates that one latent variable is significantly more informative about generative factor than the others, signifying higher degree of disentan- glement. This metric is particularly useful in scenarios where a clear and distinct representation of generative factors is desired in the latent space. MIG thus complements DCI and SNC/NK by pro- viding a measure of how well-separated the representations of different generative factors are within the model's latent space.
D EXTENDED RESULTS
Here, we present further results, in addition to those from Section Figure T0 extended Fig. reporting the correlations also with SNC, NK and the attribute classification accuracy shown in Figure g Unsurprisingly, there is a strong correlations between the three metrics designed to mea- sure disentanglement: DCI, SNC and NK. This, to some extent, verifies the reliability of these dif- ferent disentanglement metrics. SNC and NK also correlate strongly with Vendi, as DCI does. This further supports the finding in our paper of relationship between disentanglement and diversity.
19

---

Published as a conference paper at ICLR 2024
3DShapes
RPI3D-Real
Vendi
Vendi
Unfairness
0.49
0.65
0.64
0.15
0.47
0.69
Unfairness
0.58
0.74
0.83
-0.82
0.39
0.55
SNC
0.93
0.65
0.56
00.15
0.18
0.98
SNC
0.68
0074
0.85
0.54
0.85
0.89
NK
0.58
0.64
0.56
0.64
.0.83
0.51
NK
0.75
-0.8
0.85
0.59
-0.49
0.84
MLP
0.07
-0.15
-0.15
MLP
0.73
-0.82
0.59
0.42
FID
-0.18
-0.83
-0.86
FID
-0.85
-0.49
-0.14
-0.64
DCI
0.85
-0.69
0.98
0.51
-0.26
-0.11
DC1
0.84
-0.55
0.89
0.84
0.42
-0.64
Vendi
Unfaimess
SNC
NK
MLP
FID
DCI
Vendi
Unfaimess
SNC
NK
MLP
FID
DCI
(a)
(b)
Cars3D
Teapots
Vendi
.0.48
0.61
0.81
0.36
0..3
0.69
Vendi
-0.5
0.89
0.87
0.22
0.56
0.96
Unfairness
Unfairness
SNC
0.84
0.64
-0.77
SNC
-0.81
0.79
0.88
NK
0.81
-0.73
0.84
0.28
-0.88
0.95
NK
0.87
-0.3
0.79
0.41
-0.71
0.97
MLP
0.36
-0.64
0.64
0.28
-0.57
0.01
MLP
0.22
0.2
0.08
0.41
-0.68
0.35
FID
-0.93
0.72
-0.77
-0.88
-0.57
-0.71
FID
-0.56
0.5
-0.69
-0.71
-0.68
-0.7
DCI
0.72
-0.44
0.97
Vendi
Unfairness
SNC
NK
MLP
FID
DCI
Vendi
Unfairness
SNC
NK
FID
DCI
(c)
(d)
Celeba
Vendi
-0.49
0.79
0.69
0.61
0.53
Unfairness
0.49
-0.01
-069
0.36
0.79
SNC
-0.01
0.34
MLP
0.69
-0.99
0.5
-0.1
0.54
FID
0061
0.36
-0.13
-0.1
-0.15
DCI
-0.79
0.34
0.54
0.15
Vendi
Unfairness
SNC
NK
MLP
FID
DCJ
(e)
Figure 10: Correlations between all metrics measure, both for the generated images and the representations.
Figure I shows the results for neuron knockout (NK), the second metric introduced by Mahon et al.
2023 alongside SNC, which is shown in Figure 5 Similar SNC, the NK score for o-TCVAE
20

---

Published as a conference paper at ICLR 2024
3DShapes
C
MPI3D-Real
Teapots
0.3
Models
a-TCVAE
3-VAE
NK
0.2
B-TCVAE
Factor-VAE
B-VAE+HFFS
0.1
VAE
0
Figure 11: Comparison of the neuron- cuuron-knockout score of a- e-TCVAE with that of baseline models. As with other metrics presented the main paper, the improvement of a-TCVAE minor on 3DShapes and Teapots, but more substantial on Cars3D and MPI3D-Real
3DShapes
Cars3D
Celeba
MPI3D-Rea
Teapots
0.4
0.3
Models
a-TCVAE
B-VAE
0.2
B-TCVAE
Factor-VAE
B-VAE+HFS
0.1
VAE
Figure 12: Comparison of the MIG score of a-TCVAE with that of baseline models. As with other metrics presented in the main paper, the improvement a-TCVAE is minor on 3DShapes and Teapots, but more substantial Cars3D and MPI3D-Real.
3DShapes
Cars3D
Celeba
IPI3D-Real
Teapots
0.6
Models
a-TCVAE
0.4
B-VAE
B-TCVAE
Factor-VAE
0.2
B-VAE+HFS
VAE
Figure 13: Comparison of the DCI-C completeness score of a- a-TCVAE with that of baseline models. As with other metrics presented in the main paper, the performance of a-TCVAE comparable on 3DShapes, CelebA and Teapots, and better on Cars3D and MPI3D-Real.
higher than that for baseline VAE models and, while the errorbars often overlap, the superiority of q-TCVAE is consistent across all five datasets and is most substantial MPI3D-Real. Figure
12 shows the results for mutual information gap (MIG), which follows the same trend of NK. SNC and DCI scores. Figures 13 14 and 15 present the results of Completeness, Disentanglement and
21

---

Published as a conference paper at ICLR 2024
3DShapes
Cars3D
Celeba
MPI3D-RA Reaa
Teapots
0.8
Models
0.6
a-TCVAE
B-VAE
0.4
B-TCVAE
Factor-VAE
B-VAE+HFS
0.2
VAE
Figure 14: Comparison of the DCI-D disentanglement score of Q-TCVAE with that of baseline mod- els. As with other metrics presented in the main paper, the performance of a-TCVAE is comparable on 3DShapes, CelebA and Teapots, and better on Cars3D and MPI3D-Real.
3DShapes
Cars3D
Celeba
MPI3D-Rea
Teapots
0.75
Models
a-TCVAE
B-VAE
0.50
B-TCVAE
Factor-VAE
B-VAE+HFS
0.25
VAE
Figure 15: Comparison of the DCI-I informativeness score of a-TCVAE with that of baseline mod- els. As with other metrics presented in the main paper, the performance of a-TCVAE comparable on 3DShapes, CelebA and Teapots, and better on Cars3D and MPI3D-Real.
Informativeness metrics (DCI-C, DCI-D and DCI-I, respectively). The final DCI scores shown in fig. 4 is computed as geomtric mean the three scores.
22

---

Published as a conference paper at ICLR 2024
E DISCOVERING NOVEL FACTOR OF VARIATIONS
Figure 16 presents a-TCVAE traversals across 3DShapes, Teapots and MPI3D-Real datasets. The red boxes indicate the discovered novel generative factors, that are not present within the train dataset, namely object position and vertical camera perspective. While we do not have a compre- hensive explanation of why such an intriguing phenomenon is shown, we believe that the intuition behind can be explained considering the effects of VIB and CEB terms in the defined bound. Indeed, while VIB pushes individual latent variables to represent different generative factors, CEB pushes them to be informative. As result, the otherwise noisy dimensions, pushed to be informative (i.e., CEB) and to represent a distinct generative factor (i.e., VIB), resulting in the discovery of novel generative factors.
(a) 3DShapes Traversals
(b) Teapots Traversals
(c) MPI3D-Rea Traversals
Figure 16: a- a-TCVAE generated latent traversals the 3DShapes, Teapots and MPI3D-Real datasets.
The generated latent traversals reveal that a-TCVAE can Iearn and represent generative factors that are not present in the ground-truth dataset, namely vertical perspective and object position. The discovered generative factors are indicated with red box.
F RELATIONSHIP BETWEEN CEB AND DIVERSITY
F.1 FISHER'S 'S DEFINITION OF CONDITIONAL ENTROPY BOTTLENECK
Fisher's approach to the Conditional Entropy Bottleneck Fischer Alemi (2020) an extension of the Information Bottleneck (IB) principle Allmii et al. (2017), aimed at finding an optimally compressed representation of variable that remains highly informative about another variable Y, under the influence of conditioning variable Z. The CEB objective, according to Fisher, is formalized as trade-off between two competing conditional mutual information terms:
min 2(222
Here, I(X; ZIC) quantifies the amount of information that the representation Z shares with X, conditioned on C. Simultaneously. I(Y; Z(C) measures how much information retains about Y, also under the condition of C. The parameter 3 serves erucial tuning parameter, balancing these two aspects.
F.2 ADAPTING CEB TO VAES WITHOUT CONDITIONING VARIABLES
In the realm Variational Autoencoders, where the training strategy is to reconstruct the input data X using latent representation Z without any external conditioning C, the CEB framework
23

---

Published as a conference paper at ICLR 2024
undergoes a significant simplification. Given that X = Y in typical VAE setup, the CEB objective reduces to form where the focus shifts to optimizing the mutual information between X and its latent representation Z:
min [(1 (1--8)(C) (X) p(ala)
where H(X) represents the entropy of the input data, and ZZ is the conditional entropy of the input given its latent representation. This formulation underscores the trade-off between compressing the input data in the latent space and retaining essential information for accurate reconstruction.
This objective can be further broken down as (1- H -1-
F.3 INCORPORATING DIVERSITY INTO THE CEB OBJECTIVE
Following Friedman & Dieng (002)) Diversity can be quantitatively expressed as the exponential of the entropy of the latent space distribution (Z)
Diversity = exp (z)(H))|X))
To understand how the CEB framework relates to this notion of diversity, we utilize the entropy
chain rule H(Y H(X, Y) H(X). which allows to decompose (X) (2) in terms of the
joint entropy (X, Z) and the conditional entropy HZ)). Consequently, the CEB objective evolves into more comprehensive form that explicitly accounts for the diversity of the latent space:
min [1 ())(X H(X, H[))] <(>
min - [1-)
The latter one, makes clear the connection between the CEB term and Diversity as defined in Fried- man & Dieng (2022) ) Indeed, we can see that when minimizing the CEB term the Diversity term maximized.
G DISENTANGLEMENT AND VARIATIONAL INFORMATION BOTTLENECK
Disentanglement in VAEs, following Higgings' 3-VAE framework, seeks to learn representations where individual latent variables capture distinct, independent factors of variation in the data. This is achieved by modifying the traditional VAE objective apply stronger constraint space information bottleneck, controlled by hyperparameter 8. The 3 V-VA,, introduced by Higgins et al. (2016), represents seminal approach disentanglement, promoting the learning of factorized and interpretable latent representations.
the latent
On related front, the Variational Information Bottleneck (VIB) method, formulated by Alemi al. (017)). extends the Information Bottleneck principle to deep learning. The VIB approach seeks to find optimal trade-off between the compression of input data and the preservation of relevant information for prediction tasks. By employing variational approximation, VIB efficiently learns compressed representations that are predictive of desired outcomes. Interestingly, Alemi formulates VIB objective that is equivalent to Higgins' B-VAE one. Such result makes evident how imposing a higher information bottleneck leads higher disentanglement.
H SENSITIVITY ANALYSIS OF &
In this section we present a sensitivity analysis of how a affects Vendi and FID results across the considered datasets. To be consistent and analyze how Alpha influences disentanglement scores, also report a sensitivity analysis of the DCI metric and a correlation study showing how alpha is statistically correlated with FID, Vendi, and DCI metrics.
24

---

Published as a conference paper at ICLR 2024
H.1 DIVERSITY AND VISUAL FIDELITY SENSITIVITY ANALYSIS WITH RESPECT TO
To analyse how influences the presented results, performed evaluation of FID, Vendi and DCI using a E [0.00, 0.25, 0.50, 0.75, 1.00], where for a = 0.00 we obtain 3-VAE model, while for a 0.25 get the results presented in the main paper. Figures 17 and 18 [0.25, 0.50] a a-TCVAE presents the highest diversity scores, while keeping a FID score comperable 3-VAE.
s8 show that, when
3DShapes
MPI3D-Real
Cars3D
17
Teapets
Celeba
14.5
17
16
Vendi
12
13.5
Sampled from
11
14
13
10
15
13
12.5
0.0
0.25
0.25
0.25
Figure 17: Sensitivity Analysis of the Diversity of generated images with respect a. Only one sampling strategy is considered: sampled from traversals. The green dashed line represents ground truth dataset diversity. It can be seen that the higher alpha the higher will the Vendi Score.
3DShapes
MP13D-RAR Reall
Cars3D
Coleba
275
100
90
250
80
150
FID
70
225
60
Sampled from Traversals
200
40
50
0.0
0.25
0.5
0.73
1.0
0.0
0.25
0.5
0.75
0.25
0.75
077
Figure 18: Sensitivity Analysis of the Faithfulness of generated images to the data distribution, as
measured by FID score, with respect to a. Only one sampling strategy is considered: sampled from traversals. can be seen that for E [0.25, 0.50] the model presents higher visual fidelity.
Interestingly, the two sensitivity analyses show two main trends:
Diversity increases when using higher values of Alpha.
FID score improves when using smaller values of Alpha.
Indeed, when using higher values of we increase the contribution of the CEB term in equation 6 which enhances diversity at the cost of visual fidelity. As result, the higher the value a, the more diverse the generated batch of images, and the lower will be the generation quality. However, it can be noticed that when using values of a between 0.25 and 0.50, we get a set of generated images that more diverse and still have better comparable visual fidelity than 8- 3-V (i.e., a0)).
H.2 DISENTANGLEMENT SENSITIVITY ANALYSIS WITH RESPECT TO
Here, we present a sensitivity analysis of the DCI metric. Figure 19 shows that the interval [0.25- 0.50] presents higher values of disentanglement, following Diversity and Visual Fidelity analyses that show the best results the same range. Such a trend can be explained by considering that a weights the contributions of VIB and CEB terms. While the CEB term enhances diversity, the VIB term encourages disentanglement. As a result, we can see that DCI scores decrease when a gets closer to I. Interestingly. when is in [0.25,0.50], the combination of CEB and VIB terms produces a better bound for the Total Correlation objective than when using, which results in higher DCI scores.
25

---

Published as a conference paper at ICLR 2024
3DShapes
Cars3D
Celeba
MPI3D-Real
Teapots
0.8
0.6
Q
0.0
DCI
0.25
0.4
0.5
0.75
0.2
1.0
0
Figure 19: DCI scores sensitivity analysis with respect a. On average when [0.25, 0.50]
a-TCVAE presents the best DCI scores.
H.3 CORRELATION STUDY: HOW IS CORRELATED WITH VENDI, FID, AND DCI METRICS?
Here, we present correlation matrices for all the considered datasets. We computed them using the models trained for the alpha sensitivity analyses. The correlation matrices fig. 20 confirm the trends observed in the other sensitivity analyses (i.e., Vendi, FID, and DCI). Indeed, a has strong positive correlation with both FID and Vendi, showing that when a increases, diversity increases and FID deteriorates. On the other hand, has strong negative correlation with DCI for all datasets besides the Cars 3D dataset, showing that, on average, the higher the value of a, the lower the disentanglement.
26

---

Published as a conference paper at ICLR 2024
3DShapes
MPI3D-Real
a
0.86
0.88
-0.76
0.96
0.68
-0.5
Vendi
0.86
0.51
-0.41
Vendi
0.96
0.72
-0.56
FID
0.88
0.51
-0.92
FID
0.68
0.72
-0.92
-0.76
-0.41
-0.92
DCI
-0.92
Vendi
Vendi
FID
DCI
Cars3D
Teapots
a
0.98
0.91
0.36
0.96
0.96
-0.69
Vendi
Vendi
FID
0.91
0.81
0.05
FID
0.96
0.86
-0.84
DO
0.36
0.53
0.05
DCI
-0.69
-0.5
-0.84
Vendi
FID
Vendi-
FID
DCt
Celeba
a
0.94
-0.92
Vendi
FID
0.8
0.53
-0.92
DCI
-0.92
-0.74
-0.92
Vendi
FID
DCI
Figure 20: Correlations between a, diversity (Vendi score), generation faithfulness (FID score), and disentanglement (DCI). Correlations are computed using the results from all models across different seeds.
27


Reviews:
[{"review_id": "27253aSOSj", "reviewer": "Reviewer_wnca", "summary": "The paper proposes a new method for learning disentangled representations, alpha-TCVAE. \nIt is based on a new lower bound for TC loss, that convexly combines VIB and CEB terms.\nFor alpha = 0, the alpha-TCVAE reduces to beta-VAE.\nExperiments show that alpha-TCVAE brings some improvement in disentanglement quality measures and diversity of generated samples. \nComparisons with beta-TCVAE, beta-TCVAE, FactorVAE, beta-VAE+HFS, VAE and StyleGAN are provided.\nAn additional experiment shows usefullness of alpha-TCVAE for reinforcement learning.", "questions": "1. In Figure 2, what does green horizontal line mean?\n2. In Figure 3, you write that \"The scores for the images of our model, α-TCVAE, are consistently better than baseline VAE models (lower FID is better), and only slightly worse than StyleGAN.\"\nBut from Figure 3, I see that blue circles (sampling from \"noise\") for all VAE-based models have significantly higher FID than StyleGAN. \nProbably, you compare traversal from VAE and \"noise\" samples from StyleGAN, but it seems not fair to me.\n3. How the matrix for the Vendi score is formed?\n4. FID from \"traversals\" image generation is much lower than from straightforward generation from noise. Do you have an explanation?\nAlso, it is interesting to compare VAE models via Precision/Recall (Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., & Aila, T. (2019). Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32.)\n\n**post rebuttal**. Most of my questions have been addressed, authors even did some additional computations. So, I am raising my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. A new approximation to the TC loss is provided.\n2. In Appendix E, authors show that α-TCVAE can learn and represent generative factors that are not present in the ground-truth dataset. This is a very surprising observation, probably authors shoud consider making it more visible by moving it to the main part of the manuscript.\n3. Have you used augmentations during training? Because new factors of variations (like camera elevation) mights be side effect of some augmentations like crop. \n4. A comparison with the recent model β-VAE+HFS is provided.\n5. The paper contains interesting insights about connections between disentanglement and diversity.\n6. An application to RL is provided.\n7. The paper is well written and easy to follow, the language is fine.", "weaknesses": "1. The improvements over other VAE-based methods are very moderate, sometimes lower than std.\n2. The popular dSprites dataset is missing; also popular measures like MIG, DCI-C, DCI-I are missing.\n3. The term Iθ(z, x) is not defined in Section 3.\n4. The experimental setting, when a diversity is evaluated by doing traversals by +-6,8,10 std. looks strange for me.\nSuch points will be never sampled from the noise. \n5. Where did hyperparameters from Appendix B come from? For example, Roth et al, 2023 provided a grid search for them. As far as rival methods are concerned, you can take hyperparameters from original papers (but in this case original training pipeline and architectures must be also used). If the hyperparameters for rival methods are selected arbitrary, experimental results are not valid!\n6. Figure 1 is not convincing. For example, row 3 for alpha-TCVAE, contains an image with a defect: non-round violet border. \nHigh diversity obviously can be achieved by generating wrong images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cT5mrWm4mc", "reviewer": "Reviewer_hWoC", "summary": "In this paper, the author introduce α-TCVAE, a variational autoencoder optimized using a new total correlation (TC) lower bound that both maximizes disentanglement and latent variables informativeness. The new TC bound is grounded in information theory and can be reduced to a convex combination of the known VIB and CEB terms. Moreover, the paper also presents quantitative analyses and correlation studies which support the idea that smaller latent domains lead to better generative capabilities and diversity.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The total correlation between x and z has not been previously used in disentangled representation learning, although similar concepts exist, such as mutual information in InfoGAN.\n- The paper  first paper propose to discuss diversity in the context of disentangled representation learning, drawing from previous research in GANs (e.g., FID).\n- The paper considers reinforcement learning as a downstream task for disentangled representations.", "weaknesses": "- The paper proposes maximizing an upper bound of the correlation between x and z for disentanglement, but β-TCVAE penalizes total correlation between different z's. The connection between these two approaches is unclear, and the use of α-TCVAE may be misleading. If there is a connection, it should be further clarified.\n- The authors should explain why maximizing the upper bound of the correlation between x and z promotes disentanglement, and why increasing variable informativeness enhances diversity.\n- The traversal results shown in Appendix Figure 12 are poor in terms of both disentanglement and generation quality. Additionally, the paper lacks traversal images results for real world natural images (not MPI3D-real), such as CelebA.\n- The paper should provide an analysis of the sensitivity of the $\\alpha$ parameter.\n- If the answer to weakness 1 is the latter, it seems that this work is relatively incremental, as it only adds a conditional TC term to the β-TCVAE. Considering this point, the authors should clarify it in the rebuttal stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DfPb7MjemU", "reviewer": "Reviewer_4hmE", "summary": "From the perspective of information theory, this paper decomposes the Total Correlation term into two distinct bounds: the information bottleneck and the conditional entropy bottleneck. It assigns specific weights to these bounds to balance their influence. Furthermore, the authors evaluate their learning objective across multiple datasets.", "questions": "- How does the parameter alpha influence the results?\n- What about the traversal outcomes for the CelebA dataset?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- From an information theory perspective, the authors derive a lower bound of total correlation. They simultaneously address both the disentanglement and informativeness of the latent variable.\n- The writing throughout the paper is lucid and well-structured, making it accessible for readers.\n- The research showcases effectiveness by employing diverse evaluation metrics and testing their approach across various datasets.", "weaknesses": "- The novelty is limited. Earlier research, referenced by the authors, already explored the decomposition of Total Correlation. The primary contribution seems to be the weighted combination of these two decompositions, which might appear incremental.\n- They claim that the VIB term promotes compression of the latent representation, and the CEB term promotes balance between the information contained in each latent dimension. However, the paper lacks a clear illustration of:\n  - How the information bottleneck correlates with disentanglement.\n  - The connection between diversity and the conditional entropy bottleneck.\n- A more profound point of contention is the origin of the VIB and the CEB terms. Both terms are derived from the same TC term, yet they supposedly represent different characteristics. The paper doesn't sufficiently elucidate why this is the case, which could have made for a compelling discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)