Disentanglement of Correlated Factors via Hausdorff Factorized Support
Authors: Karsten Roth
Mark Ibrahim
Zeynep Akata
Pascal Vincent
Diane Bouchacourt
Venue: ICLR 2023 poster (2023)
Keywords: disentanglement
representation learning
generalization

Abstract: A grand goal in deep learning research is to learn representations capable of generalizing across distribution shifts.
Disentanglement is one promising direction aimed at aligning a model's representation with the underlying factors generating the data (e.g. color or background). Existing disentanglement methods, however, rely on an often unrealistic assumption: that factors are statistically independent. In reality, factors (like object color and shape) are correlated. To address this limitation, we consider the use of a relaxed disentanglement criterion -- the Hausdorff Factorized Support (HFS) criterion -- that encourages only pairwise factorized support, rather than a factorial distribution, by minimizing a Hausdorff distance. This allows for arbitrary distributions of the factors over their support, including correlations between them. We show that the use of HFS consistently facilitates disentanglement and recovery of ground-truth factors across a variety of correlation settings and benchmarks, even under severe training correlations and correlation shifts, with in parts over +60% in relative improvement over existing disentanglement methods. In addition, we find that leveraging HFS for representation learning can even facilitate transfer to downstream tasks such as classification under distribution shifts. We hope our original approach and positive empirical results inspire further progress on the open problem of robust generalization. Code available at https://github.com/facebookresearch/disentangling-correlated-factors.

Published as a conference paper at ICLR 2023
DISENTANGLEMENT OF CORRELATED FACTORS VIA HAUSDORFF FACTORIZED SUPPORT
Karsten Roth Mark Ibrahim², Zeynep Akata Pascal Vincent Diane Bouchacourt
University of Tübingen, ELLIS Meta AI, FAIR "MPI for Intelligent Systems MILLA, Université de Montréal/DIRO, CIFAR
Primary contact: Karsten. roth hww.::n.......uumm--uu..........
ABSTRACT
A grand goal in deep learning research is to learn representations capable of gen- eralizing across distribution shifts. Disentanglement is one promising direction aimed at aligning a model's representation with the underlying factors generating the data (e.g. color or background). Existing disentanglement methods, how- ever, rely on an often unrealistic assumption: that factors are statistically indepen- dent. In reality, factors (like object color and shape) correlated. To address this limitation, we consider the use of a relaxed disentanglement criterion - the Hausdorff Factorized Support (HFS) criterion that encourages only pairwise factorized support, rather than factorial distribution, by minimizing a Hausdorff distance. This allows for arbitrary distributions of the factors over their support, including correlations between them. We show that the use of HFS consistently facilitates disentanglement and recovery ground-trufe -auuh factors across a variety of correlation settings and benchmarks, even under severe training correlations and correlation shifts, with parts over +60% relative improvement over existing disentanglemed methods. In addition, we find that leveraging HFS for representa- tion learning can even facilitate transfer downstream tasks such as classification under distribution shifts. We hope original approach and positive empirical re- sults inspire further progress on the open problem of robust generalization. Code available http://www...cmcccccoooeeeerarae www.reeccccccccccennniioooonreae wewrr--aeeeo--fattttooo
INTRODUCTION
Z2
Real data is correlated!
Disentangled representation learning (Bengio et al., 2013; Higgins al., 2018) is promising path facilitate reliable generalization to in- and out-of- distribution downstream tasks (Bengio et al., 2013; Higgins et al., 2018; Milbich et al., 2020; Dittadi al., 2021; Horan al., 2021), on top of being more interpretable and fair (Locatello et al., 2019a;
Rano possible
Landscape
Sand
Raw, bus paodile
Grass
Träuble et al., 2021). While Higgins et al. (2018) propose formal definition based on group equiv- ariance, and various metrics have been proposed to measure disentanglement (Higgins al., 2017; Chen et al., 2018; Eastwood & Williams, 2018) the most commonly understood definition is follows:
Cow L-
Camel hi
Z1
Animal
Figure 1: Real data exhibits correlations be- tween generative factors: cows are likely grass, camels on sand. This contradicts disentanglement methods assuming statisti- cally independent factors. Instead, we show that merely assuming and aiming for fac- torized support can yield robust disentangle- ment even under correlated factors.
Definition 1.1 (Disentanglement) Assuming data generated by a set of unknown ground-truth latent factors, representation is said be disentangled if there exists a one-to-one correspondence between each factor and dimension of the representation.
Work done during an internship at Meta AI, FAIR.
1 Equal contribution

---

Published as a conference paper at ICLR 2023
The method by which to achieve this goal however, remains an open research question. Weak and semi-supervised settings, e.g. using data pairs or auxiliary variables, can provably offer disentan- glement (Bouchacourt et al., 2018; Locatello et al., 2020b; Khemakhem et al., 2020; Klindt et al., 2021). But fully unsupervised disentanglement - our focus in this study - is in theory impossible to achieve in the general unconstrained nonlinear case (Hyvärinen & Pajunen, 1999; Locatello al,, 2019b). In practice however the inductive biases embodied in common autoencoder architectures allow for effective practical disentanglement (Rolinek et al., 2019). Perhaps more problematic, stan- dard unsupervised disentanglement methods (s.a. Higgins et al. (2017); Kim & Mnih (2018); Chen et al. (2018)) rely on an unrealistic assumption of statistical independence of ground truth factors. Real data however contains correlations (Träuble et al., 2021). Even with well defined factors (s.a. shape, color or background), correlations are pervasive yeellw bananas are more frequent than red ones; cows more often on grass than sand. In more realistic settings with correlations, prior work (c.g. Träuble et al. (2021); Dittadi et al. (2021)) has shown existing disentanglement methods to fail.
To address this limitation, we propose to relax the unrealistic assumption of statistical independence of factors (i.e. that they have a factorial distribution), and only assume the (bounded) support of the factors' distribution factorizes dataset of animal images (Fig. 1). background and animal are heavily correlated (camels most likely on sand, cows on grass), resulting in most datapoints being distributed along the diagonal as opposed to uniformly. Under the original assumption of factor independence, a model likely learns a shortcut solution where animal and landscape share the same latent correspondence (Beery et al., 2018).
much weaker but more realistic constraint. For example, in
On the other hand with a factorized support, learned factors should be such that any combination of their values has some grounding in reality: a cow on sand is an unlikely, yet not impossible combination. We still rely. just as standard unsupervised disentanglement methods, on the inductive bias of encoder-decoder architectures to recover factors (Rolinek et al., 2019). However, we expect our method to facilitate robustness to any distribution shifts within the support (Träuble et al., 2021; Dittadi et al., 2021), as it makes no assumptions on the distribution beyond its factorized support. We arrived at this factorized support principle from the perspective of relaxing the independence assumption to be robust to factor correlations, while remaining agnostic to how they may arise. Remarkably, the same principle was derived independently in Wang & Jordan (2021)" from a causal perspective and formal definition of causal disentanglement (Suter et al., 2019), that explicits how factor correlations can arise. To ensure computationally tractable and efficient criterion even with many factors, we further relax the full factorized support assumption to that of only a pairwise factorized support, i.e. factorized support for all pairs of factors. On this basis, we propose concrete pairwise Hausdorff Factorized Support (HFS) training criterion to disentangle correlated factors, by aiming for all pairs of latents to have a factorized support. Specifically we encourage a factorized support by minimizing a Hausdorff set-distance between the finite sample approximation of the actual support and its factorization (Huttenlocher et al., 1993; Rockafellar & Wets, 1998).
Across large-scale experiments on standard disentanglement benchmarks and novel extensions with correlated factors, HFS consistently facilitates disentanglement. We also show that HFS can be implemented regularizer for other methods to reliably improve disentanglement, up to +61% in disentanglement performance over baselines as measured by DCI-D (Eastwood & Williams, 2018) 164.1, Tab. 1). On downstream classification tasks, we improve generalization to more severe distri- bution shifts and sample efficiency ($4.2, Fig. 2). To summarize our contributions:
[1] We motivate and investigate a principle for learning disentangled representations under corre- lated factors: we relax the assumption of statistically independent factors into that of a factorized support only (independently also derived in Wang & Jordan (2021) from a causal perspective), and further relax it to a more practical pairwise factorized support.
[2] We develop a concrete training criterion through a pairwise Hausdorff distance term, which can also be combined with existing disentanglement methods ($2.3).
[3] Extensive experiments on three main benchmarks and up to 14 increasingly difficult correlations settings over more than 20k models, show HFS systematically improving disentanglement (as mea- sured by DCI-D) by up to +61% over standard methods
CTTCC///attr//Aneaaaaee VAF, c.f. $4.1). )). [4] We show that HFS improves robustness to factor distribution shifts between train and test over disentanglement baselines on classification tasks by up to +28%, as well as sample efficiency.
We were initially not aware of this work, whose preprint predates ours. We consider it strong positive sign when the same principle (of support factorization) is arrived at independently from two quite different angles (causality versus relaxed factor independence assumption).
2

---

Published as a conference paper at ICLR 2023
2 PROPOSED APPROACH
2.1 DISENTANGLEMENT VERSUS INDEPENDENCE
We are given a dataset D {x |i=1 (e.g. images), where each x' is realization of random variable, e.g., an image. We consider that each x' is generated by an unknown generative process, involving a ground truth latent random vector z whose components correspond to the dataset's un- derlying factors of variations (s.a. object shape, color, background, ...). This process generates an observation x, by first drawing a realization z = (Zı..... . zx) from a distribution p(z), i.e. & ~ p(z). Observation x is then obtained by drawing
p(x'z). Given D, the goal of disentangled repre- sentation learning can be stated as learning a mapping fo that for any x recovers as best as possible the associated i.c. (x) Re E[z[x] up to permutation of elements and elementwise bijective transformation. In unsupervised disentanglement, the z are unobserved, and both p(z) and p 2 a priori unknown us, though we might assume specific properties and functional forms. Most unsupervised disentanglement methods follow the formalization of VAEs and employ parameterized probabilistic generative models of the form po(x, 2) pe pe(z) (z)po'' (x z) estimate the ground truth generative model over z, x. As in VAEs, these methods make the strong assumption that ground truth factors are statistically independent:
p(z) ((=1)p((2))- p(zx))
and conflate the goal of learning disentangled representation with that of learning representation with statically independent components. This assumption naturally translates to a factorial model prior (z). Successful variants of VAE for disentanglement (Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018) further modify the original VAE objective to even more strongly enforce elementwise independence of the aggregate posterior (i.e. the encoder outputs) than afforded by the VAE'S optimized evidence lower bound. However, as explained in the introduction, the assumption of factor independence clearly doesn't 't hold for realistic data distributions. Consequently, methods that enforce this unrealistic assumption suffer from that discrepancy, as shown in Träuble et al. (2021); Dittadi et al. (2021) and confirmed in our own experiments. To address this shortcoming. we develop novel method to relax the unrealistic assumption of factor independence.
2.2 RELAXING THE INDEPENDENCE ASSUMPTION INTO THAT OF FACTORIZED SUPPORT
Instead of assuming independent factors (i.e. a factorial distribution on z as in Eq, 1) we will only assume that the support of the distribution factorizes. Let denote by Sp((z) the support of p(z), ie. the set {z E Z p(z) > 0}. We say that S(p(z)) is factorized if equals to the Cartesian product of supports over individual dimensions' marginals, i.e. if:
S(p(z)) = S(p(21)) x S(p(22))
S(p(z+)) def S (p(z))
where x denotes the Cartesian product. Of course, 1 (independence) = 2 (factorized- -uupoot))))) but 2 (factorized-supporr)) + (independence). Assuming a factorized support thus relaxation of the (unrealistic) assumption of factorial distribution, (i.e. of statistical independence) of disentangled factors. Refer to the cartoon example in Fig. 1, where the distribution of the two disentangled factors would not satisfy independence assumption, but does have factorized support. Informally the factorized support assumption is merely stating that whatever values 21 and 22, etc... may take individually, any combination of these is possible (even when not very likely). In the next section we will develop a concrete training criterion that encourages the obtained latent representation to have a factorized support rather than a factorial distribution.
2.3 A PRACTICAL CRITERION FOR FACTORIZED SUPPORT
Based on our relaxed hypothesis, we now define concrete training criterion that encourages factorized support. Let us consider deterministic representations obtained by the encoder z = (x). We enforce the factorial support criterion on the aggregate distribution (z(z) [x)], where (z) is conceptually similar to the aggregate posterior (z) ) in e.g. TCVAE, though we consider points produced by deterministic mapping fø rather than stochastic one. To match our factorized support assumption on the ground truth we want to encourage the support of q,) ( to factorize, i.e. that S(qs(z) and the Cartesian product of each dimension support, S (go (z)), are equal. For
3

---

Published as a conference paper at ICLR 2023
clarity shorthand notations S and 8 to denote S* (z) and (z)) respectively when it is clear from context. To guide the learning, we thus need a divergence or metric to tell us how far S is from S* Supports are sets, it is natural to set distance such the Hausdorff distance.
Hausdorff distance between sets Given a base distance metric d(z, z') between any two points in Z (e.g. the Euclidean metric in 2 = RR*,, the Hausdorff Distance between sets (here, S and S), is then defined as
(H( max xES
sup
inf d(z, d(z,z') E'
sup ZES L''ESk
inf d(z, x()
sup =ES*
inf d(z, x(x x'EE
(3)
with the second part of the Hausdorff distance equating to zero since S S*
Monte-Carlo Hausdorff Distance Estimation In practice only have a finite sample of obser- vations {x) and can only estimate the support and Hausdorff distances from the finite number of representations (x)), We thus introduce a practical Monte-Carle saach-approximationn : with access to batch of b imputs X yielding k-dimensional latent representations Z (x) E RR estimate Hausdorff distances using sample-based approximations to the support: S Z and
S ~ Z X Z x x Z., = ((21,...,z,)) 21 E Z.,,,.., E Z, )- Here Z,5 must be understood as the set (not vector) of all elements in the jth column of Z. Plugging into Eq. 3 yields:
d (2) max xZ.,xx...x........ xZ,, [min Ez d(z, z']
(4)
where by noting z' E Z we consider the matrix Z as a set of rows, over which we find the min?.
Further relaxing the assumption to pairwise factorization In high dimension, with many fac- tors, the assumption that every combination of all latent values is possible might still be too strong an assumption. And even if we assumed all to be in principle possible, we can never hope observe all in a finite dataset of realistic size due to the combinatorial explosion of conceivable combina- tions. However, it is statistically reasonable expect evidence of factorized support for all pairs of elements². To encourage such a pairwise factorized support, we can minimize sliced/pairwist Hausdorff estimate with the additional benefit of keeping computation tractable when le is large d
(Z)
Σド Σド m ZZ., [ming'ez.. d(z, d(z,z')
(5)
where (i.)) denotes the concatenation of column and column j. yielding set of rows.
Avoiding collapse and retaining input information We will be learning representations z (x( by learning parameters Φ that optimize training objective. Because the Hausdorff distance builds on base distance d(z, zz) if we were to minimize only this, it could be trivially minimized to by collapsing all representations to single point. Avoiding this can be achieved in several ways, s.a. by including a term that encourages the variance of Z.; to be above 1 (a technique used e.g. in self-supervised learning method VICReg (Bardes et al., 2022)) or more in line with traditional VAE variants for disentanglement by using a stochastic autoencoder (SAE) reconstruction error:
εSAE xx: o, 0) = (x))())x) [log (x|[z)]
(6)
with mean given by our deterministic mapping fa- [x(x) producing diagonal covariance parameter, and e.g. (x)2)(((2z) ||ra(z) z||² with ra a parameterized decoder. The autoencoder term ensures representations (x) retaining as much information possible about x for reconstruction, preventing collapse of representations single point. A minimum scale can also be ensured by imposing E (x) to be above a minimal threshold.
where typically (z/x) (x)
2.4 PUTTING IT ALL TOGETHERR
Our basic training objective for Hausdorff-based Factored Support (HFS) can thus be formed by simply combining the stochastic auto-encoder loss of Eq. 6 and our Hausdorff estimate of Eq. 5:
CHES (D; CuFs(D;oo)) Ex (d ((2) (X)) + I ExEX (SAB(x; o, 0)
(7)
One alternatively use softened max and min operations, as defined in Appendix A.4. In practice, Saw no robustness benefit to this, likely because we compute dx over batches, not the entire dataset.
"ttraightforward to generalize larger tplles, but computational and statistical benefits shrink accordingly.
4

---

Published as a conference paper at ICLR 2023
where X 2 D denotes a batch of b imputs, (X) the batch representations Z, and y the trade- off between the Hausdorff and SAE terms. To compare with existing VAE-based disentanglement methods s.a. (-VAE (Higgins et al., 2017), we can also use Eq. 5 as regularizer on top:
B (D; = Ex2D [d
1 (Z) + + Exxx (LSAE(X; o, 0) + 3 (90l
(8)
where DKL is the Kullback-Leiin
divergence, and (z(z) the usual VAE factorial unit Gaussian
prior. This hybrid objective recovers the original 3- VAE with y = 0, and Luss (Eq. 7) with / = 0, showing that the plain HFS objective replaces the 3-VAE KL term by encouraging Hausdorff term and removes the factorial prior p(z). We can similarly extended other VAB-based variants (Chen al., 2018; Kim & Mnih, 2018; Burgess et al., 2018) by adding our Hausdorff term as regularizer to focus more on its support than a precise factorial distribution.
factorized-support
3 RELATED WORK
Disentangled Representation Learning aims to recover representation spaces where each ground- truth generative factor is encoded in unique entry subspace (Bengio et al., 2013; Higgins al., 2018) to benefit subsequent downstream transfer (Bengio et al., 2013; Peters et al., 2017; Tschannen et al., 2018; Locatello al., 2019b; Montero et al., 2021; Mancini et al., 2021; Roth et al., 2020; Funke et al., 2022), interpretability (Chen et al., 2016; Esser et al., 2018; Niemeyer & Geiger, 2021) and fairess (Locatello et al., 2019a; Träuble al., 2021: Dullerud al., 2022) via compositionality of representations. Methods often rely on Variational AutoEncoders (VAEs) variants (Kingma & Welling, 2014; Rezende et al., 2014) constrain the (aggregate) posterior of the encoder, e.g. via penalties on the bottleneck capacity (3- VAE (Higgins et al., 2017)) with progressive constraints or network growing (AnnealedVAE (Burgess et al., 2018), 8), ProVAE (Li et al., 2020)), the total corre- lation (8-TCVAE (Chen et al., 2018), 8), Factor (Kim & Mnih, 2018)) or the mismatch to some factorized prior (DIP-VAR VAE (Kumar et al., 2018), Dou (Mita et al., 2020)). These approaches assume statistically independent factors, which is invalid for realistic data motivated in $1.
Disentanglement under correlated factors. Consequently, while most methods have been shown to perform well on toy datasets and ones with known independent factors such Shapes3D (Kim & Mnih, 2018), MPI3D (Gondal et al., 2019), DSprites (Higgins et al., 2017), SmallNorb (LeCun et 2004) Cars3D (Reed et al., 2015),011,,,e,) recent research (Montero et al., 2021; Träuble et al., 2021; Montero et al., 2022; Funke et al., 2022; Dittadi et al., 2021) has started to connect these setups to more realistic scenarios with factor correlations: Träuble al. (2021) introduce artificial correla- tions between two factors, and Montero et al. (2021) exclude value combinations for recombination studies. In such settings, Montero et al. (2021; 2022); Träuble et al. (2021); Dittadi al. (2021) show that unsupervised disentanglement methods that assume independent factors fail to disentan- gle, with potentially negative impact to OOD generalization. Suter et al. (2019) propose a causal metric to evaluate disentanglement when assuming confounders between the ground-truth factors. Choi et al. (2020) introduce a Gaussian mixture model for dependencies between continuous and discrete variables in a structured setup with number of mixtures known. By contrast, we investigate generic remedy without explicit auxiliary variables or prior models by relaxing the independence as- sumption to only a pairwise factorized support. Pfau et al. (2020) propose geometrically motivated non-parametric unsupervised disentanglement following the symmetry- based definition in Higgins al. (2018) by leveraging holonomy of manifold geometries as learning signal to find disentangled subspaces. This does not assume statistical independence, but requires non-trivial holonomy for each factor manifold, and struggles in high-dimensional spaces and generalization new data. For domain adaptation shifts, Tong et al. (2022) propose adversarial support matching, highlighting that operating on the support can be beneficial for related settings as well. To evaluate disentanglement, we utilize DCI-D (part of DCI - Disentanglement, Completeness. Informativeness, see Eastwood & Williams (2018)) as leading metric. As opposed to other metrics s.a. Beta-/FactorVAE scores (Hig- gins et al., 2017; Kim & Mnih, 201 8), MI Gap (Chen et al., 2018) 8), Modularity (Ridgeway & Mozer, 2018) or SAP (Kumar et al., 2018), Locatello et al. (2020a); Dittadi et al. (2021) have indicated DCI- D as the potentially most suitable disentanglement metric (and as also done e.g. in Locatello et al. (2019b; 2020b); Träuble et al. (2021)), with generally strong correlation between metrics (Locatello et al., 2019b). Finally, Wang & Jordan (2021) independently also arrived the idea of support fac- torization for disentanglement from a causal perspective, for which they propose a similar Hausdorff distance objective, providing orthogonal validation to support factorization for disentanglement. On
5

---

Published as a conference paper at ICLR 2023
the contrary, we derive it from relaxing the assumption of factor independence, and propose fur- ther pairwise relaxation, which performs and scales much better (see Supp. A), alongside much more expansive experimental study on the impact downstream disentanglement, adaptation and generalization under various correlation shifts.
4 EXPERIMENTS
We start with experimental details listed below, before studying HFS on benchmarks with and without training correlations ($4.1). These results are extended in $4.2 evaluate the transfer and downstream adaptability ($4.3) of learned representations across different correlation shifts and link HFS during training to various downstream metries ($4.4). We include variant, qualitative and hyperparameter robustness studies in appendix $B. SE and $D all favouring our HFS objective. Across experiments, we re-implemented baselines ( VAE (Higgins al., 2017), Factor (Kim & Mnih, 2018 8), Annealed (Burgess et al., 2018), 8-TCVAE (Chen et al., 201 8)) as done e.g., in Locatello et al. (2019h; 2020b); Träuble et al. (2021). To investigate methods under correlated ground truth factors, we use and extend the correlation framework introduced in Träuble et al. (2021) who introduce correlation between pairs of factors as p(21, 22) x exp (-(21 - f(z2))² where higher notes weaker correlation between normalized factors 21 and 22, and f(z) 2 or [()) 1 for inverted correlations when necessary We extend this framework to include correlations between multiple factor pairs (either 1, 2 or pairs) and shared confounders (one factor computed
correlated to all others). All reported numbers
at least 6 seeds (with 2 10 seeds used for key experiments s.a. Tab. I or Fig. 2). Similar to existing literature (Locatello et al., 2019b; 2020b; Träuble et al., 2021; Dittadi et al., 2021) we cover at least 7 hyperparameter settings for each baseline. Further experimental details are provided in SH.
4.1 FACTORIZATION OF SUPPORTS FOR DISENTANGLENT ON STANDARD BENCHMARK
We study the behaviour of HFS and baselines on standard disentanglement learning benchmarks and correlated variants thereof (see $4) Shapes31D (Kim & Mnih, 2018), MPI3D Gondal et al. (2019) and DSprites (Higgins et al., 2017). For each setting, we report results averaged over 2 10 seeds in Tab. 1. Each column denotes the test performance on a particular correlation setting. As DSprites only has five effective factors of variation, no three- pair setting is possible. Values reported denote median DCI-D with 25th and 75th percentiles in grey. Our results indicate that a factorization of the support via HFS encourages disentanglement (as measured via DCI-D)) without relying on factorial distribution objective (s.a. 33-AE and its variants), consistently matching or outperforming the comparable 3-VAE setting both when no correlation is encountered during training ( ("No Corr.) as well for much more severe correlations ("Conf.", Even more, we find that extending existing disentanglement objectives s.a. 38VAE (or stronger extensions like 3-TCVAE) with explicit factorization of the support (+HFS) can provide even further, significant improvements. For example, without correlations we find relative improve- ments of nearly- +30%, while for some correlated settings, e.g. with a shared confounder, these go over +60%! In addition, relative increases of up +140% on 3-TCVAE further highlight both the general importance of an explicit factorization of the support for disentanglement even under training correlations, well as it being property generally neglected until now.
uncorrelated data for all methods trained
4.2
GENERALIZATION UNDER CORRELATION SHIFTS
As 3-VAB and 8-VAE + HFS models in $4.1 were trained on correlated and evaluated on uncor- related data, the performance differences provide a first indication that encouraging factorized support can benefit transfer under correlation shifts. Such changes in correlation from train to test data is commonly referred to as "distributional shift" for the model, and mark key issue interfering with generalization in realistic settings (Arjovsky et al., 2019; Koh et al., 2021; Milbich et al., 2021; Roth et al., 2022; Funke et al., 2022). While some works point to initial benefits of disentangled representations for distrii-itiiiioo (OOD) generalization (e.g. Träuble et al. (2021); Dittadi et al. (2021)). some have raised concerns about the gains from disentanglement e.g. OOD recombination (Montero et al., 2022). Conceptually, one way a disentangled representation can benefit a downstream prediction task is when the true predictor is function of only subset of the true factors. Successful recovery
Siiinonero-Candee et al., 2009) as the test
data becomes out-of-distribun

---

Published as a conference paper at ICLR 2023
Table 1: Disentanglement by explicitly factorizing the support using HFS on 3 benchmarks across various numbers of correlated factors (columns) and correlation increasing from left (no correla- tion) to right (every factor correlated to one confounder; for DSprites. three pairs are impossible, see text). Scores denote DCI-D metric computed on uncorrelated test data. (Bold) blue denotes (sec- ond) best performance per benchmark/correlation. [a. b] indicate 25/75th percentiles. The results show that relaxing the goal of a factorial latent distribution to factorized support with standalone HFS already offers competitive disentanglement. Adding HFS as regularizer over standard meth- ods (3-VAE/TCVAE) to target more factorized support yields even higher scores, beating other approaches with optimally tuned hyperparameters s.a. B. Remarkably on MPI3D, optimal tuning turned B and other TCVAE terms 0, leaving only HFS which consistently worked best.
PAIRS: 1 o 7=0. 0.1
PAIRS: 2 a = 0.1
PAIRS: 3 a= 0.1
SHARED CONE.
METHOD
No CORR.
a=
Shapes3D (Kim & Mnih, 2018)
B-VAE
70.7 [65.2, 75.4]
71.6 [60.9, 72.5]
55.6 [45.8, 57.2]
37.1
[32.3, 38.9]
38.0 [35.0, 38.6]
HFS
78.3 [75.1, 83.6]
77.8 [74.4, 78.8]
56.0 [41.0, 57.4]
47.5
[38.0, 49.1]
46.2 (3.53,47.5
B-VAE + HFS
91.2 [75.8, 100.0]
80.9 [76.4, 81.4)
67.6 [62.4, 69.3]
47.9
[44.0, 50.8]
63.5 [61.2, 65.5)
5-TCVAE
77.1 [76.6, 78.3)
71.1 [65.5, 72.5]
63.8 [59.1, 65.1)
47.3
[36.7, 50.0]
49.9 (45.8, 55.9]
3-TCVAE + HFS
85.7 [82.5, 97.3]
75.2 (63.1, 76.6)
68.3 [61.0, 71.8)
51.6
[47.7, 52.8]
61.5 [53.8, 64.2]
Factor VAE
66.1 [51.2, 69.1]
70.8 [70.5,
57.2 [55.9. 62.0]
46.8
[40.8, 49.0]
31.6 22.99,35.11
AnnealedVAE
62.2 [60.7, 63.2]
57.2 (49.5,
31,6 (26.9, 34.1]
33.6
[31.0, 38.0]
23.0 [20.1, 25.91
DSprites (Higgins et al., 2017)
3-VAE
32.2 [25.3, 37.9]
9.5 [7.9. 10.3]
7.5 [6.7, 8.3]
N/A
11.4 [9.9, 13.91
HFS
34.9 [27.4, 36.0]
13.6 [7.6, 16.7]
11.9 [9.7. 13.8]
N/A
15.1 [11.0, 16.0]
3-VAE + HFS
49.9 [30.0, 50.4]
19.7 [17.0, 21.1]
17.3 [6.0, 19.6]
N/A
15.8 [12.3, 16.7]
T-VCABB
30.9 [28.9, 35.2]
24.0 [23.6, 24.4]
11.4 (7.6, 13.6]
N/A
20.9 [17.5, 23.6]
TCVAA HFS
53.1 [41.8,
26.5 [25.6, 27.2]
27.8 (1.61,
N/A
24.8 [23.8, 26.3]
Fac
25.7 [20.9, 30.9]
15.1 [11.9, 16.3)
13.4 [12.4,
N/A
14.7 [13.5, 15.3)
AnnealedVAE
39.4
[38.7, 40.0)
14.8 [14.3, 15.9)
8.5 [6.9, 10.3]
N/A
14.3 [14.1, 14.5)
MPI3D (Gondal et
2019)
3-VAE
25.6 [24.7, 26.1]
20.5 [17.7, 20.91
23.6 (22.6,
11.6 (11.,,11
11.8 [10.0, 12.7]
HFS
32.8 (30.0, 34.3]
28.4 [26.5, 29.5]
28.0 [27.4, 28.21
14.3 [13.1, 14.8]
16.1 [15.0. 16.6]
B-VAE+ HFS
32.8 [30.0, 34.3]
28.4 [26.5, 29.5]
28.0
277.4,28.2)
14.3 [13.1, 14.8]
16.1 [15.0, 16.6]
3-TCVAE
26.6 [26.0, 27.4]
20.7 (2..4.21.3)
23.3
23.8]
11.4 [10.3. 12.6]
14.2 [13.4, 15.4]
3-TCVAE + HFS
32.8 [30.0, 34.3]
28.4 [26.5, 29.5]
28.0 [27.4, 28.2]
14.3 [13.1, 14.8]
16.1 [15.0, 16.6]
Factor VAE
26.0 [24.8, 27.5]
21.9 [20.1, 23.91
27.8 [27.2, 29.2]
10.9 [10.7, 11.91
13.6 (12.8, 13.91
AnnealedVAE
11.8 (10.8, 12.4]
11.7 [10.4, 12.91
11,8 (11.6, 12.1]
11.6 (10.6, 12.2]
13.4 (12.8, 13,91
of all factors, disentangled, enables effective subsequent feature selection (e.g. LI-regularized logis- tic regression or shallow decision trees). A downstream predictor can thus be far more sample efficient (Ng, 2004) in learning to ignore irrelevant factors, that may be spuriously correlated with the target, than if they were entangled in the representation. As we showed that explicit sup- port factorization provides stronger relative disentanglement, we leverage this for further insights into its benefits on OOD tasks. Given the strong performance of HFS on Shapes3D, we extend our experiments from $4.1 on this dataset with more training and now also test data correlations. This gives transfer grids (Fig. 2) across diverse, increasingly severe correlation shifts. For each grid, the y- and x-axis indicate training and test correlations increasing from top to bottom and left to right, respectively. Darker colors refer to a score increase. We use these grids to see (1) how different correlation shifts impact disentanglement, and (2) if improvements in disentanglement via explicitly aiming for factorized support impact downstream transferability of the learned representations.
(1) We first evaluate disentanglement of standard 3-VAE (leftmost grid; each square uses optimal parameters for given correlation and seed), and find an expected drop with increased correlation on the training data. The subsequent grid shows changes when adding HFS to B-VAE, with consistent improvements in disentanglement of test data over 3- VAE across all correlation shifts (only positive changes), and extends our insights from Tab. 1.
(2) To understand the usefulness for practical transfer tasks under correlation shifts, we train a Gra- dient Boosted Tree (GBT, sk 1l learn (Pedregosa et al., 2011) 1), c.f. Locatello et al. (2019b; 2020b)) to
7

---

Published as a conference paper at ICLR 2023
DCI-D B-VAE
ADCI-D with HFS
DCI-I B-VAE
ADCH- with HFS
No Carr
SHI
Uoge CO/r3 Uieal
www.
w
11.3
Com: Incr. Test Correlation
+
Figure 2: Out-of-Distributt Disentanglement and Generalization across large ranges of correla- tion shifts between train and test data on Shapes3D. We evaluate the impact of encouraging factor- ized support on disentanglement (DCI-D) and classification performance of test ground truth factors (DCI-I) via HFS. Y-axis denotes source correlations increasing from top to bottom, x-axis target correlations (left to right). Darker blue and green mean higher scores and absolute improvements, respectively. [Teffmost]:: DCI-D 3-VAE for all shifts, dropping with higher training correlations. [Left]: Consistent and in parts high improvements in DCI-D when explicitly encouraging factorized support via 3- VAE HFS across shifts. [Right]: DCI-I using a GBT over embeddings generated by a 8- VAE model trained on respective source correlations. Drop in performance with higher training correlation test data variation (bottom left corner). [Rightmost]: Absolute changes in DCI-I with HFS reveal higher generalization particularly when shifts are large (c.f. bottom-left). This shows that explicitly encouraging factorized support benefits generalization as shifts become more severe.
No Cor
Pairs:
Pairs:
Pairs:
Shared Conf.
Rederion imperatree 196)
60
Single eatry
Gratient Bootted
40
10"
10mt 10a
30
10 10 10" 10
10
13
30
30
10'
Trininn (8)
Figure 3: Increased accuracy and sample efficiency on downstream classification. We plot relative improvement (%) in average ground truth factor classification accuracy by using HFS on top of a 3-VAE, function of the amount of labeled training data. Classifier is GBT (for linear probe SF.2) receiving either the entire latent vector (black) or only the most expressive entry (blue). The increased disentanglement through HFS gives consistent improvements in all cases, and gets more pronounced in the low data regime for full latents, indicating higher sample efficiency, as expected from better disentanglement. Relative improvements up to +80% in the single entry case across correlation shifts highlight the better reflection of ground truth factors across correlations.
take representations of the test data and predict the exact ground truth factor values; reflected in the third grid for B- VAE baseline and measured by the DCI-I metric (Eastwood & Williams, 2018). We see that the downstream classification performance, while saturated for small shifts, drops notably both with increased training correlation more variation in the test data (drop towards bettom-left corner). If we now measure the change in downstream classification performance with HFS (last grid), we see that while for small shifts the benefits are small, they increase for larger ones (increase towards same bottom left corner). This indicates that changes in disentanglement through our sup- port factorization become increasingly important as distributional shifts increase, and highlight that benefits for OOD generalization drawn from improvements in disentanglement may be particularly evident for harder shifts.
4.3 BENEFITS UNDER VARYING DOWNSTREAM ADAPTATION METHODS
This section investigates how generalization improvements hold when the amount of downstream test data changes, and revisits the recovery of ground truth factors under correlation shifts by looking at the performance with only the single most important latent entry. We train GBTs (as we care about relative changes, xgboost (Chen& & Guestrin, 2016) for faster training) embeddings
8

---

Published as a conference paper at ICLR 2023
No Correlations
With Confounder
PZ Train Dala
No Correlations
140
With Confounder
0.75
100
0.4
120
0.50
75
JAV THE
0:25
50
80
10
10a
10
20 25 B-VAE: KLD-LosS
(2)
of (2) Train Data
B-VAE: KLD-LOSS
Figure 4: [Left, orange]: Increased support factorization train data measured by lower Haus- dorff estimate d() by increasing HFS weight y (Eq. 7)), improves disentanglement (DCI-D) across correlations Shapes3D (more results in Supp. Fig. 9). [Center, black]: Minimizing 3-VAE KL-Divergence by increasing 3 implicitly encourages a factorized support by pushing towards full independence, but hurts disentanglement because of the incorrect assumption. [Right]: We find strong correlation between HFS on train data and standard disentanglement metrics on test data ([%), darker higher) even under training correlations (top to bottom). Detailed figure in $F..
tracted either from an optimal 3-VAE or 3-VAE + HFS. For the single-latent-entry training, we first train a GBT select the most important entry to predict each respective ground truth factor, and then use said entry to train a second ground truth factor predictor. See Supp. $F2 for experiments with linear probes. all cases (Fig. 3), explicit support factorization via HFS facilitates downstream adaptation with particular benefits when only little data is provided at test time. For example in the standard uncorrelated setting, relative improvements increase from 4% to 45%, with similar trends across correlation shifts. Finally, our experiments reveal that increased disentanglement expectedly results in a better reflection of ground truth factors in single latent entries, shown in nearly +880% relative improvement when training and predicting the most expressive entry. These insights rein- force that explicit support factorization via HFS encourages disentanglement also under correlation shifts, and show potential benefits in downstream generalization especially in the low data regime.
4.4 FACTORIZATION AS A PERFORMANCE METRIC
We now explore the relationship of HFS to existing metrics across correlations. Utilizing Eq. 5 as separate evaluation metric for the factorization of the support across the whole training data facili- tated through increased HFS weighting y we find that when the factorization of the support across the training data goes down (Fig. 4 (left, orange)). the disentanglement on the test data consistently goes up. verifying again the connection between support factorization and disentanglement. Fig. 4 (center, black) shows 3- VAE implicitly encouraging a factorized support by pushing towards inde- pendence, but which hurts disentanglement and generalization, see experiments above. Finally, Fig. (right) shows that support factorization on the training data exhibits correlation with disentangle- ment metrics, consistent across also stronger training correlations, albeit lower. This is useful, as HFS neither requires access to ground truth factors nor a specific prior distribution over the support, and can thus serve as a proxy for development and training evaluation of future works.
CONCLUSION
To avoid the unrealistic assumption of factors independence (i.e. factorial distribution) in tradi- tional disentanglement, which stands in contrast to realistic data being correlated, we thoroughly investigate an approach that only aims at recovering a factorized support. Doing achieves disen- tanglement by ensuring the model can encode many possible combinations of generative factors in the learned latent space, while allowing for arbitrary distributions over the support in particular those with correlations. Indeed, through practical criterion using pairwise Hausdorff set-distances HHS - we show that encouraging a pairwise factorized support is sufficient to match traditional dis- entanglement methods. Furthermore we show that HFS can steer existing disentanglement methods towards a more factorized support, giving large relative improvements of over +60% on common benchmarks across a large variety of correlation shifts. We find this improvement in disentanglement across correlation shifts to be also reflected in improved oft-of--istributt generalization especially as these shifts become more severe; tackling key promise for disentangled representation learning.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To reproduce the results from this paper and avoid implementational and library-relate differences, have released our codebase here: factors.
To reproduce Tab. 1, we first refer to Tab. 5, which contains all Tab. 1 results with additional details on the exact correlations used (as well as other correlation settings). For each of the correlation settings, the associated factor correlation pairs are provided in SH.1, with the training, model as well as grid-search details all noted in SH. The correlation formula to introduce artificial correla- tions between respective factors follows the setup described in the experimental details noted the beginming of $4.
For the correlation shift transfer experiments used in $4.2, the same training and correlation settings are used. For our downstream adaptability results, we provide all relevant details in $4.3 and SH.
ACKNOWLEDGEMENT
The authors would like thank Léon Bottou for useful and encouraging early discussions, well as David Lopez-Paz, Mike Rabbat, and Badr Youbi Idrissi for their careful reading and feedback that helped improve the paper. We also want to extend special thanks to Kartik Ahuja for later making us aware of the work of Wang & Jordan (2021) and its close connection with our approach. Karsten Roth thanks the International Max Planck Research School for Intelligent Systems (IMPRSIS) and the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. Zeynep Akata acknowledges partial funding by the ERC (853489 DEXIM) and DFG (2064/1 Project number 390727645) under Germany's Excellence Strategy.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2019. URL /ectts://arxte org/ abs / 02873.293.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varianc-iimariine--ccvaaaa regulariza- tion for self-supervised learning. International Conference on Learning Representations, 2022. URL
net.twiwwweeeeeeeee.......eeeeee forum? x
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of
the European Conference on Computer Vision (ECCV), September 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Anulysis and Machine Intelligence,3556 (50)) 2013. doi: 10. 10..1099//M..211.......
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/aaii v22.l. URL https:/ //tp///jjj.///. aaai. org/ index..... php/ AAAI wtw.tieeeeeeeeeeeee//e/
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in beta-vae. CoRR, abs/ has//804403599, 2018. URL http::/t//// //axiii org/ abs/ 18044.18110 03599.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of dis- entanglement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. -aianhianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: http://pppeeeeeee.g..c// neurips.com 2oper/2018. /tteer//11///lee////eee
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD "16, pp. 785-794 New York, NY. USA, 2016. ACM. ISBN 978- -253---22--4222 doi: 10. 14./ 2939672.2939785. URL /ooi acm. http:/11.111//2911
10

---

Published as a conference paper at ICLR 2023
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative ad- versarial nets. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016. URL https: /proccee//e//eeeeeeeeeeeee ing. htt.:/eerrrrpp...ppppp//////// cc/papeer/201//flle hew.ba/eeeeeeeeeeeacceaaaaaaeae
haper.pdt.
Jaewoong Choi, Geonho Hwang, and Myungjoo Kang. Discond-vae: Disentangling continuous factors from the discrete. CoRR, abs/2009.08039, 2020. URL httptt////axxi.orgrggg
2009. 08039.
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wuthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer of disentangled representations in realistic settings. International Conference on Learning Representations, 2021. URL https: / //openzeview.c net wwwteeoortiee wow.mmiddd-vvviiia
Natalie Dullerud, Karsten Roth, Kimia Hamidieh, Nicolas Papernot, and Marzyeh Ghassemi. Is fairness only metric deep? evaluating and addressing subgroup gaps in deep metric learning. In International Conference Learning Representations, 2022. URL https: /oppr/eeeeee.eeeeeeeeeeeee net /ot/ffor forum? iiddiddjsse jididde-e22...ee 2366__u_DDV.
Cian Eastwood and Christopher I. Williams. framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
URL http:: /ttp:::///ppeeeeeeeeeee net/ wtwp:eiiwwweeeeeetteiiiieieee
Patrick Esser, Ekaterina Sutter, and Bjöm Ommer. variational u-net for conditional appearance and shape generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Christina M Funke, Paul Vicol, Kuan-Chieh Wang, Matthias Kuemmerer, Richard Zemel, and Matthias Bethge. Disentanglement and generalization under correlation shifts. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022. URL https: //openzeview
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Mar- tin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentan- glement dataset. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL htt.s: htt.t:sss/pproceeedingeo neuri http:/pape22211/////1 pdf.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAB: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017. URL https::/ https::/openzevierr htt.:/rrww...tttttrrr.oo.o.eeaaa
Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Lofe Matthey, Danilo J. Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. CoRR, abs/1812.02230. 2018. URL http: httttp://arxv..rrtt
abs/aalsll1 02230.
Daniella Horan, Eitan Richardson, and Yair Weiss. When is unsupervised disentanglement possi- ble? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 5150-5161. Curran Asso- ciates, Inc., 2021. URL 29586cb4 9c90e24 10044911005 211111100000000e2225aa Paper. pdf.
newreeeeeednnnn.e.eeeeeeeee.eeee cc httppssccpppppp/ppaaa..
D.P. Huttenlocher, G.A. Klanderman, and W.J. Rucklidge. Comparing images using the hausdorff distance. IEEE Transactions on Pattern Analysis and Machine Intelligence. 15(9):850-863 1993. doi: 10.1109/34.232073.
11

---

Published as a conference paper at ICLR 2023
Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural New. 12(3):429 439, apr 1999. ISSN 0893-6080. doi: 10. 00140-3.
URL https: http::://dooooorgo/o. 10. 1016 S0////00933---0 6080 (98)
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 2207-2217. PMLR, 26-28 Aug 2020. URL https: / http///oooeeeengeeee/ee mlr www.ppeesseeee...e 08 www.akeeeeknnee 0a html.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2649-2658. PMLR, 10-15 Jul 2018. URL mll. press/pees//aee /vaass/vvvvv /imllmmmmmll Bb. html.
https: https//ppcccediggge
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL CtLhttp:/// www.aaiivvvoo..................a abs/1312, 6114.
David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with tempo- ral sparse coding. In International Conference on Learning Representations, 2021. URL https: wpwnrevieeeeeeeeeeeeeeeeee net forum?21-=
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5637-5664. PMLR, 18-24 Jul 2021. URL httt::://proceedinng
wall..press
www.:.presssiiii /vtps/vv3///////// htw11////oohhee a. attohha..tll................m...
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. VARIATIONAL INFERENCE OF DISENTANGLED LATENT CONCEPTS FROM UNLABELED OBSERVATIONS. In Interna- tional Conference Learning Representations, 2018. URL https: /ttp::::////eeeeeeeeeee
forum? 1d=H1kG7GZAN
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2:114, 2004. ISSN 1063-6919. Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2004 Conference date: 27-06-2004 Through 02-07-2004.
Zhiyuan Li, Jaideep Vitthal Murkute, Prashnna Kumar Gyawali, and Linwei Wang. Progres- sive learning and disentanglement of hierarchical representations. In International Confer- ence on Learning Representations, 2020. URL / /ttp:////eeeeeeeeeeeeeeeee net /ft/ffoon forum?ic SUxpsxrYPS.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the faimess of disentangled representations. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019a. URL https:/ https::/ppcceeeings. newrips.................. cc/papper2219///1e 8ee.a55898ee8888eeee 6a Paae .df..
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.).
12

---

Published as a conference paper at ICLR 2023
Proceedings of the 36th International Conference Machine Learning. volume 97 of Pro- ceedings of Machine Learning Research, pp. 4124 PMLR, 09-15 Jun 2019b. URL
wwwpiinsss............. pww.spressssve.. www.eooaaeelllla.al...... html.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. A sober look the unsupervised learning of disentangled repre- sentations and their evaluation. J. Mach. Learn. Res., 21(1), jan 2020a. ISSN 1532-4435.
Francesco Locatello, Ben Poole. Gunnar Ractsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly Weakly-supervised disentanglement without compromises. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learn- ing. volume 119 of Proceedings of Machine Learning Research, pp. 6348 6359. PMLR, 13-18 Jul 2020b. URL /tttttt:://pooeeeeeeeeee http:///poeeennnneeee/ee//
http://111//octtell1100 .tml..
Massimilliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata. Open world compositional zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5222-5230, June 2021.
Timo Milbich, Karsten Roth, Homanga Bharadhwaj. Samarth Sinha, Yoshua Bengio, Bjöm Ommer, and Joseph Paul Cohen. Diva: Diverse visual feature aggregation for deep metric learning. In Computer Vision ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part VIII, pp. 590-607, Berlin, Heidelberg, 2020. Springer-Verlag ISBN 978-3- 030-58597-6. doi: 0000000/9778--00-58555885 177-33030-58598-33
URL
otg.///....gr.////// 1007/
Timo Milbich, Karsten Roth, Samarth Sinha, Ludwig Schmidt, Marzych Ghassemi, and Bjorn Ommer. Characterizing generalization under out-of-distriutioon shifts in deep metric learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 25006-25018 Curran Asso- ciates, Inc., 2021. URL https: / http//poooeeengeeeee/ee neur ips. ca/ cttccc///paaee pptpp///pp////1le/// d1 d2f2555377 3a ffccef77222000 f2we 9d 900ead9aaaeaaaaeeeeeeeee pdf..
Graziano Mita, Maurizio Filippone, and Pietro Michiardi. An identifialle double vae for disentan- gled representations, 2020. URL https: /awp/vaa.vvvvv/aa///// org/ abs///b////// 22100 09360.
Milton L. Montero, Jeffrey S. Bowers, Rui Ponte Costa, Casimir J. H. Ludwig, and Gaurav Malhotra. Lost in latent space: Disentangled models and the challenge of combinatorial generalisation, 2022. URL https:ttp/..o //tp/aawvvv/////////////////aavl oogg abs 2220222222222 02283.
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bow- ers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2021. URL https: www.eeewwe.....eeo//p//. net (ow.mmiddguggrr1 jKUVy.
Andrew. Y. Ng. Feature selection, vs. 12 regularization, and rotational invariance. In Proceedings of the twenty-fist international conference on Machine learning, 2004.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11453-11464 June 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp- 8024-8035. Curran Associates, Inc., 2019. URL http: pecte://papers.neuripss cc/ cc/paper/ 9015-pytorch- an wwnn--imperatiiie streeesssleeeeee weweeehighhh--eee pdf.
deep- Lew.-
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournaeeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leamn Machine learning in Python. Journal of Machine Learning Research, 12:2825-2834 2011.
13

---

Published as a conference paper at ICLR 2023
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foun- dations and Learning Algorithms. Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, 2017. ISBN 978-0-262-03731-0. URL https: /ttm/mmttsssssseeeeeeeeeeessesst .mittsssmtit.... .it..ddu/ books/ee ementts causal -nnusal-inferece...
David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by subspace diffusion. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 17403-17415. Curran As- sociates, Inc., 2020. URL https: http:::::///pooeeeingg..... neurips cc pttpp/pppprr//////e/e pdf.
Joaquin Quinonero-Candela. Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence. Dataset shift in machine learning. 2009. ISBN 9780262170055. URL http://dl acm. org/ cigationn cfm?id=1462124
Scott Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy- making. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Asso- neuriigs..euur....
ciates, Inc., 2015. URL e0741335487 (AAEL333677700011996666666662660
wwweerrps..cppppe////e... wwtpppppee//////ieeeee.ee haper.pdf.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of The 30th International Con- ference on Machine Leurning (ICML), 2014.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. In S. Bengio, H. Wallach, H. Larochelle. K. Grauman, N. Desa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https: httpss////oooeeddeeeeeee neurips...... wctppaappe/211//11eee.. 7w1.56566688222288 Pteer7681111--pppppepppep
R. Tyrrell Rockafellar and Roger J.-B. Wets. Variational Analysis, volume 317 of Grundlehren der mathematischhn Wissenschaften. Springer, 1998. ISBN 978-3-540-62772-2. doi: 10.1007/ 178-3-642-02431 02431-3. URL httpt: org/ 10.1100000000 /1t.:/0100077777-------222111111
Michal Rolinek, Dominik Zietlow, and Georg Martius. Variational autoencoders pursue pea direc- tions (by accident). In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjorn Ommer, and Joseph Paul Co- hen. Revisiting training strategies and generalization performance deep metric learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8242-8252. PMLR, 13-18 Jul 2020. URL https:/ http::://proceedingg www.:...ww.....................e wtt.ppeeee//111/tthhh000 html.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 7420-7430, June 2022.
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6065. PMLR, 09-15 Jun 2019. URL https: html.
stttps://proceednngg pttpp/edeeeeee.eo////// mlr. prwss https///////ttrr/eeeeeee
Shangyuan Tong, Timur Garipov, Yang Zhang, Shiyu Chang, and Tommi S. Jaakkola. Adversar- ial support alignment. In International Conference on Learning Representations, 2022. URL https:/ wttp///wwwweeeeeeeeeeeeeeeeeeeep net d
14

---

Published as a conference paper at ICLR 2023
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf. and Stefan Bauer. On disentangled representations learned from correlated data. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10401-10412. PMLR, 18-24 Jul 2021. URL https: http:////occcediieeoo ngs. ml press/ v139/t 11339trauble2la.htt.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-basee representation learning. ArXiv, abs/1812.05006 2018.
Yixin Wang and Michael Jordan. Desiderata for representation learning: causal perspective, 2021. URL https: /
15

---

Published as a conference paper at ICLR 2023
A ALTERNATE HAUSDORFF VARIANTS
In this section, we introduce various variants our Hausdorff distance approximation introduced in $2.3 and particularly Eq. 5, which we then experimentally evaluate in $B.
A.1
AVERAGED HAUSDORFF
First, due the sensitivity to outliers, one can also utilize the average Hausdorff distance, which simply gives:
(22
H.avg (2) = Ei= Ej=+1 x ExEZ, [minwez.....)) d(z,2')
(9)
using the same pair-based approximation introduced in Eq. 5.
A.2 SUBSAMPLING
One also operate the full approximated Sx with Sx = S* = Z,, Z, 2 x.. Z instead of a collection of S Z,, Z... by simply utilising randomly subsampled version of 8x denoted Sx (through i.i.d stitching of latent entries sampled from each dimension support):
di = max =ES
min d(z, Z'EZ
(10)
However, in practice (as shown in SB), we found di to work better, as the max-operation over a collection of 2D subspaces provides less sparse training signal than single backpropagated distance pair in s.b.
A.3 SAMPLING-BASER SOFTMIN
In addition, the latent representations and the corresponding support change during training, one can also encourage some degree of exploration during training instead of relying on the use of hard max and min operations, for example through probabilistic selection of the final distance to minimize for, allowing for a controllable degree of exploration during training:
22
H.prob
max
E Ba ( (12x77.(1.)))))) (((z,z'))
(1I)
im j=i+1 2ES* (2)
with the SoftMin-distribution
Psoftmin
(z' T) ez exxp(-d(2. x ///)
(12)
though shown in the following experimental section, found minimal benefits in doing
A.4 SOFTENED HAUSDORFF DISTANCE
To potentially better align the Hausdorff distance objective with the differentiable optimization pro- cess, it may make be beneficial to look into soft variants to relax the hard minimization and maxi- mization, respectively (note the d instead of d):
exp(-d(z,, 21)/
min (z, ZZ) Ex EZ expeepppopoo (-d(z, 2 2n) )/T)
pooft (z, ()
Σ min ( Z) d(z,z')
(13)
b
exp(d poft min [z, (..ii))/TT) groft (z, (..i.)))
(12) HH
(Z)
Σ
Σg
2 (2.1 exp( (ooft (z" Z (/(2))
1=
ZEZ.
16

---

Published as a conference paper at ICLR 2023
Table 2: Method ablations di We compare our default pairwise Hausdorff' approximation against variant using averaging instead of max H. probabilistic approximation to min (dipprr as well as subsampling of the full-dimensional factorized support without any pairwise approximations. In all cases, entries are selected with at least 5 seeds and optimal values chosen from gridsearch over 7 E {1, 3, 10, 30, 100, 300, 1000, 3000, 10000}.
Setup L
No Correlation
Correlated Pairs:
Correlated Pairs:
dz
76.9 (74.3, 81.4]
55.9 [47.6, 59.9]
48.0 [39.1, 48.99
6 (Eq. 9)
61.1 [55.6, 65.51
33.5 (26.9,36.) 36.01
42.4 43.39)
grob (Eq. 11)
((2)
74.7 [69.5, 77.4]
56.2 [46.0, 60.4]
46,9 (37.7, 48.0)
dt (Eq. 10), Subs.
56.4 [52.6, 61.7]
38.1 [21.2,
29.7 [19.7,
6y (Eq. 10), Subs. 800
60.0 (55.0, 62.8]
44.5 (2.07.48.0]
31.6 (2.14.4411
63 (Iiq. 10), Subc. 10
62.2 (59.5, 69.3]
54.71
32.1 (3..0,
d# (Fq.. 10), Sube. 8 10*
66.9 [61.3, 71.2]
(9.7)
39.2 (31.9,4388
4y (Eq. 10). Subs. 8 10
66.6 [62.2, 72.41
56.8 (50.1
40.9 (37.3,45.2]
No Correlations
Correlated Pairs: 1
Correlated Pairs: 3
Standard (+)
60
50
(-) with Mean
DCI-D
45
T2 = 1e-06
40
T1 1e-05
40
T₁ = 0.0001
20
35
T1 0.001
10-5
10-3
10-5
10-3
10-5
10-3
T1 = 0.01
Outer temperature T2
Outer temperature T2 Outer temperature T2
Figure 5: Results for our soft approximation to Eq. 5. Blue horizontal line denotes the default Eq. 5 objective, while orange denotes replaced of the max-operation with mean. We find that generally, a convergence of the soft approach to our default hard variant performs best, with large choices in the outer temperature converging towards our mean approximation.
Here, the temperature Ti controls the translation between putting more weight on the minimal dis- tance (smaller TI) versus more uniform distribution (larger 72), moving further away from the corresponding min operation.
secondary T3 then controls the transition between the max-operation over our soft distances and a more uniform weighting over all (non-zero) soft distances, with the limit case + oc approxi- mating the simple mean over soft distances.
B EVALUATION OF HAUSDORFF DISTANCE APPROXIMATION VARIANTS
As our utilized distance function dH only approximates the Hausdorff distance to the factorized support, we now move variant study of other alternative distance measures as described above.
(22)
In particular, we investigate (1) replacement of the max-operation with corresponding mean over support samples to address potential outliers better, (2) probabilistic approximation our min- operation over d(z, z') (see Eq. 11), (3) and a fully soft approximation to both max and min using respective Softmax and Softmin formulation (A.4). (4) Finally, we also revisit the impact of exlicit scale regularization as introduced in $2.3.
Method ablation. Ablation studies across the default as well as two different training correlation settings can be found in Tab. 2, with each entry computed over at least 6 seeds, and gridsearch over y E 31 10, 30, 100, 300, 1000, 3000, 10000}. Our results show that for optimization pur- poses, approximating the Hausdorff distances in "sliced", pairwise fashion suggested in Eq. 5 is noticeably better than subsampling from the incredibly high-dimensional factorized support, as instead of single distance entry that optimized for (after the max-min selection), we have vari-
17

---

Published as a conference paper at ICLR 2023
Table 3: Impact of the number of 2D approximations in d Our experiments reveal that the use of multiple 2D approximations to the full Hausdorff distances has notable merits up to a certain degree (change disentanglement performance from e.g. 64% 75% in the uncorrelated transfer setting). Each entry was chosen as the highest value in a gridsearch over E {0.01, 0.1, 1, 10, 1003.
Correlation |
Num. Phins: 1
Nam, Pains:
Num. Pains: 5
Num. Pairs: 15
Num. Pairs: 25
Num. Pairs: 35
Num. Pains: 45
No Comelation
64.0 (58.5,
66.3 [68.8 72.6)
67.9
76.6 [63.7
17.., 77.6)
75.5 1724,
74.3 171 Ro0)
Convlltted Pains:
(21.0, 29.6]
25.0 (23.0 35.4
28.7 (24.5,4 45.6]
54.2 [44.2. 67.0
$1.4
58.3]
(48.3. 57.9
53.5 (5) 99.1]
Correlated Pains:
40.9 (38.5,44 44.5]
44.3 (41.3, 48.0)
46.8 (44.) (4.9)
46.5 [45.7. 47.5]
48.7 50.21
48.8 (468,52.0) 52.0)
49.2 (45.6 518)
(22) Table 4: Impact of scale regularization (as detailed in $C) using VAE + dH with Y
100. For each setting, we perform a gridsearch over either the weight scale 8 E 1, 10, 30, 100, 300, 1000, 3000, 10000) or the L2 Regularization weight E
{10-6, 10-5- 10-1,10-1 110 10- 10-2, 10-1). Results show that scale regularization, while not neces-
sarily detrimental, does not provide any consistent benefits.
Sctup Correlation
No Correlation
Correlated Pairs:
Correlated Pairs:
Buseline Variance
74.1 [73.8, 77.6]
51.4 (42.5, 58.3]
48.7 [47.8, 50.2]
72.4 [65.4, 76.1]
54.4 [53.0, 55.6]
50.1 [49.2. 53,3]
Min. Range
75.0 [67.5, 78.5]
50.6 (38.7,
51.5 [44.2, 53.3]
L2 Decoder
73.4 (70.1. 78.21
56.3 (5144, 60.01
52.9 (44.3, 52.21
ous latent subsets that incur training gradient, and in two dimensions can cheaply compute the full factorized support.
Similarly, also find that replacing the min-selection over latent entries with a probabilistic vari- ant, as well as the max-selection over factorized support elements, offer no notable benefits. In particular the replacement of the outer max-operation severely impact the disentanglement per- formance.
These insights additionally supported when utilizing soft variant (see Eq. A.4 in the appendix), which replaces both max and min operations with a respective Sofumax and Softmin operation, each with respective temperatures 7i and 72. When utilizing this objective, we see that small temperature choices on both soft approximations are beneficial, and approximate the hard variant. Similarly, we find a consistent drop in performance when either one of these temperatures is reduced, with the soft performance converging towards the Mean variant when increasing the outer temperature 72. Overall, we don't see any major benefits in a soft approximation, while also introducing two additional hyperparameters that would need to be optimized.
Finally, we ablate the key parameter for our Hausdorff distance approximation of choice, d (Eq. the number of pairs over which we compute sliced 2D variant. Given total latent dimensionality k, we are given (5) usable combinations, which we can choose to subsample all the way down to a single pair of latent entries, which is what we do in Tab. 3. The results showcase that while minimal number of latent pairs is crucial, not all combinations are needed, with diminishing returns for more pairs included. For practical purposes, we therefore choose 25 pairs as our default setting to strike balance between performance and compute cost, which however can be easily increased needed. On the latter note, we also highlight that while the addition of CHPS does incur higher epoch training time (60s for standard VAE as used in Locatello et al. (2020b) on NVIDIA Quadro GV100) than V-E (52s), it still compares favourably when compared to e.g. ; 3-TCVAE (70s) or FactorVAE (96s). In addition, the impact the training time diminishes when larger backbone networks are utilized.
C REGULARIZING SCALE TO AVOID COLLAPSE
In the limit case, the standard Hausdorff matching problem is solved by collapsing all representations into singular point. In addition to that, the actual scale of the latent entries directly impacts the distance scale. One can therefore provide additional regularization on top to ensure both a scale- invariant measure well work against potential collapse, for example by enforcing minimal
18

---

Published as a conference paper at ICLR 2023
standard deviation p on each factor
Lucale Σ max 0, v Var [2,,0]
(14)
or simply enforcing minimal range [a, b] of the support:
Seale max [0, max(z.))] + max [0, min(z..)) a)
(15)
In real setups however, we have found regularization of scale not be necessary in the ma-
jority of cases, as the use of the additional autoencoding term alongside the Hausdorff Support Factorization sufficient avoid collapse (see part in $2.3 on collapse), as shown Tab. 4. For Tab. 4, we also investigate what happens if we apply 1.2 regularization on the decoder. In all cases, we perform a grid-search over an addi- tional scale regularization weight parameter ó (with E {0, 1,3, 10, 30, 100, 300, 1000, 3000, 100003) the L2 Regularization (10-6, 10-5, 10-1001. 11100,100 (.-10) show improvements that are both significant and consistent across correlation settings. And while these regularization may become relevant for future variants and extensions, in this work we choose to forego a scale regularizer with the benefits of having one less hyperparameter to tune for.
Grid-Parament
10-1
10
10a
104
10
80
No Correlation
weight (E Our results
40
10
109
10a
10a
10a
D HYPERPARAMETER EVALUATION
Prri-Paasseteers B
Figure 6: Robustness to factorized sup- port weighting 7 much less detrimen- tal overall training dynamics.
To understand to what extent the factorization of support parameter y impacts the learning and performance of the model, we also compare grid searches over and the stan- dard VAE prior matching weight B. The results in Fig. 6 indicate that factorization of support is much less de- pendent on the exact choice of weighting as opposed to the standard KL-Divergence to the normal prior used in 3-VAE frameworks (notice the logarithmic value grid). This stands to reason, fac- torization of the support instead of distributions is both more realistic property as well as a much weaker constraint the overall training dynamics.
E SAMPLE RECONSTRUCTIONS AND QUALITATIVE EVALUATION
We also provide some qualitative impression of the impact an explicit factorization has the overall disentangiement across different correlations. In particular, Figure visuals latent traversals both for the 3-VAE baseline (top)as well as the auggeetedd variant (bottom) for the latent entry most expressive for the first mentioned latent entry in ("Correlations addressed"). To generate these figures, we select the best performing seed for each setup, and report the respective DCI-D score within each subplot. As can be seen, beyond the increase in maximally achievable DCI-D, an explicit factorization of the support helps the disentangling method separate factors initially struggled with both when correlations exists in the training data as well as for generally failure modes when the 3- V-V fails to fully disentangle the uncorrelated setting.
"Though this enforces implicit assumption the density within each latent factor.
19

---

Published as a conference paper at ICLR 2023
Correlation: No Correlations
Correlations addressed: Object Color & Object Type/Size DCI-D Scores: BetaVAE [78 HFS [100.0]
BetaVAE
BetaVAE
HFS
Correlation: One Pair
Correlations addressed: WallColor FloorColor
DCI-D Scores: BetaVAE [65.7]
HFS [87.8]
BetavAE
BetaVAE HFS
Correlation: Two Pairs
Correlations addressed: Object Color Object Type
DCI-D Scores: BetaVAE [81.8] + HFS [99.5]
BetavAE
BetaVAE + HFS
Three Pairs Correlations addressed: WallColor & FloorColor, among others DCI-D Scores: BetaVAE [50.0] + HFS [62.9]
Correlation:
BetavAE
BetavAE + HFS
Correlation:
Shared Confounder
Correlations addressed: WallColor Confounder
DCI-D Scores: BetaVAE [41.1] + HFS [72.4]
BetaVAE
BetoVAE HFS
Figure 7: Sample traversals for the Shapes3D benchmark Kim & Mnih (2018) in latent space for latent entry most closely associated with various ground truth factors of variations across different correlation shifts. In all cases, the best seed (out of 10) was selected perform these qualitative studies. Each image also reports the associated overall DCI-D score of each respective best seed for 3-VAE and 3-VAE + HFS.
20

---

Published as a conference paper at ICLR 2023
(a) Shapes3D
Metial
Na
Part
Rac
Pauc
Parc
Paan
Poacc
Fax
Rac
Cod.
505
213
150 1050300
VAR 10%
128 64.1
1106 01-33
20.21442 m-
143
1900
188
064 940
23.
28.1
(153-227)
2191300 23
243 038
20 250
218188200
216 08 368
123 1800 160
124 16-9
153
100
(b) MPI3D
IVL V4
[VL N444
[VL
192 (22
pV2 m002)
20046
P3 1331
205
20RI 170
POPINE 1930
10:1
250
WT2
SEA
DaneMAA
15.056 9091
2211 20.0
151 16:11
20 2-41
212 15.
234 9611
13.165 1848
183175 19.1
167 13S (5.)
ArwolatVAA
184065, 1181
018 (1.3)
14001 1591
2001 2w0)
MOX
2831226 96.11
10 196
191
163 169
(c) DSprites
Table 5: Full table for Tab. 1 with detailed and extended correlation settings. To understand the exact factors correlated, please check the associated pairings from SH.
No Corr. No Corr.
1 Corr. Pair No Corr.
2 Corr. Pairs
Corr. Pairs No Corr.
Shared Conf. No Corr.
No Corr.
Relative improvement [%)
20
Linear probe + L1 Reg.
15
10
10
10
10
10-2
10a
10
10"2
10
10
10
10"1
10"
10
10
10a
Training data percentage
Figure 8: This figure shows adaptation behaviour across different amounts of test data for a LI- optimal linear probe (i.e. for each seed and entry, we selected the optimal L Ll-regularization values). Reported values show relative improvement in average ground truth factor classification perfor- mance of 3-VAE + HES versus standard 8- 3-VAE. As can be seen, the increased disentanglement through an explicitly factorized support gives expected improvements increasing with the severity of training correlations encountered.
F DETAILED FIGURES AND TABLES
In this section, we provide detailed variants of figures and tables utilised the main paper.
F.1 ADDITIONAL DISENTANGLEMENT RESULTS
For Tab. studying the impact of HFS both as standalone objective and a regularizer on disentanglement of test data across varying degrees of training correlations, we include a more detailed variant highlighting the exact splits utilised in Tab. 5, as well as additional correlation settings.
F.2 FURTHER ADAPTATIONS
Extending our adaptation experiments done in $4.3, we also investigate the average classification performance of ground truth factors of weaker, Ll-reeularized linear probe in Fig. 8. Similar to
21

---

Published as a conference paper at ICLR 2023
to.
Aami d-)
Pors 1,10 ded
Jano 2.12 Sw00)
Pain 0.1
PIR 3.191 (wd.1l
Car W (e02)
0.0
25
02
52
02
10
101
10
10'
x2120 Train Daca
han hambees
60
100
100
50
170
100
VIEE KID lo5s
Figure 9: This is the full figure for Fig. 4, showcasing that a factorization of the support on the training data is consistently linked to improved downstream disentanglement (top), and that mini- mization of the standard 8-VAE KLD-objective for a factorial distribution implicitly minimizes for a factorized support across settings.
DC-D S-VAE
ADCIG with Factorized Supoort
DCH B-VAE
ADCH with Tactorized Support
1.1. 6.0
et Pudr
ar
14 utan g 10
TV mn mn in aaA
IO IUE
Car 12
w.2
u.
PY AL BT Ia
Figure 10: This is the detailed correlation shift transfer grid utilized in Fig. 2, indicating the exact correlation settings used for training and test data.
our transfer results, we again find that the increased disentanglement through explicitly factorized support gives expected improvements which increasing with the severity of training correlations.
F.3 FURTHER PROGRESSIONS
Fig. 9 provides visualization of the relations between training support factorization and disentan- glement performance on test data (top), as well the KL.-Divergence loss in the standard 33-AAE for more training correlation settings shown in the main paper figure 4(orange and black graphs), with insights transferring from the main paper.
F.4 DETAILED CORRELATION SHIFT TRANSFER GRID
For replicability, we provide copy Fig. with the exact utilised correlation settings in Fig. 10.
F.5 DETAILED METRIC GRID
Finally, we also provide more detailed copy of the metric transfer grid in Fig. 4(right, blue) with the exact correlation settings investigated.
22

---

Published as a conference paper at ICLR 2023
Training Support Factorization
Vs. Test Metrics
Pairs: 1. [V1.
-9.77
-98.7
-9.4
-96.6
-67.1
-1.6
Hairs: 1, [V2. 0.1]
93.7
-9.4
97.5
.7.8
-95.2
.6.33
Pairs: [V2, a 0.4]
99.3
Pairs: 2. [V2, -87.7
99.1
97.4
-9.7
-03.9
22.6
Pairs: 3. [V], 0.1]
-62.0
-71.4
-58.6
-64.9
-46.1
-76.6
[V2. 0.2))
Cont.
-83.7
-43.5
-67.0
MIG
DCI-D
DCH-
DCI-C
Sap
Mod
Figure 11: Correlation of Hausdorff distance factorized support the training data various disentanglement metrics (in particular DCI and MIG) across correlation shifts. We find factoriza- tion of supports the training data to strongly relate downstream disentanglement even when experiencing strong correlation during training.
G DISENTANGLEMENT METRICS
In this section, we will provide brief introduction into various disentanglement metrics, with par- ticular emphasis on the DCI-D metric (Eastwood Williams, 2018) used as our leading measure of disentanglement.
G.1 DCI-D AND DCI-I
DCI-Disentanglement was introduced in Eastwood & Williams (2018) as part of a three-property description of learned representation spaces, alongside Completeness and Informativeness. In this work, we primarily utilize DCI-D as measure of disentanglement, and DCI-I a measure of generalization performance. In particular, each submetric utilizes multiple classification models (e.g. logistic regressor (Eastwood & Williams, 2018) or boosted decision tree (Locatello et al., 2019b)), which are trained predict each underlying ground-truth factors from representations extracted from the dataset of interest, respectively. DCI-I is then simply computed as the average prediction error (on a test-split). To compute DCI-D, for each ground-truth factor and consequently each prediction model, predictive importance scores for each dimension of the representation space are extracted from the classification model, given as R E Rdd with representation dimensionality d and number of factors k. For each row, the entropy value is then computed and subtracted from being high if dimension is predictive for only one factor, and low it is used to predict multiple factors. Finally, each entropy score is weighted with the relative overall importance of the respective dimension to predict any of the ground-truit factors, giving
&
R Σ. Ex- R
DCI-D = E( H (Nom R..))
Σ 3
G.2 MUTUAL INFORMATION GAP (MIG)
The Mutual Information Gap (MIG) was introduced in Kim & Mnih (2018) to measure the mutual information difference of the two representation entries that have the highest mutual information with respective trouh factor normalized by the respective entropy, which is then averaged for ground-truth factors. For our work, we follow the particular formulation and implementation introduced in Locatello al. (2019h), by taking the mean representations produced by the encoder
23

---

Published as a conference paper at ICLR 2023
network, and estimating discrete mutual information score, such that the overall MIG can be computed as
MIG Σ ((m(k..)) 2k) (22 21))
(zxx)
where k denotes the number of ground-truth factors of variation 2x. (=s) the respective entropy of 2k. 2 the i-th entry of the generated latent space, and mik, n) a function that returns the repre- sentation index with the n-th highest mutual information to ground-truth factor To compute the discrete mutual information, we get distribution estimates by binning representation values for each dimensions across 20 bins, doing so over 10000 samples.
G.3 MODULARITY
Ridgeway & Mozer (2018) introduce the notions of Modularity and Expressiveness as key com- ponents of disentangled representation with the former evaluating whether each representation dimension depends on at most a single ground-truth factor of variation, and the latter the predictive- ness of the overall representation predict ground-truth factor values. Similarly to Locatello et al. (2019b), we mainly focus on the property of Modularity, which Ridgeway & Mozer (2018) define for d-dimensional representation space with k ground-truth factors
Modularity 1 Σ
[m Ilj- jrapmaax, mi
(16)
(maxg mi.g) m
which, per latent dimension i measures the average normalized squared mutual information scores between the factors that do not share the highest mutual information with the latent entry i. Here, ma, denotes the discretized mutual information between latent entry and factor j similar to our implementation of the Mutual Information Gap and Locatello et al. (2019b), where we utilize discretized approximation by binning each latent entry into 20 bins over 10000 samples to compute the discretized mutual information scores.
G.4 SAP SCORE
The Separated Auribute Predictability (SAP) score was introduced in Kumar et al. (2018) as another disentanglement measure, in which the authors suggest to train a linear regressor (in the case of Locatello et al. (2019b)a linear SVM with C 0.01 and again 10000 training samples and 5000 test points) predict each ground-truth factor from each dimension of the learned representation space, and then taking the average difference in prediction errors between the two most predictive latent entries for each respective ground-trut factor.
G.5 BETA- AND FACTORVAE SCORES
The FactorVAE Score (Kim & Mnih, 2018) is extension of the BetaVAE Score introduced in Higgins et al. (2017). In both cases, a ground-truth factor of variation is fixed, and two sets of obser- vations are then sampled. The BetaVAE score then measures disentanglement as the classification accuracy of a linear classifier to predict the index of the fixed factor based on the average absolute differences between set pairs. In Locatello et (2019b), this process is repeated 10000 times to train a logistic regressor, and evaluated on 5000 test pairs. The FactorVAE score improves on this metric through the use of majority vote classifier that instead predicts based on the index of the representation entry with least variance.
24

---

Published as a conference paper at ICLR 2023
Method
Parameter
Values
B-VAE
3
11,2, 3, 4, 6, 8, 10, 12, 16]
8-TCVAR
3
[1, 2, 3, 4, 6, 10, 12, 16]
AnnealedVAE
Cmax
[2, 10, 25, 50, 75, 100, 150]
FactorVAE
3
[2, 10, 25, 50, 75, 100. 150]
HES
[20, 40, 80, 100, 200, 400, 800, 1000, 2000, 4000]
3- VAE HFS
Y
[30. 60, 100, 300, 600, 1000, 3000, 6000]
Table 6: Hyperparameter grid searches for different baseline methods well as our factorized
support objective.
H FURTHER EXPERIMENTAL DETAILS
Study design. We implement all our experiments using the PyTorch framework Paszke al. (2019). For exact and fair comparability, we re-implement all baseline meth- ods based on Tensorflow implementation provided through Locatello et al. (2019b) pectps://githee hawll.comooooll... rwwearche orgpoppleererrrddddddsstttnateee following public repositories: http:///////gggoooooooo///a/aaaa
as well as the vae,
http:://tttta..coomocccc////aaat htths://gitthb. htter com/ ubisoft/ubisoftt metrics/det: py and htt.:pppe//ggttht......o//tttt/p
For the implementation of the disentanglement metrics, follow the implementation used in Lo- catello et al. (2019b), which for the computation of the DCI metrics leverages a gradient boosted tree from the scikit-learn package. The VAE architecture used throughout our experiments follows the one utilized in Locatello et al. (2020b), which leverages the following architecture, as- suming input image sizes of 64 64 ne with ne the number of input channels, usually 3, and latent dimensionality of 10:
Encoder: (30nv(32, 4 x 4, stride 2) ReLU] x 2, [conv(64,4 4 x 4, stride 2) ReLUJ x 2, MLP(256), MLP(2 x 10)
Decoder: MLP(256), [upconv(64,
stride 2) + ReLU] x 2, [upconv(32, x 4, stride 2) + ReLUJ, [upconv(n..- 4 4, stride 2) + ReLU]
The training details are follows:
Optimization: Batchsize = 64, Optimizer Adam (31 = 0.9, 32 = 0.999, e = 10-8),
Learning = 10
Training: Decoder distribution = Bernoulli, Training steps 300000
Note that FactorVAE Kim & Mnih (2018) introduces separately trained discriminator, again utilize the setting described in Locatello et al. (2020b):
Architecture: [MLP(1000), lea 6, MLP(2)
Optimization: Batchsize = 64, Optimizer = Adam (81 = 0.5, B2 = 0.9, 1 10-8)
Finally, we provide the hyperparameter gridsearches performed for every baseline method which mostly follow Locatello et al. (2020b), as well as for HFS (though for some ablation studies more coarse-grained grids very utilised) and 3-VAE + HFS: Note that for AnnealedVAE, we also leverage an iteration threshold of 10 and ạ = 1
H.1 CORRELATION SETTINGS
We provide more detailed information regarding the specifie abbrevations used throughout the main text and for the following appendix to denote various correlation setups during training. We note that to introduce multiple correlated factors pairs, simply multiply respective p(c,, Cj) entries.
For Shapes3D (Kim & Mnih, 2018), we introduce the following correlations:
25

---

Published as a conference paper at ICLR 2023
No Corr.: :: No Correlation during training. This constitutes the default evaluation setting.
Pair:
[V]]: flcorCol and wallCol.
Pair:
[V2]: objType and objSize.
Pair:
[V3]: objType and wallCol.
Pair:
[v4]: objType and objCol.
Pair:
[inv, v4]: objType and objCol, but inverse correlation.
Pairs:
[V1]: objSize and floorCol well ob objyyp and wallCol.
Pairs:
2 [V2]: objSize and ob oTyee well as floorCol and wallCol.
and objSize. Pairs: [inv, v3]: objType and objCol as well as objType and objSize, but with inverse correlation.
Pairs:
2 [v3]: objType and ob objCol as well as
Pairs: [V1]: objsize and ob jAz (arjjAimmue immth as well as jbype and wallCol, and objCol and floorCol.
Pairs: [v2]: objCol and jAzzmuut well objType and objsize. and wallCol and floorCol.
Shared Conf. [V1]: We correlate (confound) ob jType against all other factors.
Shared Conf. [V2]: We correlate (confound) wallCo. 1Col against other factors.
For MPI3D (Gondal al., 2019), introduce the following correlations:
No Corr.: No Correlation during training. This constitutes the default evaluation setting.
Pair:
[V1]: cameraHeight and backgroundCol.
Pair:
[v2]: objCol and objSize.
Pair:
[V3]: posX and posY.
Pairs:
[V1]: objCol and objShape as well as posX and posy.
Pairs:
[V2]: objCol and posX well ob jShape and posY.
[VI]: objCol and backgroundCol as well as camera cameraHeight and posX, and jbjShape and posy.
Pairs:
Pairs: [V2]: objCol and posX as well as objShape and posY, and backgroundCol and cameraffeight.
Shared Conf. [v1]: We correlate (confound) ob jShape against all other factors.
Shared Conf. [v2]: We correlate (confound) posX against all other factors.
For DSprites (Higgins et al., 2017), introduce the following correlations:
No Corr.: No Correlation during training. This constitutes the default evaluation setting.
Pair: [V1]: shape and scale.
Pair:
[V2]: posX and posY.
Pair:
[V3]: shape and posY.
Pairs:
[V1]: shape and scale as well posX and posY.
Pairs: [V2]: shape and posX well scale and posY.
Shared Conf. [V1]: : We correlate (confound) shape against all other factors.
Shared Conf. [V2]: We correlate (confound) posX against all other factors.
26

---

Published as a conference paper at ICLR 2023
I PSEUDOCODE
Finally, we provide a PyTorch-style pseudocode to quickly re-implement and apply the factorization objective following Eq. 5.
Inputs:
# * Batch of latents <2> [os x dim]
Number of latent pairs to use for approxination <t_pairs_to_usex
import itertools as it import numpy as np import torch
Get available latent pairs.
10 pairs np np.array (list (it. cotminnainss (range (dim), 2)))
11 pairs Len (pairs)
12 pairs - [nirs[np. random. choice (n_pairs, n_pairs_to_use, replace-Palse))
14 Subsamplc batch <2> [os latent_din] into <>_2> [bs x bum_latent_palzz
2] 15 s_ z[..., pairs]
16
ixs_a 10, .... bs-1, 0, ...., bs-1]
18 ref_ranne range torch. torch.arange (len sevice=z.devica
19 ixs a = torch.tile (ref range, (ims=
20 ixs_b [0, 0, 0,
1, 1, bs-1]
ixs_b torch. ree.attinteereeeeeeeeeeeeeeeeeee (ref_range 1en len
22
23 Aggregate factorized support:
24 For every latent pair, we select all possible batch pairwise
combinations, giving our factorized support <fact_z> dim dim(fact_z)
*x2 num latent__aatleeeeeeeeee_eelllllee
2
27 fact_2 torch. cat [is_a ;, 0:1], s [ixs_b, ;, 1:2]], dim=-1)
28
29 Compute distance between facterized support and 2D batch embeddings: dim(dists) = bs 2 bs x num_pairs sraat_zz.usqueerae (1) s_z. s_z.unsqueeza (0)) 2) sum (-1)
30
dists
12
33 Compute HausdorEf distance for each pair, then sum up each pair contribution.
hfs distance dists.min (1) [0] max (0) [0] sum
Listing 1: Sample PyTorch Implementation of
(2) (Eq. 5)
27


Reviews:
[{"review_id": "ndrsA5dIEx", "reviewer": "Reviewer_e6Ls", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles a key problem in disentanglement - the independence assumption made by most methods does not hold in practice. Instead, the paper suggests requiring independent supports between factors which can be satisfied even when correlation exists. The paper proposes to measure support independence by the empirical Hausdorff distance, but find that doing so is infeasible. Instead, it measures the hausdorff distance on slices of the representations (specifically pairs of factors). While not powerful enough on its own, the authors show the Hausdorff loss term is very helpful when combined with existing approaches. Benefits are shown, particularly when strong correlations exist in the data. Good results are also shown for OOD generalization.    ", "summary_of_the_review": "Overall, I find the idea interesting and the experimental evidence positive. Please address the weaknesses above. I am leaning towards acceptance.\n\n#######################\n\nPost-rebuttal: The authors mostly addressed my concerns, I keep my rating and recommend acceptance. ", "strength_and_weaknesses": "+ Removing the assumption of independence between factors of variation in disentanglement models is of high importance.\n+ The approach of independent supports may be a promising direction for tackling this task and appears novel.\n+ The formulation of the loss term seems reasonable (but does not seem to be the entire story)\n+ Experimental results do seem to confirm the claims of the paper (but note weaknesses) \n\n- The loss term is only able to detect second order correlations, which is not quite the same as ensuring factorized support. While the authors explained why this did it, it would have been interesting to understand the performance loss due to this relaxation.\n- The experiments are somewhat self-serving in the sense the correlation is linear between pairs of terms, which is also what the loss term is optimising. How would it deal with higher order correlations? \n- The loss term, combined with reconstruction was insufficient and required the KL divergence with the normal distribution, which encourages no correlation. While the authors tried to address it in the paper, I wish that they justify this finding in a clearer way.\n- The datasets useed are somewhat easy. How does this method performance on the more challenging SmallNorb?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b4FywCpGfi", "reviewer": "Reviewer_niK9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To avoid the unrealistic assumption of factor independence in traditional disentangled representation learning, this paper proposes to relax this assumption to factorized support, and proposes a Hausdorff-distance-based regularization. The authors conduct experiments on 3 classical datasets to show the improved disentanglement brought by the proposed method. Additionally, ablation studies show that the proposed method has  promising potential on out-of-distribution settings. ", "summary_of_the_review": "I have carefully checked the main body of this paper(including the introduction, method and experiments part.) I do not check the supplementary files.", "strength_and_weaknesses": "Strength:\n1.\tThis paper is well motivated, aiming to tackle the problem of the unrealistic assumption of traditional disentangled representation learning.\n2.\tThis paper proposes a new and effective training criterion through a Hausdorff distance term, which can be easily combined with existing disentanglement methods.\n3.\tThe experiments show that the proposed method can improve the disentanglement and the out-of-distribution generalization ability.\nWeaknesses:\n1.\tSome intuitions can be further explained, e.g., in section 2.2 the situation that breaks the factorized distribution can have a factorized support. It will be more convincing to give an example which does not have a factorized support will fail to disentangle, more intuitively show the relationship between factorized support and disentanglement.\n2.\tSince this paper claims to aim at the realistic scenario of disentangled representation learning, it is better to conduct experiments on real world datasets instead of the synthetic datasets(at least for the out-of-distribution setting.).\n3.\tThe compared disentangled baselines seem to be out-of-date, it is better to incorporate the more recent disentangling methods. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1QpQGQlKkb", "reviewer": "Reviewer_rYAe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to relax the assumption of statistical independence used in many disentangled representation learning works. The paper proposes to only assume that the support of the latent factors’ distribution factorizes, which is a weaker constraint than statistical independence. Specifically, the paper proposes a training criterion that minimizes a Hausdorff set-distance between the latents’ actual support and its factorization. The paper contains extensive experiments on three datasets with varying degrees of correlation settings between ground truth generative factors. The proposed approach achieves better DCI-D scores as compared to previous works. The learnt representations from the proposed approach also achieve better classification performance in “distributional shift” settings.", "summary_of_the_review": "I think the paper tackles an important problem in disentangled representation learning research, provides a reasonable solution and supports their approach with solid experiments. I have some concerns about the reasoning behind some experimental results, the lack of any experiments with realistic data and providing a more balanced picture of the proposed approach that I have described in more detail in the weakness section. But overall, I think the paper would be useful for the research community and hence I am inclining towards acceptance.\n\n-------------------------------------------------------------------------------------------------------------------------------------------------------\nThe authors' response satisfactorily answers all my concerns. Therefore, I am keeping my acceptance recommendation.", "strength_and_weaknesses": "Strengths\n- The proposed relaxed assumption of factorization of support instead of independence provides more flexibility, allowing correlations in the training data to be learnt.\n- The paper contains extensive experiments to support the proposed approach. Consistent improvements on Shapes3D, DSprites, MPI3D show the robustness of the proposed approach.\n- Improvements in classification performance in “distributional shift” settings show that the proposed approach can facilitate generalization, especially in the case of strong correlation in the training distribution. \n- The paper is well-written and easy to comprehend.\n\n\nWeaknesses:\n\n- The paper describes a problem in previous works, proposes a solution and shows that it works. This usually covers the most important components to evaluate an approach but the paper could be much more useful for the research community if the paper can provide a higher-level picture of where the proposed approach fits in this research area, for example:\n    - A discussion on what different ways could provide the factorization of latents’ support and why the proposed HFS criterion is a better choice among them.\n    - What are the drawbacks of the proposed approach? Should everyone replace the methods from previous works with the proposed method in all cases? If not, in what settings does the proposed approach fare better than others and in what settings it might be better to not use it? Does the proposed approach take more computational resources? Is it a more difficult objective to train? \n- The paper draws its motivation from the fact that real-world datasets contain correlations between generative factors and therefore, the independence assumption is not very suitable for these real-world settings. But, the paper doesn’t include any experiments on uncontrolled real-world datasets, for example, a good first step could be the CelebA dataset.\n- The paper mentions that previous “successful” works enforce the independence assumption even more strongly when it doesn’t hold for realistic data distributions. That should result in these approaches failing poorly on realistic datasets. Can the authors provide any findings (even from other papers) to support this argument? This again emphasizes the importance of at least one experiment with realistic data for this paper (where ideally, the previous works fail poorly, perhaps even more than vanilla VAE and the proposed approach works much better).\n- The paper only considers one disentanglement evaluation metric — DCI-D. Even according to the cited justification Dittadi et al. (2021), both DCI and MIG are considered to be the best. Why do the authors choose to focus only on DCI? What are the results in terms of MIG? Also, it would be better if the evaluation metric(s) being used in the paper is explained briefly.\n- Can the authors explain why HFS is working the best for no correlation setting? Shouldn't independence assumption work perfectly in this case and therefore, existing works have an advantage?\n- Similarly, why do HFS versions of beta-VAE and beta-TCVAE work better than HFS? If the assumptions in beta-VAE and beta-TCVAE are not suitable for correlated settings, their criterion should hurt and not facilitate disentanglement, right?\n- In Figure 2, I would have expected a consistent trend from top to bottom in the first few columns but there seem to be some training correlations in between (3rd, 7th, and 8th rows) that are better than the ones surrounding them. Why is that so? How exactly is the correlation made more severe in this case? \n- Minor:\n    - Definition 1.1 seems insufficient. This definition would be satisfied by the case when there is only one dimension that contains all the generative factors, the rest all being just noise. I think the authors meant one-to-one mapping between the latent dimensions and the generative factors (as mentioned in the next sentence). If so, this definition should be fixed accordingly.\n    - It would be easier for the reader to understand if the authors can explain in more detail why the independence assumption doesn’t hold for the case in Figure 1, perhaps contrasting it with how the distribution would be in the case of independence.\n    - Table 1, column 2, HFS is second best (bold) instead of beta-TCVAE+HFS.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster