Rethinking Disentanglement under Dependent Factors of Variation
Authors: Antonio Almudévar
Alfonso Ortega
Venue: Accepted by TMLR ()

Abstract: Representation learning enables the discovery and extraction of underlying factors of variation from data. A representation is typically considered disentangled when it isolates these factors in a way that is interpretable to humans. Existing definitions and metrics for disentanglement often assume that the factors of variation are statistically independent. However, this assumption rarely holds in real-world settings, limiting the applicability of such definitions and metrics in real-world applications. In this work, we propose a novel definition of disentanglement grounded in information theory, which remains valid even when the factors are dependent. We show that this definition is equivalent to requiring the representation to consist of minimal and sufficient variables. Based on this formulation, we introduce a method to quantify the degree of disentanglement that remains effective in the presence of statistical dependencies among factors. Through a series of experiments, we demonstrate that our method reliably measures disentanglement in both independent and dependent settings, where existing approaches fail under the latter.

Published in Transactions on Machine Learning Research (02/2026)
Rethinking Disentanglement under Dependent Factors of Variation
Antonio Almudévar
almudenar@umizzar..
ViVoLab, Aragón Institate for Engineering Research (I3A) University of Zaragoza
Alfonso Ortega
ortega Erniiaaranrrrrccom .s
ViVoLab, Aragón Institute for Engineering Research (I3A)
University of Zaragoza
Reviewed on OpenReview: https: opt.reveeepr....e... net/ forum? id= PgWkNC630S
Abstract
Representation learning enables the discovery and extraction of underlying factors of varia- lion from data. A representation is lypically considered disentangled when it isolates these factors in way that is interpretable to humans. Existing definitions and metrics for disen- tanglement often assume that the factors of variation are statistically independent. However, this assumption rarely holds in real-world settings, limiting the applicability of such defini- tions and metries real-world applications. In this work, we propose a novel definition of discntanglement grounded in information theory, which remains valid even when the factors dependent. We show that this definition is equivalent to requiring the representation to consist of minimal and sufficient variables. Based on this formulation, we introduce method to quantify the degree of disentanglement that remains effective in the presence of statistical dependencies among factors. Through a series of experiments, we demonstrate that our method reliably measures disentanglement in both independent and dependent settings, where existing approaches fail under the latter.
1 Introduction
In representation learning, data are commonly assumed to be characterized by set of factors of variation (henceforth, factors) and muisances. The distinction between them lics in task relevance: factors directly influence the outcome of interest, whereas nuisances do not. For instance, the task is to describe the fruits, relevant factors may include fruit type, color, or size, while irrelevant muisances might be the background color or the shadows cast. A large body of work has emphasized the importance of learning disentangled representations, where distinct dimensions of the representation
dataset of fruit images where
correspond to different factors of variation (Schmidhuber 1992 Bengio et al., 2013; Ridgeway 2016 Lake
et al. 2017: Achille & Soatto [2118]]. Such representations are valuable across a range of applications. including: (1) interpretability of learned features and model predictions (Hsu et al. 2017 Worrall et al.
2017 Gilpin et al. 2018 Zhang & Zhu 2018 Liu et. al. 2020 Zhu et al., 2021 Nauta et al.
2023
(ii) fairer
decision-making by mitigating or removing biases sensitive attributes such as gender or race
Calders
Zliobaité 2013
Barocas & Selbst
2016 Kumar et al.
2018
Locatello et al. 2019a Sarhan et. al.
2020 Chen
et al.
2023), and (iii empowering generative models to synthesize data with controllable attributes Yan
ct al.
2016 Paige et al. 2017 Huang et al.] 2018 Dupont
2018
Kazemi et al., 2019 Antoran & Miguel
2019
Zhou et al. 2021 Shen et al. 2022
Almudévar et al. 2024).
The definition in the previous paragraph provides an intuitive, though imprecise, understanding of disentan- gled representations. It raises two fundamental questions: (i) what does precisely mean to separate the
factors, and (ii) how can we evaluate whether the factors are indeed separated? In Section 2 we review sev- eral works that attempt to address these two questions. Most of these approaches assume that the factors of

---

Published in Transactions on Machine Learning Research (02/2026)
variation are mudependent-bbott from one another and from the muisances. Consequently, the corresponding definitions and evaluation metrics are grounded in this assumption of independence. Many of the benchmark datasets used to validate these methods also exhibit (approximately) independent factors (LeCun et al. 2004
Liu et al., 2015 Reed et al. 2015 Matthey et al. 2017 Burgess & Kim 2018 Gondal et al. 2019). thereby
making the independence- based delinitions and metrics largely applicable those contexts.
However, in real-world scenarios, factors of variation are rarely statistically independent either from each other or from muisances. Returning to the carlier example, the shape of fruit is not independent of attributes such as color, size, shadow: encountering yellow banana-shaped fruit is common, whereas yellow strawberryshaped fruit is highly unlikely. When factors are independent, separating them in a representation is relatively straightforward. In contrast, when factors are dependent, it becomes less clear what it means for a representation to "separate" them. As illustrated in Figure [ even with a perfectly disentangled encoder, individual components of the representation can still carry information about multiple factors. In this work, we focus on defining and quantifying disentanglement in the sctting where the factors of variation are not independent. Our key contributions can be summarized follows:
We introduce a set of four information- properties that characterize desirable behavior for disentangled representations in the presence of dependencies and nuisances.
We prove that these properties are directly connected to the concept of minimal and sufficient representations. Moreover, we argue that evaluating minimality and sufficiency provides a more meaningful assessment of disentanglement than analyzing the properties in isolation.
We propose a method to quantify the degree of minimality and sufficiency, enabling the evaluation of disentanglement in arbitrary reprosentations.
P (02
x1) = 0.9
Z1
(~2
21) = 0.1
p (
21) 0.0
y1
Perfectly Disentangled Encoder
(Di 22) = 0.5
22
p p
222 0.4
p p
22) = 0.0
3/2
Figure 1: Tf the factors dependent, then-even even with perfectly disentangled encoder-ead sub- representation will contain information about other factors. For example, if 21 encodes the shape of a banana, the color is most likely yellow, possibly green if unripe, and rarely red. Likewise, if 22 encodes the color yellow, the shape is most likely that of a banana or a lemon, but very unlikely that of a strawberry.
2 Related Work
Definition of Discntanglement Despite being a topic of great interest, there is no widely agreed-upon
definition of disentangled representations. Intuitively, a disentangled representation separates the different factors of variation underlying the data (Dejardinss et al.) 2012 Bengio et al. 2013 Cohen & Welling 2014 Kulkarni al. 2015 Ridgeway 2016). Early evaluations of disentanglement were largely based on visual inspection (Kingma & Welling 2014). However, more precise definitions are necessary to understand the
limitations and practical relevance of disentangled representations in downstream applications. One of the
2

---

Published in Transactions on Machine Learning Research (02/2026)
earliest and most widely cited definitions is given in Bengio et al. (2013). and has been adopted by several subsequent works (Higgins et. al. 2017 Kim & Mnih 2018 Kim et al. 2019 Suter et al. 2019 This definition states that a disentangled representation is one in which a change in a single dimension of the representation corresponds to a change in a single factor of variation, while remaining invariant to changes in other factors. An alternative view argues that disentanglement should be defined such that a change in a single factor of variation leads to a change in only one dimension of the representation (Kumar et al. 2018 Locatello et al. (019bb). More recent efforts define disentanglement based on set of desirable properties. Notably, Ridgeway & Mozer (2018) and Eastwood & Williams (2018) independently proposed three key properties that a representation should satisfy to be considered disentangled. While the terminology differs slightly between the two works, the underlying ideas align: (i) Modularity (referred to as disentanglement in Bastwood & Williams (2018)): each representation variable captures at most one factor of variation; (ii) Compactness (completeness in Eastwood & Williams (2018): ): each factor of variation is captured by a single representation variable; and (iii) Explicitness (informativeness in EEstwood & Williams (2018)); the representation retains all (ask-relevant information about the factors present in the input. All of the above definitions present two key limitations: (i) they do not account for invariance to muisance factors that may be present in the input; and (ii) they assumne that the factors of variation are independent of one another and of the muisanccs-wite generally does not hold in real-world data. We propose a set of properties that a representation must satisfy to be considered disentangled, explicitly addressing both of these aspects.
Metrics for Disentanglement Due to the lack of consensus on the definition of disentanglement, there is also no agreement on how it should be measured. Below, we provide an overview of common disentanglement metrics, organized by their underlying principles, following Carbonneau et al. 2022)).
Intervention-based Metrics. These methods construct data subsets that share common factor of variation while differing in others. Representations for these subsets computed and compared to produce disentanglement score. Representative methods include 3-VAE (Higgins et al. 2017] FactorVAE (Kim & Mnih 2018), and R-Factor VAE (Kim et al. 2019). These metrics primarily assess modularity but do not account for compactness or explicitness (Sepliarskaia et al. 2019
Information-based Metrics. These approaches estimate mutual information between the factors and individual variables in the representation. Widely used examples include the Mutual Information Gap (MIG) (Chen et al. 2018) and Robust MIG (Do & Tran 2020). These metrics are mainly designed to capture compactness.
Predictor-based Metrics. These methods train regressors or classifiers to predict the factors from the learned representations. The resulting predictors are then analyzed to determine the con- tribution of each representation variable to each factor. Separated Attribute Predictability (SAP) (Kumar et al. 20188. focuses exclusively on compactness. Subsequently, more comprehensive frame- works have been proposed to evaluate multiple properties of disentanglement. Notably, Eastwood & Williams (2018) introduced DC1 (Disentanglement, Completeness, and Informativeness), while Ridgeway & Mozer (018)) proposed an equivalent set: Modularity, Compactness, and Explicitness.
Disentanglement under Dependent Factors of Variation As previously mentioned, disentanglement under dependent factors is a relatively underexplored area. However, several works have addressed related ideas, among which the following stand out: Suter et al. 2019) offers a cansal perspective on representation learning that connects disentanglement with robustness to domain shift: Choi et al. (2021) proposes a method
to disentangle variations that are shared across all classes from those that are specific each class; Montero
et al. 20200 2022) examine the effects of excluding certain factor combinations during training and evaluate models their ability to reconstruct unseen combinations at test time; Träuble et al. (2021) introduces synthetic dependencies between pairs of factors to study disentanglement in more realistic settings; Roth
eel al. (2023) presents the Hausdorff Factorized Support (HFS) criterion, which accommodales arbilrary (including correlated) distributions over factor supports; [Auua et al. (2023) explores how interventional data can support causal representation learning by revealing geometric patterns in latent variables; and yon
Kügelgen et al. (2024) proposes a nonparametric method for discovering latent causal variables and their relations from high-dimensional observations, without relying on strong parametric assumptions.
3

---

Published in Transactions on Machine Learning Research (02/2026)
Information Bottleneck in Representation Learning The Information Bottleneck (IB) framework (Tishby et al. 1999: formalizes a trade-off' between fidelity and compression in learned representations. A representation is deemed sufficient if it retains all information relevant to the target task, and minimal if it excludes all information irrelovant to it. Formally, given an input X, target variable Y, and representation Z, sufliciency corresponds to Y( = T(X; Y), while minimality requires ([;; = T(Z;Y). The IB principle has been leveraged in a range of works to design regularizers for representation learning (Alemi et al. 2017 2018 Achille & Soatto 2018b [a| Almudévar et al. 2025a), and has also been applied to promote disentanglement (Vera et al. [018 Yamada et al. 2020] Jeon et al 2021 Gao et al. 2021). To the best of our knowledge, however, this is the first work to formally define disentanglement in terms of minimal sufficient representations. In our formulation, each factor of variation is treated as a distinct target variable. Accord- ingly, we partition the overall representation vector into disjoint sets of dimensions, or Subbreepresetaaionss We define a representation as disentangled if each of these sub-representations both minimal and sufficient with respect to its corresponding factor.
3 Desirable Properties of Disentangled Representation
Notation. We denote random variables with capital letters (e.g., X) and their realizations with lowercase letters (e.g., z). Vectors are denoted by bold lowercase letters (e.g., 2) and sets by calligraphic letters (c.g.. X). We use subscript indices to refer to specific components of a vector or set.
Problem Sctup. In our representation learning sctup, we consider raw data X E X, which is fully explained by set of task-relevant factors Y {Yt)== E y and a set of nuisances N E W. The factors Y correspond to the underlying sources or causes that influence the observed data and are relevant to the task at hand, whercas the muisances N refer to sources of variation in X that are irrelevant to the task.
We define a representation Z E Z as a random variable governed by the conditional distribution p(z æ)) This setup induces the Markov chains Y + X + Z and Y, ++ X + Z for all i = L...., n. We further assume that Z can be decomposed into a set of mn variables, Z = { where each Z3 is governed by the conditional distribution p(z z²).
Desirable Propertics Next, we define four desirable properties for a discntangled representation. These properties are conceptually related to those discussed in Section 2 but differ in a crucial aspect: while previous definitions assume independence among factors and from muisances, our formulation explicitly accounts for dependencies. For the following definitions, we assume that 2j is the representation variable intended describe the factor Y,
(i) Factors-Invariance We say that a variable Z, E is factors-invarian with respect to a factor Y when it satisfies the following Markov chain:
Y, Y
(1)
where Y, = (Ya denotes the set of all other factors. This is equivalent to the conditional inde- pendence YZ Y,) O, meaning that once Y, is known, Z, provides additional information about the remaining factors V. In other words, the distribution of Z, is fully determined by the value of Y; alone. This property is directly connected to modularity (Ridgeway & Mozer 2018) (or disentanglement in Eastwood & Williams (2018) ): a representation Z, is said to be modular (or disentangled) if it captures at most one factor Y,. However, such definitions implicitly assume that the factors are independent. When dependencies exist among the factors-as is often the case real-world scenarios this definition becomes problematie. For instance, if two factors Y, and Ye are dependent, then Ys inherently contains information about Y, As result, under the original definition, it would be impossible for Z, to simultancously be modular and fully informative about Y,, since it would necessarily encode some information about Y2 as well.
(ii) Inviancs-- We say that a variable Z, E Z is factor Y; and a set of nuisances N when it satisfies the following Markov chain:
with respect to a
N ++ Y1 ++ 7
(2)
4

---

Published in Transactions on Machine Learning Research (02/2026)
This is equivalent to the conditional independence N / |Y;) 23 remains unaffected by variations in the nuisances. This property is typically neither discussed nor measured in prior work possibly due to the difficulty of estimating the distribution of N. Despite this, it is an important criterion for disentangled representations. While lhat Zy is unaflected by other factors Yx (A: * i), we would also expect 2, to be invariant to aspects of the input that are not factors of variation-i.e the muisances the representation truly disentangled (Carbonneau al.) 2022).
= =0, meaning that once Y; is known,
ensures
(in) pntpesenttations--vviiiitnn invariant for factor Y; when it satisfies the following Markov chain:
We say that variable Zy of the representation Z is represenlations-
2, 2, ++
(3)
where Z, {Za}az denotes all other variables in the representation. This is equivalent to the conditional independence I(Y;; 2, 2,) = 0, which implies that Y, can be entirely predicted from Zj alone, without any additional information from the rest of the representation. This property is useful in downstream applications: for instance, if the goal is to predict Y, one could use Zj in isolation and ignore the remaining components Z. Similarly, in controllable generative models, it would suffice to manipulate Z to control the value of Y; in the generated output X. This notion is closely related to compactness defined in (Ridgeway & Mozer, 2018) (or completeness in Eastwood & WWilliams (2018)), where representation Zj is said to be compact (or complete) if it is the sole component encoding information about factor Y, As with invarannn, this property assumes independence among factors. In realistic scenarios where factors such as Y; and Yr, are dependent, and represented by Z and Z respectively, is natural for both Z5 and Z to carry information about both factors.
(iv) Explicitness We say that a representation Z is explicil for factor Y; when satisfies the following Markov chain:
(4) This means that XIZ = 0, ie.. X provides no information about Y; when Z is known or, oquivalently, Z contains all the information about Y. This property is referred to as explicitness Ridgeway & Mozer (2018) and informativeness in Eastwood & Williams (2018). Unlike the other properties discussed, explicitness is not specifie to disentangled representations it is desirable quality for representations more generally (Bengio et al. 2013).
X+ Z
4 Measuring Disentanglement via Minimality and Sufficiency
In this section, we define minimality and sufficiency, and arguc that directly measuring these propertics is more convenient and informative than evaluating the four previously discussed properties independently. We also provide practical methods for estimating these quantities from data. We now present the formal definitions:
Definition 1. Given a representation variable Z of an input X and a target variable Y, we say that 2 is minimal with respect to Y if it satisfies 1(2; = 0 or, equivalently, if ((; = =;;))==([;;0000000 X(2;X), ie., Z contains information only about Y.
Definition 2. Given representation variable Z of an input X and a target variable Y, we say that Z is sufficient with respect to Y if it satisfies I(Y; X Z) 0 or, equivalently, if I(Z: I(Y; X), i.e., Z contains all information about Y.
4.1 Connection between Disentanglement and Minimality and Sufficiency
We connect below the properties introduced in Section 3 to the concepts of sufficiency and minimality.
Theorem 1. Let Y = (Ye)kzz denote a set of factors and Z = {Zyby= a representation. If Z, is a minimal representation of Y,, it follows that Oj is factors cecttors-ivvriand with respect to Y; and nuisances-imearion.. Equivalently, we have that:
(X Y1 24) _ (Y: Ye +s 23) (N 7,)
5

---

Published in Transactions on Machine Learning Research (02/2026)
where Y; (Ye Therefore, satisfying minimality implics jointly satistying muisances-invariane Intuitively, if Zj is fully determined by Y; (i.e., minimal), then neither the remaining factors Y; nor the nuisances N influence Zj once Y; is known captured by factors- invariance and nuisances- invariance, respectively. A formal proof of this theorem is provided in Appendix A
and
Theorem 2. Let Y; denole a factor and Z = )== a represenlation. Then, Zj is sufficient represen- tation of Y, if and only if Z3 is representations--naarre have:
for Y, and Z is explicit for Y,. Equivalently, we
2z A (X x (x
and since where 2, = [Z Thus, satisfying sufficiency is equivalent to jointly satisfying represetaaioon-invaiiinn and explicitness. Intuitively, if Y; is fully determined by Z, independently of X (i.e., sufficiency), then there is no information about Y; in 2 that is not already contained in 23
Zy E Z, this also implies that Y, is fully captured by Z (explicitness). Conversely, if all the information in and Z captures all information about Y (explicitness), then Zj must fully describe Y; that is, Zz is sufficient. A formal proof of this theorem is provided in Appendix A
Z relevant to Y; already present in Z pppresentatioos-iivraaant
4.2 Why Measuring Disentanglement via Minimality and Sufficiency?
As previously demonstrated, a representation is disentangled according to the properties defined in See- tion 3 if and only if it is both minimal and sufficient. Next, we argue that it is more practical and principled to directly assess sulficiency and minimality, rather than evaluating the four individual properties separalely.
Minimality vs. Innaiane + Nuisances-IIvariance:: When evaluating whether a representation is disentangled, the true objective is to determine whether each of its variables is influenced by single cient for a variable to be aligned with specific factor if it is also significantly affected by nuisance variables. Similarly, a variable that is invariant to nuisances but still influenced by multiple factors cannot be considered disentangled. Therefore, we argue that these two properties should be assessed jointly, and-as establishod Thcorem [ can be effectively achioved by mcasuring minimal- ity. Moreover, analyzing factors and nuisances separately overlooks joint effects, where correlations between remaining factors and nuisances affect 2j collectively despite having no independent influ- ence. Minimality, by contrast, captures these underlying correlations. Furthermore, from a practical standpoint, labels for nuisance variables are seldom available, which prevents the explicit calculation of muisacce-ivvarianc..
of the presence of other factors or muisances. It is not suffi-
Sufficiency vs. Repressetaations--aaaiian representation is disentangled, we aim to determine whether each factor can be both exclusively and fully described by a single variable in the representation. A representation in which a single variable influences a factor but captures only a small portion of its information fails to satisfy the fundamental purpose of a representation daartiionnnmmml..... to describe the underlying factors. Conversely, if a factor is well described by the representation as a whole but cqually influenced by all variables, we cannot claim that the representation is disentangled. For this reason, we argue that it is more meaningful to assess these two aspects jointly rather than separately. As established in Theorem 2 this can be effectively achieved by measuring sufficiency.
+ Explicitness: When evaluating whether
4.3 Defining metrics for Minimality and Sufficiency
At the boginning of this section, defined when representation is minimal or sufficient. In practice, however, represenlations are rarely perfectly minimal or sufficient. It is therefore desirable to design con- tinuous metrics that quantify how close representation is to these ideals. Moreover, while minimality and sufficiency can be defined for each pair (Z,, Y.), it is often useful to summarize these values into single score that characterizes the entire representation Z. We next introduce metrics that meet these requirements.
Note that + denotes "if and only if", whereas ++ represents a dependency structure in a Markov chain.
6

---

Published in Transactions on Machine Learning Research (02/2026)
Minimality Recall that variable Z is minimal with respect to factor Y, if [Z;; Y,) [(z;:)) T(Z;; Y:) = 0. To improve interpretability and facilitate comparison across metrics, we normalize this value by X() to ensure it lies in [0, 1]. We thus define the minimality of Zg with respect to Y; as
1(2;; T(Z ((2;;)) X();
myj
Herc, Maj reflects the fraction of information in 2j that is attributable to factor Y, While these values provide lactor-wise insights, it is common to report a single scalar as a disentanglement score. Assuming, as in Section 3 a. one-to-one correspondence between Zy and a specífic Yi, we define my = max, majs measuring the minimality of Z, relative its most associated factor. The overall minimality score is then obtained by averaging across all representation variables:
Σ m m
1
(6)
Sufficiency Analogously, a variable Zj is sufficient for a factor Y, if [Y;; X Z,) -(Y; 0. Applying the same normalization yields the sufficiency of Zj with respect to Y;:
Stj 1 I(Y; X Zz) I(Y;: X);
(7)
X)
Thus, Sty measures the proportion of information about Y; captured by Zj. Following the same reasoning as for minimality, we compute & = max, 8t,, which evaluates how well the most informative representation variable predicts Y. Averaging across factors yields the overall sufficiency score:
(8)
72
4.4 Computing Minimality and Sufficiency Practice
Calculating the mutual information terms requires access the ground-truth -trth factor labels Y. Furthermore, exact computation can be intractable, particularly when the factor Y, or the represcntation Zj is continuous. To address this, we proceed as follows: (i) discretize Z, by binning, common practice in representation learning (Shwartz-Ziv & Tishby 2017 Locatello et al. 2019b Carbonneau et al. 2022): and (ii) in the less common case where Y; is not categorical, we also discretize it by bimning, while leaving it unchanged when it is categorical. Once both Y, and Zj are categorical, we obtain: (i) I[Z;; Yi) can be computed directly; (ii) [(2;;X] = H(Zz), since Z, fully defined by and therefore HZZ)IX) = 0; and (iii) I(Y; = H(Y)), by the same reasoning.
5 Toy Experiments
Since our goal is to evaluate whether the proposed metrics reliably capture disentanglement, we manually construct the reprosentations in this section's experiments. This design provides full access to the ground truth and, consequently, clear expectations about the metrics' behavior.
We generate two toy experiments where a set of factors is sampled and the corresponding representations are explicitly constructed from them. This controlled setup enables a systematic comparison between our metrics and existing disentanglement metrics under conditions that are particularly challenging for prior approaches, namely: (i) dependence between factors of variation, and (ii) the presence of muisances in the representation. Accordingly, we design two simple experiments, each addressing one of these scenarios.
5.1 Disentanglement under Dependence between Factors of Variation
Definition of the factors We define a set of factors y = {ys) where each ys is constructed from a corresponding set of latent variables (e with e; 2 U100, for i I,..., n. Each factor is constructed

---

Published in Transactions on Machine Learning Research (02/2026)
follows:
1/4 = de; +
1- 5 11 Zy/
Σ
y, - B(yl)
E11
for = 1,...,7h, where 8 [+ 1] is hyperparameter that controls the level of independence among the factors, and the function B I + (1. } bins the continuous values into K discrete classes. In particular: (i) when = 1, each Y/ depends solely on 6, resulting in independent factors; and (ii) when ,,, each y/ is equally influenced by all elements of e, yielding maximal dependence.
Definition of the representations For simplicity, we define representation = { with the same number of variables as factors, where each 2j is intended to correspond to yj. We define each representation variable as:
z = ozj +
Σ (デ
2j = COS
K
for = 1,.... n, where E [ 1] is hyperparameter that controls the degree of disentanglement. Specif- ically: (i) when 1, each 2j depends only yy, resulting in perfectly disentangled representation; (īi) when a == 1 each 2j depends equally on all factors in y. yielding fully entangled representation. The cosine function serves to introduce a non-linearity in the mapping from factors to representations. We note that, since its argument lies in [0, v), the transformation remains bijective, ensuring that no information about the original factor lost.
What results should we expect? Under the described setup, disentanglement metric should satisfy the following desiderata:
(i) The metric should attain a value of 1 when a = 1, regardless of the value of 6. In this case, there is a bijective mapping between each factor y and its corresponding representation variable 2y. implying perfect disentanglement.
(ii) For a fixed value of a, the metric should increase as d decreases. Consider a scenario in which the factors are highly dependent (i.e., low 5), and each representation variable 2j is influenced by all factors (i.c., low a). In such a case, most of the information in 2j can still be attributed to its corresponding factor Mj+ resulting in higher factors-invariance. This is because the other factors contributing to 2y are themselves strongly correlated with yy. For example, in Figure ! suppose the encoder is not perfectly disentangled and 21, intended to reprosent fruit shape, also encodes color. Since banana-shaped fruits are typically yellow, 21 still contains information almost exclusively about shape, despite also carrying color information. Similarly, if does not fully capture shape because of imperfect disentanglement, the fact that it also encodes color indirectly reinforces shape information: if the fruit is yellow, its shape is much more likely to be that of a banana than that of a strawberry.
What results do we observe? Figure 2 shows the results various disentanglement metrics for n = and K- = 5 across different values of and ó. While we report results using a fixed discretization of 20 bins and 5000 data samples, we verify in Appendices B
the metrics' performance is largely insensitive to changes in bin counts and number of samples, respectively. We observe that Minimality and Sufficiency are the only metries that consistently satisfy the desirable properties outlined above. More specifically, we can see that: (i) Metrics such as MIG, SAP, Disenlanglement,, and Completeness fail to reach a value of one when a = 1 and = 1, i.e., even in the fully disentangled and independent case; (ii) Metrics such as Factor VA VAF, MIG, SAP, and Modularity never attain their maximum values as 5 == 1. mcaning they fail correctly reflect disentanglement when correlations are present; and (iii) Disentanglement and Completeness remain largely invariant with respect to d, contrary to the expected behavior discussed earlier.
8

---

Published in Transactions on Machine Learning Research (02/2026)
1
1
1
0.5
0.5
0.5
0.5
0
0
0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
(a) Pactor-VAF
(b) MTG
(c) SAP
(d) Modularity
0.5
0.5
0.5
0.5
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
(e) Disentanglement
(f) Completeness
(g) Minimality (ours)
(h) Sufficiency (ours)
Figure Comparison of different metrics (y-axis) across varying values of a (x-axis) and (color). Colors range from dark blue (highest dependence, 8 1 to dark red (lowest dependence, 1).
5.2 Disentanglement under the Presence of Nuisances
Definition of the factors In this case, define the factors using the same formulation as in Section 5.1 but restrict our analysis to the case 8 = i.c., complete independence between factors.
Definition of the representations We define the representation {2f)"= such that each 2ÿ is intended to correspond to the factor yy. Specifically:
z/ = 1/j + Be,
= cos
where ~ U[0 1] and 8 E in the representation. As before, the argument of the cosine lies in [0, n), ensuring that no information about By is lost. the transformation.
|0,1
1 is hyperparameter that controls the amount of nuisance information
K
What results should we expect? Under this setup, we expect a disentanglement metric that is sensitive to the presence muisances to attain its maximum value when 3 = 0, and to decrease as 3 increases.
What results do we observe? Figure shows 1.0 how different metries vary with respect to 3. We ob- serve that Minimality the only one that is sensi- live to the presence nuisances. While our Suffi- ciency metric remains invariant to changes in B, this is the expected behavior, shown in Theorem 2 which demonstrates that sufficiency is connected to invariance and erplicitness, but not to nuisances- ciiismmoes-invarannee invariance. Finally, we note that Minimal-- ity does not reach zero even at the highest value B, because it. jointly reflects both fuclors-imwarianee and mnisances-invarian and, in this setup, the former re- mains maximal.
Factor-V/E
MIG
Modularity
Disentanglement
Completeness
Minimality
Sufficiency
0.5
0.0
0.4
0.8
Figure Comparison of different metrics (y-axis) for different values of B (x-axis)
9

---

Published in Transactions on Machine Learning Research (02/2026)
5.3 Scalability with Respect to Number of Factors
To assess the practical leasibility of these metrics in high-dimensional scenarios, we evaluate their run- time performance as the number of generative fac- tors increases (Figure D. Regarding the experimen- tal setup, we clarify that Minimality and Sufficiency share a joint estimation step, similar to Disentangle- mend and Completeness (DCI), which why they are displayed as pairs. The results highlight three regimes: (1) Factor- VAE and SAP are the most ef- ficient, maintaining negligible and nearly constant execution times; (ii) DCT suffers from super-linear scaling costs; and (iii) our proposed metrics, while scaling almost lincarly like MIG and Modularity, demonstrate superior elficiency, requiring less com- putation time than these linear baselines.
10
Factor-VAE
MIG
SAP
Modularity
DCI
Min-Suf
5
0
o
0
8
8
8
8
14
20
Figure 4: Time (in seconds) for calculating the differ- ent metrics (y-axis) for different number of factors of variation (x-axis).
6 Neural Representations Experiments
In this section, we compare our proposed metrics against existing ones on representations obtained from neural network-based encoders. Unlike in earlier experiments, ground-truth disentanglement levels are unavailable, precluding direct analysis. Instead, noted in Section T the practical value of disentangled representations often lies in their ability to enable strong performance on downstream tasks. We therefore evaluate the metrics by examiniig their relationship with representation quality across applications typically associated with disentanglement.
To this end, we train models known to induce disentangled representations on three widely used datasets: DSprites (Matthey et al. 2017). MPI3D (Gondal et al., 2019), and Shapes3D (Burgess Kim 2018). For each dataset, we explore six encoder architectures (detailed in Appendix E] combined with six disentan-
glement losses: 8-VAE (Higgins et al. 2017), AnncaledVAE (Burgess et al.) 2018 3-TCVAE (Chen et al.
2018), FactorVAR (Kim & Mnih 2018) Ada-GVAE (Locatello el al. 2020). and HFSVAE (Roth et al. 2023
yielding 36 encoders per dataset. All models are trained using the hyperparameters listed in Appendix
After training, we evaluate how each metric correlates with downstream performance under varying levels of factor dependence. Following Roth et al. (2023), we introduce controlled dependencies among factors, even though the original datasots are independent by design. Specifically, we analyze cases with correlations between one, two, or three pairs of factors, well as confounded setting in which single factor depends all others. A metric suitable for dependent-factor scenarios should satisfy two criteria: (i) it should maintain a strong correlation with downstream performance, and (ii) this correlation should remain stable across different levels of factor dependence. The second condition is particularly critical: since the encoder itself does not change with the degree of factor dependence, a reliable metric should consistently reflect encoder performance irrespective of the underlying factor correlations.
6.1 Accuracy under Data and Computational Constraints
A central motivation for disentangled representations is their expected utility in downstream tasks, partic- ularly under constraints of limited data and computational resources (Schölkopf et al.) 2012 Bengio et al. 2013). Building on Locatello et al. (2019)), we examine whether the proposed metrics correlate with predic- tive performance measured by factor prediction accuracy under two restrictive conditions: (i) predictions roly on single dimension of the representation, and (ii) training performed with only 1000 samples.
What results should we expect? In this sotting, metries designed assess representations-innon -itroiiinnnee and eoplicitness are expected correlate strongly with predictive accuracy. The intuition is that when all the information about factor yi is encoded within a single representation dimension 2j, the mapping from
10

---

Published in Transactions on Machine Learning Research (02/2026)
2j to Ui is straightforward. This implies that even classifier with limited capacity can recover y; reliably from Ej, underscoring the practical value of such representations for downstream prediction.
What results do we observe? Figure reports the rank correlation between disentanglement metrics and predictive accuracy under the low-resource setting described above. When the factors are independent, Sufficiency achieves the strongest correlation with accuracy, while most other metrics also cxhibit moderate to high correlations. This indicales that, in the absence of dependencies, several metrics can serve as reasonable proxies for downstream performance. As factor dependence increases, however, a marked difference emerges: Sufficiency remains stable, whereas the correlations of the other metries decline sharply. These findings demonstrate that Sufficiency provides the most reliable and robust predictor of downstream accuracy among the metrics considered, especially in scenarios where factors are dependent.
No Depend.
Pairs:
Pairs: 2
Pairs: 3
Shared Conf.
1.0
1.0
1.0
0.8
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.0
-02
-02
FactorveE
MIG
Sap
Medulanty
Disentangl
Complet
Mnnmmllly
Suffelencc
FactorVAE
MG
SAP
Modularity
Disentangl
Complet
Minimally
Sufficiency
FactorvAE
HIG
SAP
Modblarity
Disentangl.
Complet.
Minimallyy
suffickency
DSprites
(b)
MPI3D
Shapes3D
Figure 5: Rank correlation between disentanglement metrics and accuracy of random forest classifier trained on 1000 samples across different levels of factor dependence.
6.2 Interventional Robustness
Another motivation for disentangled representations is their robustness to interventions on the underlying factors. Specifically, if factor Y/ is modified while all others remain fixed, the resulting change in the representation should be confined to its corresponding dimension 2j. This property particularly critical for controlled generative models: if it holds, modifying 2j ensures that the generated data changes exclusively in the associated factor ys- To quantify this, Suter et al. (2019) introduced the Interventional Robustness Score (IRS). Formally, IRS evaluates the causal link between a factor ys and latent dimension 2j by comparing post-interventions distributions. A score of 1.0 implies that zy is robust: its value changes if and only if we intervene on Yi (i.c., P(z))do(ys)) + P(z))do(v!))) but remains invariant we intervene on any other factor yayis We next compute the rank correlation between IRS and considered metrics.
What results should we expect? In this sctting, metrics designed to capture fuctors-invariance are expected to correlate positively with IRS. The underlying intuition is that if a representation dimension 2j is entirely determined by factor Vi, ie., p(z, y4) = p(2; xx), then intervention on Y/- while keeping all other factors fixed will affect zj exclusively through y4- In such cases, z; can be regarded robust to interventions, which is precisely the property that IRS is intended to measure.
What results do we observe? In contrast to the accuracy results, Figure i shows that most metrics have weak or even negative correlations with IRS, including when factors are independent, and their correlations vary substantially across dependence levels, sometimes changing sign. By contrast, Minimality exhibits a consistently strong, positive correlation with IRS across all datasets and for every dependence regime considered. The combination of stability and magnitude indicates that Minimality tracks interven- tional robustness far more faithfully than competing metrics.
11

---

Published in Transactions on Machine Learning Research (02/2026)
1.0
1.0
1.0
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.2
Faclorige
MiG
SAP
Modularity
Disentangl
Complet.
Misinality
Sufficiency
Factoria
Mig
SAP
Modularity
Disentanpl
Complet
Minimality
Sufficiency
FactorVag
MIG
SAP
Modularity
Disentangl.
Complet.
Minimality
Sufficiency
DSprites
MPI3D
Shapes
Figure 6: Rank correlation between disentanglement metrics and Interventional Robustness Score across different levels of factor dependence. Legend as in Figure 5
7 Conclusions
In this work, we identified two fundamental limitations of existing definitions and metrics of diseettanglemee (i) the assumption that factors of variation are independent, and (ii) the lack of consideration for invariance to nuisances in the representation. These limitations restrict the applicability of standard metrics in realistic scenarios.
To address them, we proposed four information cheooeti properties that characterize disentanglement in the presence of both dependent factors and muisances. We showed that these properties naturally connect to the concepts of ménimality and sufficiency, and argued that directly measuring these two quantities provides more principled and interpretable characterization of disentanglement. To this end, we introduced novel metrics for quantifying minimality and sufficiency, along with tractable estimators.
Our experiments demonstrate that the proposed metrics capture disentanglement more reliably across a wider range of scenarios than existing alternatives. Moreover, unlike prior metrics, minimality and sufficiency consistently exhibit positive correlation with downstream performance in tasks typically associated with disentangled representations, even in the presence of dependent factors of variation.
8 Limitations and Future Directions
While our framework provides a formal and rigorous definition of disentanglerentt we acknowledge two primary limitations that open avenues for future research.
Diffcrentiability and Optimization. First, the proposed motrics roly on discrete binning to compute mutual information, rendering them non-differentiable. Consequently, they currently serve as evaluation tools rather than objective functions for training. A promising direction to address this the use of variational approximations to derive differentiable upper bounds on the mutual information terms, demonstrated in Alemi et al. (2017). Similarly, adopting approach analogous to Almudévar et al. 2025bb could enable the direct optimization of models towards the minimality and sufficiency criteria.
Scalability to High-Dimensional Factors. Second, the discretipation-baat approximation does not scale well to scenarios where the factors of variation or representations are high-dimensiioal. As the di- mensionality increases, the number bins required grows exponentially, making the estimation intractable. To extend our framework to complex, high-dimensional factors, future work could leverage more robust multivariate estimators, such as the
Seaioov-Stobbue---GG Usable Information metric (Xu et al. 2020)
estimator (Kraskov et al. 2004) or the
12

---

Published in Transactions on Machine Learning Research (02/2026)
Acknowledgments
This work has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Cure grant agreement No 101007666, MCIN/AEI/I 13./999/303/////// 10001 1033 un- der Grant PID2024-1559480B-- and the Government of Aragón (Grant Group T36 23R).
Code Availability
An implementation accompanying this work is available at htt.g//hw...cca/tttttoooo/meaeae dependent dapendentt_disentaggll.. disentanglement
References
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19(50):1-34, 201 80.
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pallern analysis and machine intelligence, 40(12))2897 2905, 2018b.
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In International conference on machine learning, pp. 372-407. PMLR, 2023.
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. In International conference on machine learning, pp. 159-168. PMLR, 2018.
Alexander Alemi, Ian Fischer, Joshua Dillon, and Kevin Murphy. Deep variational information bottle- neck. In International Conference on Learning Representations (ICLR), 2017.
Antonio Almudévar, José Miguel Hernéndez-Lobato, Sameer Khurana, Ricard Marxer, and Alfonso Ortega. Aligning multimodal representations through an information bottleneck. In International Conference on Machine Learning (ICML), 2025a.
Antonio Almudévar, José Miguel Heráández-Lobtte and Alfonso Ortega. There was never bottleneck in concept bottleneck models. arXiv preprint arXin:2506. Xii:25066044877 2025b.
Antonio Almudévar, Théo Mariotte, Alfonso Ortega, and Marie Tahon. Unsupervised multiple domain translation through controlled disentanglement in variational autoencoder. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7010 7014, 2024. doi: 10.1109/ICASSPP AAAAAASSPPAAS5.22221044446666
2024..2222....46
Javier Antoran and Antonio Miguel. Disentangling and learning robust representations with natural clus- tering. In 2019 18th IEFE International Conference On Machine Learning And Applications (ICMLA), pp. 694 699. IEEE, 2019.
Solon Barocas and Andrew D Selbst. Big data's disparate impact. Colif. L. Rev., 104:671, 2016.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798- 11121799--188,, 2013.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. https///ttthbb.cmmdddennn//////
2018 Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 8-vae. arXiv preprint arXiv: 03599, 2018.
Toon Calders and Indré Žliobaité. Why unbiasod computational processes can lead to discriminative dccision procedures. In Discrimination and Privacy in the Information Society: Data mining and profiling in large dalabases, pp. 43 57. Springer, 2013.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Mare-André Carbonneau, Julian Zaidi, Jonathan Boilard, and Ghyslain Gagnon. Measuring disentanglement: A review of metrics. IEEE transactions on neural networks and learning systems, 2022.
Richard J Chen, Judy J Wang, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Ming Y Lu, Sharifa Sahai, and Faisal Mahmood. Algorithmic fairness artificial intelligence for medicine and healtheare. Nature biomedical engineering, 7(6):719 742, 2023.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Jaewoong Choi, Geonho Hwang, and Myungjoo Kang. Discond-vae: Disentangling continuous factors from the discrete. In International Joint Conference on Newral Networks (IJCNN), 2021.
Taco Cohen and Max Welling, Learning the irreducible representations of commutative lie groups. In International Conference on Machine Learning, pp. 1755-1763. PMLR, 2014.
Guillaume Desjardins, Aaron Courville, and Yoshua Bengio. Disentangling factors of variation via generative entangling. arXiv preprint arXwv:1210.5474, 2012.
Kien Do and Truyen Tran. Theory and evaluation metrics for learning disentangled represontations. In International Conference on Learning Representations (ICLR), 2020.
Emilien Dupont. Learning disentangled joint continuous and discrete representations. Advances in neural information processing systems, 31, 2018.
Cian Eastwood and Christopher KI Williams. framework for the quantitative evaluation of disentangled representations. In International conference on learning representations, 2018
Gege Gao, Huaíbo Huang, Chaoyou Fu, Zhaoyang Li, and Ran He. Information bottleneck disentangle- ment for identity swapping. In Proceedings of the IEFE/CVF conference on compuler vision and pattern recognition, pp. 3404 3413, 2021.
Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explain- ing explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80-89. IEEE, 2018
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer, On the transfer of inductive bias from simulation the real world: new disenlanglement dataset. Advances in Newral Information Processing Systems, 32, 2019.
Trina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster), 3, 2017.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable repre- sentations from sequential data. Advances in neural information processing systems, 30, 2017.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised imago-to-image trans- lation. In Proceedings of the European conference computer vision (ECCV), pp. 172 189, 2018.
Insu Jeon, Wonkwang Lee, Myeongjang Pyeon, and Gunhee Kim. Ib-gan: Disentangled representation learn- ing with information bottleneck generative adversarial networks. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 35, pp. 7926-7934, 2021.
Hadi Kazemi, Scyed Mchdi Tranmanesh, and Nasser Nasrabadi. Style and content disentanglement genera- tive adversarial networks. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 848 856. IEFE, 2019.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International conference machine learning, pp. 2649-2655 2658. PMLR, 2018.
Minyoung Kim, Yuting Wang, Pritish Sahu, and Vladimir Pavlovic. Relevance factor vae: Learning and identifying discntangled factors. arXiv preprint 1.X2::9002.111 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E Statiistiaal, Nonlinear, and Soft Matter Physics, 69(6):066138, 2004.
Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convolutional inverse graphics network. Advances in neural information processing systems, 28, 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations (TCLR), 2018.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invari- ance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. II-104. IEEE, 2004.
Wenqian Liu, Runze Li, Meng Zheng, Srikrishna Karanam, Ziyan Wu, Bir Bhanu, Richard Radke, and Octavia Camps. Towards visually explaining variational autoencoders. In Proceedings of the IEEE/CVP conference computer vision and pattern recognition, pp. 8642-8657 8651, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem. On the fairness of disentangled representations. Advances in neural information processing systems, 32, 2019a.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled represen- tations. In international conference machine learning, pp. 41244. PMLR, 2019b.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschan- nen. Weakly-supervised disentanglement without compromises. In International conference on machine learning, pp. 6348-6359. PMLR, 2020.
Loic Matthey Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://github.com wttpe//ido.oeeeeeeednddee/ee sisppiinn//dspritssddaaaseee 2017.
Milton Montero, Jeffrey Bowers, Rui Ponte Costa, Casimir Ludwig, and Gaurav Malhotra. Lost in latent space: Examining failures of disentangled models at combinatorial generalisation. Advances in Neural Information Processing Systems, 35:10136-100 10149, 2022.
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2020.
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. From anecdotal evidence to quantitative evaluation methods: A systematic review evaluating explainable ai. ACM Compuling Surveys, 55(13s):114p) 2023.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Brooks Paige, Jan- Willem Van De Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, Philip Torr, et al. Learning disentangled representations with semi-supervised deep generative models. Advances in neural information processing systems, 30, 2017.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual analogy-making. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Karl Ridgeway. A survey of inductive biases for factorial representation-learning ar preprint arXiv:1612. 05299, 2016.
Karl Ridgeway and Michael C Mozer. Learning deep disentangled embeddings with the f-statistic loss. Advances in neural information processing systems, 31, 2018.
Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentanglement of correlated factors via hausdorff factorized support. In International Conference on Learning Representa- tions (ICLR), 2023.
Mhd Hasan Sarhan, Nassir Navab, Abouzar Eslami, and Shadi Albarqouni. Fairness by learning orthogonal disentangled representations. In Computer EssionEECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXIX 16, pp. 746 761. Springer, 2020.
Jürgen Schmidhuber. Learning factorial codes by prodictability minimization. Neural computation, 4(6):
1992.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In International Conference on Machine Learning (ICML), 2012.
Anna Sepliarskaia, Julia Kiseleva, Maarten de Rijke, et al. Evaluating disentangled representations. arXiv preprint (rXie:1910.05550 2019.
Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang, Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23(241): 1 55, 2022.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprinl Xrx 000000 2017.
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, pp. 6056 6065. PMLR, 2019.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. In The 37th Annual Allerton Conference on Communication, Control, and Computing, pp. 368 377, 1999.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bern- hard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In International conference on machine learning, pp. 10401-10412. PMLR, 2021.
Matias Vera, Pablo Piantanida, and Leonardo Rey Vega. The role of the information bottleneck in repre- sentation learning. In 2018 IEEE international symposium on information theory (ISIT), pp. 1580-1584. IEEE, 2018
Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Bareinboim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. Advances in Neural Information Processing Systems, 36, 2024.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast docoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiw:1901.07011, 2019.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetor, and Gabriel J Brostow. Interpretable transformations with encoder-decoder networks. In Proceedings of the TEEE International Conference Computer Vision, pp. 5726 5735, 2017.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In International Conference Learning Representations, 2020.
Masanori Yamada, Heecheol Kim, Kosuke Miyoshi, Tomoharu Iwata, and Hiroshi Yamakawa. Disentangled representations for sequence data using information bottleneck principle. In Asian Conference on Machine Learning, pp. 305 320. PMLR, 2020.
Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Attribulelinage: Conditional image generation from visual attributes. In Computer Ssioo-ECCVV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 776-791. Springer, 2016.
Quan-shi Zhang and Song- Chun Zhu. Visual interpretability for deep learning: a survey. Frontiers of Information Technology & Electronic Engineering. 19(1):27 39, 2018.
Sharon Zhou, Eric Zelikman, Fred Lu, Andrew Y Ng. Gunnar Carlsson, and Stefano Ermon. Evaluating the disenlanglement of deep generalive models through manifold topology. In International Conference on Learning Representations (ICLR), 2021.
Xinqi Zhu, Chang Xu, and Dacheng Tao. Where and what? examining interpretable disentangled represen- tations. In Proceedings of the TEEE/CVE Conference Computer Vision and Pottern Recognation, pp. 5861 5870, 2021.
17

---

Published in Transactions on Machine Learning Research (02/2026)
A Proofs of Theorems
Theorem 1. Let y = Tu denote a set of factors and z = {zy1y= a representation. If 2j is min- imall represenlation of ye, it follows that 2y is faclors-imwarianl with respect to and muisances-imooriann Equivalently, we have that:
(+ + +)
= (ū. + + 25) A (n ++ yi + 22)
Proof. First, we demonstrate that (x + yi + 24) = (Ü: + ++ 2j): Given this Markov chain, by the DPI, we have that x(2;;)) I [(2): Since 2j is representation of æ, we also have by DPI that [[zy: v) & x)) Therefore, it follows that ((z;i)) < I (z On the other hand, by the chain rule of mutual information, we have that (z);3) (2 Ø:) Uz); + (2j Dzlvi). By the negatiiatyv of mutual information, we are left with (2)5g) 2 (2) Therefore, we have that [(z v.) (2) y) [(zi yes 9t) or, equivalently, is Y ++ 2j-
Second, we demonstrate that (æ: ++ y1 ++ -j) = (n <> yr ++ 2j): Given this Markov chain, we know from the DPI that (2z; x) I (1) On the other hand, since is representation of x, we have the Markov chain (n, 34) + x + 2j. Equivalently, we have that [(2y: n, 34) I I(2;:z). The chain rule of mutual information tells us that n(y; = [[z;: n. yi) - (=z]1))). According to the above two points, we are left with L (zz; nly;;) S ((2):=) - (z)lyt). Therefore, because of the non-negativity of mutual information, is or, equivalently,
only possible that [2zj
y/ ++ Zz.
Theorem 2. Let ys be a factor and = }m a representation. Then, 2j is sufficient representation of ys if and only if 2j is representations-inaaare invariant for ys and z is explicit for y:- Equivalently, we have the that:
(x + žj + v:)
(2, + + 11) A (xc + 2 ++
Proof. First, we demonstrate that (x + 2g + wi) - (2) + 2j ++ 3:): Given this Markov chain, by the DPI, we have that [(;;:)) I I(y: 2,). Since z is a representation of x, we also have that I(y;: x) I [[; x). Therefore, follows that I( I ((y()2) On the other hand, by the mutual information chain rule, we have that T(y;; 2) = [1g; 2,, E3) = T(y: 2j) + T(y: Eyle)). By the non-negativity of mutual information, we are left with (3)1 2 (1)62)). Therefore, we have that (y))))) - [yyi 2) [y9 zj. āz) or, equivalently, 2, 23 + V/-
Second, we demonstrate that (xr ++ Zj ++ vi) = (x + 2 + y:): Since 2j is a representation of æ, we have that p(y xr, 2;) = P(u())). Moreover, since 2j is sufficient, we have that p(y: [æ, 25) = p(wilzz). Putting the above two terms together, we are left with p(y]( = p(N(2z)) As we have already shown, if zj is sufficient, then we have the Markov chain (ž, + 2j + yi), so p(u]2)= = p(ua(zz). Therefore, we are left with p(y.lz)= p(u]az). Finally, since x is a representation of æ, we know that p(y)(x) = p(,)x, Putting all of yt-
the above together, we are left with p(yi [r, 2) p(us 12) or, equivalently,
Finally, we demonstrate that (žj + 2j + 31) A (a + = + v.) (x + 2j + n:): Since 2j and are representations of x, we have that p( = p(u z) = p(ulax). Furthermore, given the Markov chain x + z ++ Vt. we know that p(y] = p( 2) Equivalently, given E, ++ 24 + yı we have that p(u)l=,)= = = (z)) Recapitulating the above, we are left with p(u))z,) = p(u) = p(v (z,)) = p(u: x,, 23) Therefore, we have that x ++ 2j ++ yi-
18

---

Published in Transactions on Machine Learning Research (02/2026)
B Influence of Number of Bins
B.1 Toy Experiment 1 (Section 5.1
As observed in Figures and increasing the number of bins renders our metrics less sensitive to factor correlations (a), while still consistently satisfying the expected behavior described in Section
0.5
0.5
0.5
0.5
0.5
0
0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
(a) 10 Bins
(b) 15 Bins
(c)
Bins
(d) 25 Bins
(e) 30 Bins
Figure 7: Comparison of Minimality for different number of bins (y-axis) across varying values of a (x-axis) and ò (color). Colors range from dark blue (highest dependence, 8 = A to dark red (lowest dependence, ==10 1).
0.5
0.5
0.5
0.625
0.25
0.625
1.0
0.25
0.625
0.25
0.625
1.0
0.25
0.625
1.0
(a) Bins
(b) 15 Bins
(c) Bins
(d) 25 Bins
(e)
Bins
Figure 8: Comparison of Sufficiency for different number of bins (y-axis) across varying values of a (x-axis) and d (color). Colors range from dark blue (highest dependence, 6 1) to dark red (lowest dependence, 1).
B.2 Toy Experiment 2 (Section 5.2
In contrast to the previous section, Figure Snuureereveen reveals the opposite trend: increasing the number bins makes Minimality more sensitive to (the amount of nuisances the representation). Crucially, however, this increased sensitivity does not violate the expected behavior described in Section 5.2 On the other hand, Sufficiency remains invariant to the value of 3, regardless of the number of bins used.
1.0
1.0
10 Bins
10 Bins
15 Bins
15 Bins
25 Bins
25 Bins
30 Bins
30 Bins
0.5
0.5
0.0
0.4
0.8
0.0
0.4
0.8
(a) Minimality
(b) Sufficiency
Figure 9: Comparison of Minimality and Sufficiency for different number of bins (y-axis) for different values of (x-axis)
19

---

Published in Transactions on Machine Learning Research (02/2026)
B.3 Neural Representations Experiment (Section 6)
Figures 10 and 11 demonstrate that the rank correlations for key metric pairs are robust to discretization choices. Specifically, the correlation between Sufficiency and accuracy under low data and computational resources, and between Minimality and IRS, remain almost invariant to the number of bins used for calcu- lation.
No Depend.
Pairs:
Pairs: 2
Pairs: 3
Shared Conf.
1.0
1.0
1.0
0.6
0.6
0.6
0.4
0.4
0.4
0.0
0.0
0.0
Bims
20 BYS
Bins
Bins
15 Bins
20 Bins
25 Bins
30 Bins
Biss
15 Bins
20 Biss
s0 Bins
10
10
25
DSprites
(b)
MPI3D
Shapes3D
Figure 10: Rank correlation between Sufficiency for different numbers of bins and accuracy of a random
forest classifier trained on 1000 samples across different levels of factor dependence.
No Depend.
Pairs:
Pairs: 2
Pairs: 3
Shared Conf.
1.0
1.0
1.0
0.8
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
Birs
Bins
20 Bins
Birs
Bing
gine
15 Bins
Bies
25 Bina
30 Bins
Bins
15 Bina
20 Bina
25 Bira
Bina
(a)
DSprites
(b) MP13D
Shapes3D
Figure 11: Rank correlation between Minimality for different numbers of bins and Interventional Robustness
Score across different levels of factor dependence.
20

---

Published in Transactions on Machine Learning Research (02/2026)
C Influence of the Number of Samples
C.1 Toy Experiment 1 (Section 5.1
As shown in Figures and of samples, remaining consistent regardless of the factor correlations determined by a.
and 13 the proposed metrics exhibit remarkable stability with respect to the number
1
0.5
0.5
0.5
0.5
0
0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
(a) 1000 Samples
(b) 2000 Samples
(c) 5000 Samples
(d) 10000 Samples
Comparison of Minimality for different number of samples (y-axis) across varying values of a (x- axis) and 8 (color). Colors range from dark blue (highest dependence, 1 to dark red (lowest dependence,
Figure 12:
0.5
0.5
0.5
0.5
0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
0.25
0.625
1.0
(a) 1000 Samples
(b) 2000 Samples
(c) 5000 Samples
(d) 10000 Samples
Figure 13: Comparison of Sufficiency for different number of samples (y-axis) across varying values of a (x- axis) and 8 (color). Colors range from dark blue (highest dependence, _ to dark red (lowest dependence, 1).
C.2 Toy Experiment (Section 5.2
As shown in Figure 14 the proposed metrics exhibit remarkable stability with respoct the mumber of samples, remaining consistent. regardless of the level of nuisances determined by B.
1.0
1.0
1000 Samples
1000 Samples
2000 Samples
2000 Samples
5000 Samples
5000 Samples
10000 Samples
10000 Samples
0.5
0.5
0.0
0.4
0.8
0.0
0.4
0.8
(a) Minimality
(b) Sufficiency
Figure 14: Comparison of Minimality and Sufficiency for different number of samples (y-axis) for different
values of 3 (x-axis)
21

---

Published in Transactions on Machine Learning Research (02/2026)
C.3 Neural Representations Experiment (Section 6
Figures 15 and 16 demonstrate that the rank correlations for key metric pairs are robust to number of samples. Specifically, the correlation between Sufficiency and accuracy under low data and computational resources, and between Minimality and IRS, remain almost invariant to the number of samples.
No Depend.
Pairs:
Pairs:
Pairs:
Shared Conf.
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
-0.2
1000 Samples
2000 Sampies
Sotn Samples
lonop Samples
1600 Samples
2000 Samples
s00o Samples
Ianon Samples
100p Samples
2000 Samples
somn Samples
10000 Samples
DSprites
(b) MP13D
(c) Shapes3D
Figure 15: Rank correlation between Sufficioncy for different numbers of samples and accuracy of a random Torest classifier trained on 1000 samples across different levels of factor dependence.
No Depend.
Pairs:
Pairs: 2
Pairs: 3
Shared Conf.
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.2
-0.2
002
1000 Samples
2000 Samples
5000 Samples
10009 Samples
1000 Samples
2000 Samples
s0oo Samples
10n90 Samples
1009 Samples
2000 Samples
5000 Samples
10000 Samples
DSprites
MP13D
Shapes3D
Figure 16: Rank correlation between Minimality for different numbers of samples and Interventional Ro-
bustness Score across different levels of factor dependence.
22

---

Published in Transactions on Machine Learning Research (02/2026)
D Training Hyperparameters
All training hyperparameters are listed in Table
Table 1: Training hyperparameters.
Hyperparameter Value
Batch size
256
Number of epochs
50
Optimizer
Adam
Base batch size
64
Learning rate
10
Weight decay
10
Scheduler
StepLR
Step size
20
Gamma
0.1
E VAE Architectures
All our architectures are based on VAE approaches. Below, we summarize the six architectures considered, with example output shapes provided for a 64 x 64 RGB input image (C==).
E.1
Architecture Following Burgess et al. (2018)
The encoder described in Table 2 and the decoder in Table 3 are used.
Table 2: Encoder architecture
Burgess et al. 2018
Layer
Type
Parameters
Output shape
Activation
Input
Image size (C, H, W)
(B, 3, 64, 64)
Convl
Conv2d
C' + 32, kernel=4, stride=2, pad=1
(B, 32, 32, 32)
ReLU
Conv2
Conv2d
32 + 32, kernel=4, stride=2, pad=1
(B, 32, 16, 16)
ReLU
Conv3
Conv2d
32 - 32, kernel=4, stride=2, pad=1
(B, 32, 8,
ReLU
C
Conv2d
32 > 32, kernel=4, stride=2, pad=1
(B, 32, 4,
ReLU
Flatten
(B, 512)
FC1
Linear
512 256
(B, 256)
ReLU ReLU
FC2
Linear
256 256
(B, 256)
FC3
Lincar
256 latent dim x
B, latent dim x 2)
Output
Reshape
(B, latent dim, 2)
Table Decoder architecture Burgess et al.
2018
Layer
Type
Parameters
Output shape
Activation
Input
Latent vector E glatent dim
(B, latent dim)
FC1
Linear
latent dim 256 256 256
(B, 256)
ReLU
FC2
Linear
(B, 256)
ReLU
FC3
Linear
256 32 x 4
(B, 512)
ReLU
Reshape
(B, 32, 4, 4)
Conv'T 64*
Conv TonyTranspose2d
32 + 32, kernel=4, stride=2, pad=1 1
(B, 32, 8, 8)
ReLU ReLU
C1
Conv ConnTranspose2
32 + 32, kernel=4, stride=2, pad=1
(B, 32, 16, 16)
C2nvT2
Cony'Trasee Tronsrrssp
32 + 32, kernel=4, stride-2, pad=1
(B, 32, 32, 32)
ReLU
ConvT3
Conv CouwTranspos2d
32 + C, kernel=4, stride=2, pad=1
(B, 64,
Sigmoid
23

---

Published in Transactions on Machine Learning Research (02/2026)
E.2
Architecture Following
Chen et al. (2018)
The encoder described in Table 4 and the decoder in Table 5 are used.
Table MLP encoder architecture (Chen et al. 2018).
Layer
Type
Parameters
Output shape
Activation
Input
Image flattened
(B, 64 x 64 x 3 = 12288)
FC1
Linear
12288 + 1200
(B, 1200)
ReLU ReLU
FC2
Lincar
1200 + 1200
(B, 1200)
FC3
Linear
1200 - latent. dim 2
(B, latent. dim 2)
Output
Reshape
(B, latent dim, 2)
Table 5: MLP decoder architecture
Chen et al. 2018
Layer
Type
Parameters
Output shape
Activation
Input
Latent vector E IR latent dim
(B, latent dim)
FC1
Linear
latent dim + 1200
(B, 1200)
Tanh
FC2
Linear
1200 - 1200
(B, 1200)
Tanh
FC3
Linear
1200 - 1200
(B, 1200)
Tanh
FC4
Linear
1200 - 64 x 64 x 3 12288
(B, 12288)
Sigmoid
Output
Reshape
(B, 3, 64, 64)
E.3 Architecture Following Locatello et al. (2020)
The encoder described in Table and the decoder in Table 7 are used.
Table Encoder architecture
Locatello et al.
2020
Layer
Type
Parameters
Output shape
Activation
Input
Image size = (C, 64, 64)
(B, C, 64, 64)
Convl
Conv2d
C + 32, kernel=4, stride=2, pad-1
(B, 32, 32, 32)
ReLU
Conv2
Conv2d
32 + 32, kernel=4, stride=2, pad=1
(B, 32, 16, 16)
ReLU
Conv3
Conv2d
32 + 64, kernel=4, stride=2, pad=1
(B, 64, 8, 8)
ReLU
Conv4
Conv2d
64 + 64, kernel=4, stride=2, pad=1
(B, 64, 4, 4)
ReLU
Flatien
(B, 1024)
FC1
Linear
1024 + 256
(B, 256)
ReLU
FC2
Linear
256 > latent dim x
(B, latent dim x 2)
Output
Reshape
(B, latent dim, 2)
Table 7: Decoder architecture
(Loaaeelo et al.
2020h
Layer
Type
Parameters
Output shape
Activation
Input
Latent vector Iglateut dim latent dim + 256 256 + 64 4 x 4 = 1024
(B, latent dim)
FC1
Linear
(B, 256)
ReLU
FC2
Linear
(B, 1024)
ReLU
Reshape
(B, 64, 4, 4)
Conv
Conv ConyTranspose2
64 - 64, kernel=4, stride=2, pad=1
(B, 64, 8, 8)
ReLU
ConvT2
Conv ConyTransparn Transpose2d
64 + 32, kernel=4, stride=2, pad=1
(B, 32, 16, 16)
ReLU
ConvT3
Conv Transpossp
32 > 32, kernel=4, stride=2, pad=1
(B, 32, 32, 32)
ReLU
C
Conv' Thanspospe
32 + C, kernel=4 stride-2, pad-1
(B, C, 64, 64)
Sigmoid
24

---

Published in Transactions on Machine Learning Research (02/2026)
E.4
Architecture Combining Locatello et al. (2020) and Watters et al. (2019)
The encoder described in Table i and the decoder in Table 8 are used.
Table 8: Spatial Broadcast Decoder [Watters el al. 2019
Layer
Parameters
Output shape
Activation
Input
Latent vector
(B, latent latent_dimi
Spatial Broadcast
Tile latent + concal XY mesh
(B, latent dim++2 64, 64)
Conv1
(latent dim 2) 64, kernel-5,
(B, 64, 64, 64)
ReLU
stride=1, pad=2
Conv2
64 + 64, kernel=5, stride=1, pad-2
(B, 64, 64, 64)
ReLU
Conv3
64 + 64, 6 stride=1, pad=2
(B, 64, 64, 64)
ReLU
Conv4
64 + 64, kernel=ū, stride=1, pad=2
(B, 64, 64, 64)
ReLU
Conv5
C, kernel=5, stride=1, pad=2
(B, C, 64, 64)
Sigmoid
E.5
Small Architecure Following Montero et al. (2022)
The encoder described in Table g and the decoder in Table are used.
Table 9: Small encoder architecture (Montero et al. 2022
Layer
Type
Parameters
Output shape
Activation
Input
Image size (C, 64, 64)
(B, C, 64, 64)
Convl
Conv2d
C 32, kernel=4, stride=2, pad=1
(B, 32, 32, 32)
ReLU
Conv2
Conv2d
32 + 32, kernel-4, stridc-2, pad=1
(B, 32, 16, 16)
ReLU
Conv3
Conv2d
32 + 64,
stridde,,srriee--- pad=1
(B, 64, 8, 8)
ReLU
Cony4
Conv2d
64 + 64, kernel=4, stride=2, pad=1
(B, 64, 4, 4)
ReLU
Conv5
Conv2d
64 + 128, kernel=4, stride=2, pad=1
(B, 128, 2)
ReLU
Flatten
(B, 512)
FC1
Linear
512 256
(B, 256)
ReLU
FC2
Linear
256 latent dim x 2
(B, latent dim 2)
Output
Reshape
(B, latent dim, 2)
Table 10: Small decoder architecture
Montero et al.
2022
Layer
Type
Parameters
Output shape
Activation
Input
Latent vector E IR Iglatent_ding dim
(B, latent dim)
FC1
Linear
latent dim + 256 256 + 128 2 x = 512
(B, 256)
ReLU ReLU
FC2
Linear
(B, 512)
Reshape
(B, 128, 2, 2)
ConvTT
ConvTTransposel
128 64, kernel=4, stride-2, pad-1
(B, 64, 4, 4)
ReLU
ConvT2
ConyTransppar Transppspdd
64 64, kernel=4, stride=2, pad=1
(B, 64, 8,
ReLU
ConvT3
Conv Transpossd
64 32, kernel=4, stride=2, 2, pad=1
(B, 32, 16, 16)
ReLU
ConvT4
ConyTransposele
32 + 32, kernel=4, stride=2, pad=1
(B, 32, 32, 32)
ReLU
ConvT5s T5
ConyTranspann Transpose2d
32 + C, kernel=4, stride-2, pad=1
(B, C, 64, 64)
Sigmoid
25

---

Published in Transactions on Machine Learning Research (02/2026)
E.6
Large Architecure Following Montero et al. (2022)
The encoder described in Table 11 and the decoder in Table 12 are used.
Table 11: Large encoder architecture Moontero et al.
2022 ).
Layer
Type
Parameters
Outpul shape
Activation
Input
Image size (C, 64, 64)
(B, C, 64, 64)
Convl
Conv2d
C' + 64, kernel=4, stride-2, pad=1
(B, 64, 32, 32)
ReLU
Conv2
Conv2d
64 - 64, kernel=4, stride=2, pad=1
(B, 64, 16, 16)
ReLU
Conv3
Conv2d
64 + 128, kernel=4, stride=2, pad=1
(B, 128, 8,
ReLU
Conv4
Conv2d
128 + 128, kernel=4, stride=2, pad=1
(B, 128, 4, 4)
ReLU
Conv5
Conv2d
128 256, kernel-4, stride-2, pad-1
(B, 256, 2)
ReLU
Flatten
(B, 1024)
FC1
Linear
1024 256
(B, 256)
ReLU
FC2
Linear
256 + latent dim x 2
[B
2)
Output
Reshape
(B latent dim, 2)
Table 12: Large decoder architecture
Montero et al.
2022
Layer
Type
Parameters
Output shape
Activation
Input
Latent vector E platent_ding dim
(B, latent dim)
FC1
Linear
latent. dim - 256 256 256 x 1024
(B, 256)
ReLU ReLU
FC2
Linear
(B, 1024)
Reshape
(B, 256, 2, 2)
ConvT1
ConyTranspose2d
256 128, kernel=4, stride=2, pad=1
(B, 128, 4, 4)
ReLU
Conv T2
ConvTranspose22
128 >> 128, kernel=4, stride-2, pad=1
(B, 128, 8, 8)
ReLU
Conv'I T3
Conv TonnTrraspose2d
128 - 64, kernel=4, stride-2, pad-1
(B, 64, 16, 16)
ReLU
Cony
ConyTrans Transpoaspse2l
64 + 64, kernel=4, stride=2, pad=1
(B, 64, 32, 32)
ReLU
Conv
Conv CrnnTranspose2
+ C, kernel=4, stride=2, pad=1
(B, C, 64, 64)
Sigmoid
26


Reviews:
[{"review_id": "0JHCbSNF1G", "reviewer": "Reviewer_AQT3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "COPYING MY COMMENT AS A REVIEW\n\nThe authors are focusing in disentangled representations, in the setup where latent factors are dependent and nuisances appear. Drawing from the current stage of popular metrics e.g. mIG, SAP,DCI that assume independent generative factors, the authors challenge the metrics' ability to represent real-wolrd settings. For that they propose 4 information-theoretic properties that a disentangled representaton should satisfy. They run toy experiments, they construct settings with increasing nuisance information to assess which metrics react appropriately. Overall, Sufficiency tends to correlate best and most robustly with downstream accuracy as dependence increases, and Minimality correlates most reliably with IRS.\n\nPros:\n\nThe motivation is clear. The paper does a good job on presenting the problem the independent factor assumption.\nThe synthetic experiments seem reasonable. The variability of the disentaglement strength , factor dependence, and nuisance injection is well defined.\nI like the way the authors break disentanglement down into explicit information-theoretic properties. It's an intuitive way to tie them back to sufficinet representations.\nThe correlation analysis with low-data prediction and IRS is convincing.\nThe paper is clearly written and straightforward to follow.\nCons:\n\nAlthough some of these assumptions that the authors make are standard in disentanglement benchmarks, the conditional independence of factors given \n one seems unusual to me and the authors should use more justification and discussion. It would help to make clear which results really depend on it, and whether any weaker assumptions would suffice.\nthe metrics are relying on discretizing \n and sometimes \n, and then on using histogram-based MI. I'm concerned that this choice might be sensitive to a) the numer of the bins b) sample size, c) and the distributional shape of the varialbes. Since the contribution is an evaluation metric, I would really like to see at least a brief robustness study here.\nThe main link—disentanglement as minimal and sufficient representations in the information-bottleneck sense—has been touched on in prior work, so the main novelty of the paper defining concrete evaluation metrics that make these connections concise. I think this is still a meaningful novelty, but the paper might want to present itself more as “a solid and principled evaluation framework” rather than suggesting a completely new theory of disentanglement.\nQuestions:\n\nCan you calrify how siginificant is the conditional independence $p(y_i, y_j | x) = p(y_i| x) p(y_j|x) ?\nPossibly a moving-forward question on the possibility for distributed representations, rather than concern: Do you see a way to extend your framework to handle the case where a factor is represented by a group of coordinates rather than a single one?\nHow sensitive are the Minimality and Sufficiency scores to the binning and sample size? Have you tried different bin counts or alternative MI estimators?\nOverall, I'm leaning positive toward this submission: the paper discusses a real gap in how the community evaluates disentanglement when factors are dependent and nuisances are present, and the information-theoretic formulation is clean.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Explained above", "audience_yn": "Yes", "audience_explanation": "Explained above", "requested_changes": "Mentioned above", "broader_impact_concerns": "Explained above", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fpG71q8VqX", "reviewer": "Reviewer_2H2y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies disentangled representations in settings where **generative factors are statistically dependent and nuisances are present**, arguing that many popular metrics (MIG, SAP, DCI, etc.) are calibrated to independence-heavy benchmarks and can behave unreliably in these regimes.\nIt introduces **four information-theoretic properties** (factors-invariance, nuisances-invariance, representations-invariance, and explicitness) and shows that, taken together, they are equivalent to each coordinate being a minimal and sufficient representation of a single factor.\nBased on this, the authors define **two concrete metrics** implemented via mutual information estimates from discretized codes and factors.\nThrough synthetic experiments with controlled factor dependence and nuisance strength, they report that sufficiency correlates best and most robustly with downstream predictive performance, while minimality tracks an existing interventional robustness score more reliably than other disentanglement metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core positive claims are reasonably well supported within the scope of the paper. The authors clearly specify their four information-theoretic properties, derive the minimality and sufficiency scores from mutual information identities, and then design synthetic experiments that systematically vary disentanglement strength, factor dependence, and nuisance strength. I find the evidence adequate.\n\nHowever, some of the negative claims about prior work feel more rhetorical than demonstrated.\nThe paper states that existing definitions \n> (i) they do not account for invariance to nuisance factors that may be present in the input; and\n> (ii) they assume that the factors of variation are independent of one another and of the nuisances—which generally does not hold in real-world data.\n\nbut this is asserted rather than carefully proved.\nClassical notions such as modularity/compactness/explicitness are mostly structural properties of the mapping $Y \\to Z$ from factors to representations; they are typically defined without explicitly fixing which factors are \"nuisances\" or requiring independence of the factor distribution.\n\n`Bengio et al. (2013)` stated that\n> the most robust approach to feature learning is to disentangle as many factors as possible, discarding as little information about the data as is practical\n\nso \"nuisance factors\" may be useful in some other tasks.\nFurther, the definitions given in some previous works are rather vague.\nIf the author wants to criticize the existing definitions or metrics, they should first give a clear mathematical formulation that best aligns with the original papers.\nIt would strengthen the paper if the authors either (i) softened this rhetoric and framed their contribution as extending these notions to explicitly model nuisances and dependent factors, or (ii) provided concrete counterexamples showing a representation that is intuitively disentangled in their sense but necessarily scored as \"bad\" by previous definitions/metrics.\n\nFinally, the support for the proposed metrics themselves is somewhat limited by implementation choices.\nMinimality and sufficiency are estimated via discretization and histogram-based mutual information, but there is no robustness study on sensitivity to bin counts, sample size, or distributional shape.\nSince the central contribution is an evaluation metric, I would have liked at least a brief analysis or ablation here.\nMoreover, all empirical evidence is on synthetic data with known factors and nuisances; this is appropriate for an initial study, but it means that stronger claims about real-world applicability should be read as suggestive rather than definitively established.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a real and currently under-served niche: how to evaluate disentangled representations when factors are dependent and nuisances are present, instead of assuming the independent factor, fully-observed regimes of standard benchmarks.\nThis is clearly relevant to parts of the TMLR audience working on disentangled/causal representation learning, identifiability of representations, and evaluation methodology.\n\nThe idea of starting from logical/information-theoretic properties and then deriving concrete quantitative metrics also connects to a growing line of work on principled disentanglement evaluation, as well as to recent efforts on more realistic disentanglement/causal representation learning setups.\n\n- Cian Eastwood, et al. \"DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability.\" International Conference on Learning Representations. 2023.\n- Yivan Zhang, Masashi Sugiyama. \"Enriching Disentanglement: From Logical Definitions to Quantitative Metrics.\" Neural Information Processing Systems. 2024.\n- Sébastien Lachapelle, et al. \"Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies.\" arXiv:2401.04890. 2024.\n- Dingling Yao, et al. \"Multi-View Causal Representation Learning with Partial Observability.\" International Conference on Learning Representations. 2024.", "requested_changes": "### Clarify the formal status of the metrics\n\nSince the main contribution is an evaluation metric, I would like to see basic theoretical guarantees for minimality and sufficiency.\nFor example, explicit propositions of the form \"the minimality score for $(z_j, y_i)$ equals $1$ if and only if factors-invariance and nuisances-invariance hold\" etc.\nRight now these relationships are described informally; turning them into clear metric-property statements would make the framework more principled and easier to interpret.\n\n### Discuss differentiability, optimisation, and computational cost\n\nThe current implementation uses discretization and histogram-based mutual information, which is fine for post-hoc evaluation but not directly usable as a training objective.\nIt would improve the paper to comment on whether differentiable estimators of the same quantities are feasible, and whether in principle one could optimize disentanglement directly with these scores as losses.\nRelatedly, a brief discussion of computational complexity and practical scaling (in terms of number of factors, representation dimensions, and samples) would help readers understand when the metrics are usable in practice.\n\n### Be explicit about supervision and data requirements\n\nThe experiments assume access to ground-truth factors (and nuisances) to estimate mutual information.\nIt would be useful to spell out what is required in general: Do we need labels for all factors and nuisances? Can the metrics be computed with only a subset of factors observed? How do they behave under strongly imbalanced factor distributions? Even if only discussed qualitatively, this would clarify the scope of applicability.", "broader_impact_concerns": "NA", "additional_comments": "I appreciate that the paper starts from clearly stated information-theoretic properties before introducing metrics, rather than proposing a score first and only giving informal intuition later.\nThe decision to separate minimality and sufficiency, instead of collapsing everything into a single \"disentanglement score\", also makes the results more interpretable.", "extra_scores": {}, "extra_text": {}}, {"review_id": "DasBiR2EWz", "reviewer": "Reviewer_vKVM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Attention!** This article is outside of my main research area.\n\n---\n\nThis paper gives a definition of disentanglement for the case where the true latent variables (or factors) are correlated and the data contain irrelevant noise (nuisances). First, it shows that existing definitions of disentanglement are not suitable for such realistic cases, since they assume the ground-truth latents are independent. Then, it introduces four desirable properties of disentangled representations: factors-invariance, nuisances-invariance, representation-invariance, and explicitness. Each of these is formulated through conditional independencies among the data, latent variables, and noise. After that, the paper demonstrates that these four properties can be expressed equivalently through two more intuitive ones: minimality and sufficiency. For both minimality and sufficiency, the paper proposes normalized metrics that make them measurable in practice. Finally, it compares the new metrics with existing ones using two toy examples and three neural representation learning datasets, showing that the proposed metrics capture disentanglement more effectively.\n\nStrengths: The paper makes an interesting contribution and is clearly written. The experiments fulfill their purpose of comparing the proposed metrics with existing ones. The proofs appear correct.\n\nWeaknesses: The paper could be more self-contained.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The problems with existing definitions of disentanglement are well motivated in the paper.\n- The paper includes proofs of its theoretical results in the appendix, and these proofs are correct.\n- The experiments clearly show the advantages of the proposed disentanglement metrics over the existing ones.", "audience_yn": "Yes", "audience_explanation": "Learning disentangled representations is relevant for many applications. The contributions of this paper are useful for practical use of such representations.", "requested_changes": "- Footnote 1 contains a dead link. (critical)\n- To make the paper more self-contained, briefly and formally define the most important existing definitions.\n- Add a short preliminaries section summarizing the notation used.\n- The term “sub-representations” on page 4 is unclear. Please clarify. (critical)\n- State the number of bins in the main text rather than in a footnote, as done for $N$ and $K$.\n- In Figure 3, the x-axis is clipped for $K=5$ since $1 - 1/K = 0.8$. Please correct this. (critical)\n- Include a brief description of the encoder architectures used in the main text (page 10).\n- Provide a formal definition of the interventional robustness score.\n- Some cited arXiv papers have published versions. Please update the references. (critical)\n- The third part of the proof of Theorem 1 is difficult to follow. Please rewrite it for clarity. (critical)", "broader_impact_concerns": "This is a theoretical work that proposes a novel definition of disentanglement. I have no broader impact concerns.", "additional_comments": "- $x$, $y$, and $z$ should be bold in the paragraph “Information Bottleneck in Representation Learning” on page 4.\n- Both $l$ and $k$ are used for defining tilde variables; it seems possible to avoid $l$.\n- A space is missing between “$y_i$” and “is” in section 4.4.\n- $x$ should be bold in the expression “Thus, $p(z_j \\mid x)$ …” on page 17.\n- On page 17, Bayes’ theorem should read $p(z_j, \\mathbf{y}, \\mathbf{n}) = \\ldots = p(\\mathbf{n}, z_j, \\mathbf{y})p(z_j \\mid \\mathbf{y})p(\\mathbf{y})$, not $p(z_j, \\mathbf{y}, \\mathbf{n}) = \\ldots = p(\\mathbf{n}, z_j, \\mathbf{y})p(z_j \\mid y_i)p(\\mathbf{y})$.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)