Measuring Orthogonality in Representations of Generative Models
Authors: Robin C. Geyer
Alessandro Torcinovich
João B. S. Carvalho
Alexander Meyer
Joachim M. Buhmann
Venue: Accepted by TMLR ()

Abstract: In unsupervised representation learning, models aim to distill essential features from high-dimensional data into lower-dimensional learned representations, guided by inductive biases. Understanding the characteristics that make a good representation remains a topic of ongoing research. Disentanglement of independent generative processes has long been credited with producing high-quality representations. 
However, focusing solely on representations that adhere to the stringent requirements of most disentanglement metrics, may result in overlooking many high-quality representations, well suited for various downstream tasks. These metrics often demand that generative factors be encoded in distinct, single dimensions aligned with the canonical basis of the representation space.

Motivated by these observations, we propose two novel metrics: Importance-Weighted Orthogonality (IWO) and Importance-Weighted Rank (IWR). These metrics evaluate the mutual orthogonality and rank of generative factor subspaces. Throughout extensive experiments on common downstream tasks, over several benchmark datasets and models, IWO and IWR consistently show stronger correlations with downstream task performance than traditional disentanglement metrics. Our findings suggest that representation quality is closer related to the orthogonality of independent generative processes rather than their disentanglement, offering a new direction for evaluating and improving unsupervised learning models.

Published in Transactions on Machine Learning Research (10/2024)
Measuring Orthogonality in Representations of Generative Models
Robin C. Geyer
robin.geyer&inf.etfee ethli.cf.ethcchh
Deportment of Computer Science, ETH Zurich
Alessandro Torcinovich
alessandro alcsanddro. www..torccoooocccci........i..c
Faculty of Engineering, Free Universily of Bozen-Bolzane Bolzano Department of Computer Science, ETH Zurich
João B. Carvalho
jono. www.waoowwwoffeea.aaaoca.m.m
Department of Computer Science, ETH Zurich
Alexander Meyer
German Heart Center of the Charité
Joachim M. Buhmann
jbuhmann etw....feethccc
Deportment of Computer Science, ETT Zurich
Reviewed on OpenReview: https: // openreview. net/ form? id= Tggp
Abstract
In unsupervised representation learning, models aim distill essential features from high- dimensional data into lover-dimensional Icarned representations, guided by inductive biases. Understanding the characteristies that make good representation remains a topic of ongoing rescarch. Disentanglement of independent generative processes has long been credited with producing high-quality representations. However, focusing solely on representations that adhere to the stringent requirements most disentanglement metrics, may result in overlooking many high-quality representations, well suited for various downstream tasks. These metries often demand that generative factors be encoded in distinct, single dimensions aligned with the canonical basis of the representation space.
Motivated by these observations, we propose two novel metrics: Importance- Weighted Orthog- onality (IWO) and Importance- Weightedd Rank (IWR). These metrics evaluate the mutual ort ort hogonnaalyyy and rank of generative factor subspaces. Throughout extensive experiments on common downstream tasks, over several benchmark datasets and models, IWO and IWR consistently show stronger correlations with downstream task performance than traditional disentanglement metrics. Our findings suggest. that represenlation quality closer related to the orthogonality of independent generative processes rather than their disentanglement, offering direction for evaluating and improving unsupervised learning models.
1 Introduction
Humans are able to process rich data such as high-resolution images to distill and memorize key information about polentially complex concepts. Similarly, represenlation learning aims to devise procedure, often unsupervised, to encode potentially nigh-dimensional data into a lweer-iimnsioonll learned embedding space, such that classifiers other predictors can casily extract information from the Iearned representations (Bengio et al.) 2013). Understanding how to generate these convenient representations, requires the definition of desirable properties that representation learning models should enforce. In the domain of generative models,

---

Published in Transactions on Machine Learning Research (10/2024)
C22
(si)
O2
(wi)
Q2
Color
Color
Color
Size
Size
Size
Figure 1: Three configurations of data (circles) encoded in a 2-d learned latent space. The data is characterized by size and (grayscale) color factors. The blue axes represent the direction of change in the factors. (i) The factors are aligned with the basis of the space, corresponding to perfect disentanglement and perfect orthogonality. (ii) The factors are not aligned but orthogonal, corresponding complete entanglement, but still perfect orthogonality. (iii) The factors are not orthogonal and some circle configurations are not encoded, however, disentanglement is higher than in (ii), because of partial alignment with the basis. Despite its complete entanglement, we argue that latent space (ii) is just as well suited as (i) for common downstream tasks, while (iii) is not.
the ability to disentangle the explanatory factors underlying the data has long been credited to be such desirable property.
In the disentangled representation learning framework, data æ is often assumed to be generated by an underlying function driven by ground truth, generative factors {z,} and other variability factors t, that is x =g(=,t) t). A model then learns mapping r(x) R2 from the data to latent representation space. A common characterization of disentanglement posits that the generative factors are represented by single distinct components of c, implying that they manifest as orthogonal 1-d latent subspaces aligned with the canonical basis within the latent representation, up to a scaling factor and irrelevant latent dimensions (i.e., when L K).
Disentangled representations have played pivotal role in improving the performance of tasks such classification (Zhou et al. 2022). segmentation (Kalkhhof al. 2022). registration Han ct al. (2020). image- to-image translation (Fei el al. 2021), artifact reduction (Tang et al. 2022). domain adaption (Li et al.) 2021), controllable synthesis (Kelkar & Anastasio 2022) and disease decomposition (Couronné et al. 2021] However, while disentanglement represonts an intuitive and useful notion to characterize good representations, its general applicability is limited. Indeed, Locatello et al. (201 prove, by using orthogonal transformations and probability (inverse) integral transforms, that unsupervised disentanglement learning is fundamentally impossible, without inductive biases on both models and datasets. In addition, the authors show that disentanglement exhibits weak correlation with the suitability of representations for common downstream tasks. We believe that this weak correlation can be attributed to the stringent nature of disentanglement measures, which penalize many high-quality representations otherwise suitable for downstream tasks.
To get an intuition of the problem, Figure I visualizes 2-d representation spaces encoding circles varying in size and color. While in space (i) and (ii) the dimensions encoding the generative factors are orthogonal to one another, in space (iii) they exhibit strong correlation. For the downstrcam task of regressing size and color, most models find the orthogonal spaces (i) and (ii) better suited than (iii), as they properly encode all size and color combinations, while space (iii) fails to encode the largest, lightest and the smallest, darkest circles. However, according to standard disentanglement metrics, space (ii) is completely entangled (worst case), while space (iii) exhibits a better disentanglement, due to the requirement that the generative factors must align with the canonical basis of the representation space. The disentanglement properties of these representations therefore stands in contrast to their downstream task utility.
While disentangled representations can be well-suited for many downstream tasks, entangled but orthogonal representations be equally effective. In line with Wang & Isola (2020), we believe that correlation with
2

---

Published in Transactions on Machine Learning Research (10/2024)
relevant downstream tasks is necessary condition for any metric measuring representation utility. The absence of this correlation is a significant weakness of common disentanglement metrics.
We therefore propose two new metries: Importance- Weighted Orthogonality (IWO) and Importance- Weighted Rank (IWR). To compute these metrics, we devise a novel methodology, Generative Component Analysis (GCA), that identifies the weighted vector subspaces where generative factors vary. IWO then measures the mutual weighted orthogonality between the subspaces found through GCA, while IWR assesses their weighted rank. Intuitively, IWO can be thought of as expansion of cosine similarity to general vector subspaces.
We empirically assess the validity of the proposed metric in various synthetic experiments and by analysing the performance of three common downstream tasks across six benchmark datasets and six widely used models, showing that IWO and IWR consistently display stronger correlations with downstream task performance than popular disentanglement metrics such as MIG (Chen et al. 2018) or DC1 (Eastwood & Williams 2018). Our rescarch suggests that the utility of a representation may be closer related to its orthogonality than its disenlanglerent.
2 Related Work
Alongside the task of disentanglement, gauging model's performance in disentangling representation has emerged as a non-trivial problem. Beyond visual inspection of the results, a variety of quantitative methodologies have been developed to tackle this issue. Higgins et al. (2017) propose to measure the accuracy of a classifier predicting the position of a fixed generative factor. Kim & Mnih (2018) further robustify the metric by proposing a majority voting scheme related to the least-variance factors in the representations.
Chen al. (2018) introduce the Mulual Information Gap estimating the normalized difference of the mutual information between the two highest factors of the representation vector. Eastwood & Williams (2018)
propose the DCI metrics to evaluate the correlation between the representation and the generative factors. For each of them, one linear regressor is trained and the entropy over the rows (Disentanglleennt and the columns (Completeness) is computed, along with the error (Informativeness) achieved by each regressor. The Modularity metric introduced by RRdgeway & Mozer (2018) computes the mutual information of each component of the representation estimate its dependency with at most one factor of variation. SAP score Kumar et al. 2018) estimates the difference, on average, of the two most predictive latent components for each factor.
The use of metrics such as the aforementioned ones contributed to shaping several definitions of disentangle- ment, each encoding somewhat different aspect of disentangled representations, which led to a fragmentation of definitions (Locatello et al. 2019). Higgins et al. (2018) attempt instcad to propose a unified view of the anglemeet problem, by defining a principled symmeetr--bsse disentanglement framework, drawn from group representation theory. The authors established disentanglement in terms of a morphism from world states to decomposable latent representations, equivariant with respect to decomposable symmetries acting on the states/ /reprcsentations. For representation to be disentangled, each symmetry group must act only on a corresponding (multiclimensional) subspace of the representation. Following this concept ua careeptuaiizatie lization, Caselles-Dupré et al. (2019) demonstrate the learnability of such representations, provided the actions and the transitions between the states. Painter et al. (2020) extend the work by proposing a reinforcement learning pipeline to learn without the need for supervision. Noteworthy are the two proposed metrics: (i) an independence score that, similarly to our work, estimates the orthogonality between the generative factors the fashion of a canonical correlation analysis; (ii) a factor leakage score, extended from the MIG metric to account for all the factors. Tonnaer et al. (2022) formalize the evaluation in the symmetry-basee setting and proposed a principled metric that quantifies the disentanglement by estimating and applying the inverse group elements to retrieve an untransformed reference representation. A dispersion mcasure of such representations is then computed. Note that while most works focus on the linear manipulation of the latent subspace, the symmetry-based framework can also be used in non-linear cases, However, it requires modelling the symmetrics and the group actions, a challenging task in scenarios with no clear underlying group structure Tonnaer et al. 2022). We aim to develop a metric which does not require such a group structure and works in more general cases.
3

---

Published in Transactions on Machine Learning Research (10/2024)
Recently, several works [Montero et al. 2021 Träuble et al. 2021 Dittadi et al. 2021 have proposed to go beyond the notion of disentanglement, ad advocating for the relaxation of the independence assumptiion among generative factors perceived as too restrictive for real-world data problems and modelling their correlations. Reddy et al. (2022) and Suter et al. (2019) formalize the concopt of causal factor dependence, where the generative factors can be thought of as independent. or subject. to confounding factors. The latter work introduced the Interventional Robustness Score assessing the effects in the learned latent space when varying its related factors. Valenti & Bacciu (2022) define the notion of weak disenlangled representation that leaves correlated generative factors entangled and maps such combinations in different regions of the latent space.
Additionally, Eastwood et al. (2023) relax the notion of disentanglement, by extending the DCI metric with an Explicitness (E) score related to the capacity required to regress the representation to its generative factors. Instead, our metric measures the mutual orthogonality between subspaces associated with generative factors, bypassing the non-linearities of generative factor and its latent subspace. For more comprehensive understanding refer to Appendix A
3 Methodology
Our goal is to establish a metric that quantifies the total orthogonality of representation. This involves estimating the orthogonality between the latent subspaces corresponding to each generative factor. However, this raises two challenges.
First, the identification of the latent subspaces is not straightforward since the generative factor can exhibit non-linear behaviour with respect to the learned representation. We propose Generative Component. Analysis (GCA), procedure where, for each generative factor, multiple non-linear regressors are used to identify progressively smaller subspaces where most of the generative factor's information resides. These subspaces are then used to identify the generalive componenls, that is, the set of orthonormal vectors spanning the latent subspace. In a similar fashion to procedures such as linear principal component regression, we weight each vector by a corresponding importance score to obtain an importance-ordered orthogonal (i.o.o.) basis, for each generative factor.
Sccond, prominent methods for measuring orthogonality between subspaces do not provide sufficiently discriminative results for our use case. The conventional definition of orthogonal complement is binary and too restrictive for nuanced applications. Conversely, methods like canonical correlation analysis, which determines the similarity between vector spaces, typically operate under optimal conditions and may assign high scores even to spaces sharing only a single dimension. These approaches, while useful, tend to be overly permissive for our specific objectives. We therefore propose Imporlance- Weighted Orthogonalily (IWO) to characterize the orthogonality between generative factors. This is done by computing an average weighted projection of each generative factor's latent subspace onto all the others. To further discriminate between representations with similar orthogonality properties, we also compute Imporlance- Weighted Rank: (IW R), which estimates the spread of the importance weights, awarding representations that give more importance to few dimensions.
3.1 Generative Component Analysis (GCA)
Consider a latent representation or code, c E R', encoding the generative factors (ik E RK with I. 2 K. The generative factors are inducded through zj = fj (c), where fī is a potentially non-linear function. However, not all changes in e imply a change in Zj. In particular, we define the invariant latent subspace of 2, to be the largest linear subspace I, S RR,, such that (c =f (c), Ve E I,. Accordingly, the variant latent subspace (simply latent subspace) of 2j is defined to be the orthogonal complement of I, and will be denoted as Sj, with dimensionality Ry. In the next paragraph, we describe how to find an imporleenc-weeighled basis for Sz.
Subspace learning Starting from a code c E R', we project it onto progressively smaller dimensional subspaces, removing the least important dimension for regressing zj at each step, until the subspace is
4

---

Published in Transactions on Machine Learning Research (10/2024)
W
W
W1
W
I
Importancee Weights
Subspace Learning
Importauce Ordered Orthonottal Basis
II Basis Generation
W3
W₂ WW=WWW,
WF
by
bg
14
Basis Completion
Normelization
Expected Loss
Right-sidod Miltiplication
Importance Computation
Through iterative multiplications with Wi e R(x(++1,, the input is projected to subspaces of decrcasing dimensionality. The resulting outputs wa are directed into NN heads, trained to minimize the expected loss terms L. The importance or is gauged by the loss decrease between consecutive NNs heads. II - Basis Generation: The least important dimension, ba, corresponds to the null space W3. For each subsequent dimension b, the composed projection matrix Wi- Wi-1.... W3 is computed. then corresponds the dimension in the null space of Wi which is orthogonal to all previously found basis vectors. Finally, bi is retrieved by normalizing and transposing Wi.
Figure 2: Overview of GCA. I
Subspace Learning:
1-dimensional. In particular, design Linear Neural Network (LNN) composed of set of projective transformations W.-....., W1, with W; E Rxx(++1) which reduce the dimensionality of e step-by-step. No non-linearities are applied, therefore each layer performs projection onto a smaller linear subspace. The entire learning process depicted in Figure 2
To capture the most informative latent subspace of dimensionality every layer of the LNN, we feod the intermediate projections wi E R into non-linear neural networks fzz. These networks are tasked with regressing 2,, thereby guiding the selection of which latent dimension to discard in each projection step.
When the training has ended, cach regressor fa can be associated with an expected loss of regressing the factor:
L. E. (e)
(1)
where l specific loss term. In particular, note that L 2 Ci because of the potential information loss due to dimensionality reduction. Let us now quantify the loss increment by each of the LNN projections ACI- Li-1 L. We define RG the smallest dimensionality at which the lowest achievable loss is reached, that is, AC: 0 for 1 >> Rg. For I = 1, we compute AL1 as the difference between L1 and baseline loss Co E,, [((E,, =5
Basis generation Using the trained projection matrices Wi..i.... Wi, along with the layer-specific loss differences AC,, we now describe how to construct an i.o.o. basis spanning 2j 's latent subspace Sy. Formally, want to devise set of orthonormal vectors B, [6(") e | .......,, R,), ) along with their respective importance weights {a()) (j) E R | - I,... R In the following, we drop the superscript (z) to ease the
notation.
Each projection matrix Wi, 1 1...., L eliminates dimension from the data representation. The adopted training methodology ensures that each step the dimension discarded the least important for regressing zj among the remaining ones. This is in turn based which removed dimension results in the
5

---

Published in Transactions on Machine Learning Research (10/2024)
minimum increase of AC The removed dimension represents the 1-d mull space of its corresponding projection matrix.
We begin by finding the basis vector by, corresponding to the least important dimension, by identifying the null space of matrix WL-1 E R(1-1)x For each subsequent dimension bi+1 E R', I= 1,..... 2, we iteratively compute the composed projection matrix W W, WL-1 E R The null space ker (Wi) contains the target dimension bi+1 along with the previously found basis vectors b1+2...., dr. We retrieve 11+1 by finding the dimension in the mull space of W, that is also orthogonal to the previously identified basis vectors, for example through QR decomposition. Finally, the remaining dimension b1, corresponding to the most significant basis vector, can be directly retrieved by computing W1 RIxL and normalizing it. This process is depicted in Figure 2 and a pseudocode implementation can be found in Appendix
Using the loss difference defined on the basis of equation ! basis vector b, by the relative loss increase associated to its layer in the LNN:
we can quantify the importance weight ay of each
ALI
at Co LR,
1=
Finally, we normalize each vector b, thus obtaining an i.o.o. basis Bj = {6,0 ..... bb,) for Sj, with its
corresponding importance weights d1 (j)
,....,oR.. (3)
GCA allocates an i...o. basis for each generative factor of learned representation. However, when comparing representations, we also have to account for differences in CR,, as this loss corresponds to the best possible regression of 2j from the representation. In the DCT framework, this aspect is captured by the Informativenes metric. In order not to favour representations with low Rg and high LR, over those with low LR, and higher Ry, we adjust the importance weights of any factor 2j whose Cn, 0. To do that, we first complete the factor's basis By to span the whole latent space, then we distribute the loss CR, among the importance of the basis vectors equally:
ACi + LR,/LL
1= L.
(3)
Lo
3.2 Importance Weighted Orthogonality (IWO)
Orthogonality commonly treated as a dichotomous attribule; that is, vectors are classified as either orthogonal or non-orthogonal, and similarly, subspace is considered to either reside in the orthogonal complement of another not. The concept of cosine similarity provides contimuous mcasure of the degree of orthogonality between two vectors. Analogously, we aim at a continuous measure that evaluates the degree of orthogonality between two subspaces. Consider two orthonormal bases B, = (b()).... ...., ((5) and
B& = {b(4) ...., Ri
(A)) spanning two subspaces S, and Sg. Let r((kk be the overall projection of By basis
vector bf onto all the basis vectors in BR:
Rx
(ik)
Σ
6
1= I= Ry.
(4)
where the square applied to guarantee non-negativity. With these projections, a continuous interpretation of orthogonality between Sy and Sy can be expressed as
Sk) min(R, R&)
(R))
O(S,, Sx) E [0, 1] with its maximum reached if S, is a subspace of S₃ or vice-versa, and its minimum reached when Se lies in the orthogonal complement of Sy. This definition of orthogonality can be interpreted as the average squared cosine similarity between any vector pair from S, and St.
This formulation of orthogonality ignores the "importance" of dimensions within subspaces S, and St. To illustrate why this is problematic, consider two generative factors, zj and zk, sharing the same subspace S. If

---

Published in Transactions on Machine Learning Research (10/2024)
most of their variation concentrated in two dimensions, b and by while the rest of S encodes minimal variation, our metric should certainly distinguish between by and bj being orthogonal vs. identical.
Metrics As the name suggests, in addition to encapsulating the orthogonality between factor subspaces, IWO also takes into consideration the importance of the dimensions spanning them. Let us consider K different generative factors 2K.. For each one of them, we are able to allocate an i.o.o. basis
B& = (b) ,...,,bbb b with the importance weights {aj (E) {ai ,.....akk (8)). For any ground truth factor 2j, we define p(ah) as b projection onto the latent subspace of another ground truth factor 2ks scaling the projection by the importance of the respective basis vectors:
Rs Σ
Va
a
(1) (6)²)
1=1....,Ry
(6)
Analogously to subspace orthogonality, IWO is then defined using the sum over all projections dimensions spanning zy's subspace.
P(()) of the
IWO 2x) =
(7)
Note that, with respect to Equation 5 IWO does not require normalization factor as the square root in Equation i guarantees that IWO E 10. In addition, we subtract from 1 to simplify the comparison with standard disentanglement metrics. Therefore, the maximum of is reached if 2j lies in the orthogonal complement of z and vice versa. The minimum of 0 is reached when and žk, in addition to lying in the same subspace, also share the same importance along the same dimensions (ef. Appendix B for the proofs). Both Orthogonality and IWO can be efficiently calculated using matrix operations (cf. Appendix D
Together with IWO, define Importance Weighted Rank (TWR). TWR measures how the importance of generative factor's subspace is distributed among the dimensions spanning it:
IWR(2,) H66,
where H,6 -- a
log, (ag and we always assume L > 1. IWR thus measures how the importance is distributed among the L dimensions of the representation space. Note that, IWR(2;) = if the importance is distributed equally along all L dimensions spanning the representation space, while IWR(2;) importance is concentrated in single dimension only (ef. Appendix B for the proofs). We denote the mean over all generative factors of IWO and IWR IWO and IWR.
the
4 Experiments
To evaluate the offectiveness of our IWO and IWR implementations, first test whether we can (i) recover the true latent subspaces using GCA and (ii) correctly assess their orthogonality and importance spread. For that purpose, we set up synthetic data generation scheme, providing us with the ground truth IWO and IWR values. For comparison, we also test how other metrics capture different aspects of the representation. In particular, Disentanglemend, Completeness, Informativeness and Explicitness, as measured by the DCI-ES framework using its official implementation.
We further evaluate IWO/IWR through the disentanglement, 1ib framework (Locatello et al. 2019 and measure how strong they correlate with downstream task performance for three different tasks deployed the learned representations of six widely used variational autoencoder models trained on six benchmark disentanglement datasets and over wide range of seeds and hyperparameters. More details are listed in the appendix and in our open-source code implementatior!]
Training of all neural networks in the LNN performed in parallel. In principle, the gradient flow from each individual regressor fą can be stopped after the corresponding Wi, however, we attested a faster convergence
http:///gahubb.mm///ggggg//////
7

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Synthetic experimental results: comparison between (D) Disentanglement, (C) Completeness, (1) Informativeness, (E) Explicitness, IWO and IWR for Polynomial (Poly.) and Trigonometric (Trig. encodings.
Experiment (K = 5, always)
R f ROP
IWO
TWR
1 Perm. X
0.98
0.98
1.00
0.90
0.98
1.00
(1) Permutation and additive Gaussian noise
1 Noisy X
0.98
0.98
1.00
0.90
0.98
1.00
10 2 Poly. X
0.99
0.69
1.00
0.75
0.98
0.69
(2) Low rank
polynomial mapping
10 2 Poly. /
0.21
0.15
1.00
0.75
0.98
0.69
10 5 Poly. x
0.41
0.30
1.00
0.74
0.61
0.31
10 5 Poly. <
0.06
0.04
1.00
0.74
0.61
0.31
High rank
stayoommi//rrggoommetrrd mapping
10 5 'Trig.
x
0.41
0.30
0.99
0.73
0.62
0.30
10 5 Trig.
0.06
0.04
1.00
0.73
0.62
0.31
Poly:
0.98
0.53
0.99
0.76
0.97
0.54
4 Poly:
0.12
0.07
1.00
0.76
0.97
0.54
4 Trig. x
0.96
0.52
0.99
0.73
0.98
0.53
20 4 Trig.
V
0.12
0.07
0.99
0.74
0.98
0.53
(4) High rank
mapping
20 8 Poly. x
0.56
0.30
0.99
0.74
0.76
0.31
20 8 Poly. v
0.05
0.03
0.99
0.74
0.76
0.31
20 8 Trig-
x
0.54
0.29
0.98
0.71
0.76
0.31
20 8 Trig.
/
0.07
0.04
0.98
0.70
0.76
0.30
5 Poly. x
0.98
0.58
0.98
0.75
0.99
0.57
5 Poly.
0.11
0.05
0.98
0.75
0.99
0.56
100 5 Poly. x
0.98
0.64
0.97
0.74
0.98
0.63
(5) High dimensional latent space + poly./trig. mapping
100 5 Poly. <
0.09
0.04
0.98
0.73
0.98
0.63
250 5 Poly.
x
0.97
0.67
0.97
0.72
0.98
0.68
250 5 Poly.
0.09
0.04
0.97
0.72
0.98
0.68
when letting the gradient of each ffr flow back up to The aim of simultancously exploring all nested subspaces is to facilitate the identification of the smallest subspaces by guidance through the larger ones. Additionally, we assume reduction factor of 1, so that Wi e R(x(++1) for / == l, L, however, higher values can also be considered for larger representations. Further speed-up techniques are presented in Appendix !
4.1 Synthetic Experiments
We introduce a synthetic data generating scheme, which generates vectors of i.i.d Gaussian distributed latent representations e E R Then, on the basis of the latent represenlations, we synthesize K generative factors [2)...,ūk]. For simplicity, we choose as a multiple of K.
For simulating disentangled latent space, we define each 2j to be lincarly dependent on single, distinct element of c. To assess higher dimensional cases with non-linear relationships, we consider non-linear commutative mapping f R2 R. In particular, we experiment with polynomial (Poly.) and trigonometric (TTrig.) f. Notice that the commutativity enforces that the distribution is spread evenly across all dimensions, such that we can casily assess the performance of IWR. For simplicity, we always set R, R for all 1, K. Together, (latent space dimension). K (number of factors) and R (latent subspace dimension) determine how many dimensions each generative factor shares with the others. We consider the shared dimensions to contiguous. For more details about the synthetic data generation scheme,, refer Figure and the pseudocode in Appendix
To test representations that not aligned with the canonical basis, apply Random Orthogonal Projections (ROP) R E L to In line with commonly used datasets such as Cars3D or dSprites, we rescale and quantize zy to the range [0, 1]. All experiments are run four times with differing random seeds. The standard deviation was smaller than 0.02 for all reported values. The results are displayed in Table I
(1) Permutation and additive Gaussian noise We define two setups, both with L= K = and R = First, we let be a mere permutation of c, second we let be e c, with ~ N/ 0.01) being Gaussian noise. As expected, under conditions of low-complexity and (quasi-)perfect disentanglement we obtain high scores for all the metrics.
8

---

Published in Transactions on Machine Learning Research (10/2024)
(2) Low rank + polynomial mapping We choose L 10, = 2 No dimensions are shared between the generative factors. The mapping f is polynomial, adding more complexity. We also apply ROP in one of the experiments, We see that the C' score decreases due to dimension sharing. Additionally, ROP dramatically lower both D and C, while E, IWO and IWR are resilient to it.
mapping We set L, = 10, K R = 5 and vary ROP. We test with a polynomial and a trigonometric f. Each generative factor now shares, on average, two out of five dimensions with the others, determining a decrement in D, C, IWO and IWR scores. E is slightly sensible to the change in the mapping used.
(3) High rank + polynomial//rigoononnn
(4) High rank + polynomia//tiggonometti mapping We set L 20, - R (4.8), varying ROP and f. Increasing L mitigates the mutual dependence between generative factors, with IWO and IWR, partially recovering. In the presence of ROP, D and C' keep low scores. E stays almost constant assessing only the complexity of the function used.
(5) Himn-dimensional latent space + = R- = 5 and vary ROP, keeping f polynomial, The large dimensionality of the representation mitigates the effect of dimension sharing. We observe that IWO and IWR assess good orthogonality reliably, while the other metrics perform similarly to the previous cases.
mapping We E (50, 100, 250),
The experimental results reveal that both the D and C metrics from the DCI framework exhibit pronounced scnsitivity to ROP, making them unsuitable for evaluating generative factor scparability in the manner we propose. Although the E metric from the DCI-ES framework demonstrates robustness against ROP, it still falls short of reliably capturing generative factor separability. This shortcoming arises because E primarily measures the complexity of the mapping between generative factors rather than their actual separability. In contrast, our proposed metrics, IWO and IWR, offer accurate assessments of orthogonality and the importance distribution across latent subspaces. They provide a more reliable evaluation of factor separability, demonstrating both resilience to orthogonal projections and immunity to mapping complexity.
4.2 Downstream Experiments
For the systematic evaluation of IWO's and IWR's correlation with downstream task performance, we use the disentanglement _ib framework (Locatello et al. 2019). We consider six benchmark datasets, namely
dSprites, Color dSprites and Scream dSprites (Matthey et al., 2017]. Cars3D (Reed et al.) 2015).
smalINORB (LeCun et al. 2004) and Shapes3D (Burgess & Kim 2018). These datasets cover wide
range of complexities and variations, for more information on the individual datascts refer to Appendix F.2
On each of these datasets, six different commonly used Variational Auto Encoder (VAE) models are trained:
3-VAE Higgins et al. (2017), Annealed VAE (Burgess et al. 2018), 3-TCVAE (Chen al. 2018)) Factor-
VAE (Km & Mnih 2018), DIP-VAE-I and DIP-VAE-II [Kmarr et al. 2018) These models represent diverse set of approaches to disentanglement, each introducing unique mechanisms to encourage factorized
representations. For each model, we consider six different regularization strengths (cf. Appendix F.I 1 each with ten different random seeds, resulting in a total of 2160 learned representations.
The representations are evaluated for their utility in regressing the generative factors. To this end, we consider three distinct downstream models trained on the learned representations: (i) random forest, (ii) logistic regression, and (iii) multi-layer perceptron. Correlations between downstream task performance and the commonly used metrics, DCI-D, DCI-C, and MIG, together with IWO's and IWR, are calculated throughout the considered regularization strengths.
Our main objective is to investigate whether the orthogonality of a representation is indicative of the performance on a variety of downstream tasks and thus a good metric for the utility or quality of the ropresentation, as elucidated by Wang & Isola (2020). The primary distinction between Orthogonality and Disentanglement, as measured by the considered metrics, is that the former does not require alignment with the canonical basis of the representation space. The three downstream models are chosen to distill this key
9

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: (a) Average correlation between metrics and downstream task performance aggregated by model. Each model is trained on six different datasets. For cach dataset six hyperparameters on 10 random seeds are trained for a total of 360 learned representations per model. (b) Average correlation between metrics and downstream task performance aggregated by dataset. For each dataset, six different VAE models are trained. For each model six hyperparametees on 10 random seeds are trained for total of 360 learned representations per dataset.
(a) Per dataset correlation aggregated by model
(b) Per model correlation aggregated by dataset
Model
DCI-D DCL-C
MIG
IWO
IWR
Model
DCI-D
DCI-C
MIG
IWO
IWR
(1) Random Forest
(1) Random Forest
AnnealedVAE
0.74
0.64
0.66
0.74
0.82
Cars3d
-0.52
-0.59
-0.64
0.63
0.46
3-TVAAE
0.66
0.63
0.64
0.78
0.66
Color dSprites
0.73
0.79
0.69
0.33
0.77
8-VAE
0.65
0.65
0.64
0.26
0.83
dSprites
0.94
0.96
0.94
0.55
0.79
DIP-VAE-I
0.56
0.59
0.52
0.43
0.40
Scream dSprites
0.88
0.93
0.76
0.51
0.44
DIP-VAE-II
0.47
0.32
0.44
0.09
0.66
Shapes3D
0.88
0.76
0.66
0.13
0.63
Factor VAE
0.62
0.54
0.18
0.68
0.61
smallNORB
0.78
0.54
0.67
0.82
0.88
(2)
Logistic Regression
(2) Logistic Regression
AnnealedVAF
0.43
0.20
0.26
0.77
0.85
Cars3d
-0.39
-0.57
-0.70
0.75
0.69
3-TCVAE
0.13
0.06
0.13
0.45
0.09
Color dSprites
0.66
0.50
0.64
0.61
0.50
3-VAE
0.18
0.19
0.16
0.44
0.51
dSprites
0.38
0.35
0.30
0.58
0.36
DIP-VAE-I
0.73
0.63
0.59
0.73
0.63
Scream dSprites
0.79
0.55
0.86
0.86
0.67
DIP-VAE-II
0.07
-0.53
0.01
0.88
0.16
Shapes3D
-0.30
-0.49
-0.61
0.52
-0.06
Factor
0.14
-0.05
0.07
0.71
0.69
smallNORB
0.54
0.17
0.73
0.65
0.78
(3)
Multi- Multi-Layer Layer Perceptron
(3)
Multi-Tayer Perceptron
AnnealedVAE
0.35
0.17
0.08
0.41
0.58
Carsäd
-0.12
-0.36
-0.39
0.56
0.63
3-TCVAE
-0.25
-0.31
-0.28
-0.28
-0.16
Color dSprites
-0.25
-0.48
-0.26
-0.25
-0.77
8-VAE
-0.14
-0.20
-0.18
0.61
0.20
dSprites
0.18
0.17
0.13
0.35
0.08
DIP-VAE-
0.54
0.46
0.43
0.64
0.56
Scream dSprites
0.18
0.01
0.23
0.59
0.78
OIE-VAE-II
0.00
0.57
0.00
0.71
0.08
Shapes3D
-0.38
-0.51
-0.66
0.60
0.05
VAC.
-0.04
-0.19
-0.39
0.51
0.44
smalINORB
0.84
0.52
0.62
0.77
0.92
difference. While we expect alignment with the canonical basis to benefit downstream task (i) Random forest, we do not expect such a benefit for tasks (ii) logistic regression or (iii) multi-layer perceptron.
we present. the results aggregated by model and dataset respectively. We refer the reader the complete results in Table 3 in Appendix F for a detailed breakdown of all experiments and their outcomes. Figure 3 serves as an illustrative example, visualizing the comparison between the generative components identified by GCA and the dimensions found through the DCI framework for one of the 2160 considered representation spaces.
In Table 2a 2b
(1) Random forest High correlations are present for all models and datasets except for Cars3D, where the DCI and MIG metrics fail to correlate entirely. However, DCI-D correlates more reliably for the other datasets and models with the downstream task than IWO does. This can be attributed to the nature of random forests in defining axis-aligned discriminative rules. Therefore, the dowmstream task benefits from the alignment of the generative factors with the canonical basis. Indoed a random forest might have harder time fitting latent space (ii) from Figure | than fitting latent space (iii), as in the latter, one generative factor aligns with the canonical basis. However, we also notice that DCI-C and IWR correlate even stronger than DCI-D and IWO do. This suggests that random forests benefit even more significantly from low-dimensional representations of generative factors than from their aligument with the canonical basis.
(2) Logistic regression For logistic regression, where lgrregularization is applied to the weights, there is no reason to assume that alignment between generative factors and canonical basis should lead to higher downstream task performance. Indoed, we can see that IWO and IWR correlate higher and more reliably than DCI or MIG do. This leads assume that measuring the orthogonality detached from canonical basis alignment provides better metric for the evaluation of downstream logistic regression.
10

---

Published in Transactions on Machine Learning Research (10/2024)
GCA
DCI
Floor color
Wall color
Object color
Object size
Object type
Azimuth
Figure 3: Samples from 3-VAE trained on Shapes3D dataset. In each row, the same latent code is modulated along a different dimension. The reconstruction of the resulting latent codes through the decoder network is depicted. Left: The dimensions of modulation correspond to the most important generative components of each generative factor as found by GCA. We attest that modulation along the generative components indeed predominantly varies the respective generative factor. Right: The dimension of modulation corresponds to the most important dimensions for each generative factor as found by DCI. However, modulation along the dimension supposedly encoding floor color also changes azimuth and vice versa. Modulation along the dimension supposedly encoding wall color also changes floor color. The same form of entanglement goes for most other dimensions identified through the DCI framework.
(3) Multi-layer perceptron A similar behavior can be attested when fitting a multi-layer perceptron, initialized using Kaiming uniform initialization (He al.) 2015)) There is, again, no reason assume that aligument with the canonical basis would be beneficial. Indeed, see that neither DCI-D, DCI-C nor MIG correlate reliably. IWO and IWR correlate reliably with this downstream task, except for the 3-TCVAE model and the Color dSprites dataset.
5 Discussion and Conclusions
In our investigation, pivoted from the conventional focus the disentanglement of generative factors to a novel examination of their orthogonality. Our approach, geared towards accommodating non-linear behaviors within linear subspaces tied to generative factors, fills a gap in existing literature, as prior method aptly addressed this perspective. The proposed Generative Component Analysis (GCA) efficiently identifies the generative factor subspaces and their importance. Leveraging GCA, we formulated Importancee Weighted Orthogonality (IWO) and Importance- Weighted Rank (IWR), two novel metrics offering unique insights into subspace orthogonality and importance ranking. Throughout experiments, our implementation emerged as a robust mechanism for assessing orthogonality, exhibiting resilience across varying latent shapes, non-linear encoding functions, and degrees of orthogonality.
Disentanglement has long been credited for fostering fairness, interpretability, and explainability in generated representations. However, as pointed out by Locatello et al. (2019). ; the utility of disentangled representations invariably hinges on at least partial access to generative factors. With such access, an orthogonal subspace could be rendcred as useful as a disentangled one. Through orthonormal projection, any orthogonal representation discovered can be aligned with the canonical basis, achieving good disentanglement.
In conjunction with IWO's stronger downstream task correlation across datasets, models and tasks, this underscores our asscrtion that latent representations, which successfully decouple generative factors, are crucial for a wide range of rownstream applications, regardless of their alignment with the canonical basis.
In conclusion, our work lays the groundwork for a fresh perspective on evaluating generative models. We hope GCA and IWO may help identify models crafting usefuul orthogonal subspaces, which might have been overlooked under the prevailing disentanglement paradigm. We hope that IWO extends its applicability across broader spectrum of scenarios compared to traditional disentanglement measures.
11

---

Published in Transactions on Machine Learning Research (10/2024)
References
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell, 35(8):1798-11225 2013.
Chris Burgess and Hyujji Kim. 3d shapes dataset. https///gttubb.com//een/////
2018.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in beta-vae. CoRR, abs/ 0bs/1804003399 2018.
Hugo Caselles- Dupré,, Michaël Garcia Ortiz, and David Filliat. Symmetry-based disentangled representation learning requires interaction with environments. In Advances in Newral Information Processing Systems 32: Annoua Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 4608-460 4617, 2019.
Tian Qi Chen, Xuechen Li, Roger B. Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, NewrIPS, 2018.
Raphaël Couronné, Paul Vernhet, and Stanley Durrleman. Longitudinal self-supervision disentangle inter-patient variability from disease progression. In Proc. International Conference on Medical Image Computing and Computer-Assisterr Intervention (MICCAI), pp. 231-241. Springer, 2021.
Andrea Dittadi, Frederik Träuble, Francesco Locatello, Manuel Wuthrich, Vaibhav Agrawal, Ole Winther, Stefan Bauer, and Bernhard Schölkopf. On the transfer disentangled representations in realistic settings. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.noo net, 2021.
Cian Eastwood and Christopher K. I. Williams. A framework for the quantilative evaluation of disentangled representations. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenRovicw net, 2018.
Cian Eastwood, Andrei Liviu Nicolicioiu, Julius von Kügelgen, Armin Kekic, Frederik Träuble, Andrea Dittadi, and Bernhard Schölkopf. DCI-ES: an extended disentanglemenl framework with connections to identifiability. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Yuchen Fei, Bo Zhan, Mei Hong, Xi Wu, Jiliu Zhou, and Yan Wang. Deep learning- based multi-modal computing with feature disentanglement for MRI image synthosis. Medical Physics, 48(7))3778-3788) 2021.
Xu Han, Zhengyang Shen, Zhenlin Xu, Spyridon Bakas, Hamed Akbari, Michel Bilello, Christos Davatzikos, and Mare Niethammer. A deep network for joint registration and reconstruction of images with pathologies. In Machine Learning in Medical Imaging: 11th International Workshop (MLMI), pp. 342 352, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR, 2017.
Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loic Matthey, Danilo J. Rezende, and Alexander Lerchner. Towards definition of disentangled representations. CoRR, abss/1812.02230, 2018.
Lee W. Johnson and Dean R. Riess. Numerical analysis, 1982.
John Kalkhof, Camila González, and Anirban Mukhopadlyay. Disentanglement enables cross-domain hippocampus segmentation. preprint arxi::200 0rx0:22001.00500, 2022.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Varun A Kelkar and Mark A Anastasio. Prior image-based medical image reconstruction using a style-based generative adversarial network. preprinl or arXiv:2202.0889 08936, 2022.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising, In Proceedings of the 35th International
Conference on Machine Learning, ICML, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In 6th International Conference on Learning Represenlations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.nee net, 2018.
Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Compuler Sociely Conference on Compuler Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. II II-104. IEEE, 2004.
Hongwei Li, Sunita Gopal, Anjany Sekuboyina, Jianguo Zhang, Chen Niu, Carolin Pirkl, Jan Kirschke, Benedikt Wiestler, and Bjoern Menze. Unpaired MR image homogenisation by disentanglod representations and its uncertainly. In Uncertainly for Safe Ulilization of Machine Learning in Medical Imaging, and Perinatal Imaging, Placental and Preterm Image Analysis. Springer International Publishing, 2021.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114-4124, PMLR, 2019.
Loic Matthey, Trina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing 2017.
sprites datasct. https:/ wtwgiituu..oood/eeeee cowm://epnind///sptttssddaaaiddd
Milton Llera Montero, Casimir H. Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey S. Bowers. The role of disentanglement in generalisation. In 9th International Conference on Leorning Representations, ICLR 2021, Virtual Event, Austrio, May 3-7, 2021. OpenReview.net, 2021.
Matthew Painter, Adam Prügel-Bennete, and Jonathon S. Hare. Linear disentangled representations and unsupervised action estimation. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NearIPS 2020, December 6- 12, 2020, virtual, 2020.
Abbavaram Gowtham Reddy, Benin Godfrey L, and Vineeth N. Balasubramanian. On causally disentangled representations. In Thirty-Sizth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Four Fourth Conference on Innovative Applications of Artificial Intelligence, TAAT 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 8089 8097. AAAI Press, 2022.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual shalogy-making Advances in neural information processing systems, 28, 2015.
Karl Ridgeway and Michael C. Mozer. Learning deep disentangled embeddings with the f-statistic loss. In Advances in Neural Information Processing Systems 31: Annual Conference on Newral Information Processing Systems 2018, NewrIPS 2018, December 9-8, 2018, Montréal, Canada, pp. 185-194, 2018.
Raphael Suter, Djordje Miladinovic, Bernhard Schölkopf, and Stefan Bauer. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6056-6068 PMI.R, 2019.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Fan Tang, Xinyu Zhu, Jinrong Hu, Juhong Tie, Jiliu Zhou, and Ying Fu. Generative adversarial unsupervised image restoration in hybrid degradation scenes. 2022.
Loek Tonnaer, Luis Armando Pérez Rey, Vlado Menkovski, Mike Holenderski, and Jim Portegies. Quantifying and learning linear symmetry-based disentanglement. In International Conference Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 21584 21608. PMLR, 2022.
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from correlated data. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 10401 10412. PMLR, 2021.
Andrea Valenti and Davide Bacciu. Leveraging relational information for learning weakly discntangled representations. In International Joint Conference Neural Networks, IJCNN 2022, Padua, Italy, July 18-23, 2022, pp. 1-8. IEEE, 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929-9933 PMLR, 2020.
Lei Zhou, Joseph Bae, Huidong Liu, Gagandeep Singh, Jeremy Green, Amit Gupta, Dimitris Samaras, and Prateek Prasanna. Lung swapping autoencoder: Learning a disentangled structure-texture representation of chest radiographs. preprint arXiw:2211 22440034444 2022.
14

---

Published in Transactions on Machine Learning Research (10/2024)
C3
C3
C3
C3
2/2
22
(ii)
2/2
(itt)
22
(iv)
C2
C2
C2
C2
21
2
21
21
C1
C1
C1
C1
Figure 4: Four configurations 3-dimensional latent space. The planes represent the latent subspace where generative factors 21, 22 lic. The color mapping on cach subspace represents the relationship between the generative factor and the latent components (e.g., blue indicating large values for 21, red indicating low ones). Cases (i) and (ii) characterized by a good explicitness score both subspaces encode and 22 simple quadratic functions, contrary to cases (iii) and (iv) where the relationship is trigonometric and more complex to recover. In contrast, cases (i) and (iii) are characterized by better IWO score compared to (ii) and (iv). Indeed, in configuration (i) and (iii), there are dimensions within each generative factor's ' subspace that are orthogonal to one another. Consequently, any variation along any such dimension will leave the other factor unchanged.
A Explicitness IWO
The Explicitness (E) metric aims evaluate the capacity needed for representation to regress its generative factors. Formally,
E(zj, c AULCC(E,.GF)) F)z;F)
where and e represent a generative factor and the latent space respectively, and F a class of regressors (e.g., multilayer perceptrons or random forests). AULCC the Area Under the Loss Curve, computed by recording the minimum losses achievable by regressing 2j from e with models in of increasing capacity. The denominator, easily computable, acts normalizing constant that E E [0, 1]. As an example, F suggests that linear regressor is sufficient to reach zero error, proving that the representation is efficient. To account for bias toward large representations, the explicitness is paired with the Size (S), computed as the ratio between the number of generative factors and the size of the latent representation.
In Figure we depict four situations of a 3-dimensional latent space where two generative factors 21, 22 lie in a separate 2-dimensional plane each, and the relationship between each generative factor and its corresponding latent subspace is non-linear, as hinted by the coloring. In particular, =)t) ((") = f(n) (P(")) c), for
E 21 and n E {i, ii, iii, iv}, with p(() E g2x3 being projection matrix. For cases (i) and (iii), the projections span orthogonal plancs, contrarily to cases (ii) & (iv) where the planes have a different inclination. Case (i) (ii) are characterized by quadratic relationship, i.e., f(n) = A(c))² + B(c))² (with A, B being paramcters), while case (iii) and (iv) encode trigonometric relationship, i.c., f(x) cos(22t)c) + (2x (with À being parameter).
Explicitness evaluates the capacity required by model to regress the generative factors 21, 22, starting from the reprosentation c. Given that the effect of the lincar transformation is present in all four situations, the differences are delermined only by the non-linearity f(a). Therefore the metric is able to discriminate cases (i) & (ii) from (iii) (iv). Instead, IWO quantifies the orthogonality of the planes, regardless of the non-lincarities the generative factors, so it discriminates the orthogonal cases (i) (iii) from the non-orthogonal ones (ii) & (iv). Finally, note that the DCI-Disentanglement metric would penalize all four configurations as they are not disentangled.
15

---

Published in Transactions on Machine Learning Research (10/2024)
B Theoretical Results on IWO/IWR
Theorem B.1. Given representation E R², IWO =1 if and only if the latent subspaces , the generalive faclors 2K all lie in each other's orthogonal complement.
Sx of
Proof. Let By and B& represent basis for the latent subspaces Sy and Sg. the subspaces lic in cach other's orthogonal complement, this implies that b bm 0, vl = =,...., R2, Vm =1, l...., Rg- The computation of IWO is therefore
R2
R
R2
R
[wO 2k)
(k) am (b)
(2
(K) am
=
(10)
m=1
m=1
From proving that IWO zk)= 1 for any * k. it follows that IWO = 1.
Conversely, if at least one subspace does not lie in the orthogonal complement of at least one other subspace, this implics that 6(" (k) + for at least combination of I, m. k. Because of (5) 0 for s Rg for any j, this implies that IWO(=j, zx) < 1.
Theorem B.2. Given a representation e E RL, TWO if and only if the latent subspaces St,... Sg of the generative factors 2g all share the same importance along the same dimensions.
have that at 2 a111 for any and any E {j. k). Let us first consider the special case at Q+11 for any 1, i.e. the basis are ordered from most important to least important basis vector. Assuming all subspaces share the
Proof. Let Bj and B& represent basis for the latent subspaces Sy and Sg. From Section 3.1 (5)
(4)
same importance along the same dimensions, it must follow that B, Bs. Then b(²) b, V.. Also, 6(2) bl = implics that b(5) b.a for I + m. Then,
(k) and a aj (k)
R
IWO(2j,22 24) =
(k) (b.
(
)
1=1
m=
ft
(1)
(k)
=1
a
R
= = Σ
1=1
1111 0.
(11)
Let us now consider the special case where S, = S₁ = RR and a03== =
1/R, Vl, m. Note that, since all importance weights are equal, there no order the basis. Tet B, and B₁ be two orthogonal bases spanning S, and S₃ respectively. Because S, St, each basis vector 6(5) in B, can be expressed as linear combination of the basis vectors in Bg- We can then Parseval's identity (Johnson & Riess 1982 for the
(k)
16

---

Published in Transactions on Machine Learning Research (10/2024)
finite case: |2 Σm= (b)
(b( . ).
R
R
IWO 24)
-1
am
(k)
(b)
)
1=1
m=1
=
( )
R
R
R
(2)
R
16
51
12=
R 0. R
(12)
R
The general case for ( 2 1+1 then follows from the two special cases described above. From proving that IWO(2), 24) for any j # k, it follows that IWO 0.
(3)
Conversely, if two generative factors do not share the same importance along the same dimensions, this either mean that their subspaces are different or it means their subspaces are the same, but the importance is spread differently, i.e. (i) # aj cannot be 0.
(k) for least different l. For both cases follows that IWO(z 2x)
]
Theorem B.3. Given representation E RL with L then 7WR = if and only if the latent subspaces $1,...,SK of the generative factors 21, are all uni-dimensional.
Proof. Let B, = (b(²),, then af" = and
IWR(2;)= = 1
Σ
( (a
logz (1) 11 1
(1) (1)=1 1-000 11
(13)
1=1
From proving that IWR(2;) = for any j, follows that IWR 1.
() Conversely, if a. 0 for >> 1, under the constraint that a 2 a/41 and Et cannot be 1.
(1)
Ht af 1, then IWR(2) (2))
Theorem B.4. Given representation E B- with L> 1, then IWR = 0 is and only if for each generalive factor 21...., 2K the importance is spread equally among all dimensions of the representation space.
Proof. Let B₁ (bi,..., bL} and aj (5) 1/L, 1 L,..., L, then:
4
L = 1
IWR(2;)= = 1+ ( logg a 1 + Σ
logı
1
(-1)==11-- 0.
(14)
1=1
1=1
From proving that IWR(2;) 0 for any j, follows that IWR
Conversely if any ( + 1/L, under the constraint that Σt aj (i)
1, then IWR(z;) cannot be 0.
C Complexity and Differentiability
For GCA, the subspace learning slep complexity is dependent on the backpropagation algorithm used, while the basis generation step complexity is dependent on the reduced QR decomposition algorithm used. The worst-case complexity of IWO O(K33), for IWR it OKLL). GCA and the metrics are composed of differentiable operations and are therefore differentiable. While the metrics are optimizable, GCA requires supervision from the generative factors, information that may not always be available.
17

---

Published in Transactions on Machine Learning Research (10/2024)
D Efficient IWO Calculation
For the efficient calculation of Orthogonality, consider two matrices B, E BR,xL, Bg E RR whose rows compose the i.o.o. basis vectors spanning 2j and zt's latent subspaces respectively. We define the orthogonality between the two latent subspaces in terms of these matrices as:
Tr B,BJ) min min(R, Rx)
Sk)
(15)
Note that the trace Br B)) equals the sum of the squared values B, iee, (i, Pm Lm (bz) b where byt and bum are the I-th and m-th rows of Bj and BN respectively. The maximum of the trace is therefore min( Ra), reached if Sy is subspace of S or vice-versa. This definition of orthogonality can be interpreted as the average absolute cosine similarity between any vector pair from Sy and St.
To efficiently calculate the weiggtte projection of zj²s subspace onto zg's subspace, we first scale the corresponding bases vectors in B, BR with their respective importance before projecting them onto another. IWO is the sum of all individual projections. Using U,- D, where Dj E RB is diagonal with the I-th diagonal entry corresponding to the square root of importance, Vor(zz), we can efficiently calculate IWO as:
IWO(2), 24) 1 -
(16)
E Pseudocode
E.1 Basis Generation
Full QR dccomposition
Given rectangular matrix A R=x(n-1), its full QR decomposition is
R 0
A A QQ el
(17)
with R [B(x-1x((n-1) being upper-triangular, and the columns of Q R=x(n-1) and v E Rex1 being orthonormal. In particular, note that A 0, that is (er(AT). We denote QR
We report the pscudocode for generating an orthonormal basis as described in Scction 3 Require: Projection matrices (Wi) W, E (Ix(1-1)
Ensure: Orthonormal basis matrix B L
W + W
W E (I
2: b + QRker (W)
D Get least important basis voctor by-
3: B+br bl
for 1 I--2 2 do
W W. W.T
W Exxi Concatenate WT and B. W (I
W + liT
6:
7:
b <- QR W)
Get basis vector
B bi B)
Concatenate by to B
end for
10: W <- ( W)T
& W E ERII
11: W
D Get most important basis vector b
12: B b B
Concatenate b
13: return B
18

---

Published in Transactions on Machine Learning Research (10/2024)
2
R 2
R 5
C
c
Two syuthetic experimental settings with L 10, K = and differing ranks R. Left: R each 2j is function of two successive elements of (2), ..... 2s =(c, Right: R cach 2j is function of five successive elements of f(eg, C10, cI, c3) ..... 25 f((7, 68, Co, C10, ci).
Figure 5:
E.2 Synthetic Data Generation
We report the pseudocode for generating the synthetic represenlations discussed in Section 4.1 For simplicity, we have assumed L as a multiple of K. Figure 5 describes two examples of the used schema.
Require: Representation size L, # of generative factors K, # of shared dimensions R, mapping Ensure: A representation e Re and its related generative factors R
NN(O,, IL) 3: la + 1
Get L i.id. Gaussian distributed samples D. Concatenate three times, clrep) RR
c(rep) t- cal (c,
La +
5: z f-1 []
6: for + L to 2L by
& do
K
7:
x f c-
1ltl 2 fx)]
(rep)
Get values of (rep) from ( la to l + la Concatenate the generative factor f(x) z
end for
10: return c,
F Experimental Details
In this section, we provide detailed description of the correlation analysis of our orthogonality metric with downstream tasks on six different datasets and six different models listed in Table For each model, learned representations for six different regularization strengths for cach reg. strength). All these representations are directly retrieved from or trained with the code of disentanglement _ For the ES metric, we utilized the official codebase provided by the authors of DCI-ES Each dataset we investigate has independent generative factors associated with it.
considered (ten different random seeds
F.1 Hyperparameters
The hyperparameters considered for each model are the following:
V-VAE [Higgins
2017): with e (1,2,4,6,8,)
Annealed VAE Burgess et al. 2018 with Cmaz E {5, 10, 25, 50, 75, 100)
3-TCVAE Chen et al.
2018 with 3 E 10)
Factor- VAE
Kim & Mnih
2018
with y E {10, 20,30, 30, 40, 50, 100)
DIP-VAE-I Kumar et al.
2018
with Add E {1, 5, 10, 20, 50}
DIP-VAE-II Kumar et al.
2018 with Aod 21,2,5, 10, 20,50)
19

---

Published in Transactions on Machine Learning Research (10/2024)
Figure Samples from the Datasots used in our studies. From left to right: DSprites, Color DSprites, Scream DSprites, Cars3D, Shapes3D, smallNORB
F.2 Datasets
dSprites Dataset The dSprites dataset collection of 2D shape images procedurally generated from six independent latent factors. These factors are color (white), shape (square, ellipse, heart), scale, rotation, and and positions of sprite. Each possible combination of these latents is present exactly once, resulting in total of 737280 unique images.
Color dSprites Dataset This dataset retains the fundamental characteristies of the original dSprites dataset, with the distinct variation that each sprite, in the observation sampling process, is rendered in a color determined by random selection.
Scream dSprites Dataset The dataset mirrors the original dSprites dataset but introduces a unique modification in the obsorvation sampling process: A random segment from the Scream image is selected the background, and the sprite is integrated into this image by inverting the color of the chosen segment. specifically the sprite's ' pixel locations.
Cars3D Dataset The Cars3D dataset is generated from 3D computer-aided design (CAD) models of cars. It consists of color renderings of 183 car models from 24 rotation angles, each offset by 15 degrees, and from different camera elevations. The images are rendered at a resolution of 64 x 64.
smalINORB Dataset The smallNORB datasct designed for 3D object recognition from shape, featuring images of loys calegorized into five lypes: four-legged animals, human figures, airplanes, trucks, and cars. The images were captured under six lighting conditions, nine elevations, and 18 different angles. The dataset is split into training set with five instances of each category and test set with the remaining five instances.
Shapes3D Dataset The Shapes3D dataset, initially presented in Kim & Mnih 2018 is specialized collection designed for the study of factorized variations in 3D object representation. This dataset is characterized by its systematic variation across six ground-truth factors, making it particularly suitable for experiments disentanglement and reprosentation lcarning. Those factors include floor color with 10 variations, wall color also with 10 variations, object color featuring 10 distinct options, object size represented in different scales, object type with unique categories, and azimuth with 15 varied positions. Such a diverse range of factors allows for comprehensive analysis and experimentation in 3D object recognition and disentanglement tasks. The dataset is thoughtfully split into training set and test set, each containing balanced mix of these variations to facilitate robust model training and evaluation.
F.3 IWO on Limited Data
To assess the impact of smaller sample sizes on IWO and IWR's correlation with downstream task performance, we repeat the experiments detailed Section 4.2 for the smallNORe dataset, but with only 50% and 10% of the data. The results are illustrated in Table We observe that IWO and IWR are resilient to changes in dataset size.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Individual correlations between metrics and downstream task performance. Each model is trained on six different datasets. For each dataset, six different hyperparameters on 10 random seeds are trained for a total of 2160 learned representations. Correlations between downstream task performance and the three commonly used metrics DCI-D, DCI-C and MIG together with IWO and IWR are calculated. Correlations are calculated between averaged task performance and averaged metrics, where the average is taken over the seeds and grouped by hyperparameter. As such, we assess the capability of the metrics to point us to good models and hyperparameters.
Random Forest
Logistic Regression
Multi-Layer Perceptron
MIG
IWO
IWR
D
MIG
IWO
IWR
D
C
MIG
IWO
IWR
AnnealedVAE
0.72
-0.18
-0.19
0.74
0.69
-0.40
-0.85
-0.82
0.84
0.89
0.25
-0.36
-0.33
0.98
0.97
Cars3D
8-TCVAE
-0.92
-0.91
0.80
0.11
-0.91
-0.97
-0.90
0.75
-0.15
0.06
-0.01
0.16
-0.48
-0,90
B-VAE
-0.92
-0.94
-0.93
0.67
0.80
-0.93
-0.95
-0.92
0.60
0.91
-0.66
-0.67
-0.67
0.22
0.82
DIP-VAE-I
-0.84
-0.77
-0.61
-0.09
-0.18
0.61
0.39
-0.10
0.73
-0.07
-0.32
-0.50
0.98
0.99
DIP-VAE-IT
-0.89
-0.96
-0.76
0.90
0.59
-0.49
-0.72
-0.74
0.74
0.83
-0.49
-0.70
-0.49
0.67
0.94
Factor VAE
-0.26
-0.17
-0.81
0.78
0.74
-0.17
-0.31
-0.71
0.91
0.91
0.22
-0.08
-0.49
0.97
0.98
AnnealedVAE
1.00
1.00
0.98
0.97
0.97
0.86
0.85
0.90
0.91
0.94
-0.80
-0.80
-0.82
-0.86
-0.85
Color dSp.
8-TCVAE
0.99
0.99
1.00
0.78
0.97
0.14
0.54
0.45
0.70
0.55
-0.81
-0.88 89
-0.85
-0.68
-0.86
8-VAE
1.00
-0.80
0.76
-0.78
0.32
-0.95
DIP-VAE-
0.46
0.39
0.61
0.96
0.94
0.73
0.68
0.70
0.78
0.63
0.89
0.91
0.75
0.04
-0.18
DIP-VAE-IT
0.07
0.50
-0.07
-0.73
0.26
0.37
-0.78
0.90
-0.81
0.71
-0.45
0.73
0.50
-0.92
FActorVAE
0.91
0.85
0.63
0.81
0.75
0.86
0.86
0.62
0.82
0.72
-0.79
-0.80
-0.60
-0.80
-0.87
AnnealedVAE
0,99
0.99
0.97
0.89
0.89
0.84
0.82
0.85
0.87
0.87
0.47
0.45
0.54
0.73
0.72
8-TCVAE
1.00
0.99
0.99
0.92
0.97
0.37
0.38
0.25
0.66
0.54
-0.14
-0.18
-0.25
0.07
0.03
8-VAE
0.98
0.99
0.97
0.75
0.96
0.03
0.07
0.01
-0.18
0.02
-0.04
-0.06
0.02
0.40
0.08
DIP-VAE-I
0.96
0.94
0.28
0.11
0.35
0.91
-0.10
-0.22
DIP-VAE.L
0.77
0.88
0.80
-0.33
0.93
-0.20
-0.46
-0.43
0.69
-0.59
-0.92
-1.00
-0.02
0.14
-0.98
VactorVAE
0.96
0.94
0.96
0.86
0.43
0.35
0.99
0.98
0.79
0.93
0.49
0.86
0.84
VAE
0.81
0.79
0.85
0.86
0,87
0.92
0.85
0.95
0.92
0.91
0.81
0.86
0.71
0,84
0.84
8-TCVAE
0.91
0.98
0.90
0.32
-0.12
0.93
0.83
0.94
0.65
-0.46
-0.88
-0.75
-0.89
-0.78
0.37
Seregen
8-VAE
0.94
0.97
0.96
0.53
0.54
0.86
0.83
0.89
0.78
0.79
0.17
0.13
0.23
0.87
0.90
DIP-VAELI
0.79
1.00
0.28
0.68
0.78
0.83
0.50
0.94
0.95
0.89
0.86
0.61
0.89
0.98
0.94
DIP-VAE-II
0.92
0.81
0.42
0.46
0.53
1.00
0.49
0.03
0.72
FactorVAR
0.87
0.92
0.80
0.23
0.11
0.65
0.14
0.68
0.79
0.89
-0.38
-0.79
-0.30
0.66
0.64
AnnesladVAE
0.95
0.62
0.21
0.91
0.96
0.60
-0.13
-0.51
0.91
0.83
0.87
0.45
-0.01
0.97
1.00
smell B Shapeso
8-TCVAE
1.00
0.99
0.93
0.90
0.96
-0.91
-0.93
-0.96
-0.87
-0.87
-0.61
-0.68
-0.83
-0.69
-0.60
B-VAE
0.99
0.97
0.86
-0.57
0.92
-0.33
-0.62
-0.80
0.94
-0.55
-0.57
-0.0 06.66
-0.84
0.92
-0.63
DIP-VAEL
1.00
1.00
0.95
-0.24
-0.24
0.46
0.45
0.32
0.57
0.25
-0.33
-0.34
-0.41
0.96
0.82
DIP-VAE-II
0.99
0.99
0.97
-0.74
0.74
-0.63
-0.78
-0.83
0.98
-0.44
-0.73
-0.86
-0. 90
0.97
-0.56
PactorVAE
0.35
-0.02
0.03
0.49
0.44
-0.96
-0.90
0.43
-0.85
-0,98
-0.96
0.45
0.27
VAn
-0.04
0.28
0.77
0.09
0.56
-0.26
-0.37
0.21
0.14
0.68
0.50
0.42
0.38
-0.17
0.80
8-TCVAE
0.97
0.76
0.98
0.94
0.99
0.86
0.53
0.97
0.78
0.92
0.93
0.641
1.00
0.89
0.99
8-VAE
0.92
0.93
0.97
0.97
1.00
0.97
0.96
0.96
0.93
0.97
0.93
0.91
0.97
0.97
0.99
VIP-I
1.00
0.98
0.95
0.97
0.98
0.97
0.94
0.91
0.93
0.94
0.99
0.98
0.96
0.97
0.98
DIP-VAR-I
0.91
-0.44
0.68
1.00
0.99
0.85
-0.56
0.90
0.99
0.98
0.93
-0.45
0.86
1.00
1.00
FactorVAE
0.89
0.73
-0.55
0.95
0.79
-0.14
-0.51
0.40
0.13
0.18
0.77
0.59
-0.47
0.94
0.77
F.4 Segmentation Task
The task involves segmenting the sprite from the Scream dSprites dataset using the learned representations, utilizing decoder neural network that generates segmentation masks based on these representations. In Table 3 we list the results of these experiments. We observe that both IWO and IWR correlate stronger with downstream segmentation performance than classic disentanglement metrics.
F.5
IWO Training
Given learned representation of a dataset, we consider each generative factor independently, allocating separate LNNs respectively. On top of the LNNs, we have NN heads, which regress the generative factors from the intermediate projections. The NN heads are also independent from one another and do not share any weights.
F.5.1 Implementation Details
We use the Pylforch Lightning framework for the implementation of the models required to discern IWO and IWR. In particular, we use the implementations of the Linear and Batch-Normallzztionn layers. Whereas the setup of the LNN is equal for all models and datasets, the NN-heads vary in their complexity for different datasets and factors. As
https://githhb. http.///.....tiiinnniiiiiiiiiaia AT/1ightning
21

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: Correlation coefficients between IWO, IWR, DCI-D, DCI-C, MIG and the downstream task of recovering the generative factors with logistic regression and random forest. Examined latent representations of smallNORB dataset as learned by models: Annealed VAE (A-VAE), 3-VAE, 3-TCVAE and Factor- VAE (F-VAE) 100%, 50% and 10% of the dataset
IWO
IWR
Logistic Regression
Random Forest.
Logistic Regression
Random
Model
Forest
A-VAE
0.14
0.09
0.68
0.56
100%
3-VAE
0.93
0.97
0.97
1.00
3-TCVAE
0.78
0.94
0.92
0.99
F-VAE
0.13
0.95
0.18
0.79
3-VAE
0.20
0.02
0.43
0.75
50%
A-VAE
0.97
0.98
0.97
1.00
3-TCVAE
0.82
0.95
0.94
0.99
F-VAE
0.13
0.72
-0.09
0.95
A-VAE
-0.02
-0.30
0.50
0.31
%0"
3-VAE
0.94
0.99
0.96
0.99
3-TCVAE
0.87
0.95
0.92
1.00
F-VAE
0.32
0.56
0.12
0.58
Table 5: Correlation coefficients between IWO, IWR, DCI-D, DCI-C, MIG and the downstream task of segmenting the sprite in the Scream DSprite dataset.
Model
DCI-D
DCI-C
MIG
IWO
IWR
Annealed VAE
0.74
0.80
0.65
0.79
0.79
B-TCVAE
-0.91
-0.78
-0.91
-0.65
0.48
3-VAE
-0.20
-0.27
-0.19
0.52
0.51
VAE-I
0.01
0.33
-0.31
0.01
0.08
DAP-VAE-II
0.45
0.02
0.69
1.00
1.00
Factor VAE
0.09
-0.33
0.19
0.63
0.62
Average
0.03
-0.03
0.02
0.38
0.58
all considered models operate with 10-dimensional latent space, each LNN has 10 layers. The output of each LNN layer is fed to the next layer and also the corresponding NN-head.
Table B holds the NN head configuration per dataset and factor. These were found using simple grid search on one randomly selected learned representation. This is necessary as factors vary in complexity and so does the required capacity to regress them. It is worth mentioning, that the Explicitness pipeline, as proposed by Eastwood et al. 2023 could actually be employed on top of the NN-hwads, integrating both metrics.
For the initialization of the LNN layers and the NN heads, we use Kaiming uniform initialization as proposed in He et 2015 We further use the Adam optimization scheme as proposed by Kingma & Ba 2015) with a learning rate of 5 x 10 and batch size of 128 for all optimizations. Data is split into training (80%) and test set (20%). During training, part of the training set is used validation, which importance scores used for IWO should be allocated using the test set. In experiments, the difference between the importance scores computed on the training set and the test set was small. For further details on the implementation, please refer to our official c
turn used as an early stopping criterion. The
hhtpe///gahubb.mm///ggggg//////
22

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: Implementation Details for neural network heads operating on LNN layers. For each factor, ten NN heads with the specified number of hidden layers and their respective dimensions are trained in parallel.
Dataset
Factor
Layer dimensions Batch Norm
Factor Discrete
Shape
256, 256
X
Scale
256, 256
X
dSprites
Rotation
512, 512, 512
x-position
256, 256
X
y-position
256, 256
X
model
256, 256, 256
X
Cars3D
rotation
256, 256, 256
X
elevation
256, 256, 256
X
category
256, 256
/
lightning condition
256, 256
smallNORB
elevation
256, 256
rotation
256, 256
Floor color
128, 128
X
Wall color
128, 128
X
Object Color
128, 128
X
Shapes3D
Object Size
128, 128
X
Object Type
128, 128
X
Azimuth
128, 128
X
Table 7: Average correlation between metrics and downstream task. Comparison between IWO, IWR and unweighted orthogonality (0)
Random Forest
Logistic Regression
Model
IWO
IWR
IWO
IWR
O
AnnealedVAF
0.74
0.82
0.53
0.77
0.85
0160
3-TCVAE
0.78
0.66
0.12
0.45
0.09
0.02
3-VAE
0.26
0.83
-0.03
0.44
0.51
0.09
DIP- VAE-1
0.43
0.40
0.30
0.73
0.63
0.18
DIP-VAE-IT
0.09
0.66
-0.27
0.88
0.16
0.64
Factor VAE
0.61
0.59
0.71
0.69
0.56
G Importance Analysis
In order test the effectiveness of IWO IWR's importance weighing, we compare their downstream task correlation with the downstream task correlation of pure orthogonality (without any weighing). Table 7 holds the results of the correlation analysis. We can see, perhaps unsurprisingly, that importance of weighing plays an important role in why TWO and TWR can serve representation quality metries, whereas orthogonality might be questionable contender at best.
H Loss Analysis
In Figure 7 we depict the loss of neural network heads at different projection steps for single run of synthetic experiment 3 (I = 10, K R 5). La to L10 are omitted, as they are almost zero (similar Ls). Fach generative factor is analysed using GCA, which means for each generative factor we train an LNN spine with nine matrices Wo E 90110 Wi 1xx and 10 NN heads acting on the projections. Because of the symmetry of the synthetic experiments, all five generative factors are similarly encoded in the latent space. In Figure 7 we are therefore depicting the mean and standard deviation over the generative factors. Au entire pass through each LNN projects the representation to the most informative dimension for the respective generative factor. When recovering the generative factor from that projection, we incur a loss of Ci. The fraction L Lo tells us that this is z 20% better than naive guessing (assuming the expectation value) of the factor. We see that we almost perfectly recover the generative
23

---

Published in Transactions on Machine Learning Research (10/2024)
0.8
10-
L/
AC,/Co
0.7
La/La
Relative Loss
0.6
ACy/Ro
Relative Loss
C//Co
0.5
0.4
0.4
AC//Co
0.3
0.2
0.1
AC
4x/Eo
Ea/Co
Ca/Ra
Cxffa
Cg
Iterations
Figure 7: Loss of neural network heads at different projection steps for single run of synthetic experiment (L= 10, K = and R₃- = 5). We only depict La to L1, Ls to Lı0 almost (similar to Ls)- Left: Magnitude of relative loss after convergence for different projection depths. Right: Loss evolution over training iterations.
factors from projections to 5 and more dimensions. This is expected as the experiment is set up with R = 5. We see that each subsequently removed dimension increases the loss by = 20%. follows that ACi/Ca 0.2, i.e. or 0.2 for S < 5. The right-hand side of Figure / depicts the relative validation loss of the NN heads during training. The relative loss the 72nd iteration corresponds to relative loss depicted the left plot.
Computational Resources Analysis
This section details the computational resources utilized for evaluating DCI, IWO and IWR, specifically applied to the SRa dataset from disentanglement__e using a 3-VAE framework. the GCA model specifications please refer to section F.5.1
1.1 Experimental Setup
Data: Learned representations a 3-VAE trained on the smalINORB dataset from the disentonglement
Objective Assess orthogonality 60 learned representations, by performing GCA and calculating IWO/IWR. Note that the computational resources for the calculation of IWO/IWR are negligible compared to GCA.
Objective 2: Assess the disentanglement 60 learned representations, by computing DC1 metric.
1.2 Resource Utilization
In Table we list the compulational resources used for each run of GCA.
Table 8: Average resource utilization for each run of the GCA + IWO pipeline. No GPU used.
Resource
Usage
Process Memory
400 MB
CPU Process Utilization
50%
GPU Process Utilization
0%
1.3 Runtime Analysis
GCA Average Duration: minutes per run (20 Epochs)
DCI Average Duration: minutes per run.
24

---

Published in Transactions on Machine Learning Research (10/2024)
1.4
Computational Cost Considerations for GCA
For GCA computational costs scale with the number of linear layers in the LNN spine and the capacity of the NN heads.
For large latent spaces, one should avoid step wise dimensionality reduction in the LNN spine; larger reductions between consecutive LNN layers are preferred.
The first LNN layer size need not match the dimensionality of the representation. For large representations, a smaller first LNN layer is recommended.
GPU usage is beneficial for larger representations and models
Performing GCA on pretrained smalINORB representations shows small computational costs, comparable those necessary for computing DCT.
25


Reviews:
[{"review_id": "OTUrscctz4", "reviewer": "Reviewer_YKwd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe perspective that disentanglement is a better measurement for feature representation is interesting.\n\nThe proposed Generative Component Analysis (GCA) and IWO/IWR are empirically validated to be effective in most experimental settings.\n\n\nWeaknesses:\n\nIn Table 2, it shows that IWO and IWR have distinct performances in some cases, which raises the question: which metric should be used to measure the representation quality when a downstream task is not known or cannot be trivially evaluated? \n\nOnly simple classification tasks are tested to show the representation quality. It is unknown how the representation is useful in different tasks like object detection, segmentation, etc.\n\nIn Color DSprites, all disentanglement metrics fail to correlate with the representation utility, and there is no discussion on this. Similarly, in Randon Forest, IWR/IWO is not as good as DCI-C in Table 2b.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes two new metrics, Importance-Weighted Orthogonality (IWO) and Importance-Weighted Rank (IWR), to evaluate the disentanglement of representations, as the existing methods fail to address the issue in Fig. 1, where some certain bases are better than others to capture the disentanglement. Through extensive experiments, the paper shows that these metrics correlate more strongly with downstream task performance compared to traditional disentanglement metrics. The paper's finding indicates that the utility of a representation is closer related to the orthogonality than its disentanglement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Explain which metric should be used to measure the representation quality when a downstream task is not known or cannot be trivially evaluated.\n\nMore tasks should be tested to demonstrate the representation quality.\n\nThe diverse results in Table 2 should be discussed more.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sPGtHKbE6y", "reviewer": "Reviewer_UrTf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The motivation of this paper is clear, and the author proposed a reasonable solution.\n- This paper is contextualized. The related work section is useful for the readers.\n- The author proposed \"Generative Component Analysis (GCA)\" to identify the subspaces where generative factors vary.\n\n## Weaknesses\n\n- Given a metric, I think it is important to state clearly:\n\n  - what property it exactly measures,\n  - **if having a perfect score (either $0$ or $1$) implies that a representation must have the property, and if all representations with the property are assigned perfect scores**,\n  - what supervision is needed to calculate the metric,\n  - if it is differentiable so that we can directly optimize the property,\n  - what supervision/annotation is needed, and\n  - the computation cost of the metric.\n\n  The author mentioned some of them, but it would be better to summarize them as theorems (with proofs).\n\n- The author explained the design choice of the proposed metrics, but not the reasons. For example, the benefit of \"taking the importance of the dimensions spanning them into consideration\" is unclear to me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed two metrics, Importance-Weighted Orthogonality (IWO) and Importance-Weighted Rank (IWR), for measuring the orthogonality of representations such that each factor is not required to be aligned with the basis. The paper argues that these metrics correlate better with downstream task performance than traditional disentanglement metrics. The proposed metrics were evaluated on synthetic datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Locatello et al. (2019) used not only orthogonal transformations but also probability integral transforms and their inverses.\n- p. 2 \"we utilize Generative ...\" sounds like \"Generative Component Analysis (GCA)\" was proposed in a previous work.\n- As far as I know, the original paper on the group-theoretical definition (Higgins et al. 2018) never used the term \"Symmetry-Based Disentanglement Representation\".", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q9Gdrp2y5r", "reviewer": "Reviewer_QLKU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall it seems that the metric computation might be complex (and potentially costly) to compute, some parts of the paper need to be clarified. However, overall this is a very interesting paper, **challenging the correlation between disentanglement representation and performances on downstream tasks**.\n\nSome parts of the paper were unclear to me:\n- the General Component Analysis method has to learn a lot of neural networks, is the number of learned neural nets equal to the number of latent components?\n- The 'subspace learning' part of 3.1 is very clear, but I did not understand the 'basis generation' part. Would it be possible to introduce a more rigorous/mathematical definition of the $Q_l$-s and  $q_l$-s. Is it clear that $Q_l$-s are full rank? i.e. that  $q_l$-s are uniquely defined? Is it even clear that the $Q_l$-s are of size $(l + 1) \\time l$?\n- What does $b_l$, $q_l$, $\\Delta_l$ represent?\n- where are the 'importance weights' $\\alpha^{j}$ and $\\alpha^{k}$ mathematically defined? Especially since the correlation applied between the latent can make the task especially easy (or hard)\n- the synthetic data generation process could be explained more clearly, e.g, by writing the exact equation for each synthetic data.\n- What are the conclusions of Table 1? I do not understand how it makes the point of the paper, and no explanation is provided. The only reference to Table 1 in the text is 'The results are displayed in Table 1'. Could you elaborate?\nIs Table 1 displaying the correlation between downstream task performance and the metrics? If yes it seems to me that this is written nowhere.\n- Results of Table 2 are extremely interesting! How do you understand the fact DCD-I, DCI-C and MIG correlate negatively with the performances for Random Forest?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles an extremely interesting topic: in the last decades, there has been an increasing interest in 'disentangled representations', with the hope to obtain 'better' representations, 'more interpretable'. A huge amount of empirical and theoretical work can now be found on disentangled representations. However, the practical interest in disentanglement remains to be proven. The paper brings empirical evidence that disentanglement might not be the right notion. The authors introduce two new metrics, IWO and IWR, based on the orthogonality of the latent factors, and show the proposed metrics correlate better with performance on downstream tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarify all the points of the previous section, in particular:\n- Clarify the 'basis generation' part\n- Detail the cost of the proposed method, and give insights on how much the hyperparameters need to be handtuned (as I understand, one has to learn multiple networks to compute the proposed metric) \n- Comment more Table 1", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Post-discussion, all three reviewers are in favor of acceptance and are satisfied with the revised version of the paper. The AE is in favor and recommends accepting the paper.
Decision: Accept (poster)