Cross-Entropy Is All You Need To Invert the Data Generating Process
Authors: Patrik Reizinger
Alice Bizeul
Attila Juhos
Julia E Vogt
Randall Balestriero
Wieland Brendel
David Klindt
Venue: ICLR 2025 Oral (2025)
Keywords: supervised learning
representation learning
identifiability
linear representation hypothesis
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We prove that models trained with cross-entropy in supervised learning can recover latent factors of the data-generating process up to a linear transformation, supported by both theoretical results and empirical evidence.

Abstract: Supervised learning has become a cornerstone of modern machine learning, yet a comprehensive theory explaining its effectiveness remains elusive. Empirical phenomena, such as neural analogy-making and the linear representation hypothesis, suggest that supervised models can learn interpretable factors of variation in a linear fashion. Recent advances in self-supervised learning, particularly nonlinear Independent Component Analysis, have shown that these methods can recover latent structures by inverting the data generating process. We extend these identifiability results to parametric instance discrimination, 
then show how insights transfer to the ubiquitous setting of supervised learning with cross-entropy minimization. We prove that even in standard classification tasks, models learn representations of ground-truth factors of variation up to a linear transformation under a certain DGP. We corroborate our theoretical contribution with a series of empirical studies. First, using simulated data matching our theoretical assumptions, we demonstrate successful disentanglement of latent factors. Second, we show that on DisLib, a widely-used disentanglement benchmark, simple classification tasks recover latent structures up to linear transformations. Finally, we reveal that models trained on ImageNet encode representations that permit linear decoding of proxy factors of variation.
Together, our theoretical findings and experiments offer a compelling explanation for recent observations of linear representations, such as superposition in neural networks. This work takes a significant step toward a cohesive theory that accounts for the unreasonable effectiveness of supervised learning.

Published as a conference paper at ICLR 2025
CROSS-ENTROPY IS ALL YOU NEED TO INVERT THE DATA GENERATING PROCESS
Patrik Reizinger*1, Alice Bizeul=1.2, Attila Juhos"
Julia E. Vogt", Randall Balestriero Wieland Brendel David Klindt
[patzik.reizinge attila. juhos, wieland.brendel11 wrw.nnnellytuuueiieee...ooo (alice.bizeu) julia. vogt]einf..eeeeeee ch, rbalestr@brown edu,, klindt@cshl..... edu
ABSTRACT
Supervised learning has become comnertooe of modern machine learning, yet a comprehensive theory explaining its effectiveness remains elusive. Empirical phenomena, such neural analogy-making and the linear representation hypothe- sis, suggest that supervised models can learn interpretable factors of variation in a linear fashion. Recent advances in self-supervised learning, particularly nonlinear Independent Component Analysis, have shown that these methods can recover latent structures by inverting the data generating process. We extend these identifia- bility results to parametric instance discrimination, then show how insights transfer to the ubiquitous setting of supervised learning with cross-entropy minimization. We prove that even in standard classification tasks, models learn representations of ground truth factors of variation up to a linear transformation under a certain DGP. We corroborate our theoretical contribution with series of empirical studies. First, using simulated data matching our theoretical assumptions, we demonstrate successful disentanglement of latent factors. Second, show that on DisLib, widely-used disentanglement benchmark, simple classification tasks recover latent structures up to linear transformations. Finally, reveal that models trained on ImageNet encode representations that permit linear decoding of proxy factors of variation. Together, our theoretical findings and experiments offer compelling explanation for recent observations of linear representations, such as superposition in neural networks. This work takes a significant step toward a cohesive theory that accounts for the unreasonable effectiveness of supervised learning.
INTRODUCTION
Representation learning is central task in machine learning, underpinning the success of extracting and encoding meaningful information from data (Bengio et al., 2013). Among the various paradigms, supervised learning- parttnllllrrl classification tasks using cross-entropy minimization-has become the dominant method in deep learning (Krizhevsky et al., 2012). Despite its simplicity, this form of supervised learning has led to several intriguing and widely-observed phenomena, including: neural analogy making (Mikolov et al., 2013), where models seemingly map between related concepts; the linear representation hypothesis (Park et al., 2023), which posits that interpretable features can be linearly decoded from neural representations, recent work on superposition in neural networks (Elhage et al., 2022), showing evidence that interpretable features are linearly represented in neural activations (Templeton et al., 2024); and the success of transfer learning (Donahue et al., 2014), where a linear readout can be trained on top of learned representations to solve new tasks. These phenomena suggest that deep learning models encode various features in manner that allows for linear decoding. Yet, a comprehensive theory that explains why these properties emerge in deep learning models has remained elusive (Arora et al., 2016; Park et al., 2023).
We address this gap by building on the theory of Independent Component Analysis (ICA), which studies the conditions under which latent variables in probabilistic models can be uniquely identified (Comon, 1994; Hyvarinen et al., 2001). Recently, ICA has been extended to nonlinear models (Hyvärinen et al., 2023), providing a theoretical foundation for recovering latent variables in a broad class of machine learning tasks (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Gresele et al., 2019; Khemakhem et al., 2020a; Klindt et al., 2021; Khemakhem et al., 2020b; Locatello et al., 2020;
Joint first authorship: 1 Max Planck Institute for Intelligent Systems, Tübingen Al Center, ELLIS Institute, Tübingen. Germany: DDepartment of Computer Science, ETH Zürich and ETH Al Center, ETH Zürich, Zürich, Switzerland; DDeaarmentt of Computer Science, Brown University, Rhode Island, USA; "Cold Spring Harbor I Laboratory, Cold Spring Harbor, New York, USA;

---

Published as a conference paper at ICLR 2025
Rel
LNEr(X,) XEnt (Ef()))()))""
What
= ... N c= C.,
Wank
Re-1
Figure 1: DIET (Ibrahim et al., 2024) learns identifiable features: given samples and a d- dimennsiioaal latent representation, DIET learns a linear (N di-dimensional classification head Won top of nonlinear encoder f through an instance discrimination objective (1). For unit-normalized f (x.n), DIET maps samples and their augmentations close to the cluster vector ve corresponding to the class as if sampled from a von Mises-Fisher (vMF) distribution, centered around e.. For duplicate samples, i.e., matching class labels, the corresponding rows of W will be the same, as shown for 1 and æ; with w Wi.
Morioka al., 2021; Hälvä et al., 2021; Morioka & Hyvarinen, 2023). Most of these advances have focused on self-supervised learning (SSL) (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Zimmermann et al., 2021; von Kügelgen al., 2021; Rusak et al., 2024), i.e., when neural networks are trained by solving a surrogate (classification) task to learn from unlabeled data-the exceptions that study supervised learning, though either in the multitask setting, with single task with additional assumptions, include (Ahuja et al., 2022: Lachapelle et al., 2023; Fumero et al., 2023). However, we seek to understand whether similar identifiability guarantees can explain under what conditions cross-entropy- -based supervised learning, i.e., when the labels for the classification task are provided in the dataset, recovers interpretable and transferable representations.
Our journey starts with recent development in SSL: nonlinear ICA has been shown to provide iden- tifiability guarantees in contrastive learning, where models invert the data generating process (DGP) and recover latent variables up linear transformations (Hyvarinen et al., 2019; Zimmermann et al., 2021). Building on this insight, we first extend nonlinear ICA to a simple form of SSL-i.e. para- metric instance discrimination (PID) (Dosovitskiy et al., thrught the DIET method (Ibrahim et al., 2024), which streamlines the auxiliary task into an restance-discrmiing model the DGP in a new, cluster-centric way, and show that DIET's learned representation is linearly related to the ground-truth representation.
paradigm. We
From this foundation, we take the crucial step of extending the theoretical framework to the more common paradigm of supervised learning. Specifically, we show that models can recover ground- truth latent variables up to a linear transformation even in standard classification tasks using the cross-entropy loss, which is the most prevalent setting in modern machine learning. By doing so, aim to explain why deep learning, particularly supervised classification, is so effective in learning interpretable and transferable representations, offering a unifying framework to explain phenomena such as linear representations and neural analogy-making Thus, our theoretical insights offer potential explanation for the extraordinary success of supervised deep learning across a wide variety of tasks. Our contributions are
We propose a cluster-centric DGP as a model for the parametric instance discrimination method of Ibrahim et al. (2024) and prove the DGP's linear identifiability (Thm. 1);
We use our insight to extend the identifiability guarantee to standard cross-entropy- based supervised
classification under the cluster-centric DGP (Thm. 2);
We provide a "genealogy" cross-cnts pross-entrooyybeee based classification methods to connect our identifiabil- ity results in instance discrimination and supervised classification to auxiliary- vaaiiblee nonlinear Independent Component Analysis (ICA) (Hyvarinen et al., 2019) and self-supervised learning (SSL) ($ 3.4) (Zimmermann al., 2021);
We corroborate our findings in synthetic experiments matching our cluster-centric DGP, the DisLib disentanglement benchmark (Locatello al., 2019), and real-world ImageNet-X data (Idrissi et al., 2022), showing that the cross-entropy loss, irrespective of the meaningfulness of labels, can lead to linear identifiability of the features ($ 4).
BACKGROUND Empirical evidence of linear latent representation. The linear representation hypothesis (Park et al.. 2023) has lately received lot of attention. A weak version of this hypothesis could mean that there directions in neural activation space that correspond to interpretable features. In the
2

---

Published as a conference paper at ICLR 2025
case of neural analogy making, Mikolov et al. (2013) showed that there exist directions in word embeddings that are interpretable and preserved across input pairs. As an example for encoder f. producing latent variables z, the direction = f (man) f(woman) seems to correspond to gender and can be added to other words such as f(king) + z R f f(queen). Several datasets, such as the Google Analogy Dataset (GA) (Mikolov, 2013) and BATS (Drozd al., 2016), have been developed to evaluate neural analogy-making. These were, for instance, evaluated in (Dufter & Schütze, 2019). Theoretical explanations of linear representations have been proposed for word embeddings by Arora et al. (2016) and Allen & Hospedales (2019). Both approaches take a statistical learning theory perspective and focus on characterizing the pointwise mutual information. They do not consider cross-entropy-bent basedd classification; and, thus, do not make connection to supervised classification, as we do in Thm. Park et al. (2023) provide a framework to specify what exactly is meant by the linear representation hypothesis. They also provide a strong, causal hypothesis where finding that a feature is linearly represented does not imply that an intervention on that linear subspace will causally remove the feature from the model output. Engels et al. (2024) point out that some latent representations are not linear. This makes intuitive sense if we consider that some latent features, such as the pose of an object, have a -on-Euclidean topology that will have to be embedded a linear subspace of the latent representation (Higgins et al., 2018; Pfau et al., 2020; Keurti et al., 2023). For instance, the quadrature pair of sines and cosines representing rotations in 2D subspace in (Klindt et al., 2021, Fig. 15) depends on the object symmetries (Bouchacourt et al., 2021). Roeder et al. (2020) prove that different models trained with a discriminative objective converge to learning the same latent representation. Importantly, their claim is about the linear relationship between any two learned representations, and not the learned and the ground-truth one, as usually the case in identifibiliiy theory (Hyvarinen et al., 2001). They also show this empirically for pairs of models trained on different datasets. Their results are corroborated even with widely varying training factors by Moschella et al. (2023). These findings are also supported by recent large scale empirical studies in the converging representations of vision models (Chen & Bonner, 2023). This could also explain the recently proposed platonic representation hypothesis (Huh et al., 2024) about the convergence of representations, the improved disentanglement across model families (Du & Xiang. 2021). and the better identifiability of biological mechanisms (Genkin & Engel, 2020). However, these insights from the literature fail to connect the linearity of learned representations to the identifiability of the assumed ground truth DGP-thi the gap our contribution aims to address.
curved manifold in
learning and ICA. Independent Component Analysis (ICA) theory studies the conditions under which latent variables in probabilistic models can be uniquely identified (Comon, 1994; Hyvarinen et al., 2001). Identifiability means that the learned representation, at the global optimum of the training loss, relates to the ground-truth representation (i.c., the ground- truth latent variables underlying the data) via a "simple" transformation, such as permutations or elementwise invertible transformations this is different to investigations relating two instances of learned representations, such as in Roeder et al. (2020); Moschella et al. (2023); Zhang et al. (2023). Recently, ICA has been extended to nonlinear models (Hyvärinen et al., 2023), providing a theoretical foundation for recovering latent variables in a broad class of learning tasks (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019 Gresele al., 2019; Khemakhem et al.. 2020a; Klindt et al., 2021; Khemakhem et al., 2020b; Locatello et al., 2020; Morioka et al., 2021: Hälvä et al., 2021; Morioka & Hyvarinen, 2023). Most of these advances have focused on SSL., (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Zimmermann et al., 2021; von Kügelgen et al., 2021; Rusak et al., 2024).
Identifiable reakly-/seff-superviss
3 THEORY
This section presents main theoretical contribution. We start with our motivation to understand self-supervised learning (SSL) with the help of the simplified DIET (Ibrahim et al., 2024) algorithmic pipeline. For this, propose a cluster-centric data generating process (DGP) that can model semantic classes ($ 3.1). Then we state our main result in $ 3.2 and discuss an intuition behind the identifiability of the representation learned by DIET. We conclude by investigating how DIET fits into the vast literature of (identifiable) SSL and auxiliary-variabl Independent Component Analysis (ICA) methods ($ 3.4). This leads significant result for proving the identifiability of the latents learned via supervised classification under the DIET DGP ($ 3.3). We provide the technical details for Generalized Contrastive Learning (GCL) (Hyvarinen et al., 2019) in Appx. B.I and InfoNCE (Chen et al., 2020; Zimmermann et al., 2021) in Appx. B.3.
Motivation. Despite significant theoretical progress (Zimmermann et al., 2021; von Kügelgen et al., 2021; Rusak et al., 2024). it remains elusive why SSL methods work well in practice. Rusak et al. (2024) highlighted two remaining gaps between theory and practice: 1) practitioners often discard
3

---

Published as a conference paper at ICLR 2025
the encoder's last few layers (termed the projector) for better performance, despite identifiability guarantees not reflecting this fact; and 2) the data is presumably clustered, not reflected in the common assumption of uniform marginal. Despite similar terminology in auxiliary-variable nonlinear ICA algorithms, such as T'imeCContrastin GCL (Hyvarinen et al., 2019). it is unclear how such methods relate to SSL at large. Interestingly, the identifiability proof' for nonlinear ICA partition the model into separate encoder and a regression function (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019) and prove identifiability for the latent variables after the encoder, but before the regression function. This aligns with the practice of discarding the projector in SSL (Bordes et al., 2023), though identifiability results do not reflect this fact (Zimmermann et al., 2021; von Kügelgen et al., 2021; Rusak et al., 2024). These observations served as our motivation to investigate
Learning (TCL. (Hyvarinen & Morioka, 2016) or
How can we extend the identifiaillity guarantees to more realistic self- sule-supe classification scenarios, and can we apply these insights to improve our understanding of supervised learning?
Results overview. We aim to advance our theoretical understanding of SSL, for this, we use the recently proposed DIET (Ibrahim et al., 2024) (detailed in $ 3.1), which, beyond its simplicity, promises the strongest and most realistic results, based on similarities to GCL (Hyvarinen et al., 2019). Namely, DIET uses a separate encoder and classification head, and solves an auxiliary classification task akin to GCL furthermore, its loss correlates with downstream performance, non-obvious and welcome fact (Rusak et al., 2024). This provides the hope to resolve the two above points by modeling the cluster structure of the data and proving identifiability for the representation used for downstream tasks (Thm. 1). Subsequently, we leverage the insights from our identifiability theory and the DIET pipeline's 's similarity to supervised classification to show how the latter is special case of DIET, where the sample indices correspond to the semantic class labels (Thm. 2).
3.1 SETUP
DIET (Ibrahim et al., 2024). DIET solves an instance classification problem, where each sample in the training dataset of size N has unique instance label Augmentations do not affect this label. We have a composite model W f. where the backhone f produces d-dimensional representations, and a linear, bias-free free classification head W E Nxd maps these representations to a logit vector equal in size to the cardinality of the training dataset. If the parameter vector corresponding to logit i is denoted as w, then W effectively computes similarity scores (scalar products) between the w;'s and embeddings f(x). DIET trains this architecture to predict the correct instance label using multinomial regression (with f. W and temperature 3 as learnable variables), i.e., solves a parametric instance discrimination (PID) task (Dosovitskiy al., 2014; Wu et al., 2018):
((w))
LPTD (J, W, B) (e,ss) In Σ, 3(w, f(x))
An important fact is that (1) is the cross-entropy loss with instance labels, which we will leverage to connect instance discrimination to supervised classification.
The proposed cluster-centric data generating process (DGP). To prove the identifiability of the latent variables, we need to formally define a latent variable model (LVM) for the data generating process (DGP). We take a cluster-centric approach, representing semantic classes by cluster vectors. similar to proxy-based metric learning (Kirchhof et al., 2022). Then, we model the samples of a class with a von Mises-Fisher (vMF) distribution (intuitively, this is an isotropic multivariate Normal distribution that is restricted to the unit hypershere), centered around the class's 's cluster vector. This conditional distribution jointly models intra-class sample selection and augmentations of samples. together called intra-class variances. In contrast to conventional SSL methods such as InfoNCE (Zimmermann et al., 2021), this conceptually separates global and local structure in the latent space: 1) the cluster-vectors describe the global structure of the latent space; and 2) the cluster- centric conditional in (2) describes the local structure. This cluster-centric conditional embodies that data augmentations are selected such that they ought not to change the sample's 's semantic class. Our conditional does not mean that each sample pair transforms into each other via augmentations with high probability. It does mean that-since we assume a latent variable model (LVM) on the hypersphere; i.e., all semantic concepts (color, position, etc.) correspond to a continuous latent variable- the latent manifold connected, or equivalently, that the augmentation graph is connected, which is an assumption used in (Wang et al., 2022; Balestriero & LeCun, 2022; HaoChen et al., 2022). We provide an overview of our assumptions, and defer additional details to Assums. IC in Appx. A: Assumptions 1 (DGP with MF samples around cluster vectors. Simplified.).
4

---

Published as a conference paper at ICLR 2025
(i) There is finite set of semantic classes 66, represented by of unit-norm d-dimensional cluster-vectors {velc E '6) - s The system {ve} is sufficiently large and spread out.
(i) Any instance label belongs to exactly one class c c= C(i).
(ii) The latent variable z E &t-1 of our data sample with instance label is drawn from vMF distribution with concentration parameter K. around the cluster vector De of class = C(T):
p(2|c) X el
(2)
(iv) Sample æ is generated by passing latent z through an injective generator function: x: = g()).
3.2 MAIN RESULT: DIET IDENTIFIES BOTH LATENT VARIABLES AND CLUSTER VECTORS Under Assums. 1, we prove the identifiability of both the latent representations and the cluster vectors, Ve. in all four combinations of unit-normalized (i.e., when the latent space is the hypersphere, commonly used, e.g., in InfoNCE (Chen al., 2020)); and non-normalized (as in the original DIET paper (lbrahim et al., 2024)) learned embeddings, &, and weight vectors, W;, We state a concise version of our result and defer the full treatment and the proof to Thm. IC in Appx. A:
Theorem (Identifiability of latent variables drawn from vMF around cluster vectors. Simplified.). Let (f, W, globally minimize the DIET objective (1) under the following additional constraints:
C3. the embeddings fx) are unnormalized, while the w; 's are unit- mniitnormalized. Then w; identifies the cluster vector ve(i) up orthogonal linear transformation O: w; Ouc(sy for any i. Furthermore, the inferred latent variables ă f(x) identify the ground-truth latent variables up to a scaled orthogonal transformation with the same O: z = OOz.
C4. neither the embeddings f(x) nor the w; 's are unit noiiinormalized Then identifies the cluster vectors Ve up to an affine linear transformation. Furthermore, the inferred latent variables identify the ground-truth lalent variables z up to a linear transformation.
In all cases, the weight vectors belonging to samples of the same class are equal, i.e., for any i. C(i) =(() implies w: x---
Intuition. DIET assigns a different (instance) label and a unique weight vector w; to each training sample. The cross-entropy objective is optimized if the trained neural network distinguish between the samples. Thus, the learned representation å = far) should capture enough information to distinguish different samples, even from the same class. However, the weight vectors w, 's cannot be sensitive to the intra-class sample variance or the sample's 's instance label (because the conditional distribution over latent variables is identical for all samples of the same class). This leads to the weight vectors taking the values of the cluster vectors. As cluster vectors only capture some statistics of the conditional (1). feature recovery more fine-gr greined than cluster identifiability. The interaction between the two is dictated by the cross-entropy loss, which is minimized if the representation ž most similar to its own assigned weight vector w,, Fig. I provides a visualization conveying the intuition behind Thm. 1.
3.3 SUPERVISED CLASSIFICATION
This section relates cluster-centri DGP to supervised classification. To see how supervised machine learning is a special case of self-supervised approaches, consider that the sample index (i.c., the target of the cross-entropy loss) can be defined arbitrarily (as long Assums. 1 are still satisfied). This means that many labelings are possible, including the one used for supervised classification. This, in hindsight obvious insight has important consequences: it can explain the success of supervised cross-entropy- -based classification. Namely, supervised learning performs non-linear ICA under our proposed DGP (Assums. 1). We demonstrate this in $$ 4.1 and 4.3. We state a concise version of our result and defer the full treatment to Appx. A:
Theorem 2 (Identifiability of latent variables drawn from vMF around class vectors). Let Assum. 3 hold, and suppose that a continuous encoder f TR. + Red, a linear classifier W with rows { Ice E E),, and 3 0 globally minimize the cross-entropy objective:
[-
f(w))
Lsupervised (f, 3, =E
Ever e (wo 1(0)
Then, the composition h = f Pogin is linear map from 8d-1 Rd.
5

---

Published as a conference paper at ICLR 2025
(a)
(b)
(d)
GCL
TCL
DIET
Supervised
InfoNCE
The simplified genealogy of cross-entropy-basee classification methods (cf. Tab. 1 for details): The labeled arrows express how to go from general to special methods. (a) The most general auxiliary-variaabe ICA framework, Generalized Contrastive Learning (GCL) (Hyvarinen et al., 2019), yields
Figure 2:
Learning (TCL) (Hyvarinen & Morioka. 2016) the special case when the latent conditional is assumed to come from an exponential family (of order one) with a scalar auxiliary variable; (b) TCL. relates to non-unit-normalize DIET by further restricting the latent conditional to vMF distribution; (c) if the neural network used in InfoNCE is partitioned into linear classifier head and backbone, the marginal is assumed be vMF instead of uniform, we get the unit-normalized version of DIET; (d) if the labeling function in DIET is assumed to assign the semantic class labels the samples, we get classic supervised training
Intuition: In the context of DIET, the cross-entropy objective encourages the learned representations to align with the cluster vectors corresponding to each class. The identifiability of the latent variables ensured by the fact that the cluster structure reflects the underlying data distribution, modeled as a vMF distribution. This leads to representation that captures the latent structure up to an orthogonal transformation. Given the same underlying structure as DIET, supervised learning can be viewed as a special case of instance discrimination, where the instance labels are replaced by class labels. The cross-entropy objective, when applied to classification tasks and assuming our DGP from Assums. 1, similarly encourages representations to align with class vectors. As result, the latent variables are recovered up to linear transformation, providing a theoretical explanation for the success of supervised classification in learning linearly decodable representations.
3.4 THE GENEALOGY OF IDENTIFIABLA CLASSIFICATION WITH CROSS-ENTROPY
Our main result in Thm. 1, and its corollary for supervised classification (Thm. 2) suggest the following surprising conclusion to invert the proposed DGP (Assums. I):
Solving (almost) arbitrary classification task by optimizing the cross-entropy objective is sufficient to invert the DGP and identify the ground-truth representation up to a linear transformation. To show how solving a cross-entropy--aasee classification task is key component invert the DGP and to achieve linear identifiability, we provide a unified treatment of makiliary-variable ICA (i.c., weakly supervised self-supervised classification) and supervised classification methods. We call this genealogy to allude to the fact that these methods can be seen as special cases, descending from each other (cf. Fig. and Tab. for overview, and Appx. B for details).
From GCL to TCL (Fig. 2a: ar- bitrary scalar labels and exponen- tial family latent variables). The most general framework we consider Generalized Contrastive Learning (GCL) (Hyvarinen al., 2019), i.e., auxiliary-variable nonlinear ICA. GCL works with conditionally independent latent variables in Euclidean space given (possibly vector-valued iary information u. It aims to classify different values of u by distinguish- ing (æ, u) from (æ, u²), where u' is an arbitrary value of the auxiliary vari- able. At the Bayes optimum of the cross-entropy loss, GCL provides iden- tifiability of the latent variables after the encoder f. but before the classifier head W. up to elementwise invertible transformations.
Table 1: Comparison of the components of different cross-entropy-basee classification methods: denotes (possibly) vector-valued auxiliary variable, / is the scalar time step, the sample index, and e the semantic class; Exp- Fam stands for exponential family, 1n for conditionally independent sources given the auxiliary variable, W is the classifier head, f the encoder, whereas N/A stands for no assumption
auxil-
Property
GCL.
TCL.
InfoNCE
DIET
Supervised
Latent space Network Aux.info
Rd
Rd
8d-1
/Sd W o f
Rd W
W f W
f
I ExpFam
Conditional Marginal
I
vMF uniform
vMF uniform
vMF uniform
N/A
N/A
When the latent variables distributed

---

Published as a conference paper at ICLR 2025
Table 2: Identifiability results for parametric instance discrimination (PID) in numerical simulations: Mean standard deviation across 5 random seeds. Settings that match and violate our theoretical assumptions are denoted as v and respectively. We report the RC score for linear maps 2 - 2 and w; + Ve with normalized (subscript o) and not normalized (subscript a) w,, For normalized w;, we verify that the Σ + z maps are orthogonal by reporting the Mean Absolute Error (MAE) between their singular values and those of an orthogonal transformation.
normalized wi
unnormalized w. (1()) W. + We
(t)
MAE
N
d
88
p(2(00))
M.
_ + 2 WE + U2
- wi -> 0.
Σ - 2
10a
5
100
vMF(K-- 10)
V
98. 90.6.
99.9
0.01
z.00
99.0=z..0
99.9
105
5
100
vMF( 10)
98. 28.220.01
99.52..00
00 0.00
2.00
99.8 20.7
10a
5
100
<MF(K = 10)
V
98.6
98.6+0.01
99.9+o.oo
+o.00
0.00=o.o0
99.0
99.0+o.00
99.9+o.00
10
10
100
vMF( 10)
22.520.01
99.620.00
0.01 20.00
2.00
93.
22..0
99.620.0
10
20
100
VMI (N= 10)
70.8 V
97.1
0.00 0.03
99.7 20.9 20.01
10a
5
10
vMF( 10)
88.6 20.00
85.7.
0.02
0.00
90.0
99.020.00
10a
5
100
vMF x== 10)
V
98.6+0.01
99.9+..01
0.01+0.00
0.00=o.00
99. 00.00
99.9+a.o
10a
5
1000
vMF(K 10)
99.3++.00 V
99.9+o.co
0.00+0.00
0.00=o.00
99. +0.00
99.9+o.0o
103
5
100
vMF(x=5)
98.620.01
99.92o.01
0.01
0.00
99.0
99.8
10a
5
100
vMF(K AMF(K=10 10)
99.0 V
99.9
0.00
0.00=o.00
99. 99.9 #0.00
10a
5
100
vMF(k=500)
40.020.06
49.72..0s
0.30
0.00=z.0o
75.520.c0 22.5
10a
5
100
vMF(x= 10)
V
98. 620.01
99.92..01 01
0.01
0.00=a..0
99.0 99.9 20.00
00
.o
103
5
100
Laplace 1.0)
85. 220.01
99.72..01
0.01 10.00
2.00
85.420.00
99.5 .o.
103
5
100
Normal (a² 1.0)
X
98. 20.00
99.8
0.01 10.00
0.00
98. 20.020000
99.6 oo.
according to an exponential family distribution and the auxiliary variable is a scalar (c.g., time), then we get the more specialized method, named Time-Contrastivy Learning (TCL) (Hyvarinen & Morioka, 2016). If the order of the exponential family is one, identifiability holds only up to linear transformation, otherwise, up to elementwise invertible transformations.
From TCL DIET (Fig. 2b: sample index as and MMF latent variables). Using our cluster- centric DGP (Assums. 1), and assuming an even more special latent distribution (i.e., a vMF), we get the identifiability guarantee for DIET, i.e., our main result in Thm. 1. The auxiliary variable is a scalar for our result, too; however, instead of time, it is the (arbitrary) sample index.
From InfoNCE to DIET (Fig. 2c: compositional model W o I and unit-normalized latent variables). Importantly, our main result also encompasses unit-normalized representations, the conventional choice in (identifiable) SSL such InfoNCE (cf. Appx. B.3 for details on InfoNCE)- this is why we illustrate both InfoNCE and TCL as being the "parents" of DIET in Fig. 2. Thus, Thm. 1 is more general in terms of latent spaces than nonlinear ICA, and it proves identifiability for the latent variables that are used post-training, as opposed to the proofs for InfoNCE in (Zimmermann et al., 2021; Rusak et al., 2024), where practitioners discard the last few layers.
From DIET to supervised classification (Fig. 2d: semantic class labels). When the labeling function assigns the semantic class labels, and not arbitrary indices, then our identifiability result still holds, yielding the case of supervised learning (Thm. 2).
4 EMPIRICAL RESULTS
In $ 4.1, we empirically verify the claims made in Thm. 1 and Thm. 2 in the synthetic setting. We generate data samples according to Assums. 1: ground-truth latent variables are sampled around cluster centroids v. following vMF distribution. Data augmentations, which share the same instance label i, are sampled from the same vMF distribution around v.. In $ 4.2, we describe our results on the DisLib disentanglement benchmark (Locatello al., 2019), and 4.3 includes our experiments on ImageNet-X X (Idrissi et al., 2022). We made our code publicly available on GitHub!.
httpps://gite com /11 ndt aab/csi
7

---

Published as a conference paper at ICLR 2025
4.1 SYNTHETIC DATA
Setup. We consider N latent samples of dimensionality d generated from the conditional vMF p(z| sampled around a set of [6] class vectors ve, which are uniformly distributed across the unit hyper-sphere sd-1, We use an invertible multi-layer perceptron (MLP) to map ground-truth latent variables to data samples. We train a classification head W = [ww [ and an MLP encoder that maps samples to representations & E Rd using the DIET objective (I), While to verify Thm. I case C4., we do not normalize W. we do unit-normalize the weight vectors to validate Thm. case C3. We verify our theoretical claims by measuring the predictability of the ground- truth from & and Ue from w; using the R² score on held-out dataset (Wright, 1921). For identifiability up to orthogonal linear transformations, we train linear mappings with no intercept, assess the R2 score and verify that the singular values of this transformation converge to 1, while for identifiability up to affine linear transformations, simply assess the R of a linear predictor with intercept.
Results for DIET. In Tab. 2, we report the R scores for the recovery of the cluster vectors v, from W's rows and of the ground-truth latent
d
88
p(z/ve)
M. R2: 2- z
100
vMF(k 10)
09.82...0
variables x from the learned latent variables ž.
10
100
vMF(R 10)
97. 97.220.00
For DIET's PID task, also consider cases with row- oow-normalized W. We observe scores close to 100% (2 98%), even with many clus- ters (2 109) and samples (~ 10²). High latent dimensionality (> 10) does impact the recovery of ground-truth latent pariables-suce scalabil- ity problems are common artifact in SSL (Zim- mermann et al., 2021; Rusak et al., 2024). For a higher concentration of samples around ve (i.e., k= 50) as well as lower number of clus- ters (i.e., |8| = 10), the R² score decreases, which is also a common phenomenon, and is possibly explained by too strong augmentation overlap (Wang al., 2022; Rusak et al., 2024). For a low number of clusters, high i and fixed number of training samples, the concentration of samples in regions surrounding centroids, Ve. increases, a setting, refered as "overly over- lapping augmentations", known to be subopti- mal and leading to a drop in downstream per- formance (Wang et al., 2022). Our results also suggest that even under model misspecification
20
100
vMF(K= 10)
82.1
5
10
<MF(<= 10)
97.520 03
5
100
vMF (K= 10)
99.81 00
5
1000
VMF (R 10)
99.8+0.00
5
10000
VMF (R- 10)
99.8+0.00
100
VMF(K=
99.7
5 100
VMF(R= 10) vMF(K=50)
99.7
5
100
65.5
5
100
vMF(K= 10)
99.8.0 00
5
100
Laplace (b=1.0)
85.410.0
5
100
Normal (o²= (10=1.0)
X
99.620.o 00
Table 3: Identifiability results for supervised learning in numerical simulations: Mean a stan- dard deviation across 5 random seeds. Settings that match and violate our theoretical assumptions are denoted as / and X, respectively. We report the R² score for linear mappings ž + and not normalized w, We used N 10 samples
(last two rows in Tab. 2 with non-vMI distributions), identifiability still holds. For unit-normalized W rows, the MAE is lower, confirming the orthogonality of the map w; U.. We additionally ablate over batch size, concentration, and conditional in Appx. D.
Results for Supervised Classification. In Tab. 3, where the semantic class labels were used instead of the sample index, we only report the R² score for the recovery of the ground-truth latent variables z from the learned latent variables ž. In all but one setting, we observe higher R² from representations learned with class labels rather than instance indices. This suggests that even a coarser classification task may suffice to learn linearly identifiable representations of the underlying latent variables.
4.2 DISLIB
Setup. Next, we evaluate our methods on the DisLib disentanglement benchmark (Locatello et al., 2019), which provides a controlled setting for testing disentanglement and latent variable recovery. includes the vision datasets dSprites, Shapes 3D, MPI 3D, Cars 3D, and SOalINORB. We train both a three-layer MLP with 512 latent dimensions and BatchNorm (which helped with trainability) and a CNN (ResNet18) also with 512 latent dimensions. We only consider latent variables with Euclidean topology, as non- non-Euclidean, e.g., periodic latent variables such as orientation, are problematic to learn and are potentially mapped to a nonlinear manifold (Higgins al., 2018: Pfau et al., 2020; Keurti et al., 2023; Engels et al., 2024). We evaluate the recovery of latent variables by computing the Pearson correlation between ground-truth and predicted factors. We detail our setup in Appx. C.2.
8

---

Published as a conference paper at ICLR 2025
Table 4: Identifiability in DisLib datasets (Locatello al., 2019): We train different models to predict the categorical variable in each setting: (z)): as a baseline, from the inputs; (f (x)); from three-layer MIP; and (ff cwv (2)); from CNN (ResNet18). All continuous latent variables can be decoded from the learned representations, corroborated by the Pearson correlation-reportet with mean + standard deviation across 3 random seeds. Including the category is informative to see how well the underlying training classification task was solved.
Model
Latent
f M
(CN)(#)
dSprites
category
0.2622..o
2.
1.00
dSprites
scale
0.62
0.98
0.9220.05
dSprites
posX
0.92+a.00
0.97+0.on
0.99
dSprites
posY
0.92+0.00
0.97+0.00
0.99
Shapes 3D
category
0.42
1.00
1.00
Shapes 3D
objSize
0.200
0.89
0.99
Shapes 3D
objAzimuth
0.04+0.00
0.85+0.02
2.93
MPI 3D
category
0.03
0. 20.01
0.97
MPI 3D
posX
0.28
0.76
0.90
MPI 3D
pos
0.46+a.00 t0.00
0.76
0.84
MPI 3D real
category
0.19
0.88#0.01
0.98
MPI 3D real
posX
0.14
20.00
0.83
MPI 3D real
posY
t.44
2..4
0.71
Cars 3D
category
0.05
2.
0. 0.77
Cars 3D
elevation
0.152.000
0.87
0.78
smalINORB
category
0,22 +0.00
0.94+0.01
1.00
smalINORB
elevation
0.15+0.00
0.83
0.79
chance level
(init)
(trained) ResNet50 (trained)
Accuracy
Input
ResNet50 linit)
0.6
0.4
object blacking
pose
color
larger
smaller
style
shape
carker
class
1.0
level
Vie.b-16 (init)
Vit (trained)
Accuracy
Input
ResNat50 linit)
Reshett50 (trained)
0.4
background
brighter
pattern
partial
su bubcategory
muitiple
texture
person
view
objects
blocking
Approximate identifiability on ImageNet-X against a random (shuffled) baseline: Using ImageNet-X (Idrissi et al., 2022), test how well linear decoders able predict each latent from the second-to- tt-ttst layer of different models, i.e., when the classification head is discarded. We train linear classifier the features, and plot the accuracy of predicting different latent variables. As baselines, we also try decoding from the raw input and from the randomly initialized model representations. Error-bars indicate standard error of the mean (SEM) across 10 seeds of balanced resampling. Asterisks indicate significant p-values (against null hypothesis of 0.5 chance level accuracy) at an 0.05/83 multiple comparison (Bonferroni) adjusted significance level.
Figure 3:
The models trained using cross-entropy were able to recover latent variables such object position, scale, and orientation with high accuracy. As shown in Tab. 4, the Pearson correlation is generally highest when predicting the latent variables from the CNN's representation, which we attribute to the CNN's suitable inductive bias for images. In few cases, such as the position in dSprites, this can be done with fairly high accuracy even on the input data. Nevertheless, in all settings the nonlinear function estimated by the model is necessary to lincarly identify the correct latent variables.
Results.

---

Published as a conference paper at ICLR 2025
4.3 REAL DATA: IMAGENET-X
Setup. Finally, we test the generalizability of our theoretical insights on real-world data using ImageNet-X (Idrissi et al., 2022). The latent variables are binary proxies, defined by human annotators (Idrissi et al., 2022). We evaluate how well linear decoders can predict latent variables from pretrained model representations. We use two architectures, a ResNet50 and a Vit-b-16 both trained on standard supervised classification using a cross-entropy loss on the full ImageNet dataset (Deng et al., 2009). As baselines, we also decode from the inputs and the randomly initialized models. After balanced sub-sampling, over 10 random seeds, we report accuracies. We use t-tests against chance level of 50% with Bonferroni adjusted significance level of A = I7-5 Detail are in Appx. C.3.
0.05
Results. Fig. 3 shows that even in complex, high-dimensional data, latents can be linearly decoded from representations learned via supervised learning, in most cases significantly above chance level. Some factors (e.g., darker and brighter) are linearly decodable even from untrained models or input space. Unsurprisingly, decoding class (binarized ImageNet labels, every index < 500 is set to 0 and every index 2 500 is set to 1) works well for the trained models. ResNet50 has slightly higher decoding performance, possibly due to the larger latent space (d=2048, compared to d=768 in ViT). While texture information may be expected (Geirhos al., 2018), the presence of shape information suggests that shortcut learning may be mitigated even after standard training (Geirhos et al., 2020).
DISCUSSION
Limitations. One limitation of our work is that we mainly focus on synthetic and controlled datasets. While the results on ImageNet-X (Idrissi al., 2022) are promising, they only provide some supporting evidence for our theory on real data. The factors in ImageNet-X are likely not the true latent variables of the data generating process, still, the linear identifiablity results on these proxy latent variables support our theoretical results. Further experiments on other large-scale datasets would support the generality of our findings. However, this would require the availability of such datasets with full latent variable annotations. Although our cluster-centric modeling of the data generating process allows capturing the inherent structure of the data, our assumption about the latent variables' geometric properties (such being drawn from vMF distribution on a hypersphere), may not hold in all real-world settings. For instance, the pose of an object in a scene is, arguably, an independent component//suhspac corresponding to point on SO(3), which has distinct topology from our assumed latent variables on hypersphere. Moreover, the assumption that a data sample and its augmented version are conditionally independent given their semantic class could be relaxed in future work, since it may be misaligned with realistic scenarios (Wang et al., 2022). Despite these simplifications, our experimental results also suggest that our assumptions can be relaxed, as linear identifiability seems to hold even when some of the assumptions are violated (cf. Tab. 5). In Appx. D, we demonstrate the remarkable robustness of latent identifiability (Fig. 6), the interaction between batch size, latent dimensionality, concentration, and latent conditional.
Implications for Deep Learning. Our results indicate that deep learning models trained using cross-entropy and assuming a certain DGP recover the underlying latent variables up to linear transformations. As our identifiability proof for parametric instance discrimination illustrates with DIET, this statement also holds when the classification task is standard supervised learning. Our analysis on the key role of cross-entropy-basee bassd classification provides a theoretical foundation for phenomena such as neural malogy-making transfer learning, and linear decoding of features.
Conclusion. We extend the identifiability results of the auxiliary-variablle variable nonlinear Independent Component Analysis (ICA) literature to parametric instance discrimination with a cluster-centric data generating process. Our modeling choice can capture the clustered structure of the data, accommodates normmomal (as in ICA) and unit-normalized (as in InfoNCE) representations (Thm. 1). Furthermore, our identifiability result holds for the latent representation used post-training, i... for the latent variables before the classification head. Our results offer new insights into the success of deep learning, particularly in supervised classification tasks, which we show is a special case of the DIET parametric instance discrimination algorithm, where the instance labels equal the semantic class labels (Thm. 2). By linking self-supervised learning-vie nonlinear ICA and DIET-to supervised classification for a specific DGP, we provide a theoretical framework that explains why simple classification tasks recover interpretable and transferable representations.
Future Work. Future research could extend these insights to connections between nonlinear ICA and other forms of supervised learning and testing the scalability of our theoretical results to larger models and datasets. To assess our theory's 's predictions beyond proxy labels (Idrissi et al., 2022), we need real world image datasets with full specification of the latent variables, e.g., in rendered scenes.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Patrik Reizinger and Attila Juhos. Patrik Reizinger acknowledges his membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: OIIS A.039A. Wieland Brendel acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. Wieland Brendel is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. Alice Bizeul's work supported by an ETH Center Doctoral fellowship.
REFERENCES
Kartik Ahuja, Divyat Mahajan, Vasilis Syrgkanis, and Ioannis Mitliagkas. Towards efficient repre- sentation identification in supervised learning. In Proceedings of the First Conference on Causal Learning and Reasoning, pp. 19-43. PMLR, June 2022. URL https:::: httpss///ccceeeengg... html. ISSN: 2640-3498. 2
press/v171
Carl Allen and Timothy Hospedales. Analogies explained: Towards understanding word embeddings. In International Conference on Machine Learning, pp. 223-231. PMLR, 2019. 3
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. A Latent Variable Model Approach to PMI-based Word Embeddings. Transactions of the Association for Computational Linguistics, 4:385 3999,,9999900 July 2016. ISSN 2307-387X. doi: 2..1162ttacla.000 URL https: //doi.orgggeeooo 10. 11 11t///tallleeeeeeeeeaeeeaaetlaaa a0100_____00_0____11 1,
Randall Balestriero and Yann LeCun. Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods, June 2022. URL http::/arxxia org/abs/ 221./22255.. 11508. arXiv:2205.1 1508 [cs, math, stat]. 4
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions patlern analysis and muchine intelligence, 1798):1798-1888, 2013.
Florian Bordes, Randall Balestriero. Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning. Transactions on Machine Learning Research, May 2023. ISSN 2835-8856. URL https:// httpss///ppeeeeeewneeeee wtwprr/rwwww.neeeffoomriooo/eeee
Diane Bouchacourt, Mark Ibrahim, and Stéphane Deny. Addressing the topological defects of disentanglement via distributed operators. arXiv preprint arXiv:2 2021. 3
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. arXiv:2002.05700 [cs, stat), June 2020. URL http: /ttt:://axxvvv org/ httgg/////////.
/ 2002 05709. arXiv: 2002.05709. 3,5
Zirui Chen and Michael Bonner. Canonical dimensions of neural visual representation. Journal of Vision, 23(9):433744933 2023. 3
Pierre Comon. Independent component analysis, new concept? Signal processing, 36(3):287-314,
1994. 3,3
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009. 10, 28
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference machine learning, pp. 647-655. PMLR, 2014.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. Advances neural information processing systems, 27, 2014. 2,
Aleksandr Drozd, Anna Gladkova, and Satoshi Matsuoka. Word embeddings, analogies, and machine learning: Beyond king man+ woman= queen. In Proceedings of coling 2016, the 26th international conference on computational linguistics: Technical papers, pp. 3519-3530, 2016. 3
Kang Du and Yu Xiang. Causal Inference from Slowly Varying Nonstationary Processes. arXiv:2012. 13025 [cs, math, stat], September 2021. URL http://///arstt wwtp:/iwwwwwwrrr///oo//e///oass/ 13025. arXiv: 2012.13025. 3
org/abs /202..
Philipp Dufter and Hinrich Schütze. Analytical methods for interpretable ultradense word embeddings. arXiv preprinl arXiv:/904.00 1904408654, 2019.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodde Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superpo- sition, September 2022. URL httt:://////i/// org abl/ 2209 10652. arXiv:2209.1080 10652 [cs].
Joshua Engels, Isaac Liao, Eric J Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. arXiv preprint arXiv:2405. 14860, 2024. 27
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE transactions on neural networks and learning systems, 25(5):845-869, 2013. 30
Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning, April 2023. URL http: http///aaivv.....a www.iiiro...o.ooe.ooooo /tg/ab//////eaa/e////////////aa/ abs ab3/2304. 2304. 07939. 9. 1.Xx::2304.07939 [cs]. 2
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 12231, 2018. 10
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(166:65,6673, November 2020. ISSN 2522-5839 doi: 10.1038//2226-000--00
URL https: / www. nature com sarticlee/s4222 020-0065660000007-----20
02
Number: 11
Publisher: Nature Publishing Group. 10
Mikhail Genkin and Tatiana A Engel. Moving beyond generalization to accurate interpretation of flexible models. Nature machine intelligence, 2(11):674-683, 2020. 3
Luigi Gresele, Paul K. Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICA. arXiv:1908 (20.06920006622 [cs, stat), August 2019, URL htttp:://axxii..or/aa/////eeeeee 06642. arXiv: 1905.06642. 3,
Jeff Z. HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations Related Subpopulations, May 2022. URL http:/ www.i.ivvo..... wrt. /204 02.002668 683. arXiv:2204.0266 [cs]. 4
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016. 27
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loie Matthey, Danilo Rezende, and Alexander Lerehner. Towards Definition of Disentangled Representations. arXiv:1872.02233 (cs, stat], December 2018. URL http: arxiv. org abs /1812. 02230. arXiv: 1812.02230.
3, 8,
12

---

Published as a conference paper at ICLR 2025
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024. 3
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA. urXiv:/605.000 [cs, stat], May 2016. URL http::t/tt///t// org/abs 605.. 06336. arXiv: 1605.06336. 2, 6,7
Aapo Hyvarinen, Juha Karhunen, and Erkki Oja. Independent component analysis. J. Wiley, New York, 2001. ISBN 978-0-471-40540-5. 1,3
Aapo Hyvarinen, Hiroaki Sasaki, and Richard E. Turner. Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning. arXiv: /oo:.8055..5 /cs, stat], February 2019. URL http: //arxiv. wwg. absg///b//// / 1800 08651. arXiv: 1805.08651. 1, 2, 3, 4, 6, 24, 25, 26
Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural-equaation models: from linear nonlinear, February 2023. URL http://arxiv. org/abs/ 2302. 02672. arXiv:2302.02672 [cs, stat]. 1,3
Hermanni Hälvä, Sylvain Le Corff, Luc Lehéricy, Jonathan So, Yongjie Zhu, Elisabeth Gassiat, and Aapo Hyvarinen. Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA. arXiv:2106. 09620 [cs, stat], June 2021. URL http:://arxaaa /ttp///aaavvv...a/eea/eeee..ae www.aiiii.rrrrr...mmmmmm arXiv: 2106.09620. 2,
2106. 09620.
Mark Ibrahim, David Klindt, and Randall Balestriero. Occam's Razor for Self Supervised Learning: What Sufficient Learn Good Representations?. June 2024. URL http: /ttp:///axxvvvooo abss2406 0743. 10774 [cs]. 2, 4.
Badr Youbi Idrissi, Diane Bouchacourt, Randall Balestriero, Ivan Evtimov, Caner Hazirbas, Nicolas Ballas, Pascal Vincent, Michal Drozdzal, David Lopez-Paz, and Mark Ibrahim. Imagenet-x: Understanding model mistakes with factor variation annotations, 2022. URL https://
arxiv. org. abs 2211222111112 01866. 2, 10, 27
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding Dimensional Collapse in Contrastive Self-supervised Learning, April 2022. URL http: stttp://att htap:///axivv. perxii..rrg// ha///ab//22 abs//110. 09348. Number: arXiv:21 10.09348 arXiv:21 10.09349 [cs]. 29
Hamza Keurti, Patrik Reizinger, Bernhard Schölkopf, and Wieland Brendel. Desiderata for Represen- tation Learning from Identifiability, Disentanglement, and Proup-Structurednes. June 2023. URL 3,
attts::/cpperret
net forum?iici (errummidrr6666JJju
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational Autoen- coders and Nonlinear ICA: A Unifying Framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, June 2020a. URL http: http:///cocceeee..../.. mlr. wwwppppssssslpe v108 hww.00///keaaaaeemmmmmm html. ISSN: 2640-3498. 1,
Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, and Aapo Hyvärinen. ICE-BeeM: Identi- fiable Conditional Energy-Based Deep Models Based on Nonlinear ICA. arXiv:2002. 1137 [es, stat], October 2020b. URL http: /aaxiviivvv..////////aa////aaav. org/ abs /2022 11537, arXiv: 2002. 1,
3
Michael Kirchhof, Karsten Roth, Zeynep Akata, and Enkelejda Kasneci. Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learning, July 2022. URL http::/arxiv.. org/ abs/ 2207. 03784. arXiv:2207.00 .03784 [cs, stat].
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov. Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards Nonlinear Disentanglementt in Natural Data with Temporal Sparse Coding. arXiv:2007. 10930 [cs, stat), March 2021. URL tittp://arxxin wwwp.oorg/aasas... /2007. 10930. arXiv: 2007.10930. 1,3
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
13

---

Published as a conference paper at ICLR 2025
Sebastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Si- mon Lacoste-Julient and Quentin Bertrand. Synergies between Disentanglement and Sparsity: Generalization and Identifiabilityy in Multi Maski Learning. In Proceedings of the 40th Interna- tional Conference on Machine Learning, pp. 18171-18206. PMLR, July 2023. URL https: //proceedings mlm press
wtw.ggrssmr.ssess.a..m.. v220//22//////// 1atp//aachppppppp ee23a html. ISSN: 2640-3498.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disen- tangled Representations. In International Conference on Machine Learning, pp. 4114-4124 PMLR, May 2019. URL http:: /ttp:////oceee..s..//....oo mmr.ssm.r..... www.mm......///.....mmm wttps////////oooceleellll 9a. html. ISSN: 6640-3498. 2, 8, 27
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen.
Disentanglement Without Compromises. WrXiv:2002.02888 [cs, stat], October 2020. URL http:://aaaaate www. aaaaabss//00 02886. arXiv: 2002.02886. 1,
Tomas Mikolov. Efficient estimation of word representations in vector space. arXiv preprini arXiv: 1301.3811, 2013. 3
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Dis- tributed Representations of Words and Phrases and their Compositionality. In Ad- vances in Neural Information Processing Systems, volume 26. Curran Associates, nips. wwtppppp.cppppppp.epeppppppepppe files /wwes/paper//0///////// httpp/pppp////////eeee / 92tccee4222eeee00 Inc., 2013. URL -tt.:9/5533.22222ee0-bbbateeeeee 9aa42b33 (etbb11822ea. Bee.B882229.3..
html. 1,3
Hiroshi Morioka and Aapo Hyvarinen. Conneciiviy--conrastiee learning: Combining causal discov- ery and representation learning for multimodal data. Proceedings of The 26th International Conference Artificial Intelligence and Statisties, pp.
PMLR, April 2023. URL mlr. salrspress v22///226//////22222 http///mrroia22232 html. ISSN: 2640-3498.
3
Hiroshi Morioka, Hermanni Hälvä, and Aapo Hyvärinen. Independent Innovation Analysis for Nonlinear Vector Autoregressive Process. arXiv:2006. 10944 [cs, stat), February 2021. URL https://arxil: ortlvvorg//bb wwtgg///////aa /2066 10944. arXiv: 2006. 1006.10944, 10944. 2,3
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication, March 2023. URL http: http//aaxvvvvv..///////e.//..avl wttp///g////a//// /200/ 15430. arXiv:2209.15430 [cs].
David Nettleton, Albert Orriols-Puig, and Albert Fornells. A study of the effect of different types of noise the precision of supervised learning techniques. Artificial intelligence review, 33:275-306, 2010. 30
Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models, November 2023. URL http:://axiiiii http:://wxxxvv.rr.rooo org/ass// 2311. 03658. arXiv:231 11.0351 [cs, stat]. 1, 2,
David Pfau, Irina Higgins, Alex Botev, and Sébastien Racanière. Disentangling by Subspace Diffusion. In Advances in Neural Information Processing Systems, volume 33, pp. 17403-17415. Curran As- sociates, Inc., 2020, URL httptttpp://pocccceeeeteooooeeee http://proooeediinn...eerrrispss cc / htppp/pppp/222 2200//220///h/2/// hash/
.tml. 3.8.2 8.
Geoffrey Roeder, Luke Metz, and Diederik Kingma. Linear Identifiability of Learned Repre- sentations. vrXivv2207.008 10 (cs, stat), July 2020. /tt.tttppp///axi.....o////////// 00810. arXiv: 2007.00810
Rolnick. Deep learning is robust to massive label noise. arXiv preprint arXiv:1705.10694,
31
Karsten Roth, Mark Ibrahim, Zeynep Akata, Pascal Vincent, and Diane Bouchacourt. Disentangle- ment of correlated factors via hausdorff factorized support. arXiv preprint arX0v:2210.07347, 2022. 27
14

---

Published as a conference paper at ICLR 2025
Evgenia Rusak, Patrik Reizinger, Attila Juhos, Oliver Bringmann, Roland S. Zimmermann, and Wieland Brendel. InfoNCE: Identifying the Gap Between Theory and Practice, June 2024. URL http://arxiv. orgoor/g////.... abs / 2070 00143. arXiv:2407.00143 [es, stat]. 2. 4, 8, 29
James B. Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. On the Stepwise Nature of Self-Supervised Learning, May 2023. URL http://
axxiv.org/abe. 2t2g/abs//300 15438. arXiv:2303. 15438 [es]. 29
Adly Templeton, Tom Conerly, Jonathan Mareus, Jack Lindsey, Trenton Bricken, and al. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, 2024. URL https:
//transformer: circuits pubb/02411 /0/44//caaliie httca/llaaaanoooooooo monosemanticitg..
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style, June 2021. URL
htttt://arzii..rrrgg http:///aai....or///////0// 04619,
arXiv: 2106.04619. 2, 29
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos a Ladder: New Theoretical Understanding of Contrastive Learning via Augmentation Overlap. May 2022. URL
http://axxiv.. wwtp:/iwwwrrrr.//.. 2203 13457, arXiv:2203. 13457 [cs, stat]. 4, 8, 10
Wikipedia. Gibbs' inequality, 2024a. URL
org/w/ iidein.pppii citle=Gibbs$$2_iing [aleee/boeselnnaaalllyylllllllaa Online; accessed 10-September- 2024. 18
Wikipedia. Tietze extension theorem, 2024b.
http://ww..ttteeee//.....caaaaoa ond./iiexx php?title-Tiete Ttt.tpp__sttessssoottooooeeeeeed 10-September:2024. 17
Online; accessed
Sewall Wright. Correlation and causation. Journal of Agricultural Research, (7), 1921. 8
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised Fea-
ture Learning via Parameett Instance Discrimination. pp. 3733-3742, 2018. URL https: opw.accessccssscpeepee/ecossssse thecwf
2vpr_2018 //tmm htth//mm /We_ html..
Qi Zhang, Yifei Wang, and Yisen Wang. Identifiable Contrastive Learning with Automatic Fea- ture Importance Discovery. October 2023. URL http: wwtp:///.rr//////////.oo/ooose/ http///bs//3......//// 18904. arXiv:2310. 18904 [cs].
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive Learning Inverts the Data Generating Process. arXiv:2102.0m 0885000000000000 [cs]. February 2021. URL http://///// /ttp/a/avvv////////////e///aaaia org/ abs 22022 08850. arXiv: 2102.08850. 2, 4, 7, 8, 29
15

---

Published as a conference paper at ICLR 2025
A IDENTIFIABILITY OF LATENTS DRAWN FROM A VMF AROUND CLUSTER VECTORS
This section contains the formal statement and proof of our main theoretical result. Appx. A.1 contains the relevant definition of affine generator systems. Appx. A.2 contains the assumptions and the proof for all four combinations of nomi-normllized and non-normllized features/cluster vectors for parametric instance discrimination. Appx. A.3 discusses special case, supervised classification.
A.1 AFFINE GENERATOR SYSTEMS
Definition 1 (Affine Generator System). A system of vectors {ve E R E } is called an affine generator system if any vector in R& is an affine linear combination of the vectors in the system. Put into symbols: for any E Rd there exist coefficients Or, E R. such that
Σavee
aave and Σa.=1.
(3)
09'
CE'6
Lemma 1 (Properties of affine generator systems), The following hold for any affine generator system {ve RdIc }
1. for any E 66 the system {v. valc }6) is now generator system of : the invertible linear image of an affine generator system also an affine generator system.
A.2 IDENTIFABILITY OF PARAMETRIC INSTANCE DISCRIMINATION
Assumptions 1C (DGP with vMF samples around cluster vectors). Assume the following DGP:
(i) There exists a finite set of classes E, represented vectors [velc }6 C Sd such that they form an affine generator system of R (i) There is a finite set of instance labels I and well-defined, surjective class function C:
set of unit-norm d-dimensional cluster-
'6 (every label belongs to exactly one class and every class is in use).
(ii)) A data sample x belongs to class C(T) and is labeled with a uniformly-chosen instance label, iee., I E Uni(F).
(iv) The latent z gd-1 of our data sample with label is drawn from a vMF distribution around the cluster vector vc, where C. = C(I):
~ p(=|C) x (x((..))
(v) The data sample æ is generated by passing the latent through a continuous and injective generator function g ::d - i.e.. x = g(z).
Assume that, using the DIET objective (6), we train a continuous encoder f TR. Rd on æ and a linear classification head W top of f. The rows of W are { | E J In other words, W computes similarities (scalar products) between its rows and the embeddings:
W (ax) + [(w, f(x)) [ser).
(5)
In DIET, optimize the following objective among all possible continuous encoders f. linear classifiers W, and 3 > 0:
f(w)
E(,,)) (-ln Ejes eas, ((m))
C(S. W. E(a.))
(6)
In the special case where the embeddings f(x are unnormalized, but the parameter vectors w, are unit-normalized, the identifiability proof will solicit another, technical assumption:
Assumption 2 (Diverse data). The system (v.lc '6} is said to be diverse enough, if the following [8] x 2d matrix has full colunn rank of 2d:
(U. V.))
V
where (x 3 y): = x:3 is the elementwise- or Hadamard product.
As long [66 > 2d, this property holds almost surely w.r.. the Lebesgue of any continuous probability distribution of v. E Sd-1
16

---

Published as a conference paper at ICLR 2025
Theorem 1C (Identifiability of latents drawn from vMF around cluster vectors). Let (f. W ,)) globally minimize the DIET objective (6) under Assums. IC and the following additional constraints:
CI. both the embeddings f() and w, 's are unit-normalized. Then:
(a) = f о g is orthogonal linear, i.e, the latents are identified up to an orthogonal linear transformation;
(b) w; h((C(() for any i F, i.e., wi's identify cluster-vectors v. up to the same orthogonal linear trunsformation;
(c) == K, the temperature of the vMF distribution is also identified.
C2. the embeddings (x) are unit-normalized, the wi 's are unnormalized. Then:
(a) f g is orthogonal linear;
(b) w; (vv(()) + +b for any i I, where up is a constant vector independent of i.
C3. the embeddings far) are unnormalized, while the w; 's are unit-normalized. If the system {velc} is diverse enough in the sense of Assum. 2, then:
(a) w; Ove(s)) for any i E I, where O is orthogonal linear;
(b) h og = 0 with the same orthogonal linear transformation, but scaled with 3.
C4. neither the embeddings f(x nor the rows of W are unit-normalized. Then:
(a) = f g is linear;
(b) we identifies ve(.) up to an affine linear transformation.
Furthermore, in all cases, the row vectors that belong to sumples of the same class are equal, ie., for any i.j I, C(i)= C(j) implies w; = u.
Remark. In cases C2 and C4, the cluster vectors are unnormalized and, therefore, can absorb the temperature parameter B. Thus 3 can be set to 1 without loss of generality. In case C3, it is f that can absorb 8.
Proof. Step 1: Deriving an equation characterizing the global optimizers of the objective.
Rewriting the objective in terms of latents: we weplug the expression æ g(z) into the optimization objective (6) to express the dependence in terms of the latents 2:
C(f. W, W 3) = EB(,,) [- -Im EjEr Sog.
C.(ffo.
(8)
where the optimization is still over f (and not = g).
We note that the generator g is, by assumption, continuously invertible on the compact set Sd-1 Therefore, its image ( is compact, too, and its inverse g- is also continuous. By Tietze's extension theorem (Wikipedia, 2024b). g can be continuously extended to a function F : R gd Therefore, any continuous function h Sd-1 + Rd can take the role of o (F gog) = h idga-1 = h. Hence, minimizing Lz(f g. W. () (and by extension L(f, W. .)) for continuous f equates to minimizing (h, W, .) for continuous h:
by substituting
f = h continuous, since now f
B(wi (()))
(h, W B) = ((,,))
=E
In
(9)
Ejer ((m,
Expressing the condition for global optimality of the objective: We rewrite the objective (9) by 1) using the indicator variable òz=; of the event {1 = and 2) applying the law of total expectation:
B ((w))
Cz,h, W.
)) W)
B(2,1)
(10)
Σ
Ejer ((()))
iE.
((w),)(())
= Ex
E
81 In
(11)
E Ejeg Cimmshhll
i
17

---

Published as a conference paper at ICLR 2025
Using the properties that [[A =E [[A that:
and that B[8 = P(I 1), conclude
C2 W 3) Ez
Er
i In
(12)
Σ EjEr el(w,,
e (m(,)((2)))
Ez
Σ ie.r
Ey
Ö1=1|
In Σ jEar eliee,
(13)
Ez
Σ
P
= 1|1)
(14)
Ejer (((x),
iE.
By Gibbs' inequality (Wikipedia, 2024a), the cross-entropy inside the expectation is globally mini- mized if and only if
((w,)(z))
Ejer (lw,,h())) PI1 = 1|2), for any E J.
(15)
Moreover, the entire expectation globally minimized if and only if the above equality (15) holds almost everywhere for z E Sd-1
Using that instance label is uniformly distributed, or PP = sample being in class can be expressed via Bayes' theorem as:
PII = i). the likelihood of the
p(z I i) P(T 1|2) Ejer p= P) Ejer p(=[T=))
p(z (I = i)
(16)
Substituting (16) into (15) yields that for any i E I and almost everywhere w.r.t. 2 E 8d-1, (((w))(()))
p(z = i) ΣSE,
(17)
Ejer e(
We now divide the equation (17) for the probability of a sample having label i with that of having label & and take the logarithm. This yields that Cz,h, W, .) is globally minimized if and only if
B(w, wk. h=)) = In p(z 1 = i)
(18)
p 1 = k) holds for any i, k I and almost everywhere w.r.t. z E Sd-1
Plugging in the vMF distribution: Plugging the assumed conditional distribution from (4) into (18) yields the equivalent expression:
((w; - We. h(z)) = k(VC() UC(k)) x),
(19)
which holds for any i E I and almost everywhere w.r.t. z E Sd-1, Since h is continuous, the equation holds almost everywhere w.r.t. if and only it holds for all 1d-1
Observe that if h idlga-1, w. (((()) for any i E f, and = k, then the equation is satisfied. Thus, we can conclude that the global minimum of the cross-entropy loss achieved.
Step 2: Solving the equation for h. W and proving identifiability.
We now find all solutions to prove the identifiability of the latent variables and that of the cluster vectors. Denote w. _ w. to simplify the above equation to:
(w: wr h(z)) (vC() CC(R), z).
(20)
is injective and has full-dimensional image: We prove that h is injective. Assume that h(z1) = h(22) ) for some 21. 22 E gd-1, Plugging and 22 into (20) and subtracting the two equations yields:
0 (w, wR h(2)) h(z2)) = (vet) - (c(R), 21 -2z),
(21)
for any k. However, the cluster vectors {valc) form affine generator system, the vectors (Pe() DC(a) k-} form a generator system of Rd (see Lem. 1). Therefore, {y. 21 22) = 0, for
any y E Rd, which holds if and only if 2 = 22. Hence, h is injective.
By the Borsuk-UN -lam theorem, for any continuous map from gd-1 to a space of dimensionality at most d-1 1 there exists some pair of antipodal points that mapped to the same point. Consequently, no such function can be injective at the same time. Since gd - d injective, the linear span of its image must be Rdd
18

---

Published as a conference paper at ICLR 2025
Collapse of w, 's: We prove that it; w if (1) C(k), i.e., samples from the same cluster will have equal rows of W associated with them.
Assume that ((t) ((k) and substitute them into (20):
(it; ii h())) = 0 for any gd-1
(22)
However, we have just seen that the linear span of the image of h is TRd, which implies that it; k. We may abuse our notation by setting w. = w. if (() = c, which yields a new form for (20):
(i wv. h (z)) = (va Lb 2)
(23)
for any b e & and any E gd-1
Linear transformation from Va Ub to w. wEs We now prove the existence of a linear map A Rdd such that A(va Vb) Wa wh for any o, b B. For this, we prove that the following mapping is well-defined:
A: Σ a.bE'
Aab(Ua - U5) + Σ
Aas(ia wb.
(24)
a
Since the system fea vbla, b) is not necessarily linearly independent, we have prove that the mapping is independent of the choice of the linear combination. More precisely if for some coefficients Xab, Nab
Aub - -v6) -b)= = Σ
(u -b)
(25)
O.
holds, then it should be implied that
Σ
Aas(Ta w=)
Σ
Yao - ins))
(26)
a.
0.6E%
Assume that (25) holds. Then, the difference of the two sides is:
0 = Σ (Aab - Xab)(Va Uo).
(27)
a.EE%
Taking the scalar product with arbitrary @d-1 and using the linearity of the scalar product gives us:
0 Σ
(Aas Nat)(Va 2)
Σ (Aah Nat) Us. 2).
(28)
a.DE'
0.EE'
Now using (23) yields:
Σ (Nab ( wb, h(2)) = Σ (Nab - Nas) w)), h(2)).
(29)
a.DE'
However, the linear span of the image of h is Rd, which implies that
Σ (Aab (wx w.) = 0,
(30)
equivalent to (26). Therefore, the mapping is well-defined and the linearity of A follows.
linear: Equation (23) becomes:
(A(u, - vb), (()) = (v. - Ub. z),
(31)
for any a, b E E and any E gd-1 Nevertheless, (va- ubla, b E)) is generator system of gd
and, hence, (31) is equivalent to
(Ay. h(z)) ().2), for any y E Rd and any 1
(32)
This is further equivalent
(y, A (b,A)(z))) ( 2).
(33)
Since is arbitrary, we conclude that A h(z) = z for any z E gd-1 Therefore AA is an invertible
transformation and h (AT)-1 is linear.
19

---

Published as a conference paper at ICLR 2025
Proving Thm. 1C case C4: We have shown that h is linear. Furthermore, from (31) it follows, by fixing and defining & Ar, Wb, that
W Av + +1, for any E C,
(34)
which proves case C4 of Thm. IC.
Proving Thm. 1C case C2: As special case of the previous one, now assume that h(z) unit-normalized and maps g to gd-1_ That amounts to h = (AT)-1 being linear, norm-
preserving, and therefore orthogonal. Consequently A is also orthogonal, h = A and (34) simplifies to 2 uh wa Av. 1/ h(va) t, which proves C2 of Thm. IC.
Proving Thm. 1C case C1: We now assume that both h and w, 's are unit-normalized. Conse- quently, h = A is orthogonal linear and w. = Av + tb.
Therefore, one hand, the w, lie a d-dimensional hypersphere of radius I and center sb. On the other hand, by definition, wa 's also lie on the unit hypersphere gd-1,
Since the system (wal la E 83 the bijective affine linear image of the affine generator system {va aa E }.. {w.la '6' is also an affine generator system (Lem. 1). Consequently, there could be at most one hypersphere id which contains the w. ss Hence 2 = 1, &b O, and w, = h(v.).
which proves C1 of Thm. IC.
Proving Thm. 1C case C3: Finally, we assume that w; 's are unit-normalized, As this is a special case of Thm. 1C C4, we know that there exists a constant vector //s such that:
K Av, + vb Wa B
(35)
for any a E 8. We are going to prove that O A is orthogonal and sb = 0.
Let O <T ΣV be the singular value decomposition (SVD) of O. Premultiplying with U yields: Uw, >VV. + Unb.
(36)
As orthogonal transformations UI and keep their arguments unit-normalized and (Ve, Vos3 still an affine generator system (Lem. 1), we may assume without the loss of generality that
w. a Ev. + 2b,
(37)
for any E 6, where all v, 's and w, are unit-normalized.
Let us assume that &b + O. In that case both sides of (37) can be scaled such that the offset sb has unit norm. this case wa's are no longer on the unit hypersphere, but they instead have mutual norm Assuming that the diagonal elements of Σ are (a), (oI,..., oa), this is equivalent to:
2 Ev, + = 1 2 + 2(E +b) - || (va Va. σ) + (v., 20 3 v) + 1,
(38)
(39)
where [x 3 y/. riy the elementwise product. Eq. (39) is equivalent the following:
(va ② v.) (ar o o) + o. (2ar 2 v) - 2 =-1
(40)
Collecting the equations for all a E 8 yields:
o
D
20 C tb
=-1/6
(41)
where D the following [8] x (2d + 1) matrix:
==
(v. ① VT)
v
-1
(42)
20

---

Published as a conference paper at ICLR 2025
By Assum. the left [6]| x 2d submatrix of D has full rank of 2d. Consequently, the solution space to the more general, linear equation Dt = =-1p E R has dimensionality of at most 1. By the unit-normality of Va. we have (va @ V&a) 1. = 1. From this, the solutions are exactly the following:
1d
=
0,
where y E R.
(43)
? +
Therefore, for any solution of (41) there exists y such that:
Y ld
(44)
// O₂-
(45)
However, as the original transformation A was invertible, all singular values are strictly positive and, thus, it follows that v/b = This is a technical contradiction to our initial assumption that abb + Thus, it follows that 2/- 0. Therefore, (37) becomes:
Wa Ev.,
(46)
where all v. 's and w, 's are unit-normalized, Following the same derivation yields:
1= 1=|EE, 1= (va @ va) Va O
(47)
or, after collecting the equations for all a E 66:
Blo @ = 161
(48)
where B is the |8| x d matrix
B=
(U. O V.)
(49)
By Assum. 2, B has full rank, thus, there is most one solution the equation Bt = 1 Due to the norm-noryylyty of v. 's,, this solution is exactly t = 1g. However, as the singular values o; are all positive, the only solution to a ④ a la is a 1 Equivalently, = A is orthogonal.
Furthermore, h (AT)-1 (60T)-1= O.
A.3 IDENTIFIABILITY OF SUPERVISED CLASSIFICATION
Assumption 3 (DGP with vMF samples around cluster vectors). Assume the following DGP: (i) There exists a finite set of classes 66 represented by set of unit-norm d-dimensional cluster- vectors [valc {) C Sd-1 such that they form affine generator system of Rd.
(i) A data sample x belongs to a uniformly chosen class C E Uni(6).
(iii) The latent E gd-1 of our data sample x with label is drawn from a vMF distribution around the cluster vector vc:
p(z|C) x cx(
(50)
(iv) The data sample x is generated by passing the latent through continuous and injective generator function g :: - R. i.e., x g(z).
We would like to point out that the assumption of the class label being uniform restricts the scope the following theorem as it cannot account for imbalanced class labels. This shortcoming did not affect Thm. 1, as the uniform distribution over instance labels a natural choice in practical scenarios with finite datasets.
Theorem 2C (Identifiability of latent variables drawn from vMF around class vectors). Let Assums. IC hold and suppose that a continuous encoder f RD. > Rdd a linear classifier W with rows (w) c 8), and 3 0 globally minimize the cross-entropy objective:
C(F, W. W,3) = E [ [Im
((wc.((e))
Ever el(my.
Then, the composition h fog a linear map from 8d-1 to Rdd
21

---

Published as a conference paper at ICLR 2025
Proof.
Step Rewriting the Objective in Terms h. We begin by expressing the loss function terms of the latent variable z. Recall that æ g(2 and h e g. Substituting into the loss function:
[-m In Ever piw h(*))
((wc
C(S, W. 3) ,,
B(x,0)
(51)
Since g is continuous and injective on the compact Sd-1 its inverse g exists and continuous 1-g-11 on g (Sd-1) ) By Tietze's extension theorem, we can extend g to a continuous function Jext RD + gd Therefore, any continuous function h Sd-1 + gd corresponds continuous -1 satisfying f(x) h(z).
encoder f = h 9exi
Step 2: Optimality Condition of the Cross-Entrod Loss. At the global minimum of the cross- entropy loss, the predicted class probabilities match the conditional probabilities almost every- where. That is, for all z E gd-1 and all E 8:
Eeer (lwyyha))) P(C el z).
(52)
Step Expressing the True Conditional Probabilities. Using Bayes' theorem and the fact that classes are uniformly distributed (P(C c) is constant"), we have:
p(z CC
P(C = c 2) Evee p(z C c))
(53)
Given that, by assumption, the latent follows von Mises-Fisher (vMF) distribution centered v.:
p CC= c) x ex(
(54)
Substituting into the conditional probability:
ek((vr1))
P(C - 2) Ed eNil
(55)
Step 4: Equating Predicted and True Probabilities. Setting the predicted probabilities equal to the true probabilities, we obtain:
k(w))
((ve.2)
(56)
Ever (
Eeee (m(m)
Dividing the expressions for classes e and d', we eliminate the denominators:
h(x))
(57)
3w h(z)) en(w,r,2)
Taking the logarithm of both sides:
3 ((we, h(z)) (w., h(x))) = x ((Ve, - (Ve..))
(58)
Simplifying:
B(w. we, h(x))) x(v. - -U,,
(59)
We acknowledge that this assumption does not hold in many realistic scenarios, where the data distribution unhalanced hetween the classes
22

---

Published as a conference paper at ICLR 2025
Step 5: Defining Scaled Parameters.
Let define:
B We w. h
(60)
Then the key equation becomes:
(w.- - We, h(z)) = (Ve ve,z), Ve,c' B.
(61)
Step 6: Establishing Linear Relationship. Define the difference vectors:
de ve -We: deed v. We.
(62)
Our key equation is now:
(Bm h(z)) ()), Ve, E 8.
(63)
Since the set (ō I c/ E 8 spans Rd (due to the affine generator system property), we can interpret this equation stating that the inner products between h(z) and ów... correspond the inner products between z and óe
Step 7: Proving Injectivity and Full Rank of h. Suppose there exist 21, 22 E Sd-1 such that h(21) h(2)). Then, for all c, e' 8:
(6e_ 21 x2) = (6 h(z1) - h(x2)) = 0.
(64)
Since (ó } spans Rd. it follows that 2 22 0, i..., 21 x2. Therefore, h is injective.
By the Borsuk-Ulan theorem, an injective continuous map from S to Rd with d' < d cannot exist. Thus, the image of h must be full-dimensional in R.
Step 8: Defining a Linear Map A. We aim to find a linear map A Rd + Rd such that:
8 AT AT doger
Ve, c' E C.
(65)
This is well-defined because any linear dependency among the p translates to the same linear dependency among the : shown below. Suppose there are scalars (Xec) } such that:
Σ Nee dw 0.
(66)
Then, using the key equation (63):
Nec (0am.) h((2)) = E Noc (Do
Σ Necd 0.
(67)
Since h is injective and its image spans Rdd the only way for this to hold for all h(x) is if:
doo 0.
(68)
Therefore, AT is well-defined linear map.
23

---

Published as a conference paper at ICLR 2025
Step 9: Concluding that h is Linear.
Using the linear map A the key equation becomes:
(AT 8 co l((z)) (6 : 2).
(69)
This implies:
(8 Ah(z) 2) =0, Ve, E B..
(70)
Since (dear } spans RA. it follows that:
Ah(z) =2, Vz E Sd
(71)
Therefore, h is the inverse of A restricted to gd-1, and since A is linear and invertible (due to the injectivity of h), follows that h is linear:
h(z) = A-12.
(72)
This completes the proof that h is linear.
Step 10: Conclusion. Under the given assumptions, we have shown that = g must be linear function. This means that the latent variables z are identifiable up to linear transformation determined by A-1
B THE GENEALOGY OF CROSS-ENTROPY-BASHH CLASSIFICATION METHODS
This section provides the necessary background on auxiliary-variable ICA and discusses the connec- tion between ICA and DIET, and InfoNCE and DIET.
B. AUXILIARY-VARIARR NONLINEAR ICA: GENERALIZED CONTRASTIVE LEARNING (GCL)
In this section, discuss the most general quxiliary-variable nonlinear ICA, termed Generalized Contrastive Learning (GCL) (Hyvarinen et al., 2019). GCL uses conditionally factorizing source distribution (given auxiliary variable u): log p(s|u) is sum of components (s(S;, u):
log/(s/u) = Σ (8; u)
(73)
For this generalized model, Hyvarinen et al. (2019) define the following variability condition: Assumption 4 (Assumption of Variability). For any y R (used drop-in replacement for the sources s), there exist 2n + 1 values for the auxiliary variable u, denoted by uy, = O... 22 such that the 2n vectors in TR. 2m given by
(w (y, un) w (y, uu)) (w (y, w (y, uo)) ... (w u2,
U22)
(y, uu))
with
w(y.u) Oqi (y1, u)
Oqn (yns u)
8q (v), u)
Pgn (yn: u)
Oy
Byn
Oyf
are linearly independent.
Assum. constrains the components of the first- and second derivatives of the functions constituting the sources' conditional log-density, given the auxiliary variable As the authors write: "[Assum. 4] basically saying that the auxiliary variable must have a sufficiently strong and diverse effect on the distributions of the independent components."
We state the required assumptions for the identifiability of GCL, adapted from (Hyvarinen et al., 2019, Thm. 1):
Assumption 5 suuxiliary-variablle ICA with conditionally independent sources (GCL)). We assume the following for latent factors z, observations æ, generative model g. encoder f(parametrized by a neural network), linear map W With (f, W) solving multinomial regression problem:
24

---

Published as a conference paper at ICLR 2025
1. The observations are generated with diffeomorphism g g(z), where dim x dim The source components 24 are conditionally independent, given a fully observed, m- -n-dimensional random variable (RV) u, i.e.
logp(z/u) Ea(t, u),
(74)
The conditional log-pdf qu is sufficiently smooth as a function of zı for any fixed
4. Assum. 6 holds the multinomial regression function
r(e, u) = Σ (ff(()))
(75)
discriminating (x:, u) vs (x,, u²) has universal approximation capability, both for th and diffeomorphic f= (fis...,, (parametrized by a neural network)
When Assum. 5 holds, Hyvarinen et al. (2019) showed identifiability up to component-wise invertible transformations.
For the special when the conditional distribution comes from the exponential family (in the case of our chosen vMF conditional, the distribution has order one), Assum. turns into a simpler form (Assum. 6).
B.2 PARAMETRIC INSTANCE DISCRIMINATION (DIET) AND TIME-CONTRASTIVE LEARNING (TCL)
Arbitrary labels: time and sample index As Hyvarinen et al. (2019) note in (Hyvarinen et al., 2019, 5.4), u can stand for many types of additional information. TCL uses the time index, which assumed to be RV. Importantly, arbitrarily defined class label, such as in DIET, can serve the same purpose. In this case, we denote the auxiliary variable u =
Adapting the assumptions between TCL and DIET. The only reason we cannot apply (Hyvarinen et al., 2019, Thm. 1) is that our exponential family has order one, violating Assum. 4. This fact, however, shows our theory's 's consistency as we cannot go beyond identifiability up to linear (orthogonal or affine) transformation.
To fit our theory into the ICA family of methods, we note that modeling the DGP in DIET with a cluster-centric approach, naturally fit most of the ICA assumptions. To compare our Assums. 1C to all the assumptions used for (Hyvarinen et al., 2019, Thm. 3) (cf. Assum. 5), we note that the vMF distribution belongs to the exponential family, and that requiring that the cluster vectors form an affine generator system (cf. Appx. A.I for a definition and properties) satisfies the special case of the general sufficient variability Assum. 4 condition:
Assumption 6 (Sufficient variability). Define the modulation parameter matrix L TR. (E-1)xxx for d- d-dimensional exponential family distributions of order k with rows as:
[L]; (0) 0²0T
(76)
Ø¹ 10] A3 3.
(77)
Then, sufficient variability means that rank(L) = dk, i.e., the modulation parameter matrix has full column rank.
To see how VMF fulfills Assum. 6, consider that the log-pdf q:(2,,) comes from a conditional exponential family, i.e.:
Os (2,, Σ
Mgy (z.) (c)(cc) - log N.(c) log Qi(2)),
(78)
1=1
=K z) + log (1))
(79)
where & is the order of the exponential family, N is the normalizing constant, Q; the base measure, is the sufficient statisties, and the modulation parameters 0; == (,(c) depend on In our cluster- centric vMF conditional in (2), l = 1 (i.c., we can drop the j index) and 0,(c) = v,. This corresponds to (79) above, where Ca(ss) is the concentration- and dimension-ddpeedentt normalization constant.
25

---

Published as a conference paper at ICLR 2025
As our DGP assumes that the cluster vectors form affine generator system, and in the above Eq. (78) the cluster vectors take the role of (c)(c), we can prove that our DGP fulfils Assum. 6.
Lemma (The cluster-based DIET DGP is sufficiently variable), Assuming that the cluster-vectors form an affine generator system (Assums. IC), then the modulation parameter matrix L (defined in Assum. 6) formed by the cluster vectors V Va has full column rank.
Proof. First we need to show that the cluster vectors ur have the same role as 0,g (c). The derivative of the log-pdf of the vMF distribution in (79) w.r.t is the exponent in the DIET conditional (we can differentiate for non-normalized z, which is the case for auxiliary-bvarri -vvaribbe ICA).
[<<wes 2) + log Ca((t)] = K3C Oz
(80)
Then, we need d + cluster vectors to use one as a pivot to calculate as defined Assum. 6. By Lem. 1, this new set of vectors (i.e., offset by Va. expressed as v. va) also forms generator system of Rd which implies that L has rank d, concluding the proof.
To apply (Hyvarinen et al., 2019, Thm. 3) to recover the identifiability result of TCL, we need to show that our setting can solve the regression problem defined in Assum. 5. What we will show, w.l.o.g.. is that our regression function akin to (Hyvarinen et al., 2019, (11)) does not have an auriiilry-variable dependent constant.
Lemma (Regression function). The regression function in (Hyvarinen et al., 2019, Thm. 3), which solves the multiclass classification problem, consists of three items: 1) scalar product of vector- valued functions of either or c, and scalar-valued functions 2) z and 3) c. Our DGP and neural network pipeline used for learning can also match this regression function, by choosing pivot (zero) value for the c-dependent scalar function. This is without loss of generality.
Proof. In Thm. 1, the identifiability of the cluster vectors is up to an affine transformation, where the bias is denoted by 1b. Calculating the scalar product of the learned cluster vector with the learned latents yields two terms:
1. a scalar product term between - and c- dependent vectors; and
2. a xb h(z) term, which depends only on
Comparing to (Hyvarinen et al., 2019, Eq. (11)), we see that a c- dedepedennt scalar function missing. Following the common practice in multinomial regression, pivot value the e- dependent scalar function to be O, thus we do not need that term. This yields the following expression for the regression function:
can, w.l.o.g,, arbitrarily choose the
r(æ, c) = h(x)) w. h(x))
+ xb) (AVx)
hxx)) Av, + h(x)TT)
(81)
= h(x) Ave + a(x:)
(82)
where h is linear, so the first term depends on both æ, c, the second term a(x) only on æ, and we can choose (as usual practice in MLR), w.l.o.g., b(c) O, which concludes the proof.
B.3 THE RELATIONSHIP BETWEEN INFONCE AND DIET
Last, we show how DIET relates InfoNCE, where we reframe InfoNCE in form of instance disrimination. InfoNCE optimizes the cross entropy between the true conditional of the underlying DGP (a vMF distribution) and the approximate conditional parametrized by an encoder network. This cross entropy can be formulated as a loss for an N -lasas classification problem, where N is the dataset size:
B C= Σ CElg(a)),e)
s.t. gx(x)) exp (f(xt) f(xx)) E exp (f(+)) f(xx))
(83)
where e; is the 8th unit vector, encoding the class label in one-hot fashion, and क;' denotes the positive pairs. Note that the last part is simply the standard softmax (.) over the innner product (f(x)) f(x)). To go from InfoNCE to DIET, we need to make the following modifications:
1. Sum over the whole dataset N, not just the batch B.
26

---

Published as a conference paper at ICLR 2025
2. Replace the encoding of the anchor sample f f(xx) with learnable linear projection W, i.e. setting ((x,5) = a (W f(x)))
remaining difference to the original DIET formulation is that InfoNCE assumes unit-normalized features. However, our theory (ef. Thm. IC) accommodate unit-normalized vectors, so this is not a problem.
Let (æ., axt) be positive pair for sample n and let there be N samples total. The InfoNCE loss equivalent to multi-class N -apair loss of the form:
(f f(zx))
L = CE(g(a)), s.t. qa(x)) Eb=I... exp (f (x( f(x)))
(84)
Now instead of having mini-batches of size B, we take the loss over the whole dataset:
N
exp (f(x)) ((xe))
L Σ CF e;) s.t. (x(x)))
(85)
Eb=I..N exp (f f((x))
Next, replace f(xx) with a learnt and normalized weight vector wkk
N
exp (f wk)
L Σ CE(g(a)), e.) s.t. (x(xtt)
(86)
Eb=I...N exp (f (f(æ²) (wb))
Note that the last part is simply the standard softmax o()) over linear projection:
N
C CE(((()))))) s.t. ((x)) (W
(87)
where W is the projection matrix for which the 1tth row corresponds to wy. Since i in this case corresponds to the sample index in the dataset, we recovered DIET up to normalization, and so is simply the linear classifier.
C ADDITIONAL EXPERIMENTAL DETAILS
C.1 SYNTHETIC DATA
The code is based on https: https::/bbendeel group. wwtphh.i./cciiieee.
C.2 DISLIB
We evaluate our methods on the DisLib disentanglement benchmark (Locatello et al., 2019), which provides a controlled setting for testing disentanglement and latent variable recovery. We used the version of the DisLib (Locatello al., 2019) dataset based on the GitHub repository from (Roth al., 2022)3. It includes the vision datasets dSprites, Shapes 3D, MPI 3D, Cars 3D, and smallNORB. Using Pytorch, we train both a three-layer MLP with 512 latent dimensions and BatchNorm (which helped with trainability) and CNN (ResNet18) also with 512 latent dimensions (He et al., 2016). We only consider latent variables with Euclidean topology, as non-Euclidean, e.g., periodic latent variables such orientation, are problematic to learn and potentially mapped manifold (Higgins et al., 2018; Pfau et al., 2020; Keurti et al., 2023; Engels et al., 2024). We evaluate the recovery of latent variables by computing the Pearson correlation between ground-truth and predicted factors. Both models were trained for 100 epochs, with the Adam optimizer, a learning rate of 0.001 and batch size of 4096.
nonlinear
C.3 REAL DATA: IMAGENET-X
Finally, we test the generalizability of our theoretical insights on real-world data using ImageNet- X (Idrissi et al., 2022). The latent variables are proxies, defined by human annotators (Idrissi et al., 2022). They are binary labels, representing the deviation a certain latent variable on a given sample from the mode of that latent variable. We evaluate how well linear decoders can predict
https::/ http://gggttt..tt/// com/ ccom/facceookreeeerccooo
sentang1 ng- correl att-raaattdf--ctorr
27

---

Published as a conference paper at ICLR 2025
latent variables from pretrained model representations. We use two architectures, ResNet50 (latent dimension d = 2048) and a Vit-b-16 (latent dimension d' = 768) both trained on standard supervised classification using cross-entropy loss on the full ImageNet dataset (Deng al., 2009). Moreover, to get a baseline decoding performance from inputs (like in the DisLib experiments), we also fix a random linear projection from the full 224 224 3 150, 528 ImageNet input dimensionality down to 2048 the ResNet50 latent dimensionality. This is purely for computational reasons and can be justified based on the Lohnson-Lindenstrauus lemma".
We randomly split the data into 70% training and 30% testing data. For some latent variables, the label distribution was heavily imbalanced with less than 1% positive examples. To compensate class imbalance, for each latent variable, we resampled both training and testing data to achieve an distribution. We repeat this and all following analysis averaged over 10 random seeds. Using the LogisticRegret iin module from sklearn', we fit linear decoder predict the latent variable. Finally, we compute p-values based on one sample (-tests against a null hypothesis of chance level (50%) accuracy with multi-comparison Bonferroni adjusted significance level of 0.05 0.0006 (17 factors and 5 models). 17-5
D ADDITIONAL EXPERIMENTAL RESULTS
Ablating the choice of the cluster vectors. In Tab. 5, we present additional ablation studies exploring the effect of varying the distribution of the cluster vectors v. on the unit hyper-sphere. We do not observe any significant impact on the I22 scores of more concentrated cluster centroids v.--
Table 5: Identifiability in the synthetic setup. Mean a. standard deviation across 5 random seeds. Settings that match our theoretical assumptions are V. We report the R2 score for linear mappings. x + and w, - for cases with normalized (o) and unormalized (a) wy For unormalized w, we verify that mappings ž + z are orthogonal by reporting the mean absolute error between their singular values and those of an orthogonal transformation.
normalized w, cases
unnormalized w; R2 2 W.
R (t) W. - 0re
(A)
N
|
p(v.)
p(z(wo))
M.
Wi + U/
- U
10a
100
Uniform vMF( 10)
08.62a..1
0.01 99.9
99.0=a.00 0.00
0.000
99.9
10 5 100
Laplace vMF(x= 10)
V
98. 08.71om 0o
90.5
0.01 20
0.00 oo
99.
1 00
99.8
103 5 100
Normal vMF(x= 10)
/
98.210.01 on
99.220.01
0.01 20.00
0.00
99. 2 o
99.8
Quantifying the violation of the assumption the conditional with a generalized normal. Tabs. 2 and 3 show that using a Laplace conditional instead of a vMF or normal distribution leads to substantially lower R² scores, though one might argue that the Laplace distribution is not that different (according to some intuitive notion) from the vMF or normal distributions. To understand why using Laplace conditional leads to such a poor performance, we ran synthetic experiments with generalized normal conditional with scale (this is conceptually similar to our concentration parameter k) and shape 3:
z ~ p(z|C' x cellec==11 where ||
||
=
x
(88)
Importantly, 8 = 1 gives Laplace, whereas B = 2 gives a normal distribution. Thus, the generalized normal can be thought of "interpolating" between these two distributions, providing the perfect testbed to determine when performance starts to break down. We show the Re scores for both recovering z (Fig. 4 Left) and ue (Fig. 4 Middle) across multiple scale (ax) and shape (3) values, averaged over 5 seeds. We also report the average representation norm across multiple scale (a) and
http:///www.///mmmmmmmmnnnaaaaaa 5/modu wttpt//eeeeeeeeeeeeeeeeeeeeeeeee /at///kkeaammmm inear_modes
logisti html
28

---

Published as a conference paper at ICLR 2025
R2
R2
1 94.7
85.7
76.6
72.1
99.5
99.1
98.6
94.7
1.2
1.4
95
120
1.6
98.7
1.8
100
3=
80
1N2
85
85
1.6
99.3
98.7
97.8
96.6
1.6
99.7
99.4
98.9
97.4
1.8
99.5
99.6
99.2
98.6
1.8
99.7
99.4
98.9
97.5
40
2 99.6
99.7
99.6
89.4
99.8
99.5
99.0
89.3
50
20
10
20
30
40
Figure 4: Quantifying the assumption violation of a Laplace conditional: Tabs. 2 and 3 show that using a Laplace conditional leads to substantially lower R² scores. Numbers are averages across 5 seeds. (Left and Middle:) using generalized normal distribution to "interpolate" between normal (3 = 2) and a Laplace (3 = distribution for different scale values (denoted as a, which conceptually akin to our concentration k) and show the Rt score for recovering z (Left) and v. (Middle). (Right:) The average norm the representation for the one oie-dimesiional case for different values. As approaches 1, the average norm increases, indicating larger spread
shape (8) values (calculated in the one-dimensional case and averaged over 5 seeds) and the crucial effect of having fat tail (Fig. Right) with truncated Laplace distribution. Our results indicate that:
1. Ve is easier to recover than z: the numbers are higher in Fig. 4(Middle) than in Fig. 4(Left)
2. More concentrated conditionals degrade identifiability for all shapes: R² scores de- crease as increases in both Fig. 4(Left, Middle)
3. The average representation norm increases with increasing scale and decreasing shape: as the conditional approaches the Laplace distribution 8 + 1, the samples have a larger norm, i.e., they are further away from the unit hypersphere, potentially leading to insufficient overlap Fig. 4(Right)
4. Fat tails worsen identifiability performance: Allow more and more of the tail of a Laplace distribution to be included in the support (truncated symmetrically between -1, and -3, shows strong anti-correlation with the R² score for multiple scales (a)).
Loss saturation: the role of batch size and latent dimensionality. Our results in Tabs. and show that increasing latent dimensionality leads to substantially lower I82 scores-in line with findings in many prior works the identifiability of SSL methods (Zimmermann et al., 2021; ; von Kügelgen et al., 2021; Rusak et al., 2024). In general, the issue of extracting large dimensional representations from practical, real-world datasets is an open question (Simon et al., 2023; Jing et al., 2022). We investigate the interaction between batch size and concentration in Fig. ((Lef)).
With increasing concentrations, intra-cluster samples become more indistinguishable. This means that achieving close to optimum instance discrimination loss is easy with relatively coarse- cuase-grainn features. The intuition is that most SSL objectives (including DIET) saturate, i.e.. get close to optimum as the underlying pretext problem (in our cases the instance discrimination) is nearly solved a very good example is learning only the content features (but not the style ones) in von Kügglggnn et al. (2021). The result is population gradient with very low norm. As the empirical loss is calculated based on a finite batch size, the variance the gradient overtakes the norm, and the training effectively stalls. Additionally, higher concentrations result in less overlap between classes, which can have detrimental effect on source recovery. However, the signal-to-na to-seese ratio improves with a larger batch size, the increasing R² scores show in Fig. 5(Left) in the columns from left to right. The results
29

---

Published as a conference paper at ICLR 2025
82
$3.00
00.37
03.37
63.00
73
60
42 85
78.50
00.43
00.20
012 2006
2048
10 100 1910
1.3
2.
Truncation
Figure 5: The role of batch size, number of clusters, and fat tails for identifiability: (Left:) Increasing batch size improves Ri2 scores, counteracting the detrimental effect of more concentrated (higher к) conditionals; (Middle:) More clusters improve R scores, counteracting the detrimental effect higher dimensional representations (10 clusters for 10 and 20 dimensions violate Assum. 2; thus, the low Ro score); (Right:) The Laplace distribution leads to low R² scores due to its fat tails. Experiments with a truncated Laplace conditional (where the support is restricted to [- -Trunccation; Truncation)) shows that the closer the truncated Laplace distribution is to the Laplace distribution (i.e., with increasing Truncation), R² scores decrease for all tested scales a. Averages and error bars are reported across 5 seeds
suggest that this issue can, at least theoretically, be mitigated-nott that identifiability results hold in the infinite-sample regime, so requiring a larger batch size does not contradict our results.
When the latent dimensionality increases, the saturating loss implies that relatively low-dimensional features are sufficient to achieve this near-optimum loss. This phenomenon can also be seen in Fig. 5(Middle), where for each column (i.c., a fixed number of clusters), the R² score deteriorates with increasing latent dimensionality.
It is important to mention that none of these contradict our theory, which holds for the global optimizer of the DIET population loss. In practice, the additional challenges of estimation error (due finite sample size and finite batch size) and algorithmic error (using GD-based methods to solve likely non-convex problem) may impose adverse effects on the evaluation.
Diversity: the role of the number of clusters. To investigate the role of Assum. 2, we investigate how the number of clusters affects the R² score. Though the number of clusters is intrinsic to the dataset, thus, it cannot be chosen arbitrarily, knowing its effect on performance can inform practitioners about potential failure cases. We ablated the number of clusters for different latent dimensionalities (Fig. 5(Middle)). For a given dimensionality. the R score improves with more clusters, although there seems be a sweet spot where further increase in the number of clusters only marginally improves the I22 score. We also see that our requirement of d + +1 clusters (affine generator systems of Rd have at least this many members) essential for good performance. This is reflected in the extremely poor R² scores for {10, 20), when the number of clusters is only 10 (first column from the left in Fig. 5(Middl))).
Robustness to label noise. In this section, we evaluate the robustness of our method under in- creasing label noise. For DIET (Fig. 6 Left) we perturbed the instance label for each sample with probability equal to the label noise ratio (æ-axis in the figure). The perturbed labels were drawn uniformly from the set all instance labels. For the supervised case (Fig. 6 Right), the cluster label was perturbed instead. In both cases, the y-axis represents the identifiability score (R² score).
We believe this setup reflects realistic scenarios, where label noise is equally likely to affect any data point as commonly assumed in the literature (Nettleton al., 2010; Frénay & Verleysen, 2013).
Despite this increasingly challenging setup, the results demonstrate remarkable robustness to label noise. Up to an 80% label noise ratio, latent recovery shows minimal degradation, and the cluster recovery performance of DIET remains perfect-though both metric substantially decrease for larger (100%) ratio. We attribute this robustness to the symmetry of label noise across all instances and labels. While increased uncertainty does reduce the accuracy of individual label predictions, the
30

---

Published as a conference paper at ICLR 2025
W > U
- 2
2
100
100
80
60
60
R
40
40
20
20
0
0.5
0.5
Proportion of label noise
Proportion of label noise
The x z-axis shows the proportion of instances with perturbed labels; the y y-axis the R score of learning the ground truth latents (and the cluster vectors on the left). Perturbed labels are uniformly resampled from the whole instance label (left) or cluster label (right) sets, respectively. (Left:)) DIET perfectly recovers the cluster vectors V. up 80% label noise, and shows only small degradation for the latents (Right:) Supervised classification robustly recovers the latents up to 80% label noise with only a small degradation. Averages and error bars reported across seeds.
Figure 6: The robustness of DIET and supervised classification to label noise:
optimal logit values predicted by the encoder shouldn't 't change under symmetrical label noise. The stability of deep learning models label noise has also been shown by Rolnick (2017).
E
ACRONYMS
DGP data generating process
PID parametric instance discrimination
GCL Generalized Contrastive Learning
RV random variable
ICA Independent Component Analysis
SSL self-supervised learning
LVM latent variable model
TCL Time-Contrastive Learning
MAE Mean Absolute Error
vMF von Mises-Fisher
F NOMENCLATURE
R² coefficient of determination
S hypersphere
31


Reviews:
[{"review_id": "96S6bahFvh", "reviewer": "Reviewer_8oBq", "summary": "This paper provides a theoretical framework to further advance the linear identifiability of self supervised learning, and further to extend to general supervised classification tasks where cross-entropy loss is used.  The theoretical framework is constructed on a clever data generating process on the clusters of von-Mises-Fisher distribution.  Experimental results are included to empirically demonstrate the validity of the theoretical analysis. Empirical experiments on ImageNet-X dataset are also included.", "questions": "Reiterate the need of additional experimental results, please consider doing the following:\n\n1. After generating data in each cluster, randomly swap a percentage of samples between two clusters and report recovery statistics for all clusters.\n2. Increase the percentage of randomly swapped samples and observe the trend.\n3. Repeat this process with more clusters involved.\n\nThis would provide a more comprehensive evaluation by demonstrating both when the hypothesis holds and when it breaks down under violations of the clustering assumptions.  I'd like to see the experimental results discussed in the section above. In my view, it should be easy to conduct and will strengthen the paper. \n\nThe details of the experiments on DisLib and ImageNet-X should be included in appendix and/or a open github repo. DisLib (or at least a subset of it) has been widely used in many disengagement papers. The code or details explanation of the experiments should be provided for other researchers to compare, it is understandable that this experiment is not the main focus of this manuscript, including:\n\n1. Hyperparameters used\n2. Data preprocessing steps\n3. Evaluation metrics\n4. Any modifications made to standard implementations", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The framework poses significant progress towards better theoretical understanding on machine learning tasks, particularly classification tasks. If the theory holds, it may have a substantial influence on theories of adjacent research areas as well as many applications. Although this is a theoretical paper, the author(s) have made efforts to make the writing approachable, e.g., drawing from practical observations such as interpretable embeddings across input pairs from Mikolov 2013, and linear superposisiton in Elhage 2022.", "weaknesses": "My main concern is on how the author(s) conduct empirical validation. Table 2 and 3 appears to be the main empirical validation of the proposed hypothesis. The author(s) have devised a clever method for validation. However the statstics in Table 2 and 3 only shows that when the Assumptions 1 is true, the DIET method can successfully recover the cluster vectors with high probability. Another set of the experiments should be conducted to demonstrate that the hypothesis can be \"falsified\" when the assumptions are no longe true. This can be easily done with the current experiment setup:\n1. After the data in each cluster are generated, choose 2 clusters, and random swap a percentage of the samples between the two, report the statistics of the recovery of all clusters. If the hypothesis holds, the outcome should be clear\n2. Increase the percentage of the samples randomly swapped, the trend should be clear. \n3. Repeat 1 and 2, but with more clusters involved. The trend represented by the statistics should also be quite clear", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z8PBkKPIFl", "reviewer": "Reviewer_vcau", "summary": "- This paper extends identifiability results for self-supervised learning techniques to include classifiers trained with cross entropy, under certain Assumptions.\n- The Assumptions include a strong condition on the underlying data generating process that is unlikely to hold in practice: a clustering model with von Mises-Fisher samples around the cluster centers to yield the latent factors.\n- Under their Assumptions, they present a theorem claiming that such cluster-center-plus-vMF latents are identifiable up to a linear transformation using the DIET objective (which is cross entropy where the labels are training sample indices).\n- The paper validates the theoretical results on synthetic data where the latent factor is known, showing that in many of the settings covered by the theory, the R^2 score between the true and predicted latents are over 98%.\n- It further explores empirical applicability of the theory on the DisLib, a disentanglement benchmark with known factors of variation. While the data generating process does not match with the theoretical model in this setting, they still demonstrate generally strong Pearson correlations between the learned representations and the factors of variation.\n- A more challenging study using human-labeled factors of variation in the ImageNet-X dataset gives much weaker, but still positive, correlations between the factors and the learned representations.", "questions": "- What would be the impact of repeated elements in the training data when using the sample index classification head? In this case, the representation would try to map one input to two (or more) arbitrary indices. Would this impact the identifiability results in any way?\n- I’m not sure I understand the explanations for why a high concentration parameter is so detrimental to the identifiability. Could you give some additional intuition?\n- Relatedly, high concentration doesn’t seem to violate any of the listed Assumptions (and you don’t mark it with a red X in the tables) – is this poor performance an indication of a missing Assumption?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is clear and well-written.\n- The experiments are appropriate and demonstrate that the theoretical results at least occasionally apply beyond the setting of the fairly restrictive assumptions about the data generating process.", "weaknesses": "- Tables 2.3/Synthetic Data experiments: The identifiability suffers substantially as the dimensionality increases above 10, even if the perfect model is well-specified. This seems like potentially a major issue in practice, as the true latents of interest may often be very high-dimensional, and the model will often be misspecified. You acknowledge that this is an issue, but perhaps some additional discussion could be beneficial.\n- I’m not sure that the two synthetic misspecification examples are very convincing about the broad generality of the theory. Using a Laplace distribution in place of the vMF degrades performance substantially already, and I’m not sure it should be considered a particularly bad case of misspecification. Perhaps including experiments with mixtures of Gaussians, with varying numbers of mixture components would help understand how well the theory holds up over more challenging misspecification settings?\n- The results on ImageNet-X appear quite weak, which seems to indicate that violation of the Assumptions by the data generating process leads to substantial reduction in identifiability. Perhaps some additional discussion to help the reader understand why these results are weak for most of the factors of variation would help?\n- Line 71: Extraneous comma after the hyphen.\n- Line 457: “underlying training classification training task was solved” => “underlying training classification task was solved.” (Extra “training”, missing “.”)\n- Line 527: “Conlusion” => “Conclusion”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5S3GVdJO9r", "reviewer": "Reviewer_wFAY", "summary": "In this work they show that under some assumptions on the distribution of the ground truth latent variables and the \ndistribution of the labels then a model trained with cross-entropy is able to recover the ground truth latents up to \na linear transformation.", "questions": "Q1: In Table 2 with $N=10^3$, d=20, 100 classes, vMF($\\kappa = 10$) and $N=10^3$, d=5, 100 classes, vMF($\\kappa = 50$)\nsome of the $R^2$ values are much lower. You write that this is \n\"possibly explained by the content-style partitioning of latents (von Kügelgen et al., 2021) and insufficient \naugmentation overlap (Wang et al., 2022; Rusak et al., 2024).\" What do you mean by this? \nI am guessing that you would give the same explanation for table 3, d=5, 100 classes, vMF($\\kappa = 50$)?  \n\n\nQ2: In table 4, what is your interpretation of some of the pearson correlations being less than 0.8 and a few even \nless than 0.6?  \n\nQ3: In fig. 3, most factors of variation are captured with better than chance performance, but a few are not and many \nseem quite close to chance. Do you have an idea of why this might be?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1: The result that under Assumptions 1C Appendix A.2 we get linear identifiable representations is both new and interesting.", "weaknesses": "W1: The article makes too strong a claim that cross-entropy on its own gives linearly identifiable representations. \nThis claim is too strong, because additional assumptions (Assumptions 1C Appendix A.2) are needed to get the result. \nPlaces where this claim is made:\n- The title\n- The abstract: \"We prove that even in standard classification tasks, models learn representations of ground-truth factors of variation up to a linear transformation.\"\n- line 255-256: \"Namely, supervised learning performs non-linear ICA.\"\n- Line 304-305: \"providing a theoretical explanation for the success of supervised learning in learning linearly decodable representations.\" \n- Line 310-311: \"Solving an (almost) arbitrary classification task by optimizing the cross entropy objective is sufficient to invert the DGP and identify the ground-truth representation up to a linear transformation\" \n- Line 521-525: The \"Implications for Deep Learning\" paragraph. \n- Line 534-536: \"By linking self-supervised learning—via nonlinear ICA and DIET—to supervised classification, we provide a theoretical framework that explains why simple classification tasks recover interpretable and transferable representations.\"\n \n\nW2: The assumption made that \"instance label I is uniformly distributed\" (Assumptions 1C, (iii)) is important \nbecause many classification tasks have unbalanced labels. Therefore, this assumption should be made more clear. \n\n\nW3: In line 485, 518-519, you write: \"Our experimental results also suggest that our assumptions can be relaxed, \nas linear identifiability seems to hold even when some of the assumptions are violated\". However, there are also \nsome examples in tables 2 and 3 of $R^2$ being surprisingly low even though all the assumptions should be satisfied. \nThe authors should be more clear about where this discrepancy comes from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)