Statistical and structural identifiability in representation learning
Authors: Walter Nelson
Marco Fumero
Theofanis Karaletsos
Francesco Locatello
Venue: ICLR 2026 Poster (2026)
Keywords: identifiability
self-supervised learning
disentanglement
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We define a notion of near-identifiability, and show that a broad class of representation learning models have near-identifiable internal representations.

Abstract: Representation learning models exhibit a surprising stability in their internal representations. Whereas most prior work treats this stability as a single property, we formalize it as two distinct concepts: **statistical identifiability** (consistency of representations across runs) and 
**structural identifiability** (alignment of representations with some unobserved ground truth). Recognizing that perfect pointwise identifiability is generally unrealistic for modern representation learning models, we propose new model-agnostic definitions of statistical and structural near-identifiability of representations up to some error tolerance $\epsilon$. Leveraging these definitions, we prove a statistical $\epsilon$-**near-identifiability** result for the representations of models with
nonlinear decoders, generalizing existing identifiability theory beyond last-layer representations in e.g. generative pre-trained transformers (GPTs) to near-identifiability of the intermediate representations of a broad class of models including (masked) autoencoders (MAEs) and supervised learners. 
Although these weaker assumptions confer weaker identifiability, we show that independent components 
analysis (ICA) can resolve much of the remaining linear ambiguity for this class of models, and validate and measure our near-identifiability claims empirically. With additional assumptions on the 
data-generating process, statistical identifiability extends to structural identifiability, yielding a simple and 
practical recipe for disentanglement: ICA post-processing of latent representations. On synthetic 
benchmarks, this approach achieves state-of-the-art disentanglement using a vanilla autoencoder. 
With a foundation model-scale MAE for cell microscopy, it disentangles biological variation from technical batch 
effects, substantially improving downstream generalization.

Published as a conference paper at ICLR 2026
STATISTICAL AND STRUCTURAL IDENTIFIABILITY IN REPRESENTATION LEARNING
Walter Nelson', Marco Fumero', Theofanis Karaletsos & Francesco Locatello' Institute of Science and Technology Austria
Chan Zuckerberg Initiative
ABSTRACT
Representation learning models exhibit surprising stability in their internal rep- resentations. Whereas most prior work treats this stability as a single property, formalize two distinct concepts: statistical identifiability (consistency of representations across runs) and structural identifiability (aligmment of represen- tations with some unobserved ground truth). Recognizing that perfect pointwise identifiability is generally unrealistic for modern representation learning mod- els. we propose new model-agnostic definitions of statistical and structural near- identifiability of representations up some error tolerance c. Leveraging these definitions, we prove a statistical r-mear-identifiabliiy result for the representa- tions of models with nonlinear decoders, generalizing existing identifiability theory beyond last-layer representations e.g. generative pre-trained transformers (GPTs) to hear-identifiabilityy of the intermediate representations of a broad class of mod- els including (masked) autoencoders (MAEs) and supervised learners. Although these weaker assumptions confer weaker identifiability. show that independent components analysis (ICA) can resolve much of the remaining linear ambiguity bear-identifiabilityy claims
for this class of models, and validate and measure empirically. With additional assumptions on the data-generating process, statistical identifiability extends structural identifiability, yielding a simple and practical recipe for disentanglement: ICA post-processing of latent representations. On syn- thetic benchmarks, this approach achieves state-of-the of-thh-arrtt disentanglement using a vanilla autoencoder. With a foundation model-scale MAE for cell microscopy, it disentangles biological variation from technical batch effects, substantially improv- ing downstream generalization.
INTRODUCTION
Despite the massive variety of data modalities, pretext tasks, training procedures, and datasets, disparate self-supervised learning models whole seem be converging on shared set of representations of the natural world (Huh et al., 2024) which are useful for a surprising variety of
downstream tasks (Kraus et al. 2024 Hayes et al. 2025 Baevski al.) 2020; Brohan et al. (023). A classical lens for studying this phenomenon is the notion of identifiability (Reizinger et al. 2025a). In likelihood- statistical inference, identifiability the condition that data are sufficient to completely characterize the parameters of the model (Casella & Berger 2001). The situation is considerably trickier for neural network models: the parameter space large and invariant to e.g. permutations of the neurons, and the training procedures might lack likelihood-baste Instead, recent work in identifiability focuses on finding conditions such that infinite data is sufficient to characterize the trained model's representations of that data (Reizinger et al. 2025a) up some equivalence class such as linear transformation (Roeder et al. 2021).
interpretation.
We begin by sharpening existing definitions of representation identifiability, recognizing that existing results fall into two categories. The first is what we refer to as statistical identifiability, or the condition that optimizing given representation learning model will yield the same representations up to some simple transformation. The second is what we refer to as structural identifiability, or the condition that optimizing a given representation learning model will yield a particular representation every time, corresponding some latent component of the data-generating process. We provide

---

Published as a conference paper at ICLR 2026
definitions of statistical and structural identifiability that relax the requirement that the representations are exactly identifiable, making them the first general-purpose formulations which are applicable to the case where the representations are "nearly" identifiable up to error tolerance and extending prior model-specific cases (Nielsen et al. 2025 Buchholz & Schölkopt 2024).
Leveraging these definitions, we prove several new identifiability results. Our first result shows that for models which have statistically identifiable outputs, such as generative pre-trained transformers, supervised classifiers, and encoder-decoder models (Roeder et al., (02I), the intermediate-layer representations are also statistically e-nearly identifiable up to a rigid transformation. Unlike several recent results, these representations can be mapped non-linearly by mild function class condition on the mapping from the intermediate layer to the identifiable outputs. Our second result shows that linear independent components analysis (ICA) can resolve this rigid indeterminacy, yielding near-idende cra-ideetifiabilityy up to signed permutations. Notably, our sharper definitions of statistical and structural identifiability reveal that these results are available without strong assumptions on the data- geteraene process, instead requiring only this mild function class assumption on the model. Our final result shows that one is willing to make a similar assumption on the data-genetn generating process, encoder-decoder models which are statistically identifiable and achieve perfect reconstruction are also structurally identifiable. Notably, perfect reconstruction is another assumption which can be relaxed if statistical identifiability is all that is required.
the loss, and e is governed
In addition to our theoretical contributions, we perform a series of experiments to validate our claims. In synthetic experiments on autoencoders, we show that hyperparameter selection and regularization impacts the statistical identifiability nearness in ways that are predicted by our theory. Subsequent experiments show that near-identifiaiiiiyy also holds in off-the-shelf pre-trained models, and that the linear indeterminacies predicted by our theory can in practice be resolved by ICA. Next, we investigate whether our structural identifiability result can be applied to the special case of disentanglement (Locatello et al.) 2020), finding that the simple combination of vanilla autoencoders and linear ICA applied to the latent space yields disentanglement on several benchmark datasets, competitive with some of the best existing models. Finally, we show that linear ICA applied to the latent space of a masked autoencoder for cell imaging successfully disentangles batch effects from biological variation, a key problem in the application of machine learning to biology.
RELATED WORK
Statistical identifiability of representations Prior representation identifiability results make strong assumptions on the data-generating process (Zimmermann et al., 2021 Reizinger et al.) 2025b [Cen et al. 2024| Lachapelle et al. (2033) or assume a linear relationship Khemakhem et al., 20hen
between the representations and the loss (Roeder et al. [202]] Marconato et al. 2025 Nielsen et al., 2025), and generally do not distinguish between statistical and structural identifiability as we do here. Our work directly addresses this gap by proposing a concrete definition of statistical c-near- identifiability which is provably met by the general-purpose models in widespread use today, with only mild assumptions on the model class and few assumptions the data- getteaeeneatiigg process. We directly measure the consequences of our statistical identifiabilit results by assessing the & convergence of representations in real-world models, extending prior work on representation similarity (Roeder et al. 2021: Huh et al. 2024,] Klabunde et al. [0255] Nielsen et al.
2025 Marconato et al. 2025).
Nielsen et al. (2025) relaxes the identifiability theory of Roeder et al. (2021) for generative pre-traines
transformers, showing that the Kullback-L Leibler divergence on the next-token distribution fails to
serve as a witness for differences in the penultimate-layer representations, and proves a sufficient condition for divergences that do, which could be viewed particular form of near-identifiabiiiy for a particular model class. Reizinger et al. (0024) formalizes e-non-identifiabiiity with respect to the KL divergence, showing that this failure generalizes beyond representational identifiability to other properties of interest for large language models. For a history of the term identifiability, see Appendix A.I.
Structural identifiability of representations Ours is also the first work to make clear the dis- tinction between such model-specific identifiability results and structural identifiability results such as disentanglement. We formalize assumption the data- mata-generating process that allows to extend our key statistical identifiability result to structural identifiability of encoder-decodee models, and define and characterize rich class of data-generating processes which meet this assumption.
2

---

Published as a conference paper at ICLR 2026
This result is similar to other works that aim to "invert the data- data-generating process" Zimmermann
et al. 2021; Reizinger et al. 2025b; Von Kügelgen et al., 2021]. including work on the isometry
assumption (Horan al. [021]] some of which uses a different, average- notion of near- near-isometry
to claim near-recovery of the true latents in a nonlinear ICA model (Buchholz & Schölkopi 2024). These prior works are far from real-world practice, with only Reizinger et al. (2025b) presenting any results on a real model with real data, showing that linear concept decoding is possible via parametric instance discrimination in ImageNet-X, but lacking a clear disentanglement result. In stark contrast, we illustrate practical application of our theory by showing that we can improve biological foundation model for cell microscopy via disentanglement.
generalization in Our work also differs from prior work on causal representation learning (see Yao et al. (2025) in that our structural identifiability result is completely unsupervised, relying on inductive biases rather than supervision in the form of interventions.
STABILITY THEORY
We begin by providing novel theory of the stability of neural representations, with an eye toward self-supervised models. All theorems are presented informally with important constants in the main text, with full theorem statements, lemmata, proofs, and model-specific treatments in Appendix A.3
3.1 STATISTICAL NEAR-IDENTIHABILITY
We consider a data distribution supported on an arbitrary space X. A representation learning model can be fully characterized by its parameter space O, its loss function, and the deterministic mapping from parameters to representation functions. Concretely, for each E O, let Ca : X + R denote the corresponding loss function, and define the model as M (Lg 8 E θ). If some component of 0 parameterizes a representation function fo X - RD, then identifiability theory in machine learning aims to characterize the properties of so after training by minimizing [=())(z))] yields set of parameters 0.
Definition 1. Let P(x) denote some data distribution supported on X. Consider machine learning model M = =Lo A E θ}, and let F 8 + fo be some deterministic transform of the model parameters yielding representation function fa : X + B D. Let S O be the set of minimizers of (u)) For some group H of functions from ReD to itself, we say that (P, 9, La, F)) is
statistically e-nearly identifiable in the limit up to H if every 0 O' E S. we have Il ho fo I e for some E H.
Remark. For the remainder of this paper, we will use the the 1 norm (essential supre- mum with respect to P) for functions taking values in the Euclidean space IR" endowed with the & norm. When e 0, we drop the e-nearly and refer to the triple simply as statistically identifiable the limit up to H. When the identifiability is pointwise, we refer to the model as statistically identifiable in the limit.
This generalizes prior definitions of representation identifiability by the introduction of the "slack" term e. Definition ssyy that model's representations as given by independent retrainings fe and fer are near-identifiable if they are the same up to a simple transformation group H (c.g. rotations) and small amount of distortion e. this way, we also generalize the classical definition identifiability from mathematical statistics (Casella & Berger 2001). see Appendix A.1 for a history.
Intuition.
In this paper, we will mainly deal with near- idnntifibbbii of latent representations up to the function classes Minears Higia and H., Hinear is the group of invertible linear transformations on R2, while Hrigia is the class of rigid linear transformations on DRP, which consists of compositions of rotations, reflections and translations (Hrigid C Hinear). practice, translations can be ignored by assuming e.g. zero mean of the representation distribution. Similarly, reflections only flip signs, and can usually also be ignored. It is therefore useful to imagine the main indeterminacy of Hrigid as being a special orthogonal matrix in SO (D), i.e. a rotation. H. is the class of signed permutations of IR. D, which generally not resolvable because there is reasonable way to specify an ordering to the latent variables or signs to individual latent variables (should the x coordinate of an object in a scene be represented left-to-right, or right-to-left?).
3

---

Published as a conference paper at ICLR 2026
Connection with prior results Existing representation identifiability results can easily be recast in our framework. For example, contrastive learning models using the InfoNCE loss with augmentation distributions satisfying particular isotropy condition in latent space are identifiable up to Hrigid (Zimeerman et al., 2021). This isotropy condition is impossible to validate in practice without access the ground-truth data-generating factors, reflecting the fact that it is strong assumption on the true data-genenne generating process.
Another result is due Roeder et al. (2021), which applies models whose losses take the following form and can be interpreted as exponential family negative log-likelihoods:
Lo(x, v) = - (2o + Ao(x) = loog go xx)
where go is the approximating distribution, to is sufficient statisties function, to0 is the natural parameter function, and Ag is the log partition function. As an example, to might map categorical labels to their corresponding vectors in final linear weights matrix (covering the case of supervised multi-class classification, including next-token prediction). Te maps inputs to their representations, which are shown to be identifiable up Hiinaar when pointwise equality of the losses attained. In a short proof in Appendix A.3.I we show that simple sufficient richness condition on the approximating class extends this result to our definition of identifiable in the limit. Put together, this result means that "perfect optimization" on infinite data will yield a supervised or GPTT-class model with the same penultimate-layer representations every time, up to some unknown linear transformation. An extension of this result provides for the same kind of identifiability on linear subspace of the representation space, allowing for the case of models with representations of different ambient dimension (Marconato al. 2025). NNelssn et al. (2025) provides for further generalization, deriving a notion of distance on the space of modeled likelihoods Q = {go(v x) 0 E Q such that closeness in distribution implies closeness in the penultimate-layer representations given by no. form of near-identifaabiiity (see also Appendix A...II.
Even these GPT results are limited because they only treat these penultimate- layeer representations given by n, which are mapped linearly to the loss. For many models, we're 'r interested in repre- sentations from earlier layers which are mapped the loss nonlinearly, such with nonlinear decoder or head. Our key result, captured in the following theorem, provides idettfiiiiff up to rigid transformations in such cases. The level of nearness e governed by the degree of local bi-L bi-Lipschitzness of this nonlinear mapping. This result allows us to treat earlier-layer representations in GPT or supervised classification models, for example, or the latent representations of (masked) autoencoders.
Theorem 1. (Informal) Let P(x) be a data distribution, and let M be a model with a parameter space ④ and loss function La. Let F + fe. G + ge and H : 0 + ge fe. Then, if (P,0, La, H) is statistically identifiable in the limit, then (P, O, La, F)) is statistically e-nearly identifiable in the limit up to Hrigid e = + where 1 + L is local bi bi-sihhchitz constant bound for go, and en and A are constants independent of the model (and L.).
Intuition. Here, we give the first general-purgee identifiability result for the internal representations (i.e arbitrary-layer) of a broad class of models, including (masked) autoencoders, next-token predictors, and supervised learners. H parameterizes the end-to-end neural network go so. which is assumed to have identifiable outputs: for example, when the loss is the mean squared error, the neural network learns the optimal function, namely the conditional mean. The identifiability of the internal representations given by fe (the "encoder") is then governed by the local bi-Lipschitzness of the function go (the "decoder") mapping them to these identified outputs. The bi-Lipschitz constraint controls the degree to which go deforms distances. Intuitively, a bound on the local bi- bi-Lipschitz constant is small when small changes in the latent variables result in small changes in the outputs of the network. The proof is given in Appendix A.3.2 and we provide concrete examples for number of architectures, including masked autoencoders, supervised learners, and GPTss in Appendices A.3.3 and A.3.4
This result is the most general we are aware of for quantifying representation identifiability. The local bi-Lipschitz condition is difficult to test empirically, but prior work has shown that many popular regularization techniques push neural networks toward a state of "dynamical isometry", " which can be viewed as bi- bi-Lipschitz condition (Xiao et al. 2018 Bachlechner et al. 2020 Miyato et al. 2018 Karras et al., 2200)) Zhang et al. 2019)) due to the singular lar values of the Jacobian concentrating near one. We derive the precise relationship in Appendix A.4
4

---

Published as a conference paper at ICLR 2026
3.2 RESOLVING LINEAR INDETERMINACIES WITH ICA
Later, we will illustrate applications of the near-identifiabiity result Theorem T to both vanilla autoencoders and masked autoencoders. To do this, we will find it useful to resolve the remaining linear indeterminacy in the latent space posed by Himea or Heigid- We propose to do this by applying independent components analysis to the latent representations. We do not provide any novel ICA identifiability results in this work. Rather, we show that our conception of c-nearness in identifiability poses no further complications for the downstream application of ICA.
Theorem 2. (Informal) Suppose (P, P, Lo, F) statistically c-nearly identifiable up to Hiinear for F : 0 + fe. Then, for new model with parameter space O' and loss Lo which applies whitening and contrast function-based independent components analysis to the latent representations given by fg. and F" + S% which yields the transformed representations, (P, 'P,0', ,', F*) is statistically é' enear-identifaabee up to Ha for e' = Ke + K'è, where K and K' are constants free of e that depend on the spectrum of the covariance matrix of the representations and the properties of the ICA contrast function.
Intuition. Consider some representations in a model which are lincarly identifiable, such as the penultimate layer of GPT-class model or the latent tokens of masked autoencoder covered by Theorem T Whitening reduces the linear indeterminacy to a rigid one, while ICA (if sufficiently well- converged) resolves the final rigid indeterminacy to a signed permutation, with nearness preserved (up to new constants) along each step.
3.3 FROM STATISTICAL TO STRUCTURAL IDENTIFIABILITY
While statistical identifiability on its own may be a useful property for reliability and analysis, there has been recent interest in structural identifiability of representations, the ability of the model to recover some latent component of the data generating process which is useful for some downstream tasks. Below, we formalize the distinction between the two.
Definition Let Puu) denote a distribution over some unobservable parameters with support u CR. Let P(x u) denote some conditional distribution such that u(x) arg uet P( u) is well-defined almost everywhere with respect to P(r), where P(z) = P(x P) du is the marginal distribution of the data with support X. Consider machine learning model M [Lo :
}}, with solutions S C θ to the minimization of EE>P(())Loo (fo))\((())))))))))))))))))))) (3)}. Let F + S' be a deterministic transform of the parameters yielding representation function Sa X - TR. For some group of functions H. from RD to itself, we say that (P, ,, θ Lo, F) e-nearly identifies the structure u up to H. for all 0 E S, we have that fo satisfies [|h o fo ull S e for some h E H.
Intuition. Statistical identifiability is in some sense weaker than structural identifiability. Statistical identifiability is the condition that the representations consistent, while structural identifiability is the condition that the representations are consistently "correct". In order to define utility. or correctness, we need assume the existence of some latent component of the data-generating process that we're aiming to recover, u. The well-studied setting of disentanglement (Locatello et al. 2019) represents a special case where P(u) assumed have independent components. As an example, in Section 4.4 we consider the situation where P(u)) is distribution over natural latent biological factors along with independent technical variates, and x is generated via smooth function of with smooth inverse, leaving u(x) well-defined. In Appendix A.5 we provide simple proof that structural identifiability implies structural identifiability, including in the e-near case provided that H is bounded in the sense of an operator norm.
Finally, we make precise the assumptions on the data gata-geenrating process necessary to extend our statistical identifiability result in Theorem to structural identifiability. We show that bi-Lipschitz data-generating processes are structurally identified by reconstructing encoder-decoder models which nearly identifiable up to Hagia in the sense of Theorem I and up H. when combined with ICA as in Theorem 2 This is partly a generalization of an earlier result by Horan et al. (2021), which covers the perfectly identifiable case.
Theorem 3. (Informal) Let P(u) be some multivariate non-Gann oon-Gaussian distribution with independent components and consider data P(x) generated by pushforward through a smooth diffeomorphism such that g is (1 + -pssii--pss Let M be a model with a sufficiently rich parameter space θ.
5

---

Published as a conference paper at ICLR 2026
Let F G f go and H 8 > go Then, (P. .. Co. H) structurally identifies the identity function in the limit (i.e. attains perfect reconstruction), we have that (P θ Co. F) e-nearly identifies the structure g01 up Hrigid- and furthermore that new model M' which applies whitening and independent components analysis to the latent representations given by so e' e'-nearly identifies the structure 9 up to Ho where and é' depend on ó and Lipschitz bounds on go. and depends additionally on the spectrum of the covariance matrix of the representations and the properties of the ICA contrast function employed.
Intuition. Structural identifiability is stronger than identifiabiliyy, we require additional as- sumptions on the data-generating process to achieve it. In particular, here we assume that the gatt-gener function mapping "true" latents to observables is locally bi-Lipschitz, which com- bined with independence and non-Gaussianity is sufficient to nearly recover the true latents via ICA for any nearly identifiable reconstructing model with a locally bi-l.ipschitz decoder. The proof given in Appendix A.3.6
3.3.1 BI-LIPSCHITZ DATA MATA-GENERATINGG PROCESSES
Naturally, it's useful to characterize what kinds of data-generating processes might be covered by Theorem Several interesting image data-generating processes are known to approximately satisfy a Euclidean isometry condition (which is equivalent to a local 1-bi-Lipschitz constraint for smooth mappings) such as smooth articulations of cartoon faces [Tenenbaum et al.) 2000; Horan al.) 2021). Furthermore, the success of regularization techniques similar to isometry constraints in diverse classes of neural network models real- wolld settings suggests it is a useful inductive bias in practice as well (Karras et al., (2000) Lee et al.) (022). In the rest of this section, we aim to better characterize what these assumptions mean. Specifically, we give some examples nearly isometric data-generating processes inspired by the popular dSpri tes dataset (Matthey et al. (017)) and show that disentanglement in this setting implies the structural identification of the true aata-generating factors, using technique developed by Grimes (2003).
Example Consider continuous relaxation of images, where square black- and-white image is represented by an L function & [-1, x [-1,1] + {0, 13. with (xz 3) giving the value of the (x, y)th "pixel". As an example, the image of a white square with radius 0 < r < in the centre of the "frame" given by (z, y) I[[ S r. yy S r] where I the indicator function. One can first imagine manifold of such images where the centre E [a, b] of the square is moved from left to right. We write this as a continuum of images produced by the smooth function f [a, b] + L where +
and 1- b 2 r to ensure that the square does not leave the frame. In this case, each "image" on the continuum f {p) is the function (I, y) + I[[x- I r, lyl r.. where p is the square's s x coordinate.
Figure 1: A sim- ple isometric data- generating process.
We'll show that f is local isometry, meaning that it preserves notion of distance perfectly. This is equivalent to 1-bi-Lipschitz constraint. To see this, consider the Gateaux derivative of f. which is just the limiting behaviour of articulating the square units the right:
f(p + c) f(p)
f'(p) = lim c00
E
lim e00
<e<<xpprr+ar -ptr<a<ptrtet..
<T) -lp---l-p-------- - <
"gained" white pixels
"lust" white pixels
where the indicators in the limit represent the pixels that change when articulating the square units to the right. Intuitively, shifting the square e units to the right only changes er pixels to the right (flips them from black to white) and pixels to the left (flips them from white to black) of the original square. The situation is drawn in Figure I where the white square has a gray border for clarity and the red shaded areas show the white pixels which gained and lost by shifting the square units to the right. The isometry condition considers the situation as e - 0 (intuitively, as the width the red rectangles shrinks to zero).

---

Published as a conference paper at ICLR 2026
By argument made formal in Appendix A.6
can rely something like preservation of the L norm under limits to have [ = 2r, which constant when r is fixed. In the univariate case, this is sufficient for f to locally preserve notion of distance. In particular, for any two values of the latent Po and pr we have
[ (1F' 2r|p1 Pol
P1
Pol
where the integral is the usual geodesic distance along the manifold of "images". Due to the constant typically
2r, some literature refers to these scaled isometries (Lee et al. 2022). In practice, ignore the scaling constant.
While sprites datasets are known to be overly simplistic, this analysis provides some intuition that interesting real-world image manifolds might usefully be approximated by isometries, or functions that are nearly isometries. For example, similar analysis defer to Appendix A illuminating for the multivariate case where both the radius r of the square and its z-coordinate p are varied. In this setting, have that the Gateaux derivative is non-constant in (specifically, the map is conformal). However, if we assume a compact support for the latents, i.e. that [o, b] and E (0, RJ. the data-gen geneeatnngg process is additionally B-bi-Liipchitz with the constant B dependent on a and b, and Theorem applies.
EXPERIMENTS
We perform four of experiments: direct validation of Theorem on MNIST using vanilla autoencoders (Section ).1). direct validation of Theorems T and in off-the-shelf pretrained self- supervised learning models (Section 3 an application of Theorem 3 classic disentanglement problem in several synthetic datasets (Section 4.3). and real-world application to deconfounding for out-of-distriuution generalization in a real-world foundation model for cell microscopy in biology
(Section 4.4)
4.1
WARMUP: CONTROLLING IDENTIFIABILITY
We begin with experiments in a regime where the local bi- Lipschitz constant can be controlled directly possible, and examine whether our theory correctly predicts the level of sraaiidetiiiibiiiity. We consider conneccd autoen- coders with 3-layer encoders and decoders, and orthogonal linear layers with LeakyReL.U activations (with leak parameter E [o, 1).. The local bi-s.ipschitz constant of the decoders is therefore bounded by 1/a where K
0.8
emrr
0.35
g
0.05
0.2
the number of layers in the decoder. For a = 1, the network is linear, while for a 0, it's ReLU network. We fit pairs of autoencoders with different initializations and seeds to MNIST (LeCun et al.) 2010), and assess the relationship between reconstruction error. empirical near-identifiability. and empirical measurements of the local bi-Lipschitz constant, which manipulated by vary- ing between 0 and 1. According to the proportionality in Theorem I we estimate how well measured by the
0.0 mean(1] (bi-Lipschitz constant)
0.3
Figure 2: Controlling the bi- Lipschitz constant leads to im- proved identifiability (reduced & error).
the empirically estimated bi-Lipschitz term vL + L predicts identifiability, average & error from the optimal rigid transformation between the pair of latent spaces. Results are summarized in Figure 2 Full experimental details available in Appendix A.7.1
4.2 MEASURING IDENTIHIABLIITY OF PRE-TRAINED MODELS
Our next aim is to validate the statistical near-identifiabiiiy up to linear (for GPT-class models, Theorem of Roeder al. (202T) and rigid (for autoencoder-clas. and supervised models, Theorem D transformations predicted by theory, and the ability of ICA to resolve the remaining linear Indeterminacy (Theorem 2)
Matching our theory, we examine pairs of models that have the same architecture, loss, and are trained the same dataset independently. Rigid similarities (rigid transforms with scaling constant
7

---

Published as a conference paper at ICLR 2026
to allow for varying regularization across model pairs), linear transformations and ICA transforms estimated between representation spaces. We measure near- seariiddniifiaiiiity with the average lz error in the self-supervised model's representation space, along with the efficiency of the ICA transform as the percentage reduction of &a error relative to the rigid transform (since the degrees of freedom are roughly the same). Results are shown in Table [
GPT-class models exhibit ex- cellent linear alignment, as predicted by the theory of
Supervised Rigid
Model Pair
Permutation
ICA (% eft.)
Linear
Roeder et al. (202T). As pre- dicted by our theory, MAEs exhibit rigid alignment up to similar level of error, notably including one example across model sizes. In all cases, ICA mitigates a substantial portion of the indeterminacy due to the linear variation, notably without any supervision. In particular, for MAE models, ICA is nearly 60% as efficient as computing the optimal rigid transform between the two models in a fully-supervised fashion. Full experimental details are available in Appendix A.7.2
Pythia- 160M-0 > Pythia-16000 160M-1
0.219
0,150
0.131
0.202 (25%)
MAE-t mm - MrF-origiall
0.197
0.109
0.036
0.145 (59%)
CheXpert-smull + Che CheXpert-base
0.218
0.104
0.048
0.175 (38%)
ResNet-18-fe--1 ResNet- 18-fc-2
0.382
0.206
0.175
0.312 (40%)
Table 1: Supervised and unsupervised alignment scores between pairs of models which measure empirical identifiability. The optimal transforms (permutation, rigid, linear, or ICA) are estimated between the two models, and average lg errors normalized by latent diameter are reported. For ICA, we also report its efficiency as the reduction of Eg error from the permutation to the rigid transform.
4.3 DISENTANGLEMENT USING VANILLA AUTOENCODERS
Next, we assess whether our theory correctly predicts structural identifiability of the ground-truth data-generating factors in synthetic datasets matching the assumptions of Theorem We examine vanilla autoencoders, an appealing architecture due to their simplicity. In such simple models, weight decay is known be sufficient regularize the Lipschitz constant of the decoder, thus making it good testbed for our theory (Zhang et al. 2019).
We use a well-established experimental testbed for assessing unsupervised disentanglement, specifi- cally following the exact experimental protocol from Hsu et al. (2023). As baselines, we include a 3-VAE (Higgins et al., 2017), f-toota correlation VAE (Chen et al.) 2018), and BioAE (Whittington et al.] 2023), all of which leverage specialized regularization to achieve disentanglement. For compar- ison, we follow the supervised model selection strategy of Hsu et al. (023)) which shows best-case performance (Locatello et al. 2020).
For each dataset, we train a vanilla autoencoder with the only hyperparameter we vary being weight decay. Then, ICA is applied to its latent space. Models are evaluated on four datasets. Shapes3D is a toyish sprites dataset (Hsu et al.) 2023] Burgess & Kim 2018). Falcor3D and Isaac3D consist of rendered images of a living room and kitchen, respectively (Ne et al. 2020). MPI3D consists of real images of real-world robotics setup (Ciondal et al. 2019). Disentang eement of the learned latents is evaluated according InfoMEC (Hsu et al. 2023) which consists of three complementary metrics: modularity (the degree to which each learned latent encodes only one true source), explicitness (the degree to which the latents capture all information about source), and less important, compactness (the degree to which each source is encoded in only one latent). InfoMEC aims to resolve many of
Table 2: Disentanglement metrics (InfoM, InfoE, InfoC), of which InfoM and InfoE are the most
important. AB + ICA performs comparably to some of the best disentanglement-speciiii neural networks, with almost no tuning. Results marked with (*) are quoted without reproduction from Hsu et al. (2023). Bolded metrics have the highest point estimates. Full details in Appendix A.7.3
model
aggregated
Shapes3D
MP13D InfoF:
F
Isaac3D
(InfoM
InfoC)
AE
(0.39 0.76 0.25)
(0.34 0.99 0.16)
(0.42 0.40 0.31)
(0.37 0.83 0.20)
(0.41 0.80 0.34)
J-VAE*
(0.59 0.81 0.55)
(0.59
0.99 0.49)
(0.45 0,71 0.51)
(0.71
0.73 0.70)
(0.60 0.80 0.51)
3-TCVAB
(0.58 0.72 0.59)
(0.61 0.82 0.62)
(0.51 0.60 0.57)
(0.66 0.74 0.71)
(0.54 0.70 0.46)
BioAE*
(0.54 0.75 0.36)
(0.56 0.98 0.44)
(0.45 0.66 0.36)
(0.54 0.73 0.31)
(0.63 0.65 0.33)
AE + ICA (ours)
(0.65 0.83 0.40)
(0.79 0.99 0.52)
(0.44 0.66 0.31)
(0.71 0.83 0.33)
(0.64 0,82 0.43)
8

---

Published as a conference paper at ICLR 2026
the issues with the DCI framework of disentanglement evaluations, including removing the need to select hyperparameters which can affect results (Hsu et al. 2023).
Vanilla autoencoders with ICA in latent space outperform specialized disentanglement models most of the time (Table and on average perform better than all. Experimental runtime roughly 6 hours per autoencoder hyperparameter setting a single GPU (roughly 720 GPU hours total).
4.4 DECONFOUNDING AT FOUNDATION MODEL-SCALE
treh-throughput screens have become a critical tool in modern biology, particularly for drug discovery (Chandrasekaran et al. 2023). One example of such screenings is cell painting (Chandrasckaram et al. 2023] Sypetkowski et al. 2023). where a perturbation is applied (or not) to collection of cells, which are then stained and imaged. A key challenge in the application of machine learning these
data is the presence of complex technical variation ("batch effects") that is not biologically significant
(Arevalo al. 2024. Chandrasekaran
2023 Lin Lu 2022 Sypetkowski al. 2023 Ando et 201 For example, data collected from different microscopes, labs, or even just in different
experiments can exhibit variation that not of interest the practitioner, potentially confounding results. A fundamental task in this setting is to quantify the degree to which a perturbation has significant effect, which is challenging given that we have access only nigh-dimensional, noisy observations in the form of images confounded by batch effects (Bereket & Karaletsos) 2023). In particular, we almost never have the ability to measure sources of batch variation, and so it is of specific interest to be able to disentangle technical from biological variation without supervision.
We explore an application Theorem 3 by applying independent component analysis to the latent space of OpenPhenom (Kraus et al.) 2024], large, open masked autoencoder trained on Rxix3-core, large library of cell painting images (raus et al. (025). Downstream, we consider the task where the inferred embeddings "control" vs. "perturbed"). Because we are primarily interested whether the procedure enhances out-of-distribution generalization, consider downstream classifiers trained on a subset of batches and evaluated on held-out subset of batches.
used prediet whether given perturbation has been applied (i.e. classify
Inference & evaluation We perform inference on all images from Rxix3-core and estimate the whitening and independent components analysis models at the patch level by randomly subsampling a patch from each image. We consider four conditions: the original embedding (Base), the embedding whitened with principal components analysis (PCA), the whitened embedding rotated using ICA (PCA + ICA), and as baseline, the whitened embedding rotated randomly (PCA + Rand). Plates are used the batch indicator, while single patch-level embedding is used for each image. For each embedding condition (raw, whitened, whitened + ICA, whitened + random rotation), we hold out 20% of plates and train a gradient boosting classifier (Ke et al., 2017) on the remaining 80% of plates in k-fold cross cross-valides validatii scheme. Because some plates do not contain any perturbed samples (i.e. are entirely controls), we ensure that this split is roughly stratified on the label ("perturbed" vs. "control"). Perturbation classification is evaluated by the area under the receiver operator characteristic curve (AUROC). Embedding inference took approximately hour on a single GPU, while ICA estimation took about hour 128 CPU cores with 128 GB RAM.
Table 3: Results from a downstream perturbation classification task. Each row represents gene consisting of (#) separate experiments with different CRISPR guides. All experiments use the same set of 22,062 controls. Base untransformed embeddings, PCA = whitened embeddings, PCA + ICA whitened embeddings with ICA rotation applied, PCA + Rand whitened embeddings with random rotation applied.
Mean AUROC (t)
Sparsity (1 more sparse)
Gene
Base
PCA
PCA + ICA
PCA + Rand
Base
PCA
PCA ICA
PCA + Rand
CYP1IBI (1)
0.663
0.692
0.709
0.678
0.184
0.204
0.237
0.188
BIF3H (1)
0.682
0.724
0.749
0.725
0.192
0.224
0.268
0.214
HCK (1)
0.670
0.693
0.711
0,668
0.156
0.208
0.241
0.166
MTOR (6)
0.663
0.690
0.705
0.679
0.166
0.201
0.233
0.186
PLK1 (6)
0.803
0.811
0.815
0.792
0.251
0.307
0.305
0.262
SRC (1)
0.660
0.694
0.706
0.676
0.170
0.214
0.240
0.184

---

Published as a conference paper at ICLR 2026
Results Whitening alone often enhances the performance of downstream classification from the embeddings. The application of ICA consistently improves the performance even further (Table 3) To understand the source of the improvement,
measure sparsity with a measure called Hoyer sparsity which characterizes how biased trees are toward selecting a particular subset of features
VD- VD-Thh Tel2 for D-dimensional vector of split fractions
(Hoyer 2004) defined as Sparsity(e]
VD-1
such that | = and where ca measures the fraction of times the dth variable was used in a split (higher more important). The sparsity score is zero when all features used equally often.
Sparsity increases markedly with both whitening and ICA.
However, this measure does not specifically show that the informa- tion being ignored as a result of the increased sparsity specifically has to do with the distinction between technical and biological variation. To assess this, we measure how well the information useful for predicting the biological effect is concentrated in the top 1% of predictors. Denote by 74% the top &% most important features for the prediction of the biological effect y of interest, and by 2 the remaining features. Then, the concentration is given AUROC (:
Model
Concentration (1)
Base
0.163
PCA
0.332
PCA ICA
0.386
PCA + Rand
0.287
Table 4: Concentration of biolog- ical variation in the top 25% of features.
1, i.e., the improvement in predicting perturbation y from the top features versus the bottom. The concentration increases uniformly with whitening and with ICA (Table even in the case of PLK1 guides where it does not confer a substantial gain in out-of-distribution AUROC. The results are not sensitive to reasonable values of & (Appendix A.7.4). Interestingly, the results suggest that whitening alone biases the representation toward becoming axis-aligned even without ICA.
by Concentration [v] = AUROC 1
5 DISCUSSION
We have developed theory of statistical hear-identifiabiiity of neural representations which is applicable to the internal representations of real-works self-supervised models. Notably, in contrast to prior work, our result requires few assumptions on the getta-eeerating process, instead trading these off for assumptions on the model alone, and applies to a broad class of models including supervised learners, next-token predictors, and self-supervised learners. Additionally, we have shown that additional assumptions on the data-generating process can confer an even stronger result: namely, provable structural identifiability of the latent variables which generated the observables. We directly test our theory in wooldworld, off-the-shelf, pretrained sulf-super models. Furthermore, we leverage our theory to motivate the application of ICA to the latent spaces of self-supervised models and show that it can achieve tate-of-thhe-rr disentanglement results, including some of the first. disentanglement results for ofitt--dittiiuution generalization in real-world data.
Limitations & future work The primary limitation of our work is the difficulty in empirically testing the bi-Lipschitz assumptions necessary for our theory. Instead, we test the downstream effects of our theory in four sets of experiments, and offer arguments from prior work which show that common regularization techniques which enable training of practical-scale neural networks (often referred to as "dynamical isometry", see also Appendix A.4 may lead to this condition. Interestingly, because the local bi-Lipschitz assumption is largely agnostic to data modality and model implementation details, it potentially applies to a broad class of both data-generating processes and models. As such, it may be an interesting lens for studying the phenomenon of cross-model representation convergence (Maiorca et al. 2023 Fumeeo et al. 2024). which is largely unaddressed by existing theory because existing identifiabile bibity results each require different assumptions on the lata-generating process for different models (Huh et al. 2024 Reizinger et al.) 2025a). Although we echo the calls of Reizinger et al. (2025a) for extensions the practical regime (e.g. finite samples, imperfect optimization), we do not treat this case here, although it could be an extension of our framework. Additionally, because ours are the first identifiability results that apply to the intermediate layers of transformer-base models (Basile et al. 2025). In particular, Liu et al. (2025) show that discrete concept models learned atop last-layer GPT representations render the entire model end-to-ent linearly identifiable, and the results of our paper suggest that this technique may work for intermediate layer representations as well.
next-token predictors, they may be useful for the interpretation of these
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Chan Zuckerberg Initiative (CZI) through the Al Residency Program. We thank CZI for the opportunity to participate in this program and the CZI Al Infrastructure Team for support with the GPU cluster used train our models.
REFERENCES
Alestalo, D. A. Trotsenko, and J. Väisälä. Isometric approximation. Israel Journal of Mathematics, 125(1);61 82, Dec 2001. ISSN 1565-8511. doi: 10.1007/BF02773375 https://dooi
000/10.1007/800277333
D. Michael Ando, Cory Y. McLean, and Mare Berndl. Improving phenotypic measurements in high-content imaging screens. bioRxiv, 2017. doi: 10. 1011101/1 URL https://ww>
John Arevalo, Ellen Su, Jessica D. Ewald, Robert van Dijk, Anne E. Carpenter, and Shantanu Singh. Evaluating batch correction methods for image-based cell profiling. Nature Communications, 15(1):6516, Aug 2024. ISSN 2041-1723, doi: 10. 11.111//41447-02-55001133333 1038 /433///4446/ 024 0044-5061-55
URL https
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, and Julian McAuley. Rezero all you need: Fast convergence at large depth, 2020. URL
https://arxiv.orgg wttp://///////...////////
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balean, and H. Lin (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 12449-12460. Curran Associates, Inc.. 2020. URL https://proceedingss wwwplu/nssnuurriss.oocommm cc/pappp.pppppppppepppeppp stapperrffieespppeeee fttppe/ee..eeeeppep/////// File/ waw.aaaeeeeccoceeeaaa22606666600
Lorenzo Basile, Valentino Maiorca, Luca Bortolussi, Emanuele Rodolà, and Francesco Locatello. Residual transformer alignment with spectral decomposition, 2025. URL httpss://arx:
arg/abs. 2411. 00246
Michael Bereket and Theofanis Karaletsos. Modelling cellular perturbations with the sparse additive mechanism shift variational autoencoder. In Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. 1-12. Curran Associates, Inc., 2023. URL https::/pproceedinggg www.neuurrinccm wwtpppppp..ppp....eeeeepppppepee files//aperr2023/ Etee:00laabbaaeee3444444444----3
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gon- zalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alex Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao. Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision-languagg-acttion models transfer web knowledge to robotic control. In arXiv preprint arXiv:2307.15811 2023.
Simon Buchholz and Bernhard Schölkopf. Robustness of nonlinear representation learning. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.). Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 2024.
PMLR, 21-27 Jul html
ngs. mlr wtt.....e.eeee//////oooooo......
Chris Burgess and Hyunjik Kim. 3d shapes dataset. https///ghhhbboomm/eeepmm//a/aap 2018.
11

---

Published as a conference paper at ICLR 2026
George Casella and Roger Berger. Statistical Inference. Duxbury Resource Center, June 2001. ISBN 0534243126.
Mare Castella, Selwa Rafi, Pierre Comon, and Wojeiech Pieczynski. Separation of instantaneous mixtures of a particular set of dependent sources using classical ica methods. EURASIP Journal on Advances in Signal Processing, 2013(1):62, Mar 2013. ISSN 1687-6180. 80. doi: 10.1 1166 (667-6180-2013-44 URL //oiipp::/ddi...o org. 10. 11 8 687 61 -00-800-211--0
Srinivas Niranj Chandrasekaran, Jeanelle Ackerman, Eric Alix, D. Michael Ando, John Arevalo, Melissa Bennion, Nicolas Boisseau, Adriana Borowa, Justin D. Boyd, Laurent Brino, Patrick J. Byrne, Hugo Ceulemans, Carolyn Ch'ng, Beth A. Cimini, Djork-Arne Clevert, Nicole Deflaux, John G. Doench, Thierry Dorval, Regis Doyonnas. Vincenza Dragone, Ola Engkvist, Patrick W. Faloon, Briana Fritchman, Florian Fuchs, Sakshi Garg, Tamara J. Gilbert, David Glazer, David Gnutt, Amy Goodale, Jeremy Grignard, Judith Guenther, Yu Han, Zahra Hanifehlou, Santosh Hariharan, Desiree Hernandez, Shane R. Horman, Gisela Hormel, Michael Huntley, Ilknur Icke, Makiyo lida, Christina B. Jacob, Steffen Jaensch, Jawahar Khetan, Maria Kost-Alimova, Tomasz Krawiec, Daniel Kuhn, Charles-Hugund Lardeau, Amanda Lembke, Francis Lin, Kevin D. Little, Kenneth R. Lofstrom, Sofia Lotti, David Logan, Yi Luo, Franck Madoux, Paula A. Marin Zapata, Brittany A. Marion, Glynn Martin, Nicola Jane McCarthy, Lewis Mervin, Lisa Miller, Haseeb Mohamed, Tiziana Monteverde, Elizabeth Mouchet, Barbara Nicke, Arnaud Ogier, Anne- Laure Ong, Mare Osterland, Magdalena Otrocka, Pieter J. Peeters, James Pilling, Stefan Prechtl, Chen Qian, Krzysztof Rataj, David E. Root, Sylvie K. Sakata, Simon Scrace, Hajime Shimizu, David Simon, Peter Sommer, Craig Spruiell, Iffat Sumia, Susanne E. Swalley, Hiroki Terauchi, Amandine Thibaudcau. Amy Unruh, Jelle Van de Waeter, Michiel Van Dyck, Carlo van Staden, Michal Warchot, Erin Weisbart, Amélic Weiss, Nicolas Wiest-Daessle, Guy Williams, Shan Yu, Bolek Zapiec, Marek Zyta, Shantanu Singh, and Anne E. Carpenter. JUMP Cell Painting dataset: morphological impact of 136,000 chemical and genetic perturbations, March 2023. URL httpps:/wwit wioiiiiooooo viooiii...... org/ wontont//////cccccccc /ww.nt/11..... 11..0.11022203333 23..333...... 0.30.33403322 Pages: 2023.03.23.534023 Section: New Results.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan-
glement in variational autoencoders. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 31.
Curran Associates, Inc., 2018. URL http:::///roceeenngggsss... cttp:///nw..rrrrii..........oo.. ccpppppeeeppppppprrrppppppppp wttee/eeedfffeeeeeeeeeee #iles/paper/20185 /ttp:////...1...2....002233333..
Wenlin Chen, Julien Horwood, Juyeon Heo, and José Miguel Hermández-LLoatoo Leveraging task structures for improved identifiability in neural network representations, 2024. URL https
sarxxiv.org/ab...
14861
Pierre Comon. Independent component analysis new concept? Signal Processing. 36(3):287-314 1994. ISSN 0165-1684. doi:
URL https://
www. htt.://cccccccccoooooooeeeeeeeea Order Statistics.
Higher
Oswaldo de Oliveira. The implicit and the inverse function theorems: Easy proofs. Real Analysis Exchange, 39(1):207, 2014. ISSN 0147-1937. doi: 10. htaaa/aaaaaaalcc...11.1.00 URL exch .39.1 0207
http://dxx doi. or./10. 14321/real
Marco Fumero, Marco Pegoraro, Valentino Maiorca, Francesco Locatello, and Emanuele Rodolà. Latent functional maps: a spectral framework for representation alignment. A. Glober- son, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Ad- vances in Neural Information Processing Systems, volume 37, pp. 661 66178-66203 Curran Asso- ciates, Inc., 2024. 00tp0//0000011011330000000006000
witppppppppeeeeepppppeepeeeepp pdf
Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Mar- tin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: a new disentan- glement dataset. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and
12

---

Published as a conference paper at ICLR 2026
R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL htl/aattaaa///rocdediiii www.iee..urpuaaaeeeee d97c404b6 1911 0tt.000061122eee1101111111621211
Carrie Grimes. New Methods in Nonlinear Dimensionality Reduction. PhD thesis, Stanford University, 05 2003.
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q. Tran, Jonathan Deaton, Marius Wiggert, Rohil Badkundri, Irhum Shafkat, Jun Gong, Alexander Derry, Raul S. Molina, Neil Thomas, Yousuf A. Khan, Chetan Mishra, Carolyn Kim, Liam J. Bartie, Matthew Nemeth, Patrick D. Hsu, Tom Sercu, Salvatore Candido, and Alexander Rives. Simulating 500 million years of evolution with a language model. Science, 387(6736):850 858, 2025. doi: 10.1126/sceence. /00.1///sccimmc..dd00..... ads0018. URL https: /tt:://wwwww /ww.. science org/doi/abs /0.0 1121111666/9
twwp:/1iiicci/i.. scwence ads0018
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, net/ mtt.me/foorommm.d3322222222
2017. URL httpt:ttttppeeeeeeeeeeee
Daniella Horan, Eitan Richardson, and Yair Weiss. When unsupervised disentanglement possi- ble? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 5150-5161. Curran Asso- ciates, Inc., 2021. URL /ttp///tss///prreeeei......aarrr
wttp:erwwpppppcppprr.eepep http/papee____e lateerr_illsspppepeeeeeee /aper/
pdf
Patrik O. Hoyer. Non-negative matrix factorization with sparseness constraints, 2004. URL https
Kyle Hsu, William Dorrell, James Whittington, Jiajun Wu, and Chelsea Finn. Disentanglement via la- tent quantization. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. http://paprr_ffeesleeeeeeee fitpseffeeeeeeeperrreeepaaaaaeee ciates, Inc., 2023. URL wttp///tpss//pprreeedena.....ees 2023/file/ 00221111//6657771cc1133377666662
Curran Asso-
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic represen- tation hypothesis. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 20617-20642. PMLR, 21-27 Jul 2024. URL htt.ttttt//pocceeiiiitnee 1alh24ahhtm.
mlr. www.r.press/v233
A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applica- tions. Neural Networks, 13(4):411-44 2000. ISSN 0893-6080. doi: https://oii.rr/// 1016/30899360080000000002222
URL https: www.. scwencedi rect. wtw.:///eeeeccttttcococ/cceeeeee cwwpscien.c./////ooccccccoccoc
ariicle/ss1/000933000000000
Tero Karras, Samuli Laine, Miika Aittlla, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Yan Liu. Lightgbm: highly efficient gradient boosting decision tree. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 3149-3157, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational autoencoders and nonlincar ica: A unifying framework, 2020a. URL
abs/ 04809
Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma, and Aapo Hyvärinen. Iee-beem: Identifiable conditional energy- based deep models based on nonlinear ica, 2020b. htthtt
7/ rxxiv
11537
13

---

Published as a conference paper at ICLR 2026
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimizaton........ In Yoshua Bengio and Yann LeCun (eds.), ICLR (Poster). 2015. URL http:://bbl: uni-trier de/ db/ Donf /ont//clr//cc:2000
html WKincmaBiii2
Max Klabunde, Tassilo Wald, Tobias Schumacher, Klaus Maier-Hein, Markus Strohmaier, and Florian Lemmerich. Resi: A comprehensive benchmark for representational similarity measures, 2025.
URL htt.tttts::/aaaee http///xiiiiorrraa/////a/
00531
T. C. Koopmans and O. Reiersøl. The identification of structural characteristics. The Annals of Mathematical Statistics, 21(2):165-181 1950. ISSN 00034851. URL http: / /wwww//ww/..//////////////.....t jstor
org/ org/ stwale/eaeeeeeeeeeeeeeeeee 223699
Oren Kraus, Kian Kenyon- Deaon, Saber Saberian, Maryam Fallah, Peter McLean, Jess Leung, Vasudev Sharma, Ayla Khan, Jia Balakrishman, Safiye Celik, et al. Masked autoencoders for microscopy are scalable learners of cellular biology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11757-11768 2024.
Oren Kraus, Federico Comitani, John Urbanik, Kian Kenyon-Dead Lakshmanan Arumugam, Saber Saberian, Cas Wognum, Safiye Celik, and Imran Haque. Rxix3-core: Benchmarking drug- target interactions in high-content microscopy, 2025. / 20018
Sebastien Lachapelle, Tristan Deleu, Divyat Mahajan, Ioannis Mitliagkas, Yoshua Bengio, Si- mon Lacoste-Julien, and Quentin Bertrand. Synergies between disentanglement and spar- sity: Generalization and identifiability in multi-task learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 18171-18206. PMLR, 23-29 Jul 2023, URL httpss:/pproceeiing..ut mlr www.spreseeee...ec /22///22222/////2222// wwchapellehaaa.ccc html
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://www..cc....o////////aaiaa
2, 2010.
Yonghyeon Lee, Sangwoong Yoon, MinJun Son, and Frank C. Park. Regularized autoencoders for
isometric representation learning. In International Conference on Learning Representations, 2022.
URL /BRL:tttps:://peeeeevee h prrruummm--m@00 id=m@xt8177IL
Alexander Lin and Alex Lu. Incorporating knowledge of plates in batch normalization improves generalization of deep learning for microscopy images. In David A Knowles, Sara Mostafavi, and Su-In Lee (eds.), Proceedings of the 17th Machine Learning in Computational Biology meeting, volume 200 of Proceedings Machine Learning Research, pp. 74 93. PMLR, 21-22 Nov 2022. URL http:: http::::://pprceecinngg /tt.:/eeedeaaaa...rrrree////////
ml
html
Yuhang Liu, Dong Gong, Yichao Cai, Erdun Gao, Zhen Zhang, Biwei Huang. Mingming Gong, Anton van den Hengel, and Javen Qinfeng Shi. predict therefore am: Is next token prediction enough to learn human-interpretable concepts from data?, 2025. URL httpt::///// /ttp:////xxvvvvo.. org
abs/ abs/2503 2203. 08980
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Leurning, volume 97 Pro- ceedings of Machine Learning Reseurch, pp. 4124 PMLR, 09-15 Jun 2019. URL fittps://procest prrer
m ww..r.pess/vvee v97 www.eclall..................
html
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations, 2020.
Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment. In A. Oh, Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural
14

---

Published as a conference paper at ICLR 2026
Information Processing Systems, volume 36, pp. 55394 55414. Curran Associates, Inc.,
2023. URL https::/ppoooeedinss.euuuue neurss wttpepecpppppeeeeeeeeeee http:/papr__llssppaaaeaeee hwtp://appeseeea/e222222 http://aaaaaaaalllo..a.aa2aaaaa0
Emanuele Marconato, Sébastien Lachapelle, Sebastian Weichwald, and Luigi Gresele. All or
none: Identifiable lincar properties of next-token predictors in language modeling, 2025, URL
https: wtt.ariww....g...... 2410./241.... 24300230111
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. http:///stttb..commmeeepii///sss
2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, 2018. BRLhhtts::// http:////aaa...oo///////////0/ 05957
Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit Patel, and Animashree Anandkumar. Semi-supervised StyleGAN for disentanglement learning. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7369-7369. PMLR, 13-18 Jul 2020. URL https: http::s//rroceeiiii ngs ml www.s / wtt./eee00...... .tt.00...mmmll...mmm
Beatrix M. G. Nielsen, Emanuele Marconato, Andrea Dittadi, and Luigi Gresele. When does closeness in distribution imply representational similarity? an identifiability perspective, 2025.
httptttts///xii....//m wwtpxxiiii.rrgaab.oooaoo http:/bs//200000 03784
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-688, 1995. ISSN 00063444, 14643510. URL http::/www. wtwp::jssto..ogstaaaaa... wttp://///aabb/23322222222 /ttt:///www.....ooooog////2222//
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn; Machine learning Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Patrik Reizinger, Szilvia Ujváry, Anna Mészáros, Anna Kerekes, Wieland Brendel, and Ferenc Huszár. Position: Understanding LLMs requires more than statistical generalization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 42365 42390. PMLR, 21-27 Jul html
2024. URL https: httts::/proceeeinnge wwa.nnsmmll.......... www.sppeessvvvv.. / w
Patrik Reizinger, Randall Balestriero, David Klindt, and Wieland Brendel. Position: An empirically grounded identifiability theory will accelerate self-supervised learning research, 2025a. URL
https::/arxxie org. abs 2504. 13101
Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia E Vogt, Randall Balestriero, Wieland Brendel, and David Klindt. Cross-entropy is all you need to invert the data generating process. In The Thirteenth International Conference on Learning Representations, 2025b. URL https
Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representations. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9030-9039, PMLR, 18-24 Jul 2021. URL /tt..u//ttttt////oooeeeaaaaa htt.:/eiiiii..n...ppsss/////eeee
html
Bernard Russo. Maps which preserve equality of distance. must they be linear? Lecture notes, Fuller- ton College Colloquium, November 2017. BtLLhttss://ww/ wwwpwwww.......mmmmmmmmm.wwwwwww www..attt..iii.eeebbbrrssss..e
slic111617full. pdf
Shohei Shimizu, Patrik O. Hoyer, Aapo Hyvärinen, and Antti Kerminen. A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7(72):2003-200 20300, 2006. URL /tttt:///mll wtt.mm/..rr//pppp//////// shimi z html)
15

---

Published as a conference paper at ICLR 2026
Maciej Sypetkowski, Morteza Rezanejad, Saber Saberian, Oren Kraus, John Urbanik, James Taylor, Ben Mabey, Mason Victors, Jason Yosinski, Alborz Rezazadeh Sereshkeh, Imran Haque, and Berton Earnshaw. Rxrxl: A dataset for evaluating experimental batch correction methods. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4285-4294 4294, June 2023.
Joshua Tenenbaum, Vin Silva, and John Langford. A global geometric framework for non- linear dimensionality reduction. Science, 29065500):2319-2333,, 2000. doi: 10. 126/science 2309.2319. URL
www.waciien
5500 2319
Jussi Vaisala. survey of nearisometries, 2002. URL http:ss//arxw..oorg/a////ee
0201 098
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve. and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances neural information processing systems, 34: 34:16451-16467, 2021.
Martin J. Wainwright and Michael I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, doi: 10.1561/22000000001.
2008. ISSN 1935-8237. URL /ttp://dxxxxo doi org 10. 1561 1561/2203000001
James C. R. Whittington, Will Dorrell, Surya Ganguli, and Timothy Behrens. Disentanglement with biological constraints: A theory of functional cell types. In The Eleventh International Conference on Learning Representations, 2023. URL httt::://oonneeeeee.eeee//aaaaa
GfhZnGH
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S.
Schoenholz, and Jeffrey Penning-
ton. Dynamical isometry and mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks, 2018. URL https://arxiv http:///aaxii.ooooa//////e// 05393
Dingling Yao, Dario Rancati, Riccardo Cadei, Marco Fumero, and Francesco Locatello. Unifying causal representation learning with the invariance principle, 2025. URL htl/tttps//arxi..otm
abs/2409, 02772
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In International Conference on Learning Representations, 2019. URL
https://openreview net htte:/ooormmiidddmem
3Ret7
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12979-12990 PMLR, 18 24 Jul 2021. URL https://prrceedi.ng mlr www.sppeeseeeee
www.pppsssssv...eee vvtp//v1/////////// http://aaeddnnns....peeee///i/ia imme rmaan22a. html
A APPENDIX
A.1 A HISTORY OF THE TERM "IDENTIFIABILITY"
Statistical identifiability Identifiability has long history in statistics. For example, consider the following definition from Casella & Berger (001)) [p. 548], a canonical textbook in mathematical statistics:
Definition. parameter 0 for a family of distributions {f(x 8) : A E +) is identifiable if distinct values of correspond to distinct pdfs pmfs. That is, if + 0', then f(x 0) is not the same function as f(x 0').
Our Definition D in the main text makes a straightforward generalization from likelihoods f to losses f. Further, because of the dominance of empirical risk minimization and the fact that, unlike most
16

---

Published as a conference paper at ICLR 2026
likelihoods in statistical inference, losses non-convex, we define identifiability at the minimizers of the expected loss. In other words, our definition of identifiability agrees with the statistical one, modulo some small changes to make it useful for talking about non-convex optimization, specifically the empirical minimization of non-convex risk functions. Indeed, this means that not only does Definition T agree with the statistical definition, but also other recent attempts at defining and proving representation identifiability results for practical machine learning models (Roeder et al. 2021
Reizinger et al. 2024).
Structural identifiability On the other hand, there is a similarly long history of "structural identifi- ability" from econometries. For example, consider Koopmans & Reiersøl (1950), which partly led to the Nobel prize:
Identifiability of structural characteristics by model. It therefore question of great practical importance whether statement converse to the one just made is valid: can the distribution H of apparent variables, generated by a given structure S contained in model S, be generated by only one structure in that model?
Clearly, this is different definition. Indeed. Koopmans & Reiersøl acknowledge that it different from the statistical definition above. It implicitly assumes the existence of "true model" S living within the estimable model class S (and that this model generated the data), which flies in the face of mathematical statistics' common quip "all models are wrong."
Pearl 1995) inherits this definition, perhaps contributing to its later infusion into the ICA literature:
DEFINITION 4 (Identifiability). The causal effect of X on Y is said to be identifiable if the quantity P(Y]X) can be computed uniquely from any positive distribution of the observed variables that is compatible with graph G.
Put simply, both these definitions of structural identifiability assume that the true data-genete generating process matches the model, and under this assumption, consider whether some structural parameter of interest can be identified. As a consequence, we refer our Definition[aa "structural identifiability" The only generalization we make is that we do not specifically enumerate S, indeed require the true structure S E S. We do not enumerate S because instead we enumerate θ the space of possible parameters of the neural network model, and specify the mapping F which generates the learned structure from a setting of the parameters F : + fo- We refer to the "true structure" as u. The relaxation of the requirement that the model class contains the "true structure" allows for other interesting cases, as illustrated in Example 2 below.
Identifiability in linear ICA Occasionally, these definitions of identifiability are used somewhat interchangeably. This is particular true in the case of recent developments in independent components analysis, such as extensions original seminal linear ICA paper (Comon) 1994) is a statistical identifiability result, not structural (we edit the quoted Corollary slightly so that self-contained):
the nonlinear mixing regime. Interestingly, the main result of the
Corollary 13. Let no noise be present in the linear ICA model with observations y. and define y = .Mx and y Fz for a random variable x with independent components such that at most one is Gaussian. Then if V (a contrast function taking densities as inputs) discriminant, V (px) = (px)) and only if F MAP where A is an invertible diagonal matrix and P a permutation.
Of note, linear ICA a case where most existing identifiability results depend on well-specification of the model. In particular, if the data is not generated by linear mixing from independent components, there is no guarantee that a linear decomposition into independent components exists (Castella et al.] 2013)) and therefore the statistical identifiability claim above is vacuous. In other words, there are few or no results available for statistical identifiability of linear ICA that don't 't also imply structural identifiability. This perhaps helps explain why the two concepts have been used somewhat interchangeably in this area of the literature. However, as we will see in the examples below, this is not necessarily the case for all models of interest.
Notably, LiNGAM, a classical approach which extends linear ICA to causal discovery Shimizu et al. (2006), provides an early example of this distinction between statistical and structural identifiability
17

---

Published as a conference paper at ICLR 2026
extending to graphical structures. In particular, they show that the independent components estimate can uniquely recover a causal graph under the assumption of linear functional relationships and additive noise (statistical identifiabilityy. Structural identifiability follows under the assumption that LiNGAM generated the data.
Identifiability in non-linear ICA Noting that in general, non-linear extensions of ICA impossi- ble (Locatello et al. 2019), recent works have attempted to find sets of assumptions that render the task tractable. One line of work utilizes side information, showing that independence conditional on this side information leads to identifiability [Khemakhem et al. (220aa). Notably, Khemaahem et al. (2020a) explicitly define their identifiability as statistical identifiability (up to an equivalence class), but point out that if the true data-generating process takes the same form, structural identifiability of the true latents is achieved.
Below, we show how two other examples fit into these definitions.
Example I: mis-specified linear regression The first example comes from traditional statistics.
Consider the identifiability of the usual linear model y zT + n when the model is mis-specified in the sense that the true data-generating process is y = f f(x + v/' for some non-linear f. meets certain conditions, the following facts are true:
1. 3 is statistically identifiable under the usual OLS assumptions (i.e. identifiable according to our Definition [
2. f is not structurally identifiable (i.e., e-nearly structurally identifiable with 0) according to Definition [
3. the model is not absolutely identifiable according to Reizinger et al. (025a), because the data-generating process does not match the model,
4. a function is structurally identifiable (i.c., e-nearly with e 0) according to our Definition 2 where h in some sense the "nearest" linear function f.
We emphasize that in this first example, do not exploit the e-nearness relaxation in our definitions, only the distinction between statistical and structural identifiability.
Example 2: masked autoencoders under imperfect reconstruction Now, consider observations generated by some arbitrary non-linear mixing of latent factors X = g(Z). For a masked autoencoder, the following facts could be true all at once, such as in the case of imperfect reconstruction:
1. the internal representations given by the encoder f are statistically identifiable according to our Definition
2. the true data- gentraaenn process g is not structurally identifiable according to Definition 2 because of imperfect reconstruction, but
3. some approximation to the data- tettaageneraing process is structurally identifiable, but + g because masked training prevents perfect reconstruction,
4. the model is not necessarily absolutely identifiable according to Reizinger et al. (2025a).
depending on the assumptions on and g (i.e. they might not reside in the same set of possible models).
In this case, we might not have the simple analytical form for h that we do in Example 1, but it's clear that something about the datergnnn process is structurally identifiable. We emphasize that our theory does not cover this case, because it requires notion of "closeness" between h and y which is not necessarily obvious (nor is it necessarily covered by our conception of e-nearness, because the similarity is in observation space, not representation space).
In particular, Theorem I does not require perfect reconstruction to yield statistical identifiability (and there perhaps is structural identifiability of some process h which is the nearest near-isometric approximation to the manifold of masked inputs, although we don't 't make this argument explicit), but Theorem B] does require perfect reconstruction to yield structural identifiability of g. A similar observation can made about statistical versus structural identifiability in exponential family models like GPTs, discussed in Appendix A.3.1
18

---

Published as a conference paper at ICLR 2026
A.2 THEORY ROADMAP
We provide a brief table of contents to this set of appendices, which cover our theoretical contributions.
In Appendix A.3.I we show the utility of our Definition by showing that the statistical identi- fiability result from Roeder et al. (202T) meets definition of statistical identiffiblity for the penultimate layer of exponential family models such as GPTs.
Appendix A.3.2 we prove Theorem] 1 our key statistical near-identifiabiliyy result up to Hrigid for the internal representations of general self-supervised models.
In Appendices A.3.3 and A.3.4 we provide model-specific treatments for masked autoencoders and GPTs, showing that Theorem holds for these models.
In Appendix A.3.5 prove Theorem 2 showing that linear ICA can resolve the rigid indeter-
minacy left by Theorem [ (or indeed any other linear identifiability result), yielding statistical near-identifiabilityy up to H.r (the space of signed permutations).
In Appendix A.3.6 we prove Theorem 3 showing that for bi-Lipschitz data-generating processes. statistical near-identifiability extends to structural near-identifiability.
In Appendix A.4 we show that the dynamical isometry condition used for characterizing practical neural network training regimes implies the bi-Lipschitz assumption necessary for Theorem !|
Appendix A.3 we prove that structural identifiablityy (Definition 2 is strictly stronger than statistical identifiability (Definition [ including in the e-near case provided the transformation
class H is bounded.
A.3 PROOFS
A.3.1
STATISTICAL NEAR-IDEREEE IDENTIFIABILITY OF EXPONENTIAL FAMILY MODELS
The goal of this section to contextualize two important statistical identifiability results (Roeder
et al. (02T)) which apply to the penultimate layer of exponential family models, discussed in Section
(. Connection with prior results"). Specifically, we show that the result of Roeder et al. (2021)
meets the requirements of our Definition [ and explain how the result of Nielsen et al. (2025) relates to our definitions. In particular, we are interested in models which have losses taking the form in Equation from Section [.1] Co y) (z)) + Ao(x) = log (o xx)
where 8 are the parameters of the model and 77e give the representations of interest. They key result the assumption of sufficient diversity, which in the case of next-token predictors is the condition that the final linear classification head in some sense "full rank".
of Roeder et al. (2021) hinges
First, we prove a simple lemma showing that sufficient diversity can be recast as a mild property of the data distribution, combined with assumption the model.
Definition A.3. The mapping t y + RD satisfies sufficient diversity with respect to P(x, y) if repeated sampling from the marginal distribution ya ~ Pyy) yields linearly independent vectors {t(ua) - t(yo)) d=1
) for some yoo E y.
Lemma A.1. t only if Cov(t(v)] 2 17 for some >>
R satisfies the sufficient diversity assumption with respect to P(y) if and
Proof. Assume satisfies the sufficient diversity assumption. Let v E RR be any nonzero vector. By linear independence, we have v" (t(Ba) - t(yo)) o 0 almost surely for any fixed yo, and therefore o t'yly) # almost surely for any constant c E R. Thus, " Cov(t(y)] Varle Tar])) required.
as
To see the other direction, note that any draws from the distribution (((())) are linearly indepen- dent almost surely by the positive definiteness of the covariance matrix.
To see why this condition on the data distribution is extremely mild, note that even P(y) is a finite categorical distribution over labels, long there least D of them, it is possible that t satisfies
19

---

Published as a conference paper at ICLR 2026
sufficient diversity. Therefore, it is best regarded condition the model. Now, we can state the key result of Roeder et al. (2021) as a lemma.
Lemma A.2. (Theorem 1 of Roeder et al. (2021) Let P(r, v) be data distribution, and let M = {Ca(x,)) y) = - -no(x) (ty()) A(z): E be a model with an exponential family loss.
Then, if Lu = La almost everywhere with respect to P, we have that ne(x) = Lm (x) almost everywhere with respect to P for some L E Hiinear whenever to and to satisfy Definition A.3
The only additional requirement to meet our definition of e-ncar-identiiiabliiy (Definition is that θ is sufficiently rich to approximate a unique minimum of the loss. We make this concrete in the following definition. Importantly. this assumption is not equivalent to requiring that the true data distribution in the model. Instead, it amounts to an assumption that the pointwise projection of the true conditional P(y æ) onto the space of exponential family distributions is unique and in the model. It turns out that this projection exists and is unique under the sufficient diversity assumption, together with very mild assumptions on the data distribution. Therefore, this parallels the situation of statistical identifiability of masked autoencoders (see Appendix A...3 and Appendix [..]] Example 2), where perfect reconstruction is not required for statistical identifiaiiltyy (but might be required for a notion of structural identifiability).
Definition A.4. For exponential family loss Co(x, y) - (2) + Ao(x), a data distribu- tion P(x, y). and a sufficient statistics function satisfying sufficient diversity, consider the following minimization problems:
k" (z) = arg min (L(P(y 2) Q..)
KER
where (x)y)))))((( + A(x) is the upproximating distribution. Then, a parameter space O of an exponential family model of form in Equation
7 is sufficiently rich to exactly model the minimizer if this minimizer exists for every x, is unique for almost every with respect to P(z), and there exists 0 E e such that no = K² (z).
Remark. For categorical likelihoods, the only requirement on the true data distribution P(y x) is that all categories must have positive probability for the above to hold (for almost every x). This means that the assumption is trivially satisfied for e.g. GPT-class models assuming that any token might be next for any given state. This condition is called minimality (Wainwright & Jordan 2008).
Now we give the full statement of the proposition showing that the Lemma A.2 combines with the above assumption to yield mnea-ideentiffabiiiyy in the limit according to our definition (with c c= 0). T + A²(z): : A E θ} be model with an exponential family loss. Then, for F :0 no.
Proposition A.1. Let P(x, y) be data distribution, and let M = (Co v)
F) is indentifiable in the limit up to Hineer when the label distribution P is sufficiently diverse and the approximating class is sufficiently rich according to Definition A.4
Proof. Minimizing the expected loss of the model via empirical risk minimization is equivalent to minimizing the following cross- coss-y:tropy:
arg min Ex En (Co v)] arg min E, Ex log go [)] dEe
dee
arg min Ez- (CF((q/ x)) || P(y x))]
BEe
where - log qa (y | x)) x (=) + (z) parameterizes an exponential family approximating class. By sufficient richness, call qa. the unique distribution induced by any minimizer 0*, So, for any two minimizers 0, 0' we have Lg = Lg almost everywhere and Lemma A.2 yields identifiability see Remark) as required.
in the limit (Definition) I
Nielsen et a (2055) extends this result to the case where it is not guaranteed that qo gor for any two optimizers 0, 0. For example, this might arise due to the sufficient richness condition not being met, or due to imperfect optimization. Instead, a closeness condition can be placed go and qa, yielding a notion that is similar to our notion of statistical c- c-near- identifiability. However, we note that Nielsen et al. (2025) does not aim to characterize the optima of particular loss, instead relaxing this requirement to cover any procedure yielding likelihoods. This makes it an interesting extension of Roeder et al. (202T) in different direction than ours.
20

---

Published as a conference paper at ICLR 2026
A.3.2 RIGID HEAR-IDENTIFIABILIT OF MODELS WITH BI-LIPSCHITZ MAPPINGS
We begin by briefly outlining the approach for proving Theorem [ The key idea to leverage modern results in isometric approximation (Vaisala 2002 Alestalo et al. 200T). In particular, consider the latent spaces of two models which produce the same outputs. Call the mappings from latents to "stitch
outputs in these two models 9 and g'. If the mapping from latent output is invertible, together" the latent spaces of the two models by a single function g o g'. So, g o g' maps the representations of input under
model to its representations in the other model.
We assume that both and g' are locally bi-Lipschitz, meaning that distances between nearby points are not too badly deformed, and that g and g' are smooth C' diffeomorphisms. When this is the case, _ o o' is also locally bi-Lipschitz and smooth, and therefore nearly an isometry, with "nearly" determined by the bi-Lipschitz constants. The convexity of the latent spaces yields global bi- Li-Lipschiteness with the same bounds.
The isometric approximation theory outlined in |Vaisala 2002 Alestalo et al. (001]) then tells us how far g' g-_ is from an actual isometry, and allows us to construct bounds accordingly. Isometries on R. are rigid transformations, and together, these facts yield rear-identifiabiliyy up to Higidd
rely on the following definition of locally bi-Lipschitz, which allows us to treat latent-to toant-tooobeevable mappings which might induce manifold structure in the ambient space, allowing the taking of tighter constant L. In what follows, |I-I| denotes the usual & norm unless otherwise stated.
Definition A.5. A function f Z - RN is locally (1+ L-)-ii-ipschiss if for every Z C R. there exists an open neighbourhood U₂ e z such that for all E Uni we have
1 + 12 z'1l (15(2) - f(z')1 S (1+ D)|l z'1l
When L = O, distances are preserved exactly, a notion referred to as isometry. When the bi-Lipschitz constraint is global (i.e., holds for any U C 2) and 2 is bounded, there is nice relationship between the bi-L.pschhtz property and the notion of a near-isometry, which allows for additive distortion of distances.
Definition A.6. A function f Z N for Z R is an e-near-isometry for every z, z' e Z we have ||z z'1l - ll f(z') < z z'| +E.
In particular, if f is globally (1+ -e------ipscciit we have that f is also an e-near-isometry where 8 AL where A- sup2,z'zz ||z x'| || is the diameter of Z.
We require two lemmas to prove our main statistical identifiability result Theorem 1 The first is fundamental result which shows that presennving transformations on e.g. Euclidean spaces (or convex subsets thereof) are always rigid motions.
Lemma A.3. (Mazur- Mazur-Ulam Theorem, Russo 2017)) Let Z and Z' be closed, convex subsets of R with non-empty interior. Then, if f :
is a bijective isometry, then f is affine.
We also leverage the following more recent result (see Vaisala (002) for a history of isometric approximation) which shows that near-isometric mappings (such globally (1 + -))-b---ppschhtzz functions on bounded domains) have bounded deviation from a truly isometric mapping.
Lemma A.A. (Near-Isometre are Near Isometries, Theorem 2.2 of Alestalo et al. (2007)) Suppose 2, C RRD with Z compact and f - Z' is AL--ner-isoomnt where supz,''ezll z'Il.
Then, there exists an isometry U : RDD + TR. such that supzez|(U(s) [()) < CDV
With these two tools in hand, we can provide intuitive overview of the assumptions required and concrete statement of Theorem I proving statistical near-identifiabiiity of the representations.
Assumption Summary. We assume that the representations have convex support (i.e., the pushfor- ward of P(x) by any encoder fo has convex support). We assume that any "decoder" ge is injective, smooth (in particular, at least C.),, and is locally bi- bi-Lipschitz with constant 1 L. Note that injectivity, smoothness, and local bi- bi-Lisschitzness of go (with any constant) implics that go is diffcommrphic onto its image.
21

---

Published as a conference paper at ICLR 2026
Theorem. Let (z)) be data distribution, and let M be model with parameter space θ. Let F + fo. G 9 - 9e and H + 90 o fo. where go is a smooth diffeomorphism and the pushforward of P through fo has convex support. Then, if (P, G Lg. H) is statistically identifiable in the limit, then (P, ④ La, F) is statistically e-near identifiable in the limit up to Hrigid for = CDVVL + L A where 1 + L is local bi- bi-Lipschitz constant bound for go, and cp and A are constants independent of the model (and L).
Proof. Let 8 and 6' be optima of the infinite- data limit of the empirical risk minimization problem for M with respect P.
Without loss of generality, assume that [o(x) = 0 = fo for some x E supp P. noting that translating the latent spaces of both models do not alter any of our hypotheses. Then, by the identifiability of (P, θ Lg. H), we have that = 9ø' 9e satisfies T(0) = O, is clearly smooth and diffeomorphic onto its image, and is therefore globally (1 + L) bi-iishht by the convexity and compactness of the latent spaces and is ([[===2)-nerrisomeeee fo(x')l is the diameter of the latent manifold. Furthermore, we have that T so fo. By the above fact about bi-Lipschitz functions and Lemma A.4, we have that there exists an isometry U such that
where A supx,x'xx IIfo (x)
suff (x) (x)( ess sup - p(a)
p(x))
S <DV 2L + 12A
where ep is a constant depending on D. By convexity of the latent spaces and the Mazur- Ulam
theorem for convex bodies (Lemma A.3), U E Hrigia as required.
Remark. The constant ep depends only on the latent dimension D and can be computed by the following recursive formulae given in Alestalo el al. (2007)
@1 3.3
T = 6.2
(1) = 0.1 + (1+ V 12 6.2)
(A) = 3.02 + 7, (A) (A)/X2 Σ ex(2))2 e6(a)/23
(A) = T. (A) + 21+1 (A) La+1 (A)/22)
(l) = min (max min(max 7a (A), 3n- (X, [)] : >>> 0)
n+1
Bn+1(A,t) t)² 0.1 + (t /22 Tx+1 (2)) + Σ
ex (A)²
k=
with cp (D) for any D. As an example, C3 18.8.
A.3.3 MODEL-SPECIFIC RESULTS: MASKED AUTOENCODERS
The following assumptions are sufficient for a masked autoencoder meet the criteria of Theorem
Model. A masked autoencoder consists of a distribution P (m) defined over a space of masking functions A [mm= X' + L). encoder fo X' U X + RD, and decoder ga RDD R where we assume X C R is compact. For convenience, we write = go o So. The loss function with respect data point and mask is (MAE(h; m, x [l/(m(=)) z112. We assume that X is sequentially dense in X' U &, and that marginalizing the joint distribution over the masks and data P(m. m(z)) yields distribution P (mm()) with full support on X. Further, assume that at the optimum, go is diffeomorphic and locally (1 + -j-bi-Lipschitz for some constant L > 0. Finally, we assume that the image of X' pushed forward through the encoder is convex, and that the
22

---

Published as a conference paper at ICLR 2026
parameterization of h is continuous on X' U X and M is sufficiently rich contain the function which attains the optimal value of the loss, as derived below.
Proof. We begin by showing the existence and uniqueness of the conditional expectation operator B|x T]. By the fact that the masking function distribution P(m) is independent of the data, we have that there is a well-defined conditional density P(x σ) arising from P(š x) = (m ) m(x)) dm, where 8 is the Dirac delta. By compactness of X, z is integrable and therefore the conditional expectation exists and is unique almost everywhere in X. In particular, have the following expression:
Lx xx [m(()) x] dr Ix P [m(x) ã] dæ
E ø
(2)
Then under the masked autoencoding model, (()) R2( x] is the unique minimizer of LMAE by the standard fact that the conditional mean estimate minimizes the mean squared error loss. By the continuity of f on X X and sequential density of X in X' U U. consider an arbitrary sequence in X, 7, + x E X, noting that f(xr) lime, -xx f=xxf((a (āā, is well-defined, unique, and agrees regardless of the choice of sequence. Note that this continuity is by hypothesis. We remark that it would be of interest to study properties of P(m) which lead to conditional expectation operators with certain properties including continuity.
Remark. We emphasize that any such minimizer is also minimizer when an arbitrary subsetting operation applied to both the prediction and the target, such as the usual subsetting to masked tokens in the loss for computational efficiency.
A.3.4 MODEL-SPECIFIC RESULTS: GPTs & SUPERVISED LEARNERS
The results for earlier-layer representations of GPTs and supervised learners take advantage of the penultimate-lay identifiability result in Proposition A.I We formalize this with the assumptions below.
Model. Let P and M be a model satisfying Proposition A.1 with F + fo yielding the representation function of interest, G r go yielding the map between the outputs of fo and the penultimate layer. Let 1 + be a bound on the local bi-Lipschitz constant of the smooth diffeomorphism go. and suppose the pushforward of P through fe has convex support.
Proof. By Proposition A.I we have statistical identifiability in the limit up to Hlinaar of (P, (, H) for : + go o fo. Thus, Theorem applies.
A.3.5 INDEPENDENT COMPONENTS ANALYSIS IN LATENT SPACE
We consider the class of independent components analysis algorithms which optimize a contrast function, such as fastICA (Hyvärinen Ojal 2000 In general, fastICA enjoys good convergence properties despite lack of guarantees, and even in the face of missspccifiiation (Castella et al. 2013). Of particular concern in our setting missspecification of the kind such that there exists no orthogonal transformation in which the components are truly independent. This assumption is non-trivial assess, and furthermore sufficient
necessary for convergence (Castella et al., 2013), so we opt for something more relaxed, aimed at practical optimization. Specifically, we show that near-isometries are simple enough functions that they do not substantially alter the recovered components when ICA converges well, in the sense that they preserve
optima.
Model. For encoder f with whitened outputs f(x),, independent components analysis con- sists of maximizing the contrast J Eb [(q where is a contrast function, over (qi.... ........)?' qD)² = = SD(D), the space of special orthogonal matrices. As technical condition, we require sup, [| (latents must have bounded correlation). The contrast function J must be C² with [J'(y) < L1 and (J"(y)) < L2 for rigid transformations of elements of a convex body y = fx). The optima of the ICA objective I must also be locally convex for s (xx), in the sense that for sufficiently large samples
2 X for some X > 0, where the supremum is over any possible encoder
23

---

Published as a conference paper at ICLR 2026
the Riemannian Hessian Hesso J > for some μ 0 at the optimum, under any perturbation of the data not larger than E. Finally, the only indeterminacy to the optima of an individual ICA problem must be the usual invariances: signed permutations in H.g.
Under these conditions, we can show that e-nearness does not introduce any further complications to the identifiability of the ICA model. We begin with brief lemma about the stability of the whitening operation which typically precedes contrast-based ICA.
Lemma A.5. Let X and X' be zero-mean random vectors in IDD such that || - -X'Il S e almost almost surely. Furthermore, suppose the smallest eigenvalues of the
surely and [X]|, ||X'||| covariance of X and X' are bounded below by X 0. Let W E-112 and W E'-1f2 be the usual whitening matrices for X and X' respectively, Then [W' - W S Ce almost surely where C== x
Proof. First, note that we have the following bound almost surely:
|Xx x & |X(X X1 (X x')x
S |X|le + X S 2aE
Taking expectations yields | E1| S 2as. Now, using the resolvent equation B-1 A--1 B- -B A)A-1, we have the following bound on the difference in operator norms of W and W"
W WI|
12-1/2 2'-
|/ 1/2 ||I
| - الل
2A3/2
ar
A3/2
where the second-to-last inequality follows from the fact that the matrix square root is (1/(2v X))-
Lipschitz when À > O. Operating on the desired norm directly:
[W'X' WX| < W'X - W + W'X WXI
(X'X'-X) IIW [WW'-W)A W)
+ a²e
VĀ X3/2
almost surely so the proposition holds.
The next lemma simplifies the exposition of our main theorem. We show that PCA resolves the first-order dependence structure (i.e. the non-rigid component of linear transformations), leaving only a rigid transformation to be resolved by ICA.
Lemma A.6. Suppose there exists functions := X Z such that for p(x) supported on X, we have Ep(()) (f()) EEp(a) [f'=)) = 0, full-rank covariance of f(x) and (x) and furthermore supzexl1 fx) Ax)( for some invertible matrix A with positive determinant and e 2 O. Then, there exists U Hrigid such that supzexll [f - (x)( S Ce where f are the whitened outputs of J and I' respectively, E Hngid, and C X, (1 (1+ Lixae) where A A and XA the largest and smallest singular values of A respectively. X is lower bound the smallest eigenvalues of the covariance of f(x) and (x)) and fx) [=) S a almost surely with respect P(x).
24

---

Published as a conference paper at ICLR 2026
Proof. Denote and W E-122 and W E'-1/2 the usual whitening matrices for and f respectively. Let WA be any whitening matrix for Af". Let U = WAAW--1 Then is orthogonal (and can be made to have determinant 1 with sign flip by the freedom to choose Wa because UU = WA WW-1ww-- W = W W J, and we have:
= [[[((x) - FaAW
[[W((z) (A)
A (1 (1+ Ar
A3
almost surely with respect to ((z) where the final line follows by application of the previous lemma the usual whitening matrices. The determinant of U is positive by the fact that the usual whitening matrices can be made unique by selecting their positive definite forms and A has positive determinant, so U E Hrigia-
Lemma A.7. (Implicit Function Theorem, de Oliveira 2014 Theorem 2) Let F E C (22; RT) where C RN x R open. Suppose there exists a point (a, b) E Q such that F(a, b) = 0 and By (f,bb) b) is invertible, where represents the part of the argument J in Ik"". Then, there exists an open set X R. such that a X and open set Y Th" such that b Y and:
For each x E X, there is unique y [xx E Y such that f(x)) =
f(a)= and is C with Df(x)
BF by (s. [(x)))
(
(xx)) for all X
Finally, this lemma shows the crux of our argument: if ICA converges well in any pair of latent spaces, the solutions can't differ too much.
Lemma A.8. Sinite-Sample ICA Under Perturbations) Consider whitened observations {xn J and corruptions {En \x (such that the corrupted observations yn. Xn. + En also whitened) both in RRD such that ||x,|| S a and [|5 || S for all n =1, 1....,N N. Let Q+ denote a stationary point of the optimization problem
N
D
jigh
max QEBO(D)
N
d=1
Then, there exists stationary point QE1
Enn of the perturbed optimization problem
D
max QESO(D)
Σ n= d=1
ΣΣ
M(q (x,, + En))
N
such that 1|Qxx, - (.( + Ew)ll S C b for all N where C Lo(otb)+VDL, ab. Furthermore, (E1,,.....5 e is stationary point of the perturbed optimization problem attaining the same value for any signed permutation matrix P E Hy such that det P 1.
EN
Proof. We treat (D(D) as Riemannian manifold and consider the properties of the perturbed optimization problem. For notational convenience, write (* ET noting that then
||||| < Vbb.
We consider first the Euclidean gradient with respect to Q, where we write the perturbed sample as ..,,).,,,,,,(,,...... for any n. Then:
Yn X En. Denote (Q)
N
1
VoJ(Q,e) = N
Σ
gm (Q)x
1=1
25

---

Published as a conference paper at ICLR 2026
The tangent space at point Q SO(D) can be parameterized by the vector space of skew-symmettee matrices. Denoting skew (A) 2 (A A²), the Riemannian gradient is given by the projection of VQJ onto the tangent space:
grado J(Q,E) = skew (eoes(QQVQQaa (QJ(Q,E)
With this machinery, we can apply the Euclidean implicit function theorem (IFT). Let Q. (O(D) be an unperturbed optimum. Adopt the matrix exponential parameterization in a neighbourhood about Q., i.c., let N + SD D) be given by ((2) = Q. exp(It) for some sufficiently small
N R D(D-)) for the map to be injective. Let
F(E, (2) grad [()()),)=)
(eee)sswwwwnr (
and apply the IFT to F. noting that the hypotheses hold at Q 0 and 0. We thus have some neighbourhoods E C NxD and O c D(D-11/ such that for any E E there exists S2 (z) E O such that
aF (c, 22, (E))
DDs (x ||Hess (T)(
0=
skew [((0)--Ved
11
0E,
1
@VQJ
/1
dEn
Lg (a b) + LIVD
NN
Here, the bound on the Hessian comes by hypothesis, while the second term follows from the fact that the skew operator and multiplication by an orthogonal matrix is norm-preett preeerreeeeenn. combined with following expression for the Euclidean directional cross-derivative:
d. VOJIh
N ((R,(Q)Qh)y + (Q)()²
where (Q) diag J" ( ym) ...,, J" J"(aEY")) and have 1 VoJ Lo(o+b)+VDL, required. Aggregating across all n, have ( & La(a+t)+LvvV By hypothesis, the additive symmetry between x,, and en is sufficient for this bound to hold for any sufficiently small perturbation.
RvN
As result, we have
(IQx(e((x, +ex) Q-(0)x,
< [IQx(e) - (0) x + [e@=(e)enl1
< ([,(()) - (0)
(Lz(a b) + LIVD)ab +
1
as required.
Below, we provide a complete overview of the assumptions we make for Theorem
Assumption Summary. We assume that the representation distribution (i.e. the pushforward of ((x) by any encoder fo) has full-rank covariance, and that the support of the distribution has diameter bounded by A. The eigenvalues of the covariance matrices assumed be bounded below by X.
26

---

Published as a conference paper at ICLR 2026
Furthermore, assume that the identifiability up Huiearr of (P. O Lo, F) (where F F + fo) is satisfied for linear maps with singular values bounded between XA and A Finally, we assume that the contrast function used for ICA is Lipschitz (with constant L1 and has Lipschitz derivative (with constant L2), and that ICA converges such that the Riemannian Hessian at the optimum has eigenvalues bounded below by jr 0.
With these lemmata, Theorem 2 in the main text follows easily.
Theorem. Suppose (P, θ La. F) is e-nearly identifiable up Hainear for F : + fo- Then, new model M' which applies whitening and contrast function- function the latent representations given by So is é' -neaa-iiennifaalle up to Has for = K + K'e² where K and K' are constants free of e that depends on the maximum diameter of the latent space A. the spectra of the covariance matrix of the representations, and the properties of the ICA contrast function.
-aased independent components analysis to
Proof. Let fe and for be two optimal encoders. By e-near identfiability up Huncar, there exists A such that ess supp fozz) (z) S e. A has positive determinant, Lemma A.66 with the usual whitening applied to both encoders yields Ce-near-identiffabliyy up to Hrigid for C [1+
A 1/122
If not, the sign of single latent can be flipped, which flips the sign of a
single column of A, allowing the application of Lemma A.6 Denote the whitened encoders by fo and for (a:). Lemma A.8 then applies with a = A and b = Cc, where A is the maximum diameter of any latent space, yielding a bound C² L=(A+Co++/Dta ACe. Taking the limit as N + oo and denoting f&AA (z) and facA (z) the outputs of the encoders with whitening and ICA applied, we have ess supp(a) |1 (x) Pf pICA
PP(C(()) (z)Il < Ke + K'c² for K = /DD and K' = 12CPa and the indeterminacy P E H. arises by the fact that any signed permutation of the latents is a maximum of the ICA objective, as required.
A.3.6 STRUCTURAL NEAR-IDENTIFIABIIA VIA ICA
Here, we are able to give a short proof of Theorem by leveraging identical arguments to the proof of Theorem 2
Assumption Summary. We assume that the representation distribution (i.e. the pushforward of P (a) by any encoder fe) has full-rank covariance, and that the support of the distribution has diameter bounded by A. The eigenvalues of the covariance matrices are assumed to be bounded below by Furthermore, assume that the identifiability up Husear of (P. O, Lo, F) (where F + fo) is satisfied for linear maps with singular values bounded between A and A Finally, we assume that the contrast function used for ICA is C² with Lipschitz first (with constant L1) and second (with constant I.2) derivatives, and that ICA converges such that the Riemannian Hessian at the optimum has eigenvalues bounded below by μ 0. Finally, the end-to-end model 9s Se must reconstruct its inputs perfectly at the optimum and the true data-generating process must be locally (1+ + Lipishit,, smooth, and injective (and therefore diffeomorphic onto its image), with the Mata-generating factors being white (i.e. zero mean, unit variance). being non- non-Gaussian and having independent components. Finally, we assume that Q is sufficiently rich so that E M. i.e. the model can approximate the ground-truth data-generating structure.
Theorem. Let P()) be some multivariate distribution with independent non- non-Gaussian components with zero mean and unit variance, and consider data P(x) generated by pushforward through a smooth diffeomorphism g such that 9 is locally
(1 + b)bii--ipsshiitz Let M be a model with a sufficiently rich parameter space O. Let F 0 + fo. G 0 - go and H O ++ 90 fo- Then, (P, Le, H) structurally identifies the identity function in the limit (i.e. attains perfect reconstruction), we have that (P. θ, Lo, F) e-nearly identifies the structure g- up to Hrigid and furthermore that a new model M' which applies whitening and independent components analysis to the latent representations given by fo é' é'-neally identifies the structure g_ up to Ha where and e' depend on 8 and Lipschitz bounds on go. and depends additionally on the spectrum of the covariance matrix of the representations and the properties of the ICA contrast function employed.
27

---

Published as a conference paper at ICLR 2026
Proof. Take 8 = max L} as the maximum of the two local bi-Lipschitz constants of the data- generating process and the bound on the local bi-Lipschitz constant of the decoders in the model class. Theorem yields 8' sicar-identifiabiiity in the limit up to Hrigid for 8' = cD 26 + A²A. With the inverse of the true generaainn map taking the place of one of the encoders, the same argument from the proof of Theorem yields the first claim, namely 8' near structural identifiability of g- up to Hrigid-
The rest of the proof follows similarly to the proof of Theorem 2 in Appendix A.3.5 with the inverse of the true data-generating map taking the place of one of the encoders. In particular, let be an optimum and consider fo and go. Without loss of generality, assume that fe(x) noting that the encoder fa and decoder ge can be translated arbitrarily without altering any of our hypotheses. The same argument then applies, yielding the result. The precise constant the same, with = K8' + K28 for K and K' as defined in the statement of Theorem
g (x).
A.4 DYNAMICAL ISOMETRY AND BI-LIPSCHITZNESS
Proposition. Suppose f D - IR is once differentiable and satisfies dynamical isometry, in the sense that the singular values x of the Jacobian J satisfy {>, 1| f i = .....,..ii,... D. N) for some > c 2 O. Then, f is locally L- Aabi-Lpscchit,,rl for L 1+c
1-c'
Proof. Let x. y E DR Then the mean value theorem yields the bound (l](c) - s(v)l < || 11| vll $ e) x y|l- By the same argument, (1 +} - yll <II f(v)l., and taking T. = (a bound on the condition number of J) yields suitable bi-Lipschitz constant.
Remark. Many popular regularization techniques spanning architectures and tasks optimize im- plicitly or explicitly for dynamical isometry, with the level of evidence ranging from theoretical to empirical. For example, weight decay has been shown theoretically and empirically to do so (Zhang al.) 2079), normalization techniques in generative adversarial networks optimize for it directly Karras et al.) 2020;; Miyato et al. 2018), residual layers yield this property (Bachlechner et al. 2020), and specialized techniques have been developed to yield it at initialization (Xiao et al. 2018).
A.S STRUCTURAL IDENTIFIABILITY IMPLIES STATISTICAL IDENTIFIABILITY
Below, we prove that statistical (near- (ieat-iidentiffabblity is implied by structural (near-)identiffablityy
Theorem. Suppose (P, O La. F) 8-nearly identifies the structure u up to H. for 0 + So. Then, (P, O La, F) is e-nearly identifiable up to H for some e E IR provided that H. is bounded by C R as in the sense of an operator norm.
Proof. Let 0,0' E S be solutions to the minimization problem mingce E[Co] where M = (Le 0 E Q). By structural identifiability, we have that there exist and h' relating and 6' respectively to u. More concretely, we have:
| o fe u/| 5 || so ull S 8
Take h- = -1 a h', where the inverse and composition are well-defined because H. is group of functions mapping IR'' to itself. h" can be understood as mapping for as close to fo as possible under the available assumptions. Then, have
llfe h" for = ||A h fo h- o' ler
= ||/1 h fo - h- ou + h ou - o hi' o fo
< ||b- llop (1 o fo - ul|er + nN o for - ull
< 2C8
28

---

Published as a conference paper at ICLR 2026
where the third line follows by the triangle inequality, and the final line by structural identifiability of u and boundedness of H. yielding the result.
The constant C' arises because solutions might exist different scale from the Vata-generating process u (and therefore from each other). Partly as a result of this fact, this theorem likely is not useful for certain identifiability classes such as Hineer, which even if bounded by assumption can likely yield more fruitful identifiability results via a direct approach. On the other hand, for identifiability classes like Hrigid. the bound is in some sense tight.
Finally, we emphasize that the result is largely agnostic to the choice of reasonable norms, although have in mind the usual operator norm and an L. norm for 1 S p S oo, taken with respect the data distribution P(r).
Remark. Taking 8 shows that structural identifiability implies statistical identifiability in the limit.
A.6 CONFORMAL, MAPS AS NEAR-ISOMETRIES
In this section we make rigorous the arguments outlined Section 3.3.1 First, we outline the mollification argument which allows us take derivatives in L without hassle.
Mollification Let 0, be the isotropic zero-mean Gaussian density with variance σ², For any image represented by function F, note that F2 = F oga is a "smooth" version of that image without hard edges. Accordingly, mollifying all images by the same Gaussian means that taking the Gateaux derivative of image-valued functions becomes possible in L², and only introduces multiplicative constant dependent on the variance o² which we would like to ignore. To see that this is possible, consider the data-generating process for square articulating along the z-axis according to coordinate p outlined Section 3.3.1
(()) = [(x,, v) + 1ps-lss E ((-(1,,
Denote fo(p) = f(p) * 022. Then, convolving with the distributional derivative gives
(2( M (ő(x p-
6(x -ppp-p--0 -(x))++r)
2r
(p)
+ O(cxp(-o2))
Vioo
Pick any two distinct latents po and P1- For any smoothed manifold, we have that their geodesic +0(epp(-o2)) Ip Pol. The distance between any two points can
distance is given by VTo then be "renormalized" against this distance by dividing through it, ensuring that as - what left is constant. This then implies that what we actually computing in the subsequent sections is not a metric inherited from the L norm at all, but rather defines a whole new geodesic distance on the limiting manifold of unsmoothed images. For notational clarity, ignore mollification in the rest of this section and return to a heuristic argument for the next sections. We direct the interested reader Grimes (003), Section 2.6 for fully rigorous treatment.
2r
Two-dimensional manifold Now, we fully characterize the 2-dimensional manifold described in the main text. Let 2 = ((,, r) | a < p < b, Ro < < R} denote the manifold of latent variables of the position the square and its half-side length. Each point z E Z can be identified with an image:
f(p [(x, 3) + pa--pls.iiss - (²([[1,
Directional derivatives The derivative with respect to p remains the same as in the main text, and the derivative with respect follows similarly:
29

---

Published as a conference paper at ICLR 2026
|0, 2r [0,f(p,r)])== 8r
However, checking that f conformal map also demands that 0pf and f are orthogonal in L To see this, note that finite difference approximation to 0,f are the (negative) left and (positive) right edges of the square, while the finite difference approximation of the square. Therefore, the top edges contribute nothing to the inner product (0pf. 0, while the contributions of the left and right edges cancel. Thus, the Riemannian metric can be written
fr are all (positive) edges
0) G(p,r) 2r ( 4
G(p'rr) =r'
where the reparameterization p' = p/4 and r' = 2r allows us to recover the isotropy.
Near-isometry remains to show how f can be viewed as locally bi-Lipschitz. To see this, note that the we have the following global bound the differential:
2n u|l < |I 2R||u|]
for u in the tangent space at any point along Z. Accordingly, we have the following bound on the geodesic distance:
2R [2 - zoll S dgeo ff(z),,(zz) S /2R zoll
where dgeo is the geodesic distance along the image manifold. As a result (assuming w.l.o.g. that Ro 1), f is locally / R/ Roo- Furthermore, for any f-' from the image manifold to convex subset of TR2 which is also /R/ Ro-bi--Lipschitz, f f- is globally R/ Rogbi-Lipschitzz (with respect the l norm both spaces, where the constant follows by convexity) and is therefore
near-isometry with constant (R/Ro 1)A where A /4(R Ro)² (a 6)3 simply the diameter of Z.
A.7
EXPERIMENTAL DETAILS
A.7.1 WARMUP EXPERIMENT: MNIST
We conducted experiments on the MNIST dataset (LeCun et al. 2010) to validate our theoretical predictions regarding the identifiability of latent representations as function of the bi-Lipschitz constant of the decoder.
Training
We trained pairs of orthogonal LeakyReLU autoencoders with the following architecture:
Encoder: 784
784
18
784
784
I
Decoder:
R2
R
IR 784
R
T 784
All linear layers used orthogonal weight parametrization (no bias terms). LeakyReLU activations with leak constant a E {0.0, 0.25, 0.5, 0.750.7.00. 0.9, 1.0] were applied at all intermediate layers. The latent dimension was D 2. All models were fit using the Adam optimizer (Kingma & Ba 2015) with learning rate 7 = 5 10 - for up to 2000 epochs, minimizing the mean squared reconstruction error. Early stopping was applied with
Figure A.3: The distribution of sample-level bi-Lipschitz constant estimates B(2) tightens around as -
30

---

Published as a conference paper at ICLR 2026
patience of 50 epochs and minimum improvement threshold of 10 Gradients were clipped to unit norm for stability. We repeated each configuration with 10 random seeds for robust- ness, yielding 6 x 10 = 60 experimental runs. We filtered experimental runs exclude poorly converged autoencoders. Specifically, we removed runs where the reconstruction error of either autoencoder exceeded the 95th percentile observed at the reference leak value 0.9. This removed 1/60 runs.
Results For a given autoencoder, we compute the represen- tations of random subset of 1000 samples. For the decoder g. the bi-Liischitz constant a given latent point is given as (2) = ( 1/1la.1//l////// (z)]l(
where the maximum is taken over 10 randomly sampled unit vectors. We plot the distribution of (z)) as a function of the 1, the distribution typically well-concentrated around mean not much larger than one, with relatively small number of outliers. Figure 2 (in the main text) plots the samplewise estimate of L Ep(:) [=) 1] versus identifiability. Although more robust to outliers, this is not a formal bound because Theorem relies on global L, i.e.
leak constant a in Figure A.33 As
Figure A.4: Controlling the bi- Lipschitz constant leads to im- proved identifiability (reduced l error). The proportionality does not appear to differ whether the max mean bi- bi-Lipschitz constant is esti- mated.
ma (2) 1]. For completeness. we plot using the maximum in Figure A.4 with the maximum taken over all 1000 samples. Both plots include fitted curves of the form lg error a + L + b to the identifiability measurements, consistent with the theorem.
As a - 1, the estimated bi- bipsihhchit constants do indeed shrink toward as expected. This suggests validity of the exper- imental testbed, but not Theorem [ itself.. However, as + 1, we see that identifiability does indeed improve significantly (l2 error + 0) as predicted by the Theorem. Notably, this despite the fact that perfect reconstruction does not hold (Figure A.5)
We also completed the same experiment with a model per digit. This allows us to assess whether class-level indeter- minacies play role in identifiability in this problem setting like in (Nielsen et al. 2025). All hyperparameters and setup remains the same, except we fit three seeds per digit for a total of 10 digits x 6 leak values
0.00
Figure A.5: Reconstruction error improves as the bi-Lipschitz con- stant grows (leak - 1). Notably, poor reconstruction does not inhibit identifiability.
seeds 180 runs. Filtering using the same rule for reconstruction removed 5/180 runs. Re- sults are consistent for these per-digit autoencoders as well, suggesting that for this setting of hyperparameters, class-level indeterminacies do not play substantial role in the level of empirical identifiability (Figure A.6
A.7.2 MEASURING NEAR-IDENTIRABILITY
We describe here the setup for the alignment experiments reported in Table 1 All reported statistics are in-sample, with representations taken from the same data on which the models were trained. For each model, we extract the following hidden states:
GPT-class models (Pythia-160M)) penultimate-layer hidden states.
MAE models: latent representations obtained by averaging patch embeddings.
Supervised models (ResNet-18): penultimate-layer representations.
We apply the following representation alignment techniques:
representation dimensions were matched using the Hungarian algorithm to resolve signed permutation indeterminacies.
Permutation:
31

---

Published as a conference paper at ICLR 2026
12 bus Barts
1121
135
124
Mooe
Lo Stant degss
Bror dherr Liges
128 Meees
Srams Ligss
1
200
apter
Figure A.6: In digit-specific models, controlling the bi-Lipschitz constant L leads improved identifiability (reduced lg error) with pattern similar to the full-dataset models.
Rigid: estimated via the Procrustes algorithm with a global scaling constant (rigid similarity transform).
Linear: estimated via least-squares regression with no additional regularization or constraints.
estimated using FastICA (scikit-learn implementation) applied to the full representation dimension with all components retained. The Hungarian algorithm resolves the remaining signed permutation in latent space.
ICA:
Alignment quality is reported as the mean per-example f2 error, normalized by the latent diameter, defined as the maximum pairwise distance among representations. For ICA, efficiency is reported as the proportion of error reduction relative to the supervised rigid transform:
Permutation ICA
ICA efficiency
Permutation Rigid
A.7.3 DISENTANGLEMENT EXPERIMENTS
We reproduce only the autoencoder (AE) results, enough to validate that we obtain similar perfor- mance, and apply ICA to these models. Specifically, each hyperparameter setting is re-fit 3 times, and training step where the average performance is the best is selected (according modularity, after filtering for reconstruction as in Hsu et al. )2023)).
As in the original experiments in Hsu al. (023), the latent spaces overparameterized with the number of latents equal to twice the number of ground-truth sources nı = 2n,, As a result, the latent space rank deficient. To avoid the introduction of additional hyperparameters for pruning inactive latents which could bias the performance of our approach (via e.g. whitening with dimensionality reduction), we fit the FastICA model without whitening. Inspection that inactive latents are obvious (corresponding singular values are very near zero) and the remaining components of the Jacobian have similar scale (likely due to high weight decay), suggesting that full-rank whitening would have minimal impact here except to drive up the noise from inactive latents. As a result, we perform no whitening and preserve all latent dimensions.
the decoder Jacobian reveals
Table 5 is an augmented version of Table 2 with standard errors computed across the 3 models with the best hyperparameter setting for each model. Results for all models other than AE (reproduction) and AE + ICA are quoted from Hsu et al. (202)), which averaged over 5 seeds instead of 3.
A.7.4 OPENPHENOM EXPERIMENTS
The whitening and FastICA Hyvärinen & Oja (2000) algorithms are from the scikit- -lain package Pedregosa et al. 2011). I. These models trained at the patch level on the patches from the first channel of images in Rxix3-core Kraus et al. (2025), where patches are specifically subsampled from the top left-hand corner. To ensure computational feasibility, a single patch is sampled from
32

---

Published as a conference paper at ICLR 2026
each image, yielding approximately 222K patches. Both models use the default hyperparameters (all principal components are kept and the contrast function is log cosh).
Gradient boosting models are trained using LightGBM Ke et al. (2017) (hyperparameters in Table 6). Models are trained to predict whether the image patch embedding is from an image which is perturbed or Models evaluated using under the receiver operator characteristic curve.
separate model is trained on each plate, which then evaluated on all remaining plates. Below, we report augmented version of Table with the standard error of the mean computed across folds. When more than one independent experiment (corresponding to guide) is available for a given gene, the standard error is computed across all folds from all guides.
Menn AURCC s.c. (T)
Sparsity s.e. (f more sparse)
Gone
Base
PCA
PCA + ICA
PCA + Rand
Rase
PCA
PCA
ICAA
PCA
Rand
CYPTIBL (1)
0,063 0.008
0,692
0.008
0.709 1: 0.000
0.679 1: 0.010
0.184 1: 0.006
0.204
0.007
0.237
0.006
0.188
0.007
0.0003
HCK (1)
0.670 0.012
0.693
0.007
0.711
0.005
0.008
0.007
0.156
0.004
0.208
0.004
0.241
0.000
0.166
0.005
MTTOR (6)
0.663 0.003
0.800
0.000
0.705
+ 0.003
0.670 + 0.003
0.166
+ 0.002
0.201
0.003
0.233
0.003
0.186
0.002
PLK1 (6)
0,803 + 0.002
0.811
0.002
0.815
+ 0.002
0.792 + 0.002
0.251
+ 0.003
0.307 0.004
0.305 + 0.003
0.262 0.003
(RC(1)
0,000
0.706 0.004
0.676
0.214
0.008
0.005
In Table 7
we report an augmented version of Table 4.4 with the concentration scores, assessing
sensitivity to the parameter k.
33

---

Published as a conference paper at ICLR 2026
Table 5: Full results across datasets.
Shapes3D
model
InfoM
InfoE
InfoC 1
AE
0.41 + 0.03
0.98 0.01
0.28 0.01
3-VAE
0.59 = 0.02
0.99 + 0.02
0.49 + 0.03
3-TCVAE
0.61 = 0.03
0.82 - 0.02
0.62 0.02
BioAE
0.56 = 0.02
0.98 + 0.01
0.44 = 0.02
AE (reproduction)
0.36 = 0.05
1.00 0.00
0.18 0.06
AE ICA
0.78 = 0.02
1.00 = 0.00
0.42 0.09
Discrete latent models
VQ-VAE
0.72 = 0.03
0.97 0.02
0.47 - 0.03
VQ-VAE w/ weight decay
0.80 = 0.01
0.99 = 0.01
0.46 0.02
QLAE
0.95 = 0.02
0.99 + 0.01
0.55 - 0.02
MPI3D
model
InfoM
InfoE
InfoC t
AE
0.37 + 0.04
0.72 + 0.03
0.36 0.03
3-VAE
0.03
0.71 0.03
0.51 = 0.03
3-TCVAE
0.51
0.60 0.04
0.00
BioAE
0.45
0.66
0.36 1 0.03
AE (reproduction)
0.42 + 0.05
0.66 + 0.28
0.31 0.12
AE ICA
0.44 + 0.12
0.66 + 0.28
0.31 0.14
Discrete latent models
VQ-VAE
0.43 + 0.06
0.57 + 0.04
0.22 - 0.04
VQ- VQ-VAE w/ weight decay
0.50 = 0.04
0.81 0.04
0.41 0.04
QLAE
0.61 + 0.04
0.63 + 0.05
0.51 + 0.03
Falcor3D
model
InfoM f
InfoE
InfoC 1
AE
0.39 = 0.03
0.74 0.03
0,20 0.03
3-VAE
0.71 + 0.05
0.73 04
0.70 + 0.03
3-TCVAE
0.66 + 0.02
0.74 0.04
0.71 0.04
BioAE
0.54 = 0.05
0.73 0.04
0.31 = 0.01
AE (reproduction)
0.37 = 0.14
0.75 0.01
0.21 = 0.06
AE+ICA ICA
0.68 = 0.09
0.75 0.01
0.37 0.32
Discrete latent models
VQ-VAE
0.61 + 0.04
0.83 + 0.05
0.42 + 0.02
VQ-VAE w/ weight decay
0.74 = 0.02
0.86 = 0.04
0.40 0.03
QLAE
0.71 = 0.03
0.77 = 0.02
0.44 = 0.02
Isaac3D
model
InfoM f
InfoE
InfoC 1
AE
0.42 + 0.04
0.80 + 0.02
0.21 + 0.05
3-VAE
0.60 + 0.03
0.80 + 0.02
0.51 0.03
8-TCVAE
0.54 0.02
0.70 + 0.02
0.46 1 0.03
BioAE
0.63 0.03
0.65 = 0.03
0.33 0.04
0.04
AE (reproduction)
0.41 = 0.11
0.80 - 0.05
0.20 - 0.07
AE ICA
0.64 + 0.17
0.80 0.05
0.34 0.05
Discrete latent models
VQ-VAE
0.57 + 0.04
0.87 + 0.05
0.45 0.04
VQ-VAE w/ weight decay
0.73 + 0.03
0.81 + 0.03
0.44 - 0.04
QLAE
0.78 + 0.03
0.97 + 0.03
0.49 0.03
34

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
is_unbalance
True
learning rate
0.05
leaves
feature fraction
0.6
reg reg.alpha reg_lambda
5.0
1.0
min.gain__o sppit
0.8
min_data_in_lea leaf
30
Table 6: LightGBM hyperparameters used in all experiments.
Table 7: Sensitivity of concentration k.
Concentration (t)
Model
k = 25%
k: = 33%
k = 50%
Base (none)
0.163
0.134
0.133
PCA
0.332
0.307
0.314
PCA + ICA
0.386
0.372
0.334
PCA + RandRot
0.287
0.280
0.235
35


Reviews:
[{"review_id": "y1I1VFdfLo", "reviewer": "Reviewer_hzYt", "summary": "A quite detailed analysis around self supervised learning and representation stability. There are technical contributions that expand the applicability of identifiability theory to a broader class of models. While the experiments remain quite limited and synthetic, there is a lack of more challenging data with known ground truth hence making it a challenge for the authors.\n\n** Strength **\n- The paper introduces quite a few theoretical contributions that take a step towards better understanding self supervised learning which is one of the most prominent solution today\n- The writing and technical quality is above acceptance level with clear novelty and insights\n- While the scope is not as wide as it could be, the empirical section is quite detailed and feel quite reproducible\n\n** Weakness **\n- Numerous formatting and typos throughout the manuscript that makes the paper feels a bit rushed in some parts, e.g., `multiviarate`, `identfiability`, `c̃itepjumpcp` and so on\n- The paper is quite dense mathematically and the appendix doesn't help much as it doesn't provide more introductory materials to help unfamiliar readers get started. While this is not a major weakness, it would be great to bridge a bit more the paper's content with the practitioners unfamiliar with those concepts\n- It would also be great to better connect the theory and results with practical settings for SOTA models", "questions": "Please see summary", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Please see summary", "weaknesses": "Please see summary", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eoHKZf3qUl", "reviewer": "Reviewer_tnLY", "summary": "In this work, the authors provide a novel identifiability result which characterizes the similarity in parameter space for a broad family of models up to rigid transformations in terms of a bound which is a function of the bi-Lipschitz constant of a model. The authors then show a relationship between this notion of a identifiability and the idea of achieving identifiability w.r.t. a latent variable model, which they refer to as recoverability. Several experiments are run across large scale models and various datasets highlighting that applying ICA on learned representations yields disentangled representations.", "questions": "* Can the authors clarify the positioning/motivation of their results within the identifiability community?\n\n* Do the authors believe it would be possible to conduct experiments on toy data which aim to more rigorously test the theoretical statements?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I believe this work makes contributions which are potentially of substantial interest to the identifiability community.\n\n*  Namely, providing clarity on the the relationship between identifiability w.r.t. a latent variable model and models being identifiability w.r.t. each other is an important nuance to explore. \n\n* Furthermore, Theorem 1 provides very general result characterizing model identifiability which I believe is of interest even if the result relies on a Lipschitz constant which may be difficult to measure in practice.\n\n* The authors make a notable effort to conduct experiments which highlight representation similarity and disentanglement in diverse settings and across different large scale models.", "weaknesses": "* While I find the author's contributions noteworthy, I found it very difficult to parse the contributions by reading the abstract and introduction of the paper. In the introduction, the authors introduce several purported issues with existing identifiability results and then present their contributions as solving these issues. I found this relationship between these issues and the authors contributions difficult to connect and ultimately obfuscating of the authors main contributions. I would suggest the authors rework the abstract and intro to better clarify their contribution.\n\n\n* While I understand that the authors wish to explore the implications of their results on image data, the experiments feel a bit disconnected from the theory. I believe a simple toy experiment validating the authors theory in a setting in which the bi-lipschitz constant can be measured would be of value.\n\n\n* Additionally, I would appreciate if the authors could discuss the relationship between their results and prior results [1] which have results related to epsilon near identifiability.\n\n\n**References**\n\n[1] Nielsen et. al 2025, When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wiccjm2TO8", "reviewer": "Reviewer_QUzt", "summary": "The authors study $\\epsilon$-near identifiability and recoverability in self-supervised models. They define near-identifiability of a model as satisfying an identifiability criteria up to $\\epsilon$. They then show that, for a model with identifiable output, the internal representations are near-identifiable up to rigid transformations and up to $\\epsilon$ a function of the Lipschitz constants of the intermediate layers. They then show that ICA post-processing can be used to preserve near-identifiability, which results in near identifiability up to permutation and sign flips.", "questions": "- Are any of the results really specific to self-supervised models, or do they have some special qualities (e.g., in the loss)? Why is the emphasis placed on this setting in the framing?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Existing works in identifiability usually treat functions $f$ as fully black-box, but in practice are parametrised as neural networks. Arguably the black-box perspective has been taken to its limit and there are not many substantially new developments in identifiability here. If I understand correctly, what this paper does is leverage the neural network (or compositional) structure, noticing that outputs are usually identifiable (e.g., assuming optimization is well-posed), and leveraging that structure to obtain a result on near identifiability of intermediate layers, which is how many representations are extracted in practice. This is a really neat, original idea that should have significant impact for the representation learning community.", "weaknesses": "- The recoverability section, unless I'm missing something, seems to just be talking about how well-specification is usually assumed in these papers, but it is nice to have it spelled out with examples specific to the theory here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R0I7fERVT5", "reviewer": "Reviewer_p3of", "summary": "This paper provides relaxed conditions for representational identifiability in self-supervised learning, extending previous results beyond the (pen)ultimate layer. For their weaker results, the authors proposed a further preprocessing step by linear ICA to remove the linear indeterminacy. The extension of identifiability to the approximate regime is an important contribution, especially for practial applicability of identifiability.\n\n**The paper seems to be very interesting, and I believe it has great potential. However, I have concerns, especially about the imprecise use of he word \"identifiability.\"** Indeed, what the authors call \"recoverability\" is what most of the literature calls identifiability. \n\nAlthough I think this paper has the potential to be a spotlight/oral, given my current concerns, I chose a conservative score. **If the authors can address my concerns (or correct me if I have misunderstood any of their claims), then I will raise my score.**", "questions": "- L138: why do you use the $L^{\\infty}$ norm?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The paper aims to relax identifiability conditions, thus, increasing the practical relevance of the field of identifiability. Thus, it is very timely and useful.\n- The extension to approximate identifiability is definitely underexplored (though not *entirely*, novel, see, e.g., [1])\n- The theory is contextualized for practically relevant model families (MAEs/GPTs)\n- The empirical evaluation is detailed (though a bit unclear, see weaknesses)\n\t- I especially liked the PCA+rand control in Table 3\n- A.1.3-4 are nice discussions on architecture-specific cases of the theory\n- A.1.5 is a nice discussion on ICA in latent space\n\n- [ 1 ] Reizinger, Patrik, Szilvia Ujváry, Anna Mészáros, Anna Kerekes, Wieland Brendel, and Ferenc Huszár. \"Position: Understanding LLMs Requires More Than Statistical Generalization.\" In Forty-first International Conference on Machine Learning.", "weaknesses": "My main concerns are about the clarity/phrasing, and NOT the technical contributions.\n\n### Major points\n- **My biggest concerns is the improper use of identifiability/recoverability**: when the authors refer to identifiability, they do not refer to the commonly used term in the ICA literature (which is formulated in terms of the ground-truth data generating process). The authors introduce (superfluously) recoverability for the (as far as I can tell) same concept. When they talk about identifiability, they refer to the definition used by Roeder et al., 2020. I strongly suggest to correct the use of the term to avoid confusion with the majority of the identifiability literature. Note that this does not devalue any contributions, only makes their presentation presumably confusing. For a comparison between the two identifiability notions, see, e.g., [ 1 ] (where the authors call it relative - Roeder - vs absolute - Hyvarinen - identifiability)\n- Thm. 1.:\n\t- the theorem would benefit from a more-detailed explanation. In my opinion, the current intuition section is more like a sketch. At least I couldn't figure out the intuition.\n\t- E.g., why are all $F, G, H$ needed?\n\t- What is the role of bi-Lipschitzness?\n\t- Small remark: there is a identifiability result for next-token prediction, see, http://arxiv.org/abs/2503.08980\n- Def. 2: rephrase the intuition with the mainstream identifiability definition (the connection between the two concepts in A.3 is nice)\n- Table 1: it is missing context, it is unclear from the caption what the metrics mean, what was measured, etc. Please make the caption self-contained\n- The proof of the Theorem in A.3 would benefit from more steps and inline explanations of what is happening between the steps\n\n### Minor points\n- L045: \"Second, model-specific identifiability results fail to predict the empirical alignment seen across different architectures and modalities (Huh et al., 2024):\" \n\t- This is unclear how it would hold for the most widely-used identifiability definition, as assumptions generally require assumptions that do not constrain the modality or the architecture. Please elaborate\n- L053: In this paper, we argue that many current identifiability results in representation learning are overloaded with recoverability guarantees of some latent component of the data-generating process,\n\t- I don't understand this sentence\n- L143: here, the \"All or None\" paper by Marconate et al. seems to be very relevant, please discuss it. Also, [ 1 ] proposes a simialr notion, termed $\\varepsilon-$non-identifiability\n- L277 - Example: I like the example, though it's a bit too involved and I am missing the intuition for it. If you could add such an explanation (preferably with a figure), that'd help the reader\n- L363: the citation showing Anonymus et al is a typo, I presume?\n- Bottom of page 7: there seems to be some typo/latex error\n- Maybe formalize all your assumptions into an assumption latex environment (it's OK to have this in the appendix), such that readers can refer to it\n\n\n- [ 1 ] Reizinger, Patrik, Szilvia Ujváry, Anna Mészáros, Anna Kerekes, Wieland Brendel, and Ferenc Huszár. \"Position: Understanding LLMs Requires More Than Statistical Generalization.\" In Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)