When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective
Authors: Beatrix Miranda Ginn Nielsen
Emanuele Marconato
Andrea Dittadi
Luigi Gresele
Venue: NeurIPS 2025 poster ()
Keywords: Identifiability
Representational Similarity
Distributional Distance
Robustness
Representation Learning
Primary area: probabilistic_methods
TL;DR: Small KL-divergence fails to ensure similarity of representations; we propose a distance which does and demonstrate it empirically.

Abstract: When and why representations learned by different deep neural networks are similar is an active research topic. 
    We choose to address these questions from the perspective of identifiability theory, which suggests that a measure of representational similarity should be invariant to transformations that leave the model distribution unchanged.
    Focusing on a model family which includes several popular pre-training approaches, e.g., autoregressive language models, we explore when models which generate distributions that are close have similar representations. We prove that a small Kullback--Leibler divergence between the model distributions does not guarantee that the corresponding representations are similar. This has the important corollary that models with near-maximum data likelihood can still learn dissimilar representations---a phenomenon mirrored in our experiments with models trained on CIFAR-10.
    We then define a distributional distance for which closeness implies representational similarity, and in synthetic experiments, we find that wider networks learn distributions which are closer with respect to our distance and have more similar representations. Our results thus clarify the link between closeness in distribution and representational similarity.

When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective
Beatrix M. G. Nielsen 1
Emanuele Marconato
Andrea Dittadi 1,3,4,5
1.3,4
Luigi Gresele
'Technical University of Denmark 2 University of Trento, Italy HHelmholtz AI, Munich
"Technical University of Munich MMunich Center for Machine Learning (MCML) University of Copenhagen
Abstract
When and why representations learned by different deep neural networks are similar is an active research topic. We choose to address these questions from the perspective of identifiability theory, which suggests that measure of repre- sentational similarity should be invariant to transformations that leave the model distribution unchanged. Focusing on model family which includes several popular pre-training approaches, e.g.. autoregressive language models, we explore when models which generate distributions that are close have similar representations. We prove that small Bellbacck-Libbee divergence between the model distributions does not guarantee that the corresponding representations are similar. This has the important corollary that models with near-maximum data likelihood can still learn dissimilar representations phenomenon mirrored in our experiments with models trained on CIFAR-10. We then define a distributional distance for which closeness implies representational similarity, and in synthetic experiments, that wider networks learn distributions which are closer with respect to our distance and have more similar representations. Our results thus clarify the link between closeness in distribution and representational similarity.
find
1 Introduction
How to compare and relate the internal representations learned by different deep neural networks is a long-standing and active research topic [3. 30, 33, 38, 45, 49, 65]. Understanding when and why var- ious kinds of representational similarity emerge has implications for model stitching [12, 16, 36, 41, 46], knowledge-distilatton [53, 58, 73, 74] and interpretability for concept-bass basedd and neuro-symbolic models [5, 15, 42], see [61] for review. A theory of w when similarity occurs will chiefly depend on how similarity measured. As argued by Bansal et al. [3], similarity measure "should be invariant to operations that do not modify the 'quality' of the representation, but it is not always clear what these operations are". Indeed, this depends on how one defines the 'quality' of representation, and previous works have debated the pros and cons of different choices [3, 14, 30, 31, 45, 49, 65, 67].
We choose address these questions from the perspective of identifiability theory. In identifiability of probabilistic models, representations are called equivalent if they result in equal likelihoods, and
"Correspondence bmgi dtu.dk heat@itu.dk. i Joint last authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Pe
/ L
Fanninnros f())
uell
Pra
dic.
Pre
[rr xi)
Pr
[1
fi(e)
f1(0)
Bar St
drg
Theorem 4.7
Figure I: When closeness in distribution does and does not imply representational similarity. On the left, we show two distributions gr Pr'.'' E Pe which are closer than w.r.t. the distance dÊ (Definition 4.4), illustrated by the shaded blue ball. We use [ff.g]] E 0/ ~u denote the identifiability class (Footnote 3) of ~ug -ddeniifialle model with embedding and tnembedding g (Section 2). Theorem 4.7 implies that the identifiability classes [(f, g)] and [ gr',g" g')], within the shaded orange arca, will have similar representations i.c., their dissimilarity under de. (Definition 4.6) is bounded above by 2M We also consider third distribution Pi, E Pe which, while e-close in dxL. (in magenta), falls outside the blue region and has representations that are very dissimilar from those of [(f, g)] and [(f', .g)). as described by our Theorem 3.1. On the right, we plot the three model embeddings: Taking as reference, we find the best linear the points according to the residual error (brighter colors denote larger errors). The embeddings f are ncarly a linear transformation of while shows substantial deviation -iviatiio-vviibllyy farther from being linearly related.
and f, and then color each of
all models with equivalent representations together form an identifiability class [27]. Therefore, to preserve the "quality' of representations, our notion of representational similarity needs be invariant to precisely those transformations which leave the model likelihood unchanged. We focus on a model family including many popular pre-training approaches, e.g., autoregressive language models [8, 48], contrastive predictive coding [47], as well as standard supervised classifiers. Identifiability results for this model class show that, under a diversity condition, equal- likelihood models yield representations which are equal up to certain invertible linear transformations [28, 34, 51, 52].
As first step toward theory of representational similarity, we ask: For what distributional and representational distances is it true that models whose output distributions are close have simi- lar internal representations? Addressing this requires going beyond classical identifiability, which assumes exact likelihood equality. We do so by proving in Theorem 3.1 that a small Kullback-Leibler (KL) divergence does not guarantee that the corresponding representations are similar-i.e. close to the identifiability class of our model family. As an important corollary, two models which are arbitrarily close to maximizing the data likelihood can still have dissimilar representations (Corol- lary 3.2). We then introduce new distributional distance (Definition 4.4) and prove (Theorem 4.7) that closeness in this distance bounds representational dissimilarity (Definition 4.5).
We conduct classification experiments on CIFAR-10 10 and find substantial representational dissimilarity between some trained models with similarly good performance (Section 5.1). This dissimilarity appears to stem from a mechanism analogous to that identified in our KI. divergence theory. We also run synthetic experiments showing that wider neural networks tend learn distributions closer under our distance and have more similar representations (Section 5.2). This suggests that the inductive biases of larger models may promote representational similarity, something already observed in previous works. Altogether, our findings indicate that the robustness of identifiability results [28, 51, 52] under the KL divergence may require additional assumptions, and that whether distributional closeness implies representational similarity depends critically on the choice of distributional and representational distance.
The structure and main contributions of our paper are as follows:
We prove that small KL divergence between two models from considered class (Equation (1)) does not guarantee similar representations (Section 3, Theorem 3.1); ); as corollary, models with mear-maximum data likelihood can have entirely dissimilar representations (Corollary 3.2).
3 For model family O and equivalence relation ~.., 0/ ~ denotes the quotient space whose elements are identifiability classes [[f.g)] = ((f', ((f''g') E θ (f. ~ (f'.g') [19, 27J.
'E.g., Roeder et al. [52]: "as the representational capacity of the model and dataset size increases, learned representations indeed tend towards solutions that equal up to only linear transformation."

---

We define distance between probability distributions (Section 4. and dissimilarity measure between representations (Section 4.2), and prove that small distributional distance implies represen- tational similarity up to specific class of invertible linear transformations (Section 4.3).
We empirically show that: (1) models trained CIFAR-10 despite similarly good performance through a mechanism similar to that in the proof of Corol- lary 3.2 (Section 5.1); (2) on synthetic data, wider networks yield lower mean and variance of our distributional distance, and also have more similar representations (Section 5.2).
exhibit dissimilar representations
2 Model Class and Identifiability
We consider input variables x E X, which can be real-valued (e.g., images) or discrete (e.g., text strings). We assume that the input data is sampled i..d. from a distribution px()). Given an input x, we consider the task of defining conditional distribution over categorical outputs y Y. which can be class labels in classification tasks or next-tokens following a context in language modeling.
Model class. Following Roeder et al. [52], we consider a model to be pair of functions (f.g) g) O where the codomain of both f and g is a vector space RM,, which we will also refer to as the representation space. Here, θ entails the space of all such pairs that can be generated by arbitrarily large neural networks. We will refer to f fxx as the model embedding and to g(y) as the umembedding. The conditional distribution" defined by the model is given by
exp(f(x) g(v))
Pr. (rlg( = Eveyeep(f(( exp(f(xx g(3'))
We will also indicate the model distribution with pr.g E Pe. where Pe is the space of conditional distributions that be constructed in Equation (1) using models in +. This model family captures a variety of models for different learning contexts [52] among which: (i) auto-regressive language models like GPT-2 [48] and GPT-3 [8]: (ii) supervised classifiers like energy-based models [28] and models trained with DIET [26]: (iii) pretrained language models like BERT [13]; (iv) self-supervised pretraining for image classification with Contrastive Predictive Coding (CPC) [47].
Identifiability. It useful to fix pivot input point X E X' and a pivot label yo E Y and consider the displaced embeddings fx(x)= == f(x) f(x)) and unembeddings (o g(y) g(y)). The conditional probability distribution generated by the model can then be rewritten as
(vlx) x cxp (x)((((x) g(y) (xp(-f(x) g(yo))
(2)
exp (xx) go()))
Next, we review identifiability of the model class in Equation (1). Identifiability theory characterizes the set of transformations of representations that leave a model's output distribution unchanged; two models are therefore deemed equivalent when one can be obtained from the other by such a transformation. To proceed, we introduce an assumption about our considered models. This condition, also known as diversity [28, 34, 52]. corresponds to only considering models that span the whole representation space, in the sense that there is no proper linear subspace of RM containing the embeddings or the unembeddings [43]. Formally, this can be written as:
Definition 2.1 (Diversity condition). A model (f. g) E θ satisfies the diversity condition there exists XI...., XM E X and ......,yy yM E y such that both the displaced embedding vectors { (1x and the displaced unembedding vectors (go(y,)) are linearly independent.
The diversity condition can hold when the total number of unembeddings in y is strictly higher than the representation dimension M [52]. For example, in GPT-2 [48], while the representation dimensionality varies in the order of thousands (depending on the model size), the number of tokens in y is of the order of ten thousands (~ 50k tokens), implying that diversity may be easily satisfied. With this assumption, we can prove the identifiability of the model in Equation (1) up to invertible linear transformations (a detailed proof can be found in Appendix B):
$ Both and g are therefore treated as nonparametric functions in the following, similar to [28. 34, 43. 52]. "We use the term "distribution" generically. In Equation (1), Pr.g (v(x) is a softmax over a finite label set, ie.. a probability mass function.
3

---

Model (f.g)
Model (f", g')
EMBEDDINGS f(-)
UNEMBEDDINGS g(-)
EMBEDDINGS (-())
UNEMBEDDINGS
B
Riys
6/3
0.5
80yel
-077
-30
-10
giya
Figure 2: Two models with small KL divergence but highly dissimilar representations. We construct two models (f. g). (f',g') gg') E θ whose representations are related by non-linear transformation: The embeddings and unembeddings of the (f', g') model sponding unembedding vectors the (f, g) model. As a result, the nearest unembedding vectors in g(-) (dashed lines) are mapped away from cach other in g (-). In Theorem 3.1, we show that as the norm of the unemhedding vectors grows for both models. their distributions pr.g and pr become closer in KL divergence, whereas their representations remain dissimilar -ie., far from being equal up linear transformation, also Table
constructed by permuting the embedding clusters and the corre-
Theorem 2.2 (Linear Identifiability [28, 34, 52]). Let (f,g), (f', ,') E O, and (f, satisfy the diversity condition (Definition 2.1). Let (resp. L') be the matrix with columns go (y) (resp. go (y)), and let L LL'T MxM Then:
Pr. x) p (y x), V(x, y) E x x (f.g) ~ (f'.g)
(4)
where the equivalence relation ~i, is defined by
f((x) = Af Bo(()) A gú (v)
(f.g) ~ g''
(5)
The result of Theorem 2.2 is central to the scope of our analysis because it shows that representa- tions of two models generating the same conditional distribution related by invertible linear transformation. This also means that, under the diversity condition. to each probability distribution Pr.g Pe corresponds a unique (up equivalence) choice of embeddings and unembeddings. See Fig. for an illustration.
Representational similarity. Theorem 2.2 suggests that natural notion of similarity between representations for our model class should account for the linear equivalence relation in Equation (5). Hereafter, we say two models g), ,') θ have equivalent representations if their embeddings and unembeddings are related by an invertible linear transformation of the kind in Theorem 2.2, i.e., gf,g) ~ (f", g'), and call representations similar if they are "close' to having such relation- which we will make precise in Section 4.2. We note that several measures of representa- tional similarity [30, 45, 49] are based on Canonical Correlation Analysis (CCA) [22], and are thus invariant to any invertible linear transformation. They thus attain their maximum when applied to rep- resentations that are equivalent in the sense above, but also for other invertible linear transformations. We discuss alternative choices of similarity measures in Section 6.
3 Models Close in KL Divergence Can Have Dissimilar Representations
Given Theorem 2.2 next ask whether similar conclusion holds approximately when the models have nearly the same distributions. More precisely, we ask: For a given choice of divergence between distributions, does small difference in distributions imply representational similarity?
As natural starting point, we consider the widely used Kullback-Leibler (KL) divergence. Specifi- cally, we quantify the difference between two models by the KL divergence between their conditional distributions, averaged over the data distribution pD(x), This is equivalent to the KL divergence between the corresponding joint distributions, assuming both share the marginal PD (x). Formally:
dKL (p) Ex-pp [KLL (p(v)x) || pl py(y))))] DKL (p(x, v) 1/ (x, y))
(6)
When DKL. (pylt) // (vlx)) 0 for all x, we obtain models that are equivvall, as specified in Theorem 2.2. However, only making the KI. divergence small is insufficient to conclude much about the similarity of representations, The following result formalizes this claim (proof in Appendix C.

---

Theorem 3.1 (Informal). Let k = | and assume k 2 M + 1. Then Ve > 0, there exist models (f. linearly equivalent.' More precisely, one can construct a sequence of model pairs, depending smoothly on real-valued parameter such that, as p - 0o, the KL divergence between the models tends to zero while their embeddings remain fixed and not equal up to any linear transformation.
(f'. (f'.). θ for which (gI, Pr x S 6, and whose embeddings are far from being
Proof sketch. We here sketch the construction used for k 2 M + (see Appendix C.1 for a construction which holds with k = M + 1) Let gf) E θ model which has unembeddings with fixed norm, i.e., [lg())] p. for all 3 and o E Rt. Assume the unembeddings have non-zero angles between them and let the unembeddings satisfy diversity (Definition 2.1). For E X, let the embedding f(x) have strictly largest cosine similarity with one g(ò). for some 9 E V. and such that y = yey ()). As a consequence, note that the model (f. g) also satisfies the diversity condition for the embeddings. Let (f', g'' E θ be another model which is constructed starting from (f, g) and considering a permutation, of the label indices such that g' (1s) = g (V+()) for i = 1..... k, with corresponding shift of the embedding clusters." For every E X. we let [x)
[|f(x)] || and let the angle between f' (xx and g'()) be equal to that between f(x) and gg)). We illustrate this construction in Fig. 2 for f(x).
E R2. One can then find a permutation for which f(x) is not a linear transformation' 9 of f' (xx (thus the KL divergence
between pr,g and pr.g' non-zero). However, as p -> oo, we have that dgi. (Pr.g. Pr .r'gg although the embeddings stay constant and not linearly equivalent.
Theorem 3.1 shows that, even when dKL (Pr.g, Pr g S e, there is no guarantee that the models close to being -egeevvvall The proof constructs p-parameterized sequence in which the embeddings remain fixed and not equal up to a linear transformation; consequently, the error of the best linear regression of one embedding set onto the other is strictly positive and stays constant as - 00, while dkL - 0.
In Theorem 3.1, we consider the KL divergence between two distributions from our model family. However, when practitioners study representational similarity in trained models, they fit each model to the data distribution sentations. We therefore also formulate the following corollary which is closer to the usual setup:
by mexiimum-likelhooe estimation and then compare their repre-
Corollary 3.2 (Informal). Let ls | and assume k: 2 M + 1. Consider a dataset of (x, y) pairs, where only one label is assigned to each unique input. Then, we can find two models that obtain an arbitrarily small cross-entropy loss the data while having representations which are far from being linearly equivalent in the same sense as in Theorem 3.1.
The proof is in Appendix C.2, and an illustration through synthetic experiments can be found in Appendix F.4. This corollary shows that two models having close optimal classification loss training data, possibly also equal, may not possess similar representations at all. Importantly, we find that this situation can arise in practice with discriminative models trained on CIFAR-10 (Section 5), which can have dissimilar representations despite assigning high probability to the same labels.
4 When Closeness in Distribution Implies Representational Similarity
In this section, we introduce distance between probability distributions (Section 4.1) and measure of representational similarity (Section 4.2) and prove that, for our model family (Equation (D), small values of the former imply high similarity under the latter (Section 4.3).
"We show in Section 5.3 how the construction from our proof can lead to high dissimilarity both according to MCCA [49, 52] and under the measure introduced in Definition 4.5.
*This is not a permutation of the representation dimensions-i.e. it is not f'(x) = Pf( for some permuta-
tion matrix P. The permutation of label indices, and corresponding shift of embedding clusters, is in general not a linear transformation of the representations (see Fig. 2); it also changes the model's output distribution. since it alters the rank-ba by predicted probability of every label except the tighest-probabliig one for each input x. "In Table 1, we show that one can construct models this way and such that the mean canonical correlation is close to zero, whereas it would be equal to one if their representations were linear transformations of each other.
5

---

4.1 A Distance Between Probability Distributions
We start by studying what relation holds between models in θ which need not have the same probability distributions. Hereafter, we will also consider the variance over input samples and outputs
and denote with Varx = Varx<pp [+] and Vary[] Vary-Unif(y)]) respectively. With this in
mind, it is useful to introduce the following functions:
(1x()))) = Varx log pl (8)(x) log pl and (x(:)))) = Varyllogp(viz, log (x)]
(7)
We also denote with S E MMxd the diagonal matrix with entries Sμ 1/x (y:: p) We now have everything we need to state a Lemma relating any two model embeddings (the full statement and proof, including the relation between the unembeddings can be found in Appendix E.1):
Lemma 4.1. Let (f, g), (f', .g') E θ be models satisfying the diversity condition (Definition 2.1). Let L. L' be defined as in Theorem 2.2. Let A = - S'L'' and hr hr(x) == -L- S- where the i-th entry of cy(x) is given by
log Pegg..&.00 (Vv..((Wib xx) p
log Pr ..r.ggl. (4/1x) (x) log Pr (yolx)
'y (x)
(8)
(xx: pr.g)
Ux pri
Then, we have
f(x) = (x) + hr(x).
(9)
Remark 4.2. In the special case where the two models (f, g). (f', g') E O have the same distribution, the error term in Equation (8) vanishes and we recover (f, g) ~ (l''g) (see Corollary E.I).
We note that the error term in Lemma 4.1 depends entirely on differences of log probabilities, We also see that if cy, (x) is constant (or equivalently, Varle,, (x) = 0), the embeddings will be invertible linear transformations of each other. This suggests that if we want a distance between distributions which is related to the similarity of the embeddings, we can measure how far this error from being a constant. We therefore define a distance which includes these weighted differences of log-likelihoods.
To proceed, we restrict our analysis to subset of distributions that satisfy the following assumption: Assumption 4.3. We consider probability distributions p such that for sets XLLV C X and VLLV C CY containing all labels except one, the following conditions
satisfied: (1) For all x E XLLV we have that log pyy (x) log p(y Xx)) is not constant in y and pD (x() > 0; and (2) For all E Vuay we have that log ((y) x) log p(yo (x) is not constant in
Note that this assumption guarantees that, for any such p. the terms (x((1 p). and (Xy:))))) are non-zero for x; E XL.x and y: E VIIx. Under Assumption 4.3, we exclude those probability distributions that assign the same distributions over the labels for all inputs, that is, we need at least one input to result in different distribution. We also exclude those distributions which assign equal probability to two or more labels for all inputs.
Definition 4.4 (Log-likelihood variance distance between distributions). For any two probability distributions p.y/ for which there exists a common choice of XIAN C and VLaN C y such that they both satisfy Assumption 4.3, for a fixed X E RRt, and by considering the following terms:
logp(y|x)
log (v)x)
log log p(vn(x) xx)
log 1/ (yo xx)
41 = max MEDiaN \ou))
Varx
Varx
(x:
(x(u;p')
(x(v:p)
<>x p')
log ((ul())))
[x(x)
log logp(y|xx))
(ogr((v/xx)
12 = max
Var,
Vary y
xEXLav (xx)
Viy (x:p))
(x;,'))
(y(x;p)
(x:x')))
ty
max Ve
(es((::)) p) (s((:)))) p'),, t4 ==
max
[yy p) -x:xx'')
XERANN \xx)
the log bog-iikeidood variance (LLV) distance between p and p' is given by d1.(p. max (t1, t2, Xt3, Xt1}
(10)
In Appendix E.2, we show that d distance metric between sets of conditional probability distributions." 10 The non-zero weighting constant X is introduced because, as we will show in
Speciiicallyy dirv p') d') negatigative; zero iff p p'. symmetric; and satisfies the triangle inequality.

---

Theorem 4.7, t1 and t2 can be used alone to bound the similarity between model representations, but to and ta need also be considered to make dLyv a distance metric. In our experiments, we set À to a small non-zero value, so that the ( and l2 terms dominate (see Appendix E.3 for details).
Note that the log-likelihood variance distance, d'iv. requires choice of input, 2L.v. and label, FLiV, sets, and the exact value of the distance depends on this choice when the distributions are not equal. However, d is a metric for any choice satisfying Assumption 4.3. Therefore in the experiments, we draw a sample of possible sets and choose the ones which give the smallest value.
4.2 A Dissimilarity Measure Between Representations
Having defined distance between distributions, turn to distance between representations which is related to invertible linear transformations. We define a dissimilarity measure based on the partial least squares (PLS) approach called PLS-SVD [54, 60]. For two random vectors z, w, let Ezw denote the covariance matrix whose (i.j)-th entry is
(II) where Pz,w is the joint distribution of z, w. and Pw are the respective marginals. PLS-SVD seeks pairs of unit-length vectors uy E Rdz and V TR (E = 1, ... that maximize the covariance between the one-dimensional projections u z and vž w. subject to mutual orthogonality of successive directions. This is equivalent to finding the left and right singular vectors of Σzw (more about PLS-SVD in Appendix D). Leveraging this procedure by PLS-SVD, we introduce a similarity and dissimilarity measure:
Ez )ag = Cov [24 wy] == () [[=; Ezz-p (x;) - -ww-p-ush
Definition 4.5 (PLS SVD distance between vectors). Let 2, w be two M-dimensional random vectors, and define z', w' by standardizing their components: 2 = (2j B[z+))/ (z.) w, (w, - E[ww]) / std( w.). Let {u and [v.) be the left and right singular vectors of the cross- covariance matrix Ez'w'. We define
mSvD(z, (z. w) == EM Covz [u] 2., V. w'), ds w)
msvD w)
(12)
Since msvD Zz w) S mSvD(z, z) = 1, it follows that dsv w) 2 for all random vectors z, w. We also have that the dissimilarity measure is invariant to rotation followed by dimension-wise scaling (see Appendix E.4): if mSvD(z. w) or, equivalently, dSvD w) = 0, then there exist an orthonormal matrix R and diagonal matrices S,S' scaling 2, w to unit variance such that z = S RS'w.
With this distance between vectors, we can define a dissimilarity between representations for models from our model family by seeing the embeddings and unembeddings as random vectors.
Definition 4.6 (Representational dissimilarity measure de de Let g); (f', g') O. Let L, L' be defined as in Theorem 2.2. Let N (resp. N') be the matrix with columns (x() (resp. (x)). The representational dissimilarity measure de.g is defined as follows:
(r.g((f. g), == mmax/ (I T f(x), L.).L". (TT'((x)), dsv)) (N N 8'(1)) (13)
4.3 Bounding Representation Dissimilarity via Distributional Distance
We now prove that it is possible relate a bound on the distance metric deLv between model distributions (Definition 4.4) to a bound on the dissimilarity measure dr.g between model repre-
sentations (Definition 4.6). For the result below, We denote with 2 == L f(x), Z2 == L f'(().
W := N g(y) and W2 N" (w) (proof in Appendix E.6).
Theorem 4.7. Let (f.g), (f', g') E θ be two models such that: (1) There exist XLIN C. X and MnI C y, consisting of a pivot point and all labels but one, such that all L, and N, N' matrices constructed from these sets are invertible;"! (2) Both pt.g and g' satisfy Assumption 4.3 for KLIN and VLLV; (3) The covariance matrices Exizi: Ew w. and the cross cross-covariance matrices E2,,,, Ew, ws are non- singular... Then, for any weighting constant X E Rt, we have
di Pr' ) S = dr.g((f,g), (f', g')) < 2Me
(14)
"Thhs is slightly stronger than the diversity condition (Definition 2.1) in the sense that we need diversity to hold for all sets using labels from VLLV and the chosen pivot point.
7

---

Sirpane automotilko
1.0-
classes
63-
moam moamm--e
deer
622
toog
61-
ha150
ship
35-
600
truck
3.5
ENG (X)
EMBEDSINGS (t))
with
Figure 3: (Left) Embedding representations of two models trained on CIFAR-10. 10. Representations for some of the labels are permuted, and mcca (f(x), (x)) = 0.55. (Right) Mean d and de. vs network width. Shaded area is standard deviation. Both mean and standard deviation decrease as the network width increases.
(x)) remains constant for models that are linearly equivalent to the members in the expression. That is, for any (f" (f",g") E such that (f", .g") ~ (f,g), have that dgvD (L f(x), L' L dsv L
Remark 4.8. can be shown (Lemma E.8) that dL f(x), L' ('
A similar argument also holds for dsvD(N g(v), g'))..
Theorem 4.7 shows that for the measures di d and dr.g. closeness in distribution (i.e., small values of d bounds representational dissimilarity (as measured by dr.g)-
Notice that Theorem 4.7, we can freely choose any set of inputs XLIV C X as long as they satisfy (1) and (2). Hence, if there is a choice making to in Definition 4.4 as small as possible, it will also
T make dsvD (NT g(y), N'T g' (v)) small. There also some flexibility for VLLV C Y, where careful choice of pivot point yo and the left-out label can make the term & Definition small and, as consequence, dsvD [L f(x), L L'T'' f'()) also small.
Experiments
In this section, we present our key experimental findings: (1) For models trained for classification on CIFAR-10 [32]. we observe cases of similarly well-performing models with highly dissimilar representations, arising from a mechanism analogous the constructive proof of Theorem 3.1 (Section 5.1); (2) On synthetic data, training wider neural networks results in a reduction of both the mean and variance diLv between models and leads to higher representational similarity (Section 5.2). Additional experiments illustrating the phenomena in Theorem 3.1 and visualizing the bound in Theorem 4.7 are summarized Section 5.3. For experiments we use 10-5, See Appendix F for implementation details. The code can be found on GitHub." 12
5.1 Dissimilar Representations in CIFAR-10 Models with Similar Performance
Experimental setup. We trained classification models CIFAR-10 [32] with two-dimensional embedding and unembedding representations. The embedding network a ResNet18 [21], and the unembedding network consists of three fully connected layers of width 128, followed by the final representation layer.
Results. Embeddings of images with the same labels form clusters, and certain label pairs pairs-such as "truck" and "automobile" consistently appear as neighbors across retrainings (See Appendix F.5). However, some clusters permuted: Fig. (Left) shows two retrainings where, in the first model, the "truck" cluster lies between "automobile" and "ship", while in the second it lies between "automobile" and "horse". This mirrors the construction in Theorem 3.1, where models close in KI. divergence have dissimilar representations. The models in Fig. (Leftt) have dRL. of 1.13 (0-+1) and 1.12 (1>0), similar test losses (1.19 and 1.18), but a large d (~ 1.55) and small mCCA (0.55).
"github. www..//wwww/mmell.....ooo distssse---it--eee rep-s----p--- sim, DOL: 10. www.628812zodde 17249361.

---

5.2 Wider Networks Learn Closer Distributions and More Similar Representations
Experimental setup. We train models on a 2D classification task with 4, 6, 10 or 18 labels. Data are samples from 2D Gaussian (Ii = 0, α 3), and labels are based on the angle to the first axis, consisting of slice of the circle and the opposite slice (see Appendix F.2). We train twenty random seeds of models with wwo-dimensional representations. Fach model consists of three fully connected layers with widths 16, 32, 64, 128 or 256. We retain only models achieving over 90% accuracy. For classes, this yields 19 models at width 16 and 20 models for all other widths. For 6 classes, we retain 16, 17, 19, 19, and 20 models across the five widths, respectively.
Results. We find that diyy and dr,g between the models trained with wider networks have smaller both mean and standard deviation. In Fig. 3 (Right) we show the trend for models with 4 and 6 classes. Similar plots for 10 and 18 classes can be seen in Appendix F.7. While the bound in Theorem 4.7 is only non-vacuous when (r.g((f. g). (f', g')) v 2M < (i.c., when e < 1/2M) our plots suggest that there is a broader relationship between dr, and di since the values of dL and de,g seem to be related even before the bound becomes non-vacuous.
5.3 Visualizing Representation Dissimilarity and the Bound from Theorem 4.7
Following the construction in Theorem 3.1, we simu- late two models with fixed embeddings and unembed- dings with increasing norm. We measure the KI. diver- gence and dL between distributions; between repre- sentations, we measure the mean canonical correlation (moCa) [45, 49, 52] and the maximum deyD of the em- beddings over the input and label sets for d (since this is what Theorem 4.7 bounds). The results are in Table 1: We find that as the unembeddings' norm, p. increases, dkL decreases, while Lvv and dgvD remain high and mccA remains low and almost constant. This aligns with the result in Theorem 3.1, while it displays that di is robust to these model constructions.
Table I: Kullback-Libbeer divergence rapidly vanishes while other measures stay constant. Cells in the dKL column shaded red in proportion to their magnitude (darker = larger value). As the unembed- ding norm & grows, dkr. drops hy almost four orders of magnitude, whereas di and the maximal devn between embeddings re- main virtually unchanged.
dxL
dra
MCCA
maxdavD
3
0.8866
1.3176
0.0006
0.9996
0.2230
0.0007
0.9998
0.0369
1.3171
0.0004
0.9995
0.0055
1.3178
0.0006
0.9998
Also, for these model pairs and models trained on synthetic data, we find that the bound in Theorem 4.7 is always
1.5
0.0008
1.3175
0.0008
0.9999
18
0.0001
1,3172
0.0010
0.9998
satisfied (refer to Appendix F.6 for details and to Fig. 16 for visualization). However, for models trained on CIFAR-10, the distributions are not close enough for the bound to be non-vacuous.
6 Discussion
Limitations. To the best of our knowledge, this work provides the first theoretical result that upper- bounds representational dissimilarity by distributional distance (Theorem 4.7) a relationship that Theorem 3.1 shows is far from obvious. The bound is nonetheless not tight: it grows linearly even though the empirical trend of max dsvD is clearly sub-linear (Fig. 16), and it presently requires the label set to contain every class but one. We conjecture that a tighter, potentially non-linear bound could be obtained and that dA would remain a valid distance when computed on a suitably diverse subset of labels. The distributional distance itself (Definition 4.4) inherits the same label- diversity assumption (Definition 2.1), which prevents its applicability to the whole model family of Equation (1): diversity is more likely to hold in language models, but may fail in standard image classifiers
when the number of classes is smaller than the representation dimension [52]. It was recently proved by Marconato et al. [43] that, when relaxing the diversity assumption, the model family in Equation (1) can be identified up to an extended- equivalence relation. Extending our theory to that setting would enable comparing models with different representation dimension [43].
Alternative similarity measures. The invariances of our dissimilarity measure, dr.g. comprise a subset of all invertible linear transforms and always consider embeddings and unembeddings in combination with each other (see Appendix E.7). By contrast, much prior work employs CCA-based measures that are invariant to any linear transformation [45, 49] and only measures (dis (dis)similarity

---

between embeddings. Kornblith et al. [31] argue that similarity measures should be invariant only to orthogonal transformations, based on the task-relevance of relative activation scales and the invariance of gradient-descent dynamics [35]. What measure best captures representational similarity still an open question [3], with different approaches reflecting distinct goals and trade-offs [30]. Our goal is not to claim our measure is the right one, but to present a pairing of dissimilarity measure and distributional distance that enables theoretical analysis and captures key empirical aspects of representational similarity. Extending such guarantees to other (dis (dis)similarity measures and distributional divergences is an important direction for future work.
Do similarly retrained models learn similar representations? This has been extensively explored in empirical studies [31, 38, 46, 51, 52, 56, 65]. Retraining with a different seed can result in different likelihoods, and we show that even models with near-zero KL divergence can have dissimilar internal representations. While the existence of constructions such as the one in Theorem 3.1 does not imply that model will learn them during training, we find it interesting that a mechanism resembling our construction actually emerged in our CIFAR-10 experiments (Section 5.1), where we observe large representational dissimilarity despite close distributions in terms of KL divergence. Such dissimilarity can arise when most output labels have negligible probability for example, an image might be confidently classified "cat" or "dog". with all other classes (e.g., "spaceship" "chair") having near-zero probabilities; and in language modeling, often only few next-tokens are plausible. Overall, these observations suggest that similar performance alone is insufficient to ensure representational similarity [6]. though higher network capacity may help, see Section 5.2 and 56].
Identifiability and robustness. In nonlinear ICA [10, 18, 20, 24, 28, 57, 72], disentangled and causal representation learning [2. 7, 37, 39, 40, 44, 63, 64, 69-71] models are typically constructed solutions are unique up to fixed (and 'small') set of ambiguities,
so that their aeelihooddmaximiise as guaranteed by identifiability [24, 68]. Robustness when the likelihood is only approximately maximized has been studied less, though Buchholz and Schölkopf [9] analyzed ssometric-mxxing ICA; Sliwa et al. [55] empirically probed independent mechanism analysis [19]; and Träuble et al. [62] showed that independence- based disentanglement lacks robustness when true factors are correlated. Our results further highlight the importance of understanding robustness in nonlinear representation learning. 13 In particular, our Corollary 3.2 suggests that relying solely on identifiability results -uchh as those by Reizinger et al. [51]-may be insufficient to guarantee that similar representations are practically attainable when optimizing the cross-entropy loss: additional assumptions may be needed.
Conclusion and Outlook. We have studied the link between distributional closeness and repre- sentational similarity for the embedding and unemhedding layers of our model family, focusing on different instances of the same model class. Extending our analysis to earlier layers would first require identifiability results for intermediate network representations, which remains an open problem in representation learning [25, 28]. In practice, learned representations often appear surprisingly robust across datasets, architectures, and training objectives [3, 4, 17, 23, 31, 46), and explaining this broader robustness is an exciting direction for future work By clarifying when distributional closeness does- and does not-imply representational similarity for embeddings and unembeddings within our model family, we take initial step toward principled theory of representational similarity.
Acknowledgments and Disclosure of Funding
We thank Simon Buchholz for helpful discussions and feedback. Beatrix M. G. Nielsen was supported by the Danish Pioneer Centre for AI, DNRF grant number P1. E.M. acknowledges support from TANGO, Grant Agreement No. 101 120763, funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Health and Digital Executive Agency (HaDEA). Neither the European Union nor the granting authority can be held responsible for them. A.D. acknowledges funding from the Helmholtz Foundation Model Initiative. L.G. was supported by the Danish Data Science Academy, which is funded by the Novo Nordisk Foundation (NNF21SA0069929).
13 e- n-n- identifiability [50] is closely related: our work illustrates it for learned representations and highlights that it depends the choice of divergence- in deL (Theorem 3.1) but not in di (Theorem 4.7).
10

---

References
[1] Ani Adhikari and Jim Pitman. Probability for Data Science. UC Berkeley, 2025, URL htttps://textbook.pcce .rgg [Cited on page 43.]
[2] Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal repre- sentation learning. International Conference on Machine Learning (ICML), pages 372-407. PMLR, 2023. [Cited
page 10.]
[3] Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in Neural Information Processing Systems (NeurIPS), 34: 225-236, 236, 2021. [Cited on pages and 10.]
[4] Boaz Barak, Annabelle Carrell, Alessandro Favero, Weiyu Li, Ludovic Stephan, and Alexander Zlokapa. Computational complexity of deep learning: fundamental limitations and empirical phenomena. Journal of Statistical Mechanics: Theory and Experiment, 2024( (224(10):04000 2024.
(Cited on page 10.]
[5] Samuele Bortolotti, Emanuele Marconato, Paolo Morettin, Andrea Passerini, and Stefano Teso. Shorteuts and identifiability in concept- based models from a mouro-symbolic lens. arXiv preprint arXiv:2502. 1245, 2025. [Cited on page 1.]
[6] Lukas Braun, Erin Grant, and Andrew M. Saxe. Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks. In International Conference on Machine Learning (ICML), 2025. [Cited on page 10.]
[7] Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal representation learning. Advances in Neurul Information Processing Systems (NeurIPS), 35: 38319-38331, 2022. [Cited on page 10.]
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:
1877-1901, 2020. [Cited on pages and 3.]
[9] Simon Buchholz and Bernhard Schölkopf. Robustness of nonlinear representation learning. In International Conference Machine Learning (ICML), pages 4785-4821, PMLR, 2024. [Cited on page 10.]
[10] Simon Buchholz, Michel Besserve, and Bernhard Schölkopf. Function classes for identifiable nonlinear independent component analysis. Advances in Neural Information Processing Systems (NeurIPS), 35: 16966,, 2022. [Cited on page 10.]
[11] Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing. Advances in Neural Information Processing Systems (NeurIPS), 36: 45462, 2023. [Cited on page 10.]
[12] Irene Cannistraci, Luca Moschella, Mareo Fumero, Valentino Maiorca, Emanuele Rodolà, et al. From bricks to bridges: Product of invariances to enhance latent space communication. In International Conference on Learning Representations (ICLR), 2024. [Cited on page 1.]
[13] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. (Long and Short Pupers), pages 4171-4186, 2019. [Cited on page 3.]
[14] Frances Ding. Jean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity through statistical testing. Advances in Neural Information Processing Systems (NeurIPS), 34: 1568-1568, 2021. [Cited on page 1.]

---

[15] Hidde Fokkema, Tim van Erven, and Sara Magliacane. Sample-efficiem learning of concepts with theoretical guarantees: from data to concepts without interventions. arXiv preprint M.Kiv:2502.065536 2025. [Cited on page 1.]
[16] Marco Fumero. Marco Pegoraro, Valentino Maiorca, Francesco Locatello, and Emanuele Rodolà. Latent functional maps: a spectral framework for representation alignment. Advances in Neural Information Processing Systems (NeurIPS), 2024. [Cited on page 1.]
[17] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Matthias Bethge, Felix A Wich- mann, and Wieland Brendel. On the surprising similarities between supervised and self- supervised models. arXiv preprint 20X1:20010.08377, 2020. [Cited on page 10.]
[18] Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete Rosetta Stone problem: Identifiability results for multi-view nonlinear ICA. In Uncertainty in Artificial Intelligence (UAI), pages 217-227. PMLR, 2020. [Cited on page 10.]
[19] Luigi Gresele, Julius Von Kügelgen, Vincent Stimper, Bernhard Schölkopf, and Michel Besserve. Independent mechanism analysis, a new concept? Advances in Neural Information Processing Systems (NeurIPS), 34:28233-282222 2021. [Cited on pages 2 and 10.]
[20] Hermanni Hälvä and Aapo Hyvarinen. Hidden markov nonlinear ICA: Unsupervised learning from nonstationary time series. In Uncertainty in Artificial Intelligence (UAI), pages 939-948.
PMLR, 2020. [Cited on page 10.]
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. [Cited on pages 8 and 49.]
[22] Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3-4):321-377, 1936.
[Cited on pages 4 and 35.]
[23] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: representation hypothesis. In International Conference on Machine Learning (ICML), 2024. [Cited on page 10.]
The platonic
[24] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 859-868 PMLR, 2019. [Cited on page 10.]
[25] Aapo Hyvärinen, Ilyes Khemakhem, and Ricardo Monti. Identifiability of varial-v and structural-equaation models: from linear to nonlinear. Annals of the Institute of Statistical Mathematics,
2024. [Cited on page 10.]
[26] Mark Ibrahim, David Klindt, and Randall Balestriero. Occam's 's razor for self supervised learning: What is sufficient to learn good representations? In NeurlPS 2024 Workshop: Self Self-Supervised Learning-Theory and Practice, 2024. [Cited on page 3.]
[27] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligence and Statisties (AISTATS), pages 2207-2217. PMLR, 2020. [Cited on page 2.]
[28] Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Iden- tifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems (NeurIPS), 33:12768-111 12778, 2020. [Cited on pages 2, 3, 4, 10, and 25.]
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations (ICLR), 2015. [Cited on pages 49 and 50.]
[30] Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. ACM Computing Surveys, 1-59) 2025, [Cited on pages 4, and 10.]
12

---

[31] Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning (ICML), pages 3519-3529. PMLR, 2019. [Cited on pages and 10.]
[32] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. [Cited on pages 8, 49, and 54.]
[33] Aarre Laakso and Garrison Cottrell. Content and cluster analysis: assessing representational similarity in neural systems. Philosophical psychology, 13(1):47-76, 2000. [Cited on page I.]
[34] Sébastien Lachapelle, Tristan Deleu, Divyat Mahajan, loannis Mitliagkas, Yoshua Bengio, Simon Julie,, and Quentin Bertrand. Synergies between disentanglement and sparsity: Generalization and identifiability in multi-task learning. In International Conference on Machine Learning (ICML), pages 18171-18206. PMLR, 2023. [Cited on pages 2, 3, 4, and 25.]
[35] Yann LeCun, Ido Kanter, and Sara Solla. Second order properties of error surfaces: Learning time and generalization. Advances in Neural Information Processing Systems (NeurIPS), 3, 1990. [Cited on page 10.]
[36] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 991-999, 2015. [Cited on page 1.]
[37] Adam Li, Yushu Pan, and Elias Bareinboim. Disentangled representation learning in non- markovian causal systems, In Advances in Neural Information Processing Systems (NeurIPS), 2024. [Cited on page 10.]
[38] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Feature Extraction: Modern Questions and Challenges, pages 196-212. PMLR, 2015. [Cited on pages and 10.]
[39] Wendong Liang, Armin Kekić, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, and Bernhard Schölkopf. Causal Component Analysis. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024. [Cited on page 10.]
[40] Phillip Lippe. Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning (ICML), pages 13557-13603. PMLR, 2022. [Cited on page 10.]
[41] Valentino Maiorca, Luca Moschella, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via inverse relative projection. arXiv preprint arXiv:2406.15057 2024. [Cited on page 1.]
[42] Emanuele Marconato, Andrea Passerini, and Stefano Teso. Interpretability is in the mind of the beholder: A causal framework for numan-interpretable representation learning. Entropy, 25 112):1574, 2023.
[Cited on page 1.]
[43] Emanuele Marconato, Sébastien Lachapelle, Sebastian Weichwald, and Luigi Gresele. All or none: Identifiable linear properties of next-token predictors in language modeling. International Conference on Artificial Intelligence and Statistics (AISTATS), 2024. [Cited on pages 3, 9, and 25.]
[44] Gemma Moran, Dhanya Sridhar, Yixin Wang, and David M. Blei. Identifiable deep generative models via sparse decoding. Transactions Machine Learning Research, 2022. [Cited on page 10.]
[45] Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018. [Cited on pages 1, 4, and 9.]
[46] Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodola. Relative representations enable zero-shot latent space communication. International Conference Learning Representations (ICLR), 2023, [Cited on pages and 10.]
13

---

[47] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0377 2018. [Cited on pages and 3.]
[48] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9 2019. [Cited on pages 2 and 3.]
[49] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017. [Cited on pages 1, 4, and 9.]
[50] Patrik Reizinger, Szilvia Ujváry, Anna Mészáros, Anna Kerekes, Wieland Brendel, and Ferenc Huszár. Position: Understanding LLMs requires more than statistical generalization. In International Conference on Machine Learning (ICML), pages 42365-42390, PMLR, 2024.
[Cited on page 10.]
[51] Patrik Reizinger, Alice Bizeul, Attila Juhos, Julia Vogt, Randall Balestriero, Wieland Brendel, and David Klindt. Cross-entropy is all you need invert the data generating process. In International Conference on Learning Representations (ICLR), 2025. [Cited on pages 2 and 10.]
[52] Geoffrey Roeder, Luke Metz, and Durk Kingma. On linear identifiability of learned representa- tions. In International Conference on Machine Learning (ICML), pages 9030-9039. PMLR, 2021. [Cited on pages 2, 9, 10, and 25.]
[53] Aninda Saha, Alina Bialkowski, and Sara Khalifa. Distilling representational similarity using centered kernel alignment (CKA). In Proceedings of the the 33rd British Machine Vision Conference (BMVC 2022). British Machine Vision Association, 2022. [Cited on page 1.]
[54] Paul D Sampson, Ann P Streissguth, Helen M Barr, and Fred L. Bookstein. Neurobehavioral effects of prenatal alcohol: Part II. Partial least squares analysis. Neurotoxicology and teratology, 11(5):477-49,, 1989. [Cited on pages 7 and 35.]
[55] Joanna Sliwa, Shubhangi Ghosh, Vincent Stimper, Luigi Gresele, and Bernhard Schölkopf. Probing the robustness of independent mechanism analysis for representation learning. In 1st Workshop on Causal Representation Learning at the 38th Conference on Uncertainty in Artificial Intelligence (UAI CRL 2022). OpenReview. net, 2022. [Cited on page 10.]
[56] Gowthami Somepalli, Liam Fowl, Arpit Bansal, Ping Ych-Chiang, Yehuda Dar, Richard Bara- niuk, Micah Goldblum, and Tom Goldstein. Can neural nets learn the same model twice? investigating reproducibility and double descent from the decision boundary perspective. In IEEE/CVF Conference on Computer Vision and Patter Recognition (CVPR). pages 13689-
13698. Institute of Electrical and Electronics Engineers, 2022. [Cited on page 10.]
[57] Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by nonlinear ICA with general incompressible-fon networks (GIN). arXiv preprint xxxv:2001.00877. 2020. [Cited on page 10.]
[58] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems (NeurIPS), 15:6906-6919 2021. [Cited on page 1.]
[59] Gilbert W Stewart. Perturbation theory for the singular value decomposition. Citeseer, 1998.
[Cited on page 42.]
[60] Ann Pytkowicz Streissguth, Fred L Bookstein, Paul D Sampson, and Helen M Barr. The enduring effects of prenatal alcohol exposure on child development: Birth through seven years, partial least squares solution. The University of Michigan Press, 1993. [Cited on pages 7 and 35.]
[61] Ili Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Erin Grant, Jascha Achterberg, Joshua Tenenhaum, et al. Getting aligned on representational alignment. arXiv preprint arXiv:2310.13011 2023. [Cited on page 1.]
14

---

[62] Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer. On disentangled representations learned from comrllated data. In International Conference on Machine Learning (ICML), pages 10401-10412. PMIR, 2021. [Cited on page 10.]
[63] Burak Varici, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. General identifiability and achievability for causal representation learning. In International Conference Artificial Intelligence and Statistics (AISTATS), pages 2314-2322. PMLR, 2024. [Cited on page 10.]
[64] Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Kekić, Elias Bareinboim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from unknown interventions. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024. [Cited on page 10.]
[65] Liwei Wang, Lunjia Hu, Jiayuan Gu, Zhiqiang Hu, Yue Wu, Kun He, and John Hopcroft. Towards understanding learning representations: To what extent do different neural networks learn the same representation. Advances in Neurul Information Processing Systems (NeurIPS), 31, 2018. [Cited on pages and 10.]
[66] Jacob A Wegelin et al. A survey of Partial Least Squares (PLS) methods, with emphasis on the two-block case. University of Washington, Tech. Rep, 2000. [Cited on page 35.]
[67] Alex H Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. Advances in neural information processing systems, 34:4738 47500,
2021. [Cited on page 1.]
[68] Quanhan Xi and Benjamin Bloem-Reddy. Indeterminacy in generative models: Characterization and strong identifiability. International Conference on Artificial Intelligence and Statistics (AISTATS), pages 6912 6939. PMLR, 2023. [Cited on page 10.]
[69] Danru Xu, Dingling Yao, Sebastien Lachapelle, Perouz. Taslakian, Julius Von Kügelgen, Francesco Locatello, and Sara Magliacane. A sparsity principle for partially observable causal representation learning. In International Conference Machine Learning (ICML), pages 55389-55433. PMLR, 2024. [Cited on page 10.]
[70] Dingling Yao, Dario Rancati, Riccardo Cadei, Marco Fumero, and Francesco Locatello. Unifying causal representation learning with the invariance principle. arXiv preprint arXiv:2409.02772. 2024.
[71] Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shan- mugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024. [Cited on page 10.]
[72] Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. Advances in Neural Information Processing Systems (NeurIPS), 1116611116422 2022.
[Cited on page 10.]
[73] Zikai Zhou, Yunhang Shen, Shitong Shao, Linrui Gong, and Shaohui Lin. Rethinking centered kernel alignment in knowledge distillation. In Proceedings of the Thirty-Thire International Joint Conference on Artificial Intelligence, pages 5680-5688, 2024. [Cited on page 1.]
[74] Martin Zong, Zengyu Qiu, Xinzhu Ma, Kunlin Yang. Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In International Conference on Learning Representations (ICLR), 2023. [Cited on page 1.]
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We list our three main claims in Section and they are mainly supported by Section 3, Section 4 and Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Some proof ideas and sketches are in the main article. Full proofs are in the appendix due to size constraints. Proofs for Section 2 are in Appendix B, proofs for Section 3 are in Appendix C and proofs for Section 4 are in Appendix E.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give implementation details of our distance Appendix E.3 and details on how recreate the experiments in Appendix F. All the relevant code is available at github 14
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
14 "github. www.///mmgii///le//eeeooe wttp:///ccooeooooeeeeoooooe dist-rep-stit
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code will be released publication of the article.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
woweeeSubmssiioooooiooooooo for more details. public/guide httee//iide//eeeeeniiiieeeeeeeee
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips. cw/ccc/pp.. mublic/guidese /ttp///ddde//oeeummiimiii poodeeSumisssionPee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 5 provide the necessary details understand the results and in Appendix F and Appendix we provide the details important for implementation. The code is also available github, gi thub. cwm/bemiiiccccccccooccccmm.....e wteeeiggni//oie hetinni//olleeeeeeeeeeeeeeeeeeee dist-rep---t--------e www--eeppeeeeeeeee sim.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We do not have error bars our plots.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We report computing resources in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http///eerrpp/eeeeee/eee
Answer: [Yes]
Justification: Ethics.
This work conforms with the principles outlined in the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work focuses on improving our understanding a popular class of discrim- inative models including autoregressive language models. We hope our work can lead to safer deployment of such models the future, however for now this fundamental research that has direct societal impact.
19

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
We do not release anything with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite original creators when relevant. Available licence information is in Appendix H.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi sewrrswithcodee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We describe how to generate the synthetic data used in Appendix F.2. Our code also includes the necessary documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The work does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs have not been used for any core components of this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httpps://neuripsss wwtp://cccccccccccccc/cccaaacaee for what should or should not be described.
22

---

Appendix
Table of Contents
A Notation
24
B Identifiability Result and Proof
25
Models close in KL divergence can have dissimilar representations details
C.I KL goes to Zero but Representations Dissimilar
27
C.2 Loss goes Zero but Representations Dissimilar
33
D Partial Least Squares (PLS) and Canonical Correlation Analysis (CCA)
35
D.1 PLS-SVD
35
D.2 Canonical Correlation Analysis
35
When Closeness Distribution Implics Representational Similarity Details
36
E.1
The Connection Between Representations and Log- Likelihoods
36
E.2
Distance Between Distributions
39
E.3
Implementation of the Log Likelihood Variance Distance
41
E,4
Using PLS SVD to Define a Dissimilarity Measure
41
E,5
Lower Bound on Mean PLS--VV
42
E.6
Bounding Representational Similarity with Distribution Distance
44
E.7
Invariances our Representational Distance and CCA
F
Experimental Details
49
F1
Constructed Models
49
F2
Models Trained on Synthetic Data
49
F3
Models Trained on CIFAR-10
49
R4
Loss Difference vs Embedding MCCAA
50
E5
All Two-dimensional Representations of CIFAR- 10 Models
50
Illustration of Bound on Constructed Trained Models
53
F.7
Wider Models Have more Similar Distributions Extra Plots
53
G Computing Resources
54
Assets
54
23

---

A
Notation
We here present table (Table 2) with the notation used in the paper.
Table 2: Notation. We bold letters for vectors and non-bold letters for scalars.
Symbol
Description
Reference
vector and vector-valued variables
Section 2
y
scalar-valued variable
Section 2
(f, (f,g)
model with embedding and unembedding functions f and g, see Equation (1) distributions of model (f, g)
Section 2
Pr.g
Section 2
(x)
the displaced embedding functions f(x) f(xu)
Section 2
8o(y)
the displaced unembedding functions g(y) ggb))
Section 2
L
the matrix with columns (y). see Theorem 2.2
Section 2
dRL.
the distance between distributions arising from the KI. divergence the variance over the inputs from some input distribution, Px the uniform distribution over the elements of y
Section 3
Var [
Section 4
Unif())
Section 4
divv
the log-likelihood variance distance with weighting parameter À the mean PLS-SVD distance between random vectors distance between representations for our model class
Section 4
dsvD
Section 4
dr.s
Section
Ezw
the cross-covariance matrix of random vectors z, the matrix with columns (x)
Section 4
N
Section 4
mcCA
the mean canonical correlation coefficient
Appendix D.2
24

---

B Identifiability Result and Proof
We present a unified and slightly adapted version of the identifiability results from [28, 34, 52], see also [43, Corollary 6].
Theorem 2.2 (Linear Identifiability [28, 34, 52]). Let (f, g), g') E O, and (f.g) satisfy the diversity condition (Definition 2.1). Let (resp. L') be the matrix with columns go (y) (resp. go (y)).
and let A = L L RMMM. Then:
(V. x) PF g' (u x), v(x,v) EXXY x = (f,g) ~ (f'
(4)
where the equivalence relation ~r, is defined by
f(x) = (x) Bu(y) A gg(())
(f. ~ (f',
(5)
Proof. We first prove that pr,x (y/ x) Pr g' (y) x)) ((x xX f(x) (x) for
A - invertible.
By assumption, the models (f. g), (') have equal likelihoods. Moreover, by construction, the two models have representations in RM,. Let (xx = Evsey exp(f(x) g(y;)), and similarly for Z' (') Then
(v x) = Pr x (1 x) f(x) g(v) - logg D)) = f' fx() (x)T)'(y) log (x,
(15)
(16)
for all x E A' and all E V. In particular, this equation holds for any fixed y. From the diversity con- dition (Definition 2.1), we can choose M y ......., yw. such that the displaced unembeddings (go [ are linearly independent. By subtracting the log-conditional distribution for the pivot yo E Vi we obtain for each y/ E y the following:
f(x) T g()) f(x) T (16) + (x,((((x))) - log(Z(x, 37)
= fxx) T g'( (w:) f'(x) g (No) + log( (og(Z"(x)))) ()) - log(Zz(x, (17)
f(x) g(v) f(x) g(va) f'(x) g'(()) (x(x) T (v(o))
(18)
f(x) (wr) f'(x) (v:),
(19)
where the last passage holds by definition of go (1()). see Section 2. Let L be the matrix which has go (V:) as columns and L' be the matrix which has (v:)) as columns. Notice that the diversity
condition (Definition 2.1) implies that L is an invertible matrix. We can then stack the equations to
get
L f(x) = [ [f((xx)
(20)
and since L is invertible,
(21)
If we set A = (I',-1),, we only need to show that A is invertible. Using the diversity condition,
we pick points Xu, ..., ZM & such that the displaced embedding vectors {f,();)), M are linearly independent. Let N be the matrix with (x,) as columns and N' be the matrix with (x,) as columns. Notice that, from the diversity condition N is an invertible matrix. Then
N = AN'.
(22)
Since any two matrices, B, C MXM have the property rank(BC) I min(rank(B), and N has rank equal to M, we have that necessarily also A and N' have rank M. In particular, A is an invertible matrix.
rank(C))
Next we prove that pr.g(y] x) = Pr Pr (y x). v(x, v) E xx = Bu(y) = A A-T - for A L --T''T invertible.
25

---

As before, we have that
f(x) g(y) log (xx = fxx) g'(() log (Zx(
(23)
holds for all x E X and all y E V. In particular, this equation holds for any specific x. From the diversity condition (Definition 2.1), we can choose M + 1 x's, , XM, such that the displaced embeddings (fo(x;)]==, are linearly independent. By subtracting the log-conditional distribution for the pivot X0 E X, we obtain for each x; E A' the following:
f(x,) g(y) (gyy) f(xo) g(v) + (xg( D1)) - log log(Z(x;;,)))
f'x;))] g (v) f'(x)) T g'()) + log( Z (xp. (x,,
(24)
f(x,) T g(y) f(xo)) g(y) f'(x)) Tgx.)))( f'(x)) Tg'(v) +
(25)
fx(x;) g(y) (x,)) T g'((() + Ci,
(26)
where
2'(xo:31) 7(xo:)))
2 (x:,
G log
+ log
(27)
Z'xx;,
Let N be the matrix with (xx) as columns, let N' be the matrix with (xx) as columns and let c be the vector with G2 as entries. Then, since N is invertible
N N g(y) = N'
g (3()) +
(28)
g(v) = N N gNy
(29)
Since we found Equation (22) that A invertible, we have that
N' A'
(30)
Therefore, we get
g(y) N N'T - g'v) T
(31)
g(y) N (A-'N) g(()) T
(32)
g(y) N N A +N-
(33)
g(y) = A (g)())
(34)
where b = N-Tc. c. So, considering the displaced unembedding vectors, get:
go( g(( () g(vo) A T g')) + b A- -g'o) - b T g)())
(35)
(36)
A-T
(37)
This proves the claim.
26

---

C Models close in KL divergence can have dissimilar representations details
C.1 KL goes to Zero but Representations are Dissimilar
Here, we restate Theorem 3.1 in full detail:
Theorem (Formal statement of Theorem 3.1). Let (f, 8p) E θ be model with representations in RRA,, Assume M 2 2. Let k == [ be the total number of unembeddings and assume k 2 M + 1. Assume the model satisfies the the following requirements:
The unembeddings have fixed norm, that is [[g)(1))]l (1 ps for all yi: yj V where E RR.
ii) For every E X, let there be one element Yj E y for which the cosine similarity cos()(x;), Bp (Us)) > and cfs(f(x;); (,))))))) cos(f(x)),glu) g(ye)) for all ye # Yj.
Let (f', k,) be model which also satisfies i) and ii) and which is related to (f, {p) in the following way:
iii) For every E [|f'(xx)]l [[f(x;))] c(x;))
iv) For yj argmaxyex(p((/lx)) we have that the angle between f' f(;;) and g% is equal the angle between f(x;) and gp(U;), in particular, cos(f(x)). 8,(())) cos( (x)),,(
Then, making pairs of models satiisfying assumptions i) to and with increasing values of p gives us a sequence of pairs of models for which we have:
dKL dKL(Pr.g. gr. PP sx)) + 0 for p p
(38)
Also, it is possible to construct a model (f',g)) E θ which satisfies the requirements above, but where as - oc, the embeddings stay fixed and (xx is not an invertible linear transformation of f(x).
Proof. Let the models (f, gp), ( (f', .p) E be as described above. Below, we use the shorthand p = pr.gr and p' = pf'.""
We prove the results in two parts. (1) We show that, for any two models satisfying the requirements to iv) above, we get that dkr. (p, p') 0 as م (2) We then specifically show how construct model gz) and model (f', gy), which satisfies the requirements i) to iv) but where as p - oc, the embeddings stay fixed and f' fx(x is not an invertible linear transformation of f(x).
(I) Since (KI(p(y (l)ppl)))))l(((() (Ip((x)x.) + 0 for all x; E X' implies that (p, p') - O, we will show that DKL (p(y) x.) llp' (y)x,)) + for all x; X. Fix x; X. For notational brevity, we denote with e, E R+ the value of the norm of the embeddings on X,, i.e. |[f(x:)l [[f'(x;)) = c(x.) (assumption ii))). We then consider the KL divergence
DRL( = Σ (() log
Plu, x.)
(39)
p' xx)
Since (pL(p())x))l (l) involves a sum over j. DKL (p(y)x/ xx) Ilpp (y/ (x) will go to zero if cach term in the sum goes zero. To show that each term of this goes to zero, we first consider the term for yj = arg max ye> (p(y) x))) In this case, we have that
exp(f(x.) T (x))))
plus x;)
(40)
Σk exp(f(x;)) (x()) (x)(())
=(1+ 1+
Σ exp(f(xx)) go(() f(x,) ((xx))T,((,))
(41)
E=1
27

---

From iii) we have ||f(x))l C2 and from i) we have that l|g. (uy)l We can therefore write
f(x:) Ep(U;) cos (f(x.), (o(3))) Gi p,
(42)
and substituting this into Equation (41), see that
p(yj(x,) = (1+
Σ
exp (os (cos(f(x;), co)(((xx)) Bo (43)
E=
Now since by assumption ii) cos(f((;) go (v:)) > 0 and for all ye + y)
cos(f(x;) g, cos(((x)). (be)))
(44)
we have that
go)((()))))) < O,
(45)
Taking the limit in A. we get
exp(c- p (cos(f(xx), co
for p oo
(46)
Since every term of the sum in Equation (43) goes to zero for p - co, the whole sum goes zero, which means that from Equation (43) we get that
x)))(x;)
1+0 for p-00.
(47)
Similarly, we also have that
p' (W]]lx)) for p 00.
(48)
This means that for Vj Aargnaxxyyy p(y xx), from combining Equation (47), Equation (48) and Equation (39) we get that
p(y,(x;) 1- log 0 for - 00.
px,, log
(49)
p' (2)))(x))
Next, we consider the case where yj f argmax p(y( In this case, we consider
p(u)(x;)
p(ys|x) log
- log (1)((())x,))
(50)
p' (x:) -((,)xx)(f(xx)) go(U;) - -log 7(x:) f'(x;) (0,,)),,(())) + log 7. 7'(x:)
1,((5))
(51)
p(u,)lx)(((x))) (v,)) - fx(x)) T (1,)) + p(u, (x) log 6'(x:)
(52)
Z(x;)
We will consider the two terms in Equation (52) separately. First, we consider
p(u,)xx(((xx: (T(x))TTT(T))) - fxxx,)
T (u))).
(53)
Similarly as for Equation (42), we have
f(x,) T 8o(2;) =
c4 p.
(54)
We use this to rewrite Equation (53) and see that
f(x,) (vs)) - f((x;) T (v,)) = (fos(f(x,)) (u;)) a4 (os((f()x))))))) (, 9 M
(55)
= |q p. (xx((f()),, (x
(56)
< 2c p.
(57)
This gives us that
p(vxlx))(((x)) (p(())
(B, - I' (xxx)) gy (w:)) < 2cpp p(u)(x))
(58)
28

---

Since |=| + O implies that x: - 0 and we have shown that the absolute values of the term is upper bounded by 2cpp- p(y,)x,), we will show that the term goes to zero by showing that 2cpp ply x;) goes to zero. We see that
2G P exp(f((xx; TT (y()))
pcspp(u))x))
(59)
E exp(f(xx.) [B,(ee)
20 p
(60)
1+ Ei=1, exp(f(x,) 8o (ve) - f(xx) T 8D((s))
20, P
(61)
1+ Ei=1,e exp(c p 8x (0)
exp (a p (cos((((x)) (v(()) co 8o (w:))))
= 2cg
1/0+
Σ
(62)
E=,,
Consider now y, = argmax ply|x;), recall we have Yj + yr- Considering this yr. we have by assumption ii) that cos(f(x;), Ep (Vr)) > and
cos (f (xx ()(())
> 0,
(63)
Since for a E R we have (x
for xc, for this yr. we have that exp(e; p (cos((((x)) 8, cos(ff (,
Do for p - xc.
(64)
We can now define the denominator of Equation (62) as
1
exp(cap(oos((((x)_..
(
d(p)
P
(65)
k
(xp(c((coo((((x)) g0 (w()) cos(ffx;)) 8o
+
P
8=1, Efjr
and see that the first term of Equation (65) goes to zero, the second term goes to infinity Equation (64) and the third term is always positive since exp(x: is always positive. Therefore we have that
d(p) for p -
(66)
which means that
2c, d(p)
2cpp plu, (x.)
- for p
(67)
Thus, get for the first term of Equation (52) that
p (f(x;) go (yr) - f' (xxx) ( (v,)) + for - 00
(68)
which was the desired result. We now consider the second term in Equation (52)
P(U, (x) log 7'(x;)
(69)
Z(x;)
We that
Z'(x;)
exp(cos((((x)))))))))))))) (y))) e4 p)
(70)
2(x:)
m=1 exp(cco(p((x)) 8% (ym)) Cs
Σ Di exp(1 Gi p)
(71)
Σm exp( -1 11.c:-1 .P)
expla p)
(72)
exp(-c p)
= exp(2c p).
(73)
29

---

Which means we have that
(xx)
p(U)(x;) log [(x,)
S pluy x;) log exp(22 p) = 2cap p(u)(xx).
(74)
We also have that
2'(x:)
Σp l- (xp(cos((((x)))gg())) c.- p)
(75)
((x,)
m= exp(cos(((xx)) (ym) pp)
D exp( 1.. ...
(76)
Em=1 exp(1 Q: PP)
exp(-cc p)
(77)
explc: p)
exp( -2G pP)
(78)
Which means have that
Z'(x,)
p(y) log
Z(x,) 2 p(y))xx) log exp(-2c; p)
(79)
2cpp p(u)(xx).
(80)
which gives us
Z'(x:)
p(y xx) log
S 2cp p(us(x))
(81)
Z(x,)
Now Equation (74) and Equation (81) together give us that
Z'(x;) (xx)
p(y, (x:) log
S 2cpp- xx))
(82)
Thus, we have upper bounded the absolute value of the term with 2c,p- p(y, xx)) like in Equation (58). Using again that 2c,p- p(y)(x;) (x;) + 0 for p - (Equation (67)), get that
p xx) log 2'(x;)
for p - co.
(83)
7(x:)
Since both terms in the sum (Equation (52)) go zero for - oc, the entire sum goes to zero and for y # argmaxyey p(y),,), we have that
p(u X+) p'( x4)
p(U,(x) (x4) log
+ 0 for p 00.
(84)
Since we now have this result for both y, argmaxye yey p(v(x;) and y * argmax p(y|x;), the sum over all j in DKL (p(y)x)) llp (x/x)) (Equation (39)) also goes to zero. Thus, Vx, E X.
DKL
0
(85)
which means that
(p. p¹)
(r/x))] + 0 for - 00
(86)
proving the first claim of the theorem.
(2) We now present two constructions which satisfy the requirements i) to iv) above, but where f'(x) is not an invertible linear transformation of fx)) In the following, we use e; to denote the i-th unit vector in M whose i-th component is equal to 1, and all other components are equal to zero
Construction for 1: 2 M + 2. For i = 1,.... M, let (y:)) and Bp (yM+2)
e;- Let (y(+++1) = -p e
-p- e2. For any remaining labels with index > M 2, let them be such #g (y)) for j * l and more than x //2 radian angle away from é2, -é1 and -62. Let that Bp (Uj) 7
{x,, 100 n=1 X and (xm) ) m= C X be two sets of inputs. We define
cos((T (1 sin( π
(1
f(x,))
(87)
30

---

Now (f(x,)) is sequence with a well-defined finite limit, which we for case of reference shall call a
COs S sin 3et 1
lim f(xn) = n-000
(88)
We now define f for the other set.
cos( sin
4ym
f(xm)=
(89)
(f(xm))mm mm is now sequence with the same limit, that is,
cos( 4 sin
lim f(xm)
0
=a
(90)
For every remaining x X. construct f(x) such that there exists a label yy ) such that we have the cosine similarities cos(f(x) gy())) and cos(f(x), go (11)) cos(f(x), (y(e)) for ye # yj- With this (f, g) satisfies i) and ii).
We now construct the second model (f", (f'.g'). Let (v:) (u.) for 1..... . and M 2. Let g (UM+1) P e2 and (3a+22) -p- p- e1. Note that swapped the unembeddings for g' of yMs+1 and VM+2 compared to g. For all x E X and the corresponding yjj argmax p(y /x) such that y₁ + VM+1, MM422. let (x(
f(x). However, for all x X and the corresponding Yy = argmax p(y(x) such that y/ = M + 1. let [[f((x)]| = [[(]]] and let f'xx) be rotated counterclockwise with respect to the first two axes (i.c., the first
components of the representa-
tions) such that cos (x),,(( (1;)) = cos(f(x) g, (U,)) (), Also, for all x; E X and the corresponding
V = ygy p(y)x;) such that yy = M + 2, let [x,))x,)] = [[f(x,)] and let (xx, be rotated clockwise with respect to the first two axes such that (os(f((x)) g (31)) = cos cos(f) (1,)). This construction satisfies the requirements i) iv).
We will show that f' f'(x) is not an invertible linear transformation fxx by showing that any function + R.M with T T(f(x)) f'(x), Vx E X' cannot be continuous.
T: M
Let RMM. M be such that (f((x) (x(),, Vx X. We recall that function h: RM is by definition continuous at E Rd if
RM
hxx) + h(c) for x x
(91)
Therefore, we consider the limits of the sequences {T (f((x)))) and (f(((x,)) 1m We first note that by construction, we have for the rotation matrix
cos
sin
sin
cos
0
R
(92)
that
f((x,) Rx(xx,.
(93)
Which means that since Tff(x.)) = f'(xn), we have that
3x cos(
2
sin
(1 -)
T(f(x,)) Rff(x,)
0
(94)
31

---

and we see that when - 20,
cos Smt 2 sin det 2
T(((xn)) - b.
for f(xn) + a
(95)
However, we also have that Tf((Xm)) f' (xm) f(xm). This means that for m oo, we have = ((((xm)) + for f(xm)
(96)
Now since for b from Equation (95), we have b + a. cannot be continuous. In particular, it cannot be an invertible linear transformation, and thus (x( not an invertible linear transformation of fxx).
Note that if either f' is injective, then both f and f' be smooth. However, if both f and f' are injective, then either f or f' has to be non-continuous.
Notice also that transformation. See for example Fig. which has been constructed by permuting multiple labels.
permute additional labels to bring the embeddings further from linear
Construction for k M + 1. For = 1.... M, let gp (3() p e; and (NM++))) -p- e Let
[xn]== 100 C X' and (xm
1000 m=1 C X be two sets of inputs. We define
cos(T (1 sin( π (1
f(xx)
(97)
Now (f(x,))ae is sequence with a well-defined finite limit, which we for ease of reference shall call a
cos sin da
lim f(xn)= =
0
(98)
We now define f for the other set.
cos( sin
4ym
37
f(xm)
(99)
(f( is now sequence with the same limit, that is,
cco(( 3r
3x sin I
lim f(xm) m=+00
a
(100)
For every remaining x, E X, construct f(x:) such that there exists label yj E y such that we have the cosine similarities cos(f(x;), (y;))) ( > 0 and (os(fx)),g,(()) > cos(f(x;;) By())) for ut F yj- With this gf,g) satisfies i) and ii).
We now construct the second model (f',g'). Let (v)) go():) for i = 1...., M and let g ()M+1) = e2. For all X; E X' and the corresponding y₁ = argmax p(y|x;) such that UJ # yM+1, let (xx)) = f(x;). For all x; E A' and the corresponding = argmax,e> p(y) xx.) such that = yMs-1, construct fx(x, from fxx;) by rotating it counterclockwise with respect to the first two axes such that cos(f' (x;), g (u,)) cos (f((f)x), g,(u))) This construction satisfies the requirements i) iv) and by the same argument as used for the with k > M 2, f' f'(x) is not invertible linear transformation of f(x).
32

---

C.2
Loss goes to Zero but Representations are Dissimilar
We denote the negative log- lellikilihood for model p with respect data distribution pD as:
NLLpp (p) == ((x.)))pll-oog
x)]-
(101)
In the following. it is useful to establish the link between the negative log reg-likelhhodd and the DKL divergence with the empirical distribution. To this end, we introduce the empirical distribution over inputs and outputs pD(x, y), whose marginal on the input is given by pr (x). This can be rewritten
as:
pD(x, y) = PD(y x))p
(102)
Notice that the conditional distribution underlies how labels are associated with the input. We can connect the negative log likelihood the KL divergence with the empirical distribution, a well-known fact in the literature, as shown in the next Lemma:
Lemma C.I. Let pD(x, y) = PD (y x (x)) be the ground-truth data distribution. Let ply (x) be model such that, for all in the support of pD(x), we have poly x) & ply x). Define the negative likelihhodd as NLI pp (p) == Bxxuipp (- -loogry x)). Then:
NLLpp (p) dKL (PD.P) + Ex-ro [Hl (pD(y x))],
(103)
where H(q) == -Ba [lo (<x) denotes the entropy of distribution q.
Proof.
From the definition of KL divergence and entropy:
DKL (PD(y) x))|p(y x)) yyr (ulx) (ogp x) - logp(y x)]
(104)
H (pD (y x)) + Bympp((lx) I- log x)]
(105)
Moving the entropy to the other side and taking expectations w.r.t. px(x). we get:
(xx,)>>pp logg p(y x)] = Ex-pp [DKL (pD(y x)]lp(y) x))] + Ex-pp [H (PD((y x))] (106)
The statement follows by the definition of dkt. (Equation (6)) and NLL.
Notice that, when the conditional distribution pD (y x) degenerate, i.e., to each x E X' a single E y is associated, the expectation of the conditional Shannon entropy vanishes.
Next, we restate Corollary 3.2 and prove in full details:
Corollary (Formal version of Corollary 3.2), Assume we have a data distribution where only one label is associated with each unique input. Let (f, g) O be model with representations in R.M. Assume M 2 2. Let k == | be the total number of unembeddings and assume k: 2 M + 1. Assume the model satisfies the following requirements:
i) The unembeddings have fixed norm, that is Ig(y)))l = (lg(y))) = P. for all yı Uj E y, where R
ii) For every E X, let there be one element y E ) for which the cosine similarity cos(f(x)), g(U;)) > and cos(f(x,), g(y,)) > cos(f(x;)) g(y)) for all ye + yy. Also, assume Yj is the only label associated with x,.
Let g')) be a model which also satisfies i) and ii) and which related to (f, g) in the following way:
iii) For every x; E Ex [|P'(x;)]1 [[(x))]l
iv) For ý = argmax (p(y)x)) have that the angle between f'(x:) and g'(g) is equal to the angle between f(x;) and g(g), particular, (fs((( g()) cos( (f' (x;) ) g ()))
Then, making sequence of pairs of models indexed by p. we have NLL(PF.) + 0 for - 0o
(107)
and
NLL(Pr + 0 for + 00.
(108)
Here & denotes absolute continuity.
33

---

Proof. Below, we use the shorthand = Pr.g and p' = g'. We make use of the same calculations as for the proof of Theorem 3.1. We consider p(y [x,) for yj the label assigned to x, according to the data. We denote with E IRt the value of the norm of the embeddings on x,, i.e., ||f(x:)]
||f' (xx)ll = Cç- this case we have that
plus x,) = exp(f(x,) g(v,))
(109)
Ee= exp(f(x;) g(ve))
(110)
1+ Lt=re ( g((e)) - ((xx))
By construction, [|f(x:)] = C.. We that
f(x,) "((s)) = cos(f(x;). a.
(111)
and see that
p())(x;)=
1
exp(c: p (cos((()))).((())
(112)
(=1,6p
Now since by construction, cos(f (f( ((,))) and for all {
cos(f((x:) (3;)) cos(f(x)).g(v)))
(113)
we have for all & that
cos(f(x),,g({)) cos(((()). <0
(114)
and
exp(c. p (ooo(f(x)), (fxx) g(v,) + for p (115)
which means that combined with Equation (1 12) we get
1 for p
p(U)(x;)
(116)
+
Similarly, we have that Therefore, have for the NLL:
p' p'())1x;) + for P - 00.
(117)
NLL(p) = Σ logp(us(x:) +0 for p x
(118)
and
NLL(D') Σ
logp' (,, + for p 00.
(119)
34

---

D Partial Least Squares (PLS) and Canonical Correlation Analysis (CCA)
Partial least squares (PI.) algorithms techniques that, for two random variables z and w, derive the cross-covariance matrix Ezw using latent variables. We give here a short description of the two variants that are relevant to our work. more detailed description can be found in [66].
In our case, we will work with random variables of the same dimension. So in the following, we consider z, E I We will mostly work with the centered and normalized versions of the random variables. So if 2, is the i'th I'th dimension of the original random variable, we will consider
21 E(2
2/
(120)
Var
D.I PLS-SVD
PLS-SVD [54, 60) a variant of PI.S where the latent vectors are simply the left. and right singular vectors in the singular value decomposition of the cross-covariance matrix, Ezww
In Algorithm 1, we report the algorithm in the case where we have n samples of our M-ddimensional random variables, In this case, we work with (n x .M) matrices Z, W, where each row is sample.
The aim of the algorithm is to get projection vectors (r)) v(r) such that Covzw (uu(), Wr(r) is || = The original algorithm includes
as large as possible under the constraint that | choice of signs for the covariances. In this version of the algorithm, we have chosen all signs to be positive. R. > 0 denotes the maximal rank of the algorithm.
Algorithm 1 Iterative SVD Projection Extraction from Cross-Covariance Matrix
1: Set f-
2: Center and scale Z and W
3: Compute cross-covariance matrix: + Z W
4: Set C(I) + C
5: while ((r) * and r < do
6: Perform SVD: ((+) UDVT
7: Extract leading singular vectors: ui (r) + first column of U, v( <- first column of V
8: Save (r) and v as the r-th projection vectors
9: Let + leading singular value of ((+)
10: Update matrix: ((++1) <- c(r) o
(M) ())
11:
r +1
Considering this algorithm, we see that the projection vectors we get, (((),, v(r), are exactly the first m singular vectors Z W corresponding to the largest m singular values, where m the value of at the exit point. Note that this relates to principal component analysis (PCA) in the sense that when doing PCA, one uses the singular value decomposition that capture the variance in dataset (a random variable). When doing PLS-SVD, one uses singular value decomposition of the cross-covariance matrix to obtain components that capture the covariance between two random variables.
the covariance matrix to obtain components
D.2 Canonical Correlation Analysis
Canonical Correlation Analysis (CCA) [22] seeks pairs of unit-length vectors u, E Rd and Va Rdw (E = 1,.... TT) that maximize the correlation between the one- mne-dimensional projections u, z and v. w, subject to mutual orthogonality of successive directions. In other words, CCA finds the singular
value decomposition of the matrix Q Σn Ezw Ewer where Ezw is cross-covariance matrix and Exz, Eww are covariance matrices. Equivalently, Q the cross-covariance of the whitened variables ΣnA and Σ w, which have identity covariance. Whitening removes the influence of marginal variances, so the eigenvalues found by SVD are the canonical correlations. The results of CCA and PLS-SVD will differ if there are high covariances between the 2,'s or the w;'s.
We can use CCA to define similarity measure between random variables and w, known as the mean canonical correlation mccA (z, w), which is the mean of the maxima correlations for the
35

---

centered and rescaled random variables. Because of these centering and rescaling operations, MCCA will be invariant any invertible linear transformation.
E When Closeness in Distribution Implies Representational Similarity Details
We here include the technical details of Section 4.
E.1
The Connection Between Representations and Log-Likelihhode
Let (f.g) E + be model satisfying the diversity condition (Definition 2.1) and consider XMs X and y1...,, UM E y such that both the vectors (fo(x)) and (g)( are linearly independent. We recall some notation for use in the Lemma. We denote with L and N the following matrices:
L (B.
N (fo(x))....... ................................ fo(xM))
(121)
We recall the definition of the following functions:
thx (1 = [Varx(logp((sxx)) log (ogg((u(x)) and (x(x)))) = /\arlllogp((/x)) - log xx)
(122)
We also denote with S', D, D' MMM the diagonal matrices with entries Sa (x(ppp)) D
(,(x, 5/2 ( Djs म. We now have everything need state Lemma
relating any two model representations.
Here, we provide the full statement for Lemma 4.1 and prove it:
Lemma (Complete version of Lemma 4.1). For any (f,g) (f', θ satisfying the diversity
condition (Definition 2. Let = L L-TS-'S''' and hy(x) = L-- --e where the i-th entry of Ey (x) is given by
log & x) log Pr (20lx)
log x (U: log Pe x (yol xx)
Ey, (x)
(xx pr,x)
(x(1i pr
B D D-DD-'DA ''N'' and high)) = N--N-- T DD' (x),, where the j-th entry of exly) is given by
(x) = Pr (v)x)) log (ulxo) Pogrr'.e (,)xx) log Pr' (y) Xn)
(x,; pr.g)
(y(x;; Pr pr
Then we have:
f(x) Af hx(x), g(y) Bg'(y) + hg
(123)
Proof. Using the definition of E(x), we can relate the embeddings of the two models as follows:
SLf(x) S'L''' SLF - (')'
(124)
S'L''( (x).
(125)
Therefore, multiplying by the inverse of S and of L, we get
f(x) L ---L--SS''''ER fx)) + L-TS--ee,x
(126)
Af (x) hr(x).
(127)
With similar steps, obtain the relation between the unembeddings of the two models, proving Equation (123).
Notice that the error term ey (x) is function of and comprises the y, from the diversity condition (Definition 2.1). Also, B results from the product of the scaling matrices and those constructed from the diversity condition applied to f. hg(v) depends (x.), which is function of y using the x,'s from the diversity condition.
The following corollary connects Lemma 4.1 to the identifiability results of Theorem 2.2:
36

---

Corollary E.1. Under the same assumptions of Lemma 4.1, we assume that the distributions of the models (f,g), (f', .g) are equal, then we get (as in Theorem 2.2):
f(x) = (x
(128)
So Bo = A 8%(0)
(129)
where A LL.
Proof. Below, we use the shorthand p = pr.g and p' pp When the two models entail the same distribution, we get that
(x(())) = Warx(logp log 1ogg(1o = (x((x)))
(130)
which means that
Cy. (x) = f(x) T (o((:)
f(xx) (1))
Varx [ f(x))
V Varx [L," [(()]
log pr. ( x)
log Pr. pr.../mmll (vnlx)
log pr xx' (y6 xx log pr (yol xx)
(x(1/s pr.g)
2/ (Y6: Pr,,e')
(log (r.( log (V)lx) log Pe Pr (volx) log Pr.g ( (v01x)) xx)
Vx (y4: pr.g)
0.
Also, since
Varx f(x]]
Varx log log p(yo x
(131)
Varx log p/ (wixx) log p' (wo x)]
(132)
Varx f(x))
(133)
and S is diagonal matrix, we have that
-'
(134)
Finally, we get
A A = -
(135)
and since the error term vanishes, we obtain:
f(x) = Af'
(136)
showing the identifiability of the embeddings. The result for the unembeddings proceeds in a similar way, by noticing that also Ex is zero, therefore getting the same result of Theorem 2.2.
The result of Lemma 4.1 also gives us the following corollary, showing connection between the distributions and the representations:
Corollary E.2. Under the same assumptions as in Lemma 4.1, we have:
Varx (x,, = 2(1 Cor([L,f(x) (x)).
(137)
Proof. We can write
SL f(x S'I ('x) S'L
S''''T((x (()).
(138)
So setting
ey SL -
(139)
gives 6, (x) in Equation (8). Since L and S are invertible, the inverses exist, and can left multiply with these inverses. We now consider the variance of cyi (x). Recall that
Var Zarzule [2 + w| = Var + Var [w] + 2 Cov Covzu [2, w]
(140)
37

---

This results in
L,f(x)
Iff((x
Varxlex = Varxl
(141)
v [a
[ar[L;
L,f(x)
I/F((x
Vaax
Varxl
(142)
/Vax [xx(x)]
VVarx [x)
1,f(x)
2 xovxx
(143)
[ar[L,
[ar(L[F((
L,f(x)
[xf(xx
2 xxx
(144)
= 2(1 - (x)
(145)
where we use Equation (140) for the second equality and that Var, Var=(z/std(z)) in the third equality. We thus have the result.
There three things that are important from this result.
Firstly, as seen in Corollary E.I, when the distributions are equal, the error term becomes zero and we are back in the case of Theorem 2.2.
Secondly, the shape of the eтrоr term suggests what distance should measure if we want it to give guarantee for how far representations are from being invertible linear transformations of each other. We notice that Equation (8) can be rewritten as
log loy
log p' (3: x)
Eys (x)
(146)
Var log ((y x)
log x)))
V Varx llogr (y: x) log (1o0l))
log p' (yolz) xx)
logp(yol xx)
(147)
Varx (log (1s xx) log
(oggrloo log logr((llx)) x)]
that is. entirely in terms of the logarithm of distributions. We also see that if Cz. (x) is constant (or equivalently, Varlcy, (x)] 0), the embeddings will be invertible linear transformations of each other.
Thirdly, Lemma 1 fits with the observation made in Section 3. The KL divergence depends the difference of the logarithms of distributions, but it also multiplies that difference by the likelihood of the label. Which means that can have relatively large difference logarithms of distributions, even though the KL divergence would result in a small value.
A similar result for the unembeddings can be found below.
Corollary E.3. Under the same assumptions as in Lemma 4.1, we have:
Var, [Exi (v)] 2(1 (or) (v))).
(148)
Proof. We can write
SN g(y) = S'N'T S'N'T''(') + SN g(y) - S'N g'().
(149)
So setting
(x) SN g(y) -
(150)
gives Ex) as the error term for the unembeddings.
38

---

We now consider the variance of Ex, (t).
Nig(v)
Ngg' (v)
Vary|exi(3)] Vary
(151)
Varr (1)(
Var, (')
(1g
N'g'(y)
Varyl
Varyl
(152)
V Var,, (
Var,
N.g(v)
Níg(u)) (v)
Covvvl
(153)
VVar [N))(
Var,, (aay(
N,g(y)
N'g'(y)
2 2 Cov vel
(154)
Varr, (N,g))))())))
[Vary|Ngg((v
2(1 - Corr[N.g(y), N,g, (r)'
(155)
which gives us the result.
E.2 Distance Between Distributions
Recall that we the following notation:
(x: = v Warx ((x())) log p(yo x)] and (x;;));))) = / Varr, (Varyloog(uxxs) - log (()(x)))
We restate Definition 4.4 and then prove its distance metric.
Definition 4.4 (Log-likelihood variance distance between distributions). For any two probability distributions p. p' for which there exists a common choice of XLLV C X and VLLV C y such that they both satisfy Assumption 4.3, for a fixed E R and by considering the following terms:
logp(y/x)
log (v/2)
logp(vu|>)
log p/ (yo1x)
ti max
Varx
Varx
VEV Yoau
(x(()))
Vxx (x(xpp)
(x((Gp)
<x((1:
log logp(u(x)
logp'(y(x)
log logp(y(Xo)
log (wlo()))x))
to =
max XEXavV(Xe)
Var,
(x;:p))
(x;p'))
VV Var, y
(x(x)p)
(x;p)))))
03
max
Je' (e(( (x(x))')), 44 ==
max
(y (x;
(x;1'))
XE (xx)
the log- likelikeooood variance (LLV) distance between p and p' is given by
d'vv(p,p)) = max (11, f22, Ata, At4}
(10)
To show that d is distance metrie between sets of conditional probability distributions, have to prove that: i) it is non-negative, ii) it is zero if and only if the models are equal (that is, the distributions equal for all labels and all inputs) iii) it is symmetric and iv) it satisfies the triangle inequality.
Proof. i) Since variance is non-negative, the first two terms non-negative. The second two terms are absolute values, so also non-negative. Thus, the maximum of these four terms is also non-negative.
ii) From the expression of d if the distributions are equal for all x E X' and all y E V, then the distance becomes zero.
Now, assume the distance is zero. Then, in particular, 44 = 0. So we have
V VVar,llog p(v| [anz]|log((/x;))
- log p(y|x))] V
Var, Var, ((((x)) - log xo]]
(156)
for all X, E XLIV \ {xo}. So since to O, we have
Var, [log p(y1x,) log (ylx;) x1 0,
(157)
for all x, E XLLV. This means that for each xj, we have that the log difference of the distributions is a constant for all ys E YLiV, and so
log x,) - log (xjxx)) Cj.
(158)
39

---

for all ys. This means that
log (()l)(()))))))) log (U: xx) + c
(159)
p(ui(z;) (ys xz)) exp(e,)
(160)
Now since and p' are probability distributions over the yi's, 's. we have
1 Σ plullx)) exp(ex) Σ p()))))) exp(e,)
(161)
This means that p(y|x;) = (x)x)) for all y D' and for xh E VLLV. Since we have 1 13 we get that
0,
Varx pogp((u /x) - log (o (w/[x)] 0,
(162)
for all ys E VLLV and all inputs E X. Thus, we have
log xx) Jogp((y:
(163)
for every choice of y, except one left out of the label set and all x. This results
p xx) p' (v/(x) exp(c.) for all x E X. In particular, this is true for x, E XLLV- So we have
(164)
p(yi(x;) p' (ys (x)) exp(ci) p' (ıx(x;)
(165)
and thus, exp(c) = for all E Vaav Since the probability of the last label, yh is fixed by the other labels, i.e.
p(ux(x) Ex(ss)x
(166)
this gives us that
pluil)) = 1/ (vilx)
(167)
for all ys E y and all x X.
iii) d is symmetric because it depends on variances and on absolute values, which are symmetric.
iv) We prove that di satisfies the triangle inequality by showing that all terms in it satisfy the triangle inequality. Assume we have three random variables X, Y. 2, then
VarlX 2] Var[X Y + 2
(168)
WarlX Y] + VarlY 2| + 2Cov -Yy - Z
(169)
S Var(X r1 VarlY Z + Var - Y1 VarlY Z
(170)
Varlx - Yl + VarlY Z])
(171)
Var[X Y] Varly -2].
(172)
So the square root of the variance of differences of random variables satisfies the triangle inequality. Taking the maximum also preserves the triangle inequality. Therefore, t and f22 satisfy the triangle inequality. Concerning the last two terms and t4, see that for models p. p', p" we have
Varg/log p(ylx,) log((())x))
Var, flog poogp''((iz)) (vlx,) log (v/xx))
Var, Var, p(ylx,) (ogp((/xx))
Var,, [log p* (v|x,) - log pa (8/xx)]
Var, (vlx;) log (v(x)))))))
v Var, flog (v1x)) log o' (v)x*))
Var, p(ulxx) logp(v/xx)
Var, plog - log (gg
Var, [oog( (v) x)) - log logx"()(xx))
(vlx)) log (v|xx))
So the second two terms also satisfy the triangle inequality. Thus, the sum of these four terms satisfies the triangle inequality.
40

---

E.3 Implementation of the Log-Likelihood Variance Distance
Here, we collect the implementation details of d'v-v set the weighting constant to 10-3, To choose the best pivot yo E Y and the left-out label yæ E V, we evaluate l for all possible choices of (yo. )2), then select those giving the smallest value of dıyy. For the imput set &LLV, we use M + 1 inputs. In experiments, the representational dimension is 2, draw 200 samples of sets from X' containing inputs, and choose the set giving the smallest tz.
collect 3 inputs. We randomly
In our experiments, we only make pairwise comparisons of models. Notice that, in the case where one would like compare three or more models, the same pivot yo. left-out label ye. and input set XLIV must be chosen for all models.
E.4 Using PLS SVD to Define Dissimilarity Measure
We restate Definition 4.5 and then prove some important properties.
Definition 4.5 (PLS SVD distance between vectors). Let w be two M-dimensional random vectors, and define z', w' by standardizing their components: 2/2 = (2, E(2z)) std(2,). w
(w, - B(wn)) std(w)). Let M. and [V be the left and right singular vectors of the cross- covariance matrix Ez'w' We define
msvD(z, w) = ΣM Cove lu, z', vz,v w'], dsvD(z, w) =1 (2.
(12)
show that this measure zero from a vector to itself and invariant to multiplications of scaled variables with orthonormal matrices. Before proceeding, we recall the following property of the trace two matrices:
tr (r(AB) = (r(BA).
(173)
In the next Proposition, we prove that dz, 22
Proposition E.4, Let z be an M M-dimensional random variable and assume that the covariance matrix Ex'r of the centered and scaled variable z' is non-singular. Then
M
mSvD(Z,2) () = M Σ
Covezzuue [u z', u, z' =
(174)
where is the i'th singular vector of Exe.
Proof. Let
Exe U
(175)
be the singular value decomposition of the covariance matrix of z' with itself. Then, using Equa- tion (173), we have that
tr tr(xx)= = (r(DU") (U tr(D) Σ oi
(176)
where are the singular values of Eziz. Since Var[2!] = for all i {1,..., M). we also have that
tr (EE')) Varz(2] M.
(177)
Combining the two results, we get
M
Σ
σ; = M.
(178)
41

---

Therefore, the mean of the covariances becomes
1
msvD 2)
Cov [u, 2', u, z']
(179)
M
1
tr(UT Cov (o 2
(180)
M
M
1
M
0
=1
(181)
M
M
which proves the claim.
Next, we prove the invariance to translation and orthogonal transformations the members msvD: Proposition E.5. Let z and w be M M-dimensional random variables. Then, the measure
msvD (z, w)
M Σ Covz'w (u z', v. [w]
(182)
invariant to translations and multiplications with an orthonormal matrix after the scaling.
Proof. Invariance translations. Since MsVD (z w) is based covariances, and covariances invariant to translations, (z, w) is invariant to translations.
Invariance to multiplication with orthonormal matrix. Assume T and R orthonormal matrices. Let = Tz' and k Rw'. Let the singular value decomposition Ex'w' be:
Σg w -UDV.
(183)
then
Σnk TUDV R
(184)
(TU)D(RV)T,
(185)
Since T and R are orthonormal, TU and RV are also orthonormal, so Cov [h, k] has the same singular values as wy' Thus, the measure is invariant to multiplications with orthonormal matrices (rotations) after the scaling.
F.5 Lower Bound on Mean PLS-SVD
To show how the probability distributions connected to the representations, need intermedi- ate result, where we bound męvD. To prove this bound, we will make use of Weyl's Incquality and therefore restate from [59].
Theorem E.6 (Weyl's Inequality). Let A be
matrix. Let A E. Let be the i'th singular value of A and let ãg be the i thth singular value of B (ordered from largest to smallest). Then
0
0z <
[||s
(186)
where ||-Ile is the spectral norm defined
max [|]|==
|Ev|
(187)
and I1- is the Euclidean vector norm. Or equivalemty::
El|= F (E),
(188)
where Tmaae (E) is the largest singular value of E.
We now state and prove bound MgvD-
42

---

Lemma E.7. Let z, w be two M-dimensional random vectors, and define z', w' by standardizing [[zz))/st((z;) std(2;), w/ = =(; /lw,l)/ std(ww, w;). Assume the M x M
their components: 2 (2; -
matrices Ez'z' and En'w' IR x are full-rank. Let {ur J and {vili=1 be the left and right singular vectors of E'. Then, the
the following bound holds:
M Covz'w'
M Varz w/ (2j w/l
msvD (Z, w)
I z. v. w 21
M
(189)
M
where Varz, ww] denotes the variance over the joint distribution over z' and w'.
Proof. For ease of notation, assume z and w are M -idimension random variables which are already centered and scaled. So Var[z;] for all components i I....,M M. Below we will make use of the covariance inequality, which says that for scalar variables 2, w,
Var,
Var [w] S Co w| S Var, Varw(w)
(190)
Let A == Ezw be the cross-covariance matrix of and w, and let B == Exz be the covariance matrix of z. Then
B = A + [B A), B A. We can write the i
(191)
and we define
j" 'th entry of as
E EEi,. = Cov [2] - Cov [24, wy] = Cov [25, 2j - wyl+
(192)
because of the bilinearity of the covariance, see section 13.2.7 of Adhikari and Pitman [1], "The Main Property: Bilinearity".
The singular value decompositions of A and B always exist and can be written follows:
A = UDVT, B = WEWT.
(193)
where U, V and W are orthonormal matrices and D and Σ are diagonal matrices containing the singular values. Note that since B is symmetric and positive definite, the left and right singular vectors coincide. Let o, be the i'th singular value of A (sorted from largest to smallest) and let a be the i'th singular value of B. Now Weyl's inequality E.6 gives us that
0 - ās < [||
(194)
Since the spectral norm of E is the largest singular value of E
E||: = σ max (E) and the frobenius norm is the square root of the sum of the squared singular values
(195)
E
of E V trace E E).
(196)
have that
E <
E trace E E).
(197)
Next, we bound the trace of E ETE:
E:
M
trace E E)
Covzw [24, w/i
(198)
M
v [2 V Varz, - wi))²
(199)
M Σ
M Maarz -
(200)
43

---

where in the first step we use the covariance inequality (Equation (190)) and in the second we use that Var[2k] 1 by construction. Combining the inequalities from Equations (194), (197) and (200)
we obtain
M
00 - 0:| S
MVarz, - wil:
(201)
which means that the difference between d; and og is most VIM M
M Mar,, wi). This
means that if or < os. we still have:
M o:
2 oi
Var,, w wil
(202)
and if o; 2 di, we also have
M MVarz,wla wi]
0.2 0:
(203)
V Ei M M Varla wi] 2 Let v, be the i'th column of V and recall that from the SVD of
since
we have
M
o, u Av. Covzw
Σ
U
Vaw!
(204)
[=1
Co [ut 2, w
(205)
because of the bilinearity of the covariance, Thus, Equation (202) gives us the bound
Covzw [ V. W 20:
M Marz, [[ wi]
(206)
and therefore
1 Σ m
msvD(Z, w)
Cov 2, w]] 2
M Marz. w/]
(207)
where we used that 1 EM 0 from Proposition E.4.
E.6
Bounding Representational Similarity with Distribution Distance
We can now use Definitions 4.4 and 4.5 and Lemma E.7 to show how a bound on the distance between probability distributions can give us a bound on the distance between representations. For the result below, let L, L' be defined in Theorem 2.2. Let (resp. N') be the matrix with columns (x) (resp. (x)). We denote with 21 = L f(x). 22 = L L'),'(x), wT == N NT g(y)) and W2 N gy)).
Theorem 4.7. Let (f,g), (f',g') E two models such that: (1) There exist XAAN C X and LLLN Y, consisting of pivot point and all labels but one, such that all L, L' and N, matrices constructed from these sets are invertible:" (2) Both pr.g and pr.g g satisfy Assumption 4.3 for
XLAV and VLIV; (3) The covariance matrices Σz,z, Ew, w. and the cross-covariance matrices Σ2,z, Ew,wa are non singular. Then, for any weighting constant E RR we have
d Pe
pr.ge)) g:) Se
dr ((f.g) (f'' < 2Me
(14)
16This is slightly stronger than the diversity condition (Definition 2.1) in the sense that we need diversity to hold for all sets using labels from VLLV and the chosen pivot point.
44

---

Proof. We start from
dsvD(L (L f(x), L [x))) mLvD(L (L)()x (x))
(208)
and consider the components L,, L/2 which are l'th row of L L"T, respectively. In the following. notice that
(1G Varx Trrx:logpuun (()))) log logp(())x) xx] Var [ff())
(209)
Using Lemma E.7, we have that
Lxf(x)
[xf(xx
(ovD(LL((x),L
Σ MVar
(210)
Var[L,
Var[L,f
Therefore, we have
M
Lxf(x)
(xf(()
dsvD ((X.T s
MVar
(211)
[fr(L,f(x)
[Vr[[[P[()]]
Considering the variance term, we can rewrite it as follows:
L,((X)
(x)
log pgpp(ylee (x) log ((vol
log p/ (w0
log p' (30 (x)
Var
Var
Var [
Var [
log p(gr(())2))
log (un(
Var
Var [x)
log p' (20 x))
log ( [z)
+ Var
Var [
v Var [xx)x]]
log p(m (r)
log (m)(x)
log p' (vo [x)
log p(y0 (x)
+ 2 Cov
/ar
/arlL (x)
Var [x)
[ar[L,
log
logl'(u((z))
Var
Var [x
Var
log p' (yul
log Dogp(yola)
+ Var
Var Lff
[[
log (ggr(yl
logg((m/l))
log
log logr(1o(2)
+ 2, Var
Var
/ar
[Var([LP((x)
[ar[L,f(x)]
By assumption, (p.((p.p)) SG, which means that for all E VLIV:
logp(u)(=)
log (11 (x)
Var
Se.
(212)
Var Lf(x)
Var
[
Hence, we obtain
M
L,f(x)
[x
dLvD(L f(x), (' <
Mar
(213)
[ar(LLi(x))
[arl(LPP(x
M(e² + 2
(214)
2 = 2Mc,
(215)
45

---

giving the first part of the result. Next, we consider
dN g(v),N N' g' g'()) mSvD (N') g(y). g'(v))
(216)
Let Nj, N be the j'th row of N N'T T Using Lemma E.7, we obtain:
M
N (gg(v)
Ny8((v)
mIvDD (N 8(v), g')))
MVar
(217)
Var
[g(()
/aar
This implies the following:
M
N,g(v)
Nig'(v) (v)
devD(N g(y). (00)) g'())) S
Σ
MVar
(218)
Var(N,g(y)
Var(N
Considering the variance term, we can rework follows:
N (gg(v)
(yg(())
log xj) logp(u/xo)
log (1/p/////))))
log (ulxo)
Var
Var
Var N (gv)
Var Var[N (')
v Var(N,g(w)
v Var((N,g(())
logp(y|xy)
log log''( (ylxy)
Var
Var N ((()
Var [i''(v)) (v)
log (3/xx)
log logp(y/x)) xb)
+ Var
Var(N/1'(11)
log (og(()))))
logr((y/x)
log (2/x))
log p(y)22))
+ + Cov
Varr(N IN g(y)
/VarLL [
Var NNg(())
log (ogp(ulx)) x1,)
logg'(ulx))
< Var
/arrNN
Var(NN (v))
log (w/ Xo)
logylvi Xo)
+ Var
NNr g'())
Varl N [g()]]
log p(y xj)
log (y/ x))
log (y/ Xo)
log p(y (xo)
+ 2
Var
Var
Var (N,((N))))
Var(N
Var N'g(N)
Var
Since by assumption d (p.p') p/) S e, this implies that l2 S and that for x, E XLLI:
log Hogp(ylx x,)
log (1) xx))
Var
Se
(219)
Var [gy)
Var ( (v))
and
Pogp(yl Xn)
log p' (y(x))
Var
Se.
(220)
Var [gg())
VarlN
46

---

Therefore, we get:
M
Ngg(v)
Njg'()
dsvD(N' g(u), N' g'())) (((()) =
MVar
(221)
/aa(NNg(u)
[ar[[N
M M(e 2 262)
(222)
1=1
2 2Mc
(223)
showing the second part of the result. Taking the maximum between Equation (215) and Equa- tion (223), we get the result of the claim.
To see how this result connects to how far the embedding functions f(x), f'(x) are from being invertible linear transformations of each other, notice that if L, and L are both invertible, then if there exists invertible linear transformation, B, such that L f(x) BL fx(x,, then also have invertible linear transformation A = - BL such that f(x) (x))
Since mxvD(xx y) is always non-negative, for this bound Theorem 4.7 to be non-vacuous, we need
1
dLvD(L (lv)(Lf(x),.I))
f(x), L' (x())) 2Mc <1
<< 2M
(224)
This means that for higher dimensional representations, need the variance of the differences of log-likelihan log-likelihoods to be smaller, if we want a guarantee from this result.
Next, we prove that the dissimilarity between representations induced by Theorem 4.7 is invariant to substituting the members with models that are ~u--quivalent.
Lemma E.8. For any two models (f. (f.g), (f', g') E O, and for any other model (f", (g')) E θ such
that (f" (f) ~ g), we have that:
(x), dLvD(L T (eTTT((xL, L (())) = dSvD P LT f(()), dSvD(N ( g'v), g g'(w) dN g') N' g'())).
(225)
(226)
Proof. The proof follows using the linear equivalence relation Theorem 2.2. For any two models (f,g) ~ (f",g") gg" we have that:
(v) f(x) (v) FF [x(),,
(227)
for all x E X' and E y. By considering M elements in V, we get
f((()) = L' ('))
(228)
where L' M are the matrices constructed with columns the vectors go(y) and g)
respectively. Therefore, we get Equation (225);
dL f(x), f'(x)) = dSvD (x),, f()). Lf()).
(229)
To obtain Equation (226), notice that with similar steps we can write
N gg(()) N' g g'()) +b.
(230)
where b E IR is displacement vector. We have
dSvD g(y). N' (v)) = dNv (v) b, g(()))
(231)
and given that dsvD is invariant to translations, we arrive at the final result
dN Ny g'(r) dN g'()) N' g(()))
(232)
This shows the claim.
47

---

E.7 Invariances of our Representational Distance and CCA
As noted in Appendix D.2, mCcA(f, f") invariant any invertible linear transformation of and f'. In contrast, when considering our dissimilarity measure, dr.g. and the transformations to which it is invariant, it is important to note that it relies both dsvD (L f(x), L'T f'(x)) and dSvD (N' g(y), N g" (v)) In both these expressions, embeddings and unembeddings are coupled, since the matrices L, L' (resp. N, N') depend on the unembeddings g, g (resp. the embeddings f, f').
By contrast, MCcA (f, f') can be computed independently of the unembeddings g, g.g'.
Consequently, the two similarity measures (dissimilarity in the case dr.g) differ in their invariance properties, as demonstrated below. Consider two models (f, g), (f", g') E θ such that f(x) = Af"(x) and (y) = Bg. (v), with and B invertible matrices but such that B & A i.e., (f, g) and (f', are not in the identifiability class. Since A and B are invertible matrices, have MCCA (f, f')= = mcca g') 1. In contrast, we get:
-T
g(v), dr (ff, g), g')) = max { (L [(xxx, L L (x))), (N g(v), N gT(()) 20, 0,
(233)
where the equality holds if and only if there exist orthogonal matrices O O' and diagonal ma- trices S,S', D, D' MxM with entries S (vx(1:))) -1 Dji == (x,((x;;))
S =
(1x(())')) D/ (x;;:
and displacement vectors a, such that:
L L'x(x)= = S OS''' Ts' + N g(y) = D'D''' g'()) +b.
(234)
Because the set of matrices described above is a subset of all linear invertible transformations, it is then possible to find cases where, Equation (233), the equality does hold for careful choice of (f, g), (f", g') E θ.
48

---

F Experimental Details
This section contains details of the implementation of all our experiments. repository containing the code to reproduce our experiments is available GitHub."
E.1 Constructed Models
In the following, we will detail how to construct the models which we will use to illustrate Theorem 3.1 and generate Table 1; and illustrate the bound in Theorem 4.7 (see Appendix F.6).
We choose classification models with a representation space equal to R To construct the reference model (f, g) E O we distribute its unembeddings uniformly on the unit circle, ensuring that the angle between any two adjacent unembeddings is equal. We then sample its embeddings such that each embedding corresponding to given label lies mleser-measured by angular distance to its associated unembedding than to any other.
We construct another model (f'g') E è by permuting the unembeddings and their associated embeddings. For both models, we then vary the norm of the unembeddings and measure dgL. and dixx between models and the maximum of dSVD between embeddings.
To illustrate how dx and the bound derived in Theorem 4.7 increase with increasing differences in representations, compare reference model with perturbed version constructed by adding small amount of Gaussian noise to the reference model's 's embeddings. By increasing the amount of noise, d grows, as well as the maximum of dgvD.
F.2 Models Trained on Synthetic Data
Synthetic data generation. We consider data with two-dimensional input and with classes, where each class consists of slice of the circle together with the opposite slice. See Fig. for an example with e = 6. We construct the data by drawing 20, 000 samples from a two-dimensional Gaussian (μ 0, = and assigning labels based on angle the first axis.
Model training. We trained classification models with {4, 6, 10, 18) classes, using representation space equal to R Each model consists of three fully-connected layers, with layer sizes chosen from {16, 32, 64, 128, 256). We use LeakyReLU activation functions. We train four types of models: one where the norms of both the embed- dings and unembeddings are constrained to be equal to 20, one where the norm of the embeddings are equal to 20 and with unconstrained unembeddings, one where the norm of the embeddings is unconstrained and the norm of the unembeddings is 20, and one with no constraints on the norms. To obtain the results in Section 5.2, we only consider models with unconstrained norms. For each combination of the number of classes, the layer size, and whether the restriction is applied or not, we train with 20 random seeds. All models are trained with a batch size of 128 for 15, 000 steps using the ADAM optimizer [29].
Figure 4: Illustration of training data for classes. Each color represents different class label.
F.3 Models Trained on CIFAR-10
We trained classification models on CIFAR-10 [32], where the embedding network consisted of ResNet18 [21] but choosing the representation space be or 5-dimensional. For each dimension we trained 10 seeds. The unembedding network consisted of three fully connected layers of width 128, followed by an output layer of size 2, 5, thus giving us representations in the desired number of dimensions. The models were trained for 20, 000 steps with a batch size of 32
gi thub. www.b/mwgiii//eeeeetcctppeeaaeee wwwppiiiiiiiiieeeeeeeeeoooooooom hatta///oooooosstrreeeeeee dist-rep-sitt
For this, we used code based on htt.ah/wwweeaaaagggoooeoeeeeeeee models/resner
49

---

2
5
test loss difference
0.20
0.15
0.10
0.05
0.00
0.7 0.8 0.9
0.5
0.6
0.7
0.8 0.9
embedding mCCA
embedding mCCA
embedding mCCA
Figure 5: For models trained on CIFAR-10 with representational dimensions of 2, 3 and 5, difference in test loss mcca of the embeddings of the models. We that there both be small difference in loss and larger difference in representations or a larger difference in loss smaller difference in representations.
EMBEDDINGS f(-)
UNEMBEDDINGS g(-)
airplane
automobile
10
bird
cat
0-
deer
-10
dog
frog
-0
horse
-30
ship
truck
-50
-25
25
2
Figure 6: Illustration of representations of model trained on CIFAR-10, seed O.
using the ADAM optimizer [29]. The ResNet model used ReLU activation functions, while the networks for the unembeddings used LeakyReLU activation functions.
E.4 Loss Difference Embedding mocA
To illustrate Corollary 3.2 and that a small difference in test loss does not guarantee similar represen- tations for our models trained on CIFAR-10, we present Fig. 5. Here we compare difference in test loss with MccA of embeddings of models, where the dimension of the representations are 2, and We see that there can both be a small difference in loss and a larger difference in representations or larger difference loss and a smaller difference representations.
F.5 All Two-dimensional Representations of CIFAR-10 Models
We here present all the embedding and unembedding representations of our models trained on CIFAR- 10 with 2-dimensional representations: seed in Fig. 6, seed 1 in Fig. 7, seed Fig. 8, seed 3 in Fig. 9, seed 4 in Fig. 10, seed 5 in Fig. 11, seed 6 in Fig. 12, seed 7 in Fig, 13, seed in Fig. 14, seed in Fig. 15.
We see that some labels are neighbours for all ten models. For example, "automobile" and "truck" are neighbours for all seeds and "cat" and "dog" are neighbours for all seeds. However, other labels sometimes have varying neighbours. For example "airplane" and "bird" are neighbours for seeds 2, 3, 5, 6, 7, but not for the remaining seeds. seeds 1, 4, and 9, the "frog" label is put between them, and in seed 0 the labels are permuted even further. This behaviour might arise because inputs for some labels are so similar that would aversely affect the performance of the model to place them far apart, while the embeddings of inputs for other labels can be placed in several equally good ways.
50

---

EMBEDDINGS f(-)
UNEMBEDDINGS g
airplane
30
automobile
bird
10
cal
deer
0-
dog
-10
frag
-0
horse
ship
truck
50
25
2
Figure 7: Illustration of representations of model trained on CIFAR-10 10, seed 1.
EMBEDDINGS r(-)
UNEMBEDDINGS g(-)
airplane
automobile
bird
20
cat
deer
dog
frog
horse
-20
ship
-3-
truck
50
25
25
Figure 8: Illustration of representations of model trained on CIFAR-10, seed 2.
EMBEDDINGS f(-)
UNEMBEDDINGS g(-)
airplane
automobile
bird
20
cat
deer
dog
frog
-20
horse
ship
truck
-50
-25
25
-2
0
Figure 9: Illustration of representations of model trained on CIFAR-10, 10, seod 3.
EMBEDDINGS f(-)
UNEMBEDDINGS g(-)
20
airplane
automobile
3-
bird
cal
deer
dog
-20
frog
30
horse
0-
ship
10
truck
40
20
0
20
40
-2
Figure 10: Illustration of representations of model trained on CIFAR-10, seed 4.
51

---

EMBEDDINGS f(-)
UNEMBEDDINGS g
1.5
airplane
automobile
1.0
bird
20
cal
0.0
deer
0-
dog
frag
-00
1.0
horse
-40
1.5
ship
truck
20
0
20
Figure 11: Illustration of representations of model trained on CIFAR-10, 10, seed 5.
EMBEDDINGS I(-)
UNEMBEDDINGS g(-)
20
airplane
automobile
20
bird
cat
0-
deer
dog
frog
horse
ship
-0
truck
40
20
20
40
-2
Figure 12: Illustration of representations of model trained on CIFAR-10, seed 6.
EMBEDDINGS f(-)
UNEMBEDDINGS g(-)
airplane
automobile
40
bird
cat
20
deer
dog
frog
horse
ship
-2
truck
50
-5
25
50
Figure 13: Illustration of representations of model trained on CIFAR-10, seed 7.
EMBEDDINGS f(()
UNEMBEDDINGS g(-)
airplane
automobile
bird
20
cat
deer
0
dog
-20
-1
frog
horse
ship
.0
truck
-00
20
20
2
Figure 14: Illustration of representations of model trained on CIFAR-10, seed 8.
52

---

EMBEDDINGS [((-)
UNEMBEDDINGS g!-)
30
airplane
20
automobile
bird
10
cat
deer
0-
dog
0-
frog
horse
200
ship
-30
truck
Figure 15: Illustration representations of model trained on CIFAR-10, seed 9.
1.0
1.0
bound
bound
0.4
0.4
0.2
0.2
0.0
0.00
0.05
0.10
0.15
0.20
0:25
0.00
0.05
0.10
0.25
Figure 16: (Left) We evaluated and de, for a reference model and a constructed one where we progressively increase the noise added the embeddings. more noise is injected, the two models display higher di (up 0.25), while representation dissimilarity de, does not exceed 0.1. (Right) observe similar trend for trained models, with slight increase in de.g when dλiv approaches 0.25. We note that the bound given by Theorem 4.7 remains valid for both cases.
F.6 Illustration of Bound on Constructed and Trained Models
Experimental setup. We compare both constructed models and models trained on synthetic data. For the constructed models, we compare reference model to perturbed version constructed by adding Gaussian noise to the reference model's embedding representations (as described Appendix F.I). For models trained on synthetic data, the setup is the same described in Appendix F.2
Results. In Fig. 16 (left), observe that the maximum distance between representations gradually increases with the distance between probability distributions while staying below the bound of the Theorem 4.7. We find that not all the trained models have distance d small enough for the bound to be non-vacuous. In this case, we report only those distances that are small enough for the bound to be non-vacuous between trained models in Fig. 16 (right) and see that the bound remains valid.
E.7
Wider Models Have more Similar Distributions Extra Plots
We here report the empirical trend that wider networks induce more similar distributions and more similar representations for models with 10 classes (Fig. 17 dLLv and max (right) and 18 classes (Fig. 18 dLLy (left) and max (rig(tight)). Note that with more classes, we have less models among the most narrow networks which achieve more than 90% accuracy. We have therefore left out results in the plots where we had fewer than 5 models for the comparison. For 10 classes, this means that starting at width 32, we have 9, 16, 19, and 19 models for the comparisons in the plot. For 18 classes, starting width of 64, we have 10, 12, and 19 models for the comparisons in the plot.
53

---

10 classes
0.8
mean d
0.6
0.2
mean deg
0.1
0.0
0.0
150
200
250
150
200 250
width
width
Figure 17: We display how mean dLav and max dLLv varies with increasing the neural network width for models trained to classify among 10 classes. (Left) We observe decreasing mean dLLv as the network width grows. The shaded represents the standard deviations evaluated from different random seeds retraining. (Right) similar trend also observed max dLav when increasing the network width.
0.030
18 classes
0.35
0.025
mean di
0.30
0.020
mean deg
0.25
0.015
0.010
0.005
0.15
100
150
200
250
100
150
200
250
width
width
Figure 18: We display how dLav and max dLav varies with increasing the neural network width for models trained to classify among 18 classes, (Left) We observe decreasing mean dLLv as the network width grows. The shaded area represents the standard deviations evaluated from different random seeds retraining. (Right) A similar trend is also observed for max dLLv when increasing the network width.
G Computing Resources
Each model was trained using a single NVIDIA RTX A5000. For each number of classes (4, 6, 10, 18), training 20 seeds the synthetic data took about hours, summing to total of 136 hours to train the models synthetic data. For the models on CIFAR-10, training 10 seeds took ~27 hours.
The distances calculated on CPU-only machine: computing the distances for the models synthetic data required less than 2 hours, whereas the evaluation on models in CIFAR- 10 required around 4 hours. Evaluating the accuracy of models on synthetic data also done on the CPU, taking less than 20 minutes in total.
H Assets
CIFAR-10. We used the CIFAR-10 dataset as loaded with torchvision package 19 The dataset from [32] and contains 50, 000 images for train and validation, and 10, 000 images for testing.
Python Packages. All experiments conducted with Python 3.11 and used pytorch 2.5.1. Other packages are reported the repository at ctt...ccmbbbeeeeeeeeeee. httpe////eagggi///eeeeee-e-----e
19 htttps://docs pyttorc...cccc.ccommm http:///i/////t///////////////rr stt.
datasets. CIFAR10. html
54


Reviews:
[{"review_id": "9oPl6tVVcL", "reviewer": "Reviewer_TFNc", "summary": "In this work, the authors aim to analyze whether similarity between two models at the level of behavior (predictions made by a model) implies similarity at the level of representations. To this end, the authors contextualize results from identifiability theory (which show equal likelihood implies representational similarity up to invertible transformations under certain diversity conditions) and argue that developing a meaningful analysis requires first formalizing what similarity metrics even make sense. The authors thus propose a new \"distributional distance\" that bounds representational dissimilarity, and enables statements on similarity of representations based on closeness of predictions.", "questions": "See weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths.** The question of how and when similarity in the predictions space implies similarity in the representation space is extremely useful: if true, training different models to merely learn the distribution will yield the same representations and be of similar utility. The paper is very well-written, though partially lacking in experiments that could have been helpful for emphasizing the theory. \n\n**Weaknesses.** The weaknesses below are based on my interpretation of results. Please correct me if I'm wrong in how I understood the results and I'll happily change my scores in accordance with that.\n\n- Construction-centric nature of Theorem 3.1 and related arguments: The primary argument in the formal analysis is based on construction of a model class that is equivalent in predictions, but not in representations. While fair, the argument ignores learning dynamics: if a model does not ever learn solutions beyond a narrow subset of the elements of this prediction-equivalence class, should we worry about the *possibility* of representational dissimilarity? \n\n- Permutation based representation dissimilarity in Theorem 3.1: My understanding of the arguments up to theorem 3.1 was that authors deem models with representations equivalent up to linear transformations as representationally similar. Wouldn't permutations fall under the scope of this definition? If so, I don't follow why the intuitive figure (Fig 2) relies on permutation of representations to produce representationally dissimilar models. More broadly, isn't a permutation transformation too trivial to consider models representationally dissimilar? I would in fact argue that representational similarity analyses should by default ignore all trivial symmetries of neural network architectures (e.g., permutations of neurons yielding a behaviorally equivalent model).\n\n- Empirical characterization: I regard the paper mostly as a theory work and hence this is a minor weakness. Specifically, I found the empirical analysis somewhat underwhelming. For example, the authors argue based upon their defined measure that representational similarity increases with width, an arguably known result from past work on CCA / CKA based similarity analyses [1]. However, offering a tool should ideally be grounded in some interesting conclusion or application.\n\n[1] https://arxiv.org/abs/1905.00414", "was_revised": true, "final_justification": "The comments were very helpful in helping clarify my questions! I'll update my score accordingly.", "rebuttal": "We thank the reviewer for their helpful questions and for emphasizing the significance of our research question.\n\n> “Construction-centric nature of Theorem 3.1…”\n\nIt is true that just because these constructions exist doesn't mean a model will learn them during training. We will make this clearer in the article.    However, identifying these examples is essential for mathematical understanding of the problem: identifiability research has similarly made progress by starting from constructed counterexamples (e.g. [1]) to understand non-identifiability issues.  Moreover, we find it interesting that a mechanism resembling our construction actually emerged in the CIFAR-10 experiments (Section 5.1). Understanding when and why such constructions are learned in practice would be an interesting direction for future research.\n\n> “Permutation based representation dissimilarity…”\n\nWe appreciate the comment and suspect the confusion may be due to unclear phrasing on our part. \nThe permutation in 3.1 should be understood as a permutation, $\\pi$, of the _label indices_, $\\mathbf{g}’(y_i) = \\mathbf{g}(y_{\\pi (i)})$ with a corresponding shift of the embedding clusters; and _not as a permutation of the representation dimensions (i.e., the axes in the figure)_, $\\mathbf{f}’(\\mathbf{x}) = \\mathbf{P}\\mathbf{f}(\\mathbf{x})$ for some permutation matrix $\\mathbf{P}$, which would indeed be a linear transformation, as the reviewer points out. Unlike the latter, the permutation of label indices, and corresponding shift of embedding clusters, is not a linear transformation of the representations: In Table 1, we show that when constructing models in this way, the mean canonical correlation is close to 0, which means their representations are very far from being linear transformations of each other. The permutation of label indices also changes the model’s output distribution, since it alters the rank—by predicted probability—of every label except the top one for each input $\\mathbf{x}$.  Conversely, permutations of embedding neurons  $\\mathbf{f}’(\\mathbf{x}) = \\mathbf{P}\\mathbf{f}(\\mathbf{x})$ (and accordingly of unembedding ones) in a way that lead to the same probabilities will not alter the model’s output distribution and will thus yield equivalent model instances according to our identifiability perspective (models that have equal distributions are deemed equivalent). We will include the clarification above in the revised version of the paper.\n\n\n> “The authors argue based upon their defined measure that representational similarity increases with width, an arguably known result from past work”\n\nWe believe this comment may reflect a misunderstanding of the primary novelty of the experimental findings in Section 5.2. That higher-capacity models tend to produce more similar representations is indeed a known result, which we acknowledge on page 2 (see Footnote 2). The experiments in Section 5.2 examine this phenomenon through the lens of our novel theoretical results. Specifically, they show that wider neural networks yield lower mean and variance of our distributional distance $d_\\mathrm{LLV}$ (second plot from the right in Figure 3), and that this correlates with lower values of our representational dissimilarity measure $d_\\mathrm{SVD}$ (rightmost plot in Figure 3)—both in the extreme regime where the bound from Theorem 4.6 is non-vacuous (thereby illustrating our theory) and, intriguingly, even beyond that regime, suggesting that a tighter bound may be attainable. These results are inherently novel, as they rely on our newly introduced dissimilarity measure $d_\\mathrm{SVD}$ and distributional distance $d_\\mathrm{LLV}$. Importantly, prior experimental studies have used different similarity measures—most notably those based on CCA. However, similarity under CCA does not imply similarity under $d_\\mathrm{SVD}$, since CCA is invariant to arbitrary linear transformations, whereas $d_\\mathrm{SVD}$ is only invariant to a restricted class of such transformations (see Appendix E.8 and our response to reviewer `Hbxo`).\n\n> “Empirical characterization”\n\nOur primary goal is to advance the mathematical understanding of representational similarity through the lens of identifiability. We appreciate that the reviewer considers lack of largeer scale experiments a “minor weakness,” and we agree that a more comprehensive empirical investigation is a valuable direction for future work. At the same time, we are encouraged by reviewer `r8r8`’s assessment that, “while not large scale, [our experiments] are well-designed to illustrate [our] argument.” In addition, we have conducted further experiments to strengthen the empirical validation of our approach; please see our response to reviewer `66wY` for details.\n\n> “offering a tool should ideally be grounded in [...] application”\n\nWhile our current work is primarily theoretical and only represents an initial step towards applications, we believe it is a significant and original one; in particular, it relaxes a strongly unrealistic assumption underlying many prior identifiability analyses—namely, the assumption of equal model distributions. We see this as an important step toward developing methods with empirical relevance and applicability. Encouragingly, several reviewers offered concrete suggestions for how this line of work could evolve toward practical applications (e.g., for distillation, as suggested by reviewer `r8r8`). We take this as a sign that our theoretical contribution opens up a promising avenue for future development with real-world impact.\n\n[1] Hyvärinen, A., & Pajunen, P. (1999). Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3), 429-439."}, {"review_id": "KHlINZmF2m", "reviewer": "Reviewer_66wY", "summary": "The paper looks at co-embedding-based classification models, where inputs and outputs are mapped to a shared space and then inner products are used in a softmax to obtain conditional distributions.  The central question is whether, given access to all the conditional distributions for two models trained on the same data, one can claim anything about the similarity of the representations of the two models in the shared space (pre-softmax).  \n\nAs a first guess, one might try using one of the popular statistical distances between corresponding conditional distributions, e.g. KL divergence, to assess classification similarity and infer that the models’ representations are closely related if the distance vanishes.  The authors show that this is not the case in general -- there are ways to obtain arbitrarily close output distributions with representation spaces that are not relatable through linear transformations.  \n\nThe paper then proposes a statistical distance -- which additionally satisfies the four properties of a metric -- that essentially inverts the softmax operation by operating on log probability ratios and can bound how similar the representation spaces are up to linear transformation.  Some experiments on CIFAR10 and synthetic examples are included to bolster the theory, with an interesting result suggesting larger intermediate dimensions increase the final representation space similarity.", "questions": "- It would be helpful to understand whether the use of 2D embedding spaces in the experiments was driven by practical considerations (e.g., visualization) or if similar behaviors are expected in higher-dimensional regimes as well.\n\n- The CIFAR point clouds (Fig 3 and Supp) look like radial information of the image embeddings might not be playing a role (due to the high density near the origin).  A quick look at the code suggests the norm might be fixed for these experiments.  While fixing the norm for the label embeddings serves a purpose for the different synthetic experiments, applying norm constraints to the *image* embeddings for CIFAR10 seems less directly motivated by the theory.  This may weaken the empirical alignment with the theoretical results and limit the already constrained experimental scope.  Could the authors clarify whether this design choice is discussed in the text?  And more generally, can they speak to the rationale for norm-fixing in this setting?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper is clearly written and well-structured.  The flow from proposing KL divergence, to providing an intuitive counterexample, to proposing the LLV distance is effective.  Prior work and limitations are addressed with transparency.\n\n- Linking the set of classification probability distributions to the underlying representation space, for a format that is broadly used in machine learning (the co-embedding space matching x to y), offers a foothold into novel assessments of representation similarity.  The identifiability angle is powerful.  The metric properties of d_LLV, though not utilized in this work, are an added bonus.\n\n## Weaknesses\n\nThe experimental scope is limited, especially in that all embedding spaces are restricted to 2D, raising concerns about whether the observed behaviors extend to higher-dimensional settings typical of practical applications.", "was_revised": true, "final_justification": "The authors have addressed the majority of my concerns.  While the experiments' low dimensionality remains a weakness, it is outweighed by the rest of the paper.  I raise my score to a solid accept.", "rebuttal": "We thank the reviewer for their constructive feedback and for recognizing the potential of our analysis.\n\n> “all embedding spaces are restricted to 2D”\n\nRegarding the CIFAR-10 experiments in Section 5.1: to guarantee the diversity condition, the representation dimension needs to be smaller than the number of labels, and we specifically chose a dimension of 2 for ease of visualization. This makes it easier to compare our findings in Figure 3 (Left) with Figure 2, which illustrates the construction used in Theorem 3.1. \nTo address the reviewer’s concern, we will add results from models we have trained on CIFAR-10 with 3- and 5-dimensional representation spaces. For 3- and 5-dimensional representations, we cannot provide a visualization like in Figure 3 (Left). Instead, we will include scatterplots in the appendix showing the difference in test loss versus the mean canonical correlation (mCCA) of embeddings from models trained on CIFAR-10. In these scatterplots, we see that it is possible to get both smaller loss differences and less similar representations, and models with higher loss difference and more similar representations. For 2 and 3 dimensions, we see that the minimum representational similarity is lower than for 5 dimensions, and the maximum similarity achieved is slightly higher.\nAll in all, these scatter plots suggest that a small loss difference does not imply that the underlying representations are similar, in agreement with our Corollary 3.2. \n\n> “The experimental scope is limited”\n\nThe primary goal of our work is to deepen the mathematical understanding of representational similarity through the lens of identifiability. We appreciate reviewer `r8r8`’s comment that “while not large scale, [our experiments] are well-designed to illustrate [our] argument,” and we believe that the additional results described above and prompted by this review further strengthen our experimental validation. We agree that an empirical study with larger-scale experiments and models is an important direction for future research.\n\n\n> “CIFAR point clouds… the [embedding] norms might be fixed for these experiments”\n\nWe thank the reviewer for pointing out that this aspect of the code was unclear. **We did not constrain the embedding or unembedding norms of the models presented in the main paper**: \nThe embeddings and unembeddings are plotted in Figure 5 to Figure 14 in Appendix F, and it can be seen that their norms are not fixed. However, we have some additional experiments in Appendix F where either embedding or unembedding norms, or both, were fixed. These ablations were not included in the main paper, but only in Appendix F.5, \"Illustration of Bound on Constructed and Trained Models\". We will clarify this point once we publish the code."}, {"review_id": "fEqlTgrywa", "reviewer": "Reviewer_Hbxo", "summary": "A typical identifiability treatment tells us that equal likelihoods imply similar representations, up to an identifiability class. What about unequal but similar likelihoods? The authors in this work study model and representational similarity in a large class of models, and show that similar likelihoods do not necessarily imply similar representations. The authors define the identifiability class of their model to establish their results, and use it to define a new distributional distance. They then use this distributional distance to bound representational similarity. Their results are supported by numerical results on the CIFAR dataset and synthetic experiments.", "questions": "See weaknesses above for some questions, with just one more here:\n- Shape metrics from Williams et al (https://arxiv.org/abs/2110.14739) use **groups** to define invariance in shape classes, going beyond CCA -- can this be leveraged for your identifiability class, to be more specific than the CCA-based measures that consider any linear transformations?", "limitations": "Limitations are discussed in the last section. Perhaps including an example of the failure modes mentioned could strengthen the text?", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Well-motivated and well-framed question on identifiability and its relaxations. \n- Impressive theoretical results and new definitions. \n- A simple synthetic example corroborates a proof.\n- Numerical experiment on CIFAR is non-trivial. \n\nWeaknesses:\n- Lacks comparison to the conclusions that could be drawn from existing approaches. For instance, CCA is the most raised alternative -- could the relationship to this model be further discussed? What would we miss by just using CCA?\n- Some assumptions are not discussed to their fullest extent. For instance, does the pivot $x_0$ (especially its probability $p_{\\mathcal{D}}(x_0)$) influence the distance or log-likelihood variance distance terms?", "was_revised": true, "final_justification": "Satisfied by comments, keeping acceptance score.", "rebuttal": "We thank the reviewer for their encouraging feedback and for acknowledging the strength of our theoretical results.\n\n> “What would we miss by just using CCA?\"\n\nThis is an important technical point. To summarize:\n\n(1) Similarity measures based on CCA take only pairs of embeddings, or unembeddings, as input, whereas our **$d_{\\mathrm{SVD}}\\left(\\mathbf{L}^{\\top} \\mathbf{f}(\\mathbf{x}), \\mathbf{L}^{\\prime \\top} \\mathbf{f}^{\\prime}(\\mathbf{x})\\right)$ takes both the embeddings and the unembeddings of both models as inputs**, since the matrices $\\mathbf{L}$ and $\\mathbf{L}^{\\prime}$ depend on the unembeddings $\\mathbf{g}$. Note in fact that the distribution of the model (Equation (1) in the paper) depends on both the embeddings and the unembeddings.\n\n(2) CCA-measures have thus different invariances and can lead to different conclusions than our dissimilarity measure $d_{\\mathrm{SVD}}\\left(\\mathbf{L}^{\\top} \\mathbf{f}(\\mathbf{x}), \\mathbf{L}^{\\prime \\top} \\mathbf{f}^{\\prime}(\\mathbf{x})\\right)$. For example, mCCA between model embeddings is invariant to any linear transformation of the embeddings.\nIn principle, one could find two models giving rise to different distributions where the embeddings of a model are a linear transformation of the embeddings of a second model, whereas their unembeddings are not equal up to the same linear transformation. These two models would thus have maximum mCCA (equal to 1) between embeddings, but they would have a positive $d_{\\mathrm{SVD}}\\left(\\mathbf{L}^{\\top} \\mathbf{f}(\\mathbf{x}), \\mathbf{L}^{\\prime \\top} \\mathbf{f}^{\\prime}(\\mathbf{x})\\right)$ (not the minimum value of zero).\n\nThanks for prompting this explanation: We will clarify this in the revised version of the paper, also based on a discussion of (2) we currently have in Appendix E.8. \n\n> “...does the pivot… influence the distance or log-likelihood variance distance terms?”\n\nWe thank the reviewer for pointing out that this requires clarification. When models have different distributions, the choice of pivot (as well as the input and label sets) can indeed affect the value of the distance $d_\\mathrm{LLV}$. As long as the diversity condition is satisfied, any choice of pivot is valid. If, on the other hand, the distributions of two models are equal, then the distance $d_\\mathrm{LLV}$ will be zero no matter the choice of pivot and sets. We will make this clearer in the paper. In the experiments, we draw a sample of possible inputs with pivot sets and choose the set that gives the smallest value for the term $t_2$ in Definition 4.4 (we specified this in Appendix E.3: Implementation of the Log-Likelihood Variance Distance). We have not yet analyzed the effect of the pivot point probability on the value of the $t_2$ term in $d_\\mathrm{LLV}$. It might be interesting to explore this in future work. \n\n> \"Shape metrics from Williams et al…”\n\nWe thank the reviewer for this interesting reference and will include it as related work. We will also consider whether its ideas can inform the design of other similarity measures with the desired invariance properties."}, {"review_id": "eXeZRpc1lU", "reviewer": "Reviewer_r8r8", "summary": "This paper studies when two independently trained neural networks learn similar internal representations if their output distributions are close. It argues that a good similarity measure should be invariant to transformations that leave the model's output distribution unchanged. The paper argues that a small KL divergence between two models output distributions does not guarantee that their learned representations are similar, specifically that for any arbitrarily small $\\varepsilon$, one can find two models with $d_{KL} \\le \\varepsilon$ whose embeddings are far from linearly equivalent. Even models with arbitrarily low classification loss can learn entirely dissimilar features. It introduces a new log-likelihood variance (LLV) distance between distributions and a PLS-SVD-based dissimilarity between representations. LLV is tailored to identifiability and is a proper metric. Using these, they prove that if two models distributions are close under the proposed LLV distance, then their representations are guaranteed to be similar. Concisely, closeness in the usual KL sense is insufficient to ensure representational similarity, but closeness in the new metric does imply it.\n\nExperimentally, the authors train two ResNet18 models with a 2D bottleneck on CIFAR-10. Although both runs reach virtually identical test loss, the embeddings of the data are distinct, making CCA moderate and KL low while LLV distance remains high, validating their arguments. On a synthetic 2D classification toy dataset they show that as width grows, the mean and variance of LLV across seeds fall steadily, and the corresponding dSVD gap shrinks, suggesting higher-capacity models converge toward a common representation when their distributions align under LLV.", "questions": "- Have you looked at LLV for larger models / higher dimensional representations? Do the same conclusions hold? Curious about practicality of the metric.\n- It seems like this might be intractable for something like language models, do you see anyway of accelerating / approximating this?\n- I assume other f-divergences would have a similar issue as KL, was this looked at?\n- Any thoughts on whether LLV could be used as an explicit penalty (loss) in something like distillation?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "# Strengths\n- Theoretical: The paper proposes a rigorous perspective on representational similarity which is novel. Theorem 3.1 demonstrates a counter intuitive fact that even extremely small differences in output distributions can correspond to large differences in internal representations. This a relevant result w.r.t. efforts such as model distillation. Theorem 4.6 is likewise a meaningful contribution: a new distributional distance specifically designed to measure representation similarity under some assumptions.\n- Experiments: While not large scale, they are well-designed to illustrate the argument. Section 5.1 shows a concrete example of two standard models with indistinguishable performance metrics but significantly different internal representations. This example makes the theoretical phenomenon tangible. The synthetic experiments in Section 5.2 show a clear trend that increasing model capacity yields closer distributions and more aligned representations.\n- Clarity: The paper is generally well-written, albeit a bit dense.\n- Novelty: LLV is a novel, theoretically validated distance that shows promise in experiments.\n- Reproducibility: code included in supplementary materials, public release promised.\n\n# Weaknesses:\n- Assumptions: Theorem 3.1 requires certain conditions that somewhat limit its scope ($k \\geq M+1$). It's not entirely clear how severe this limitation is, and perhaps is a reasonable assumption to make. Language models frequently have far more tokens than the embedding dimension, but it may be much closer (or reversed) with a vision model on ImageNet. Similarly, Theorem 4.6 comes with some assumptions, but again not clear on the severity.\n- Bounds: The bound on dSVD ($2M\\varepsilon$, will worsen with dimension) is not very tight, though the authors acknowledge this. The CIFAR experiments didn't enter the setting where the bound is non vacuous.\n- Experimental: While the experiments do support the claims, they could be considered limited in scope. All experiments use relatively low-dimensional representations and simple datasets (e.g., CIFAR). Validation in higher dimensional cases or on a larger dataset (i.e., ImageNet) would be a valuable addition. Combined with the loose bounds, assessing the overall practical utility of LLV is a bit harder.", "was_revised": true, "final_justification": "Authors addressed my questions appropriately, and I had little issue with the paper in the first place.  Showing that representational dissimilarity can be bounded using a distance between distributions is a nice contribution. Despite the loose bounds and computational complexity I think this is a promising direction of work. Maintaining my score of accept.", "rebuttal": "We thank the reviewer for their encouraging feedback, and we are pleased that they found our results relevant and meaningful.\n\n> Clarification on the assumptions of our theorems:\n\nA central assumption of our work is the _diversity condition_ (Definition 2.1). As the reviewer rightfully noted, this condition is more likely to hold for language models, whereas for regular image classification models, it might not be satisfied (especially when the number of classes is smaller than the representation dimensionality). We will clarify this point in the paper. As we noted in Section 6, recent work [1] may serve as a starting point for exploring how to proceed when the diversity condition does not hold, potentially extending the results to typical image classifiers. However, we leave this direction for future work.\nWe remark that Theorem 4.6 has an assumption slightly stronger than the usual diversity condition (all $\\mathbf{L}, \\mathbf{L}’, \\mathbf{N}, \\mathbf{N}’$ constructed from the input and label sets are invertible). However, we expect that if the diversity condition can be relaxed, then this slightly stronger assumption may also be relaxed while retaining a qualitatively similar result. \n\n\n> “The bound on dSVD [...] is not very tight”\n\nWe agree with the reviewer’s observation and remark that this point is also acknowledged in Section 6 (“Limitations” paragraph). Nonetheless, we believe this result provides an important proof of principle: It demonstrates that **representational dissimilarity can be bounded using a distance between distributions**---a fact that is far from trivial, as Theorem 3.1 illustrates. Our experiments on synthetic data indicate that this bound could potentially be tightened; however, determining how to achieve this is left for future work.\n\n> “Validation in higher dimensional cases or on a larger dataset (i.e., ImageNet) would be a valuable addition”\n\nWe thank the reviewer for the positive feedback on our experiments: “While not large scale, they are well-designed to illustrate the argument.” We agree that expanding to larger-scale experiments is an important step for future work, especially to bridge the gap to practical applications. Furthermore, we are particularly interested in experimenting with language models, as they appear to align most naturally with our assumptions. \n\n> “accelerating / approximating LLV computation”\n\nWe hope that in future work we can reduce the number of labels required so that the metric scales with representational dimension rather than the number of labels.\nThis would significantly reduce computation time for language models where the number of tokens far exceeds the representational dimension (e.g., GPT-2 has ~50k tokens vs. ~1000 dimensions).\nApproximating the computation of $d_\\mathrm{LLV}$ is another promising direction for future work and could make it more suitable as a loss function.\n\n\n> Considering other f-divergences:\n\nOne of the reasons why the KL divergence cannot be used to bound representational similarity stems from the fact that unlikely labels (with $p(y|\\mathbf{x})$ very small) give a small contribution to the KL, which enables constructions as the one in Theorem 3.1. We expect that other divergences with similar behavior would also not allow us to derive bounds on representational similarity. Conversely, divergences that do not put particularly low weight on unlikely labels (like $d_\\mathrm{LLV}$) might be better suited for deriving bounds on representational dissimilarity, as in our Theorem 4.6. We plan to test other divergences in future work.\n\n\n> “Any thoughts on whether LLV could be used as an explicit penalty (loss) in something like distillation?”\n\nWe thank the reviewer for sharing this thought. We agree that this is an interesting direction to pursue, and we explicitly mentioned distillation in the introduction. However, we believe further research is needed before recommending log-likelihood variance distance as a loss function. We hope it will be possible to either find a simplified version of the distance or identify an upper bound or approximation that can serve as an effective and efficient distillation loss.\n\n[1] Marconato, E., Lachapelle, S., Weichwald, S., & Gresele, L. All or none: Identifiable linear properties of next-token predictors in language modeling. AISTATS 2025"}]

Author rebuttal:

Meta-review: This work considers the problem of comparing the representations of models trained to perform the same task. The authors show that small statistical divergences between model distributions (specifically, the conditional distributions of the outputs given the inputs) does not imply that their representations are similar. The authors then propose a distance (a proper mathematical metric) on the model distributions that has the property that bounds how similar the model representations are. They illustrate their theory empirically and show that larger intermediate dimensions lead to more similar representations. Reviewers stated that the problem is interesting, the theoretical results were impressive, and the paper was well written; though they felt that the empirical tests could be stronger. All of the reviewers ultimately gave the paper positive reviews, so I recommend acceptance.
Decision: Accept (poster)