Challenges in Explaining Representational Similarity through Identifiability

Authors: beatrix m g nielsen
luigi  gresele
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-20 17:39:59

Abstract: The phenomenon of different deep learning models producing similar data representations has garnered significant attention, raising the question of why such representational similarity occurs. Identifiability theory offers a partial explanation: for a broad class of discriminative models, including many popular in representation learning, those assigning equal likelihood to the observations yield representations that are equal up to a linear transformation, if a suitable diversity condition holds. In this work, we identify two key challenges in applying identifiability theory to explain representational similarity. First, the assumption of exact likelihood equality is rarely satisfied by practical models trained with different initializations. To address this, we describe how the representations of two models deviate from being linear transformations of each other, based on their difference in log-likelihoods. Second, we demonstrate that even models with similar and near-optimal loss values can produce highly dissimilar representations due to an underappreciated difference between loss and likelihood. Our findings highlight key open questions and point to future research directions for advancing the theoretical understanding of representational similarity.

Main paper: 
 introduction: 
 There is ample evidence of similarity in the representations learned by different deep learning models (Lenc and Vedaldi, 2015; Kornblith et al., 2019; Bansal et al., 2021; Ding et al., 2021) , which has given rise to conjectures on why the phenomenon occurs (Huh et al., 2024; Teney et al., 2024) as well as work on exploiting it (Moschella et al., 2022; Cannistraci et al., 2023; Maiorca et al., 2024) , for example for model stitching (Lenc and Vedaldi, 2015; Bansal et al., 2021) . One instance of representational similarity, perhaps the simplest and most fundamental one, is when different models trained on the same data and with the same learning objective produce representations that are equal up to a simple transformation, for example linear. 3 For highly nonlinear models, it is not obvious that this should occur: in fact, work on nonlinear independent component analysis (ICA) offers many examples where different models, despite assigning equal likelihood to the data, yield representations related by nonlinear transformations (see, e.g., (Hyvärinen and Pajunen, 1999) ). However, Roeder et al. (2021) and Khemakhem et al. (2020) show that for nonlinear models in a broad discriminative class-including several models popular in representation learning, those which assign equal likelihood to the observations extract representations which are linear transformations of one another, provided a suitable diversity condition is satisfied. One might therefore be inclined to use these findings to account for some of the empirical observations of representational similarity.In this work, we identify two key challenges in using existing identifiability theory to explain representational similarity. Firstly, identifiability results require equality of the likelihoods of the considered models as a premise: in practice, this assumption is rarely fulfilled for trained models, since even initializing with different random seeds will typically result in models with likelihoods which might be close, but not equal. It would therefore be desirable to relax the assumption of equality and explore whether the results in (Roeder et al., 2021; Khemakhem et al., 2020) can be extended to prove approximate representational similarity4 for models achieving close to equal likelihoods. Furthermore, as we will show in Section 4, minimizing the loss is not the same as minimizing the difference of log-likelihoods, which means that models with close to optimal loss can have representations which are very dissimilar. By pointing out these challenges, we hope to inspire further research on the theoretical underpinnings of representational similarity.Our contributions can be summarized as follows:• In Section 3, we prove a relation between the difference of log-likelihoods entailed by two models and their extracted representations. This generalizes the analysis in (Roeder et al., 2021; Khemakhem et al., 2020) , which requires vanishing log-likelihood difference.• In Section 4, we introduce a construction showing that it is not sufficient that both considered models have close to optimal (zero) loss for their representations to become similar. 
preliminaries: 
 Model Class. We will consider a model class defined by the probability of a label, y, given an input, x, from a domain, X , and a collection of possible targets, S, where we must have y ∈ S.EQUATIONwhere θ is the parameters of the model. We will often write "the model θ" as short for the model with parameters θ. For two models, θ * and θ ′ , we write f * and f ′ for their embedding functions and g * and g ′ for their unembedding functions, following the terminology in (Park et al., 2023) . In the following, we let the codomain of f and g be R M . So the embedding functions f : X → R M take inputs into R M , and the unembedding functions g : S → R M take the labels into R M . This model class is the same as the one considered in (Roeder et al., 2021) : it is particularly interesting because many common models and pre-training objectives can be written in this form, for example autoregressive language models and a common supervised classification objective. For more examples, see (Roeder et al., 2021, Section 4 & Appendix D) .Diversity Condition. We define the diversity condition like in (Khemakhem et al., 2020) .Definition 1 (Diversity condition). We say that a model θ ′ from the model class Eq. ( 1) satisfies the diversity condition for g ′ if there exists y 0 , ..., y M ∈ S such that the M vectors {g ′ (y i ) -g ′ (y 0 )} M i=1 are linearly independent. Similarly, we say that a model satisfies the diversity condition for f ′ if there exists x 0 , ...,x M ∈ X such that the M vectors {f ′ (x i ) -f ′ (x 0 )} M i=1 are linearly independent.This assumption, or variations thereof, plays a crucial role in the identifiability results in, e.g., (Roeder et al., 2021) , (Khemakhem et al., 2020) and (Lachapelle et al., 2023) .Identifiability Result. For our purposes, the identifiability results found in (Khemakhem et al., 2020) , (Roeder et al., 2021) , and (Lachapelle et al., 2023) can be summarized in the following statement (a detailed proof can be found in Appendix B). Theorem 1. Let θ * be a model of the form in Eq. (1) and satisfying the diversity condition on f * and g * . Assume θ ′ is another model of the same form. ThenEQUATIONwhere following (Lachapelle et al., 2023) , the equivalence relation is defined byEQUATION)where A is an invertible matrix and b is a vector.Measuring Similarity with Canonical Correlation Analysis. It is possible to define a measure of how close two sets of vectors are to being linear transformations of each other, based on Canonical Correlation Analysis (CCA) (Hotelling, 1936) . Given two random vectors z ∈ R N and w ∈ R M , CCA finds vectors s k and t k , where k ≤ min(N, M ), such that the Pearson correlation ρ k = corr s ⊤ k z, t ⊤ k w is maximized, with the constraint that s ⊤ k z, s ⊤ j z and t ⊤ k w, t ⊤ j w are linearly independent for k ̸ = j. In our setting, we would like to measure how close the embeddings and unembeddings from two models θ * , θ ′ are to being linear transformations of each other. Our vectors will thus be f * (x), f ′ (x) or g * (y), g ′ (y) for inputs x and labels y (thereby N = M ). As our similarity score, we will use the mean canonical correlation, as in (Klabunde et al., 2023) :m CCA (z, w) = 1 M k ρ kIf m CCA (z, w) = 1, z and w are linear transformations of each other (see Appendix A).The Difference Between Loss and Likelihood. The distinction between loss and likelihood is crucial, since we would like to understand whether different models achieving small loss (i.e., close to optimal) will extract similar representations. We say that two models, θ * , θ ′ , have equal likelihood if p θ * (y|x, S) = p θ ′ (y|x, S) for all x ∈ X and y ∈ S. When training such models, the loss we optimize is E (x,y)∼q D [-log (p θ (y|x, S))], where (x, y) is an input-label pair from the data distribution q D . Note that the identifiability results by, e.g., Roeder et al. (2021) and Khemakhem et al. (2020) require that the likelihood entailed by the two models, θ * and θ ′ , should be equal. However, the observed data distribution q D (y|x) will likely not include all possible combinations of inputs values x ∈ X and label values in S. For example, in the case of an autoregressive language model, given a sequence of words x, many among the potential next token-candidates y would result in nonsensical sentences, thereby having a low probability of appearing in the training data. Because of this, for each x, minimizing the loss does not uniquely constrain the conditional likelihood for all targets in S. In short, equal loss does not mean equal likelihood. One might nevertheless think that if the loss is close enough to optimal, and there is very little density assigned to improbable targets, we would still get models with representations which are close to being linear transformations of each other. However, as we show in Section 4, this is not the case. 
close-to-identifiability result: 
 As a first step towards generalizing the results in (Roeder et al., 2021; Khemakhem et al., 2020) , we prove that for models as in Equation ( 1), it holds that the representations extracted by a model θ * can be written as a linear transformation of those from another model, θ ′ , plus an error term which can be seen as the non-linear part of the relationship between the functions. When the likelihoods of the two models are equal, the error term vanishes, and we recover the results of Theorem 1.Theorem 2. Let θ * and θ ′ be two models, and let θ * satisfy the diversity condition (Definition 1) for both f * and g * . Let y 0 , ..., y M be the y i s from the diversity condition on g * . Let L * be the matrix with columns g * (y i ) -g * (y 0 ) and L ′ the matrix with columns g ′ (y i ) -g ′ (y 0 ). ThenEQUATIONwhereA = L * -⊤ L ′⊤ , h f * (x) = L * -⊤ ϵ y (x), and ϵ y (x) is a vector function with each entry equal toϵ yi (x) = f * (x) ⊤ g * (y i ) -f ′ (x) ⊤ g ′ (y i ) + f ′ (x) ⊤ g ′ (y 0 ) -f * (x) ⊤ g * (y 0 ). So ϵ y (x) is a function of x using the y i s from the diversity condition on g * . B will be a similar product of matrices, only using the diversity condition on f * . Also, h g * (y) will contain a ϵ x (y) which is a function of y using the x i s from the diversity condition on f * .The proof can be found in Appendix C. A consequence of Theorem 2 is that there is the following relationship between the embeddings f * (x), f ′ (x):f * (x) = L * -⊤ L ′ ⊤ f ′ (x) + ϵ y (x) (6)and a similar one for the unembeddings g * (y), g ′ (y). This equation shows that whether we can say that f ′ (x) is close to being a linear transformation of f * (x) depends on the relative size of ϵ y (x)compared to L ′ ⊤ f ′ (x). If ∥ϵ y (x)∥ << ∥L ′ ⊤ f ′ (x)∥, then the representations will be close to linear transformations of each other. Since ϵ y (x) and ϵ x (y) can be expressed in terms of differences of log-likelihoods entailed by the two models (see Appendix C), we see that they will be small if the models assign likelihoods which are close to equal to the observations. In particular, we see that for the embedding representations to be close, we need log-likelihoods to be close for all x and for all the y i 's from the diversity condition on g. Conversely, for the g representations to be close, we need log-likelihoods to be close for all y and for all the x i 's from the diversity condition on f . If the distributions for the models are equal, ϵ y (x) and ϵ x (y) will be zero, and we recover the identifiability result of Theorem 1. However, optimizing p * (y|x, S) and p ′ (y|x, S) for the correct label, y, is not enough to make this difference of log-likelihoods small. Below we present an example of this. In this example, we have M = 2 and a classification task with four labels, y 0 , y 1 , y 2 , y 3 . The example relies on the fact that if we fix non-zero angles between the unembedding vectors, and we let the embedding representations be closer to the correct label in terms of angle than the incorrect ones, then we can make the likelihood of the correct label arbitrarily close to 1 by only changing the lengths of the unembedding vectors (see Appendix E, and Appendix D for more details).For the first model θ ′ , we let the the angle between the unembeddings be very small, and the lengths of the unembeddings be very large. We generate our embedding vectors such that they have Euclidean norms larger than 1 and such that they are very close in terms of angle to the unembeddings with the correct label (see Fig. 1 ). For the second model, θ * , we spread out the unembedding representations such that three are on the axes and one is slightly off. For the embedding representations, we place them such that they are closer in terms of angle to the unembedding with the correct label, but more spread out than the ones from model θ ′ (see Fig. 1 ).We can now calculate the negative log-likelihood (N LL) for these two models using Eq. ( 1).For model θ ′ , we get N LL ′ ≈ 9 • 10 -10 and for model θ * , we get N LL * ≈ 7 • 10 -10 . In fact, for the correct labels, we get a small difference in log-likelihood for all datapoints. The maximal difference for the two models is 8 • 10 -7 . As mentioned above, we could make this loss arbitrarily small, by increasing the lengths of the unembeddings. Now both of these models have close to zero loss, and their loss thus is close to equal: however, they are very far from being linear transformations of each other. For example, the mean canonical correlation between the embedding representations is m CCA (f ′ (x), f * (x)) ≈ 0.42, which is very far from the value of 1 which would indicate a perfect linear relationship. See Appendix D for further insights on this value of m CCA and the degree of dissimilarity it indicates. It is possible to construct an example with even smaller m CCA , for example by making the angles smaller and the lengths longer for g ′ (y) for model θ ′ , while keeping the other model θ * as it is. 
conclusion: 
 We showed that the representations (embeddings and unembeddings) extracted by two models of the form in Eq. (1) will be close to being linear transformations of each other for all x and y, if the log-likelihoods entailed by the two models are close for all x and y. We also introduced a construction to show that, for the representations of two models to be close to equivalent, it is not sufficient that the losses of both models are close to each other and small. These results point to interesting questions for future research: for example, how a non-vanishing difference in log-likelihood can be connected to a measurement of representational similarity; and under what additional assumptions similarity should be expected if the difference in expected log-likelihood or in loss is non-vanishing. A More About CCA 
 
 Appendix: We show that if f * (x) = Af ′ (x), where A is an invertible matrix, then m CCA (f * (x), f ′ (x)) = 1 and that if m CCA (f * (x), f ′ (x)) = 1, then f * (x) = Af ′ (x) + bwhere A is an invertible matrix and b is a vector. So if we assume that the representations are centered, we have that ifm CCA (f * (x), f ′ (x)) = 1, then f * (x) = Af ′ (x). Proof. Assume f * (x) = Af ′ (x), where A is an invertible matrix. Then the k'th entry of f * (x) is equal tof * (x) k = A k f ′ (x)where A k is the k'th row of A. Lets k = e k , t k = A ⊤ k (7)where e k is the standard basis vector, we get the vectors used by CCA. Note that since A is invertible, the rows A k are linearly independent. Thenρ k = corr s ⊤ k f * (x), t ⊤ k f ′ (x) = 1 (8) for all k ∈ {1, ..., M }. Thus m CCA (f * (x), f ′ (x)) = 1.Assume now that m CCA (f * (x), f ′ (x)) = 1. Then there exist from CCA, vectors s k , t k such thatρ k = corr s ⊤ k f * (x), t ⊤ k f ′ (x) = 1 (9)for all k ∈ {1, ..., M }. Therefore,EQUATION)where σ is the standard deviation and c is some constant. Therefore, if we stack these equations, we getEQUATIONwhere S and T σ are invertible matrices, therefore we getEQUATIONso letting A = S -1 T σ and b = S -1 c, we have the result.We here present a merging of the identifiability results found in (Khemakhem et al., 2020) , (Roeder et al., 2021) and (Lachapelle et al., 2023) . Let θ * be a model satisfying the diversity condition on f * and g * (like in (Khemakhem et al., 2020) and not (Roeder et al., 2021) ). Assume θ ′ is another model. Thenp θ * = p θ ′ =⇒ θ * ∼ L θ ′ (13)where the equivalence relation is defined as in (Lachapelle et al., 2023) byEQUATION)where A is an invertible matrix and b is a vector.Proof. We first prove that p * = p ′ =⇒ f * (x) = Af ′ (x) for A invertible.Assume the models have equal likelihoods. Let the codomains of f * , g * , f ′ , g ′ be in R M . Let Z * (x, S) = yj ∈S exp(f * (x) ⊤ g * (y j )), and similarly for Z ′ (x, S). ThenEQUATIONEQUATIONfor all y. In particular, it is true for the M + 1 ys, y 0 , ..., y M which exist according to the diversity condition on g * . So we can write up M + 1 equations of this kind. If we subtract the equation with y 0 from these, we are left with M equations of the formEQUATIONEQUATIONLet L * be the matrix which has g * (y i ) -g * (y 0 ) as columns and L ′ be the matrix which has g ′ (y i ) -g ′ (y 0 ) as columns. We can then stack the equations to getEQUATIONand since L * is invertible,EQUATIONIf we set A = (L ′ L * -1 ) ⊤ , we only need to show that A is invertible. Using the diversity condition on f * , we pick points x 0 , ..., x M such that f * (x i ) -f * (x 0 ) are linearly independent. Let N * be the matrix with f * (x i ) -f * (x 0 ) as columns and N ′ be the matrix withf ′ (x i ) -f ′ (x 0 ) as columns. Then N * = AN ′ (25)Since we know that for any two matrices, B, C, rank(BC) ≤ min(rank(B), rank(C)), and N * has rank M , we see that A and N ′ must both also have rank M . Thus, A is invertible.Next we prove that p * = p ′ =⇒ g * (y) = A -⊤ g ′ (y) + b for A invertible and b a vector.As before we have thatEQUATIONholds for all x. In particular, it is true for the M + 1 x's, x 0 , ..., x M which exist according to the diversity condition on f * . So we can write up M + 1 equations of this kind. If we subtract the equation with x 0 from these, we are left with M equations of the formEQUATION)whereEQUATIONLet N * be the matrix with f * (x i ) -f * (x 0 ) as columns, let N ′ be the matrix with f ′ (x i ) -f ′ (x 0 ) as columns and let c be the vector with c i as entries. Then since N * is invertibleN * T g * (y) = N ′⊤ g ′ (y) + c (31) g * (y) = N * -⊤ N ′⊤ g ′ (y) + N * -⊤ c (32)Since we found before that A is invertible andA -1 N * = N ′ (33)we have thatEQUATION)where b = N * -⊤ c, and we have the result.We here provide the statement and proof of our main contribution. We show that for models as in Equation ( 1), the representations extracted by a model θ * can be written as a linear transformation of those from another model, θ ′ , plus an error term which can be seen as the non-linear part of the relationship between the functions. When the likelihoods of the two models are equal, the error term vanishes, and we recover the results of Theorem 1. Moreover, the connection we present between representations and difference in log-likelihoods can also be seen as a first step towards a "similarity-quantifying" measure (Sucholutsky et al., 2023) of representations based on the likelihood of the models.Let θ * and θ ′ be two models, and let θ * satisfy the diversity condition (Definition 1) for both f * and g * . Let y 0 , ..., y M be the y i s from the diversity condition on g * . Let L * be the matrix with columns g * (y i ) -g * (y 0 ) and L ′ the matrix with columns g ′ (y i ) -g ′ (y 0 ). Let x 0 , ..., x M be the x i s from the diversity condition on f * . Let N * be the matrix with columns f * (x i ) -f * (x 0 ) and N ′ the matrix with columnsEQUATIONwhereA = L * -⊤ L ′⊤ , h f * (x) = L * -⊤ ϵ y (x), and ϵ y (x) is a vector function with each entry equal to  and ϵ x (y) is a vector function with each entry equal toϵ yi (x) = f * (x) ⊤ g * (y i ) -f ′ (x) ⊤ g ′ (y i ) + f ′ (x) ⊤ g ′ (y 0 ) -f * (x) ⊤ g * (y 0 ). Also, B = N * -⊤ N ′⊤ , h g (y) = N * -⊤ (ϵ x (y) + c),ϵ xi (y) = f * (x i ) ⊤ g * (y) -f ′ (x i ) ⊤ g ′ (y) + f ′ (x 0 ) ⊤ g ′ (y) -f * (x 0 ) ⊤ g * (y), and c is a constant vector.Proof. We first show that f* (x) = Af ′ (x) + h f * (x).Let the codomains of f * , g * , f ′ , g ′ be in R M . Let y 0 , ..., y M be the ones which exist according to the diversity condition on g * . Let Z * (x, S) = yj ∈S exp(f * (x) ⊤ g * (y j )), and similarly for Z ′ (x, S). Thenp * (y|x, S) = p * (y|x, S) p ′ (y|x, S) p ′ (y|x, S) f * (x) ⊤ g * (y) -log(Z * (x, S)) = f * (x) ⊤ g * (y) -log(Z * (x, S)) + f ′ (x) ⊤ g ′ (y) -log(Z ′ (x, S)) -f ′ (x) ⊤ g ′ (y) + log(Z ′ (x, S))for all y. In particular, it is true for y 0 , ..., y M . So we can write up M + 1 equations of this kind. If we subtract the equation with y 0 from these, we are left with M equations of the formf * (x) ⊤ g * (y i ) -f * (x) ⊤ g * (y 0 ) + log(Z * (x, S)) -log(Z * (x, S)) = f * (x) ⊤ g * (y i ) -f * (x) ⊤ g * (y 0 ) + log(Z * (x, S)) -log(Z * (x, S)) + f ′ (x) ⊤ g ′ (y i ) -f ′ (x) ⊤ g ′ (y 0 ) + log(Z ′ (x, S)) -log(Z ′ (x, S)) -(f ′ (x) ⊤ g ′ (y i ) -f ′ (x) ⊤ g ′ (y 0 ) + log(Z ′ (x, S)) -log(Z ′ (x, S))) f * (x) ⊤ g * (y i ) -f * (x) ⊤ g * (y 0 ) = f ′ (x) ⊤ g ′ (y i ) -f ′ (x) ⊤ g ′ (y 0 ) + f * (x) ⊤ g * (y i ) -f ′ (x) ⊤ g ′ (y i ) + f ′ (x) ⊤ g ′ (y 0 ) -f * (x) ⊤ g * (y 0 ) f * (x) ⊤ (g * (y i ) -g * (y 0 )) = f ′ (x) ⊤ (g ′ (y i ) -g ′ (y 0 )) + f * (x) ⊤ g * (y i ) -f ′ (x) ⊤ g ′ (y i ) + f ′ (x) ⊤ g ′ (y 0 ) -f * (x) ⊤ g * (y 0 ) (g * (y i ) -g * (y 0 )) ⊤ f * (x) = (g ′ (y i ) -g ′ (y 0 )) ⊤ f ′ (x) + f * (x) ⊤ g * (y i ) -f ′ (x) ⊤ g ′ (y i ) + f ′ (x) ⊤ g ′ (y 0 ) -f * (x) ⊤ g * (y 0 ) (g * (y i ) -g * (y 0 )) ⊤ f * (x) = (g ′ (y i ) -g ′ (y 0 )) ⊤ f ′ (x) + ϵ yi (x)whereEQUATIONso it is a difference of log-likelihoods.Let L * be the matrix which has g * (y i ) -g * (y 0 ) as columns, let L ′ be the matrix which has g ′ (y i ) -g ′ (y 0 ) as columns and let ϵ y (x) be the vector which has ϵ yi (x) as entries. We can then stack the equations to getEQUATIONand since L * is invertible,EQUATIONIf we set A = L * -⊤ L ′ ⊤ and h f * (x) = L * -⊤ ϵ y (x), we get the result.Notes on when A is invertible We see that if the diversity condition is satisfied for g ′ with the same y i as for g * , A is invertible.If we do not assume that g ′ satisfies the diversity condition, we can use the diversity condition on f * to pick points x 0 , ..., x M such that f * (x i ) -f * (x 0 ) are linearly independent. Let N * be the matrix with f * (x i ) -f * (x 0 ) as columns, let N ′ be the matrix with f ′ (x i ) -f ′ (x 0 ) as columns and let E be the matrix with ϵ y (x i ) -ϵ y (x 0 ) as columns. Then we haveEQUATIONIf E = 0, we get N * = AN ′ . Since we know that for any two matrices, B, C, rank(BC) ≤ min(rank(B), rank(C)), and N * has rank M , we see that A and N ′ must both also have rank M . Thus, A and N ′ are invertible.If E ̸ = 0, we get from the same argument that L ′ ⊤ N ′ + E is an invertible matrix. However, this does not give us invertibility of L ′ or N ′ .Proof. Next we show that g * (y) = Bg ′ (y) + h g (y).As before we have thatEQUATIONholds for all x. In particular, it is true for the M + 1 xs, x 0 , ..., x M which exist according to the diversity condition on f * . So we can write up M + 1 equations of this kind. If we subtract the equation with x 0 from these, we are left with M equations of the formEQUATIONandEQUATIONwhich is a constant.Let N * be the matrix with f * (x i ) -f * (x 0 ) as columns, let N ′ be the matrix with f ′ (x i ) -f ′ (x 0 ) as columns, let c be the vector with c i as entries and let ϵ x (y) be the vector which has ϵ xi (y) as entries.ThenEQUATIONLetting B = N * -⊤ N ′⊤ and h g (y) = N * -⊤ (ϵ x (y) + c), we have the result.In this example, we have M = 2 and a classification task with four labels, y 0 , y 1 , y 2 , y 3 . The example relies on the fact that if we fix non-zero angles between the unembedding vectors, and we let the embedding representations be closer to the correct label in terms of angle than the incorrect ones, then we can make the likelihood of the correct label arbitrarily close to 1 by only changing the lengths of the unembedding vectors. We will first describe the unembedding vectors, g(y i ), and then describe the embedding representations, f (x), based on the g(y i )s. For model θ ′ , we let ∥g ′ (y i )∥ = 1100 for all i (see Appendix E for how this was chosen) and the angle between g ′ (y i ) and g ′ (y i+1 ) be π/16 radians, starting with g ′ (y 0 ) = (1100, 0). So we have g ′ (y 1 ) = (1100•cos(π/16), 1100•sin(π/16)), g ′ (y 2 ) = (1100 • cos(2π/16), 1100 • sin(2π/16)), g ′ (y 3 ) = (1100 • cos(3π/16), 1100 • sin(3π/16)).To generate our f ′ (x) representations, we draw 1000 samples for each label from a uniform distribution from [-π/128, π/128]. This represents the angle between f ′ (x) and the representation of the correct label g ′ (y i ). To get the length of the f ′ (x), we draw 4000 samples from a standard normal distribution, z ∈ N (0, 1), and transform it to be a value larger than 1 in the following way: ∥f ′ (x)∥ = |z| + 1. See a visualization of the f ′ (x) for model 1 in Fig. 2 .For model θ * , we let ∥g * (y i )∥ = 100 for all i, the angle between g * (y 0 ) and g * (y 1 ) will be 2π/6 radians and the angle between g * (y 2 ), g * (y 3 ) and g * (y 3 ), g * (y 0 ) be π/2 radians, starting with To generate our f * (x) representations, we take the samples we drew from the uniform distribution for the first model and multiply them with 16, so we get samples from a uniform distribution from [-π/8, π/8]. These represent the angle between f * (x) and the representation of the correct label g * (y i ). We use the same lengths for the f * (x)s as in model 1. See visualization in Fig. 3 .We can now calculate the negative log-likelihood (N LL) for these two models using Eq. ( 1). For model θ ′ , we get N LL ′ ≈ 9 • 10 -10 and for model θ * , we get N LL * ≈ 7 • 10 -10 . In fact, for the correct labels, we get a small difference in log-likelihood for all datapoints. The maximal difference for the two models is 8 • 10 -7 . As mentioned above, we could make this loss arbitrarily small, by increasing the lengths of the unembeddings. Now both of these models have close to zero loss, and their loss thus is close to equal: however, when we calculate the mean canonical correlation between the embedding representations, we get m CCA (f ′ (x), f * (x)) ≈ 0.42, which is very far from the value of 1 which would indicate a perfect linear relationship. To get an idea of how dissimilar this is, let f ′ noise (x) be f ′ (x), where we have added Gaussian noise to each dimension with four times the variance that f ′ (x) has in that dimension. We then get m CCA (f ′ (x), f ′ noise (x)) ≈ 0.44. If we only add noise with twice the variance, we get m CCA (f ′ (x), f ′ noise (x)) ≈ 0.57. Thus, we can see that the representations of the models are quite far from being similar.We can also calculate L ′ ⊤ f ′ (x) and ϵ y (x) (the equivalent of Eq. ( 6)) for the models and consider the relative size of ϵ y (x) compared to L ′ ⊤ f ′ (x) in each dimension. We use y 0 , y 1 and y 3 for the diversity condition and let′ ⊤ = (g ′ (y 1 ) -g ′ (y 0 )) ⊤ (g ′ (y 3 ) -g ′ (y 0 )) ⊤ (53)Let L ′ ⊤ f ′ (x) i be the i'th dimension of L ′ ⊤ f ′ (x). Then This means that on average the value of the "error term" in one of the dimensions is more than twice as large as the representation term and thus it makes sense that the representations are not close to being linear transformations of each other.Buchholz and Schölkopf (2024) also consider approximate identifiability, but for a different model class. 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: The authors have highlighted the shortcomings of applying Identifiability theorem to two different models with near-optimal loss value. The work is of great interst as it could be employed in enhancing the understanding of model's representations.
main_review: main_review: ### Summary
The paper highlights the shortcomings in understanding representational similarity between two models using the Identifiability theory:
- The satisfiability of the key assumption that the two models should have the same log-likelihood is rare.
- The two converged models with near-optimal loss values may not have similar representations.

### Strengths
The mentioned key challenges are corroborated by -
- A theoretical proof of the theorem that representation obtained by one model can be written as a sum of a linear transformation of the representations obtained by another model and an error term that vanishes if two models have the same likelihood for all x and y.
- An experimental example to show that two models with near-optimal loss value can result in dissimilar representations.

### Weakness
The experiment is shown on representations that are manually created to achieve near-optimal loss. However, it would be better to verify this for neural networks trained on similar datasets with near-optimal loss value but with different initialization.

### Significance, clarity, novely
The paper highlights the shortcomings of the Identifiability theory. The work is of great significance as it could be leveraged to enhance the model's interpretability. The paper is well-written with sufficient originality.
title: title: Interesting and substantial work
main_review: main_review: **Pros:**
* This work is high quality and novel.
* This paper provides significant and interesting theoretical results for representational similarity, and reveal open questions/problems for future research. 
* The toy example provided in Section 4 showing that two models with close-to-equal losses produce very different representations is interesting and helpful.

**Cons:**
* While most of the paper is clear, there is a critical component that was not clearly defined. Under section 2, $g^*$ and $g'$ are introduced as unembedding functions. But what are unembedding functions? Do they map the labels $y$ to embeddings $z$? Given the importance of unembedding functions in the key results of the paper, I think it would be helpful to clearly explain what they are.
title: title: The paper addresses challenges in explaining representational similarity through identifiability theory but lacks empirical validation and practical solutions.
main_review: main_review: The paper addresses the question of "Why different models often produce similar data representations?". Focusing on the challenges of explaining this phenomenon through identifiability theory, the authors identify two key obstacles: First, they highlight that the assumption of exact likelihood equality is rarely met in practice. To address this, they extend existing identifiability results by relating deviations in representations to differences in log-likelihoods. Second, they demonstrate that even models with near-optimal and similar loss values can produce highly dissimilar representations due to the distinction between loss and likelihood.

Although the paper successfully identifies key challenges, it lacks concrete solutions or empirical evaluations to test its theoretical claims. Including experiments or practical suggestions on how to overcome the highlighted obstacles would have provided a more well-rounded contribution. Additionally, the paper could benefit from more detailed discussions on the potential implications of the findings for broader representation learning scenarios, particularly for models trained on different datasets or with varying objectives.
title: title: Review of Challenges in Explaining Representational Similarity through Identifiability
main_review: main_review: - I think this paper makes a nice point: two models with similar losses needn't have similar likelihoods for all pairs of inputs and outputs, and thus (at least in this particular model class) this means that two models with nearly identical performance needn't have representations that are linear transformations of one another
- I like the constructed example, but I wonder if some other example is possible with an even lower mean CCA score?
- What's missing to me is how meaningful this observation is in practice on real data
- nit: Please use `\citep{}` and `\citet{}` as appropriate in the context

Comments: title: Thank you to the Reviewers and Edits
comment: We would like to thank all the reviewers for their comments. 

In response to the reviews, we have 

- Added some explanation of the embedding and unembeddings functions in section 2 under "Model Class". 
- Added a the end of Section 4 how it is possible to construct an example with even lower m_CCA
- Changed the use of citet and citep 

We agree with the reviewers that is is a good idea to run experiments to show how something like the constructed example can occur in practice, and we will do this in future work.