Title: When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective
Venue: NeurIPS 2025
Accepted: yes
Keywords: Identifiability

Representational Similarity

Distributional Distance

Robustness

Representation Learning
Abstract: When and why representations learned by different deep neural networks are similar is an active research topic. 
    We choose to address these questions from the perspective of identifiability theory, which suggests that a measure of representational similarity should be invariant to transformations that leave the model distribution unchanged.
    Focusing on a model family which includes several popular pre-training approaches, e.g., autoregressive language models, we explore when models which generate distributions that are close have similar representations. We prove that a small Kullback--Leibler divergence between the model distributions does not guarantee that the corresponding representations are similar. This has the important corollary that models with near-maximum data likelihood can still learn dissimilar representations---a phenomenon mirrored in our experiments with models trained on CIFAR-10.
    We then define a distributional distance for which closeness implies representational similarity, and in synthetic experiments, we find that wider networks learn distributions which are closer with respect to our distance and have more similar representations. Our results thus clarify the link between closeness in distribution and representational similarity.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies when two independently trained neural networks learn similar internal representations if their output distributions are close. It argues that a good similarity measure should be invariant to transformations that leave the model's output distribution unchanged. The paper argues that a small KL divergence between two models output distributions does not guarantee that their learned representations are similar, specifically that for any arbitrarily small $\\varepsilon$, one can find two models with $d_{KL} \\le \\varepsilon$ whose embeddings are far from linearly equivalent. Even models with arbitrarily low classification loss can learn entirely dissimilar features. It introduces a new log-likelihood variance (LLV) distance between distributions and a PLS-SVD-based dissimilarity between representations. LLV is tailored to identifiability and is a proper metric. Using these, they prove that if two models distributions are close under the proposed LLV distance, then their representations are guaranteed to be similar. Concisely, closeness in the usual KL sense is insufficient to ensure representational similarity, but closeness in the new metric does imply it.\n\nExperimentally, the authors train two ResNet18 models with a 2D bottleneck on CIFAR-10. Although both runs reach virtually identical test loss, the embeddings of the data are distinct, making CCA moderate and KL low while LLV distance remains high, validating their arguments. On a synthetic 2D classification toy dataset they show that as width grows, the mean and variance of LLV across seeds fall steadily, and the corresponding dSVD gap shrinks, suggesting higher-capacity models converge toward a common representation when their distributions align under LLV.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- Theoretical: The paper proposes a rigorous perspective on representational similarity which is novel. Theorem 3.1 demonstrates a counter intuitive fact that even extremely small differences in output distributions can correspond to large differences in internal representations. This a relevant result w.r.t. efforts such as model distillation. Theorem 4.6 is likewise a meaningful contribution: a new distributional distance specifically designed to measure representation similarity under some assumptions.\n- Experiments: While not large scale, they are well-designed to illustrate the argument. Section 5.1 shows a concrete example of two standard models with indistinguishable performance metrics but significantly different internal representations. This example makes the theoretical phenomenon tangible. The synthetic experiments in Section 5.2 show a clear trend that increasing model capacity yields closer distributions and more aligned representations.\n- Clarity: The paper is generally well-written, albeit a bit dense.\n- Novelty: LLV is a novel, theoretically validated distance that shows promise in experiments.\n- Reproducibility: code included in supplementary materials, public release promised.\n\n# Weaknesses:\n- Assumptions: Theorem 3.1 requires certain conditions that somewhat limit its scope ($k \\geq M+1$). It's not entirely clear how severe this limitation is, and perhaps is a reasonable assumption to make. Language models frequently have far more tokens than the embedding dimension, but it may be much closer (or reversed) with a vision model on ImageNet. Similarly, Theorem 4.6 comes with some assumptions, but again not clear on the severity.\n- Bounds: The bound on dSVD ($2M\\varepsilon$, will worsen with dimension) is not very tight, though the authors acknowledge this. The CIFAR experiments didn't enter the setting where the bound is non vacuous.\n- Experimental: While the experiments do support the claims, they could be considered limited in scope. All experiments use relatively low-dimensional representations and simple datasets (e.g., CIFAR). Validation in higher dimensional cases or on a larger dataset (i.e., ImageNet) would be a valuable addition. Combined with the loose bounds, assessing the overall practical utility of LLV is a bit harder.\n\nQUESTIONS:\n- Have you looked at LLV for larger models / higher dimensional representations? Do the same conclusions hold? Curious about practicality of the metric.\n- It seems like this might be intractable for something like language models, do you see anyway of accelerating / approximating this?\n- I assume other f-divergences would have a similar issue as KL, was this looked at?\n- Any thoughts on whether LLV could be used as an explicit penalty (loss) in something like distillation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA typical identifiability treatment tells us that equal likelihoods imply similar representations, up to an identifiability class. What about unequal but similar likelihoods? The authors in this work study model and representational similarity in a large class of models, and show that similar likelihoods do not necessarily imply similar representations. The authors define the identifiability class of their model to establish their results, and use it to define a new distributional distance. They then use this distributional distance to bound representational similarity. Their results are supported by numerical results on the CIFAR dataset and synthetic experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- Well-motivated and well-framed question on identifiability and its relaxations. \n- Impressive theoretical results and new definitions. \n- A simple synthetic example corroborates a proof.\n- Numerical experiment on CIFAR is non-trivial. \n\nWeaknesses:\n- Lacks comparison to the conclusions that could be drawn from existing approaches. For instance, CCA is the most raised alternative -- could the relationship to this model be further discussed? What would we miss by just using CCA?\n- Some assumptions are not discussed to their fullest extent. For instance, does the pivot $x_0$ (especially its probability $p_{\\mathcal{D}}(x_0)$) influence the distance or log-likelihood variance distance terms?\n\nQUESTIONS:\nSee weaknesses above for some questions, with just one more here:\n- Shape metrics from Williams et al (https://arxiv.org/abs/2110.14739) use **groups** to define invariance in shape classes, going beyond CCA -- can this be leveraged for your identifiability class, to be more specific than the CCA-based measures that consider any linear transformations?\n\nLIMITATIONS:\nLimitations are discussed in the last section. Perhaps including an example of the failure modes mentioned could strengthen the text?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper looks at co-embedding-based classification models, where inputs and outputs are mapped to a shared space and then inner products are used in a softmax to obtain conditional distributions.  The central question is whether, given access to all the conditional distributions for two models trained on the same data, one can claim anything about the similarity of the representations of the two models in the shared space (pre-softmax).  \n\nAs a first guess, one might try using one of the popular statistical distances between corresponding conditional distributions, e.g. KL divergence, to assess classification similarity and infer that the models’ representations are closely related if the distance vanishes.  The authors show that this is not the case in general -- there are ways to obtain arbitrarily close output distributions with representation spaces that are not relatable through linear transformations.  \n\nThe paper then proposes a statistical distance -- which additionally satisfies the four properties of a metric -- that essentially inverts the softmax operation by operating on log probability ratios and can bound how similar the representation spaces are up to linear transformation.  Some experiments on CIFAR10 and synthetic examples are included to bolster the theory, with an interesting result suggesting larger intermediate dimensions increase the final representation space similarity.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper is clearly written and well-structured.  The flow from proposing KL divergence, to providing an intuitive counterexample, to proposing the LLV distance is effective.  Prior work and limitations are addressed with transparency.\n\n- Linking the set of classification probability distributions to the underlying representation space, for a format that is broadly used in machine learning (the co-embedding space matching x to y), offers a foothold into novel assessments of representation similarity.  The identifiability angle is powerful.  The metric properties of d_LLV, though not utilized in this work, are an added bonus.\n\n## Weaknesses\n\nThe experimental scope is limited, especially in that all embedding spaces are restricted to 2D, raising concerns about whether the observed behaviors extend to higher-dimensional settings typical of practical applications.\n\nQUESTIONS:\n- It would be helpful to understand whether the use of 2D embedding spaces in the experiments was driven by practical considerations (e.g., visualization) or if similar behaviors are expected in higher-dimensional regimes as well.\n\n- The CIFAR point clouds (Fig 3 and Supp) look like radial information of the image embeddings might not be playing a role (due to the high density near the origin).  A quick look at the code suggests the norm might be fixed for these experiments.  While fixing the norm for the label embeddings serves a purpose for the different synthetic experiments, applying norm constraints to the *image* embeddings for CIFAR10 seems less directly motivated by the theory.  This may weaken the empirical alignment with the theoretical results and limit the already constrained experimental scope.  Could the authors clarify whether this design choice is discussed in the text?  And more generally, can they speak to the rationale for norm-fixing in this setting?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors aim to analyze whether similarity between two models at the level of behavior (predictions made by a model) implies similarity at the level of representations. To this end, the authors contextualize results from identifiability theory (which show equal likelihood implies representational similarity up to invertible transformations under certain diversity conditions) and argue that developing a meaningful analysis requires first formalizing what similarity metrics even make sense. The authors thus propose a new \"distributional distance\" that bounds representational dissimilarity, and enables statements on similarity of representations based on closeness of predictions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths.** The question of how and when similarity in the predictions space implies similarity in the representation space is extremely useful: if true, training different models to merely learn the distribution will yield the same representations and be of similar utility. The paper is very well-written, though partially lacking in experiments that could have been helpful for emphasizing the theory. \n\n**Weaknesses.** The weaknesses below are based on my interpretation of results. Please correct me if I'm wrong in how I understood the results and I'll happily change my scores in accordance with that.\n\n- Construction-centric nature of Theorem 3.1 and related arguments: The primary argument in the formal analysis is based on construction of a model class that is equivalent in predictions, but not in representations. While fair, the argument ignores learning dynamics: if a model does not ever learn solutions beyond a narrow subset of the elements of this prediction-equivalence class, should we worry about the *possibility* of representational dissimilarity? \n\n- Permutation based representation dissimilarity in Theorem 3.1: My understanding of the arguments up to theorem 3.1 was that authors deem models with representations equivalent up to linear transformations as representationally similar. Wouldn't permutations fall under the scope of this definition? If so, I don't follow why the intuitive figure (Fig 2) relies on permutation of representations to produce representationally dissimilar models. More broadly, isn't a permutation transformation too trivial to consider models representationally dissimilar? I would in fact argue that representational similarity analyses should by default ignore all trivial symmetries of neural network architectures (e.g., permutations of neurons yielding a behaviorally equivalent model).\n\n- Empirical characterization: I regard the paper mostly as a theory work and hence this is a minor weakness. Specifically, I found the empirical analysis somewhat underwhelming. For example, the authors argue based upon their defined measure that representational similarity increases with width, an arguably known result from past work on CCA / CKA based similarity analyses [1]. However, offering a tool should ideally be grounded in some interesting conclusion or application.\n\n[1] https://arxiv.org/abs/1905.00414\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}