Title: Representational Difference Explanations
Venue: NeurIPS 2025
Accepted: yes
Keywords: interpretability

explainability

XAI

graph

comparison

similarity

differences

representational similarity
Abstract: We propose a method for discovering and visualizing the differences between two learned representations, enabling more direct and interpretable model comparisons. We validate our method, which we call Representational Differences Explanations (RDX), by using it to compare models with known conceptual differences and demonstrate that it recovers meaningful distinctions where existing explainable AI (XAI) techniques fail. Applied to state-of-the-art models on challenging subsets of the ImageNet and iNaturalist datasets, RDX reveals both insightful representational differences and subtle patterns in the data. Although comparison is a cornerstone of scientific analysis, current tools in machine learning, namely post hoc XAI methods, struggle to support model comparison effectively. Our work addresses this gap by introducing an effective and explainable tool for contrasting model representations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method called Representational Differences Explanations (RDX), aimed at identifying and explaining differences between the learned representation spaces of two models. The method automatically identifies groups of input samples that are close in one model’s representation space but far apart in the other, or vice versa. The authors introduce a quantitative metric to assess how well such groups reflect the divergence between models’ internal representations.\nRDX is compared against classical clustering (k-means) and sparse dictionary learning methods (e.g., NMF, Sparse Autoencoders). The evaluation is carried out on controlled settings such as MNIST and CUB, and the paper includes an ablation study and qualitative examples that demonstrate how RDX can uncover previously unknown model differences.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n- The paper tackles the challenge of systematically explaining differences between learned representations across models, a question that has received far less attention than single-model interpretability.\n- The paper is clear and accessible. Many natural questions that arise while reading are addressed in the appendix, making the paper thorough without being dense.\n- RDX is composed of well-established components, such as spectral clustering and local neighborhood distances, combined in a novel way to address the representation difference problem. \n- The authors propose a concrete, intuitive metric to assess the quality of difference explanations. While simple, it aligns well with the goals of the method and is easy to compute.\n- The method is evaluated both quantitatively and qualitatively on controlled datasets like MNIST and CUB.\n- The case studies clearly show that RDX can uncover subtle, meaningful, and non-obvious differences between model representations.\n- The comparison to dictionary learning methods (e.g., NMF, Sparse Autoencoders) illustrates where these classical approaches fall short in identifying contrastive differences between models.\n\nWeaknesses:\n- The baselines used in the paper (e.g., k-means, basic sparse dictionary learning) are relatively simple and unsurprisingly underperform. This weakens the strength of the empirical comparisons. Incorporating more recent and relevant methods, such as those explicitly designed to compare representation spaces [1, 2], would provide a stronger benchmark. In particular, USAE [1] could be applied by training a larger sparse autoencoder over both models and then identifying concepts that are selectively activated in only one model.\n- The method relies on computing pairwise distance matrices, which scale quadratically with the number of samples. While experiments use up to 5,000 samples, this becomes problematic for large-scale datasets (e.g., millions of samples). The paper does not address how RDX would scale to such settings, where sparsity of concepts or subsampling may become essential.\n- The SAE comparison is limited to very low-dimensional hidden spaces (e.g., 3 or 5 units/neurons), which may restrict the richness of the discovered differences. Modern alternatives like top-K sparse autoencoders could offer more expressive and interpretable explanations, and should be considered\n- The method assumes that shared local concepts imply similarity in the representation space. However, this assumption may not hold in more complex settings where the relevant concept is a small part of the input (and background features differ a lot). In such cases, using input-space similarity to assess representational similarity might be misleading or insufficient.\n\nReferences:\n\n[1] Thasarathan, Harrish, et al. \"Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment.\" arXiv preprint arXiv:2502.03714 (2025).\n\n[2] Vielhaben, Johanna, et al. \"Beyond scalars: Concept-based alignment analysis in vision transformers.\" arXiv preprint arXiv:2412.06639 (2024).\n\nQUESTIONS:\n(1) Did/Can dictionary learning methods uncover any different types of representational differences compared to RDX?\n\n(2) line 57: The statement “Visual concepts are usually defined as images that the model considers to be similar” may be unprecise. I was under the impression that visual concepts rather correspond to recurring input patterns or localized semantics, not just described by overall image similarity.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method, Representational Differences Explanations (RDX), to compare representations of different models. The authors evaluate the method both on representations with known and unknown differences and show examples where RDX yields better explanations of the differences.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I think the paper proposes an interesting method that can give new insights into the differences between the representations of models. However, there are some problems that need to be addressed or justified.\n\n**Strengths:**\n- The problem, as described in the Introduction and illustrated by Figure 1, is well-motivated and easy to understand. \n- The paper is clear, well-written, and easy to understand.\n- The problem addressed in this paper is timely and important.\n- It seems to me that there are sufficient details needed to replicate the experiments.\n- Based on the figures provided, it seems that the method works and can reveal interesting insights into the differences between the models. It is able to capture differences between two models instead of focusing on the most important explanations for each model separately.\n\n**Weaknesses:**\n- The examples shown might potentially be cherry-picked. For example, in Figure 6, five different explanations were identified in each case, but images are shown only for two of them. How can one make sure that these were not shown solely because they support the claims and make a nice story? \n- Thorough evaluation of the method is missing. The binary success rate metric (BSR) seems to me a bit cyclic, since it evaluates something that the RDX method was explicitely developed to do (identify pairs that are closer in one representation than the other). But the other methods, that RDX are compared to, do not have this as a primary goal, so it is perhaps not so surprising that RDX outperformes the rest. But does it neccessarily imply that it gives better explanations? (I also acknowledge that objectively evaluating explanations is a complicated issue that perhaps goes beyond the scope of this paper.) \n- The method depends on Euclidean distances between points from a chosen dataset. I think this is problematic from two perspectives. First, there is a strong dependency on the dataset choice. Since the distances are defined as a ranking of the nearest neighbors (and not how far the points actually are), adding or removing points from the dataset would heavily influence the distance matrices. For instance, in the experiments where only a subset of ImageNet classes are used, would the explanations be different if all ImageNet classes were used for testing? Second, the representations usually live on curved manifolds in high-dimensional spaces, where Euclidean distances do not make sense. For example, two points can be close in Euclidean geometry, but very distant in manifold geometry. How are these differences taken into account in RDX?\n\nQUESTIONS:\n(see the weaknesses for more information and context)\n1) Can you provide any guarantees that changing the dataset would have only a minor effect on the final explanations? Or, at least, show it empirically?\n2) Is there any reason to believe that using Euclidean distances in your method makes sense even when the data lies on curved manifolds?\n3) How did you choose the examples of explanations shown in the paper? Were there any objective criteria used for choosing what to report (and what not to)?\n4) Regarding representational alignment (section 3.4): it is not clear to me when exactly this step happens. Do you use the transformation in all the experiments, or only in some? How does it change the resulting explanations?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes RDX, a novel explanation method that identifies differences in representations between two models and visualizes them as concepts via examples. Explanations are evaluated qualitatively, as well as quantitatively through BSR, a newly proposed metric for evaluating difference explanations.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths.** The paper is extremely well-written and structured. The proposed method is innovative and provides a new type of difference-based concept explanations that will be of significant interest to the XAI community. The addressed issue of revealing behavioral differences between models is explained in detail and supported by experimental results. Explanations are evaluated both qualitatively and quantitatively. The proposed method is compared to several state-of-the-art related methods for dictionary learning. Visualizations are informative. The large appendix ensures reproducibility and provides several additional results that answered many of my initial questions. \n\n**Weaknesses** \nBSR, the proposed metric for evaluating explanations (and similarly its variants in the Appendix) simply counts the frequence where two examples from the explanation grid have a lower representational distance in A than in B (i.e., $D_A$ < $D_B$). The sampling of difference explanation grids for RDX is done in a way that only indices with pairwise negative entries in $G_{A,B}$ (line 149), i.e. with $D_A < D_B$ according to Equation (1) are chosen. While I agree that BSR may be a desirable property for difference explanations, its formulation is too close to that of RDX to serve as the (only) quantitative comparison to other explanation methods. I would have liked some comparison via other quantitative metrics as well, e.g., evaluating how/if the found representational differences affect model performance or some of the metrics listed here [1] to offer a more holistic comparison to other explanation methods.\n\nIIUC, since RDX is not symmetric, comparing two models requires inspecting two sets of explanations: RDX(A, B) and RDX(B, A) to discover potential representational differences. I am wondering how and if this scales to comparing more than two models at once, due to this asymmetry. \n\n[1] Dreyer, Maximilian, et al. \"Mechanistic understanding and validation of large AI models with SemanticLens.\" arXiv preprint arXiv:2501.05398 (2025).\n\nQUESTIONS:\n- Please rebut my concerns regarding using only the BSR metric for quantitative comparisons or include a comparison via other evaluation metrics, to offer a more holistic view.\n- Please answer my above question regarding scalability of RDX beyond pair-wise comparisons\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduces Representational Difference Explanations (RDX), a post-hoc analytic technique that surfaces where two models disagree in their learned representations. RDX builds pairwise distance matrices for two embedding spaces, forms an asymmetric, locally-biased difference matrix that emphasises disagreements on near-neighbours, converts that matrix into an affinity graph and applies spectral clustering, then turns each high-affinity cluster into a concise image grid that itself is the explanation. Experiments showcase the metholodogies performance on CUB, MNIST and large-scale “unknown difference” settings.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper tackles an interesting problem of comparing two models' explanations through post-hoc analysis
- The paper does not require gradient access or optimization, making it practical for real world applications
 
Weaknesses:
- In sections 4.1 and 4.2, the author conduct experiments for dictionary learning a modified MNIST setting, without explaining the rationale behind this choice. MNIST is utilized for comparing digits as some of them can look similar to others.
-Authors do not compare their method with other XAI methods. The baselines include models that have been largely explored in the literature.
- The evaluation basis of the method is largely visual. Tables are placed in the appendix, making the main text of the paper difficult to follow, as it's very notation-heavy.

QUESTIONS:
Questions:
- Could the authors explain their choice for conducting experiments on a modified MNIST setting, instead of the original?
- Is there any comparison with recent explanation methods?
- Could the authors cite some metrics or numerical evaluation for their method is the main text of the paper?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02