Title: Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: representational alignment

interpretability

concept-based explanations
Abstract: Measuring the alignment between representations lets us understand similarities between the feature spaces of different models, such as Vision Transformers trained under diverse paradigms. However, traditional measures for representational alignment yield only scalar values that obscure how these spaces agree in terms of learned features. To address this, we combine alignment analysis with concept discovery, allowing a fine-grained breakdown of alignment into individual concepts. This approach reveals both universal concepts across models and each representation’s internal concept structure. We introduce a new definition of concepts as non-linear manifolds, hypothesizing they better capture the geometry of the feature space. A sanity check demonstrates the advantage of this manifold-based definition over linear baselines for concept-based alignment. Finally, our alignment analysis of four different ViTs shows that increased supervision tends to reduce semantic organization in learned representations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new method for evaluating the similarity and differences in vision models' representation by combining concept discovery with alignment analysis. As opposed to defining concepts as linear subspaces in representation space, this paper approaches concepts as being embedded in low-dimensional non-linear manifolds. Network's internal features are projected via UMAP and then clustered with HDBSCAN. Then, the distances of clusterings from different networks are calculated. This method is evaluated on multiple vision models (e.g., DINO, CLIP, etc.) with ImageNet images.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* The idea of measuring similarity/differences between networks using interpretable concepts rather than a single scalar is interesting and can be helpful in better probing what these model do and don't know. \n* Most works on interpreting representations assume the linear representation hypothesis, while this work is more unique in assuming non-linearity and leveraging non-linear tools. \n* The proposed method (NLMCD) passes sanity checks better than baselines. Additionally, some qualitative observations are made the show the utility of NLMCD. \n\n**Weaknesses**\n* The ultimate goal of a new alignment metric would be to facilitate some new insight. Examples from the paper, such as finding that earlier layers focus on lower level features, are good sanity checks that have already been established. Some findings that would not be enabled by other similarity metrics would strengthen the paper. The most interesting finding was that models which were supervised with labels have reduced semantic structure in the learned representations. I think this can be emphasized more and further developed. \n* The method itself is not completely novel. It combines existing tools such as UMAP and HDBSCAN. This makes it more important to emphasize how this combination of tools enables the discovery of new phenomena which otherwise would not have been discovered. \n* [1] seems similar in spirit. It clusters neurons across vision models based on similarity, forming concepts.\n\n[1] Dravid et al. \"Rosetta Neurons: Mining the Common Units in a Model Zoo.\" ICCV 2023.\n\nQUESTIONS:\n* The paper can be further strengthened by showing how the method enables new insights into the representations of these models. Most of the experiments just demonstrate sanity checks. The finding that models supervised with labels have reduced semantic structure in the learned representations is interesting. Why does this not happen in CLIP as it has supervision in the form of text?\n\nHere are a few questions that this method can be used to answer:\n* What is the same and different between generative and discriminative models. How does the semantic structure differ and how is it similar?\n* I do not see any reason this method cannot be applied to a language model. How is the semantic structure different between the vision and language encoder in CLIP?\n\nThese are just a few ideas that can make me more convinced of the utility of the method.\n\nLIMITATIONS:\nYes, the authors have a dedicated limitations subsection mentioning the limited scalability of HDBSCAN.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This paper proposes an interesting analysis pipeline for investigating and exploring the alignment properties of representations in Vision Transformers. Specifically, unlike existing approaches that reduce alignment to a single scalar value, the authors introduce a concept-based alignment metric that treats concepts as nonlinear manifolds. In practice, they use a soft clustering method, i.e., HDBSCAN applied on the UMAP projection, as a proxy for identifying these concept manifolds. This metric is then used to evaluate alignment across fuzzy concept clusters derived from different architectures. The proposed pipeline is applied to four ViT models, i.e., FS, CLIP, DINO, and MAE, revealing insightful structural differences. The analysis enhances the interpretability of Vision Transformers and highlights their potential in downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n\n- The proposed method shows strong potential for future extensions.\n- The approach is conceptually simple and intuitively appealing.\n- The paper presents the proposed methodology in a clear and well-structured manner.\n\n# Weaknesses\n- The proposed method appears to be time-consuming, as it requires traversing all vectors. However, the authors do not provide any analysis of the computational cost, which undermines the practical value and contribution of the method.\n- The paper relies on ImageNet class labels and coarse WordNet categories as weak proxies to annotate discovered concepts, which may not accurately capture the true semantic content of each concept manifold. Moreover, in the CLIP experiments, the analysis is confined to the visual encoder; extending the analysis to CLIP’s text encoder and to the joint image–text embedding space could reveal richer multimodal alignment patterns and better validate the method’s generality.\n\nQUESTIONS:\n- Appendix B describes the design choices for hyperparameters such as UMAP’s min_dist, n_neighbors, and embedding dimensionality, as well as HDBSCAN’s min_cluster_size and min_samples, based on maximizing the DBCV score. It would be helpful to include a hyperparameter sensitivity study to show how variations in these settings affect metrics like noise rate, DBCV, NRMSE, and CBA alignment scores. This would further clarify the robustness and stability of the proposed method.\n- The proposed analysis pipeline is suitable to be extended to multimodal foundation models, or compare the image and text encoders in CLIP, and assess how their fusion affects representational alignment?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework to analyze and compare deep visual representations at a conceptual level rather than with a single similarity score. The authors combine concept discovery (finding interpretable clusters or “concepts” in a model’s latent space) with representational alignment analysis, yielding fine-grained insights into how different layers or models align in terms of learned features. Key contributions include: (1) defining concepts as non-linear manifolds in feature space (as opposed to traditional linear directions), and developing a clustering-based method to identify these concept manifolds (using UMAP for dimensionality reduction and HDBSCAN for unsupervised clustering); (2) introducing a concept-based alignment (CBA) measure using a generalized Rand index that quantifies similarity between two sets of concept clusters, which can be decomposed into pairwise distances between individual concepts; and (3) applying this analysis to several Vision Transformers (ViTs) trained with different paradigms (supervised, contrastive, self-supervised, etc.), revealing how their internal representations differ. Notably, the paper finds that certain high-level concepts are universal across models, while others are model-specific, and that increased training supervision tends to reduce the semantic organization of learned features. The proposed approach thus goes “beyond scalars” like CKA by pinpointing which concepts are shared or divergent between representations, offering a more interpretable comparison of model internals.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. Innovative Fine-Grained Analysis: The paper addresses a clear limitation of existing representational similarity measures (e.g. CKA) which collapse complex relationships into a single number. By integrating concept discovery with alignment, the method yields rich, interpretable information about what is similar between layers or models (i.e. which concepts align). This is a novel perspective that provides insights not available through traditional scalar metrics.\n2. Novel Concept Definition: Treating concepts as non-linear manifolds in feature space is an innovative generalization beyond prior linear subspace definitions. This more flexible definition is well-motivated (aligning with the manifold hypothesis) and the authors demonstrate its advantage: in a controlled “sanity check,” their non-linear concept-based alignment outperforms simpler linear concept approaches in capturing expected layer-wise similarity patterns.\n3. Insightful Findings for Vision Transformers: Using the proposed analysis, the paper provides interesting observations about ViT models. For instance, self-supervised models (DINO, MAE, CLIP) develop strongly semantic, coherent concept structures in later layers, whereas the fully supervised ViT (FS) shows less semantic organization in its top-layer concepts. \n4. Clarity and Organization: The paper is well-structured and easy to follow. \n\nWeakness:\n1. Complexity and Scalability: A potential concern is the computational complexity of the concept discovery pipeline. The approach relies on UMAP for embedding and HDBSCAN for clustering, which required the authors to subsample the dataset (25% of ImageNet and spatial token pooling) due to scalability limits. This subsampling raises the question of whether some fine concepts might be missed or labeled as “noise” by HDBSCAN when data is limited.\n2. Scope Limited to ViTs: The experiments focus on four ViT models with a specific architecture and on a single dataset (ImageNet). It remains unclear how well the proposed alignment analysis generalizes to other network architectures (e.g., CNNs or different ViT variants) or other data domains.\n\nQUESTIONS:\n1. Semantic coherence of discovered concepts: By qualitative inspection, the discovered concept clusters (e.g. an “apples” concept in Fig. 3) appear semantically coherent, with the nonlinear (NLMCD) clusters showing high within-cluster consistency. How do the authors evaluate or ensure that these data-driven concepts are indeed meaningful and interpretable to humans?\n\n2. Fine-grained CBA vs. scalar alignment (CKA): The proposed Concept-Based Alignment (CBA) measure achieves performance on par with standard scalar alignment measures like CKA in the sanity check, while additionally providing a fine-grained view into how individual concepts align. What new insights does this concept-level alignment enable that a single scalar (e.g. CKA) might obscure? For instance, have the authors observed cases where two models or layers have similar overall alignment scores yet differ in which specific concepts are aligned (or misaligned)? Highlighting such a scenario would clarify the practical advantage of CBA’s fine-grained analysis over traditional one-number similarity metrics.\n\nLIMITATIONS:\nSee weakness and questions", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method of concept level alignment analysis in pre-trained vision transformers. The first part of the author's solution involves building a manifold based approach of concept discovery in neural network representations which they call NLMCD: first they reduce the representation dimensions using UMAP, and then perform heirarchical clustering using HDBSCAN. This leads to 'soft' assignment of feature representations to different concept clusters. In order to develop an alignment metric using their soft clustering, the authors adapt a generalization of Rand index - the distance between two concept clusters is given as an average of distances between two membership vectors. The authors define concept-based alignment (CBA) as difference of this averaged distance from 1.The authors perform several quantitative checks on their concept discovery approach, and the results from these are somewhat mixed. A sanity check on the CBA metric is performed against CKA which shows that the author's metric offers more fine-grained alignment scores. Lastly, and perhaps the most impactful section of the work, the authors perform concept level alignment analyses on four different pre-trained models on ImageNet and confirm several results from representation level analyses in the past.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. Concept-based alignment instead of representation-based alignment is a better approach towards quantifying representation similarities across neural networks, training objectives, and training distributions. The authors approach is among one of the first methods towards this, and as such quite novel. \n2. The concept alignment analysis is quite interesting, the authors' analysis helps to re-establish some representation level results for feature learn through concept level analysis too (e.g. inter-modal concept alignment in early layers, similarity in concepts across similar training objectives.) \n\n**Weaknesses**\n\n1. The algorithm used for calculating concept discovery (UMAP followed by HDBSCAN) is very compute intensive, which is reflected in the author's experimental choices - using 25% of ImageNet, pooling sequence tokens, perhaps the use of UMAP itself. It will be hard to translate this concept discovery method (and hence alignment metric) to web scale datasets, where alignment is perhaps more important as data quality is not controlled. \n2. The experimental results for concept discovery evaluation are not presented well, and also have a lot of variance across models. Why does the NRMSE drop for intermediate layers for some models? Yet other models like CLIP have their values same across all layers? Robustness decreasing with layers is generally not a positive sign, although the absolute values are themself high enough where it might not be an issue. Overall, it is not clear to me as a reader what inference and conclusions to draw from the author's results. Perhaps the section 4.1 evaluation can be broken down into evaluation and results separately to explain the results better.\n\nQUESTIONS:\n1. In Line 187 \"we test the robustness of our approach by measuring the alignment between two runs (with different initializations)\" can the authors confirm what they mean by different initializations - are these different initializations of the clustering, or of the vision transformer training, or something else?\n\n2. It is unclear to me why the authors consider the SEQ tokens at all, and especially after pooling across layers, which would lead to signficant information loss in terms of granularity of concepts learnt. Can the authors explain why they examined SEQ tokens?\n\n3. In the comparison of vision models, please include more references beyond just analysis of attention patterns, CKA, and downstream performance. A lot of research has been done on comparing vision model representations as well as concepts across different lenses (e.g. training objectives, dataset sizes, downstream tasks, transfer learning, multi-modal learning).  Some of these lines of research (not exhaustive) are in these papers:\n\n* On the duality between contrastive and non-contrastive self-supervised learning, Garrido et al\n* Objectives Matter: Understanding the Impact of Self-Supervised Objectives on ViT Representations, Shekhar et al\n* A large-scale examination of inductive biases shaping high-level visual representation in brains and machines, Conwell et al\n* Objective drives the consistency of representational similarity across datasets, Ciernik et al\n\n4. How are the concept formation graphs (CFGs) made in Fig 3? As in how did the authors figure out the layer 7-9 concept development - was this through manual observation, or was there some quantitative indicator e.g.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-12"}