Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers
Authors: Johanna Vielhaben
Dilyara Bareeva
Jim Berend
Wojciech Samek
Nils Strodthoff
Venue: NeurIPS 2025 poster ()
Keywords: representational alignment
interpretability
concept-based explanations
Primary area: deep_learning
TL;DR: Representational alignment based on concept discovery across ViTs trained on different tasks.

Abstract: Measuring the alignment between representations lets us understand similarities between the feature spaces of different models, such as Vision Transformers trained under diverse paradigms. However, traditional measures for representational alignment yield only scalar values that obscure how these spaces agree in terms of learned features. To address this, we combine alignment analysis with concept discovery, allowing a fine-grained breakdown of alignment into individual concepts. This approach reveals both universal concepts across models and each representation’s internal concept structure. We introduce a new definition of concepts as non-linear manifolds, hypothesizing they better capture the geometry of the feature space. A sanity check demonstrates the advantage of this manifold-based definition over linear baselines for concept-based alignment. Finally, our alignment analysis of four different ViTs shows that increased supervision tends to reduce semantic organization in learned representations.

Beyond Scalars: Concept-Based Alignment Analysis in Vision Transformers
Johanna Vielhaben" Fraunhofer HHI
Dilyara Bareeva
Fraunhofer HHI
Jim Berend Fraunhofer HHI
Wojeiech Samekt Fraunhofer HHI Technical University of Berlin
Nils Strodthoff Carl von Ossietzky Universität Oldenburg
Abstract
Mcasuring the alignment between representations lets understand similarities between the feature spaces different models, such as Vision Transformers trained under diverse paradigms. However, traditional measures for representational align- ment yield only scalar values that obscure how these spaces agree in terms learned features. To address this, combine alignment analysis with concept discovery, allowing a fine- mne-grained breakdown of alignment into individual concepts. This approach reveals both universal concepts across models and each represen- tation's s internal concept structure. We introduce a new definition concepts as non-linear manifolds, hypothesizing they better capture the geometry of the feature space. A sanity check demonstrates the advantage of this manifold-base definition over linear baselines for concept-based alignment. Finally, our alignment analysis of four different ViTs shows that increased supervision tends to reduce semantic organization in learned representations.
Introduction
Vision Transformers (ViTs) [11] gaining increased popularity backbones for various computer vision tasks. There is a large zoo of pre-trained models covering different learning paradigms and supervision levels, with different capabilities [19] and thereby different internal representations. Consequently, understanding and comparing these representations essential for practitioners when selecting pre-trained models, assessing robustness and generalization properties, and designing fine- tuning protocols. This opens up questions like: Where does the model representation change the most and how? Which concepts (i.e. dominant structures in representation space) are encoded in lower layers vs. upper layers? Does the model encode semantically similar concepts in spatial proximity each other? How are the representations of model A aligned to that of model B across layers? One way to address these questions is by examining patterns in hidden activations and measuring representational aligumenr[35, 44], yet existing methods often provide only single scalar indicate Aimilarity441], leaving finer details unexamined.
In this paper, propose fine- ine-ggrined aligmment analysis based concepts that structure the latent representation. This gives insights into universal concepts between representations different layers models, well as how single representation is structured. To achieve concept-basd based alignment we need solutions for concept discovery, and measuring the alignment between concept
Now at Graphcore
I Corresponding author wojciech.sames Samechh.aamee hhi fraunhofer.de *Correponniig author nils strodthoff uol..ee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Concept Discovery
Concept-based Alignment
Latent Activation
Manifold Approx.
3. Density-based clustering (HDBSCAN)
Insights into inter- and intra-layer
model semantics
grid
stripes
layer n
pa
UMAP
CBA
suncept-wise di 1m e
Medel
d(P, Q
11c AAN
Model
Modee
mredet In
laver 1-9
Figure We combine concept discovery with alignment analysis for fine-grained insights into structures within and differences between latent activations. To this end, we investigate latent activations formed by intermediate layers, which according to the manifold hypothesis can be organized in terms of low-dimensional manifolds. We recover these concept manifolds using density-based clustering (HDBSCAN) applied to UMAP embeddings the latent representations. The discovered structures in latent space allow to characterize single layer and the formation of structures between layers.
proximity scores. Previous work concept discovery has ranged from identifying individual neurons or other discrete units [1] to defining linear directions in feature space [14), with the most general approach so far viewing concepts growing evidence that linearity may be too strong an assumption [2. 8J. In contrast, we posit that to capture the underlying geometry of representation more faithfully, concepts should be treated the most general structure they can form-namely nonlinear manifolds. For alignment measurement, previous methods typically characterize the similarity of similarities (e.g.. via CKA [28]), collapsing the results into single scalar score. We instead measure fine- tne-grained distances between concept pairs. Specifically, we represent activations as proximity scores to discovered concepts and employ generalized Rand index with pseudo-metric properties [24), which we partition into pairwise concept distances.
multi-dimmnsional linear subspaces [42]. However, there is
We combine concept discovery with alignment analysis reveal not only which concepts are universal between two representations but also how a single representation is internally structured (Sec. 2.2).
We propose a novel definition of concepts as nonlinear manifolds and demonstrate via a sanity check-the superiority of this definition for alignment analysis when compared to simpler and linear concept definitions (Sec. 4.2).
In our concept-based alignment analysis of different ViTs, we find that their representations exhibit markedly different structures; specifically, increased supervision correlates with reduced semantic structure in the learned representations (Sec. 4.3).
Concept Discovery for Representational Alignment
This section consists of two parts. First, we introduce our novel non-linear concept definition motivated by the manifold hypothesis and describe our approach to discovering these concepts in latent activations (Fig. 1). Second, we show how this concept-based representation of hidden activations can be used to measure alignment across different models layers, enabling fine-grained analysis that goes beyond single-scalar metrics.

---

2.1 Concept Discovery
Motivation According to the manifold hypothesis, which is widely accepted in machine learning, many datasets, including image data that nominally lie in high-dimensional space, can be described in terms of a few underlying latent factors and are thus concentrated on a (potentially disconnected) low-dimensional manifold embedded in high-dimensional space [20]. [32] shows how neural network trained on toy classification problem solves the task by transforming the topology of the input data, and layerwise reducing the Betti numbers of the class-wise components. We hypothesize that state-of-the-ar vision models behave similarly and try to recover the connected components in the hidden representations, which we call concepts.
Definition We analyze the hidden representation at an intermediate feature layer of a neural network. To this end, we divide the model f into two parts. / = gi o hy, where h1 is the mapping to a hidden feature layer I. Our definition then relies on hidden representations fy (x:) E N of input samples T; from set S. N' is the number of spatially separable elements in the representation, i.e., the number of tokens a transformer model or the number of superpixels in a convolutional feature map. We spatially decompose the feature maps h(r;)) into a set of N N' |S| feature vectors & E Rf. Previously. concepts have been mostly defined as linear structures [14, 46]. The most general linear structure would be affine subspaces, which would already represent extension compared to the recently considered definition as linear subspaces [42]. In this work, we generalize this idea even one step further and define concepts manifolds in the F-dimensional feature space.
Definition 1 a point cloud {Øî} } consisting of the feature vectors Φ5 that lie on the concept manifold with index o.
We define concept C manifold in d-dimensional featurr space, represented by
Benefits of concept manifold definition In the following, we want to compute concept proximity scores by which we measure alignment. Incorrect assumptions about the structure of the concept manifold, e.g., assuming it has no curvature (affine subspaces) or it is spherical and the distance to the manifold can be estimated by the distance to the centroid, directly lead to distorted concept proximity scores and hence to distorted alignment. Later, in sanity check our definition performs better than linear and non-linear spherical baselines for measuring representational alignment.
Clustering for concept discovery Having established our definition of concepts as manifolds in feature space, we now turn to the challenge of discovering these concepts through clustering. As stated above, we assume that feature vectors { from a hidden representation are sampled from a set of low-dimensional concept manifolds {C"). Recovering these concept manifolds in high-ddmensiooal space (F 768 in our experiments) is a challenging clustering problem. Therefore, we revert to density-based clustering on oow-dimesioonal embedding of the data [18, 22]. For this embedding. we utilize UMAP (Uniform Manifold Approx inmroxiaaiionooo and Projection) [29]. dimensionality reduction technique that preserves local and some global structure. Given that we have no a priori knowledge about the number of clusters, we employ HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise), which can handle clusters of varying densities [3]. HDBSCAN builds hierarchy of clusters based on density, represented by condensed tree, and allows for robust handling of noise, making it suitable for the possibly intricate structure of feature representation spaces. While UMAP does not fully preserve density, its ability to maintain the overall structure of the data makes it valuable preprocessing step before applying HDBSCAN. We use the HDBSCAN implementation from [30].
Concept proximity scores We leverage soft clustering with HDBSCAN based on the condensed tree which is roughly density function over the data points to compute fuzzy cluster membership as described in [30], which we formalize in the appendix for the reader's convenience. It is based on the distance to concept anchor points of each cluster and an outlier score, both derived from the condensed tree. We now have a fuzzy clustering P P(e) = (P(d))....
{{) with n clusters, where P [o, 1]"d holds the concept proximity scores of each concept C We interpret the concept proximity scores (Ø) as the probability that a feature vector o belongs to a concept Po in clustering P. This approach contrasts with previous concept assignment paradigms [14, 42], which often rely on hard clustering, where each feature vector is assigned to single concept, or linear methods that project onto specific concept directions, limiting the representation to more rigid framework. In contrast, our soft clustering method allows for nuanced membership scores that reflect the degree
3

---

of belonging to multiple concepts. In the following, we refer to our concept discovery method as NLMCD (non-linear multi-dimensional concept discovery).
2.2 Concept-based Representational Alignment
We now address the question of measuring representational alignment based on the concept proximity scores derived from fuzzy clustering.
Pseudo-metric between fuzzy clusterings The concepts are at this point characterized by prob- abilistic clustering (t)) (P()o).... ,()))) where p( (Ø:) P (6:)). We want to measure the similarity between two probabilistic clusterings P, Q from two different representa- tions to evaluate how aligned their concepts are. For this purpose, we leverage an extension of the pair-based Rand index generalized to fuzzy clusterings proposed in [24]. The original Rand index counts the number of concordant pairs (either two points are paired or not paired both clusterings) and disconcordant pairs (two points are paired in one clustering but not in the other). The distance between probabilistic clustering P, Q is based on generalized degree of concordance that is based on the distance between two membership vectors dms (P P(b,)):
2
dcross (Pr n(n 1)
Σ (P())), - dm (Q()))
(I)
A commonly used choice for the distance d is dms (P().) P)) = 1 - I P(e,)Il [9]. Finally, we refer to the similarity between two clusterings. derived from the uncovered concepts, as Concept-Based Alignment (CBA):
CBA
deross (P, 2)
We choose this measure because derose (P, Q) is pseudo-metric satisfying desirable properties that case interpretation Also, when P, Q are crisp partitions, CBA reduces to the original Rand index.
Distance between single clusters In contrast conventional measures for representational align- ment that yield a single scalar value, our approach provides a more nuanced measure of representa- tional alignment by assessing differences between pairs of single clusters. To measure the distance between two clusters po, Q from two clusterings P, Q, we decompose the distance in Eq. 1 into the contribution of single concepts pa, Q and measure the pairwise similarity between the membership scores of each feature
deross pa -=0 m(n-Ij Σ (a: - (o) 12 (Φ.) Q- (b1))
Due to the absolute value in Eq. 1, summing over all pairs 3 does not yield the total deross Q). but by the triangle inequality Ea,s deroes (Px, (P*,Q*) 2 deross (P, Q) the sum is an upper bound for the overall distance between two clusterings.
Related work
Alignment Representational alignment measures categorized, with particular emphasis Centered Kernel Aligmment (CKA) in [28]. CKA evaluates the similarity of similaritics, either linearly under non-linear kernel. Similarly, [10] measure alignment through the similarities of binary k-nearest neighbor adjacency matrices, which resembles CKA with a narrow Gaussian kernel. Our method relates to CKA in that it condenses these similarities into clusters and subsequently measures the similarity between these clusterings. In parallel to the initial release of this work, [26] introduced a related concept-based alignment measure which relies on class-wise one-dimensional linear concepts. In contrast, we analyze alignment across classes based on non-linear, multidimensionall concepts.
Concept discovery Most existing methods model concepts linear directions [17, 46, 15, 14, 23, 16). Generalizing this definition, [42] suggest that concepts can be represented more faithfully as multidimensional linear subspaces. While the above approaches operate unsupervised, without access to concept labels, supervised concept discovery methods have explored more flexible geometries:
*1) Identity: d(2, x) = O for all x. 2) Symmetry: d(æ, y) = d(y, x) for all x, 3) Triangle Inequality: d(x,)) & d(x, 3) + d(y, 2) for all x, 2.

---

[39] estimate concept manifolds as culliidimensional ellipsoids, [7] employ kernel classifiers for nonlinear concept discovery and report improvements over linear methods, and also [8] find evidence for the existence of non-linear features. Unlike these approaches, our main goal in concept discovery representation summarization for alignment measurement, rather than interpretability or feature enumeration. For this reason, we employ the most general, non-linear concept definition.
Comparison of Vision Models On the one hand, alignment measures such as CKA have been used to compare the representations of various vision models, including ViTs and ResNets trained on different tasks or datasets [6, 5]. This has been combined with the analysis of patterns in attention maps [44, 35, 33), visualization of feature maps [12, 38], and linear probes [38]. On the other hand, downstream performance is analyzed to guide the selection of pre-trained models for transfer learning [27, 19].
4 Results
We evaluate concept discovery in Sec. 4.1, check the superiority of our new concept definition over linear and simplified baselines for concept alignment analysis in Sec. 4.2, and perform concept-alignment analysis between four ViTs in Sec. 4.3.
4.1 Concept discovery
Experimental setup First, we outline the details of the concept discovery procedure as described in Sec. 2.1 based on UMAP embeddings and HDBSCAN clustering. For concept discovery and later analysis of representational alignment, we use random subset of 25 of the ImageNet train set, stratified samples across all 1000 classes. We study four different ViTs [11] with the same architecture (base, patch size 16, input size 224) but different training objectives and training datasets: FS [40] trained supervised on ImageNet- 1 [37] classification, CLIP [34] trained to contrast images and text from WeblmageTexxt [34], DINO [4] trained on ImageNet- Ik to enforce consistency between augmented views of the same image, and MAE [21] trained to reconstruct missing pixels of input data from ImageNet-Ik. We perform concept discovery separately for the sequence (SEQ) and the CLS token. We extract activations at the last MLP layer of each of the twelve transformer blocks. Due to computational constraints of UMAP and HDBSCAN limiting the number of tokens for clustering, reduce the 196 SEQ tokens per image to single representative token, which also facilitates comparison with CLS experiments. We obtain this token by average pooling over a central 4 x 4 block of tokens, assuming the image center contains more diverse concepts, while peripheral regions may predominantly capture repetitive background elements. Further, for SEQ tokens, we discard the last block for the considered models only the CLS token in the final layer enters the loss. Hyperparameter tuning for UMAP and HDBSCAN (based on DBCV) and the final settings are detailed in the appendix.
Evaluation We assess concept discovery using four metrics. First, report the rate of points classified as noise by HDBSCAN, Second, we compute the density-based cluster validity index (DBCV) [31], which compares intra- inter-cluster density and ranges from [-1, (higher is better). Third, we evaluate how well the UMAP- clustering preserves distances by calculating the root mean squared error between the original and embedded distance matrices, normalized by the mean pairwise distance (NRMSE). Finally, we test the robustness of our approach by measuring the alignment between runs with fixed model weights and input samples, but different initializations for UMAP and HDBSCAN using CBA from Eq. 2.
Turning to the results on embedding and clustering quality in Fig.2, we find that NRMSE remains consistently low across most layers and models. Exceptions include the CLIP CLS rep- resentation in layer one and the DINO CLS representation between layers six and eleven, where NRMSE rises. The DBCV scores range between 0.4 and 0.7, indicative of medium clustering quality, yet remain fairly consistent across both models and token types (SEQ vs. CLS). Given the inherent complexity of the clustering task, this level of performance is reasonable, supported by the qualitative
Results
'e do not weight the average DBCV by cluster size proposed in Moulavi et al. [31] so that noise rate and DBCV remain independent.
5

---

DM0
CLP
MAE
DNO
CUP
-
P5
MAC
deratyy baddddd thamte uaidity (aaco)
0.7
0.923
1001 00terate
embedeinng cualtty
2.6
60
0.75
0.050
0.023
0.1-
0.800
Figure 2: Quality of concept discovery: The noise rate is the ratio of points classified as noise in HDBSCAN, DBCV a density-based clustering validity index that contrasts intra- inter-cluster density with scores in [-1,1] 1J where higher is better. NRMSE measures the root mean squared error between the distance matrix of the original and embedded activations, normalized by the average distance in the original embedding, and shows how faithfully the UMAP embedding captures the geometry of the representation. Robustness is measured between two runs by concept- allgnmmet from Eq. 2. Results are across layers for CLS (dotted) and SEQ (solid) token representations of the models introduced in Sec. 4.1.
layer 7
layer
layer
"round edlote colects
"ounn green abject
"appler
Figure 3: Concept formation graph for the concept "apple(s)" in layer 9 of the FS model. Each concept is represented by six randomly sampled images containing a token assigned to that concept (highlighted in a yellow frame).
results in Fig.3 and Fig. The noise rate is relatively high overall but diminishes in deeper layers. Insufficiently dense sampling, constrained computationally by UMAP and HDBSCAN, may prevent some noisy regions from crystallizing into valid concept clusters. Robustness decreases for all models across layers but stagnates at around 0.84 for most models in the late layers. Finally, two random baselines based on a randomly initialized model and/or randomized input (detailed in the appendix) exhibit weaker clustering validity, confirming that concept discovery relies substantially on both the learned representations and the underlying input distribution.
Concept formation graphs Finally, for a qualitative evaluation of our concept discovery method, we construct concept formation graphs (CFGs). These are unweighted, directed graphs that show how tokens transition from one concept to another across consecutive layers. In Fig. 3, we illustrate how the "apples" concept develops from layer seven to layer nine the FS model. Additional examples for other models and the detailed algorithm for CFG construction are provided in the appendix.
4.2 Sanity checking concept structure for alignment and CKA comparison
Setup We use sanity check to demonstrate how concept-based alignment analysis benefits from representing concepts as non-linear manifolds, comparing these results to alternative concept def- initions and discovery methods. Additionally, we compare CBA from Eq. against CKA as an established representational alignment measure that offers a single scalar to indicate alignment. This sanity check relies on the assumption by [25] that adjacent layers should align more strongly than distant ones. For each layer, we compute the weighted Kendall's Tau correlation [43] between align- ment scores and layer distances, separately for upstream and downstream layers. We use hyperbolic weights to prioritize alignment between closer layers (whose alignment is more informative) and separate upstream from downstream layers accommodate different rates representational change (c.g., layer six may differ more from layer seven than from layer four). Averaging these correlations

---

Table 1: Sanity check for concept alignment, based on weighted Kendall Tau [43] between alignment and layer distance. We compare the suitability of NLMCD for Concept-Based Alignment (CBA) concepts against other methods: one-dimensional linear subspaces (PCA), nulti-dimensional linear subspaces (MCD), and spherical non-linear concepts (KMeans). Additionally, we compare CBA against CKA. Results within the same standard error interval as the top score for each model are bold and those CBA results within the same interval as NLMCD-CBA are italic. NLMCD consistently outperforms other concept approaches. While NLMCD-CBA and CKA are en-par, CBA offers the advantage of fine-gran fine-grained concept-based alignment.
SEQ
CLSS
FS
CLIP
DINO
MAE
FS
CLIP
DINO
MAE
PCA-CBA
0.91(2)
0.91(3)
0.88(4)
0.84(3)
0.92(2)
0.91(2)
0.78(5)
0.78(4)
MCD-CBA
0.90(4)
0.92(4)
0.85(4)
0.87(4)
0.82(4)
0.73(4)
0.62(5)
0.73(5)
KMeans-CBA
0,94(2)
0.82(4)
0.87(4)
0.98(1)
0.96(3)
0.73(1)
0.89(5)
0.86(4)
NLMCD-CBA
0.97(1)
0.98(1)
0.92(2)
0.98(1)
0.93(2)
0.96(1)
0.91(2)
0.94(2)
CKA
0,98(1)
0.94(2)
0.99())
0.99(1)
0.93(2)
0.97(1)
0.89(3)
0.93(3)
across layers verifies whether the alignment measure reflects the expected structural relationships. Using this sanity check, we compare CBA based on NLMCD concepts against one-dimesiional linear subspaces discovered by PCA [46, 14], multi-dimensional linear subspaces discovered by MCD [42], and spherical non-linear concepts discovered by KMeans clustering [14]. For the linear subspaces, we project feature vectors onto the concept subspace and clip negative values to zero to obtain soft membership scores (arguing that vector pointing in the opposite direction of concept inactive). For KMeans concepts, we measure concept proximity by the euclidean distance to the cluster centroid. We also normalize concept membership scores po = pa / Pa as their sum is required to be less bounded by one Σ pa < 1 in Eq. 1. There is no direct way estimate the number of concepts for PCA, MCD and KMeans, so we use all F 768 components for PCA as a conservative baseline, and the number of concepts discovered by NLMCD for MCD and K.Means discovery.
Results We present the samity check Tab. for SEQ and CLS token alignment. We find that for SEQ tokens, NLMCD shows higher scores than linear concepts except for DINO, where PCA can match its performance. Further, only for DINO and MAE simple nonlinear KMeans is en par with NLMCD. Similarly, for CLS tokens, NLMCD mostly outperforms other concept methods, while only for FS can PCA, and for CLIP and MAE can KMeans match its performance. To complement this quantitative evaluation of how well different concept definitions and discovery methods suit alignment measurement, we visualize PCA, MCD, and KMeans concepts and their geometric structure (based on our concept distance measure) in the appendix. By visual inspection, we find that the non-linear NLMCD and KMeans concepts are most coherent within each concept cluster and also show the highest coherence across groups of closely aligned concepts. When comparing to CKA. NLMCD- CBA achieves similar performance for both SEQ and CLS tokens. However, it offers the additional advantage of providing fine-grain -eaiirri concept-level view of alignment, unlike the single-scalar nature of CKA.
4.3 Concept Alignment Analysis
We now investigate concept-based alignment described in Sec. 2.2 between representations across layers and models. We structure the analysis into intra-model and inter-model to answer questions 1) 4) openend in the introduction. Due limited space, we focus on SEQ representation and defer the CLS representation analysis to the appendix. First, we analyze how representations are transformed within one model and how they are structured across layers.
Where does the model representation change the most and how? First, we focus on the intra- intra-model alignment heatmaps between SEQ representations across layers measured by CBA from Eq. the upper row of Fig. In Fig. 5, we complement these findings with additional concept clustering characteristics: alignment of concepts with ImageNet-1k labels, the intrinsic dimensionality of each concept, and the number of concepts per layer. In CLIP, DINO, and MAE, observe prominent break between layers one to six and layers six to cleven. The number of
Intra-model:
7

---

CLIP
DINO
MAE
Supervised
Level of Superviiio
Self Supervised
Intra-Model Aligmmmnt
lays
peoformation
loyer
BeVEsS
materiol
equspment
plant
buildingg
clothung
5003
furnttrre
loyer
Fellme
Figure 4: Intra-model relationships of SEQ representations. Upper row: We show CBA from Eq. to visualize how representations are transformed across layers of the models introduced in Sec. 4.1 (darker pixels correspond higher alignment). We observe nucleation process between layer and 10 in FS and smoother processing split into two major blocks between layer 1-6 and 6-1 6-11 in CLIP, DINO and FS. Center and bottom row: we zoom into the representations at layer 6 and 11 and partition the scalar CBA alignment into single concepts. We show UMAP embedding constructed from the pairwise distance concept measured by deross (Po, p3) from Eq. 3. Each point this concept atlas corresponds show four random input tokens from the members of the concept cluster pa marked by a yellow box in the entire image. The stronger the supervision during ViT training ranging from FS, over CLIP to DINO and MAE, the less semantically organized are the representations at layer 11.
distinct concept pa To convey their meaning, for some concepts,
class Iebel elgimeen
2000
concept Intrino dim.
concept count
800
600
200
12
12
30
12
byyer
Figure 5: To supplement the intra-model alignment analysis, we evaluate alignment between concepts and ImageNet-Ik Ik class labels, (based on CBA from Eq. 2), concept count, and the average intrinsic dimensionality (based on [13]) across concepts.
concepts increases smoothly in the first part but rises sharply from layer seven onward. At that same point, class label alignment shows marked increase, whereas the average concept dimensionality slightly decreases for DINO and MAE but increases for CLIP. In contrast, FS shows a distinct transformation between layers nine and ten, leading to sudden jump in both class alignment and the lower
number of concepts, accompanied by higher intrinsic dimensionality.. This shift results alignment between representations in the last layers and earlier layers and suggests the formation of distinct class-specific concepts, mirroring the "nucleation process" previously reported for ResNets by Doimo et al. [10].
Intra-model: Which concepts are encoded in lower layers vs. upper layers? How structured are the representations? We now zoom on the representational alignment scores by examining pairwise distances among individual concepts. Specifically, concept atlas two-dim.

---

CINO
MAL
1oart +-1 CLIF
Tear 1-1 CLIP
-ayen + DINO
High disance
NOE COnoaNO
DIND
MAME
DIN conce
Figure 6: Inter-model relationships of SEQ representations. Left: We show CBA from Eq. 2 to visualize how representations differ between models(darker pixels correspond to higher alignment). Right: We zoom into the concept-wise distances deross (Pa, PG) from Eq. 3 between the represen- tation of layer 10 in MAE and DINO. Alongside the pairwise distance matrix deross (Pa Q²), present randomly chosen examples of concepts from MAE and their nearest matches in DINO, and the distribution of the distances between nearest matches. We illustrate some random example pairs with low to high distance (marked by a star in the distance distribution plot). Overall, concepts that show higher pairwise distance also appear slightly more distinct visually.
UMAP embedding of the pairwise concept distance matrix - to visualize the structures concepts form layers six and eleven across of all models (Fig. 6). We color-code each concept cluster according to WordNet categories derived from ImageNet- kk labels to guide the eye. At layer six, all ViT models display a structured set of concepts which mostly focus on lower-level structures, shapes, and object parts, rather than semantic WordNer-based groupings. In comparison, at layer eleven, CLIP, DINO, and MAE concepts show strong semantic organization, illustrated by clearly separated canine concepts and coherent groupings of human body parts such as neck, shoulder, and legs. In contrast, FS at layer eleven appears less semantically orgamized fully supervised training may push similar concepts apart (for instance, different dog breeds) to reduce confusion and optimize for task task-specific accuracy. This, however, may have negative implications for generalization other tasks.
Inter-model relations Second, we analyze how the representations between two different models differ and present CBA from Eq. 2 between all layers of the models in the upper part of Fig. 6. We observe higher alignment between the self-supervised models DINO and MAE than with CLIP and the FS model in the aligmment heatmaps. Further, layers of the first are more aligned than those of the second half across all models pairs. We conclude that basic foundational features are learned similarly across models, while later layers diverge as the models specialize to concepts serving their pre-training task.
Infer-model:
How is the representation of model A similar to that of model B? We now select a pair of layers from two different models (layer ten in MAE and DINO) for closer look at concept- resolved differences in their representations. Alongside the pairwise distance matrix deross ( (Pa Q²9). we present randomly chosen examples of concepts from MAE and their nearest matches in DINO, as well as the distribution of the distances between nearest matches. Overall, concepts that show higher pairwise distance also appear slightly more distinct visually. Interestingly, around 62% of concepts from MAE all match to only three singular concepts in DIÑO, which are marked in gray in the distance distribution plot in Fig. 6 and excluded from example sampling. Upon inspection, those singularly matched concepts have no visual resemblance. Instead, the singular concepts in DINO exhibit low total activation across samples. Consequently, as our distance measures not only how concepts activate similarly, but also how they similarly do not activate, they are closest match to variety of MAE concepts that have close match otherwise by virtue of their shared lack of activation. Distances to these singular concepts span a range, depending on the activation strength of the concept from the first model. We observe similar behavior for other pairs of layers.
'Thsee are derived from the ImageNet- Ik labels the images from which the patches were extracted.
first map these labels to more abstract categories (provided in the appendix). Second, we perform a majority vote among all patches in cluster to assign the category. This labeling is only proxy and may not accurately reflect the actual content of the patches e.g. patch might show grass which animal stands.

---

5 Conclusion
We propose a novel approach that combines concept discovery with representational alignment analysis in ViTs. With concept-based alignment analysis, we answer the questions raised in the introduction and examine the structure of feature spaces of different ViTS, as well as fine details be- tween the concepts of two different models. These insights are not available through traditional scalar alignment measures. Understanding the structured nature of latent spaces can guide practitioners in choosing models that not only perform well on benchmark datasets but also exhibit robust feature representations for downstream tasks. For instance, the nucleation process FS emphasizes the importance of model structure over mere classification accuracy when selecting a pre-trained model.
Limitations The computational scalability of HDBSCAN limits the sampling of feature vectors which makes undersampled concept regions appear as noise. The limited variability of ImageNet- Ik might obfuscate the meaning of a concept, e.g. when a concept represents a color but there are only dog patches of that color. Further, the computational complexity of CBA requires subsampling of feature vectors.
This work was supported by the European Union's Horizon Europe research and innovation programme (EU Horizon Europe) as grants [ACHILLES (1011 (101189889), TEMA (101093003)]; and the German Research Foundation (DFG) through research unit DeSBi [KI-FOR 5363] (project ID: 459422098) and grant SELPHY-TS (project ID: 553038473).
Acknowledgments
References
[1] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
[2] Leonard Bereska and Stratis Gavves. Mechanistic interpretability for AI safety a re- view. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //opnreview
Survey Certification, Expert Certification.
[3] Ricardo G. B. Campello, Davoud Moulavi, and Jörg Sander. Density-based clustering based on hierarchical density estimates. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, 2013. URL https //api. peppi.semann www.stiiicoooolaaa......i org/ /ogg/orpusla ID: 32384885.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650-9660, October 2021.
[5] Laure Ciernik, Lorenz Linhardt. Marco Morik, Jonas Dippel, Simon Kornblith, and Lukas Mut- tenthaler. Objective drives the consistency of representational similarity across datasets. In Forty- second International Conference on Machine Learning, 2025. URL. httph: htttps::/oopnrreiee:
[6] Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature Commmicaatinns, 15(1):9383, Oct 2024. ISSN 2041-1723. doi: 1038/ 441467-024-53147-y URL https: http:////oooooo//////// 10. 1038 00088//41447-02-5514444
[7] Jonathan Crabbé and Mihaela van der Schaar. Concept activation regions: A gen- eralized framework for concept-based explanations. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Infor- mation Processing Systems, volume 35, pages 2590-2607. Curran Associates, Inc., 2022. URL httpps:/pproceeddnggs www.wnww...cceeeeeoee httpr//se..cppeee...e//////////p /illes//paper//22///ieel 1arra2arf556633c3006666------
Conference. pdf..
[8] Róbert Csordás, Christopher Potts, Christopher D Manning, and Atticus Geiger. Recurrent neu- ral networks learn store and generate sequences using non-linear representations. The 7th BlackboxNLP Workshop, 2024. URL https::/openreviieeee
10

---

[9] Ryan We and Jeffery L. Andrews. Random models for fuzzy clustering similarity measures. ArXiv, abs/2312.110270 2023. URL https://api semanticscholar. oom.ammniiciool......ooooooaaaap 266348826.
[10] Diego Doimo, Aldo Glielmo, Alessio Ansuini, and Alessandro Laio. Hierarchical nucleation deep neural networks. In Pruceedings of the 34th International Conference on Neural Information Processing Systems, NIPS 20, pages 7526-7536, Red Hook, NY, USA, December 2020. Curran Associates Inc. ISBN 9781713829546.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learinng Representations, 2021. URL https://openrevie:: http:////pppaaaeeeee..eooorrrrra -ibbddnNT
[12] Amil Dravid, Yossi Gandelsman, Alexei A. Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in model zoo. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1934-1943, October 2023.
[13] Elena Facco, Maria d' Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports, 7(1):12140, Sep 2017. ISSN 2045-2322. doi: 10.1008//41198 19980-11198-111111777311 URL https: /tttt:::/doi.orr/1... org/10. 1038/
[14] Thomas Fel, Victor Boutin, Louis Béthune, Remi Cadene, Mazda Moayeri, Léo Andéol, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying au- tomatic concept extraction and concept importance estimation. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- Curran Associates. ral Information Processing Systems, volume 36, pages wetelfflllsapppeeeeeee /pppappee/0//////l/sss Inc., 2023. URL http::ttpt:///eet https:t////rooeeeeinge Swwpeef--apppp--offeeeeeeee Conference pdf.
[15] Thomas Fel, Agustin Picard, Louis Béthune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for ex- plainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 271 1-11-121112 1-2721, June 2023.
[16] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https://arxis: org/ abs abs/2406, 04093.
[17] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-BBus Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:: / http://prooeeeiii...eeeee -pdf. 2011/ffleee7T222t133366636333313
seaper_ffles/pee
[18] Alvina Goh and Rene Vidal. Clustering and dimensionality reduction on Riemannian manifolds. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-7, June 2008.
doi: 10.1../008. 4587422. ISSN: 1063-6919.
[19] Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, Ramalingam Chellappa, Andrew Gordon Wilson, and Tom Goldstein. Battle of the backbones: large-scale comparison of pretrained models across computer vision tasks. ArXiv, abs/2310.19990 2023. URL 2644888842.
www.nticscoolartrr...eeaacac fttp:::/app..sennntccchooooo
[20] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. org.
http://www.depleamiinghhh
[21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000-16009, June 2022.
11

---

[22] Moritz Herrmann, Daniyal Kazempour, Fabian Scheipl, and Peer Kröger. Enhancing cluster analysis via topological manifold learning. Data Mining and Knowledge Discovery, September 2023. ISSN 1573-756X. doi: 10. (0077//106180003--098880000 URL. https://doi. org/10. 1007/s10618 023-00980-2.
[23] Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL cecttps:/oopervveeee wttpw/vwww.eeetttuneeeeee net/forum? id=F76bWRSLKK.
[24] Eyke Hullermeier, Maria Rifqi. Sascha Henzgen, and Robin Senge. Comparing fuzzy partitions: A generalization of the rand index and related measures. IEEE Transactions on Fuzzy Systems, 20(3):546-553) 2012. doi: 10.1 109/TFUZZ.201 1.211993.....
[25] Max Klabunde, Tassilo Wald, Tobias Schumacher, Klaus Maier-Hein, Markus Strohmaier, and Florian Lemmerich. Resi: comprehensive benchmark for representational similarity measures. In The Thirleenth International Conference on Learning Representations, 2025. URL https://opearrever: wttprreeiiiiieeeffffoooeoe wwt. Phrdrmmnaidd
[26] Neehar Kondapaneni, Oisin Mac Aodha, and Pietro Perona. Representational similarity via interpretable visual concepts. In The Thirteenth International Conference on Learning Repre- sentations, 2025. URL httttt::pppppeerrieeeeeee wttperewwwwweeeeeeeeeeeeeeeeee
[27] Simon Kornblith, Jonathon Shlens, and Quoc Lee Do better imagenet models transfer better? 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2656-2666, 2018. URL https: /appi.app..a/// sem...issmannnchhollrrr org org/CorpusID:4000 449288547
[28] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity neural network representations revisited. In International conference on machine learning, pages 3519-3529. PMLR, 2019.
[29] Leland McInnes and John Healy. Umap: Uniform manifold approximation and projection for dimension reduction. ArXiv, abs/1 ab2.180,, 2018. URL htttps://app httpt::://pp.ssmmaiiiseeeee semanticscholii org/Corpus ID: 3641284.
[30] Leland McInnes, John Healy, and Steve Astels. hdbscan: Hierarchical density based clustering. The Journal of Open Source Software, 2(11), mar 2017. doi: 10.21 1051100s00000000 URL https://doi... org/10. 211055/2 000000
[31] Davoud Moulavi, Pablo A. Jaskowiak, Ricardo J. G. B. Campello, Arthur Zimek, and Jörg Sander. Density-based clustering validation. In SDM, 2014. URL https://api. semanticacholar. owg. : 16656312.
[32] Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of deep neural networks. The Journal of Machine Learning Research, ((11):188:7503-1844:78 January 2020. ISSN 1532-4435.
[33] Namuk Park, Wonjac Kim, Byeongho Heo, Tackyung Kim, and Sangdoo Yun. What do self- supervised vision transformers learn? ArXiv, abs/2305.007299 2023. URL https://api. semanticscholar org/CorpusID: 258426737.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-88 8768. PMLR, 18-24 Jul 2021. URL https://proceeddngg mlr, waar.press/vlle
hatess//v19///// h119///ddf999111a.eee html.
[35] Maithra Raghu, Thomas Unterthiner, Simon Korbblihh, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In Neural Information Process- ing Systems, 2021. URL https: : //api. semanticscholar .rrgg /CorpusID 237213700.
12

---

[36] Sebastian Raschka, Joshua Patterson, and Corey Nolet. Machine learning in python: Main developments and technology trends in data science, machine learning. and artificial intelligence. arXiv preprini wrXi::2002008833 2020.
[37] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, (15(3):211-252 2015. doi: (2000///1122-----08111666 URL httth:t///// /doi. axg/10.1007 s11263- 081--8
[38] Shashank Shekhar, Florian Bordes, Pascal Vincent, and Ari Morcos. Objectives matter: Under- standing the impact of self-supervised objectives on vision transformer representations. ArXiv, 00//2304.13088 2023. URL https: tattts://arxtte http///axvvvv.///// orgg
13089.
[39] Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences, 119 (43):e2200800119, October 2022. doi: N0.1073/pnas222080011 2200000199 URL https: /www. www.wo........... pnas.
2200800119.
[40] Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreever. htt.:/rw.nnt/tooom///ddddaaaaai
[41] Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng. Andreea Bobu, Been Kim, Bradley C. Love, Erin Grant, Jascha Achterberg, Joshua B. Tenenbaum, Katherine M. Collins, Katherine L. Hermann, Kerem Oktar, Klaus Greff, Martin N. Hebart, Nori Jacoby, Qiuyi Zhang, Raja Marjich, Robert Geirhos, Sherol Chen, Simon Kornblith, Sunayana Rane, Talia Konkle, Thomas P. O C'Conelll, Thomas Unterthiner, Andrew Kyle Lampinen, Klaus-Robed Muller, Mariya Toneva, and Thomas L. Griffiths. Getting aligned on representational alignment. ArXiv, 0bs/2310.1300 2023. URL https:///arai
/rg/2310 13018.
[42] Johanna Vielhaben, Stefan Bluecher, and Nils Strodthoff. Multi-dimensional concept discovery (MCD): unifying framework with completeness guarantees. Trunsactions on Machine Learning Research, January 2023. ISSN 2835-8856. URL https::/ /ttpr//veee/ereeeee.e/em net.
[43] Sebastiano Vigna. A weighted correlation index for rankings with ties. In Proceedings of the 24th International Conference World Wide Web, WWW *15, page 1166-1176, Republic and Canton of Geneva, CHE, 2015. International World Wide Web Conferences Steering Committee. ISBN 9781450334693. doi: 10.1145/2736277.271111 2736277 2741088.
URL https://dot org/10. 1 1145/
[44] Matthew Walmer, Saksham Suri, Kamal Gupta, and Abhinav Shrivastava. Teaching matters: Investigating the role of supervision in vision transformers. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7486-7496, 2022. URL https:
//api. seagpsssmanticshhllar org/ :ttg/oorpssI::::55500 2544366577.
[45] Ross Wightman. Pytorch image models.
aptorch-imagee models, 2019.
[46] Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A. Ehinger, and Benjamin I. Rubinstein. Invertible concept-based explanations for enn models with non-negative concept activation vectors. Proceedings of the AAAI Conference on Artificial Intelligence, 35(13): 1682-1 16900, May 2021. doi: 10. (609///ai.v33113333 17389. URL https::///// //ojs/////j./....////// waj..aai..a..................... org/ index.dexx./// index.ppp/ AAA/arrtice/viie 17389.
13

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We summarize the contribution of our work in the introduction and in the
abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include section limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
We do not present any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe our methods, the dataset, models, metrics, and parameters we used.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
We provide our code and use models, datasets and methods that are publicly available.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
wwwsssssiiiiiiosso olicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/
www.abimsssioooooooooooo onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specifty all models, dataset subset, methods and hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the standard error across layers for our sanity check in Tab. 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details about the hardware we used and run-time for our experi- ments in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https
Answer: [Yes]
Justification: Our work fully confirms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We briefly describe the positive impact through enhanced model understanding in the conclusion we do not see any negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release new datasets or trained models, so there is no risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all models, datasets, and methods that
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: We provided our code and use reference publicly available models, datasets and methods.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
We did not conduct studies with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We did not conduct studies with human participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We did not use LLMs in the core method development of this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: http:://eeuuuueeeeeee http://rrrps.cc//ooeeeeeeeeeeeee / for what should or should not be described.
20

---

A HDBSCAN
After concept discovery with HDBSCAN, compute concept proximity scores P(#) (P(6o).... P( ) P [0, These rely on the implementation soft clustering with HDBSCAN from [30], which we formalize here for the reader's convenience.
1|ad holds the concept membership scores Pa (Ø) of each concept CO.
Clustering HDBSCAN first transforms the feature space using a density-informed metric called mutual reachability distance
0;) max max(coreDiistanee:())) coreDistance,(6)) dl d();,
(4)
where (ore is the distance between a point & and its k-nearest neighbor. Based on the mutual reachability distance between all pairs, minimum spanning tree is constructed that connects all points and minimizes the sum of the edges weighted by MRD. From this, a hierarchical tree is constructed via robust single linkage clustering. The hierarchical tree is condensed by eliminating insignificant clusters and simplifying the hierarchy. This is achieved by selecting a range of persistence values X, which are the inverses of the mutual reachability distances (A -1/ 1/MRD). Clusters that persist over significant ranges of X, i.e. they are stable across multiple density levels, are retained, while clusters that exist only over narrow ranges of À are considered noise and pruned from the tree. The result is condensed tree that focuses on the most significant clusters. Finally clusters are extracted from the condensed tree either based on their stability across different density levels or simply the leaf nodes are identified as clusters.
Soft clustering with HDBSCAN membership with and an outlier score.
The soft cluster membership scores combine distance- based
For the distance- based membership cluster C first & exemplar points {ØT}, i [1, are extracted. A single centroid is not enough to characterize a cluster as its shape can be arbitrary. The exemplar points the points within the leaf nodes beneath cluster C with maximum persistence À in the condensed tree, i.e. the densest points where the cluster persists. Then, the distance membership score between a point & and cluster C the inverse minimum distance across the exemplar points }
Md (o) dist Imin:(d(e) Φ +;))
(5)
E> [min (d( )
normalized across all clusters. The outlier-based membership compares a point's membership persistence to the total persistence of cluster:
A0= binh hirth
M membership
10 max
Here, A the persistence value at which cluster Co first appears, i.e. birth point in the condensed tree and Ag is the persistence value at which point & would join cluster Co Fi- nally, distance and basde membership are combined with stronger emphasis on buteeer-based membership,
M (Φ) = (M" ad) 1/2 (M" membersthip
and normalized M (Φ) preted as the probability that point belongs cluster C given that the point belongs some cluster,
(0) E. ( This membership score Ma M (Ø) can be inter-
M norm
P( C 133 & C²))
(8)
We want to compute the joint probability P(Φ E C*), which includes the probability that & may be noise,
P( E C²) = P C 33 )3) & C².).
Here, P(33 $ C*) the probability that belongs to some cluster. To estimate P(3 C*)). the À value at which & would join the nearest cluster is compared to the maximum À value of that cluster.
A A0
P(3B: : E CO)
(10)
C
max
21

---

where Ap-C" is the persistence value at which point & would join its nearest cluster C and X is the maximum value of cluster C Thus, the final probability, that point & belongs cluster C is.
A0-0a
pa (Ø)
M norm (Ø)
(11)
A
B
Details on experimental setup
Here, we provide further details on the experiments.
VIT sources
We list the URL of each Vision Transformer provided by the timm library [45]:
FS: http://swggggggg.........c../cas
CLIP: hhtp://///gggggga.......ormaaaaa
o openai
DINO: http:///s./hggggoooo.ooooo/////a
MAE: http://hwgggggaaac...ootmmm///ia
Hyperparameters for UMAP and HDBSCAN We tune hyperparameters of UMAP and HDB- SCAN such that the density-basee validity index DBCV is maximized across models and layers. Here, for DBCV, the average across clusters is weighted by their respective size such that the noise rate is indirectly included. We list the effect of the most influential hyperparameters that we tune and state the final value we used:
Minimal distance in UMAP: low minimal distance in UMAP enhances local cluster density but may also increase noise. We use a value of 0.01 in all experiments.
Number of neighbours in UMAP: the number of neighbors controls the local structure, the smaller the finer it captures local neighborhoods but distorts global structure which is important for concept alignment analysis later. We use a value of 30 in all experiments.
Embedding dimensionality in UMAP: We use the practical limit for HDBSCAN of
F" = 50 in all experiments.
Minimum cluster size in HDBSCAN: a too small minimum cluster size may identify noise as cluster, whereas, when too large, distinct clusters will merge. We use a value of 50 in all experiments.
Min samples in HDBSCAN: controls how conservative the algorithm is about noise. We need this to be rather low because of sampling limitations which means that most likely some concept manifolds are not sampled densely enough. We use a value of 20 in all experiments.
Additionally, we assume that clusters are rather uniform in size and select the leaf nodes in the HDBSCAN hierarchical condensed tree as clusters. Sampling one pooled SEQ token or one CLS token from each representation of images within a 25% subset of the ImageNet1 1k train set results in 315.770 feature vectors d; for clustering. We use the cuMI. [36] versions of HDBSCAN and UMAP for computation on the GPU.
In Tab. 2 and Tab. 3, we show a hyperparameter sensitivity study (at layer 11 of DINO SEQ repre- sentations) that varies UMAP and HDBSCAN parameters one time (the other hyperparameters are fixed at the default values stated above) and observe that concept discovery evaluation metrics are reasonably stable within quite broad hyperparameter range. Only NRMSE shows a rather high range when varying the embedding dimension and UMAP minimal distance with minimum of 1.0 for small embedding dimensions (< 40) and very small minimal distance. This is likely not sign of a superior embedding, but rather an counter intuitive behavior of this metric due to extreme compression of pairwise distances the embedding.
Cluster label in Concept Atlas To assign a label from the WordNet Hierarchy to each concept cluster, we first assign the ImageNet- label of the image from which a token is extracted to its representation feature vector Φ;. Then we map this to a label higher in the WordNet hierarchy by the mapping in Tab. We then assign the most frequent label among the cluster members {Φ6) to the cluster Co
22

---

Hyperparameer
Range
NRMSB
Noise miñio
DBCV
Robustmess
Eimeekknng dim. F
20
100 9.5 15.1
0.68 0.68 < 0.66
0.56 0.58 0.59
0.84 0.84 0.8T
UMAP
0.72
0.53
0.81
UMAP neighhors
10 4u
6.4.9.5
0.05 < d.6B < 0.06
0.58 < 0.58 < 0.61
0.84 < d.8G 0.85
Table 2: Impact of UMAP hyprrarrameters on embedding and clustering quality at layer 11 DINO
SEQ representations. UMAP parameters are varied one at time while all other hyperparameters are fixed at the default values stated above. Results are reported as min-defaalt<<max
peperparameerr
Range
Noise ratio
DBCV
Robusiness
100
0.66
0.57 0.58 0.59
Min, sumples
50
0.68 0.68 < 0.67 0.50 0.58 < 0.65 0.83 < 0.84 < 0.85
Table 3: Impact of HDBSCAN hyperparameters on clustering quality at layer 11 of DINO SEQ representations. HDBSCAN parameters are varied one at a time while all other hyperparameters are fixed at the default values stated above. Results are reported as min=defaull<mxx.
Computation of alignment Our concept- alignment measure CBA is based on pairs of feature vectors (o,, Oz). To reduce run-time, we sub-sample 20% of the 315.770 feature vectors before computing CBA. Some concepts become under-sampled in this process, indicated by their total activation in the subsample being much lower than 20% of their activation over all samples. Such under-sampled concepts appear close to nearly all other concepts due to the missing activations. To address this, we exclude them by filtering out concepts whose total activation in the subsample is below 17% of their total activation across the entire dataset (this filters out around 4% of concepts in the examples shown here and in the main paper).
Runtime Collecting activations for 25% subset of the ImageNet1- k train set, concept discovery and concept discovery evaluation (NRMSE, DBCV) takes around 1.5 hours per layer. We performed around 30 runs for UMAP and HDBSCAN hyperparameter tuning. Computing scalar concept alignment for all pairs of layers depends on the total number of concepts, requiring about 1.25 hours for the FS model, which has the largest concept count. Concept-wing distance computations also scale with the number of concepts: for example, the fine-grained inter-model analysis shown in Fig. 6 in the main paper took about hours on MAE vs. DINO at layer 10 with 477 and 684 concepts, respectively. All experiments were conducted Tesla V100 GPU.
C Baselines for concept discovery evaluation
Here, we describe two baselines for our evaluation of NLMCD concept discovery quality in Sec. 4.1 of the main paper. We present them alongside the concept discovery results for FS, CLIP, DINO, and MAE in Fig.7.
Random/ImageNet Baseline We feed ImageNet samples (the same subset as in Sec. 4.1) to randomly initialized ViT. Although the input images themselves contain discriminative patterns, the model itself has no learned structure. Here, noise rate remains almost constant across layers, comparable to middle or later layers of trained models. UMAP preserves distances well in this setting, resulting in good NRMSE. DBCV scores (measuring cluster seperation by density) are overall lower than in trained models but not as low as one might expect. Visually, clusters seem to capture pixel color similarities rather than semantic concepts (see examples in Fig.7). However, robustness of concept discovery is much lower: different initializations of UMAP and HDBSCAN lead to different color-driven clusters. In short, while input structure alone can induce clustering, it mainly reflects superficial color patterns rather than meaningful learned semantics with slightly reduced cluster separation and lower robustness.
Random/Noise Baseline Here, a randomly initialized ViT receives Gaussian noise patches. With no structure in the data or the model weights, HDBSCAN labels most points as noise, leading to higher noise rate than concept discovery with ImageNet inputs or trained models. Similarly, DBCV values drop noticeably, whereas NRMSE remains only slightly higher, indicating that UMAP still preserves pairwise distances. Interestingly, robustness is relatively high this scenario, simply because most tokens consistently categorized noise. This extreme setting underscores that
23

---

0660
CUT
192
HIMSE 12
0.3
0.80
loyer
layer
0.95
0.8
0.5
0.50
0.5
0.80
0.4
0.80
lager 11
0.75
Figure 7: Two baselines for concept discovery evaluation based on a randomly initialized model with ImageNet or gaussian noise input. Left: The noise rate is the ratio of points classified as noise in HDBSCAN. DBCV is a density-based clustering validity index that contrasts intra- vs inter-cluster density with scores in [-1, where higher better. NRMSE measures the root mean squared error between the distance matrix of the original and embedded activations, normalized by the average distance in the original embedding, and shows how faithfully the UMAP embedding captures the geometry of the representation. Robustness is measured between two runs by concept aligmment from Eq. 2. Results are across layers for SEQ (upper row) and CLS (bottom row) token representations. Right: Randomly chosen examples for concepts discovered at layer 1, 6, 11 in the SEQ representations of the randomly initialized model, all corresponding to color patterns.
both structured data and learned parameters are essential for creating meaningful clusters concept discovery.
Summary Overall, the Random/ImageNet baseline shows that the input distribution alone can drive clustering based on superficial pixel color regularities in representations, but naturally fails to produce semantically rich clusters driven by the model's learned concepts. The Random/Noise baseline confirms that without both learned parameters and structured data, clustering breaks down almost entirely, highlighting the fundamental role of learned representations for robust and meaningful concept discovery.
D Concept Formation Graphs
Notation. Let Cl.a, with E .,....,. NÝ, denote a concept cluster in layer 1. For token i 1, 1,..., k. we use the notation o Cl. indicate that the token representation øf in layer is assigned that cluster. Let Ct denote the target concept for which a concept formation graph (CFG) constructed. Given & tokens sampled from the training dataset and their cluster assignments, the algorithm for constructing the CFG is defined follows:
Transition matrix calculation: First, we compute transition matrices Tr,41 E N* x for each pair of consecutive layers (1, + 1). Each entry represents the count of tokens transitioning from a concept Cdl in layer I to concept 1,, in layer + 1:
(Ti,+1)as # {1. (I....,k}
8 Chi and o!+1 E C++1,9)
(12)
where # denotes the count of tokens.
2. Recursive graph construction:
Initializing the set of CFG nodes with the target concept node C4**, we recursively add all predecessor concept nodes whose "contribution" (the proportion of incoming transitions) surpasses a specified threshold r. Formally, suppose concept C++, in layer / + has been added to the CFG. Then, for cach concept Cl. in
24

---

layer I, we include the edge (Ch,a, () and node C² to the CFG if:
(Tu+4)as Ey (T1,41)78
T.
(13)
The resulting CFG is a binary, unidirectional graph. Fig. 9, 10, and 8 illustrate additional exemplary CFGs for CLIP and DINO. The CFGs were constructed using the same k = 315,770 tokens from the ImageNet training dataset that were used for concept discovery, with the threshold parameter set to 0.05. In each image, the "concept" token is highlighted in yellow. The concepts in Fig. 3 of the main text are human-labeled.
layer 6
layer
layer 8
layer 9
XXX
Figure 8: Concept formation graph for a concept in layer of DINO. Each concept is represented by six randomly sampled images containing a token assigned to that concept (highlighted in a yellow frame).
layer 9
layer 10
layer 11
layer 12
Figure 9: Concept formation graph for a concept in layer 12 of CLIP. Each concept is represented by six randomly sampled images containing token assigned that concept (highlighted in a yellow frame).
E Qualitative comparison against linear and spherical concept discovery
Figure 11 illustrates the concept atlases for four concept discovery methods (PCA, MCD, KMeans, and NLMCD) the concept alignment sanity check of Sec. 4.2 the main paper. For a direct qualitative comparison, we also show concept atlases layer six for CLIP SEQ token concepts using
25

---

layer
layer
layer 9
layer 10
Figure 10: Concept formation graph for a concept in layer 10 of CLIP. Each concept represented by six randomly sampled images containing token assigned to that concept (highlighted in yellow frame).
each method (Fig. 12-15). We segment each atlas into 15 groups via KMeans (indicated by color) and display random examples of concepts within each group.
Visually, MCD, KMeans, and NLMCD exhibit more coherent concept clusters than PCA. Moreover, when examining groups of close concepts in each atlas, we find that NLMCD and KMeans yield more structured organization and greater semantic consistency within concept groups than MCD or PCA (e.g., note how the bright yellow cluster in the MCD atlas mixes sundown/horizon, fur, and shiny object concepts). While the difference in semantic consistency is subtle between KMeans and NLMCD in this example, the results in Sec. 4.2 show that KMeans is par with NLMCD in half of the test settings but never outperforms it. Nevertheless, KMeans can still serve as a slightly less accurate but more computationally cheap alternative to NLMCD.
PCA: One-dim, lincar
MCD: Multi-dim.lincar
KMeaas: Non-lincar spherical
NLMCD: linear
Cy
Cl
C2
c
G2
Figure 11: Illustration of the four different concept definitions that are compared for measuring concept alignment in Sec. 4.2 in the main paper (in three dimensions).
F
Concept alignment analysis
F1
CLS representations
We investigate concept-based alignment within and across models based on the CLS token represen- tations analogous to the SEQ token analysis in the main paper.
Intra-model alignment We first compare the intra-model alignment heatmaps for CLS repre- sentations across layers (top row of Fig. 16), measured by CBA, with the same analysis for SEQ tokens presented in the main paper. For the FS model, the CLS representations exhibit pattern very similar to that of the SEQ representations. Likewise, CLIP and MAE show largely comparable alignment profiles, but their earliest layers display noticeably lower alignment for CLS,
26

---

Figure 12: Detailed concept atlas for PCA SEQ token concepts at layer six of CLIP. This is based on UMAP embedding constructed from the pairwise distance of concepts measured by deross (Pro pd). Each point in this concept atlas corresponds to a distinct concept pa, To convey their meaning, we show four random input tokens from the members the concept cluster pa We dissect the concept atlas into 15 groups (indicated by color) and show four random concepts for each group.
suggesting that the model might rely less on the CLS token in early layers. With respect to class label alignment, intrinsic dimensionality, and the number of discovered concepts, the CLS patterns in Fig. 17 also broadly resemble the SEQ results, except for DINO. In DINO's CLS tokens, alignment across layers is substantially lower than in the SEQ tokens, and the concept atlas layer 11 appears less semantically organized. One plausible explanation is that the UMAP embeddings for DINO's CLS tokens (layers 7-11) exhibit higher NRMSE, indicating reduced faithfulness of the embedding to the original activations, and thereby less faithful concepts.
Inter-model alignment Second, we analyze how the CLS representations between two different models differ and present CBA alignment between all layers and model in the upper part of 18. Like for the SEQ representations, CL.S representations layers of the first are aligned than those of the second half across all models pairs, suggesting that basic foundational features are learned similarly across models, while later layers diverge as the models specialize concepts serving their pre-training task. However, the overall alignment between models is weaker for CLS representations than for SEQ, also in low layers. Next, zoom in into the distance deross (Pa _P8 between concept pairs from CLS representations of MAE layer 11 and CLIP layer 11 in the right part of 18. Next to the pairwise distance matrix deross (Pa, Q²), we present the distribution of the distances between nearest matches, and randomly chosen examples of concepts from MAE and their nearest matches CLIP. Similar to the comparison of SEQ concepts between MAE and DINO in Sec. 4.3 in main paper, we find that 80% of CLS concepts in MAE are similar only 5 singular concepts in CLIP. Lastly, among the randomly chosen examples, visual discrepancy more pronounced for high-distance concepts than for the other pairs with lower distance.
27

---

Figure 13: Detailed concept atlas for MCD SEQ token concepts at layer six of CLIP. Detailed concept atlas for PCA SEQ token concepts at layer six of CLIP. This is based on a UMAP embedding constructed from the pairwise distance of concepts measured by Meross(Po, po). Each point this concept atlas corresponds input tokens from the members of the concept cluster p We dissect the concept atlas into 15 groups (indicated by color) and show four random concepts for each group.
distinct concept pa, To convey their meaning, we show four random
F.2 Additional results for SEQ representations
Intra-model
To give more detailed view of the organization of concepts across the layers of one model, we select the DINO model and show the respective concept atlases at layer one, six and eleven in 19, 20, and 21, respectively. To give an overview of the structure within a concept atlas, we group the concepts in the UMAP embedding via KMcans and show four random concepts for each group. In layer one, many concepts correspond to color, in layer six, they represent mostly textures, and in layer eleven they correspond to abstract concepts. Moslty, concepts within group are of similar nature.
Inter-model In the main paper, we show fine- fine-grained inter-model concept distances between DINO and MAE at layer ten. Here, we add fine -neiinai concept distance analysis between DINO and MAE, as well as between MAE and CLIP in 22. Again, we show the full pairwise distance matrix well as how distances between closest matching pairs are distributed. Similar to the comparison of SEQ concepts between MAE and DINO in Sec. 4.3 in the main paper, we find that in the case of DINO vs. CLIP, 65% % of CLIP concepts most closely match with only 2 singular concepts in DINO; for MAE vs. CLIP, 86% of MAE concepts match with 4 concepts in CLIP.
Now, we examine differences in concept alignment between pairs of layers. Specifically, we compare layer 10 in MAE/DINO (see Fig. 6) and the same layer in CLIP/DINO (see Fig. 16), which exhibit the same overall concept alignment (0.74). To summarize fine- ame-ggained differences in concept-alignment we group concept nearest-neighhor pairs into the WordNet categories from Fig. 4 (88% of concept pairs are in the same category for MAE/DINO, 93% for CLIP/DINO) We then compute the mean concept distance within each concept category and find that 1) canine and insect concepts have high distance both model pairs, while equipment and material concept have low distance in both pairs; 2) fish concepts are similar in MAE/DINO but have high distance in CLIP/DINO, while sport concepts
28

---

Figure 14: Detailed concept atlas for KMeans SEQ token concepts at layer six of CLIP. Detailed concept atlas for PCA SEQ token concepts at layer six of CLIP. This based on a UMAP embedding constructed from the pairwise distance of concepts measured by deross P$). Each point in this concept atlas corresponds to a distinct concept Pa, To convey their meaning. we show four random input tokens from the members of the concept cluster pa We dissect the concept atlas into 15 groups (indicated by color) and show four random concepts for each group.
show the opposite trend. This exemplifies scenario where two representations are similarly aligned by the overall score, but fine- grained concept alignment differs.
29

---

Figure 15: Detailed concept atlas for NLMCD SEQ token concepts at layer six of CLIP. Detailed concept atlas for PCA SEQ token concepts layer six of CLIP. This is based on UMAP embedding constructed from the pairwise distance of concepts measured by derose (aa, p&). Fach point in this concept atlas corresponds input tokens from the members of the concept cluster pa, We dissect the concept atlas into 15 groups (indicated by color) and show four random concepts for each group.
distinct concept pa To convey their meaning, we show four random
30

---

Superwised
Level Supervision
Sell-Supervized
Aligommntt
Layer
Layer
layer
sport
geea penformation
fruit
organian
loyas
pzant
building
coothing
layes
Figure 16: Intra-model relationships based on CLS representations across layers. In the upper row, show CBA to visualize how representations are transformed across layers of the models (darker pixels correspond to higher alignment). In the center and bottom row we zoom into the representations at layer and 11 of each model and partition the scalar CBA alignment into single concepts. We show UMAP embedding constructed from the pairwise distance of concept measured by deross (Po, ) Each point in this concept atlas corresponds distinct concept pa, To convey their meaning, we show four random input tokens from the members of the concept cluster C marked by yellow box the entire image.
15
-
DIV
MN
CUP
signoment
1200
concept intresic dm..
10
600
400
0.1
200
10
Figure 17: Class label alignment (based on CBA), concept count, and the average intrinsic dimension- ality (based on [13]) across concepts for CLS representations supplement the intra-model alignment analysis, by providing insights into how well the model aligns with ImageNet- 1k labels, the spatial organization of tokens, and the complexity of the learned concepts as they evolve through the layers.
31

---

dista
CLP
DIN
MAC
CLIP
CUP
DINO
High distence
MARE CONNOCTT
DND
MAE
Figure 18: Inter-model relationships based on CLS representations across layers. Right: We show CBA to visualize how representations differ between the models (darker pixels correspond to higher alignment). Left: We zoom into the concept-wiss wise distances derosa (Pn, Po) between the represen- tation of in MAE and CLIP at layer 11. Alongside the pairwise distance matrix derose (Po, Q*), we present randomly chosen examples of concepts from MAE and their nearest matches in CLIP, and the distribution of the distances between nearest matches. We illustrate some random example pairs with low to high distance (marked by star in the distance distribution plot).
Figure 19:
We zoom into the SEQ representations DINO layer and show UMAP embedding constructed from the pairwise distance of concepts measured by deross (Pa, p3). Each point in this concept atlas corresponds to distinct concept pa To convey their meaning, show four random input tokens from the members of the concept cluster pa We dissect the concept atlas into 7 groups and show four random concepts for each group. Concepts representing similar colors lie within the same group, e.g. shades of blue in the blue group or red and orange in the red group.
32

---

Figure 20: We zoom into the SEQ representations DINO layer and show a UMAP embedding constructed from the pairwise distance of concepts measured by deross p Each point this concept atlas corresponds to distinct concept pa, To convey their meaning, we show four random input tokens from the members of the concept cluster pa We dissect the concept atlas into groups and show four random concepts for each group. Most concepts represent a pattern texture which are similar within each group.
33

---

Figure 21: We zoom into the SEQ representations at DINO layer 11 and show UMAP embedding constructed from the pairwise distance of concepts measured by derass po). Each point in this concept allas corresponds to distinct concept pa_ To convey their meaning, we show four random input tokens from the members of the concept cluster pa We dissect the concept atlas into 30 groups and show four random concepts for each group. For most groups, these are semantically similar.
34

---

distance
Medium
DINO
DINO
CLIP
Medium High distance
High distance
DINO concepts
DINO
CLIP
DINO
DINO
ayer
concepts
Medium distance
5000
MAE
MAS
3000
CLIP
CLIP
Medium High distance
High distance
MAE
MAE concapts
MME
MAE
CLIP
Jayer
CLIP concepts
Figure 22: CBA of SEQ concepts across layers of CLIP and DINO, as well as CLIP and MAE (darker pixels correspond to higher alignment). We zoom in into the concept-wise distances deross (Pa p3) between the representation of layer ten CLIP and DINO, as well as layer in CLIP and MAE. Alongside the pairwise distance matrix derose [Pa Q²). present randomly chosen examples of concepts from MAE and their nearest matches in DINO/CLIP, and the distribution the distances between nearest matches. We illustrate some random example pairs with low to high distance (marked by a star the distance distribution plots).
35

---

Categury
ImageNet ik class
amphibiam artifacts
European fire valumander atolodl bullfrog common new elt spulled salamundr usiled frog tree frug A Alghin hound Band Aid Dirch oven Petri dish sbocus sshcan bockpack ballyoint hunnister barrel buth lowel butheuh beacon besker beer botile heer glass bell cote binder hinthous book jacker bottleep
beenkwater breastplate
bucker
mail chainlink fence chiffonier clawvrr clift dwelling cloak clog
shiker coffee mag comic book cowbuy bool crate crih cnucch cuiness cup disper dishwasher duck envelupe expresso maker face powder fig fire screen lhupole fountain foumtain pen gremask goblet grasihopper grille grocery store guil hair spmy hund blower holster hongynomb iron "juck-o-lanten" jaysick badle lampshuke less cap library liputick lution maillig mailhox manhole cover mask mutchssicck maze measuring cup megalith mene microwave minibus mixing howl mohile hrone meetar mortarheard exoquiito net mountain vent murele necklaee obelisk packet paddle yutio pedestal pencil pencil sharpener perfume pickellussbe picket fence pier piggy
pill bottle pillow pischer plastic bug plate
rack pole pop bottle pot prayer mug purse quill quilt meket ralio hareel refrigerator rotisserie rubber craser minning shoe safe sallshaker seablurd school his schoener scorehooedd shield shoji shopping basket shower cutainn ski musk sleeping bug sliding door scup dieeemeer soup bowl space bar spollight steel arch bridge stone wall stove street sign strecher sunscreem suspension tridge swab wing lodily television thatch theater curtain thimble roof toter pole inflii light thy triumphal arch trollyyhus tub tumstile umbrella vacuum vase viaduct wallle iron wislbessin washer
bottle water jug water
web whiskey jug window
window shade bottle
fonce wreck yurt
bird
African grey American soot American egret European gallinule alburnos buld cagle bee cater biuem black grouse black stock bluck swan brambling bulbul bustand chickadee cock coacal enme dowitcher drake themingo goldiinhh goose gresi gryy oul hen humbill house finch humminghird indigo bumlnng Sacomur say juncu king penguin lise limpkin little blue hemo lorikeet mocow magpic surich oy oystercatceer partridge peliete prairic chicken plarmigan quail raf-hicked sandpiper rnd- rol-hressted mergaeser redshank robin roddy turestoee ruffed grouse spoonbill sulphur crested cuckatio toucun vulture water owsel white stork.
bailding
apiary hum bouthouse caslle church cinema greemhouse home theater momudery monque polace planetarium prison restaumnt siage stupa vault
canine
African huming dog Airodae American Staffondshire teriir Appeszeller Arctic fox Australion terrier Redlington terrier Bernese mountain dog Blonheim spaniel Bander collic
Border lerrier Bodion bull Bouvier des Flandre Brabancon griffun Britiany spuniel Cardigan Chesapeake Bay retriever Chihushua Dandie Dinmont Debernan English foxhound English English springer EmileBucher Eskino dog French bulldag German shert-sainee pointer Geedon
Greal Dane
Pyneees Greater Swiss Mountain dog Irizan
hound Irish setter frish terrier Irish water spaniel Irish welthound Italian greyhound Japanese spaniel Ketry blus tenier Labradee retriever Lakeland tarrier Leonberg Lhusa Maltese dog Maxican hairless New foundlandd Norfelk terrer Norwegian elkhound Neewich terrier Pokinese Peméecoke Pomeramian Rhudesian rigebbck Rotweeller Suini Demard Suluki Samoyed Scotch lerrier Sootishh dorthound Sealyhum hum lerrier Shellind shoepdog Shih- Ton Siburian hady StafTockhhir bullteriier Sussen spunie! Tibetan mastif Tibetan lerrier Walker hound Weimaraner Welsh springer spuniel West Highland whise terre Yorkobize terierr affenpinscher basenjii baseet beugle black and tan cocahound bloodhound bluetick buzzui beizrd bull masilf caim chow clumber cocker spuniel vollie cuyose curly posied retriveer dalmatian thole dingo flat couted retriever giant schnauzer golden retriever grey fox. groenendael byena keeshond kelpie kit fox lemonder knvner malomate milinois ministare pinscher ministure poodle ministure schnauzer otierhound papillon pug rod fox red woll' rechune schipperke silky terrier sod costed w hheuten terrier standard poodle sundard schmauzer timber wolf poodle terrier vizsla whippet white wolf wire haired fox terriet Christmas stocking Loufer English shoepág Windher te abuya academic gown apron buliing cup beinkin
dothing
bolu burnel bow li brassiore builetproof vest curdigun chest cowboy hat crash helmet dishrng feather bou for cost gown handkrrchic book hoopskirt jean jemey kimono knce pad lab coat mailiot military uniform miniskin mitten overskin pujana puper towel poncho sandal saring sest bell shower er sock sombrero stule sait sw custiir ywimming uunks trench cost velvet vestment waller wig wool accordion acoustic guilar andlog clock assauk rille hanjo barometer bessoon binoculars bow backle bullet train candle mimur car wheel cash machine cello chime combination gas pump gong slide hammer hamper hand held computer
device
lock desktop computer digital clock digital watch brake drum drumstick electrc fan electric guitz
disc
harmonica harp hatchet hom hourglass lopopp lighter loudspeaker loupe magnecic compuss maraca marimba maypols microphone missile monitor mouse mousetrap neck brace woilockkuini scale screen sewing machine ski slide rule slot slug sookee solar dish space heater spider web dram stethoscope stopwatch strainee sundial sunglass sunglasses switch syringe thresher tosser torch tripod trumbone typewriter keyboard upright vending machine violin wall clock whistle
notebook abue odometer oil filter irgan esygen mask podllewhedd pudlock puintruush pumpipe purking meter pick "potters wheel" projector puck reliator radio selescope remote control revolver rifle safety
equipmeat
CD player Polaroid camera balance bears burbell "carpesters kit" cassette player celiular telephone computer keybourd croquet ball cressword puzzle dial telsphose drilling platfocm dumbbell polt ball golfeart horizontal bar iPod jigsaw puzzle modem oscilloscope parachute parallel bars pay phone photocopier ping pong bill plate ball lape player
bag reel reflex
sstablishmaat
bakery barbershop bookshop butcher shop confectionery shoe shop tobacco shop soyshop
feline
Egyptia cat Persian cat Siamese cat catamount cheetah coal congar leopard licn punther snow leopard tabby tiger tiger cat
fish
unemune fish barazoula cohu cel electric my gar goldlish great while shark hammerhead liunfish puller ruck beanty stingna sturgeon tench diger sbark French loa! hagel burito carbonara cheeseburger chocolate sauce consomae zucumber dough eggnog espresso guacamole hay hot pot holdog meat loaf pizza potple pretzel
food
loily mashed potato
fruit
Granny Smith accen buckeye hip jackfruit lemon orange pincapple rapesead marherry
funiture
alte birber chair bassinet beaver bookcase china cabunet cradle desk dining tuble entertainment center file folding chair four tour-poster medicine chest milk can mink otter park bench pool table rocking chair studio couch table throne tuilet wardrobe alp bubble cliff coral reef dome geyser lakeside promantary sandbor seashore valley volcano
geological for-
mation
insect
ant bee cabbuge butterfly cicada cricket danseltly dragontly dung beedle fl ground boerle lacewing ladybug leaf bectle lealbopper long homnd beetle lycaenid mantis monarch
peacock Ahinnceros beetle singles sulphur tunertly tiger beetle walking sick weevil
mannal
clephant Madagascar cat scrofa amnadilo baboon bighara bison black foosed ferret brown bear capuchin chimpanzez oolobus dagong echidra fitch fox squirrel gizelle gibhon garilla grey whale guznon guinca pig hamster hare liitheesst hippopottanu thex ioe hear imqula indri killer whale knala langer lesser panda liams macaque mermoiet marmor mezrkat mongoose orangatan ox panda patas
African eleyhant American black bear Angora Arabian camel
polecat poscupine prohescis monkey ram sea lion siamang sloth bear spider monkey squirel monkey thrse teed sloth tusker wallsby warthog water buffalo wombut wood rabbut zebra chain cornet doonsat groom knot spincli tollet tissue grand pizno
material
musical
organism
America lobuter Dungeness crab German shepherd admiral aguric badger ballplayer barn spider black and gold garden spider black widow bolete boxer brain voral centipede chiton
cockroach canch cocal fungus crawfish dum carbistar fiddler crsh flarworn garden spider gyominm harvester harvestaan kin-ofthh-wwoods hemiit crah howler monkey ssopad jellylish king crab mushmom sematode nipple printer rock cnb rule scorpoon scuba diver sea cucumber sea slug sea urchin stail spiny lobstar starfish stinktionn tamanula tick vilobite weasel wing woll spider
plant
acora squssh artichike banans hell pepper brocooli buttemut squish cardoon couliflower corn custard apple daisy head carbage octsina plawhesl pomegrannte sea anemone sorrel spagitetii squasi 'yellow ladys supperr zuceluni
reptlle
African chamekean African cracndile American alligstor American chameleon Gila meester Indian cobra Komado dragon agamtn alligstor lizard bunded gocko hoa constrictor box tuttle common iguans diancudhtek frilled lizard grass snike grees lizard grees mambaa green stake hoguose snake king snake leathehhack tartle loggerhead mud turtle night sanke ringneek snake rock python sand viper sca stake sidewinder termapin thunder snake theeratops vine snake water snake whiptail boschall backethell foetball helmet mihhy ball tennis hall vallsyhall
sport
tnol
can openes chain saw carkserew lown mower letter opener lumbermill nail plane plow plunger power screw screwdriver showel Crock Por caidron coficepot frying pas spazala teapot wok woodes spoon
utansil
vehicle
Modal T aiteraft cariee air liner nirship andullance amphihian balloce barow beach wagon hicycle-built--e fort----ilt------uu boholad cab cance catamtaaan chamberred namtilus container ship convectible dagsted electric Incomsotive
eagune firchost forklift freight garbage tunk go go-kart goodela half track horse cart jeep jinrikisha lifoboat limeusine liner minivan moped motor scooter meontiin bike mosing van oxcart possenger car pickup pince racer receentional vchicle shopning cart snowmohile snowplow space shutlle speedhout sports car steam locomotivve strectear nahmarine tank now trick nacttr trailer tnuck tneycle trimam enieycle wagen warplane yos!
Table 4: Mapping between categories from the WordNet Hierarchy and the ImageNet- k classes used for assigning a category to the concept clusters.
36


Reviews:
[{"review_id": "fo8xOwqr39", "reviewer": "Reviewer_8f6e", "summary": "The authors propose a new method of concept level alignment analysis in pre-trained vision transformers. The first part of the author's solution involves building a manifold based approach of concept discovery in neural network representations which they call NLMCD: first they reduce the representation dimensions using UMAP, and then perform heirarchical clustering using HDBSCAN. This leads to 'soft' assignment of feature representations to different concept clusters. In order to develop an alignment metric using their soft clustering, the authors adapt a generalization of Rand index - the distance between two concept clusters is given as an average of distances between two membership vectors. The authors define concept-based alignment (CBA) as difference of this averaged distance from 1.The authors perform several quantitative checks on their concept discovery approach, and the results from these are somewhat mixed. A sanity check on the CBA metric is performed against CKA which shows that the author's metric offers more fine-grained alignment scores. Lastly, and perhaps the most impactful section of the work, the authors perform concept level alignment analyses on four different pre-trained models on ImageNet and confirm several results from representation level analyses in the past.", "questions": "1. In Line 187 \"we test the robustness of our approach by measuring the alignment between two runs (with different initializations)\" can the authors confirm what they mean by different initializations - are these different initializations of the clustering, or of the vision transformer training, or something else?\n\n2. It is unclear to me why the authors consider the SEQ tokens at all, and especially after pooling across layers, which would lead to signficant information loss in terms of granularity of concepts learnt. Can the authors explain why they examined SEQ tokens?\n\n3. In the comparison of vision models, please include more references beyond just analysis of attention patterns, CKA, and downstream performance. A lot of research has been done on comparing vision model representations as well as concepts across different lenses (e.g. training objectives, dataset sizes, downstream tasks, transfer learning, multi-modal learning).  Some of these lines of research (not exhaustive) are in these papers:\n\n* On the duality between contrastive and non-contrastive self-supervised learning, Garrido et al\n* Objectives Matter: Understanding the Impact of Self-Supervised Objectives on ViT Representations, Shekhar et al\n* A large-scale examination of inductive biases shaping high-level visual representation in brains and machines, Conwell et al\n* Objective drives the consistency of representational similarity across datasets, Ciernik et al\n\n4. How are the concept formation graphs (CFGs) made in Fig 3? As in how did the authors figure out the layer 7-9 concept development - was this through manual observation, or was there some quantitative indicator e.g.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1. Concept-based alignment instead of representation-based alignment is a better approach towards quantifying representation similarities across neural networks, training objectives, and training distributions. The authors approach is among one of the first methods towards this, and as such quite novel. \n2. The concept alignment analysis is quite interesting, the authors' analysis helps to re-establish some representation level results for feature learn through concept level analysis too (e.g. inter-modal concept alignment in early layers, similarity in concepts across similar training objectives.) \n\n**Weaknesses**\n\n1. The algorithm used for calculating concept discovery (UMAP followed by HDBSCAN) is very compute intensive, which is reflected in the author's experimental choices - using 25% of ImageNet, pooling sequence tokens, perhaps the use of UMAP itself. It will be hard to translate this concept discovery method (and hence alignment metric) to web scale datasets, where alignment is perhaps more important as data quality is not controlled. \n2. The experimental results for concept discovery evaluation are not presented well, and also have a lot of variance across models. Why does the NRMSE drop for intermediate layers for some models? Yet other models like CLIP have their values same across all layers? Robustness decreasing with layers is generally not a positive sign, although the absolute values are themself high enough where it might not be an issue. Overall, it is not clear to me as a reader what inference and conclusions to draw from the author's results. Perhaps the section 4.1 evaluation can be broken down into evaluation and results separately to explain the results better.", "was_revised": true, "final_justification": "The authors have provided adequate responses to the questions raised in my review. I think the main merit of the paper is in the problem its trying to address, which is both important and underexplored. The method used is quite novel, and while not scalable, the results are important enough to drive further research into concept based alignment.", "rebuttal": "We appreciate the assessment of our method as novel and our concept alignment analysis as interesting, and thank you for the helpful feedback.\n\n**[W1] Computational Complexity**\n\nWe agree that the UMAP+HDBSCAN pipeline is compute-intensive and list this as a limitation. In future work, we want to explore more scalable alternatives, such as deep density-based clustering (e.g., Beer et al., 2024), which may allow efficient clustering at scale.\n\n**[W2] Presentation of concept discovery evaluation**\n\nWe appreciate the suggestion to clarify this section and will improve separation between evaluation and results. \nRegarding the drop in DBCV for the FS model around layer 9 (and the smaller drop at layer 6 in DINO), we note that these layers coincide with a pronounced increase in the number of discovered concepts (Fig. 5). In contrast, as you notice, the increase of concepts across layers is smoother for CLIP, and also MAE.\nWe hypothesize that this reflects a sudden reorganization of concepts, which makes the clustering problem at these layers  more challenging before the representation exhibits a new structure.\n\n**[Q1] Evaluation of Robustness**\n\nWe tested robustness by re-running the clustering pipeline with different random seeds (i.e., different UMAP/HDBSCAN initializations) while keeping model weights and input samples fixed. We will clarify this in Line 187.\n\n**[Q2] SEQ token**\n\nWe analyze spatial (SEQ) tokens to localize emerging concepts in the input space. ViT patch tokens are widely observed to retain spatial correspondence (e.g., Caron et al. 2021), so clustering SEQ tokens allows us to capture localized concepts. We pool only across spatial tokens to obtain a representative sample across all ImageNet images, and never across layers.\n\n**[Q3] Related work**\n\nThank you for the suggestions, we will add these papers to our Related Works section.\n\n**[Q4] CFGs**\n\nThe textual concept description in Fig. 3 are maually crafted, but the graph itself is constructed via in algorithm described in appendix D. In short, we compute transition rates of tokens between concept clusters at two consective layers, an then construct a binary unidirectional graph with edges between those concept nodes for which the respective transition rate surpasses a thresold.\n\nBeer et al., 2024: A. Beer et al., \"SHADE: Deep Density-based Clustering,\" 2024 IEEE International Conference on Data Mining (ICDM), Abu Dhabi, United Arab Emirates, 2024, pp. 675-680\n\nCaron et al. 2021: Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand\nJoulin. Emerging properties in self-supervised vision transformers. In International Conference on Com-\nputer Vision, pp. 9630–9640, 2021"}, {"review_id": "ZEOqCrbzxZ", "reviewer": "Reviewer_ErWZ", "summary": "This paper proposes a novel framework to analyze and compare deep visual representations at a conceptual level rather than with a single similarity score. The authors combine concept discovery (finding interpretable clusters or “concepts” in a model’s latent space) with representational alignment analysis, yielding fine-grained insights into how different layers or models align in terms of learned features. Key contributions include: (1) defining concepts as non-linear manifolds in feature space (as opposed to traditional linear directions), and developing a clustering-based method to identify these concept manifolds (using UMAP for dimensionality reduction and HDBSCAN for unsupervised clustering); (2) introducing a concept-based alignment (CBA) measure using a generalized Rand index that quantifies similarity between two sets of concept clusters, which can be decomposed into pairwise distances between individual concepts; and (3) applying this analysis to several Vision Transformers (ViTs) trained with different paradigms (supervised, contrastive, self-supervised, etc.), revealing how their internal representations differ. Notably, the paper finds that certain high-level concepts are universal across models, while others are model-specific, and that increased training supervision tends to reduce the semantic organization of learned features. The proposed approach thus goes “beyond scalars” like CKA by pinpointing which concepts are shared or divergent between representations, offering a more interpretable comparison of model internals.", "questions": "1. Semantic coherence of discovered concepts: By qualitative inspection, the discovered concept clusters (e.g. an “apples” concept in Fig. 3) appear semantically coherent, with the nonlinear (NLMCD) clusters showing high within-cluster consistency. How do the authors evaluate or ensure that these data-driven concepts are indeed meaningful and interpretable to humans?\n\n2. Fine-grained CBA vs. scalar alignment (CKA): The proposed Concept-Based Alignment (CBA) measure achieves performance on par with standard scalar alignment measures like CKA in the sanity check, while additionally providing a fine-grained view into how individual concepts align. What new insights does this concept-level alignment enable that a single scalar (e.g. CKA) might obscure? For instance, have the authors observed cases where two models or layers have similar overall alignment scores yet differ in which specific concepts are aligned (or misaligned)? Highlighting such a scenario would clarify the practical advantage of CBA’s fine-grained analysis over traditional one-number similarity metrics.", "limitations": "See weakness and questions", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1. Innovative Fine-Grained Analysis: The paper addresses a clear limitation of existing representational similarity measures (e.g. CKA) which collapse complex relationships into a single number. By integrating concept discovery with alignment, the method yields rich, interpretable information about what is similar between layers or models (i.e. which concepts align). This is a novel perspective that provides insights not available through traditional scalar metrics.\n2. Novel Concept Definition: Treating concepts as non-linear manifolds in feature space is an innovative generalization beyond prior linear subspace definitions. This more flexible definition is well-motivated (aligning with the manifold hypothesis) and the authors demonstrate its advantage: in a controlled “sanity check,” their non-linear concept-based alignment outperforms simpler linear concept approaches in capturing expected layer-wise similarity patterns.\n3. Insightful Findings for Vision Transformers: Using the proposed analysis, the paper provides interesting observations about ViT models. For instance, self-supervised models (DINO, MAE, CLIP) develop strongly semantic, coherent concept structures in later layers, whereas the fully supervised ViT (FS) shows less semantic organization in its top-layer concepts. \n4. Clarity and Organization: The paper is well-structured and easy to follow. \n\nWeakness:\n1. Complexity and Scalability: A potential concern is the computational complexity of the concept discovery pipeline. The approach relies on UMAP for embedding and HDBSCAN for clustering, which required the authors to subsample the dataset (25% of ImageNet and spatial token pooling) due to scalability limits. This subsampling raises the question of whether some fine concepts might be missed or labeled as “noise” by HDBSCAN when data is limited.\n2. Scope Limited to ViTs: The experiments focus on four ViT models with a specific architecture and on a single dataset (ImageNet). It remains unclear how well the proposed alignment analysis generalizes to other network architectures (e.g., CNNs or different ViT variants) or other data domains.", "was_revised": true, "final_justification": "I thank the authors for their detailed rebuttal. Many of my concerns have been adequately addressed. The additional discussion comparing concept-based alignment and scalar-based alignment was particularly insightful. Overall, I find the paper to be of sufficient quality for acceptance, and I have decided to maintain my original score.\n\nAdditionally, I encourage the authors to consider the following suggestions for future work, which may further enhance the impact and generality of their approach:\n\n1. Extend the experiments to a broader range of model architectures and language models to better demonstrate the generalizability of the proposed method.\n\n2. Investigate potential failure cases where the scalar alignment score does not accurately reflect model behavior, but the concept-based alignment does. Such analysis would further highlight the advantages and robustness of your approach.", "rebuttal": "Thank you for your helpful feedback and describing our analysis as innovative, novel, and insightful.\n\n**[W1] Computational Complexity**\n\nWe agree that the UMAP+HDBSCAN pipeline is compute-intensive and list this as a limitation. In future work, we want to explore more scalable alternatives, such as deep density-based clustering (e.g., Beer et al., 2024), which may allow efficient clustering at scale. \n\n**[W2] Scope**\n\nWhile our experiments focus on ViTs, we note that our method is model-agnostic and applicable to other architectures. We plan to extend to CNNs and and in particular to language models in future work. \n\n**[Q1] Concept interpretability**\n\nNLMCD is not optimized for human interpretability per se — it captures concept manifolds that are meaningful from the structure of the model's representation. To assess human interpretability of the discovered concepts, we inspected the concepts ourselves. In the future, it would be interesting to evaluate this in more depth, by performing a human user study in the spirit of Ghorbani et al. (2019) or use a foundation model as a proxy human like Dreyer et al. (2025).\n\n**[Q2] Insights from concept-based alignment**\n\nThanks for this helpful prompt to further demonstrate the additional insights that fine-grained concept alignment offers. Based on your question about concept-level differences between pairs of representations that share the same overall alignment score, we performed the following analysis: \n\nLayer 10 in MAE/DINO (see Fig. 6) and the same layer in CLIP/DINO (see Fig. 16) share the same overall alignment (0.74). To summarize fine-grained differences in concept-alignment, we grouped concept nearest-neighbor-pairs into the WordNet categories from Fig. 4 (88% of concept pairs are in the same category for MAE/DINO, 93% for CLIP/DINO). We then computed the mean concept distance within each concept category and find that 1) canine and insect concepts have high distance in both model pairs, while equipment and material concept have low distance in both pairs; 2) fish concepts are similar in MAE/DINO but have high distance in CLIP/DINO, while sport concepts show the opposite trend. \n\nThis exemplifies a scenario where two representations are similarly aligned by the overall score, but fine-grained concept alignment differs. We will add this analysis to the appendix.\n\nBeer et al., 2024: A. Beer et al., \"SHADE: Deep Density-based Clustering,\" 2024 IEEE International Conference on Data Mining (ICDM), Abu Dhabi, United Arab Emirates, 2024, pp. 675-680\n\nGhorbani et al. 2019: Ghorbani, A., Wexler, J., Zou, J.Y., & Kim, B. (2019). Towards Automatic Concept-based Explanations. Neural Information Processing Systems.\n\nDreyer et al. 2025: Dreyer, M., Berend, J., Labarta, T., Vielhaben, J., Wiegand, T., Lapuschkin, S., & Samek, W. (2025). Mechanistic understanding"}, {"review_id": "TkWGwbQ3BS", "reviewer": "Reviewer_V5Qu", "summary": "- This paper proposes an interesting analysis pipeline for investigating and exploring the alignment properties of representations in Vision Transformers. Specifically, unlike existing approaches that reduce alignment to a single scalar value, the authors introduce a concept-based alignment metric that treats concepts as nonlinear manifolds. In practice, they use a soft clustering method, i.e., HDBSCAN applied on the UMAP projection, as a proxy for identifying these concept manifolds. This metric is then used to evaluate alignment across fuzzy concept clusters derived from different architectures. The proposed pipeline is applied to four ViT models, i.e., FS, CLIP, DINO, and MAE, revealing insightful structural differences. The analysis enhances the interpretability of Vision Transformers and highlights their potential in downstream tasks.", "questions": "- Appendix B describes the design choices for hyperparameters such as UMAP’s min_dist, n_neighbors, and embedding dimensionality, as well as HDBSCAN’s min_cluster_size and min_samples, based on maximizing the DBCV score. It would be helpful to include a hyperparameter sensitivity study to show how variations in these settings affect metrics like noise rate, DBCV, NRMSE, and CBA alignment scores. This would further clarify the robustness and stability of the proposed method.\n- The proposed analysis pipeline is suitable to be extended to multimodal foundation models, or compare the image and text encoders in CLIP, and assess how their fusion affects representational alignment?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths \n\n- The proposed method shows strong potential for future extensions.\n- The approach is conceptually simple and intuitively appealing.\n- The paper presents the proposed methodology in a clear and well-structured manner.\n\n# Weaknesses\n- The proposed method appears to be time-consuming, as it requires traversing all vectors. However, the authors do not provide any analysis of the computational cost, which undermines the practical value and contribution of the method.\n- The paper relies on ImageNet class labels and coarse WordNet categories as weak proxies to annotate discovered concepts, which may not accurately capture the true semantic content of each concept manifold. Moreover, in the CLIP experiments, the analysis is confined to the visual encoder; extending the analysis to CLIP’s text encoder and to the joint image–text embedding space could reveal richer multimodal alignment patterns and better validate the method’s generality.", "was_revised": true, "final_justification": "After reading the comments from the other reviewers and the authors' rebuttal, I find that most of my concerns have been addressed, as noted in my earlier response. Overall, the paper presents a clear and technically sound analysis pipeline that offers fine-grained insights beyond scalar alignment scores, and the rebuttal effectively resolves the key practical questions. Therefore, I will maintain my current rating.", "rebuttal": "Thank you for the positive assessment of our method as conceptually simple and intuitively appealing with strong potential for future extensions. We are excited to apply our method in other use-cases in the future, especially for the analysis of multi-modal models.\n\n**[W1] Computational cost**\n\nWe agree that computational cost of computing CBA is a limitation of our method, and we will add this to the Limitations paragraph. In practice, we subsample the feature vectors and use batched matrix operations to reduce runtime. In appendix B, we report runtime on a V100 GPU (concept-wise CBA for representations with around 500 concepts takes 6 h).\n\n**[W2]  1) WordNet categories**,  2) confinement to CLIP image encoder – see [Q2]\n\nWe clarify that coarse labels are used only for visualizing the concept atlas in Fig. 4. We agree that they might not capture the true semantics of the concept manifold and acknowledge their limitations in a footnote in L. 269 (we will fix the formatting issue with this footnote).  In future work, we aim to build an interactive tool to better visualize the concept atlas, where the user can click on each concept to see a visualization. At this point we also want to clarify that class label alignment in Fig. 5 is based on the more fine-grained original Imagenet labels.\n\n**[Q1] Hyperparameter sensitivity**\n\nWe performed  a small sensitivity study (at layer 11 of DINO SEQ representations) that varies UMAP and HDBSCAN parameters one at a time,  and observe that our concept discovery evaluation metrics are reasonably stable within a quite broad hyperparameter range. \n\nUMAP hyperparameters and embedding/clustering quality:\n\n\n\n| Hyperparameter           | Range       | NRMSE (min < **default** < max) | Noise ratio            | DBCV                   | Robustness             |\n| ------------------------ | ----------- | ------------------------------- | ---------------------- | ---------------------- | ---------------------- |\n| **Embedding dim (zdim)** | 20 – 100    | 0.7 < **9.5** < 15.1       | 0.66 < **0.66** < 0.66 | 0.56 < **0.58** < 0.59 | 0.84 < **0.84** < 0.87 |\n| **min\\_dist**            | 0.005 – 0.5 | 0.7 < **9.5** < 13.6        | 0.64 < **0.66** < 0.72 | 0.53 < **0.58** < 0.58 | 0.81 < **0.85** < 0.86 |\n| **n\\_neighbors**         | 10 – 40     | 6.4< **9.5**         | 0.65 < **0.66** < 0.66 | 0.58 < **0.58** < 0.61 | 0.84 < **0.85** < 0.85 |\n\nHDBSCAN hyperparamters and clustering quality: \n\n| Hyperparameter         | Range     | Noise ratio            | DBCV                   | Robustness             |\n| ---------------------- | -------- | ---------------------- | ---------------------- | ---------------------- |\n| **min\\_cluster\\_size** | 20 – 100          | 0.57 < **0.66** < 0.73 | 0.57 < **0.58** < 0.59 | 0.83 < **0.84** < 0.85 |\n| **min\\_samples**       | 10 – 50       | 0.66 < **0.66** < 0.67 | 0.50 < **0.58** < 0.65 | 0.83 < **0.84** < 0.85 |\n\n\nOnly NRMSE shows a rather high range when varying zdim and min_dist with minimum <1.0 for small embedding dimensions (<=40) and very small min_dist. This is likely not a sign of a superior embedding, but rather an unintuitive behaviour of this metric due to extreme compression of pairwise distances in the embedding.\n\nWe will add the results of this study to the appendix.\n\n**[Q2] Comparison of image and text encoder in CLIP**\n\nWe see cross modal concept based alignment as an exciting application of NLMCD CBA, which is directly applicable to the CLIP text encoder. Due to the limited rebuttal period, we cannot present results here, but here is what we would do:  We would focus on CLS token representations, as image localized concepts (e.g., a cat’s whiskers) do not directly correspond to textual mentions (e.g., “cat with whiskers”). Further, we hypothesize that cross modal concept alignment is most meaningful at the final layers, where image representations are more abstract and more likely to align with the concepts described in the associated text."}, {"review_id": "mamBROHjDW", "reviewer": "Reviewer_wzkQ", "summary": "This paper proposes a new method for evaluating the similarity and differences in vision models' representation by combining concept discovery with alignment analysis. As opposed to defining concepts as linear subspaces in representation space, this paper approaches concepts as being embedded in low-dimensional non-linear manifolds. Network's internal features are projected via UMAP and then clustered with HDBSCAN. Then, the distances of clusterings from different networks are calculated. This method is evaluated on multiple vision models (e.g., DINO, CLIP, etc.) with ImageNet images.", "questions": "* The paper can be further strengthened by showing how the method enables new insights into the representations of these models. Most of the experiments just demonstrate sanity checks. The finding that models supervised with labels have reduced semantic structure in the learned representations is interesting. Why does this not happen in CLIP as it has supervision in the form of text?\n\nHere are a few questions that this method can be used to answer:\n* What is the same and different between generative and discriminative models. How does the semantic structure differ and how is it similar?\n* I do not see any reason this method cannot be applied to a language model. How is the semantic structure different between the vision and language encoder in CLIP?\n\nThese are just a few ideas that can make me more convinced of the utility of the method.", "limitations": "Yes, the authors have a dedicated limitations subsection mentioning the limited scalability of HDBSCAN.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* The idea of measuring similarity/differences between networks using interpretable concepts rather than a single scalar is interesting and can be helpful in better probing what these model do and don't know. \n* Most works on interpreting representations assume the linear representation hypothesis, while this work is more unique in assuming non-linearity and leveraging non-linear tools. \n* The proposed method (NLMCD) passes sanity checks better than baselines. Additionally, some qualitative observations are made the show the utility of NLMCD. \n\n**Weaknesses**\n* The ultimate goal of a new alignment metric would be to facilitate some new insight. Examples from the paper, such as finding that earlier layers focus on lower level features, are good sanity checks that have already been established. Some findings that would not be enabled by other similarity metrics would strengthen the paper. The most interesting finding was that models which were supervised with labels have reduced semantic structure in the learned representations. I think this can be emphasized more and further developed. \n* The method itself is not completely novel. It combines existing tools such as UMAP and HDBSCAN. This makes it more important to emphasize how this combination of tools enables the discovery of new phenomena which otherwise would not have been discovered. \n* [1] seems similar in spirit. It clusters neurons across vision models based on similarity, forming concepts.\n\n[1] Dravid et al. \"Rosetta Neurons: Mining the Common Units in a Model Zoo.\" ICCV 2023.", "was_revised": true, "final_justification": "I was fairly borderline on this paper for two main reasons. The first one is novelty, since the paper combines existing tools such as UMAP and HDBSCAN. This method, as acknowledged by the paper, is not very scalable, which limits its applicability on larger models trained on more diverse data, where explainability methods would be more impactful and yield more meaningful insights. While the proposed definition — \"a novel definition of concepts as nonlinear manifolds\" (L45) — is a unique direction, definitional novelty alone is difficult to assess unless it leads to new types of insight or analysis. Which leads to my second main concern, that the method has not demonstrated to provide a practical application, theory, or produce new scientific insights beyond what has already been shown in explainability/interpretability/representational alignment literature (e.g., earlier layers focus on \"lower-level structures, shapes, and object parts\" (L270-271)). The analysis is also ImageNet-centric and focuses on only discriminative models, limiting its scope. In my original review, I provided multiple potential applications and questions that the method could be used for, which were **not** resolved in the rebuttal. I am confident in this assessment as I work on representational alignment and interpretability. That being said, the paper is well-executed and the method is fairly well-motivated. I will give the benefit-of-the doubt and maintain my score.\n\nAs a sidenote, I was note able to get to this during the discussion. The finding that ViTs supervised with labels have reduced semantic structure in the learned representations is in line with the well-known phenomenon of \"neural collapse\" [1], where representations in the penultimate layer of a supervised model collapses to the class mean in order to encourage max-margin separation. I think that some discussion of this would be valuable. \n\n[1] Papyan et al. \"Prevalence of neural collapse during the terminal phase of deep learning training.\" PNAS 2020.", "rebuttal": "Thank you for your valuable feedback and positive assessment of our method as helpful and unique.\n\n**[W1] New insights**\n\nWe agree that this is our most interesting finding, which we highlight it in the abstract and introduction. \nRegarding new insights, we will add a small analysis that compares two representation pairs with same overall alignment but different alignment on the concept level to the appendix, which we describe in further detail in the answer to [Q2] of ErWZ.\n\n**[W2] Novelty**\n\nWe agree that the components (UMAP, HDBSCAN) of our concept discovery pipeline are well established. Our contributions, which we describe in the Introduction, rather lie in: (1) the novel definition of concepts as nonlinear manifolds, and (2) decomposing alignment into concept-level comparisons and thereby enabling insights that single-scalar methods like CKA miss.\n\n**[W3] Related work**\n\nThanks pointing out the paper by Dravid et al., we will add it to our Related Works.\n\n**[Q1] FS vs CLIP model**\n\nThe CLIP model differs from fully supervised ViTs in that its supervision is soft and semantically structured (via textual description of images that vary across samples from one class). This encourages representational similarity between fine-grained categories (e.g., all dogs like \"Labrador\", \"German shepherd\"), in contrast to hard class separation in fully supervised setups.\n\n**[Q2] Generative vs. discriminative models**\n\nThis is an interesting question - we think that Fig. 4 points to some answers for this question by comparing the FS model with MAE, which is trained with a BERT style masked patch reconstruction objective. \n\n\n**[Q3] CLIP text encoder**\n\n\nWe agree that cross modal alignment is a natural extension, and NLMCD CBA is directly applicable to representation of the CLIP text encoder Extending this to text and image encoders that were not jointly trained (like Maniparambil et al. 2024 and Huh et al 2024 )\nis also interesting question, to which NLMCD-CBA could add by understanding alignment on a concept level.\n \nDue to the time constraints of the rebuttal period, we defer these experiments to future work. In such an analysis, we would focus on CLS token representations, as image localized concepts (e.g., a cat’s whiskers in an image) do not directly correspond to textual mentions (e.g., “A cat sitting on a table with whiskers.”). We hypothesize that cross modal concept alignment is most meaningful at the final layers, where image representations are more abstract and more likely to align with the concepts described in the associated text.\n\nMayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, Noel E. O'Connor; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 14334-14343\n\nMinyoung Huh, Brian Cheung, Tongzhou Wang, Phillip Isola \nProceedings of the 41st International Conference on Machine Learning, PMLR 235:20617-20642, 2024."}]

Author rebuttal:

Meta-review: This submission proposes to use UMAP followed by HDBSCAN clustering to cluster activations of neural networks, and then to use the similarity between the clusterings of examples across neural networks to measure representational alignment. The resulting clusterings are interpretable and seems sane insofar as it assigns higher similarity to nearby layers than to distant ones. When applied to four ViT models trained with different forms of (self-)supervision, the proposed method reveals that earlier layers are more similar across models than later ones, and demonstrating that the method allows to decompose the scalar similarity measure into similarities between pairs of concepts.

Reviewers felt that the approach to representational similarity was novel and promising, and the comparison of ViTs was interesting. They noted that the method is very computationally intensive, but did not feel that this limitation was sufficient to undermine the utility of the method.

I agree with the reviewers that the proposed method seems promising and the evaluation is convincing insofar as the general sanity of the method. It would be nice if the submission offered a clearer demonstration that the method can be used to discover new insights into the differences between models that can then be independently validated by some other means. But overall, this is a valuable contribution that introduces a novel and interpretable approach to measuring representational alignment, with clear potential for future applications.
Decision: Accept (poster)