Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometry
Authors: Thomas Fel
Binxu Wang
Michael A. Lepori
Matthew Kowal
Andrew Lee
Randall Balestriero
Sonia Joseph
Ekdeep Singh Lubana
Talia Konkle
Demba E. Ba
Martin Wattenberg
Venue: ICLR 2026 Poster (2026)
Keywords: computer vision
interpretability
convex geometry
Primary area: interpretability and explainable AI
TL;DR: We used sparse autoencoders to build a 32k-concept dictionary interpreting DINOv2. Discovering departures from pure sparse linear coding, we propose the Minkowski Representation Hypothesis: tokens form through convex mixtures of archetypes.

Abstract: DINOv2 sees the world well enough to guide robots and segment images, but we still do not know what it sees. We conduct the first comprehensive analysis of DINOv2’s representational structure using overcomplete dictionary learning, extracting over 32,000 visual concepts in what constitutes the largest interpretability demonstration for any vision foundation model to date. This method provides the backbone of our study, which unfolds in three parts. 

In the first part, we analyze how different downstream tasks recruit concepts from our learned dictionary, revealing functional specialization: classification exploits “Elsewhere” concepts that fire everywhere except on target objects, implementing learned negations; segmentation relies exclusively on boundary detectors forming coherent subspaces; depth estimation draws on three distinct monocular cue families matching visual neuroscience principles. 

Turning to concept geometry and statistics, we find the learned dictionary deviates from ideal near-orthogonal (Grassmannian) structure, exhibiting higher coherence than random baselines. Concept atoms are not aligned with the neuron basis, confirming distributed encoding. We discover antipodal concept pairs that encode opposite semantics (e.g., “white shirt” vs “black shirt”), creating signed semantic axes. Separately, we identify concepts that activate exclusively on register tokens, revealing these encode global scene properties like motion blur and illumination. Across layers, positional information collapses toward a 2D sheet, yet within single images token geometry remains smooth and clustered even after position is removed, putting into question a purely sparse-coding view of representation. 

To resolve this paradox, we advance a different view: tokens are formed by combining convex mixtures of a few archetypes (e.g., a rabbit among animals, brown among colors, fluffy among textures). Multi-head attention directly implements this construction, with activations behaving like sums of convex regions. In this picture, concepts are expressed by proximity to landmarks and by regions—not by unbounded linear directions. We call this the Minkowski Representation Hypothesis (MRH), and we examine its empirical signals and consequences for how we study, steer, and interpret vision-transformer representations.

Published as a conference paper at ICLR 2026
INTO THE RABBIT HULL: FROM TASK- CON- CEPTS IN DINO TO MINKOWSKI GEOMETRY
Thomas Fe Binxu Wang
Michael A. Lepori" Matthew Kowal
Andrew Lee"
Randall Balestriero" Martin Wattenberg' u.
Sonia Joseph
Ekdeep S. Lubana AA Talia a
Demba Ba a
"Kempner Institute, Harvard University
Harvard University
Deett. of Psychology, Harvard University "FAR.AI
Brown University
/ Google DeepMind Meta
Gooddire
kempneringg insti tute. tute.gitteet gi io/dinovision
Bhewhere Concepta
Bonder Corcapts [Seg mar (ardor))
Manoosiar Degth Cue Concepts (Depth mationl
Tokes speelle Concegts
(Reghtere CLS, Panch))
Ocomety & Butistiss
Bloon of Poottion
The shops of Image
Minkoanndd Repreeentation
ABSTRACT
DINOv2 sees the world well enough guide robots and segment images, but we still do not know what it sees. As working baseline, we operationalize the Linear Representation Hypothesis (LRH) (features as sparse combination near-orthogonal directions) using sparse autoencoders, yielding 32,000 unit con- cept dictionary what constitutes the largest interpretability demonstration for a vision foundation model to date. This method provides the backbone of our study, which unfolds in three parts. First, we analyze how different downstream tasks recruit concepts from our concept dictionary, revealing functional specialization: classification exploits "Elsewhere" concepts that implement "object negation"; segmentation relies exclusively boundary detectors forming coherent subspaces; depth estimation draws on three distinct monocular cue families aligning with visual neuroscience principles. Turning to concept geometry and statisties, we find higher coherence than random or Grassmannian baselines, sharply decaying spectra with task ass-nedg anisotropy, antipodal pairs forming signed axes, and lwwdiieesioonal token neighborhoods. These patterns reveal partially dense, struc- tured representations that question a purely ppurse-coding view of representation. Motivated by these departures, we advance different view: tokens are formed by combining convex mixtures of a few archetypes (e.g.. a rabbit among animals, brown among colors, fluffy among textures). Multi-head attention directly imple- ments this construction, with activations behaving like sums of convex regions. this picture, concepts expressed by proximity to landmarks and by regions not by unbounded linear directions. We call this the Minkowski Representation Hypothesis, and present outline, together with observed departures from LRH. We conclude by examining how this perspective changes approach studying, steering, and interpreting vision- transformer representations.
working hypothesis whose testable predictions we

---

Published as a conference paper at ICLR 2026
Classification IN1K
Segmentation ADE2OK
Depth Estimation NYU DepthV2
Figure 1: Concept importance across tasks. UMAP projection of the learned dictionary, with colors indicating the relative magnitude of each concept's contribution to three downstream tasks: (Left) classification (ImageNet- Ik), (Middle) segmentation, (Right) depth estimation. While classification recruits broad set of concepts, segmentation and depth primarily activate more restricted set of concepts. Although UMAP only preserve local geometry, functionally relevant groupings are visibly clustered in the projection. We show in later sections that different tasks consistently recruit distinct, low-dimensional regions of the concept space.
INTRODUCTION
Vision Transformers (ViT') Dosovitskiy et al. (2020) recast images as sequences of patch tokens that are linearly embedded and processed by stacked self-attention and feedforward layers Vaswani al. (2017), replacing the convolutional inductive bias found in earlier architectures LeCun et al. (2015); Serre (2006). This design scales reliably and flexibly Zhai al. (2022); Dehghani al. (2023); Alabdulmohsin et al. (2023), and when trained with contrastive Zhai et al. (2023), masked He et al. (2022), self-distillatio objectives Touvron et al. (2022); Caron et al. (2021), it learns rich, semantically organized representations without labels.
The case of DINOv2. DINOv2, trained self-supervised on massive unlabeled data, exhibits strong emergent representations Oquab et al. (2023); Darcet al. (2023) that support fine-grained classifica- tion Chiu et al. (2024), segmentation Liu et al. (2023), monocular depth Mao et al. (2024); Cui et al. (2024), tracking Faber al. (2024); Tumanyan al. (2024), and robotic perception Kim et al. (2024). Beyond discrimination, its embeddings serve as priors for generative models Yu et al. (2025), proxies for distributional similarity Stein et al. (2023), and tools to reveal blindspots Bohacek al. (2025). They are robust and transferable, powering video world models Baldassarre et al. (2025). aligning with language for open-vocabulary segmentation and zero-shot classification Jose al. (2025), and adapting across domains from satellite imagery Waldmann et al. (2025) to medical scans Ayzenberg et al. (2024); Zhang et al. (2023); Zhou et al. (2024). Yet the internal organization remains unclear: what is encoded, which features are available Hermann et al. (2023) to downstream tasks, and how are they geometrically arranged?
Vision Explainability. To address these questions, we draw on vision explainability, a field that has developed both an empirical toolkit and theoretical accounts for probing large vision mod- els Doshi- Velez & Kim (2017); Gilpin et al. (2018), 8); Fel (2024). Early work centered on attribution, answering "where" model looks Simonyan al. (2013); Zeiler & Fergus (2014); Bach et al. (2015); Springenberg et al. (2014): Smilkoy et al. (2017); Sundararajan et al. (2017); Selvaraju et al. (2017); Fong & Vedaldi (2017); Fel al. (2021); Novello et al. (2022); Muzellec et al. (2024). but these often give surface-level insight and can mislead Hase & Bansal (2020); Hsieh er al. (2021); Nguyen al. (2021); Colin al. (2021); Kim al. (2022); Sixt al. (2020). To move beyond attribution, concept-based based methods have been proposed to extract interpretable latent dimensions Kim et al. (2018); Poeta et (2023); Bau et al. (2017); Ghorbani al. (2019): Zhang et al. (2021), Fel et al. (2023c); Graziani et al. (2023); Vielhaben et al. (2023); Kowal et al. (2024a;b); Fel et al. (2023b). A natural first attempt was to identify individual neurons as carriers of concepts Cammarata et al. (2020). However, neuron-centric accounts struggle with polysemy, basis dependence, and capacity limits Arora et al. (2018); Elhage et al. (2022); Gorton (2024). These limitations motivated methods that identify concepts as distributed features, without requiring alignment to single neurons Ghorbani et al. (2017); Zhang et al. 12021);( Fel et al. (2023c). A first theoretical account soon followed in the form of phenomenology by Elhage et al. (2022): in d-dimensional space, neural networks can encode
2

---

Published as a conference paper at ICLR 2026
exponentially many nearly orthogonal features by representing each as a sparse combination of neurons. This phenomenology crystallized into the Linear Representation Hypothesis (LRH). which holds that models contain many more features than neurons, arranged as sparse, quasi-orthogonal directions Park et al. (2024). If the LRH is valid, then concept extraction amounts to an overcomplete dictionary learning problem Fel et al. (2023b). In this framework, the activation space of layer is factorized into a dictionary basis and sparse codes, Sparse autoencoders (SAEs) Makhzani & Frey (2014); Cunningham et al. (2023); Bricken al. (2023) are one practical instantiation of this idea, enforcing sparsity through a simple encoder-decoder architecture. When applied to pretrained models such as DINOv2, SAEs uncover rich library of patterns, which we refer to as concepts.
Our contributions. We operationalize LRH in DINOv2 with a stable SAE and extract a dictionary of 32,000 concepts, released as (to our knowledge) the largest interactive interpretability demo for a vision foundation model (which will be publicly released upon acceptance). On this basis, our study proceeds in three parts:
Downstream usage. We quantify how tasks recruit the dictionary and find clear specialization: (i) classification repeatedly uses "Elsewhere" concepts that implement learned negation; (ii) seg- mentation concentrates on border detectors forming coherent subspaces; (iii) depth estimation draws on three families of monocular cues (projective, shadow-based, frequency transitions). These task-aligned subsets are low-dimensional and only weakly overlap.
Concept geometry and statistics. Several diagnostics are compatible with linear sparse-coding view: atoms are distributed rather than neuron- seuron-aligned (low Hoyer scores), and we observe antipodal pairs forming signed semantic axes (so cos 0 = -1 often indicates opposite poles of one feature, not unrelated ones). At the same time, the dictionary departs from a mear-orthogonal/Grrssmmma picture: pairwise similarities have heavier tails, the singular-value spectrum of D decays sharply (anisotropic capacity), and concepts cluster along task-recruited directions. Moreover, positional features are dense yet low-norm, while per-image token clouds remain smooth and low-dimensional even after removing position. Taken together, these results challenge purely sparse, near-orthogonal "feature packing" account and point to additional geometric constraints.
Towards Minkowski geometry. Guided by these departures, we propose the Minkowski Representa- tion Hypothesis (MRH): tokens behave as sums of convex regions around archetypal landmarks (e.g.. animal/rabbit, color/brown, texture/fluffy), and concepts are expressed by proximity to landmarks rather than unbounded directions. We show that multi-head attention constructively realizes MRH (each head produces convex set; heads sum to Minkowski sum), and provide preliminary empirical signals consistent with this picture, along with practical implications for steering and structure-awan probes.
With that said, we begin by recalling the LRH and how it can be operationalized through sparse autoencoders, which provide the dictionary of concepts underpinning our analysis.
LINEAR REPRESENTATION HYPOTHESIS AND OPERATIONALIZZATION
recurring phenomenology of large models is that their representational capacity vastly exceeds the number of neurons: in a d-dimensional space, they encode exponentially many features by representing each as a sparse linear superposition Arora et al. (2018); Elhage et al. (2022). Empirically, such features behave as nearly orthogonal directions Papyan al. (2020), active only in restricted contexts, while neurons themselves are polysemantic Nguyen et al. (2016). The geometrical structure that minimize interference on e vectors in dimensions called Grassmannian frame Strohmer & Heath Jr (2003), and an activation space that we can describe using such object is said to satisfy the Linear Representation Hypothesis Elhage et al. (2022); Costa et al. (2025) (see Appendix for more details). This motivates two conclusions: (i) neurons are not the appropriate locus of interpretability. and (ii) one must recover the latent basis along which the model effectively operates.
In that view, the LRH is a useful hypothesis precisely because it can be operationalized. If activations admit such a sparse overcomplete representation, then concept discovery reduces to finding the appropriate overcomplete dictionary. While Sparse Autoencoders (SAEs) Cunningham et al. (2023); Bricken et al. (2023) have emerged as a popular choice to learn such a dictionary, they face a persistent challenge in stability: naive SAEs produce inconsistent features across runs, undermining interpretability Paulo & Belrose (2025); Papadimitriou et al. (2025). To address this, we adopt a stable SAE Fel et al. (2025), which constrains each dictionary atom to lie in the convex hull of real activations. This guarantees that atoms remain in-distribution and yields reproducible, geometrically faithful dictionaries. Formally, let (X, F, P) denote the probability space of natural images, A' C
3

---

Published as a conference paper at ICLR 2026
Elsewhere concepts
Border concepts
Rebbit
Tokens map the conceepte"
Attribution map wwottport of emgge othen
Figure 2: "Elsewhere" and border concepts. (Left) In classification, recurring "Elsewhere" concept fires off-object yet depends on the object's presence, disappearing when the object removed (via causal masking Petsiuk et al. (2018)), providing evidence suggestive of a causal effect realizing conditional negation (another interpretation being distributed off-object evidence). (Right) In segmentation, top concepts localize along object boundaries, showing consistent spatial patterns and similarity embedding space, suggesting a shared functional role on a low-dimensional submanifold. See Appendix D for more details and examples.
R x3 For a pretrained Vision Transformer f : - R any image æ ~ P yields activations f(x) Rexd i.c. t token embeddings dimension For batch X (a;))==>> concatenating all tokens gives A E Rnixd, Our objective is to factorize A into sparse codes Z E RB and dictionary D E Rexd with
min
| - 2D subject to Z 2 0, |Z < k, DE conv(A). 2.D
Here, 2 0 denotes an elementwise non-negativity constraint, i.e. entries Z are constrained to be nonnegative. Concretely, we used DINOv2-B with 4 registers as f. with d = 768, 261. We set 32,000 atoms and k = 8 active codes per token, approximate conv(A) by 128,000 centroids extracted via k-means over 1.4M ImageNet- IK images (with augmentation), and parametrize D SC with S row-stochastic. Codes Z are produced via single-layer encoder with BatchTopK projection Bussmann al. (2024); Hindupur et al. (2025). Training with Adam for 50 epochs yields R² 88% reconstruction fidelity, consistent with prior stability results.
Interpretation of Z and D. Essentially, the matrix Z encodes the statistical structure of the activation space capturing which concepts are active, how frequently, and to what degree. In contrast, the dictionary D encodes the geometric structure defining the atomic directions used to span the space and organize the model's internal feature basis. This decomposition yields a library of 32,000 concept atoms, each interpretable as linear probe on DINOv2 activations. With this basis in hand, first investigate which concepts recruited by different downstream tasks (Section 3), then discuss their statistical and geometric organization in detail (Section 4).
APECCICCC UTILIZATION OF LEARNED CONCEPTS
With the concept dictionary in place, we now ask: which concepts are actually recruited by down- stream tasks? To answer this, we express linear probes in the concept basis, allowing us to quantify how strongly each concept contributes to a given output. We use this alignment score to compare tasks, while the precise definition and its theoretical justification are deferred to Appendix C.I.
Different tasks recruit different concepts. Figure reveals that different tasks recruit different subsets of concepts, often with minimal overlap. Classification activates wide and dispersed array of concepts, while in contrast, segmentation and depth estimation draw on more compact and localized regions of the concept manifold. This may suggest functional regions in the latent space, where concepts are reused non- con-uniformly across tasks. Quantitatively, we confirm this asymmetry in Figure 11 (Left): classification draws from broader span of the dictionary than segmentation or depth estimation. In fact, we can show that the recruited concepts seem to bear geometric resemblance. We isolate the top 100 most task-aligned concepts per-head and analyze their pairwise similarities. As shown in Figure 11 (Middle), intra-task concepts are significantly more aligned with another compared to randomly selected concepts, breaking the confirm this observation by comparing the eigenvalue spectrum of each
observed globally. Finally,
in Figure 11 (Right), task's 's sub-dictionary. All three spectra decay much faster than those of random subsets of concepts, indicating that spec-fp concepts form low-dimensional subspace.
Together, these findings suggest that perceptual tasks selectively activate low-dimensional, function- ally specialized subspaces within the broader concept representation space of DINO, But what do
4

---

Published as a conference paper at ICLR 2026
Orignal
Locol Meciin
Soges
Wirsing
Local Frequency
Transition Dooth
Mautan
co
Hight delf grdpes Harrying
UMAP Perturbation results
Projective Depth
as
0.5
Local Mediar
Rlle
Seof rogeu
stats
plrping
Shadow Depth
10
0.5
Local Mediam
tlar High sanse Suft Eniire Warping
Figure 3: DINO encodes diverse monocular depth cues. Visualization of key concepts used in monocular depth estimation tasks. We identify three dominant types: projective geometry cues (e.g.. vanishing lines, converging structures), shadow-based cues (e.g., soft lighting gradients and cast shadows), and frequency-basee -based cues (e.g., transitions between high- and low-texture regions). These findings suggest that DINO learns rich basis of 3D perception primitives from 2D data alone.
these task-specific subspaces actually look like? Can we identify recurring families of concepts that characterize each task? To this end, we now turn to a qualitative examination of these patterns.
Classification and the Elsewhere Concepts. Across many ImageNet classes, the top concepts for classification include not only objects or parts, but also "Elsewhere" concept (Figure 2, left). These activate broadly across tokens but are suppressed on the object itself, appearing only in surrounding regions. Crucially, they not generic background detectors: they vanish if the object is removed, indicating a conditional negation "the object exists elsewhere, but this token is not the object". Such disjoint activations may support classification by outlining boundaries, encoding contrast, or distributing evidence, but they can also mislead attribution maps, which assume concepts are tied to the tokens where they fire. This calls for interpretability tools that combine localization with causal perturbation Shaham et al. (2024). Having explored classification, we now turn to segmentation.
Segmentation and Border Concepts. For segmentation, on the ADE20k dataset Zhou et al. (2017), we observe that all the concepts among the top-50 consistently localize along object contours or spatial boundaries. As shown in Figure 2 (right), and expanded in Figure 10, these "border concepts" activate narrowly along the periphery of objects (highlighting limbs, outlines, or silhouette transitions). Remarkably, while the precise visual features vary across classes (e.g., animal ears, tower edges, or sea urchin spines), the spatial footprints of these concepts remain strikingly consistent. Furthermore, in the concept embedding space, these border concepts form a visibly tight cluster (Figure 10, right), suggesting that DINO allocates a dedicated region of its representational geometry to encoding object boundaries. As quantitatively shown in Figure 11, their absolute cosine similarity is higher than average and their eigenspectrum decays faster than a random subset of concepts, suggesting low-dimensional structure composed of boundary detectors. Segmentation concepts reveal that DINO dedicates portions of its concept space to encoding local spatial structure. We now examine depth estimation, another spatially grounded task, but one that requires global 3D understanding rather than contour localization.
Depth and Monocular Cue Concepts. The depth head is trained on NYU depth Nathan Silberman & Fergus (2012) following the original procedure described DINO (with depth value binning to 256 classes and linear normalization, see paper for details). Despite no explicit 3D supervision, DINO shows strong performance on monocular depth tasks Mao et al. (2024); El Banani et al. (2024); Zhan et al. (2024). To understand the internal features supporting this, apply controlled perturbations that isolate specific monocular cues e.g., median blurring (removes shadows), edge- preserving smoothing (preserves contours), and high-pass filtering (emphasizes projective geometry). We measure concept activation changes and visualize them via UMAP (Figure 3). revealing three functional clusters: (i) projective geometry (e.g., vanishing lines), (ii) shadow-bar (e.g., soft
5

---

Published as a conference paper at ICLR 2026
Spectrum of Z
E: &: 7 (Condioonal
94
0.5
1.0
188
[|2]]
Fire!
500
Components
430
81
14000
Hoyer Score oE
43.5
200
400
G23
630
03 00
Destmbuted
Figure 4:
Concept statistics and geometry. (Top Left) Conditional energy E(Z, Z. > 0)
vs. firing count: most concepts follow sparse trade-off; a few outliers fire densely with low energy, encoding persistent positional features Sun et al. (2025). (Top Right) Spectrum of Z Z shows smooth decay, inconsistent with low-rank modularity but broader than shuffled baselines (Appendix E). (Bottom) Dictionary D shows anisotropic structure: (A) heavier-tailed inner products than random//rrasmamninn baselines using TAAP algorithm Massion & Massart (2025) (Appendix F); (B) sharp singular value decay; (C) low Hoyer scores confirm distributed (non-ncuron-alignel atoms; (D) UMAP reveals no strong modularity. Overall, DINO's concept space is high-dimensional, distributed, and weakly clustered.
gradients), and (ii) local frequency transitions Schubert et al. (2021); Ding al. (2025), echoing the bokeh concepts of Fel et al. (2024). Some concepts mix cues, as shown in full perturbation maps (Figure 9). These clusters suggest that DINO internally encodes a diverse, interpretable set of monocular depth cues emerging without labels-anne accessible through linear readouts. For more detail see Appendix C.3.
STASTISTICS AND GEOMETRY OF CONCEPTS
Concept Occurrence Statistics. We next examine the statistical profile of concepts. Figure 4 (left) plots conditional energy E Z; Z; 0) against firing count. Most concepts follow a triangular envelope: rare ones fire strongly, frequent ones weakly, revealing distinct norm regimes where some features dominate the activation norm while others remain minor. Three outliers break this pattern, showing dense Sun et al. (2025) activations that correspond to positional signals (investigated deeper in Appendix and Section 5). Thus the space is not strictly sparse: a few universally active, spatially grounded features cocxist with a larger ensemble of selective concepts, echoing hybrid sparse-dense regimes Jiang& & Lai (2014); Pramanik et al. (2020). Concepts Co-occurence, The Gram matrix of concept co-activation G = ZTZ Rexc as shown in Figure 4 (right), has spectrum that decays smoothly, with no gaps or dominant modes, providing little evidence for modular low-rank structure (concept co-activate without forming clear clusters). Baseline details are provided in Appendix E. Geometric Organization. Figure 4 summarizes properties of D. Relative to random and Grassmannian baselines, coherence (A) higher: most atoms are near orthogonal, with small tight clusters, departing from the LRH and suggesting structured redundancy and reuse. The singular value spectrum (B) decays sharply, indicating anisotropic capacity allocation that is consistent with anisotropy in A and local warping from normalization layers Peyré (2009). Hoyer scores (C-D) are far from the one-hot bound, so atoms are distributed rather than neuron aligned Elhage et al. (2022); Colin et al. (2024). Additional motifs such antipodal pairs are provided in Appendix G. Concept geometry is only weakly shaped by co-activation. One hypothesis is that concepts used together should lie nearby in geometry, with co-occurrence "bending" the dictionary into clusters. To test this, we compare the co-activation matrix Z Z with the geometric affinity DD As shown in Figure 13 (middle), they correlate only weakly, suggesting that usage influences geometry but is not

---

Published as a conference paper at ICLR 2026
Figure 5: PCA maps of DINO patch embeddings reveal smooth, semantically aligned structure. (Top) Original images. (Bottom) PCA heatmaps of the top three components of patch-token embeddings encoded as (r,g.b) values. Despite no supervision for localization or segmentation, the PCA projections consistently delineate object shapes with smooth transitions across the image grid. Since PCA is a linear operator, it cannot fabricate curvature; the observed smoothness therefore reflects genuine structure in the embeddings. Moreover, the alignment of token geometry across distinct similar images suggest that the representation is not purely relative (i.e., based only on distances within each image tokens).
the dominant organizing principle.' Additional evidence comes from UMAP embeddings (Figure 13 right), where top co-activations form tangled, non-local graphs rather than modular clusters.
Taken together, the dictionary is neither maximally incoherent nor uniform: high coherence, sharp spectral decay, task-aligned clusters (See. 3), and dense positional signals sit uneasily with purely sparse, oethooottogonal view. These effects are hard to attribute to simple feature packing and point to additional geometrie constraints. We therefore step beyond the SAE lens and examine the model native token geometry within single images.
THE SHAPE OF AN IMAGE
Cmap for cach postion token P.
PCA Pacross layer
Layer
Layer
Layer
Layer
Layer
Laye
Layer 13
Layer
Figure 6: Visualization of positional encoding across layers reveals smooth compression to 2D sheet. PCA projections of the positional encoding vectors at different layers show clear evolution: from high-rank, dispersed geometry in early layers (Figure 24) smooth, low-dimensional sheet in the final layers.
We now study the token geometry within a single image using per-image PCA maps, standard diagnostic tool that has been employed for feature distillation Oquab et al. (2023); Darcet et al. (2025); Kouzelis et al. (2025), as well as a popular method for showcasing DINO's 's learned features. In Figure 5, patch embeddings organize along smooth trajectories connecting small set of extremal points. We begin by testing whether positional information alone explains this structure.
Just position? To extract position each layer, we trained linear decoders to predict token coordinates Islam et al. (2024) and formed positional basis P E 256x Early layers permit near-perfect decoding with high rank, but the positional subspace compresses sharply and approaches
Algebraically, the correlation is roughly proportional to the trace of activation covariance corr( Z Z, DD") tr( 2 Z DDT) tr(D" Z Z D) & tr(ATA A) cov(A), which is guaran- teed to he positive. This may be intrinsic property of linear reconstructive methods.
7

---

Published as a conference paper at ICLR 2026
Minkowski Representation Hypothesis
== Pj
Directional steering
Pj
Stocring strength
Archetypal steering
Pi
Steering strength
Figure 7: Minkowski Representation Hypothesis. Left: the activation set X' is a Minkowski sum of head polytopes P. = conv AT;; each head outputs a convex combination of its values and the outputs sum across heads. Right: steering comparison. Directional steering (pink) follows an unbounded vector and leaves the manifold; archetypal steering (purple) moves toward landmarks within polytopes with monotonic proximity.
two dimensions in the final layers (Fig. 24, top), consistent with shift from place-like to axis coding O'Keefe & Dostrovsky (1971); Chang & Tsao (2017); PCA of P itself confirms this collapse (Figure 6). In real images, positional directions typically appear only among intermediate components (around PCs 3-5; Fig. 24, bottom), so the leading PCs capture non-positional structure. Projecting tokens orthogonally
the positional subspace leaves the PCA organization largely unchanged (Figure 25). This indicates that PCA is capturing something beyond explicit position.
Toward interpolative geometry. Since position does not account for the structure observe, ask what does. In Figure 5, across many images, patch tokens lie on a consistent low-dimensional set aligned with objects, suggesting interpolation between a few landmark representations rather than purely relative coordinates. DINOv2 offers a concrete mechanism: DINO head and an iBOT head produce soft assignments over large prototype vocabularies (128k each), yielding mixtures of prototypes for each token Oquab et al. (2023); Zhou et al. (2022). The Sorachenko-Leoneee larizer further encourages spread in representation space Sablayrolles et al. (2019). A parsimonious view is that tokens are mixtures of small number of archetypal points drawn from several prototype systems, which yields piecewise smooth variation. In the next section, we propose to formalizes this landmark baasdd account and shows how attention naturally realizes it.
regu-
6 MINKOWSKI REPRESENTATION HYPOTHESIS
We conclude by proposing the Minkowski Representation Hypothesis: a geometry where token embeddings lie in Minkowski sums of convex polytopes spanned by archetypal landmarks. This view is motivated by our observations above and connects Gärdenfors conceptual spaces which concepts occupy convex regions Gärdenfors (2004), well evidence that concept structure can be convex and compositional Park al. (2025). Mechanistically, a single attention head outputs convex combination of its values, and multi-head attention these headwise convex sets, yielding a Minkowski sum. Different polytopes can capture factors such as position, depth, or category, with probes reading out proximity to the corresponding landmarks. We now formalize this hypothesis.
Definition 1. Minkowski Representation Hypothesis (MRH). Let X Rdd be layer's 's activation set and A = (a1,.... ac) E IR Rxed an overcomplete archetype set (c >> d) partitioned into tiles (m and define P; conv(AT, with AT, {a; j E T.). MRH holds (i) X =
+ Pmi (ii)
every x X admits a block-convex code Eies AT, z, with z; E (T and [S] < m; (ii) for
codes Z E guxe stacked over samples, the Gram G Z T Z exhibits blocks aligned with (T.)
Put simply, MRH says that a point & is a sparse composition of concept regions, with only a few tiles active |S| < m. and each active tile contributing a convex combination of its archetypes. For example, a token representing a rabbit might combine: (1) a convex mix from an "animal category" tile (capturing rabbit-like features), (2) convex mix from "spatial position" tile (left/center/right),
8

---

Published as a conference paper at ICLR 2026
and (3) convex mix from "depth" tile (forgground/backrround)) The final activation is the Minkowski sum (+) of these convex contributions.
We now connect the definition evidence, with full statements and proofs deferred to Appendix K.
Theoretical account. First, we show that a single attention head outputs convex combinations of its values, so its attainable set lies in conv(V) and, under mild reachability conditions, coincides with it (Lemma 1); moreover, affine maps preserve these convex codes (Lemma 2). For multiple head, we naturally find MRH block-convex structure:
Proposition 1 (Multi-head attention realizes MRH). Let there be H heads with value sets Va. and per-head output projections wh) For any input, head h produces weights on E Am and output (8)
yh = Ei O E convv(V)) After projection and summation,
H (h) O
H -Σ h=1
(M) (1) 2% E E W (
yh
On
h=1
Thus every output admits an MRH representation with clove-convex codes z = (a1)..... O()) and archetypes VV.'....... w! VH). in addition, each head can realize any point of relint(A (* up to the softmax additive constant, then the attainable set is exactly the Minkowski sum.
Thus, tokens admit block-convex codes over head polytopes and realize MRH with output projections. This mechanism is elementary and aligns with convex partitioning and low-dimensional population geometry in deep networks Montúfar et al. (2014); Raghu al. (2017); Balestriero et al. (2018); Tvetkova et al. (2025); Tankala et al. (2023); Hindupur et al. (2025); Chung (2021); Cohen et al. (2020), as well as convex conceptual spaces Gärdenfors (2004); Park et (2025).
Empirical evidences. Next, we test the three MRH criteria on ImageNet- Ik tokens. We first compare straight-line interpolation to piecewise-linear k--N geodesics and find that only the latter remain near the data support (Fig. 26, left), consistent with paths that traverse faces of polytopes. Then we compare Archetypal Analysis (AA) Cutler & Breiman (1994) (the |S| = case) to an SAE and observe that with about ten archetypes per image AA matches exceeds SAE reconstruction (Fig. 26, middle), indicating concentration on low-dimensional polytopes Bárány & Füredi (1 (1988); Balestriero et al. (2021). Finally, we examine the Grams of the codes and observe clear block structure (Fig. 26, right), suggesting tiles effect: group of concepts working together under convex constraint K.
Implications for Interpretability. If, and this an assumption, the Minkowski Representation Hypothesis holds, three immediate implications follow. (i) Concepts are points and regions, not directions. Under an archetypal view, concept is landmark (an extremal point of a convex cell) or a small constellation of landmarks whose hull defines a coherent region. This departs from the linear-factor picture where evidence is the magnitude of an inner product with a preferred direction. (ii) Steering admits strict maximum. In landmark-based geometry, probing moves activation toward a specific point or cell; once the landmark (or its convex neighborhood) is reached, the signal saturates and further movement drives the embedding aff-manifodd. This bounded trajectory helps explain why the gain of SAE-style steering plateaus Wu et al. (2024); Mueller et al. (2025); Karvonen al. (2025) or reverse Hedström et al. (2025); Templeton al. (2024) when scaling pushed too far. Practical probes should therefore estimate proximity to the landmark set (e.g., barycentric weights or geodesic distance within the cell) and stop at convergence rather than extrapolating indefinitely. (iii) Decomposition is non-identifiable from final activations. From observations of X' alone, recovering the generating factors (the head polytopes) is ill- ill-posed.
Proposition 2 (Non-identifiabiiity of Minkowski decomposition). Let X = Minkowski sum of convex polytopes. Given only samples from X. the decomposition (P.)m generally non-unique: there exist distinet collections {0,1 with X = $)_1 2, In particular, even simple polytopes admit infinitely many decompositions as sums of line segments (zonotope generators) with varying directions and lengths.
==-= S be
See Appendix K.6 and Smilansky (1987) for details. The limitation follows from additivity of support functions under Minkowski addition, h ( = Σ, hy, (u), which admits many valid summands. Practically, this means that estimating individual concept contributions or polytopes P from final

---

Published as a conference paper at ICLR 2026
activations alone is underdetermined: exploiting intermediate signals (attention weights, per-head outputs) and architectural structure may render the factorization tractable and guides the design of structure-aware interpretability tools.
DISCUSSION
We trained a RA-SAE on DINOv2 and released a 32k-concept dictionary interactive visualization, showcased in Appendix I.. We used this dictionary to quantify how downstream tasks recruit features. We discovered clear specialization for different tasks: "Elsewhere" concepts for classification via learned negation, border concepts for segmentation, and three families of monocular depth cues, Examining concept statistics and geometry, we found the dictionary more coherent than random and Grassmannian baselines, departing from the LRH idealized assumption. We observed dense but low-norm positional concepts suggesting embeddings contain both sparse and dense components. Per-image token clouds showed smooth structure that positional information alone cannot explain. These observations challenged purely sparse, near-orthogonal accounts and led us to propose the Minkowski Representation Hypothesis: tokens behave as sums of convex regions around archetypal landmarks; geometry realizable by multi-head attention through headwise convex combinations. We provide theoretical justification and preliminary empirical evidence, with implications showing that if true, extracting concepts from single layers is insufficient- proper decomposition requires signals from the entire network structure.
While focused single architecture, these findings have immediate relevance for the substantial body of research and applications that depend on DINOv2 representations.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Ibrahim M Alabdulmobsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- attribution methods for deep neural networks. Proceedings of the International Conference on Learning Representations (ICLR), 2018.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 2018.
Lev Ayzenberg, Raja Giryes, and Hayit Greenspan. Dinov2-based self supervised learning for few-shot medical image segmentation. ArXiv e-print, 2024.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. Public Library of Science (PloS One), 2015.
Federico Baldassarre, Mare Szafraniec, Basile Terver, Vasil Khalidov, Francisco Massa, Yann LeCun, Patrick Labatut, Maximilian Seitzer, and Piotr Bojanowski. Back to the features: Dino as foundation for video world models. ArXiv e-print, 2025.
Randall Balestriero, Jerome Pesenti, and Yann LeCun. Learning in high dimension always amounts to extrapolation. ArXiv e-print, 2021.
Randall Balestriero et al. A spline theory of deep learning. Proceedings of the International Conference Machine Learning (ICML), 2018
Imre Bárány and Zoltán Füredi. On the shape of the convex hull of random points. Probability theory and related fields, 1988.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara, Beren Millidge, Gabriel Alfour, Interpreting neural networks through the polytope lens. ArXiv e-print, 2022.
Matyas Bohacek, Thomas Fel, Maneesh Agrawala, and Ekdeep Singh Lubana. Uncovering conceptual blindspots in generative image models using sparse autoencoders. ArXiv e-print, 2025.
Mark Bowick, Angelo Cacciuto, David R Nelson, and Alex Travesset. Crystalline order sphere and the generalized thomson problem. Physical Review Letters, 2002.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. ArXiv e-print, 2024.
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Anmand Joulin. Emerging properties in supe-vupe vision transformers. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.

---

Published as a conference paper at ICLR 2026
Le Chang and Doris Y Tsao. The code for facial identity in the primate brain. Cell, 2017.
Christopher Chiu, Maximilian Heil, Teresa Kim, and Anthony Miyaguchi. Fine-grained classification for poisonous fungi identification with transfer learning. ArXiv e-print. 2024.
Yuo Chung. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion in neurobiology, 2021.
Uri Cohen, Sue Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 2020.
Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What cannot predict, i do not understand: A human-centered evaluation framework for explainability methods. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Julien Colin, Lore Goetschalckx, Thomas Fel, Victor Boutin, Jay Gopal, Thomas Serre, and Nuria Oliver. Local vs distributed representations: What is the right basis for interpretability? ArXiv e-print, 2024.
Valérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, and Demba Ba. From flat to hierarchical: Extracting sparse representations with matching pursuit. ArXiv e-print, 2025.
Beilei Cui, Mobarakol Islam, Long Bai, and Hongliang Ren. Surgical-dino: adapter learning of foundation models for depth estimation in endoscopic surgery. International Journal of Computer Assisted Radiology and Surgery, 2024.
Hoagy Cunningham, Aidan Ewart, Logan Riggs. Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. ArXiv e-print, 2023.
Adele Cutler and Leo Breiman. Archetypal analysis. Technometrics, 1994.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. ArXiv e-print, 2023.
Timothée Darcet, Federico Baldassarre, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Cluster and predict latent patches for improved masked image modeling. The Journal of Transactions on Machine Learning Research (TMLR), 2025.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. Proceedings of the International Conference Machine Learning (ICML), 2023.
Philippe Delsarte, Jean-Marie Goethals, and Johan Jacob Seidel. Spherical codes and designs. Geometry and Combinatorics, 1991.
Zhiwei Ding, Dat Tran, Kayla Ponder, Zhuokun Ding, Rachel Froebe, Lydia Ntanavara, Paul G. Fahey, Erick Cobos, Luca Baroni, Maria Diamantaki, Eric Y. Wang, Andersen Chang, Stelios Papadopoulos, Jiakun Fu, Taliah Muhammad, Christos Papadopoulos, Santiago Cadena, Alexan- dros Evangelou, Konstantin Willeke, Fabio Anselmi, Sophia Sanborn, Jan Antolik, Emmanouil Froudarakis, Saumil Patel, Edgar Y. Walker, Jacob Reimer, Fabian H. Sinz, Alexander S. Ecker, Katrin Franke, Xaq Pitkow, and Andreas S. Tolias. Bipartite invariance in mouse primary visual cortex. bioRxiv, 2025.
Finale Doshi- Delei-Velez and Been Kim. Towards rigorous science of interpretable machine learning. ArXiv e-print, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. Proceedings of the International Conference Learning Representations (ICIR), 2020.
12

---

Published as a conference paper at ICLR 2026
Mohamed EI Banani, Amit Raj, Kevis-Kokitsi Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2024.
Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, "ac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCan- dlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
Niels G Faber, Seyed Sahand Mohammadi Ziahari, and Fatemeh Karimi Nejadasl. Leveraging foundation models via knowledge distillation in multi-object tracking: Distilling dinov2 features to fairmot. ArXiv e-print, 2024.
Thomas Fel. Sparks of explainability; recent advancements in explaining large vision models. PhD thesis, Université de Toulouse, 2024.
Thomas Fel, Remi Cadene, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Thomas Fel, Thibaut Boissin, Victor Boutin, Agustin Picard, Paul Novello, Julien Colin, Drew Linsley, Tom Rousseau, Rémi Cadène, Laurent Gardes, and Thomas Serre. Unlocking feature visualization for deeper networks with magnitude constrained optimization. Advances Neural Information Processing Systems (NeurIPS), 2023a.
Thomas Fel, Victor Boutin, Mazda Moayeri, Remi Cadene, Louis Bethune, Mathieu Chalvidal, and Thomas Serre. A holistic approach to unifying automatic concept extraction and concept importance estimation. Advances Neural Information Processing Systems (NeurIPS), 2023b.
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2023c.
Thomas Fel, Louis Bethune, Andrew Kyle Lampinen, Thomas Serre, and Katherine Hermann. Un- derstanding visual feature reliance through the lens of complexity. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Papadim- itriou, Binxu Wang. Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal sac: Adaptive and stable dictionary learning for concept extraction in large vision models. Proceedings of the International Conference on Machine Learning (ICML), 2025.
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Peter Gärdenfors. Conceptual spaces: The geometry of thought. MIT press, 2004.
Richard J Gardner. Geometric tomography. Cambridge University Press Cambridge. 1995.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2017.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Leilani H. Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa. Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. Proceedings of the IEFE International Conference on data science and advanced analytics (DSAA), 2018.
Liv Gorton. The missing curve detectors of inceptionvl. 1: Applying sparse autoencoders to inceptionv early vision. ArXiv e-print, 2024.
13

---

Published as a conference paper at ICLR 2026
Mara Graziani, An-phi Nguyen, Laura O' O'Mahony, Henning Müller, and Vincent Andrearczyk. Concept discovery and dataset exploration with singular value decomposition. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Chris Hamblin, Thomas Fel, Srijani Saha, Talia Konkle, and George Alvarez. Feature accentuation: Revealing" what features respond to in natural images. ArXiv e-print, 2024.
Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? Proceedings of the Anmual Meeting of the Association for Computational Linguistics (ACL), 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Anna Hedström, Salim I. Amoukou, Tom Bewley, Saumitra Mishra, and Manuela Veloso. To steer or not to steer? mechanistic error reduction with abstention for language models. Proceedings of the International Conference on Machine Learning (ICML), 2025.
Katherine L Hermann, Hossein Mobahi, Thomas Fel, and Michael C Mozer. On the foundations of shortcut learning. Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Sai Sumedh R Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting assumptions: The duality between sparse autoencoders and concept geometry. ArXiv e-print, 2025.
Cheng-Yo Yu Hsich, Chih-Kuar Ych, Xuanqing Liu, Pradeep Ravikumar, Scungycon Kim, Sanjiv Kumar, and Cho-Jui Hsich. Evaluations and methods for cxplanation through robustness analysis. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Md Amirul Islam, Matthew Kowal, Sen Jia, Konstantinos G Derpanis, and Neil DB Bruce. Position, padding and predictions: A deeper look at position information in cnns. International Journal of Computer Vision, 2024.
Xudong Jiang and Jian Lai. Sparse and dense hybrid representation via dictionary decomposition for face recognition. IEEE transactions pattern analysis and machine intelligence, 2014.
Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, et al. Dinov2 meets text: A unified framework for image-and pixel-level vision-language alignment. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2025.
Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, al. Saebench: A comprehensive benchmark for sparse autoencoders language model interpretability. Proceedings of the International Conference Machine Learning (ICML), 2025.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). Proceedings of the International Conference on Machine Learning (ICML), 2018.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source lasionlanguage-action model. ArXiv e-print, 2024.
Sunnie S. Y. Kim, Nicole Meister, Vikram Ramaswamy, Ruth Fong, and Olga Russakovsky. HIVE: Evaluating the human interpretability of visual explanations. Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2022.
Theodoros Kouzelis, Efstathios Karypidis, Ioannis Kakogeorgiou, Spyros Gidaris, and Nikos Ko- modakis. Boosting generative image modeling via joint image-feature synthesis. ArXiv e-print, 2025.
14

---

Published as a conference paper at ICLR 2026
Matthew Kowal, Achal Dave, Rares Ambrus, Adrien Gaidon, Konstantinos G Derpanis, and Pavel Tokmakov. Understanding video transformers via universal concept discovery. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024a.
Matthew Kowal, Richard P Wildes, and Konstantinos G Derpanis. Visual concept connectome (vcc): Open world concept discovery and their interlayer connections in deep models. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024b.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-traine pre-training for open-set object detection. ArXiv e-print, 2023.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Yifan Mao, Jian Liu, and Xianming Liu. Stealing stable diffusion prior for robust monocular depth estimation. ArXiv e-print, 2024.
Bastien Massion and Estelle Massart. Grassmannian frame computation via accelerated alternating projections. In 2025 International Conference on Sampling Theory and Applications (Samp'TA), pp. 1-5. IEEF, 2025.
Guido Montúfar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2014.
Edith Mooers. Tammes's problem. University of Vermont, 1994.
Aaron Mueller, Atticus Geiger, Sarah Wiegreffe, Dana Arad, Iván Areuschin, Adam Belfki, Yik Siu Chan, Jaden Fiotto-Kaufman Tal Haklay, Michael Hanna, al. Mib: A mechanistic interpretability benchmark. Proceedings of the International Conference on Machine Learning (ICML), 2025.
Sabine Muzellec, Leo Andeol, Thomas Fel, Rufin VanRullen, and Thomas Serre. Gradient strikes back: How filtering out high frequencies improves explanations. Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2012.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. Visualization for Deep Learning workshop, Proceedings of the International Conference on Machine Learning (ICML), 2016.
Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems (NeurIPS). 2021.
Paul Novello, Thomas Fel, and David Vigouroux. Making sense of dependence: Efficient black-box explanations using dependence measure. Advances in Neural Information Processing Systems (NeurIPS), 2022.
J. O'eeee and J. Dostrovsky. The hippocampus as a spatial map. Preliminary evidence from unit activity in the freely-moving rat. Brain Research, 1971.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. ArXiv e-print, 2023.
Isabel Papadimitriou, Huangyuan Su, Thomas Fel. Sham Kakade, and Stephanie Gil. Interpreting the linear structure of Dision-language model embedding spaces. Proceedings of the Conference on Language Modeling (COLM), 2025.
15

---

Published as a conference paper at ICLR 2026
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 2020.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. Proceedings of the International Conference on Machine Learning (ICML), 2024.
Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical concepts in large language models. Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Gonçalo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different
features. ArXiv e-print, 2025.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. Proceedings of the British Machine Vision Conference (BMVC), 2018.
Gabriel Peyré. Manifold models for signals and images. Computer vision and image understanding, 2009.
Elconora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitelli, and Elena Baralis. Concept- based explainable artificial intelligence: A survey. ArXiv e-print, 2023.
Aniket Pramanik, Hemant Kumar Aggarwal, and Mathews Jacob. Deep generalization structured low-rank algorithms (deep-slr). IEEE transactions on medical imaging, 2020.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl-Dickstein. On the expressive power of deep neural networks. Proceedings of the International Conference on Machine Learning (ICML), 2017.
Ralph Tyrell Rockafellar. Convex analysis. Princeton university press, 1970.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Spreading vectors for similarity search. Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Ludwig Schubert, Chelsea Voss, Nick Cammarata, Gabriel Goh, and Chris Olah. High-low frequency detectors. Distill, 2021.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Thomas Serre. Learning a dictionary of shape-components in visual cortex: Comparison with neurons, humans and machines. PhD thesis, Massachusetts Institute of Technology, 2006.
Tamar Rot Shaham, Sarah Schwettmann, Franklin Wang, Achyuta Rajaram, Evan Hernandez, Jacob Andreas, and Antonio Torralba. A multimodal automated interpretability agent. Proceedings of the International Conference on Machine Learning (ICML), 2024.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Leariing important features through propagating activation differences. Proceedings of the International Conference on Machine Learning (ICML), 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. Proceedings of the International Conference on Learning Representations (ICLR), 2013.
Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. Proceedings of the International Conference on Machine Learning (ICML), 2020.
Zeev Smilansky. Decomposability of polytopes and polyhedra. Geometriae dedicata, 1987.
16

---

Published as a conference paper at ICLR 2026
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. Proceedings of the International Conference on Machine Learning (ICML), 2017.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. Workshop Proceedings of the International Conference on Learning Representations (ICLR), 2014.
George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan L. Ross, Valentin Villecroze, Zhaoyan Liu, Anthony Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Thomas Strohmer and Robert W Heath Grassmannian frames with applications to coding and communication. Applied and compulational hurmonic analysis, 2003.
Xiaoqing Sun, Alessandro Stolfo, Joshua Engels, Ben Wu, Senthooran Rajamanoharan, Mrinmaya Sachan, and Max Tegmark. Dense sae latents are features, not bugs. ArXiv e-print, 2025.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. Proceed- ings of the International Conference on Machine Learning (ICML), 2017.
Pranay Tankala, Abiy Tasissa, James M. Murphy, and Demba Ba. K-deep simplex: Deep manifold learning via local dictionaries. IEEE Transactions Signal Processing, 2023.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey. Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Transformer Circuits Thread, 2024.
Joseph John Thomson. On the structure of the atom: investigation of the stability and periods of oscillation of number of corpuscles arranged at equal intervals around the circumference of circle; with application of the results the theory of atomic structure. Philosophical Magazine and Journal of Science, 1904.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2022.
Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self- supervised point tracking in a single video. Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2024.
Lenka Tvetkova, Thea Bruesch, Teresa Dorszewski, Fabian Martin Mager, Rasmus Ortoft Aagaard, Jonathan Foldager, Tommy Sonne Alstrom, and Lars Kai Hansen. On convex decision regions in deep network representations. Nature Communications, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Johanna Vielhaben, Stefan Blücher, and Nils Strodthoff.
concept discovery (med): A unifying framework with completeness guarantees. The Journal of Transactions Machine Learning Research (TMLR), 2023.
Leonard Waldmann, Ando Shah, Yi Wang, Nils Lehmann, Adam J. Stewart, Zhitong Xiong, Xiao Xi- ang Zhu, Stefan Bauer, and John Chuang. Panopticon: Advancing any-senso foundation models for earth observation. ArXiv e-print, 2025.
Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models. Advances in Neural Information Processing Systems (NeurIPS), 2024.
17

---

Published as a conference paper at ICLR 2026
Chih- Kuan Yeh, Cheng- Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikumar. On the (in)fidelity and sensitivity for explanations. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. Proceed- ings of the IEEE European Conference on Computer Vision (ECCV), 2014.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2023.
Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. general protocol to prohe large vision models for 3d physical understanding. Advances in Neural Information Processing Systems, 2024.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa P. Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems (NeurlPS), 2023.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept- based explanations for enn models with non-negative concept activation vectors. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017
Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning. ArXiv e-print, 2024.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. Proceedings of the International Conference on Learning Representations (ICLR), 2022.
18

---

Published as a conference paper at ICLR 2026
A TABLE OF CONTENTS
B Linear Representation Hypothesis
20
B.1 From LRH to Dictionary Learning.
20
C Task Specific Concept
21
C.1 Importance Measure for Concept-Tate Aligmment
C.2 On the "Elsewhere" Concepts
21
C.3 Monocular Depth Estimation
................................
23
Tasks Concept form low-dimensional subspaces
25
E Baseline Models for Co-Activation Spectrum Analysis
26
Baseline Models for Concept Geometry Analysis
27
Additional Geometric Diagnostics
28
H Token-Type-Speecfii Concept
29
H.I Footprint Analysis
29
H.2 Token-specific Concepts.
30
1 Qualitative Visualization Local Geometry
32
Position Embedding Analysis
33
J.1 Positional Basis Extraction.
33
J.2 Structure Persists After Position Removal
K
Minkowski Representation Hypothesis
39
K.1 Background and Related Work on Convex/Polytopal Views
39 Viw.............................
K.2 From a single head to convex polytopes
40
K.3
Robustness transformations
K.4
From multiple heads to Minkowski sums
41
K.5
Attention Concentration Effects
42
K.6
Non-identifiability of Minkowski decomposition.
43
K.7
Empirical evidence
44
Visualization tool
46
19

---

Published as a conference paper at ICLR 2026
B LINEAR REPRESENTATION HYPOTHESIS
We recall here, briefly, some aspect and motivation behind the Linear Representation Hypothesis (LRH).
Beyond empirical characterization, the geometry suggested by LRH can be motivated by compression: when features are arranged with minimal coherence, it maximizes the number of linearly retrievable features while minimizing destructive interference, kind of "feature packing".
This principle extends beyond neural activations to fundamental problems in discrete geometry and optimization theory. It connects to classical sphere packing problems: Tammes's ' problem Mooers (1994), which seeks optimal angular separation of points sphere (originally motivated by pollen grain morphology, where surface protrusions must be optimally spaced for acrodynamic dispersal efficiency); Thomson's problem Thomson (1904); Bowick al. (2002), which minimizes Coulomb electrostatic repulsion energy between charged particles constrained to sphere; and the spherical code problem Delsarte et al. (1991), which maximizes minimum distance between codewords on the unit sphere for eгror correction.
These problems share common mathematical substrate: the geometric structure formalized as Grassmannian frames in signal processing theory Strohmer & Heath Jr (2003). This object underlies the optimization principle of minimizing mutual coherence (the maximum absolute inner product between distinct normalized vectors). These mathematical convergences suggest that neural networks may naturally approximate such optimal geometric configurations to (i) maximize representational capacity while (ii) minimizing cross- cross-feature interference. This geometric phenomenology crystallizes into our LRH Elhage al. (2022); Costa et al. (2025) that we formally state here:
Definition 2. Linear Representation Hypothesis (LRH). A representation a E Bd satisfies the linear representation hypothesis if there exists a spursity constant k, an overcomplete dictionary
D= (di....,e dr) E Raxe, and a coefficient vector E RC such that Dzz under the following conditions:
Overcompleteness:
< d,
i Quasi-ortoogonaliyy
D) max dd ||d/22 1,
(ii) Sparsity:
supp(=)]$$k<
B.1 FROM LRH TO DICTIONARY LEARNING.
The LRH induces natural inverse problem: recover the latent basis on which activations are sparsely expressed. Concretely, given activations A E 1 (we adopt the row-atom convention D Red used in the main text so that A ZD with E mxx)) concept extraction becomes Nectionary-learnine problem with method-specific constraints on Z and D:
(Z*, Dz* arg min || 2.D
ZDTIP2
Vi, Di E {er fe,,...... ex}.
(ACE K-Means), (ICE PCA), (CRAFT -RAFT-NMF),
D D I, 2 0, D20
s.t.
ž= = (A),, K, (SAEs).
Here I is the identity, e, denotes canonical basis vector, and Vo an encoder and a sparsity projec- tion (e.g., TopK, Jump-ReL.U, or simply ReL.U) producing sparse codes. This formulation unifies the previous clustering-basee concept extraction Ghorbani et al. (2019), orthogonal factorization (PCA/ICE) Zhang et al. (2021), nonnegative concept extraction (CRAFT/NMF) Fel et al. (2023c), and modern SAEs Cunningham et al. (2023); Bricken et al. (2023). In practice. these approaches trade off fidelity A ZD [[)) against sparsity (e.g., || llo,, yielding Pareto frontier; SAEs are attractive at scale because the encoder V enables amortized, batched inference while retaining LRH's sparse, overcomplete structure.
20

---

Published as a conference paper at ICLR 2026
C TASK SPECIFIC CONCEPT
In this section, we review additional results and observation on the differents task-specific concepts discussed in Section 3. We will start by giving details the theoretical root of the importance measure, then we will briefly expand on the "Elsewhere" concept before delving into the monocular depth estimation.
C.I IMPORTANCE MEASURE FOR CONCEPT-TASK ALIGNMENT
In the Section 3, we ask which concepts in the dictionary are actually recruited by downstream tasks. We describe here precisely the importance measure we used, which has some appealing property as the linear probe allow us to directly interpret the importance for any linear probe as a linear combination of concepts.
Let A E IR xd denote token activations (over n images and tokens per image). We factor A = ZD with codes & nkn and dictionary D E TR For linear probe with weights W E Rexd and predictions Y AWT E Ti substituting the factorization gives
Y W Z (DW))
W'E
The matrix W encodes the alignment between dictionary concepts (rows) and task outputs (columns). We define the concept-importance vector for the probe as the expected concept activation weighted by this alignment:
E Z W IR"
where the expectation is taken over tokens (and samples) in the evaluation set. Class-wise scores correspond to the components of d;; concept-wise scores be read from EZ)) together with the corresponding rows of W'
In linear regimes, this coincides with canonical attribution functionals when expressed in the concept basis. Specifically gradientx Shrikumar et al. (2017); Simonyan et al. (2013), Integrated Gradients with zero baseline Sundararajan et al. (2017); Ancona et al. (2018), Occlusion Zeiler & Fergus (2014), and RISE Petsiuk et al. (2018) reduce to linear functionals that are proportional to W())W, when aggregated across tokens. Under standard faithfulness criteria such as C-Deletion, C-Insertion Petsiuk et al. (2018), and C-uFidelity Yeh et al. (2019), this is proven to be the optimal attribution when concept are linearly linked to class score; see Theorem Fel et al. (2023b) (and Ancona et al. (2018) for the initial discussion). We thus use this formulation principled and canonical measure concept importance for linear readouts.
C.2 ON THE "ELSEWHERE" CONCEPTS
We have observed and discussed in Section 3 consistent and intriguing pattern: across wide range of ImageNet classes, the top few most important concepts typically include not only interpretable objects or object-parts, but also an intriguing "Elsewhere" concept. As detailed in Figure these concepts activate broadly across the tokens, but crucially not on the object itself. Their firing is suppressed exactly where the object appears, and prominent surrounding regions or background areas. Importantly, "Elsewhere" concepts are not generic background detectors: their firing depends critically on the object's presence, and they vanish entirely if the object is removed from the image.
This phenomenon reveals that certain concepts do not fire where the relevant information is located, but instead are able extract information from one region to fire in spatially distant locations. The Flsewhere concepts represent extreme example of this spatial decoupling, where the relationship between what drives the concept's firing and where it actually fires exhibits a complete spatial inversion. Rather than simply detecting local features, these concepts implement form of distributed spatial reasoning that can be characterized as implementing the logical relation "not the object, but the object exists". This suggests that DINO has implicitly learned sophisticated form of fuzzy spatial logic, systematically distributing class-relevant evidence across both object-centric and contextuall--rlatee off- off-object tokens.
The utility of this distributed representation becomes particularly evident when considering the architectural constraints of vision transformers. Since DINO's final classifier operates the spatial
21

---

Published as a conference paper at ICLR 2026
Elsewhere Concepts
Rabbit
Fox
Image
Tokens map Wwitcch tokeess contain the concepts
Attribution map part image when masked removes concept"
Figure 8: "Elsewhere" concepts reflect off-object activation conditioned on object presence. Visualization of recurring concepts pattern, consistently among the top-3 most important concepts for several ImageNet classes (rows: rabbit, fox, cat), using token-level attribution (middle row) and causal masking Petsiuk et al. (2018) (bottom row). These "Elsewhere" concepts consistently activate in tokens disjoint from the object, yet their presence is conditional on the object itself being present elsewhere in the image: they vanish when the object removed. Rather than capturing background texture, they express a structured logical relation: "not the object, but the object exists". This suggests that DINO implicitly learns form of fuzzy spatial logic, distributing class evidence across both object-centric and off-object tokens.
average of patch embeddings (concatenated with cls token), having class-relevant information distributed across tokens that do not directly contain the object provides several computational advantages. This strategy enhances robustness to partial occlusion, as class evidence remains available in unoccluded regions even when the primary object is hidden. also provides invariance to viewpoint changes and spatial transformations, since class information is not concentrated solely object locations. Furthermore, this distributed approach allows the model to integrate multi-scale contextual information that may be crucial for disambiguation in challenging visual scenarios.
To causally verify these interpretations, we employed RISE Petsiuk al. (2018) analysis, applying random perturbations that mask portions of the image while measuring the resulting changes in concept activation values. The causal attribution maps (bottom row of Figure 8) demonstrate that despite firing in off-object locations, these Elsewhere concepts are causally dependent on the object itself. The RISE attribution computed as:
(
RISE (f. x)) Em-P.
(f(x m) m: =
where f represents the composition of the model and the SAE, using 8000 forward passes for each explanation. The results consistently show that the object or animal itself is the most causally important region responsible for concept firing, even though the concept manifests its activation in spatially disjoint locations.
This finding challenges the implicit assumption underlying most heatmap-based concept visualiza- tion approaches: that concept is primarily about the spatial tokens where the concept fires most strongly. The Flsewhere phenomenon demonstrates clear dissociation between activation localization and causal attribution, revealing that the most informative regions for understanding concept's 's behavior may be spatially distinct from where the concept exhibits its strongest activations. This spatial decoupling would have implications for interpretability research and practice, as it warns against the common tendency to overtrust activation based visualizations as direct indicators of what information a concept requires or processes.
The prevalence of Elsewhere concepts across diverse object categories indicates that this distributed spatial reasoning not an artifact specific classes but represents fundamental computational strategy employed by vision transformers. This pattern suggests that these models naturally evolve sophisticated spatial logic capabilities that go beyond simple local feature detection, instead develop- ing context-dependent activation patterns that are dynamically modulated by global image content. The discovery of this phenomenon highlights the need for interpretability tools that explicitly account for the potential disconnect between concept activation locations and their causal dependencies, incorporating both spatial activation analysis and causal perturbation methods Shaham et al. (2024) provide accurate and complete characterizations of learned representations.
22

---

Published as a conference paper at ICLR 2026
C.3 MONOCULAR DEPTH ESTIMATION
Still in Section 3, We have seen that despite being trained without explicit 3D supervision, DINO exhibits surprising aptitude for depth-related tasks. We have conducted targeted perturbation analysis of depth-relevant concepts using five controlled image manipulations: local median blurring to suppress shadows, global blur remove fine details, high-pass filtering to emphasize geometric patterns, soft edge enhancement to retain contours, and geometric warping to distort perspective cues. We measure concept activation profiles across these perturbations and project results onto UMAP embedding showcased in Figure 3.
This systematic analysis reveals three coherent clusters with distinct sensitivity profiles. The local frequency transition cluster responds to blur and median filtering. capturing spatial detail and texture gradients. The projective geometry cluster shows sensitivity to warping and high-pass filtering, detecting perspective lines and structural convergence. The shadow-basee baseed cluster exhibits primary sensitivity to median filtering, responding to lighting gradients and cast shadows. Many concepts exhibit mixed sensitivity profiles, suggesting DINO learns composite depth representations integrating multiple visual channels. This taxonomy aligns with classical monocular depth cues from visual neuroscience, demonstrating that interpretable 3D perception primitives emerge through self-supervised learning. Full perturbation profiles are available in Figure 9.
23

---

Published as a conference paper at ICLR 2026
Orsat TM
CRAPE
Seace 1AX
DRaip XI1
OenAk 19RO
Olind 3012
LEROUT
SIONN OI
LEXNUER
2004
BIZK
Figure 9: Complete perturbation analysis for depth-relevant concepts showing sensitivity profiles across five image manipulations. The systematic clustering into three main families demonstrates organized depth representation structure within the concept dictionary.
24

---

Published as a conference paper at ICLR 2026
Most Important Segmentation Concepts
Segmentation relies on spatially localized border concepts. Examples of the most important concepts across segmentation tasks, visualized via token attribution (colored overlays). Most of these concepts activate along object boundaries, whether biological (e.g., limbs, heads) or architectural (e.g., domes, rooflines). Despite differences in content, these border concepts exhibit consistent spatial patterns and nontrivial similarity in embedding space (right), suggesting shared functional role and a possibly low-dimensional submanifold within the concept geometry.
Figure 10:
D TASKS CONCEPT FORM LOW-DIMENSIONAL SUBSPACES
To better understand how different tasks recruit concepts from our learned dictionary, we examine the geometric organization of task-specific concept subsets. While Section demonstrated that different tasks draw from distinct regions of the concept space, here we investigate whether these functional specializations exhibit coherent geometric structure.
We extracted the top-500 concepts most aligned with each task (classification, segmentation, depth estimation) based on their importance scores, well random subset of 500 concepts control. For cach subset, we computed 2D PCA projections to visualize their geometric arrangement within the concept space.
The results in Figure 12 reveal differences in geometric organization across tasks. Classification concepts are broadly scattered across the projection space, consistent with the diverse range of features needed for multi-class recognition. In contrast, segmentation concepts trace coherent low-dimensional arc. Depth estimation concepts exhibit clear bimodal structure, reminiscent of the bi-modal organization observed in Figure 11, potentially reflecting the distinct families of monocular depth cues identified (projective geometry, shadows, and frequency transitions). The random concept subset shows discernible structure, confirming that the observed patterns reflect genuine functional organization rather than artifacts of the projection method.
These geometric signatures support the hypothesis that task-specific concept recruitment follows principled patterns: each task draws from geometrically distinct subregions of the concept space, with the local geometry reflecting the underlying computational requirements. This functional- geommeric correspondence suggests that the concept dictionary exhibits hierarchical organization, where related computational primitives cluster together in representational space.
25

---

Published as a conference paper at ICLR 2026
Importance scores
viaaba Orsire Sendrrt Couceds
Easragedrum o'D last
Classification
Segmentation
Depth Estimation
301
107
10
10
10
10--
131-
10
107
10P
1.0
2.1
Figure 11: (Left) Classification recruits more concepts than segmentation than depth. Clas- sification utilizes a larger fraction of the dictionary compared to segmentation and depth, likely reflecting the higher rank of the classification head. This supports the view that task complexity and output dimensionality shape the breadth of concept recruitment. (Middle) Intra-task concept similarity. Cosine similarity histograms of the top 100 most important concepts per task, compared to random subsets of the dictionary. Intra-task concept pairs exhibit higher mutual alignment, deviating from the guasi-orthogonalityy expected of generic dictionary atoms. This suggests that functional concepts form more coherent subspaces. (Right) Spectral analysis of task-specific subspaces. Singular value spectra of the top-100 100 task-relevant concepts reveal sharply decaying profiles for all tasks (especially segmentation and depth) indicating that each task activates a low oow-dimesiional functional subspace. Compared random concept subsets, task-aligned subspaces exhibit stronger concentration, supporting a "functional subspace" hypothesis.
Top-500 Classification
Top-500 Segmentation
Top500 Depth
Top-500 Random
0.4
2.2
3.0
-0.2
-02
-0.2
-3.4
-02
-2.4
-0.4
-025 0.50
0.5
0.50
-2.50
-0.25
0.03
0.2
0.30
-0.25
5.00
0.25
2.50
-0.50
-0.25
0.00
0.5
0.50
Figure 12: 2D projections (PCA) of top-500 concepts most aligned with each task. Classification concepts (left) are broadly scattered; segmentation concepts (middle-left) trace a low-dimensional arc; depth concepts (middle-right) show clear substructure (bimodality), remniscent of the bimodal structure in Fig. (middle); random concept subset (right) shows structure. This supports the hypothesis that each task draws from geometrically distinct subregion of concept space.
E BASELINE MODELS FOR CO-ACTIVATION SPECTRUM ANALYSIS
To contextualize the spectral properties of Z Z studied in Section 4, we construct two baseline models that preserve structural characteristies while eliminating semantic organization.
Random Baseline. mass:
We generate a random symmetric matrix R with identical sparsity p and total
R, Us- .((V) p) where Unj Vi ~ ((0,1)
(1)
R (R + R 1/2
| + R ||F -| Z
(2)
with diag = diag( ZT ZTZ). This tests whether the observed spectral structure results from random co-occurrences expected with identical activation sparsity.
Shuffled Baseline. We preserve the exact empirical distribution of co-activation strengths while
destroying semantic organization through random permutation. Let = Z be the original co-activation matrix. We construct the shuffled baseline as follows:
1. Extract all upper triangular entries: 2/ = {G.y :1 <
2. Apply random permutation æ to obtain shuffled values: 21' (())
26

---

Published as a conference paper at ICLR 2026
3. Construct matrix S where:
Ga
if = j (preserve diagonal)
S1y =
4 if < j (where U/g is the k-th element of of)
(3)
Sji
if > (copy from upper triangle preserve symmetry)
This procedure preserves: (i) all diagonal entries (self-activations)) (ii) the empirical distribution of off-diagonal values, and (iii) matrix symmetry. However, destroys the specific concept pairs and potential block structure while maintaining the same marginal statistics the original co- activation matrix. The shuffled baseline is particularly diagnostic: substantial deviation from (ig(S) indicates that the specific pattern of concept co-activation (not merely the distribution of co-activation magnitudes) carries semantic information.
F BASELINE MODELS FOR CONCEPT GEOMETRY ANALYSIS
To contextualize the geometry of dictionary atoms D, we also build few baselines.
Random vectors on sphere We sample a Gaussian random matrix H E Rexd where Hay (0, 1) i..dd,, and then we normalized each row to have L2 norm 1. This is reminiscent of the concept vectors when randomly initialized before training. high dimension, these vectors are usually relatively isotropic the unit sphere.
Grassmannian frame Next, to stress test the LRH, we numerically computed the Grassmannian from of atoms in d dimension. This is a non-trivial computational problem, where analytical solution is only available for few scenarios, otherwise we need to rely on iterative optimization to find approximate Grassmannian frames. Brute force gradient optimization of incoherence is slow to converge at our scale. Here, we adapted the algorithm from recent work TAAP Massion & Massart (2025), to further accelerate the solver, we adapted it to CUDA, made Grassmannian frame solving feasible at our problem scale frame once still takes 6hr on an A100 GPU. the end, we reached maximal coherence of 0.065897. Indeed, more isotropic and less coherence than the Gaussian random vectors.
32000, d = 768. To note, even with GPU acceleration, solving the
27

---

Published as a conference paper at ICLR 2026
G ADDITIONAL GEOMETRIC DIAGNOSTICS
In Section 4, we studied the geometry of the dictionary and found that while most dictionary atoms are quasi-orthogonal, we observe deviations that hint at additional non-trivial geometric structure. Some concept pairs are nearly antipodal (D; ~ -D)), yet correspond to semantically opposed features such as left vs. right or white vs. black. This suggests that DINO may use polarity encode fine-grained meaning along shared axes (see Figure 13, left).
In addition, still in Section 4 we wonder whether co-activation statistics shape the geometry of the learned concepts. Comparing the concept co-activation matrix Z Z with the geometric affinity matrix DD we find only weak correlation (r 0.28, R 0.08). This indicates that features often used together are not necessarily close in embedding space. A UMAP embedding overlaid with high co-activation links confirms this (Figure 4): connections are non-local, with no strong modularity. Together, these results suggest that DINO's s concept geometry is only partially shaped by usage patterns, and that polarity and function may act as latent organizing principles.
Antipodal Concepts
Vertical lines
Horizontal lines
0-0.28, p-0, R7
White shirt
Black shirt
Person on the Left
Person on the Right
10
10
30
272
Figure 13:
Emergent geometric patterns in the dictionary. (Left) Antipodal concept pairs
(D, ~ DD,) encode semantically opposed features (e.g., vertical vs. horizontal). (Right) Geometry is weakly shaped by co-activation: correlation between Z Z and DDT is low (r 0.28), and UMAP overlays reveal non-local structure with no clear modularity.
28

---

Published as a conference paper at ICLR 2026
H TOKEN-TYPE-SOE
CONCEPT
In the first part of this work, specifically the Section 3, we have studied DINO concepts primarily through their semantic content and task alignment. However, this overlooks fundamental structural aspect in Vision Transformers: the token types. In ViT architectures Dosovitskiy et al. (2020), not all tokens play the same role: cls and reg tokens are explicitly designed for global processing Darcet et al. (2023), while spatial tokens correspond to local image patches. This raises natural question: are some concepts specialized for specific token types? And if so, do they occupy distinct subspaces of the concept geometry?
H.1 FOOTPRINT ANALYSIS
To address this, we study the footprint of each concept: the distribution its activations across token positions. For every concept, we compute the entropy of its token-wise activation over 1.4 million images. Concepts with low footprint entropy are highly localized activating consistently on specific token subsets whereas high-entropy concepts are spatially diffuse and positionally agnostic.
Average Concept Token Map sorted by Entropy
Figure 14: Spatial footprints of 2,500 concepts sorted by entropy. Each row shows the average activation pattern of one concept across token positions, arranged from most localized (low entropy. top) most uniform (high entropy, bottom). Localized concepts exhibit strong positional preferences, firing predominantly in specific regions on particular token types (cls, register, spatial positions), while uniform concepts show broad, distributed activation patterns across the entire image.
29

---

Published as a conference paper at ICLR 2026
Formally, to understand the spatial organization of concept activations, propose the notion of "footprint" of each concept: its distribution of activations across token positions within images. With Z Rutxe that we reshape to the tensor Z Rextxe then for the concept we compute the footprint w E as:
N
W
where Zno. is the concept map (a vector 261 scalars for DINOV2-b) of the concept on input n, and N is the total number of inputs. This captures the average activation strength of each concept at each spatial location, revealing whether concepts exhibit positional preferences or fire uniformly across the image.
We characterize each concept's 's spatial specificity using the entropy of its empirical footprint distribu- tion. Low entropy indicates highly localized concepts (e.g., firing only at specific positions or token types), while high entropy suggests spatially uniform activation patterns.
The analysis reveals a spectrum of spatial behaviors. Most concepts exhibit relatively uniform firing patterns across spatial tokens, but a significant subset shows strong positional biases. These include concepts that fire exclusively on register tokens (capturing global scene properties like illumination), position-specific concepts that consistently activate in particular spatial regions (potentially encoding geometric or compositional biases), and single concept that fires exclusively on the cls token (likely encoding its positional embedding). This spatial specialization provides further evidence for the functional organization of the concept dictionary, with different concept families optimized for different computational roles within the architecture.
1044
Average Activation
Avarage Activation Spatal Taker
102
Concept Types by Positional Usage
Nambo Concorts
10-
0.4
12
922
0 30 14
C.5.orly
RES onty
Spotia only
10
Figure 15: Statistical analysis of concept footprints. (Left) Average activation intensity per token position, showing elevated activity on register and cls tokens compared to spatial tokens. (Middle) Same analysis restricted to spatial tokens only, revealing subtle positional biases within the image grid. (Right) Distribution of concepts by token-type exclusivity, confirming the findings from Figure 16: one concept fires exclusively on cls, hundreds specialize for register tokens, and many are restricted to spatial positions, indicating substantial functional specialization beyond uniform activation patterns.
H.2 TOKEN-SPECIFIC SPECIFIC CONCEPTS.
Figure 16 reveals continuum of footprint entropy across the concept dictionary. While most concepts are positionally agnostic (high entropy). distinct subset exhibits highly localized activations, We identify three main categories among low-entropy concepts: (i) Position-specifee concepts that consistently activate on narrow spatial regions, such as "left-only" or "bottom-only." These may reflect residual positional encoding, biases in the training data, or local geometric primitives. (ii A unique cls-only concept, which fires persistently on the cls token across all images. This concept is closely tied to the positional embedding of the cls token and may act an "ID" or "passport" -like concepts in the network. (Mi) much more diverse (and surprising) group of reg-only concepts that activate exclusively the registers tokens. Unlike the cls case, the variety and number of reg-specific concepts cannot be explained by position alone. This suggests the register tokens encode of non-spatial features that explore in the next section.
30

---

Published as a conference paper at ICLR 2026
Low trenery Awarago Conoent
Medius Entrrey Awargg Coraapt
Ineraay Ane Doneaat
Concapt Types Bortto Usage
Figure 16: (Left) Token-wiss footprint of concepts. Distribution of concept entropy across token positions. Most concepts are not token specific (high entropy), but a significant tail exhibits strong localization. The average activation maps for low-entropy concepts often reveal spatial edges (e.g.. left/right, top/bottom) or special-token specificity. (Right) Number of concepts by token-type exclusivity. While only one concept fires exclusively on the cls token, hundreds are reg-only, indicating substantial specialization beyond positional embedding concept.
Motion blur
Illumiationn Worping Distortions (e.g.
Randering style: Games, Drawings...
Object and Scone Context
Register-only concepts capture global, non-local scene properties. Visualization of selected concepts that activate exclusively on reg tokens. These concepts are not object or region specific, but instead encode global properties such as motion blur, illumination, caustics reflections, lens effects, and style. Their emergence suggests that the register token acts as a potential conduit for abstract scene level information.
Figure 17:
Register-Only Concepts and Global Scene Properties. Upon inspecting the register-only concepts, an interesting pattern emerges: these concepts do not align with localized object parts semantic categories. Instead, they seem to encode holistic, global attributes of the image. As shown in Figure 17, register-only concepts respond to phenomena such as lighting style, motion blur, caustic reflections, or artistic distortion. Some even appear sensitive to camera properties (e.g., wide-angle warping or depth-of-fieh -feld effects).
Interestingly, this specialization is highly asymmetric: while only single concept activates exclu- sively on cls, hundreds specialize for reg. This suggests that DINOv2 not only encodes high-level features, but also distributes them across structurally distinct token pathways.
31

---

Published as a conference paper at ICLR 2026
1 QUALITATIVE VISUALIZATION OF LOCAL GEOMETRY
Figure 18: Local embedding geometry of patch tokens across random ImageNet samples. Each panel shows ImageNet image alongside its token-level geometry. Patch embeddings from DINO are projected via PCA (on per-image basis) and visualized by assigning the first three principal components RGB color (normalized to [0, IJ). Despite no supervision, token embeddings seems to lie smooth, low dww-dimnniioal manifolds, with transitions often aligned to object boundaries or perceptual contours.
We provide further visualizations to illustrate the structure of DINO representations. at the per-image level. In Figures 18 to 22, patch tokens are projected into their top PCA components, computed independently for each image. The resulting RGB visualizations highlight smooth embedding transitions that often align with object boundaries.
To complement these local views, Figure 19 shows controlled example across five similar inputs (snow leopards), using a shared PCA basis trained on the entire token corpus (of those 5 images). While individual token positions vary slightly due to pose lighting, the embeddings align within common geometric frame. This suggests that DINO not only builds smooth manifolds locally, but does in globally consistent latent space.
32

---

Published as a conference paper at ICLR 2026
Figure 19: Detailed structure of token geometry in snow leopard images. Here project all patch tokens from five different snow Icopard images into the PCA basis trained on all the DINO token pooled across the five images. Each token is visualized in RGB according to its coordinate along the principal components. This shared projection reveals how mnnilarly-positione objects (here, the leopards) align across images within the same geometric manifold. Despite slight variations in pose and lighting, the representations remain coherent and consistent across instances. This provides evidence for global manifold structure, within which local image tokenizations trace smooth trajectories.
point in 3D PCA space, and colored
J POSITION EMBEDDING ANALYSIS
To investigate the role of positional information in the observed smooth token geometry, we conduct two experiments: (i) extracting and characterizing the positional basis across layers, and (ii) verifying that smoothness persists even after removing positional information.
J.1 POSITIONAL BASIS EXTRACTION.
For each layer, we extract positional embeddings from 1 million ImageNet images, yielding A E xtxx token representations where the number of images, t 261 tokens per image, and d = 768 is the embedding dimension. We employ two approaches to recover the positional basis: Direct averaging. We compute the average embedding for each spatial position across all images:
p, = 3 E An,i where An.i is the embedding of token at position in image n. We repeat this procedure for each layer. Linear classification. We train linear classifier predict token position from embeddings, yielding weight vectors w; for each position at each layer.
Both methods produce highly consistent results: the stable rank profile is similar and the accuracy yield the same results. We therefore choose to use the classifier weights as our primary positional basis for the rest of the experiment.
The analysis in Figure 24 reveals that positional information undergoes systematic compression across layers. Early layers maintain high-rank positional representations that allow precise spatial
33

---

Published as a conference paper at ICLR 2026
SPHINE
Figure 20: (cont.) More examples of PCA-colored patch token embeddings in Fig. 18.
localization, but this progressively collapses to low-dimensional (approximately 2D) subspace in the final layers, consistent with transition from place-cell-like to coordinate-based encoding.
J.2 STRUCTURE PERSISTS AFTER POSITION REMOVAL.
Still in Figure 24, we showed that the position basis is not responsible for the main part of the structure observed the PCA visualization. To test qualitatively this effect, we project token embeddings orthogonal to the positional subspace, completely removing positional information by projecting the token on the orthogonal subspace of the classifier.
Remarkably, PCA visualizations (in Figure 25) of the original image of Figure 19 embeddings continue to exhibit the same structure, with smooth patterns that align with object boundaries and semantic regions. This demonstrates that the interpolative geometry we observe reflects genuine semantic organization rather than artifacts of positional encoding. The structure emerges from the model's 's representation of visual content itself, not from spatial coordinate information.
34

---

Published as a conference paper at ICLR 2026
0:28 8
BELEGT
CONTROL
Figure 21: (cont.) More examples of PCA-colored patch token embeddings as in Fig. 18.
35

---

Published as a conference paper at ICLR 2026
Figure 22: (cont.) More examples of PCA-colored patch token embeddings in Fig. 18.
36

---

Published as a conference paper at ICLR 2026
Figure 23: (cont.) Final examples of PCA-colored patch token embeddings as in Fig. 18..
37

---

Published as a conference paper at ICLR 2026
Excleined Veriance of Position Dasis
Accuroc)
Stacke Ront
0.6
0.4
CLS
0.2
REG
Spatial
0.0
0.65
12
Layers
Layer
1.0
Per-image PCA: cumulative variance captured
Cosine similarity PCA/Position basis
0.25
Pos comp. 1
Curultetvv expaired vaiterre
0.8
0.20
Pos camp. 2
0.15
0.4
0.10
0.05
1000
150
200
50
100
150
200
250
Number PCA components
PCA component
Figure 24: (Top) Positional encodings compress over layers (Left) Linear decodability of token position across layers: precise spatial coordinates are recoverable up to layer 8, after which accuracy collapses. cls & reg tokens linearly decodable until the end. (Right) The rank of the positional embedding subspace steadily drops, converging to a subspace by the final layers-indien indicating strong compression. (Bottom) Few PCA components of single images tokens explain the variance of tokens across the entire image, suggesting they are lying in a low dimensional subspace. On average, the image-wise PCA shows that position basis correlates with components 3 and 5, but not the dominant directions. The "smoothness" of the PCA maps persists even when positional components are removed, suggesting that this reflects deeper geometric organization beyond position alone.
Figure 25: Smooth token geometry persists after removing positional information. PCA visual- ization of patch token embeddings from rabbit image after projecting orthogonal to the positional basis to completely remove spatial coordinate information. Despite the absence of any positional cues, the token embeddings (colored by their first three PCA components) continue to exhibit smooth, structured patterns that align with object boundaries and semantic regions. This demonstrates that the interpolative geometry reflects genuine semantic organization rather than artifacts of positional encoding.
38

---

Published as a conference paper at ICLR 2026
K MINKOWSKI REPRESENTATION HYPOTHESIS
The hypothesis stated in Section 6 is an hypothesis about how the representation space is composed. Such assumption is crucial, as it determines what the method, and what the method can validly recover, (what we can see). To put it simply, if sparse autoencoders implicitly answer to the Linear Representation Hypothesis, then we understand the importance of specifying the right ambient geometry it not only conditions our interpretation but also determines the very methods we use to extract concepts.
Armed with the observation of Section 4, contend that alternative account explain the phenomena we documented, in particular the interpolative geometry within single images.
K.1 BACKGROUND AND RELATED WORK ON CONVEX/POLYTOPAL VIEWS
As explained in the main text, our motivation for this hypothesis traces to Gärdenfors' conceptual spaces, where concepts inhabit convex regions along geometric quality dimensions Gärdenfors (2004). Put plainly, have reasons to believe that the observed interpolation is the surface of a deeper organization: the activation space behaves as a sum convex hulls' 2. A single attention head already performs convex interpolation over its values, creating archetypal geometry; multi-head attention then aggregates these convex pieces additively, yielding a Minkowski sum. It is natural to imagine one hull reflecting token position, another depth, another object or part category, activation is the sum of these convex interpolations, and the "concepts" available to probes are the archetypes governing each hull. We will start by making this idea explicit, then we will review some theoretical evidence based on previous works and showing how simple attention blocks generate such geometry. Then, we will follow showing some empirical signals that make the proposal plausible candidate and the implication of such geometry. We first formally recall the Minkowski Representation Hypothesis stated Section 6.
that the final
Definition 3. Minkowski Representation Hypothesis (MRH). Let X Rdd be a layer's activation space and x E X. Let A = (a1...., ae) E Rexd be Overcomplete Dictionary of A Archetypes (c >> d). We partition the archetypes into m disjoint tiles {Ti) To (l,...,c), and define the tile polytopes:
P, conv (AT)) with AT
Then X satisfies MRH if
(i) Minkowski sum:
X' m Pi
(i) Block-convex coding:
x E iES ZiAT, 2, E A |S] < m.
(iii) block-structured Gram:
G = Z Z E Rexe with blocks corresponding to {T))
where Z E gaxe stacks archetype codes over n samples and A = (=
We have formally defined MRH motivated by an empirical observation of interpolative geometry and now ask basic question: can single attention block generate such structure? The answer is yes, and the mechanism is elementary. We first review relevant prior work, then show how attention naturally generates such structure. Previous work provides two lines of supporting evidence. First, neural networks naturally partition input spaces into convex regions through their piecewise-linear activations Montúfar et al. (2014); Raghu et al. (2017); Balestriero al. (2018), 8), suggesting that convex decompositions may be fundamental to deep architectures Tvetkova et al. (2025); Tankala et al. (2023); Hindupur al. (2025). Analyses have also approached neural networks explicitly through the polytope lens, showing how piecewise-linee (2022). Second, recent work on neural population geometry shows that deep networks organize representations into low-dimensional manifolds with rich geometric structure Chung (2021); Cohen et al. (2020), which is consistent with a small number convex factors combining to yield the observed variability. Third, in the language domain, categorical and hierarchical concepts have been shown to admit convex (polytopal) representations, with semantic relations reflected directly in geometric structure Park et al. (2025).
partitions structure representation space Black et al.
2This aligns with evidence that concept structure can be convex and compositional in other domains Park al. (2025)
39

---

Published as a conference paper at ICLR 2026
We now demonstrate that standard attention mechanisms naturally generate the geometry described by Definition 1. The argument proceeds in three steps, and full proofs are provided in Appendix K. showing how elementary operations compose to create the previously describe geometric structure.
We will show that the elementary operations available in DINO, and in particular a standard multi- head attention block, already generate the geometry described by the Minkowski Representation Hypothesis. We then state and prove a few basic properties of this geometry and discuss its robustness. We begin by recalling some basic fact and with elementary results.
Notations. For a finite set of vectors V = {(i,,... Um C IRd the convex hull conv(-) (-) is
m
m
conv(V)
0jo; aEAA] where
A m E .aj 20,
Σ aj
j=1
And denote by R - Rd the standard softmax:
aι)) =
-== e
and recall two basic properties that will be used repeatedly: (i) softmax is invariant under adding a constant along the all-ones direction, a (x=+A) (ar), (ii) maps RRm /span [1) diffeomorphically onto the relative interior of the simplex relint( A "., with inverse given by the log map up to an additive constant, namely if a E relint( A ") then log + A1 satisfies o (xx)
K.2 FROM A SINGLE HEAD TO CONVEX POLYTOPES
We start with the most basic question: what does a single attention head produce geometrically? A single head takes query, forms attention weights over fixed set of value vectors, and returns weighted sum of those values. We now make precise the fact that the range of this map is a convex polytope, namely the convex hull of the values, and that under mild reachability condition every point of that hull can be attained.
Lemma 1 (Single head creates convex polytopes). Consider one atlention head with values V (V1...., vmm C Rd and attention weights a E A., The attainable output set is
m
0g aEA") Tm S conv(V).
y= j=1
Moreover. suppose the pre-softmax logit map can generate any vector in in(KT) + span(1) R as the query varies, where K denotes the matrix of keys and the invariance to 1 reflects softmax's s additive invariance. Then y conv(V) (V). In this case they admit a MRH representation, their codes are z = a and the archetypes are A.
Proof. By definition of single head, the output has the form y = ar with a E m Therefore y E conv(V) and y C cVnv(V). Now, let a E relint(A" be arbitrary. Define u = log entrywise and note that adding any constant e along leaves softmax unchanged. Hence a (a + A1) = a. If the pre-softmax logit map can realize any vector in Inm up to the direction, then there exists a query producing logits x + A1. For that query, the attention weights equal the prescribed a and the head output is the convex combination E, UyUj. The argument above covers all points with strictly positive coefficients, that relint( A.). The extreme points and boundary faces of nm can be attained by limits of interior points (or by sending some logits to -o)), hence their images under the affine map a + E, 2y are limits of attainable outputs. Therefore y contains all of conv(V).
Finally, for the MRH identification, this corresponds the special case of |.S| with a single polytope, where the codes are simply z = and the archetypes are the values A = V.
An interesting observation is that one could show that a strictly block-sparse attention (with V affinely independent) would induce disjoint polytopes by applying the previous lemma to each block, splitting conv(V) as the union of conv( V,) for each block b. This is particularly interesting as ViT attention patterns often observed to be block- block-sparse in practice. This establishes the base case: a single head creates a convex polytope. The next question is: does this convex structure survive the linear and affine mappings applied throughout the transformer, such as projection matrices or RMSNorm?
40

---

In that Published as a conference paper at ICLR 2026
K.3 ROBUSTNESS TO TRANSFORMATIONS
The first observation is straightforward but crucial: the image of convex combination under an affine map is the corresponding convex combination of the images. In our setting, this means that an activation admits an archetypal decomposition, then any affine layer simply moves the archetypes while keeping the codes unchanged. This is exactly the stability we need to propagate archetypal structure through projections and bias terms.
Lemma 2 (Affine transformations preserve MRH structure). Ler y(x)) = Wx + b be affine transformation and x Σÿ 2j ag with 2j 2 and E; 24 1. Then
((x) Σ 2g a with aj Wa, +
Hence convex structure is preserved: archetypes absorbe the transformation, but the codes z remain unchanged.
Proof. Starting from a = Ej 2/ ag with 2j 2 0 and E, 23 = 1, apply and use linearity of W:
7(x) W Σ 2j aj) + b Σ 3j Waj + b Σ 2 (Waj + b)
Σ 2j aj.
The right-hand side a convex combination of the transformed archetypes a'z-
Any linear projection W and any bias addition preserve convex decompositions exactly in Lemma 2. In particular, the per-head output projections and the final output projection of attention blocks do break archetypal structure. We remark that LayerNorm and RMSNorm are not globally affine because their scale factor depends on the input. However, many norm become affine in evaluation, once veean/variance are held fixed.
K.4 FROM MULTIPLE HEADS TO MINKOWSKI SUMS
We now turn to the multi-head case. Each head yields a convex polytope (the convex hull of its value vectors), and the standard attention output aggregates head outputs additively after per-head linear projection. This naturally leads to Minkowski-sun geometry as describe in Definition 1.
Proposition (Multi-head attention realizes MRH geometry). Let there be H heads. For head h, let (ma) be its value vectors, and let the head output be
V, (v)) , ",,.))
(1)
( Oh. Up
with Oh E TR
ARA
Let (hh)) be the per-head output projection. The block output is
H1
w
(h)
yn.
h=1
Then
(K) E W O (conv(Vi))
W
that is. the attainable outputs lie in the Minkowski sum of the head polytopes after projection. Moreover. under the reachability condition that each head can realize any point of relint ("")) (up the softmax additive constant), the set of altainable outputs equals this Minkowski case, the representation admits an MRH form with block-convex codes
z = (a aH) and archetypes A VW (
(H)) VR),
where each block oi,, belongs to a simplex and each block of archetypes the projected value set for that head.
41

---

Published as a conference paper at ICLR 2026
Proof. By Lemma 1, for each head h we have yn Vonv(V.).. Then, for any linear map L and finite set S, we have L(conv(S)) conv(L(S)). Therefore W (h) yh E w (conv( (v.)) =
conv (w @ Vn.. By definition of the Minkowski sum, if ps w (conv(Vi.) then Σn Ph
4b w th) (conv(Vi)). Taking pa w yh and summing over heads gives
H
w yn E o w (conv(Vi)))
y
This proves the inclusion.
Now, assume head h can realize any an E relint(A (Am). Let an arbitrary element of the Minkowski sum be given:
y Σpn Ph with Ph W! (conv(())..
For each h there exists Br nR such that ps W E, thy By reachability, choose query so that the head-h attention equals a = 3r Then the block output equals - y" was arbitrary in the Minkowski sum, the attainable set equals the sum.
For the MRH identification, we collect the per-head attention weights into the block vec- tor 2 (a),.... ('11) and the per-head projected values into archetype blocks A (W Vi...., (.. The output y thus a sum of H block-convex combinations, realiz- ing the Minkowski sum structure with block-convex coding as required by MRH.
Having established that attention produces convex polytopes and that affine transformations preserve their structure, now move toward more realistic operating regimes. What does it mean, geometri- cally, to have sparse or block-sparse attention as is commonly observed, and what happens under the elementwise nonlinearities used in practice?
K.S ATTENTION CONCENTRATION EFFECTS
We first examine the limit in which softmax sharpens. In fact, as temperature decreases, attention places nearly all mass on the highest-logit index. The attainable set contracts from the full convex hull toward its vertices.
Lemma (Low-temperature softmax selects vertices). Consider attention with values V (VI,..., Um and weights (</TT) with logits x Rm and temperature T > Then
lim Σajo
U3 where j = arg max Tj.
In the zero-temperature limit the output lies vertex of conv(V).
Proof. For 0, a, exp((x/T)) E. exp(Ix/T). If j* indexes the maximum of a and A, Ij' xj 0, then agy 1/(11 + E Ejp As/T) and EjØr" aj Ejjzs e -As/T aj+- As т -- 0, all terms e A//T vanish for A, > 0, hence ag- + 1 and aj + 0 for j # jr. The convex combination collapses to vy-.
In fact, we can derive a quantitative measure of this convergence, let diam(V) maxp.e 1| Myl denote the diameter of the value set. Then the deviation from the winning vertex vy satisfies
Σasny a, wz. diam( V)
Σ e As/T j71
showing that finite logit margin already forces the output to lie in small neighborhood of vertex, with exponentially small deviation 1/r. Another way to see the effect to direcity study the geometry created under striet sparsity contraint.
42

---

Published as a conference paper at ICLR 2026
Lemma 4 (Support restriction selects subpolytope). Fix a subset S S {I....,, m. and consider the feasible set of outputs with attention supported on S.
Y's
< 2j2 a aj = a= 0 $ JES
Then Vss conv([v, :) S}). In particular, across input regimes the support repeatedly takes values in a family S of subsets, the attainable set is the union Uses conv : E 5)), ie., union of lower-dimensional polytopes. When a given S coincides with the maximizers some linear functional over V, V/ is aace of conv(V). When S = arg max, (w, Uz) for some vector w, then V is the face of conv exposed by the supporting hyperplane with normal w.
Proof. Immediate from the definition of convex hull and the constraint a; 0 for & S. The face condition the standard supporting-
upporting-hyperplane hyperplane characterization of faces Rockafellar (1970). I
Lemma 4 formalizes the intuition: attention sparsifies, geometry collapses from the full polytope to subpolytopes, and in the extreme low-temperature limit to vertices (point).
K.6 NON-DDENTIHABILITY MINKOWSKI DECOMPOSITION
We now address question that bears directly on recoverability: given only the attainable activation set
X C Rd with X= == can we uniquely recover the summands (P:}? MRR is to be useful for analysis, we must understand when (and why) such decompositions are non-unique.
Proposition 4 (Non-identifiabiliiy of Minkowski decomposition). Let X' = -i be Minkowski sum of convex polytopes. Given only observations from X, the decomposition (PI...., Pm is generally non-unique: there exist distinct collections {21...., _22 such that X - Qj. In particular, even very simple polytopes admit infinitely many decompositions as sums of line segments (zonotope generators) with varying directions and lengths.
Proof. We argue via support functions. For a nonempty closed convex C Rdd. its support function is
he(u)= = sup x²) sec
u E Rd
Support functions are sublinear (positively homogeneous and subadditive), and they are additive under Minkowski sums Gardner (1995):
( = ha(u) + (n( for all u E RR..
Hence a decomposition X' = = is equivalent to a decomposition
hx
Σ hpp.
of the single sublinear function hx into sum of sublinear summands.
But additive decompositions a fixed sublinear function are highly non-unique in general. Indeed, fix any sublinear with 0 < lx pointwise. Then both
m
hx (hx 3) + and, more generally, (hi...., hm s.t.
h4 hh= hx
all define a valid support-function decompositions. Under lower semicontinuity, each sublinear hi itself the support function of a unique closed convex Q; containing the origin, i.e., ho. = h4- Therefore
m
hx
X'
1
Since there are infinitely many ways to split fx into a sum sublinear functions, there are infinitely many corresponding Minkowski decompositions. This proves the general non-uniqueness claim.
To build concrete intuition on why non-uniqueness appear, consider the simplest case decomposing rectangle of segments.
43

---

Published as a conference paper at ICLR 2026
Segments generating the same zonotope We work in I2 and consider the axis-aligned rectangle
X' = l- x [-b,b] = (-a,aaeer) @ (1-b, (l-b,
For any a E (0, 1),
[-a, 1- e/ = a-aa, aale; @ [- a)a, (1 - a)aae1.
80
== l-aa,a aajer + 1-11 a (1- a)aee + [-b,
Varying a yields uncountably many distinct 3-segment decompositions of the same rectangle. Also, iterating this splitting on either axis leads infinitely many distinct k-segment decompositions whose projections sum to the same total width and height. Hence even for very simple polytopes, Minkowski-sun decompositions into segments are non-unique.
K.7 EMPIRICAL EVIDENCE
We now detailed for the preliminary evidence exposed in Section 6, for each of the 3 criterion of the hypothesis. The results below should be read as compatible evidence, not proof: multiple ImageNet-
mechanisms mimic the same surface phenomena. Unless stated otherwise validation tokens from the last DINOv2-B layer; cosine distance and k-NN graphs with standard symmetrization.
Figure 26 (left) contrasts straight-line interpolation between tokens with piecewise- computed shortest paths the token k-NN graph. Straight lines depart the data support quickly, whereas graph geodesics remain close throughout. This matches what Minkowski sum structure predicts: feasible displacements arise as sums of small face-walks within head polytopes, which are piecewise linear in barycentric coordinates yet appear curved in the ambient space. The curved, on-support geodesics thus support criterion (i) by reflecting head- sead-wise convex reweighting rather than simple linear combinations.
geodesics
To test the convex coding assumption (i), we compare Archetypal Analysis Cutler & Breiman (1994) (AA) with SAE for token reconstruction in Figure 26 (middle). Note that AA is precisely the single-tile case of MRH ([S) = 1), making this a direct test of our geometric assumptions. AA imposes dramatically stronger constraints": it forces all reconstructions to lic within the convex hull of observed tokens and requires archetypes to actual data combinations. Despite these restrictive assumptions, AA matches SAE performance with remarkably few archetypes (10 archetypes per image), providing preliminary evidence that even the simplest case of MRH captures fundamental geometric structure.
This is particularly striking in high dimensions, where the probability that random point lies in the convex hull of small set of samples decays exponentially with dimension Bárány & Füredi (1988); Balestriero et al. (2021); the fact that tokens can nevertheless be accurately reconstructed from only ~0 archetypes indicates that embeddings concentrate on low-dimensional polytopes embedded within the ambient space.
Additionally, the block structure emerges naturally: Figure 26 (right) reveals that archetypal decompo- sitions spontancously organize into the block-sparse pattern predicted by criterion (iii), with distinct clusters of co-activating archetypes rather than uniform mixing.
We emphasize that these observations piece-wise linear on-manifold trajectories, efficient archetypal reconstructions, and dlock-structured co-activation - represent preliminary evidence rather than definitive proof. Multiple geometric hypotheses could generate similar surface patterns. Nevertheless, the convergent evidence motivates exploring what MRH would mean for interpretability practice.
Positioning & Claims. We treat the Minkowski Representation Hypothesis (MRH) working hypothesis, not proved theory of representation. We do not claim causal identification of concepts, head "tiles," their generative mechanisms; our evidence is observational and model-specific (DINOv2-B). Our goal is to describe robust empirical regularities (task-specific subspaces; families
Archetypal Analysis seeks Z = X B where archetypes are convex combinations of data (so Z C conv (X)), and X ZA where data are convex combinations of archetypes. Both A and B have simplex constraints (columns to 1, negatvvi......
44

---

Published as a conference paper at ICLR 2026
Z'Z of Arch. Analysis
SAE ender
Dezance to narentt Samge
0.30
Archetypal Analysis
Meen Squared
0.25
6-
0.20
0.15
graph interp.
step
Archetypes
Figure 26: Empirical support for MRH criteria ImageNet- 1k validation set. (Left) Distance to data along interpolation paths between tokens. Linear interpolation (dark line) rapidly deviates from valid embeddings, while there exist piecewise-linear paths computed via shortest paths token k-NN graphs (light blue) that remain consistently close to the data manifold. This supports criterion (i) Minkowski sum structure: feasible connections follow polytope face-walks rather than straight lines through empty space. (Middle) Archetypal Analysis (single-tile MRH with [S] = 1) achieves lower error than a sparse autoencoder (dashed line) with few as 10 archetypes, despite stronger constraints, supporting criterion (iî) convex coding assumptions. (Right) Archetypal coefficient matrix z" Z after clustering reveals emergent block structure with bright diagonal clusters. Even without knowledge of tile boundaries, archetypes naturally organize into co-activating groups, supporting criterion (iii) block-structured decomposition.
of depth cues; "Elsewhere" patterns) and propose a geometry that makes testable predictions for future work (e.g., per-head block structure, subspace additivity across heads). We therefore refrain from stating that our results "contradict" LRH; instead, we document systematic departures from purely sparse near-orthogonal picture. All conclusions should be read as conditional on our dictionary, probes, and datasets, and we report ablations where feasible.
45

---

Published as a conference paper at ICLR 2026
L VISUALIZATION TOOL
To facilitate exploration of the concept dictionary and make our findings accessible to the broader research community, release DinoVision interactive web-basee visualization tool that enables real-time exploration of the 32,000 extracted concepts. The tool presents concepts as points in a 2D UMAP projection where spatial proximity indicates conceptual similarity in the original high- dimensional space, though global clustering patterns should be interpreted with caution due to UMAP's limitations in preserving large-scale structure. The interface displays each concept
DIVCMidion
Every
detak
Figure 27: Vision interface showing the interactive 2D UMAP projection of 32,000 concepts extracted from DINOv2. Users can explore individual concepts by clicking points to reveal activation patterns across the 261 token grid (256 spatial patches plus CLS and register tokens). The visualization includes adjustable parameters for point size, opacity, and co-occurrence links between frequently activated concept pairs.
point whose size reflects activation frequency across the dataset. Users can click any point to examine detailed activation patterns showing how that concept fires across DINOv2's 261 tokens, which comprise 256 spatial patches arranged in 16 16 grid plus classification token and four register tokens. The token visualization uses color intensity to indicate activation strength, with redder regions corresponding stronger concept responses. The tool help us discovered that some concepts activate exclusively on register tokens, and they seems to capturing global scene properties like illumination and motion blur. Interactive features include adjustable visualization parameters such as point size scaling, heatmap opacity controls, and the ability to display co co-occurrence links between concepts that frequently activate together. Users can navigate directly to specific concepts by entering concept identifiers or explore neighborhoods around selected points. The co-occurrence analysis reveals structured relationships in the concept space, with connecting lines indicating statistical dependencies between concept activations. For concept visualization use Feature Accentuation (FA) from Hamblin et al. (2024), we start from maximally activating images each concept and perform 1024 steps of gradient ascent optimization parameterized in Fourier space with MACO Fel et al. (2023a) constraints on the magnitude of the spectrum, boosted according to natural image statistics approximately following 1/w² where represents cycles per image.
Importantly, the tool implements a composite (two-layer) rendering approach that maintains smooth 60fps interaction for point navigation while progressively loading high-resolution concept visualiza-
46

---

Published as a conference paper at ICLR 2026
tions needed. We believe the tool serves both research instrument for further investigation and a demonstration of the rich structure present in vision transformer representations.
47


Reviews:
[{"review_id": "cZutpRaT19", "reviewer": "Reviewer_nELv", "summary": "The paper studies what DINOv2 “sees” by operationalizing the Linear Representation Hypothesis (LRH) with a sparse autoencoder to extract a 32k-concept dictionary from ViT activations, then analyzing how downstream tasks recruit these concepts and what the concepts’ geometry suggests. Empirically, different tasks use distinct, low-dimensional subsets: classification leans on off-object “Elsewhere” concepts, segmentation concentrates on boundary detectors, and depth estimation draws on monocular cue families. they also run geometric diagnostics and find some deviations from the LRH like structured redundancy. Motivated by this, the authors propose the Minkowski Representation Hypothesis (MRH): token embeddings lie in Minkowski sums of convex polytopes spanned by archetypal landmarks, a structure they argue multi-head attention realizes via convex combinations per head and summation across heads. They provide preliminary qualitative/quantitative signals consistent with MRH - e.g. token embeddings smoothly interpolate between landmark-like prototypes instead of varying along linear feature axes.", "questions": "- How do you quantitatively assess convex regions or Minkowski sums beyond analogy to attention mechanics?\n- How sensitive are the learned concept dictionaries to SAE initialization and regularization strength?\n- How do the discovered “Elsewhere” concepts improve human interpretability or model steering compared to prior SAE- or NMF-based approaches?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is ambitious in scope, undertaking one of the first large-scale interpretability analyses of a state-of-the-art vision foundation model through SAEs. \n- Goes beyond static visualization to quantify how distinct downstream tasks (classification, segmentation, depth) selectively recruit different subsets of the concept space, revealing functional specialization. e.g. the paper identifies interpretable and generalizable phenomena—e.g., “Elsewhere” (off-object) concepts, border detectors, and monocular depth cues—illustrating that meaningful, task-specific structure emerges spontaneously in DINO.\n- Moves beyond the traditional “sparse direction” view of representations to analyze the geometric structure (anisotropy, coherence, spectra) of learned concepts\n- The work also makes a creative theoretical leap—proposing the Minkowski Representation Hypothesis to reinterpret transformer geometry in terms of convex archetypal regions, an idea that elegantly ties attention mechanics to cognitive theories of conceptual spaces.", "weaknesses": "- The paper attempts to do too much—spanning SAE implementation, large-scale task analysis, and a new geometric theory (MRH)—without a unifying throughline. The connection between these parts often feels narrative rather than logically necessary.\n- The evidence for the Minkowski Representation Hypothesis is largely qualitative and circumstantial (e.g., UMAPs, smooth PCA maps, block structures). Stronger quantitative tests or falsifiable predictions are needed to substantiate the claim.\n- The observed deviations from LRH (anisotropy, coherence, low-dimensional task subspaces) could be explained by simpler mechanisms such as structured sparsity or normalization effects—yet the paper moves quickly to a new geometric framework without ruling these out.\n- Key hyperparameters (e.g., SAE sparsity level, dataset etc.) and their sensitivity are not well explored; it is unclear how robust the reported concept structure for the 3 tasks is to these design choices.\n-  While task specialization is interesting, it remains correlational; it’s unclear whether manipulating the discovered concepts truly affects task behavior. I think some causal manipulation could be interesting too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "irpm15UdKb", "reviewer": "Reviewer_F23h", "summary": "In this work, the authors analyze the internal representations of the DinoV2 model to provide interpretability insights into how the model performs tasks. They begin by characterizing tokens using a sparse autoencoder to create a concept dictionary. They then investigate how different tasks use this dictionary to accomplish their distinct goals and highlight their distinct differences. For example, they find that classification uses elsewhere concepts while segmentation uses boundary detectors. Secondly, the authors find that the representations are partially dense demonstrating behavior that may be inconsistent with sparse coding. Finally, the authors propose a new hypothesis called the Minkowski Representation Hypothesis which they suggest as an alternative to the LRH. They provide initial evidence for the hypothesis, then leave room for investigations in the future.", "questions": "One weakness of this paper is that it primarily is descriptive as opposed to providing an understanding of why each task has such different utilization of concepts. Can the authors describe why they believe this asymmetry is happening? Is it relative complexity between the task types etc?\n\nThe authors suggest that Dinov2’s representations follow their proposed Minkowski Representation Hypothesis. One thing that isn’t clear is whether the authors believe that other networks should follow this hypothesis. It seems to stem very naturally from attention layers. Do the authors expect other networks to follow this hypothesis? If so, when do they expect it to emerge over LRH or do they consider it an alternative hypothesis to LRH in general?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper demonstrates clear novelty and depth while providing a useful tool for other researchers to investigate further. These qualities constitute a clear accept.", "weaknesses": "The analysis in this paper is mostly descriptive as opposed to describing why different tasks within DINOv2 have different representational sparseness etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wthlnE2UJ0", "reviewer": "Reviewer_PH2n", "summary": "This paper introduces a 32k-concept dictionary extracted from DINOv2 using Sparse Autoencoders (SAEs) and presents an interactive web-based visualization platform. The authors analyze how downstream tasks selectively recruit concepts from this dictionary, finding that classification relies on \"Elsewhere\" concepts implementing object negation, segmentation uses boundary detectors forming coherent subspaces, and depth estimation draws on three families of monocular cues (projective geometry, shadows, and frequency transitions). The work proposes the Minkowski Representation Hypothesis (MRH) as an alternative to the Linear Representation Hypothesis (LRH), suggesting that token embeddings behave as sums of convex regions around archetypal landmarks rather than as sparse combinations of near-orthogonal directions. However, the authors acknowledge that MRH remains a working hypothesis requiring further empirical validation.", "questions": "- Sec. 3: what's the reason for isolating the top 100 most task-aligned concepts per-head when analyzing their similarities?\n- Sec. 3: how do you defined the top concepts for classification and how important or frequent is the \"elsewhere\" concept? why only this concept is analyzed for the classification task?\n- Sec. 3: you cite Fig.10 (right) in appendix but the same can be seen in Fig. 2\n- Is \"the largest interactive interpretability demo\" (line 112) an advantage? isn't complexity a problem for interpretability?\n- What's the x-axis of Fig.11 left?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**Clear introduction and methodological foundation**: The paper provides adequate background on ViTs, DINOv2, vision explainability, SAE adoption, and task-specific learned concepts, making the work accessible to the broader audience.\n\n**Interpretability contributions**: The interpretation sections are particularly well-executed, and the release of a web-based visualization platform for navigating DINOv2's 32k-concept dictionary represents a valuable resource for the community.\n\n**Technical rigor**: The submission is mostly clear, technically correct, and results appear reproducible given the extended mathematical explanations, despite code not being provided.\n\n**Novel theoretical perspective**: The paper proposes MRH as an interpretability framework for understanding ViTs, suggesting future research should examine these models through this geometric lens.", "weaknesses": "**Limited scope of downstream task exploration**: Section 3 examines only three downstream tasks, while DINOv2 supports many more applications. For classification, only the \"elsewhere\" concept is explored in detail, limiting the generalizability of findings.\n\n**Insufficient empirical foundation for MRH**: The authors propose MRH based on intuitions from task-specific geometrical organization but acknowledge it still needs rigorous proof (line 445). Given the limited number of tasks and task-specific concepts explored (see point above), the theoretical proposition rests on unstable empirical grounds.\n\n**Underdeveloped discussion of implications**: The paper lacks sufficient explanation of what consequences adopting the MRH concept has for interpreting ViTs and what concrete benefits this perspective provides to the research community. The discussion section should match the verbosity and clarity of the introduction.\n\n**Minor:**\n- Misleading characterization of SAE contribution: The authors state they \"operationalize\" LRH using SAE, when they actually adopt an SAE previously introduced by Fel et al. (2025). This overstates their methodological contribution.\n- Excessive reliance on appendix: Figures in the appendix are cited frequently in the main text (e.g., \"Different tasks recruit different concepts\" in Sec. 3, \"Empirical evidences\" in Sec. 6), resulting in sections that are difficult to follow without constant back-and-forth navigation.\n- Citations should stay in brackets, use \\citet.\n- TL;DR is broken.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QpghnXDpj6", "reviewer": "Reviewer_vpmQ", "summary": "Using a stable sparse autoencoder, the paper builds a large concept dictionary for DINOv2 and shows task-specific feature usage. Analyses of concept statistics and geometry reveal distributed, partially sparse structure that only partially matches the Linear Representation Hypothesis. Finally, the authors propose the Minkowski Representation Hypothesis: token embeddings lie in sums of convex regions (aligned with multi-head attention’s convex mixing), which have implications in mechanistic interpretability such as model steering.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "Clarity:\n- The paper follows a mostly logical narrative structure: Testing LRH with Stable SAE -> Empirical findings on concept statistics and geometry -> MRH\n- Once released, the interactive concept explorer can make the work much easier to visualize.\n\nQuality:\n- The methodology is rigorous, relying on quantitative measures for its major claims (singular-value spectra, coherence vs Grassmannian/random baselines, etc).\n- The convex-hull constraint on the SAE makes it more reproducible by keeping concepts in-distribution. \n\nOriginality:\n- While concepts such as superposition and polysemanticity have been widely explored in mechanistic interpretability, the idea of convex regions is quite novel in this area, to the best of my knowledge.", "weaknesses": "l161: Because the stable SAE constrains dictionary atoms to the convex hull of activations, convexity is guaranteed for the learned concepts but not necessarily for the native representation. Maybe explicitly separate SAE-induced claims from “model-native” claims, and add ablations with unconstrained SAE, showing the same qualitative geometry without the convex prior.\n\nl064: The importance definition is clear, but the figure mapping isn’t. Maybe specify exactly what dot size and color encode in Figure 1. Adding a concise legend in the main text, a one-line formula linking $\\phi$ to the plotted quantity would make the visualization reproducible and easier to see.\n\nl259: UMAP is fine for intuition, but it shouldn’t be used to make claims. Maybe caution that UMAP as visualization only, add a brief hyper-parameter sensitivity (n_neighbors, min_dist, seed) in the appendix, and keep the main analysis on PCA/spectral metrics.\n\nl293: \"dense Sun et al. (2025) activations\" -> \"dense activations Sun et al. (2025)\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)