Register and [CLS] tokens induce a decoupling of local and global features in large ViTs
Authors: Alexander Lappe
Martin A. Giese
Venue: NeurIPS 2025 poster ()
Keywords: Vision Transformer
Register Token
Primary area: applications
TL;DR: Register tokens denoise attention maps but do not make them more interpretable.

Abstract: Recent work has shown that the attention maps of the widely popular DINOv2
model exhibit artifacts, which hurt both model interpretability and performance
on dense image tasks. These artifacts emerge due to the model repurposing patch
tokens with redundant local information for the storage of global image information.
To address this problem, additional register tokens have been incorporated in which
the model can store such information instead. We carefully examine the influence
of these register tokens on the relationship between global and local image features,
showing that while register tokens yield cleaner attention maps, these maps do
not accurately reflect the integration of local image information in large models.
Instead, global information is dominated by information extracted from register
tokens, leading to a disconnect between local and global features. Inspired by these
findings, we show that the [CLS] token itself leads to a very similar phenomenon
in models without explicit register tokens. Our work shows that care must be taken
when interpreting attention maps of large ViTs. Further, by clearly attributing
the faulty behavior to register and [CLS] tokens, we show a path towards more
interpretable vision models.

Register and [CLS] tokens induce a decoupling of local and global features in large ViTs
Alexander !. Martin A. Giese'
Hertie Institute, University Clinics Tübingen IIMPRSII al alexander. Lappeeuni-tuebingenn
Abstract
Recent work has shown that the attention maps of the widely popular DINOv2 model exhibit artifacts, which hurt both model interpretability and performance on dense image tasks. These artifacts emerge due to the model repurposing patch tokens with redundant local information for the storage of global image information. To address this problem, additional register tokens have been incorporated in which the model store such information instead. We carefully examine the influence these register tokens on the relationship between global and local image features, showing that while register tokens yield cleaner attention maps, these maps do not accurately reflect the integration of local image information large models. Instead, global information is dominated by information extracted from register tokens, leading to a disconnect between local and global features. Inspired by these findings, we show that the [CLS] token itself leads very similar phenomenon in models without explicit register tokens. Our work shows that care must be taken when interpreting attention maps large ViTs. Further, by clearly attributing the faulty behavior to register and [CLS] tokens, we show path towards more interpretable vision models.
Violation of the patch integration assumption
Patch features
Global representation
Self-attention layers
[CLS] attention
Figure 1: The global image representation computed by Vision Transformer is usually understood as a weighted average of the local patch features, where weights are given by the [CLS] token attention scores. This notion, which we refer to as the patch integration assumption, underlies attribution methods like plotting the attention maps to identify patches that strongly contribute to the global output. In this work, we show that both register tokens and the [CLS] J token lead to violations of this assumption in large ViT models.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
In recent years, the application of the Transformer architecture 1 to computer vision has given rise to powerful, highly general feature extractors which can be used for numerous downstream vision tasks. These models are pretrained on large datasets in either supervised [3
4 S or self-supervised 7 9 [0] fashion, and then fine-tuned or used directly as uff-the-shellf feature extractors. A useful property of the Vision Transformer architecture that it simultaneously extracts global and dense local features, making it suitable for both classification, as well as more fine-grained tasks such as object detection [T, segmentation [2] or depth estimation 10 13 Further, the fact that the global image representation is computed as convex combination of the dense features enables direct comparisons between patch and global representations [14] [6]. The weights used for this convex combination are the attention outputs of the global [CLS token to the patch tokens. Caron et al. 7 popularized the investigation of the attention scores as a way of attributing model outputs to image regions, as can seemingly clearly identify the patches from which the global information is extracted. After the publication of the now widely used DINOv2 model [1O). Darcet et al. [T7] showed that the model's attention maps exhibit artifacts with very large weights. Upon further investigation, they noted that these artifacts also appear in other models trained with different training strategies, and that the patch tokens corresponding to the artifacts contain global image information rather than information about the image patch as originally assumed. Since this phenomenon made the attention maps less interpretable and degraded the model's performance on dense tasks, the authors proposed the introduction of additional register tokens in which global information can be stored, to prevent it from being encoded in the patch tokens. The inclusion of such register tokens achieved the desired effect of removing artifacts from the attention maps. Further, while there has been some debate on how informative attention maps are in general [8 19,, the resulting attention maps have also been argued to be more interpretable [17 (2). Subsequently, further work proposed alternative ways of including register tokens in the model without repeating the expensive pretraining stage 20 21).
However, the encoding of global information in the patch tokens points towards a deeper underlying problem. Even if this information is stored in register tokens instead, it is unclear whether the denoised attention maps over the patch tokens actually reflect the global model output faithfully. Since evidently some global information is integrated before the last layer and can flow from the register tokens to the [CLS] token, it must be studied to which degree the patch features still contribute to the final output. In this work, we show that in large models, the global image representation relies primarily on information extracted from the register tokens, which needs to be considered when studying the denoised attention maps. This effect does not occur in the smaller model variants, which display a tight correspondence between local and global features. Inspired by this finding, we examine whether a disregard of the local patch features can also occur in overparameterized models without explicit register tokens. We observe that the [CLS] token itself may be interpreted as register and show that the implicit backward-attemi attentioon mechanism introduced by the residual connection leads to the same effect.
Contributions. Our results show that the intuitive correspondence between global and local image representations holds in smaller variants of the ViT trained with DINOv2, but breaks down for larger models. Importantly, the last-layer attention maps do not faithfully represent the mechanism through which global image representations are formed in large DINOv2 models with register tokens or residual connections in the [CLS] J token. These effects need to be taken into account when studying attention maps to attribute model outputs to image regions, ground global feature representations in local patch features. Our findings suggest that future iterations of generalist vision models should be built without register tokens and residual connections in the [CLS] ] token to ensure model interpretability.
2 Preliminaries
Vision Transformers. Given an image x E Rhxaxx3, vanilla Vision Transformer divides it into small patches and feeds these patches through network of self-attention layers. This process yields a feature map z E RR xp2xd, where pı and p2 denote the spatial dimensions and the model's feature dimension. To learn a global representation of the image, an additional [CLS] token is introduced, which is treated exactly like the image patch tokens. The output of the [CL.S] ] corresponds to the global image embedding. Mathematically, the self-attention mechanism for
token at the last layer
2

---

single attention head works as follows: The [CLS] J token outputs a query vector q E I which is compared to the keys output by each token denoted by ki.... kp IR to yield the attention vector
a = softmax ((q, ki),..., (q. kpipu+)))
(1)
The output of the attention mechanism for the [CLS] J token is then given by the convex combination
Ocla := Σ a.V
where v, denotes the value vector of the i-th token. The final processing after the attention mechanism differs slightly between models. In the DINOv2 model we consider here, the attention is followed by a residual connection, layer normalization and shallow multi-layer perceptron yield the final output. As the residual connection is applied after self- attentionnoo and does not affect the features extracted via the attention map, we exclude it from the computations when comparing contributions of register tokens and patch tokens in Section 3 separately Section
We treat the influence of the residual connection
The patch integration assumption. Vision Transformers are therefore usually understood patch feature extractors, where the [CLS] token learns to select the final features from the most relevant patches for global image understanding. We refer to this notion the patch integration assumption. Based on the fact that the [CLS] ] token undergoes the same transformations as the patch tokens and can be written as a weighted average thereof, the patch integration assumption has been used to attribute global model behavior to specific image regions [14] 13 16]. In particular, the attention vector of the [CLS] token is popular object of study to determine the patches that the model relies on for its global image representation 2 7 22 23 24]. A body of work on interpretability of ViTs has suggested alternative methods for attributing model behavior, going beyond studying the [CLS] attention scores 255 26 27 28 However, only Darcet et al. [17] challenged the patch integration assumption itself by observing the emergence of high-norm tokens in large models.
Transformers with register tokens. As mentioned previously, Darcet et al. 17 found that very large models trained with DINOv2 store global image information patches that otherwise contain redundant information, degrading the quality of the dense patch features. As a remedy, they introduced additional register tokens that are appended to the patch tokens just as the [CLS] token. These are supposed to be used for storing global information to avoid the use of the patch tokens themselves for this purpose. In this case, the output of the [CLS] token becomes
Σ
a
ajVz.
iE (patches)
jE (registers)
patch contribution
register contribution
Decoupling of patch and global features. Darcet et al. [1 show that the resulting attention maps of DINOv2 models including registers are less noisy, and the resulting patch features show better performance on dense prediction tasks. However, it is possible that the inclusion of registers decouples the global image embedding from the local patches, as the [CLS] J token is no longer convex combination of only the local features. In case the [CLS] token attends primarily to the register tokens, the attention maps on the patches, while shown to be less noisy, might be uninformative due to dominance of non-local features. Further, if global information is obtained from the registers instead of integrating information from the patches, feature spaces between local and global representations might not be aligned, hurting the ability precisely specify the local grounding of global information
3
The influence of register tokens on the patch integration assumption
3.1 Larger models attend more to register tokens and less patch tokens
High-norm patch tokens have so far only been observed in the larger variants of the ViT architecture. This raises the question of whether only larger models rely heavily on additional register tokens. To answer this question, we test how much attention the last-layer [CLS] token places the register
3

---

a) Attention of [CLS] token register and patch tokens
b)
Pre-Layer Norm Activations
Value
1000
0.8
Attention
0.6
50
100
Pre-Norm Feature rank
0.4
Post-Layer Norm Activations
0.2
Value
small
base
large
giant
Model
50 Pre-Norm Feature rank
Register tokens
Patch Tokens
Figure 2: a) The amount of attention placed by the [CLS token the last layer onto the patch and register tokens, respectively. Smaller models attend primarily to the patch tokens, whereas bigger models attend more strongly to the register tokens. b) Mean activations of the highest-nore register token in the last layer of the 'giant' model. The 100 dimensions with highest activations before the layer norm are shown. Register tokens show large activations a small subspace, making them measured by pairwise cosine similarity (top panel). However, the
seemingly image- sange-independent layer norm downscales these dimensions before the self-attention mechanism (bottom panel).
and patch tokens, respectively. We study the DINOv2 models with register tokens as published by the authors huggingface.com and probe them using the validation images of the MS COCO dataset [30]. We study the 'small', "base", 'large' and 'giant' models, which differ in number of self-attention layers, hidden dimension and number of attention heads. They have 21M, 86M, 300M, and 1, 100M total parameters, respectively. We extract the post-softmax attention vectors of the (CLS) token as presented in Eq. (3). average them across attention heads and then sum the resulting scalars for the register tokens and the patch tokens respectively, to examine how attention is partitioned between the two types of tokens, For each image, the sum of register attention and patch attention is one. Results are shown in Fig. 2
We observe clearly that smaller models attend primarily to patch tokens, and the 'large' and 'giant' variants place more attention on the register tokens. Strikingly, the variance of preference for either register or patch tokens in the largest model is considcrable, with some images attending mainly to patch tokens, and some almost ignoring them entirely.
3.2 Information register tokens is only seemingly image independent
Image dependency high-norm tokens. The role of information encoded by high-norm tokens in the original DINOv2 model is not entirely clear. Darcet et al. [17] trained linear probes on the high- norm tokens, concluding that they contain more global information and less positional information than the regular patch tokens. On the other hand, Wang et al. [31 claimed that high-norm tokens are image independent and can predicted by the first singular vector of linear approximation of the attention layer itself. They support these findings by noting that the pairwise cosine similarity between high-norm tokens different images is extremely high. This conclusion raises the question of why the [CLS] token of the original DINOv2 model would be subject to image-nddependenden information. By studying representations within the self-attention layer, we first reveal that these previous findings also apply to the register tokens, corroborating the assumption that they indeed take over the role of high-norm patch tokens. Further, we show that the previous results are not at all at odds, and that high-norm register tokens do carry image-dependent information.
Influence of layer norm. To understand in how far register tokens encode image-dependent information, we study their hidden states at the penultimate layer of the "giant' model. The penultimate layer the most relevant, since the final [CT.S] output is formed by attending to these representations. First, note that the hidden states of the register tokens do have abnormally high norm, and that the

---

a)
CKA with global representations
b)
One-shot classification accuracy
1.00
Register tokens
Register tokens
0.75
Patch tokens
Accuracy
0.6
Chance level
Patch tokens
CKA
0.50
0.4
0.25
0.2
0.00
0.0
small
base
large
giant
small
base
large
giant
Model
Model
Figure 3: a) Centered kernel alignment between the global [CLS] token output, and [CLS] token output computed while only attending to either register tokens or patch tokens. Patch tokens yield faithful representation of the global output for smaller models, but the connection between local and global features breaks down with increasing model size. One-shot classification accuracy on the 1000 Imagenet classes, The classifier is trained the global [CLS] output and then tested on output based on patch and register tokens, respectively. Attending only to the patch tokens yields poor performance in the larger models, corroborating the finding that global representations are not formed by attending to the patch features.
same token has the highest norm among all tokens for any input image by order of magnitude. Second, we observe that the finding from Wang et al. [31], that activations are very similar across images, hold for the register tokens as well. The average pairwise cosine similarity between images in the highest-nont token is 0. 0.9989, seemingly confirming that these tokens are image independent. However, studying the feature activations more closely shows that this need not be the case. Fig. shows the top-100 100 feature dimensions of the highest-nore token (features with highest activation), averaged across all images. The plot clearly shows that the vector is dominated by extreme outliers which in turn dominate the cosine similarity. This leads us to two important points: First, note that the hidden state itself does not appear the self-attention mechanism via a dot-product comparison. Instead, keys and values, which are compared via the dot product, are computed linear maps of the hidden state, allowing model ignore these dimensions and focus on potentially image dependent dimensions instead. This mirrors exactly the methodological differences between [17] and [3I], demonstrating why the authors came to opposite conclusions. Further, before entering the self-attention mechanism, the VIT applies a layer norm to all tokens, thereby reweighting the features and downscaling high-norm tokens. Fig. 2 shows the same data after applying the layer norm, showing that outlier dimensions scaled down before computing self-attention.
3.3 Registers tokens dominate global image representations in large models
So far, we have shown that larger models attend strongly to register tokens, and have discussed how register tokens can encode -pegesspecific information. It remains to study how strongly the register tokens actually influence the output of the self-attention mechanism. After all, the information extracted by register tokens might be redundant w.r.t. to the patch tokens, or their value vectors could have low norm, leading weak contribution despite large attention.
To examine the influence of patch tokens and register tokens on the [CLS] output, we constrain the final self-attention layer attend only one of the token types. Specifically, attentions to one of the two token types to zero, yielding a [CLS] output that is based only on the other. We also compute the unaltered model output based all tokens and investigate the similarity of the resulting global representations using linear centered kernel alignment (CKA) [32). Given two (mean-centered) matrices of neural network activations X E [aax, Y ng computed on n test samples, linear CKA is defined as
all
tr (XXTYTT)
CKA(X,Y) VIXXXXXXXXX((XXXXX))
and can interpreted as measuring the alignment of pairwise similarities between the two matrices. Fig. a shows the CKA between the unaltered output, and the patch-based and register-brat based output,
5

---

respectively, CKA was again computed using the samples from the MS COCO test set. We observe that for the two smaller models, the patch-based representation shows toose-to-perfect alignment with the [CLS] output. This indicates that the notion of the global output being an aggregate of last-layer local patch features is accurate. However, with increasing model size, the global output and the patch-based output become highly disconnected, and outputs based on register tokens represent the global output more faithfully in the giant model. The same analysis for the Imagenet test set 33 is presented in the appendix, mirroring the qualitative results. Fig. 3 b) displays one-shot
top-5 classification accuracy when training the classifier on the global [CLS] output and evaluating on either the patch- basch- register- based output. Results were generated by randomly sampling one training image and one test image per class from the 1000 classes in the Imagenet test set. In accordance with the results from panel a), we observe that classifiers relying only on patch features perform poorly for the two larger models. This demonstrates that class-relevant global information is nof extracted from the patch features.
3.4 Attention maps with registers are clean but do not reflect global image representations
The original purpose of including register tokens the DINOv2 model was to remove high-norm tokens from the image patches to obtain better dense representations and cleaner attention maps. However, Fig. casts
castss doubt on how informative the cleaned-up attention maps are if the patch features extracted via the attention maps do not accurately represent the global model outputs. We show an example in Fig. 4 As expected, we observe that the attention map output by the giant model with registers is visually cleaner than the one output by the vanilla model. Next, we examine how faithfully the features extracted according to this attention map represent the total layer output by computing the cosine similarity between the extracted patch features, and the total layer output including register tokens. The cosine similarity is -0.0092, meaning that the features extracted according to the attention map are completely orthogonal to the features extracted by the [CLS] token when including the register tokens. While not all images elicit disconnects of this magitude, the example demonstrates the violation of the patch integration assumption in large models with registers, showing that its attention maps should not be relied on.
At the global level, let us consider the connection between the attention maps over patches and the CKA results shown in Fig. 3 The purpose of the [CLS] token attention is to extract global image features that facilitate downstream tasks. Relevant for task performance is how images embedded relative to each other, which, as is done in CKA, can be measured by the Gram matrix XXT, where X E Rxd contains the feature representations of a of test samples. This Gram matrix encapsulates the similarity structure of learned image representations. Writing X = X, + X, as the sum of patch contributions and register contributions Eq. (3) we can decompose the Gram matrix as
(5) The attention maps only give insight into which patches are relevant for computing the patch-based representation and thus embedding an image into the representational geometry determined by X..X..
XXT = + X, + X + X
However, Fig. 3 shows that the geometries of X and XXT are misaligned, indicating that the attention map does not yield sufficient insight into how an image embedded in the models's 's global representational geometry.
3.5 Connections to overparameterization and neural collapse
The analysis of Darcet et al. [17 showed that high-norm patch tokens only appear in the larger variants of the ViT architecture. Our results on register tokens are similar, showing that both (CLS]tokep attention to registers, as well as their influence on the global image representations increase substantially with model size. These findings relate to literature on how representations are formed in overparameterized models. Work on neural collapse (34] demonstrates that models with large capacity tend to learn simple, very low-dimensional representations at the last layer and that such representations yield better performance [35] 36 37) and generalization abilities (34). Further, it has been shown for overparameterized classification models, that low-intrinsic dimensionality as well as linear separability of classes emerge already before the final layer findings with this body of work, we hypothesize that larger VIT variants are overparameterized to the point at which DINOv2 training yields simplistic representations at the last layer, relying on a small number of tokens in which global information has already been integrated. The introduction of
[38 39 Integrating our

---

DINO
DINOv2-giant
DINOv2-giant with registers
Figure 4: Attention maps of the final token A convex combination of the corresponding patch features yields the patch-based contribution to the global image representation. As noted by Darcet et al. (17), the attention map of the DINOv2 model exhibits large artifacts. These are removed by including register tokens in the model, seemingly leading to a more interpretable attention map. However, when computing the [CLS] output based on the convex combination of patch features in the model with registers, its cosine similarity to the total output of the final layer is -0.0092. In other words, attending the patch tokens yields representation completely orthogonal to the one including the register tokens, showing that the attention map fails to attribute global information to image patches.
register tokens merely shifts this mechanism outside of the patch features, preserving simple last-layer representations in the register tokens and thus still allowing the model to violate the patch integration assumption.
The influence of the [CLS] token on the patch integration assumption
4.1 Very large models attend primarily to features from the skip connection
So far, we have discussed how register tokens lead disconnects between local and global image representations and thereby the violation of the patch integration assumption. We have hypothesized that this behavior is due to overparameterization, allowing the model solve image-level training objective before the final layer, and storing the information in the register tokens. This line of reasoning motivated an inquiry into the mechanisms through which similar behavior could arise in overparameterized models without explicit register tokens. We have already discussed how the original DINOv2 model adapts patches containing little local information to instead store global information. In this section, we explore another mechanism that allows the model to ignore patch information the last layer, namely residual connections within the [CLS] token.
Attending to the previous layer. Previous work has shown that overparameterized networks may form more less final representations of the input relatively early in the model hierarchy 38 39]. Since the [CLS] token in the VIT is appended to the patch tokens before the first layer, the standard ViT already contains one register token, which can be used to store emerging global information. Importantly, recent implementations of ViTs include skip connections within the attention block, resulting the [CLS] token output
(cas)) = Σ
a.V; +
prev CLS]
a.V; + A(CLS) V(CLs] +
prey CLS
IE [patches)
skip contribation
patch contribution
non-ppacch contribution
meey where [ccss] denotes the output of the [CLS] token the previous layer, and a(cus) and V(cus) denote the attention value to the [CLS] J token and value vector, respectively. This allows the output to attend to the [CLS] token via two different mechanisms: The first is part of the standard self-attention as given in equations (T) and (2). The second is given implicitly through the relative scales of the contributions of the attention mechanism and the skip connection the output O given in Eq. (6). These relatives scales depend on the model parameters; therefore the model learns during training how strongly to attend to the skip connection.

---

a)
Norm of skip connection and attention output at last layer
Similarity of CLS token to last layerl CLS) token
b)
1.0
Skip features
giant
0.8
Patch features
Cosine similarity
0.8
large
base
Norm ratio
0.6
0.6
small
0.4
0.4
0.2
0.2
0.0
small
base
large
giant
0.00
0.25
0.50
0.75
1.00
Model
Normalized Depth
Figure 5: a) After the self-attention mechanism, skip connection sums the attention output and the hidden states from the previous layer, providing an alternative way for the (CLS) token to attend to itself. Since the attention weights are given only implicitly, we plot the L2-norm of the contributions of the skip connections and the patch features to the [CLS] token as a proxy. We observe that "giant' model, the global output is primarily determined by previously computed features, rather than the patch features. b) We show the cosine similarity of the [CLS] token at all model layers to the [CLS] token at the last layer. The three smaller models exhibit large jump the very last layer, indicating that the [CLS] token at the last layer is strongly influenced by the patch tokens. Conversely, the [CLS] token of the 'giant' model converges the final output more smoothly, explaining its low reliance on the patch features at the last layer.
Measuring attention to patch features and skip features. To examine how strongly mod- els without register tokens attend to the skip connection, we study the original implementa- tion of DINOv2 without registers, as well as DeiT III O for supervised model, CLIP 3 for language supervision, and iBOT 9 For
CKA with global representations
1.0
0.8
CKA
0.6
readability we present findings for DINOv2 in main text and show the generality of the find- ings using the other models in the Appendix. Since attention weights for the skip connection not computed explicitly, compute proxy the norms the patch contribution and the skip contribution from Eq. (6). The results of this analysis on the MS COCO vali- dation set shown in Fig. 5 We observe that the three smaller models attend to both patch and skip features, but the 'giat' model output is strongly dominated by the skip features. Panel b) sheds further light this phenomenon: The similarity of the [CLS] token of the last layer and all previous layers is low for the smaller models, indicating that the final (CLS) output determined by integrating the final patch fea- tures. This is in line with the patch integration assumption. However, the structure of the out- of the "gint' model emerges earlier, and the penultimate output is already very similar to the final one. Therefore, the last layer receives the quasi-final output through the skip connection and only needs to attend weakly to patch features to compute small corrections. We show that other VIT models trained with different recipes mimic this behavior in the Appendix (Fig
0.4
0.2
0.0
small
base
large
giant
Model
Skip features
Patch features
Figure 6: Centered kernel alignment between the standard [CLS] token output, and [CLS) token output computed using either only patch token fea- tures the skip connection features from DINOv2. We observe that the findings from Fig. 5 that the gi- ant model attends primarily to the skip connection. does indeed lead disconnected representations between global features and patch features. This shows that the skip connection for the [CLS] token by itself lead to violation of the patch inte- gration assumption.

---

Earlier, we found for models with register tokens that large attention to registers leads to disconnect between local and global image features. Does the same hold for large attention skip features? To answer this question, we compute the centered kernel alignment between the global model output, and model output based only on patch or skip features, respectively. The results are shown in Fig. [ (results for other models are presented in the Appendix, see Fig. [). Again, we find that for the larger models, the features extracted from the patches do not yield a faithful representation of the global model output. We conclude that the skip connection of the [CLS] token indeed disconnects the global output from the patch tokens in the larger models.
5
Discussion
tight correspondence between local and global image features computed by Vision Transformers is desirable for both interpretability and tasks combining local and global image information like object detection. For smaller DINOv2 models, including the widely used 'base' model, we found no evidence for disconnects between local and global features which is in line with previous findings regarding high-norm tokens [7]. On the other hand, we have observed that the global output of 'giant' model variants does not correspond to the final patch features extracted via the self-attention mechanism in models with registers, making their attention maps unreliable. Therefore, we hypothesize that these disconnects arise as consequence of overparameterization, which gives models the flexibility to integrate global information already in intermediate layers. Combining results from Darcet et al. with the work at hand, we identify three mechanisms through which the patch integration assumption may break down in overparameterized models. First, patch tokens can be repurposed to store prematurely emerging global information. Second, an approach of alleviating the first issue based on introducing register tokens seems to only move the problem outside of the patch tokens, still allowing the model to disregard patch features. Third, the skip connection in the self-attention layer enables the [CLS] token to extract image representations gradually, resulting in the fact that the last attention layer does not extract features that accurately represent the final output.
Future design choices. We have shown that while register tokens remove artifacts from the patch representations, they do not fully solve the problem of degenerate attention maps as they dominate the final attention-layer's output. This effect is particularly strong for the [CI.S] ] token itself, which can be interpreted as register token with an especially strong bias to attend to itself via the residual connection. We therefore argue that models that satisfy the patch integration assumption should be built without register tokens and hidden-layer [CLS] ] tokens. Another approach might be explicit regularization, although experiments on DeiT III show that drop-path regularization on it's own does not prevent the violation of the patch integration assumption. In any case, it remains to ensure that global information is not stored in repurposed patch tokens. Here, approaches that impose direct regularization on the patch tokens appear to be the most promising. Along those lines, Wang et al. 31 proposed enforcing patch tokens to have similar representations as their neighbours, effectively removing high-norm patch tokens without the introduction of an additional mechanism for storing global information. The authors test their approach by finetuning on a smaller dataset and show that the resulting model performs well dense tasks.
Limitations and future work. Our analysis focuses on diverging global encoding geometries between register tokens and patch tokens. Since the variance over samples with respect to the registers is high (see Fig. 2 future work could study the properties that lead image to be processed primarily by patch/register tokens. As we have seen that centered kernel alignment between register tokens and patch tokens is low, a systematic analysis of the respective directions of variance in the representations could shed further light on the effects we have discussed. Since both attention to the skip connection and attention to explicit registers emerge in larger models, a more focused inquiry into the interdependence of these effects would be interesting. Finally. it remains to study the impact of violations of the patch integration assumption on performance on global image tasks. As mentioned in Section 3.5, previous work has shown relationship between simplistic last-layer embeddings and model performance. A comparison of complexity-matchnt to satisfy the patch integration assumption and models with registers is necessary for determining whether there is a tradeoff between image-level task performance and correspondence of local and global features.
models that are forced

---

Broader impacts. We do not foresee malicious use of our work, or tangible potential negative impact as a whole. On the contrary, we hope that our contribution will aid in better understanding of foundation vision models.
6
Acknowledgements
AL and MG are supported by ERC-SyG 856495. MG is supported by HFSP RGP0036/2016, BMBF FKZ 01 GQ17Q44 The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Alexander Lappe. Finally, we are grateful to Vojta Smekal, Marta Poyo Solanas, Prerana Kumar, and Lucas Martini for valuable discussions and support.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[2] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021.
[3] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 8748 8763. PMLR, July 2021.
[4] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. Proceedings of the 38th International Conference on Machine Learning, pages 4904-4916. PMLR, July 2021.
[5] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, pages 10347 10357, PMLR, July 2021.
[6] Hugo Touvron, Matthicu Cord, and Hervé Jégou. DeiT III: Revenge of the ViT. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022, volume 13684, pages 516-533. Springer Nature Switzerland, Cham, 2022. ISBN 978-3-031-20055-- 978-3-031-2005332 doi: 10. (007/978-3-031-200533 30.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self- SuperSuue Vision Transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, October 2021, doi: 10. 10.1009/CCV49922.2001 00951.
[8] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. 2022 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 15999-15988, June 2022. doi: 10. 10.1109/CVPR526888..8 2022.01553.
[9] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBOT: Image BERT Pre-Training with Online Tokenizer, January 2022.
[10] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec. Vasil Khalidov, Pierre Fernandez. Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby, Mido Assran. Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research, July 2023. ISSN 2835-8856.
[11] Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon W Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing Objects with Self sulrvviue Transformers and no Labels. In Proceedings of the British Machine Vision Conference 2021, page 365, Online, 2021. British Machine Vision Association. ISBN 978- 177-1-998 1-910725-68-1 doi: 10.5244/C.35.665.
10

---

[12] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C. Berg, Wan-Yet Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything, April 2023.
[13] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision Transtormer Adapter for Dense Predictions. The Eleventh International Conference Learning Representations, September 2022.
[14] Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay Attention Your Neighbours: Training-Free Bven-Vocabulary Semantic Segmentation, September 2024.
[15] Chong Zhou, Chen Change Loy. and Bo Dai. Extract Free Dense Labels from CLIP. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision ECCV 2022, volume 13688, pages 696-712. Springer Nature Switzerland, Cham, 2022. ISBN 978-3-031--181444 978-3-031-19815-11 doi: 10. 1007/997 3- 19815- 1_4..
[16] Andrew F Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M Henderson, Leila Wehbe, and Michael Tarr. BRAIN MAPPING WITH DENSE FEATURES: GROUND- ING CORTICAL SEMANTIC SELECTIVITY IN NATURAL IMAGES WITH VISION TRANSFORMERS. In ICLR, 2025.
[17] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision Transformers Need Registers, April 2024.
[18] Sarthak Jain and Byron C. Wallace. Atlention is not Explanation, May 2019.
[19] Sarah Wiegreffe and Yuval Pinter. Attention is not not Explanation. September 2019.
[20] Nick Jiang, Amil Dravid, Alexei Efros, and Yossi Gandelsman. Vision Transformers Don't Need Trained Registers, June 2025.
[21] Yinjic Chen, Zipeng Yan, Chong Zhou, Bo Dai, and Andrew F. Luo. Vision Transformers with Self- Disttilitlld Registers, May 2025.
[22] Lennart Brocki, Jakub Binda, and Neo Christopher Chung. Clas--Discrmminative Attention Maps for Vision Transformers. Transactions on Machine Learning Research, August 2024. ISSN 2835-8856.
[23] Yaodong Yu, Tianzhe Chu, Shengbang Tong, Ziyang Wu, Druv Pai, Sam Buchanan, and Yi Ma. Emergence of Segmentation with Minimalistic White-Box Transformers, August 2023.
[24] Matthew Walmer, Saksham Suri. Kamal Gupta, and Abhinav Shrivastava. Teaching Matters: Investigating the Role of Supervision Vision Transformers. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7486 7496, Vancouver, BC, Canada, June 2023. IEEE. ISBN 9798350301298 doi: 10. 10.1109//V/R52222,... wtt.:/22229..22.007777 2023.00723.
[25] Bill Psomas, Toannis Kakogcorgion, Konstantinos Karantzalos, and Yannis Avrithis. Keep It SimPool: Who Said Supervised Transformers Suffer from Attention Deficit?, September 2023.
[26] Hila Chefer, Shir Gur, and Lior Wolf. Transformer Interpretaiility Beyond Attention Visualization. In 202/ IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 782-791, Nashville, TN, USA, June 2021. IEFF. ISBN 978-1-66554-509-2. doi: 10.1 109//VV9444433.222211 2001444337.2200008...
[27] Weiyan Xie, Xiao-Hui Li, Caleb Chen Cao, and Nevin L Zhang. ViT-CX: Causal Explanation of Vision Transformers.
[28] Alexandre Englebert, Sédrick Stassin, Géraldia Nanfack. Sidi Ahmed Mahmoudi, Xavier Siebert, Olivier Cornu, and Christophe De Vleeschouwer. Explaining through Transformer Input Sampling, In 2023 IEEE/CVF International Conference Computer Vision Workshops (ICCVW), pages 806-815, Paris, France, October 2023. IEEE. ISBN 9798350307443. doi: 10.109//CCV66679320003 www.:/66679332.200003300
[29] Feng Wang, Jieru Mei, and Alan Yuille. SCLIP: Rethinking Self-Attention for Dense Vision-I Language Inference. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision ECCV 2024, volume 15079, pages 315-332. Springer Nature Switzerland, Cham, 2025. ISBN 978-3-031-72263-77 978-3-031-7226444, doi: 10.10077978-3-00 --66--------------4--- 18.
[30] Tsung- Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C, Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision ECCV 2014, pages 740-755, Cham, 2014. doi: 10. 10.10007/78-33 3-9 10602- 1_48.
Springer International Publishing. ISBN 978 3-99---99-0000211

---

[31] Haoqi Wang, Tong Zhang, and Mathieu Salzmann. SINDER: Repairing the Singular Defects of DINOv2, July 2024.
[32] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton, Similarity of Neural Network Representations Revisited. In Proceedings the 36th International Conference on Machine Learning. pages 3519-3529, PMLR, May 2019.
[33] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248 255, June 2009. doi: 10.1 1009/C/PR.209995266660
[34] Vardan Papyan. X. Han, David L. Donoho. Prevalence neural collapse during the terminal phase of deep learning training. Proceedingy of the National Academy of Sciences, 117(40):24552-2466 October 2020, doi: (073//pnas.20155009
[35] Alessio Ansuini, Alessandro Laio, Jakob Macke, and Davide Zoccolan. Intrinsic dimension data representations in deep neural networks. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[36] Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How Far Pre-trained Models Are from Neural Collapse the Target Dataset Informs their Transferability. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pages 5526 5535, Paris, France, October 2023.
IEFE. ISBN 9798350307184 doi: 10.1 10000CCC5510000022200011
[37] Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse Engineering Self- Seff-Supervieed Learning.
[38] Wojciech Masarezyk, Mateusz Ostaszewski. Ehsan Imani, Razvan Pascanu, and Piotr Milos. The Tunnel Effect: Building Data Representations in Deep Neural Networks.
[39] Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso Poggio. Feature learning in deep classifiers through Intermediate Neural Collapse. In Proceedings of the 40th International Conference on Machine Learning, pages 28729-28745. PMLR, July 2023.
12

---

A Supplementary Results
Norm of skip connection and attention output at last layer
DINOv2
DeiT III
Skip features
0.8
0.8
Patch features
Norm ratio
0.6
0.6
0.4
0.4
0.2
0.2
small
base
large
giant
tiny
small
base Model
large
huge
Model
smaller train
CLIP
iBOT
0.8
0.8
Norm ratio
0.6
0.6
0.4
0.4
0.2
0.2
base
large
huge
giant
small
base
large
Model
Model
Figure 7: Attention to the [CLS] token discussed in Section 4 for Vision Transformers trained with different recipes. Note that attention to the skip connection consistently grows with model size. For DeiT III, we also include the tiny model, which trained smaller data set. Interestingly, the tiny model mimics the behavior of the huge one trained on more data, demonstrating the importance of the the relationship between model capacity and data size discussed in the main text.
13

---

CKA with global representations
DINOv2
DeiT III
1.0
1.0
Skip features
Patch features
0.8
0.8
Li
CKA
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
small
base
large
giant
tiny
small
base Model
large
huge
Model
smaller train set
CLIP
iBOT
1.0
1.0
6
0.8
0.8
0.6
0.6
CKA
0.4
0.4
0.2
0.2
0.0
0.0
base
large
huge
giant
small
base Model
large
Model
Figure 8: CKA results for the skip connection as discussed in Section for different pretraining recipes. A disconnect between the local and global image features discussed in the main text occurs across all pretraining recipes with growing model size.
14

---

Similarity of cls token to last layer cls token
DINOv2
DeiT III
1.0
1.0
small
Cosine similarity
0.8
base
0.8
large
0.6
giant
0.6
0.4
small
0.4
base
0.2
large
0.2
huge
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
CLIP
iBOT
1.0
1.0
base
small
Cosine similarity
0.8
large
0.8
base
huge
large
0.6
giant
0.6
0.4
0.4
0.2
0.2
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Normalized Depth
Normalized Depth
Figure 9: Layerwise similarity between the [CLS] token and the last-layer [CLS] ] token discussed in Section 4 CLIP and iBOT mirror the DINOv2 behavior, whereas DeiT III does not exhibit a clear relationship between model size and early [ [CLS] integration. Note that DeiT III is the only model trained with drop-path regularization, which may explain why all model sizes integrate final output information early in the hierarchy.
15

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims in the abstract and introduction are that including register tokens in ViTs detaches the [CLS] output from the patch features, which is shown in the experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The ahstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include dedicated section limitations of work indicating deeper inquiries which are not part the paper. as well as the necessity further, computationally expensive. experiments on models without register tokens.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings. model well- sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice
what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
applicable, the authors should discuss possible limitations of their approach to address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [NA]
Justification:
No theoretical assumptions necessary to support the claims made in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
16

---

model All the theorems, formulas, and proofs in the paper should be numbered and cross reffeeeeferenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All models used in the analysis publicly accessible, and describe where download them in main text. believe that experimental details are sufficient reproduce results. Additionally, we provide code that reproduces the paper's ' results and is inexpensive to run.
Guidelines:
The answer NA mcans that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contrihution or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., the case of a large language model), releasing checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (c.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but should be possible for other rescarchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, as described supplemental material?
the data and code, with sufficient instructions
Answer:
[Yes]
Justification: We provide scripts automatically download the models used and run all analysis an image dataset of choice.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines puttts::/niipen cc/
publ cc/pbl1l11pp11
guides Codee ubmii ssionnsi onioonnooooiooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
17

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips... cc/
public/qui /etpg/ideeeeeeeeeeeeeeeeee
www.emissssssoiisoioooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Datasets and models are described in the main text, training details are not applicable to our work.
Guidelines:
The answer NA that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: When presenting strength of attention register tokens, we report the entire distribution so that the reader can appreciate the high variance. We do not report statistical significance or error bars for our experiments as there is no sampling aspect, except for the one-shot-classficee experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
96% CL. if the hypothesis of Normality errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [No]
Justification: We don't 't explicitly state compute times/resources we believe it is obvious to the field that the experiments shown are inexpensive
18

---

Guidelines:
answer means the paper does not include experiments.
The paper should indicate the type compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics
Answer:
[Yes]
Justification:
We are not aware of any ethics violations regarding this work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
include a short section dedicated to this question.
Guidelines:
The answer means that there is no societal impact of work performed.
the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address socictal impact.
Examples negative socictal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not foresee any malicious applications of our work.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines restrictions to access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make hest faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g.. code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected??
Answer: [Yes]
Justification:
We clearly indicate the authors models and datascts used for our experiments.
Guidelines:
The answer means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset,
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should he provided.
If assets are released, the license, copyright information, and terms of use in the package should datasets.ppape
be provided. For popular datasets, paperr www.twoddeeeooooooo.....oooooooe com/ cat catasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should be provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New assets
Question: Are new assets introduced paper well documented and is the documentation provided alongside assets?
Answer: [NA]
Justification: not provide new assets.
Guidelines:
The answer means that paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
There no experiments including human subjects in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
There are no experiments including human subjects this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- blc), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage of LLMs it important, original, or non non-standard component
core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: There is non-standard contribution of any LLM to this work.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs any important, original, non-standard components.
Please refer LLM policy http::///aauups..c./////eeeeeee/ what should should not be described.
for
21


Reviews:
[{"review_id": "eQisiOzJYy", "reviewer": "Reviewer_qxac", "summary": "This paper investigates the patch integration assumption in Vision Transformers (ViTs), focusing on how register tokens and [CLS] skip connections in large models (e.g., DINOv2) decouple global and local feature representations. Through empirical analysis (e.g., centered kernel alignment, attention distribution probing), the authors demonstrate that register tokens dominate global representations in large ViTs, rendering attention maps unreliable for attribution. The work further links this phenomenon to overparameterization and suggests architectural refinements for interpretability.", "questions": "* **(Q1)** Could the authors' findings hold for *non-DINOv2 architectures* (e.g., DeiT, Swin Transformers) or other pretraining methods (e.g., masked image modeling or supervised pre-training)? If not, how might this limit broader applicability?\n\n* **(Q2)** The authors claim register tokens harm dense-task performance (Sec. 3.5), but provide no metrics. Can the authors benchmark patch-feature quality on tasks like segmentation using standard datasets (e.g., ADE20K)?\n\n* **(Q3)** Instead of removing registers/skips (Sec. 5), could pre-norm / post-norm regularization or drop-path regularization (Sec. 4) preserve interpretability?", "limitations": "The paper acknowledges scope limitations (Sec. 5) and computational/resource constraints (Sec. A). However, methodology limitations (e.g., narrow architecture coverage) require expanded discussion. Suggestion: Explicitly state that conclusions are DINOv2-specific and may not generalize.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\n\nThis work tries to address a critical issue in ViT interpretability and provide recent findings about attention-map artifacts based on DINOv2 and [1]. The authors present several interesting findings based on empirical studies and visualizations. Overall, this manuscript is well-structured with clear visualizations (e.g., Fig. 4 illustrating orthogonal feature spaces).\n\n### **Weaknesses**\n\nThis paper does not provide any new designs but conducts empirical studies with several variants of ViTs using different types of special tokens (e.g., CLS tokens or register tokens).\n\n* **(W1)** This work is not contributive enough to be published as a conference paper. Compared to previous work like DINOv2 with register tokens, this work only provides incremental findings (e.g., skip-connection effects) without transformative solutions. It limits the novelty of this work. Such empirical studies should provide some benchmarks or technique contributions. I recommend that the authors resubmit the manuscript to some workshops or the track of findings, or enhance it with more benchmarks or ablation models.\n\n* **(W2)** Drawbacks in the approaches of the empirical studies. On the one hand, this work relies heavily on the analysis methods used in [1] , e.g., feature similarity metrics (CKA), without downstream task evaluation (e.g., segmentation / detection performance). On the other hand, the authors only evaluated the classical Vision Transformers with contrastive learning (e.g., DINO and DINOv2), how about other similar Transformer architectures and different pre-training methods? Based on the limitations and drawbacks above, the provided conclusion and findings might be limited to certain research domains rather than being useful to general vision network designs and pre-training.\n\n### Reference\n\n[1] Vision Transformers Need Registers. ICLR, 2024.", "was_revised": true, "final_justification": "Overall, this submission offers a careful and useful empirical clarification of when attention maps fail as attributions in large ViTs, including a non-trivial analysis of the [CLS] residual. However, in its current form the evidence base is narrow and the downstream impact is not quantified. If the authors incorporate the rebuttal-reported cross-model experiments and add a small, focused downstream evaluation or residual-gating ablation, I would be inclined to raise my score. I lean Borderline Reject on significance/external validity grounds.", "rebuttal": "Thank you for your comprehensive review and constructive comments on our work. \n\nPlease allow us to comment on your suggestions:\n\n>  Compared to previous work like DINOv2 [17] with register tokens, this work only provides incremental findings (e.g., skip-connection effects)...\n\nWe would argue that the findings presented in this work are not merely incremental with respect to previous work; we will summarize our opinion on how the main contributions set our work apart from previous efforts. To the best of our knowledge we are the first to show that:\n\n- large DINOv2 models rely on register tokens to a surprisingly high degree for forming their [CLS] output; there is a clear relationship between this effect and model capacity. While previous work showed that high-norm tokens seem to encode more global image information, the contribution of high-norm tokens to downstream information integration was not studied. The computational role and relevance for [CLS] output of register tokens has not been studied before either. \n- previous work has yielded disagreement on whether high-norm tokens encode global image information ([17], [SINDER]); we show that this is due to methodological differences. Further, we show that high-norm register tokens live primarily in a small subspace which is downscaled during the layer-norm operation, explaining diverging findings and shedding light on the structure of these outliers\n- representations based on register/patch tokens are surprisingly orthogonal which, we argue, has important implications for the field. Several recent works [...] have claimed that incorporating register tokens yields more interpretable attention maps:\n\t-'Register tokens enable interpretable attention maps in all vision transformers...' [17],\n\t'Shifting the outlier tokens outside of the image area mimics register behavior at test-time (...), resulting in more interpretable attention patterns' [101],\n\t'Integrating registers has been shown to improve interpretability...' [102]\n\tHowever, these claims do not hold when the attention map does not reflect the model output in a veridical way. We believe that an awareness of these limitations makes an important contribution to the field.\n- The [CLS] token itself can produce the same effects\n- We also explicitly connect findings for high-norm and register tokens to literature on overparameterization (Please note also additional novel results described below)f\n\n> This work relies heavily on the analysis methods used in [17] , e.g., feature similarity metrics (CKA), without downstream task evaluation (e.g., segmentation / detection performance).\n\nWe would kindly point out that, as far as we can tell, CKA and similarity of patch/register representations to the full model output have not been studied in previous work, including [17]. To the best of our knowledge, we are the first to analyze the representations of patch/register tokens using feature similarity metrics. Downstream task performance (e.g. segmenation, LOST) has been studied relatively extensively, and we do not claim that models with registers have poor performance on these tasks. The focus of our contribution is rather on understanding the computational role filled by register tokens, and pointing out potential pitfalls like seemingly more interpretable attention maps.\n\n>  On the other hand, the authors only evaluated the classical Vision Transformers with contrastive learning (e.g., DINO and DINOv2), how about other similar Transformer architectures and different pre-training methods?\n\nThis is a very good point and we have tried to address it by running further experiments. First, the reason why the study on register models focuses only on DINOv2 is simply that, to the best of our knowledge, no other large-scale models with register tokens have been released openly. The paper on DINOv2 with registers [17] has been cited over 500 times and the model is being used by the community which we believe warrants an interest in this model specifically. Further, due to training cost, other work on register tokens also focuses on DINOv2 rather than retraining other models from scratch ([102], [102]).\n\nTo connect findings between models with register tokens and those only employing a [CLS] token, we also studied DINOv2 in the second part of the manuscript which does not rely on register tokens. We do agree with your comment that this approach needlessly narrows down the architecture coverage. To remedy this, we additionally ran the experiments in Section 4 for OpenCLIP and DeiTIII to cover contrastive language training and supervised training as well. Unfortunately, NeurIPS doesn't allow plots in the rebuttal this year - however, the findings mirror the ones for DINOv2: Attention to the skip connection grows dramatically with model size in all model families, and similarity between full model output and model output based on patch features declines with model size for all models. Therefore, the findings shown in Fig. 5,6 do seem to be representative for a variety of training schemes. Interestingly, as hypothesized in section 3.5, we observe that the tiny DeiTIII model trained on Imagenet1k shows similar metrics to the giant one trained on Imagenet22k, indicating that high attention to the skip connection arises from model capacity relative to complexity of the training data/task rather than model complexity itself.\n\n> - (Q1) Could the authors' findings hold for _non-DINOv2 architectures_ (e.g., DeiT, Swin Transformers) or other pretraining methods (e.g., masked image modeling or supervised pre-training)? If not, how might this limit broader applicability?\n\nWe hope to have answered this question above - if part of the question remains, we are of course open to discussing further.\n\n> - (Q2) The authors claim register tokens harm dense-task performance (Sec. 3.5), but provide no metrics. Can the authors benchmark patch-feature quality on tasks like segmentation using standard datasets (e.g., ADE20K)?\n\nWe have tried not to claim poor performance of register models on tense tasks as it's been shown that performance on segmentation and localization is quite good. We were rather referring to tasks that require a tight connection between local and global features (e.g. [15], [16]). However, these statements serve more as motivation to our work rather than aiming to provide quantitative evidence. Therefore we do agree that strong claims in that regard are not justified and have made sure there are no overclaims in the manuscript.\n\n> - (Q3) Instead of removing registers/skips (Sec. 5), could pre-norm / post-norm regularization or drop-path regularization (Sec. 4) preserve interpretability?\n\nThis is an interesting question and indeed it should be explored how strongly drop-path regularization can inhibit extreme attention to the skip connection at the last layer. The only statement that we can make with certainty is that DeiTIII, which does use linearly increasing drop-path regularization, shows the same patterns as displayed for DINOv2 in the manuscript (see discussion above). Therefore, at least it does not seem to be a 'quick fix' that works by just including it. With your permission, we would include a brief paragraph in the discussion of the manuscript should it be accepted for publication.\n\nThanks again for your review - we hope that we were able to address some of your concerns and have perhaps increased your confidence in our work. We are of course open to further discussion on the manuscript.\n\n[17] Vision Transformers Need Registers\n\n[101] Vision Transformers Don’t Need Trained Registers\n\n[102] Registers in Small Vision Transformers"}, {"review_id": "zcE6rLlL7R", "reviewer": "Reviewer_inCN", "summary": "This paper identifies a fundamental issue in ViTs, such as DINOv2: a decoupling between global image representations and local patch features. Traditionally, global features are assumed to be formed by aggregating local patch embeddings via [CLS] token attention. However, the authors show that this assumption breaks down in large models. The authors point out that [CLS] often attends to special register tokens rather than image patches, making the global representation insensitive to local details. Even without explicit register tokens, residual connections allow [CLS] to bypass patch information, effectively acting as an implicit register. Besides, the root cause is attributed to overparameterization, allowing global information to be compressed early and isolated from the image structure.", "questions": "The authors may also provide some robustness/variance evaluation on the experiment results.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The authors present a rigorous and well-structured analysis of the interaction between the [CLS] token and register tokens. The paper is clearly written and easy to follow. I like the authors to identify a fundamental issue—overparameterization—and links it to the phenomenon of neural collapse, offering a novel and insightful perspective. I believe this work can inspire further research toward improving the interpretability and structure of Vision Transformers.\n\nWhile the authors suggest avoiding residual connections between register tokens and the [CLS] token, it would strengthen the paper to include empirical evidence supporting this claim. Are there any experiments or architectural designs specifically aimed at mitigating this issue? A proof-of-concept evaluation would greatly enhance the contribution. Also as mentioned in the limitation part, it would be important for the community to understand how these discussed relationships can improve the performance of these ViTs.", "was_revised": true, "final_justification": "The authors addressed my concerns, I recommend accept.", "rebuttal": "Thank you for you thorough review - we appreciate your confidence in the value of our work. Please allow us to respond to your comment:\n\n> The authors may also provide some robustness/variance evaluation on the experiment results.\n\nTo strengthen the empirical results of the paper, we have included two more models for the experiments in Section 4 (DeiTII for supervised training, OpenCLIP for contrastive language training). While, unfortunately, we can not share results visually this year, the findings mirror the ones for DINOv2: Attention to the skip connection grows dramatically with model size in all model families, and similarity between full model output and model output based on patch features declines with model size for all models. Therefore, the findings shown in Fig. 5,6 do seem to be representative for a variety of training schemes, therefore demonstrating the robustness of the findings. Interestingly, as hypothesized in section 3.5, we observe that the tiny DeiTIII model trained on Imagenet1k shows similar metrics to the giant one trained on Imagenet22k, indicating that high attention to the skip connection arises from model capacity relative to complexity of the training data/task rather than model complexity itself."}, {"review_id": "TL7D96e8Ng", "reviewer": "Reviewer_Cr6A", "summary": "The paper studies the observations from Darcet et al. (Vision Transformers Need Registers) further in depth. They introduce the term “patch integration assumption” which describes the theory that the [CLS] token learns to select the final features from the most relevant patches for global image understanding. The authors show that this assumption holds for smaller models but not for larger models (ViT-H, Vit-g) through multiple experiments. They hypothesize that: “ larger ViT variants are overparameterized to the point at which DINOv2 training yields simplistic representations at the last layer, relying on a small number of tokens in which global information has already been integrated.”. Further, they also show that in models without registers the skip connections on the CLS token are used to store global information.", "questions": "* I find this hypothesis: “we hypothesize that larger ViT variants are overparameterized to  the point at which DINOv2 training yields simplistic representations at the last layer, relying on a small number of tokens in which global information has already been integrated.” interesting. Did you inspect earlier layers in the larger models if maybe the patch integration assumption holds more there and these global aggregated features are simply passed on?\n\n* Why do we want the patch integration assumption to hold? To get better interpretable models? Why can we not use proper explanation methods?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* They introduce the term “patch integration assumption” which poses an interesting question, is described well and picked up again and again throughout the paper.\n* The paper is embedded well into related work.\n* The disconnection between local and global features is studied from multiple angles (attention distribution, CKA with global representations, accuracy when only aggregating from one set).\n* The experiments are well designed and hypotheses and conclusions drawn are easy to follow.\n* The results are nicely connected to works on overparameterization (section 3.5).\n\n\n**Weaknesses**\n\n* The study focuses only on the representations and attention weights at the last layer. Especially as the hypothesis is that for larger models, global information gets aggregated earlier, it would have been great to also analyze the attention weights in earlier layers and how this is progressing throughout the model.\n* In the Explainable AI literature, it is well known that attention is not a good explanation [1, 2, 3]. This connection is missing in this paper and it also limits the insights/novelty of the paper a bit.\n* It should be better discussed why the patch integration assumption should hold (There is not really something in the model to encourage this).\n\n[1] Wiegreffe, Sarah, and Yuval Pinter. \"Attention is not not explanation.\" arXiv preprint arXiv:1908.04626 (2019).\n\n[2] Jain, Sarthak, and Byron C. Wallace. \"Attention is not explanation.\" arXiv preprint arXiv:1902.10186 (2019).\n\n[3] Ali, Ameen, et al. \"XAI for transformers: Better explanations through conservative propagation.\" International conference on machine learning. PMLR, 2022.", "was_revised": true, "final_justification": "I definitely would advocate for accepting this paper as already outlined by my initial score as it provides quite interesting insights to the computer vision community!\n\nHowever, as some parts could benefit from clearer discussion (e.g. do we want the patch integration assumption to hold and why is this case for smaller models from the beginning) and more zooming into the evolution of interaction between global and local tokens throughout the model (beyond Fig. 5b), I don't feel strongly enough to raise my score to a strong accept.", "rebuttal": "Thank you for your insightful review and expressing your support for our work. While we believe that this is a fair evaluation, please allow us to briefly address the points of criticism and questions you've raised:\n\n> - The study focuses only on the representations and attention weights at the last layer. Especially as the hypothesis is that for larger models, global information gets aggregated earlier, it would have been great to also analyze the attention weights in earlier layers and how this is progressing throughout the model.\n\nOur reasoning for focusing on the last layer was that\na) The attention maps of this layer were studied in previous work\nb) Usually both dense and global features are extracted from this layer for downstream tasks - therefore we believe that the integration of local information is most relevant at this layer.\nRegarding intermediate layers, for models without explicit register tokens we did  analyze how the [CLS] representations evolve with depth in Fig. 5b) which is one step in the direction you've raised.\nFurther, in response to your comment, we've run an additional experiment on DINOv2 with registers, measuring the ['CLS] attention to patch and register tokens at each layer. Interestingly, the [CLS] token in models of all sizes attends to both patch and register tokens in intermediate layers. The clear relationship between model size and attention to registers seems to emerge only at the last layer. Subsequent work should study this phenomenon in depth - it should be noted though that the intermediate [CLS] tokens are of course not directly trained to extract global information, which is why their attention patterns may differ from the last layer.\n\n> - In the Explainable AI literature, it is well known that attention is not a good explanation [1, 2, 3]. This connection is missing in this paper and it also limits the insights/novelty of the paper a bit.\n\nWe do agree with you statement, however we would argue that our analysis of the attention maps still has merit to the field, as several works have claimed that attention maps of models with registers _are_ more interpretable, see for example:\n\n'Register tokens enable interpretable attention maps in all vision transformers...' [17],\n'Shifting the outlier tokens outside of the image area mimics register behavior at test-time (...), resulting in more interpretable attention patterns' [101],\n'Integrating registers has been shown to improve interpretability...' [102]\n\nTherefore, while the downsides of studying attention maps have been studied as a whole, we believe that the field should also be aware of the specific pitfalls associated with attention maps of register models.\n\n> - It should be better discussed why the patch integration assumption should hold (There is not really something in the model to encourage this).\n\nThis does pose an interesting reversal of the hypothesis. Our results show empirically that smaller models _do_ satisfy the patch integration assumption (see also additional experiments mentioned below) and previous work has simply assumed that they do implicitly. When considering the original work on high-norm tokens [17] one can indeed consider the models that exhibit outlier tokens to be the 'standard case' and explicitly study those model that do not display them. To the best of our knowledge, analyses of that kind have not been done and we think that this would be a very interesting avenue. For this work, we hope you agree that at least a systematic treatment of this question is out of scope.\n\n> - I find this hypothesis: “we hypothesize that larger ViT variants are overparameterized to the point at which DINOv2 training yields simplistic representations at the last layer, relying on a small number of tokens in which global information has already been integrated.” interesting. Did you inspect earlier layers in the larger models if maybe the patch integration assumption holds more there and these global aggregated features are simply passed on?\n\nFor the models without register tokens, the analysis shown in Fig. 5b) does support the claim that in larger models, global feature integration starts earlier in the hierarchy and those features are passed on with small modifications. Therefore, the last layer essentially receives its output as input in the giant model.\nFor the model with registers, we did not test this explicitly but we would argue that the results in Fig. 2 also strongly indicate this conclusion. Of course, in the intermediate model layers, the model has to extract all necessary information from the image patches but in the last layer the model attends primarily to just 5 register tokens (compared to 256 patch tokens!). A layer-wise analysis like the one shown in Fig. 5b) probing whether the information stored in register tokens saturates with depth would also be interesting here, but unfortunately due to time constraints we were not able to implement this during the rebuttal phase.\n\n>- Why do we want the patch integration assumption to hold? To get better interpretable models? Why can we not use proper explanation methods?\n\nFirst, we would argue that the simplicity of being able to write the [CLS] output as a convex combination of **meaningful** patch features is itself an advantage over more involved attribution methods because, it is very intuitive and easy to apply/understand for the user.\nBeyond interpretability, joint training of dense and global features as in DINOv2 allows the user to explicitly ground the global representation in the dense feature maps as done in [15, 16]. More generally, we are referring here to tasks that rely on a correspondence between dense local features and the global image embedding. \n\n---\n\nFinally, please note that we've also run additional experiments including more model architectures for the skip connections studied in Section 4. Since there is no global response this year, and to avoid walls of text, we would kindly refer you to the response to reviewer qxac in case you are interested.\n\n[15] Extract Free Dense Labels from CLIP\n\n[16] BRAIN MAPPING WITH DENSE FEATURES\n\n[17] Vision Transformers Need Registers\n\n[101] Vision Transformers Don’t Need Trained Registers\n\n[102] Registers in Small Vision Transformers"}, {"review_id": "ZCqgTt7j8X", "reviewer": "Reviewer_WKgH", "summary": "The authors analyze the behavior of register tokens across ViT models of different sizes, highlighting that in register-based ViT models, the patch integration assumption does not hold. In addition, they provide an analysis based on skip connections, showing that register or [CLS] tokens can ignore the local information of patch tokens via skip connections.", "questions": "1. All analyses in the paper appear to be entirely based on DINOv2. It remains unclear whether similar phenomena occur in other ViT models. In contrast, the original work [17] conducted experiments across a variety of ViTs.\n2. In lines 123–125, the authors emphasize that there is large preference variance. Could the authors provide visualizations or a brief analysis to clarify which types of samples, in the largest model, tend to favor register tokens? This would help to better understand under what circumstances the model deviates from the patch integration assumption.\n3. Do register tokens exhibit similar behavior in shallow ViT networks？", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The analysis based on skip connections is quite interesting.\n2. The authors’ work serves as strong evidence to challenge the “patch integration assumption.”\n\nWeaknesses:\n1. The core conclusions of the paper appear to be a straightforward extension of prior work [17], without introducing fundamentally new insights. The paper seems more like an effort to find empirical evidence for some intuitive conclusions:\n- Lines 12–13 state that “the CLS token can be interpreted as a register,” but register tokens are essentially an extension of the CLS token as discussed in [17], so this symmetric conclusion seems rather obvious.\n- In [17], register tokens are used to store global information to obtain better dense representations and cleaner attention maps, with emphasis on improvements in downstream dense prediction tasks. Given this, it’s already expected that the attention maps from the CLS token to patch tokens would not fully express global information, since part of it is stored in register tokens.\n2. The experiments appear to be limited only to the DINOv2 ViT model.\n3. The analysis seems confined to only the last few layers of the ViT architecture.", "was_revised": true, "final_justification": "The paper analyzes large ViT models with a register and provides strong evidence challenging the “patch integration assumption,” along with an interesting analysis of the [CLS] skip connection in an attempt to explain why the patch integration assumption fails. However, since the register inherently carries high-norm global information, I believe it is expected that the aggregation of patch tokens (low-norm) by the CLS token cannot directly correspond to the global information in the representation — and this is the core conclusion of the paper. Therefore, I consider the contribution to remain somewhat incremental and my final score is borderline reject.", "rebuttal": "Thank you for your insightful review, and the helpful comments on this manuscript.\n\nPlease allow us to comment on your suggestions:\n\n> The core conclusions of the paper appear to be a straightforward extension of prior work [17], without introducing fundamentally new insights. The paper seems more like an effort to find empirical evidence for some intuitive conclusions.\n\nOur work does build heavily on [17], but we do believe that our findings are sufficiently distinct from previous work. In particular, we study the effects of the methodology proposed in [17], which are not discussed in that paper. The main findings that set our contribution apart in our opinion are:\n\n- large DINOv2 models rely on register tokens to a surprisingly high degree for forming their [CLS] output; there is a clear relationship between this effect and model capacity. While previous work showed that high-norm tokens seem to encode more global image information, the contribution of high norm tokens to downstream information integration was not studied. The computational role and relevance for [CLS] output of register tokens has not been studied before either. \n- previous work has yielded disagreement on whether high-norm tokens encode global image information ([17], [SINDER]); we show that this is due to methodological differences. Further, we show that high-norm register tokens live primarily in a small subspace which is downscaled during the layer-norm operation, explaining diverging findings and shedding light on the structure of these outliers\n- the attention maps are not as reliable as assumed (see later point)\n- the [CLS] token itself can produce the same effects as the register tokens\n- We also explicitly connect findings for high-norm and register tokens to literature on overparameterization (see also additional results described below)\n\n> - Lines 12–13 state that “the CLS token can be interpreted as a register,” but register tokens are essentially an extension of the CLS token as discussed in [17], so this symmetric conclusion seems rather obvious.\n\nWe agree that interpreting the [CLS] token as a register is not a deep theoretical insight. However, we have not seen this interpretation elsewhere and are not sure where to find this being discussed in [17]. Their analysis also does not seem to include the [CLS] token as a register - on the contrary, experiments in [17] (e.g. Fig. 8) include models with 0 register tokens even though a [CLS] token _is_ included. In any case, we would argue that the referenced statement serves rather as a device for unfolding the story of the paper and connecting the findings from section 3 and 4 rather than a theoretical contribution.\n\n> - In [17], register tokens are used to store global information to obtain better dense representations and cleaner attention maps, with emphasis on improvements in downstream dense prediction tasks. Given this, it’s already expected that the attention maps from the CLS token to patch tokens would not fully express global information, since part of it is stored in register tokens.\n\nWe do agree, however the degree to which global image information integration is pushed into these register tokens was, in our opinion, not obvious and has not been studied before. Of course one would expect some information contribution from the registers, but that [CLS] output and output computed from attention are in many cases completely orthogonal was striking to us. It has been claimed multiple times that attention maps from register models are mote interpretable, see for example\n'Register tokens enable interpretable attention maps in all vision transformers...' [17],\n'Shifting the outlier tokens outside of the image area mimics register behavior at test-time (...), resulting in more interpretable attention patterns' [101],\n'Integrating registers has been shown to improve interpretability...' [102]\n\nTherefore, we argue it is not clear to the field that attention maps from register models do not reflect global information integration. We believe that this adds strong to merit to our work, as it seems important for the community to be aware of this potential pitfall.\n\n\n> The experiments appear to be limited only to the DINOv2 ViT model.\n\nWe agree that this is a shortcoming of our work and have added further experiments to improve our contribution. First, the reason why the study on register models focuses only on DINOv2 is simply that, to the best of our knowledge, no other models with register tokens have been released openly. The paper on DINOv2 with registers [17] has been cited over 500 times and the model is being used by the community which we believe warrants an interest in this model specifically. Further, due to training cost, other work on register tokens also focuses on DINOv2 rather than retraining big models from scratch ([101], [102]).\n\nTo connect findings between models with register tokens and those only employing a [CLS] token, we also studied DINOv2 in the second part of the manuscript which does not rely on register tokens. We do agree with your comment that this approach needlessly narrows down the architecture coverage. To remedy this, we additionally ran the experiments in Section 4 for OpenCLIP and DeiTIII to cover contrastive language training and supervised training as well. Unfortunately, NeurIPS doesn't allow plots in the rebuttal this year - however, the findings mirror the ones for DINOv2: Attention to the skip connection grows dramatically with model size in all model families, and similarity between full model output and model output based on patch features declines with model size for all models. Therefore, the findings shown in Fig. 5,6 do seem to be representative for a variety of training schemes. Interestingly, as hypothesized in section 3.5, we observe that the tiny DeiTIII model trained on Imagenet1k shows similar metrics to the giant one trained on Imagenet22k, indicating that high attention to the skip connection arises from model capacity relative to complexity of the training data/task rather than model complexity itself.\n\n> The analysis seems confined to only the last few layers of the ViT architecture.\n\nWhile section 4 includes a brief explanation for how the [CLS] representation emerges throughout the model, we did focus on the last model layer to investigate how the final global representation is formed. We argue that the last layer is most interesting for our purposes because\na) the resulting local and global features are the ones usually used for downstream tasks\nb) (the attention map of) this layer was studied in previous works\nTo address your comment, we ran an additional experiment on how attention to patch and register tokens is distributed at each model layer. We will include this in the appendix of the manuscript. In brief, we find that models of all sizes attend to register tokens in intermediate layers to varying degrees. A very clear picture arises only at the last layer, where we find the monotonic relationship between model size and attention to register tokens when the final output is formed.\n\n> 1. All analyses in the paper appear to be entirely based on DINOv2. It remains unclear whether similar phenomena occur in other ViT models. In contrast, the original work [17] conducted experiments across a variety of ViTs.\n\nWe hope that this is satisfyingly addressed above - if there are further questions, please let us know.\n\n> 1. In lines 123–125, the authors emphasize that there is large preference variance. Could the authors provide visualizations or a brief analysis to clarify which types of samples, in the largest model, tend to favor register tokens? This would help to better understand under what circumstances the model deviates from the patch integration assumption.\n\nThis is a very good point, and we also spent some time trying to find structure as to which types of samples were processed primarily in the register tokens. However, this analysis did not lead to any clear conclusions - images of a variety of classes lead to strong violations of the patch integration assumption.\n\n> Do register tokens exhibit similar behavior in shallow ViT networks？\n\nFrom our experiments, it does seem like these behaviors only arise in the larger models. The 'small' and 'base' model _are_ more shallow than the 'large' and 'giant' model, but of course the depth is confounded with total parameter count. It would be interesting to study these effects in models with different depth or width while keeping the numbers of parameters fixed. However, at this point we can only tie the effects to the number of parameters.\n\nAgain, thanks for your review. We hope that our comments and additional work have sufficiently answered some of your concerns and increased your confidence in our work.\n\n[17] Vision Transformers Need Registers\n\n[101] Vision Transformers Don’t Need Trained Registers\n\n[102] Registers in Small Vision Transformers"}]

Author rebuttal:

Meta-review: The paper analyzes the representations learned by vision transformers (ViTs) of different sizes with registers or CLS tokens, in particular with respect to the "patch integration assumption" that the final representation is aggregated from patch representations, and attention maps are somewhat representative of the importance of different image regions. 

Based on the reviews, author/reviewer discussion, and the paper itself, here are the key strengths and weaknesses.

Strengths
1. Interesting and well-executed analysis, from several different angles
2. Good presentation

Weaknesses
1. The analysis is limited to one model type (DINOv2). Authors mentioned other model types in the rebuttal, but it's not in the paper atm
2. The analysis is limited to the last layer 
3. No clear practical conclusions, no application experiments demonstrating useful takeaways from the analysis

Overall, the paper is fairly borderline, with some interesting analysis, but also clearly limited in some ways. This is also reflected by review scores.

Given how the paper is well written and the results are thought-provoking and potentially of interest to the broad community of computer vision researchers, I lean towards recommending acceptance. However, I do very much encourage the authors to improve the paper.
Decision: Accept (poster)