Revisiting [CLS] and Patch Token Interaction in  Vision Transformers
Authors: Alexis Marouani
Oriane Siméoni
Herve Jegou
Piotr Bojanowski
Huy V. Vo
Venue: ICLR 2026 Poster (2026)
Keywords: representation
vision
transformer
SSL
attention
specialization
architecture
interpretability
DINO
DINOv2
CLIP
DEIT
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose and analyze a new architecture to specialize CLS and patch tokens processing in ViTs, enhancing dense tasks performances.

Abstract: Vision Transformers have emerged as powerful, scalable and versatile representation learners. To capture both global and local features, a learnable [CLS] class token is typically prepended to the input sequence of patch tokens. Despite their distinct nature, both token types are processed identically throughout the model.
In this work, we investigate the friction between global and local feature learning under different pre-training strategies by analyzing the interactions between class and patch tokens.
Our analysis reveals that standard normalization layers introduce an implicit differentiation between these token types. Building on this insight, we propose specialized processing paths that selectively disentangle the computational flow of class and patch tokens, particularly within normalization layers and early query-key-value projections.
This targeted specialization leads to significantly improved patch representation quality for dense prediction tasks. Our experiments demonstrate segmentation performance gains of over 2 mIoU points on standard benchmarks, while maintaining strong classification accuracy. The proposed modifications introduce only an 8\% increase in parameters, with no additional computational overhead.
Through comprehensive ablations, we provide insights into which architectural components benefit most from specialization and how our approach generalizes across model scales and learning frameworks.

Published as a conference paper at ICLR 2026
REVISITING [CLS] AND PATCH TOKEN INTERACTION IN VISION TRANSFORMERS
Alexis Marouani
Oriane Siméoni'
Hervé Jégou'
Piotr Bojanowski*
Huy V. Vol
FAIR, Meta
2 LIGM, Bcole des Ponts, IPParis, UGE, CNRS, 77455 Marne-la-Vallée, France
ABSTRACT
Vision Transformers have emerged as powerful, scalable and versatile representa- tion learners. To capture both global and local features, a learnable [CLS] class token is typically prepended to the input sequence of patch tokens. Despite their distinct nature, both token types are processed identically throughout the model. In this work, we investigate the friction between global and local feature Iearn- ing under different pre-training strategies by analyzing the interactions between class and patch tokens. Our analysis reveals that standard normalization layers introduce an implicit differentiation between these token types. Building on this insight, propose specialized processing paths that selectively disentangle the computational flow of class and patch tokens, particularly within normalization layers and early guery-key-valle projections. This targeted specialization leads significantly improved patch representation quality for dense prediction tasks. Our experiments demonstrate segmentation performance gains over 2 mloU points standard benchmarks, while maintaining strong classification accuracy. The proposed modifications introduce only an 8% increase in parameters, with no ad- ditional computational overhead. Through comprehensive ablations, we provide insights into which architectural components benefit most from specialization and how approach generalizes across model scales and learning frameworks.
INTRODUCTION
In recent years, significant progress has been made in developing vision foundation models capable of generating rich and highly generalizable visual representations for images. Notably, latest state- of-the-art results have been achieved using Vision Transformer (VIT) models (Dosovitskiy et al., 2021) trained under various paradigms, including fully-supervised (Touvron al., 2022), weakly
"Correspondence rouan @meta. com
Original image
(a) DINOv2 w/ regs (Darcet al., 202.3)
(a)
(b) DINOv2 w/ attn. bias (An al., 2025)
(b) + ours
Figure| Visualization of the impact of our proposed layer specialization for [CIS]] and patch tokens the patch features obtained with DINOv2 when using two strategies to mitigate artifacts, namely registers ('regs") (Darcet et al., 2023) and attention bias ('attn. bias') (An et al., 2025). We display the first PCA components of model outputs in RGB.

---

Published as a conference paper at ICLR 2026
supervised (Radford et al., 2021; Bolya et al., 2025), and self-supervised learning (Zhou et al., 2021; Oquab et al., 2023; Siméoni et al., 2025). These models capture a wide spectrum of visual semantics, enabling robust performance across diverse range of downstream tasks and data domains.
The VIT architecture (Dosovitskiy et al., 2021) processes images by dividing them into fixed-size patches, which are then embedded and fed to sequence of transformer blocks. Typically, trainable class token [CLS] is prepended to the sequence of patch embeddings and is designed to aggregate information from all patches. Patches and [CLS] tokens are trained with different objectives, if any. For instance, most pre-training methods apply a loss function solely on the [CLS] token (Chen al., 2020; Grill et al., 2020; Caron al., 2021; Radford et al., 2021; Touvron et al., 2022). Some employs an objective on patch tokens only (He et al., 2022), while others train the [CLS] and patch tokens with separate losses (Zhou et al., 2021; Oquab et al., 2023; Siméoni et al., 2025). Regardless of the specific training paradigm, recent works (Darcet et al., 2023; An et al., 2025; Siméoni et al., 2025) show that there persistent imbalance between the [CLS] and patch tokens. Proposed so- lutions to this issue include introducing additional storage tokens into the input sequence (Darcet et al., 2023), modifying the attention mechanism (An et al., 2025), or incorporating additional loss terms to explicitly constrain patch locality (Siméoni et al., 2025). In contrast,
hypothesize that the imbalance arises because models process the [CLS] and patch tokens through identical compu- tational pipelines, despite their fundamentally different roles and nature, and propose disentangling their treatment to overcome the imbalance.
In this work, we analyze the model statistics in order better understand the internal mechanisms that govern the interaction between the [CLS] and patch tokens. Our analysis reveals a surprising finding: normalization layers are already implicitly learning to distinguish between the [CLS] and patch tokens before the attention mechanism. Building on this insight, we introduce a simple yet effective architectural modification that explicitly separates the processing of the [CLS] and patch tokens, as illustrated in Fig. 5. With just minimal set of specialized layers, our approach leads to noticeably richer dense features (see Fig. 1) and delivers substantial gains on dense prediction tasks. For instance, we improve the average mloU scores on segmentation benchmarks by as much as 2.2 points with a ViT-L. This work sheds light on hidden dynamics within transformer models and also demonstrates how targeted architectural changes can translate into significant real-world performance improvements. We make the following contributions:
We analyze the interactions between [CLS] and patch tokens within Vision Transformers, and show that models implicitly attempt to distinguish them through normalization layers.
We propose an architectural modification that specializes their computations to reduce the friction between them, while keeping the number of operations constant. We study different specialization strategies for transformer block components.
We demonstrate the generalizability of our approach across model scales and learning frameworks, showing significant improvements in dense prediction tasks without compro- mising classification performance.
2 RELATED WORK
Vision Transformers Inspired by Vaswani et al. (2017) and first introduced by Dosovitskiy et al. (2021), Vision Transformer has become architecture of choice when building vision models. A typical ViT model consists of a patch embedder and a stack of transformer blocks. Given an image, the patch embedder divides it into equally-sized square patches and transforms them into patch tokens that represent local information in the image. Optionally, a learnable [CLS] token is added to the set of patch tokens in order to capture global information. All tokens are then passed through the transformer blocks which process them with various transformations, most notably the multi-head self-attention operator (Vaswani et al., 2017) that allows tokens to attend to each other. Built on the original architecture, subsequent works have introduced additional components to improve various aspects of ViTs such as data efficiency (Touvron et al., 2020; Yuan et al., 2021), 1), computational cost (Liu et al., 2021; Bolya et al., 2023) and normalization (Touvron et al., 2021). ViT architecture has enabled state-of -f-t performance in various tasks (Carion et al., 2020; Strudel et al., 2021), I), simplified multi-modas learning (Radford et al., 2021; Fini et al., 2024), and led to excellent local and global representation in foundation models (Oquab et al., 2014; Tschannen et al., 2025; Simóoni et al., 2025). In most ViTs, [CL.S] and patch tokens are functionally interchangeable in transformer
2

---

Published as a conference paper at ICLR 2026
befoce ofter
simibeity
6:2-
0.2-
0.2
0.2
6.0-
0.0-
0.0-
-0.8
-0.2
-0.2
-0.2
palSppp putchre-potchiee
(CLS)-patcha petehre-prcche
(alSpppatcls potchee-patelos
(SLS)ppatcin petchre-gateles
(a) LayerNorm pre-attn.
(b) Self-attention
(c) LayerScale post-attn.
(d) LayerNorm pre-MLP
Figure 2: [CLS] -patches separation effect within transformer blocks in vanilla DINOv2 ViT-L model. We show mean and standard deviation of cosine similarity between [CLS] and all patches, and all-to-all patches. before and after each transformer layers. 'attn." stands for attention.
blocks - they are processed in identical manner using the same operators despite their distinctive nature. We show in our analysis that the identical treatment of these tokens is suboptimal and that disentangling them leads to better local features for dense tasks.
Improving dense feature learning Visual representation learning approaches have mostly focus optimizing the global representation by primarily training the [CLS] token to summarize the im- age content either in supervised (rriiisdTuuuu (Touvron et al., 2022), weakly supervised (Radford et al., 2021; Tschannen al., 2025) self-supervised settings (Caron et al., 2021; Liu et al., 2021). As by- product, they also produce local representation that perform well on tasks that require fine-grained features such as object detection, semantic segmentation depth estimation. Most notably, the self- supervised method DINO (Caron et al., 2021) produces excellent patch features that supercharge research on unsupervised object detection and segmentation. iBoT (Zhou et al., 2021) augments DINO with masked image modeling (He et al., 2022) to optimize both global and local representa- tion. DINOv2 (Oquab al., 2023) introduces new technical components such Sinkhorn-Ke Knopp centering and untying heads to successfully scale DINO to large datasets and model sizes, achieving excellent performance on dense tasks. Learning meaningful dense features with Vision Transform- is not without challenges. Darcet et al. (2023) discusses the noisy attention maps produced by models trained at scale during longer training periods. This issue, which degrades dense prediction performance, is caused by some patch tokens losing their local context after being repurposed by the model to store global information. They propose an architectural solution with registers to mitigate these issues. Other successful attempts to enhance the quality of local features include regulariz- ing similarity to neighbor patches post-training (Pariza et al., 2024) or recovering patch similarity with Gram anchoring mechanism (Siméoni et al., 2025). Similar to these works, we improve dense features quality during training by specializing [CLS] and patch tokens treatment within the Trans- former blocks of VITs and thus reducing the friction between them.
FRICTION BETWEEN [CLS] AND PATCHES
Vision Transformers are typically trained with a trainable [CLS] token which encodes global infor- mation about the image prepended to the sequence of patch tokens. Despite the distinct nature of the [CLS] and patch tokens, current models treat them equivalently, applying the exact same operations to both. However, (Darcet et al., 2023) has highlighted potential communication issues between these two types of tokens, Icading to a severe loss of locality of patch tokens and the appearance of undesirable outliers in the attention maps. While registers help to mitigate the appearance of arti- facts, we argue that more could be done. Our observations indicate that a degree of friction persists between [CLS] and patch tokens, as discussed below.
VITs differentiate [CLS] and patch tokens for the attention We analyze the interplay between [CLS] and patch tokens by computing their similarity at different points within the model, before and after principal layers in each transformer block. In Fig. 2, we visualize the mean and standard deviation these similarities. Our results are averaged over patches of 1000 images and across all model blocks. Additionally, we present the same statistics between patches. While certain opera- tions wons-such as the LayerScale applied post-attention havi little effect on the similarity between [CLS] and patch tokens, the self-attention layer markedly increases their similarity. This increase expected, as self-attention realigns the different token types. However, our analysis uncovers a surprising phenomenon: the representations of [CI.S] and patch tokens naturally diverge specific
3

---

Published as a conference paper at ICLR 2026
6,50
before afher
0.50
0.50
0.50
simibeity
6,25
0.25
0.25
0.20
0.00
0.00
0.00
0.00
0.20
0.25
0.20
0.25
ClLS--outius pitchospaatles
[CLS)-paidees patibreegatchee
(uiSp potchospptchle
[CLSS]ppiidus patchespatchoe
(a) DINOv2
(b) DINOv2 w/ regs
(c) DINOv2 w/ attn-bias
(d) DEIT-III
Figure 3: Impact of LayerNorm before attention layer for different pre-trained models. We show mean and standard deviation of cosine similarity between [CLS] and all patches, and between all patches. Statistics visualized before and after LayerNorm (LN).
stages of the computational pipeline, particularly just before attention operations. Indeed, the Lay- erNorm applied before attention drastically reduces the similarity between [CLS] and patch tokens, bringing it close to zero. This implicit differentiation indicates that the model attempts to adapt to the distinct functional roles of these token types before the attention mechanism, despite their shared parameterization. We plot the statisties of more layers in Sec. A.2.
Role of the pre-attention LayerNorm In Fig. 3, we focus on the impact of the pre-attention LayerNorm to the similarity between the [CLS] and patch tokens in different pre-trained models, including DINOv2 (Oquab et al., 2023) and its variants with registers (Darcet et al., 2023), noted 'regs', and attention bias (An al., 2025), noted 'attn. bias', and supervised DEIT-III (Touvron et al., 2022). It can be observed that in all cases, prior to the attention mechanism, the Layer- Norm disentangles the [CLS] and patch tokens, enabling them serve distinct functions within the attention process. This phenomenon appears in all pre-trained models with different extent. For instance, the LayerNorm strongly enforces negative correlation between [CLS] and patches in DI- NOv2 and DEIT-III while keeping the correlation close to zeros in the variants of DINOv2. In contrast, the similarity among patch tokens remains positive and largely stable, with only slight decrease observed- phenomenon we interpret as regularization effect. This effect likely prevents rank collapse and promotes a more uniform distribution of tokens on the unit sphere, consistent with observations reported in Wu et al. (2024).
CLS)
[CLS]
[CLS]]
patches
patches
patches
dimensions
disnensiews
dimensions
(a) Block
(b) Block 15
(c) Block 23 (Last)
Figure 4: Dimensions with biggest magnitudes early (a), in the middle (b), the end (c) of the model for [CLS] and 5 patches with the highest magnitudes in the selected dimensions. Tokens taken the output of blocks. The considered model is DINOv2 ViT-L with attention bias.
Dimension separation To understand how separation effect can appear in LayerNorm layer, one has recall that performs woint-wise normalization and dimension-ve affine transfor- mation. Therefore, a separation effect occurs when inputs have very different magnitudes in each dimension. In Fig. 4, we plot the dimensions with biggest absolute magnitudes averaged over patches and [CLS] at the output of different blocks. We observe that some specific dimensions are leveraged only by a certain token type. For example, in Fig. 4c, the 2nd dimension presents large magnitudes for patches and almost none for [CLS]. Moreover, the deeper are in the model, the fewer token types share dimensions. This enables normalization layers to perform distinctive operations. More than just regularizing, they specialize and separate the tokens.
All the observations above indicate that treating [CLS] and patch tokens identically compels the model to allocate resources towards implicitly separating them, which could be used to learn more
4

---

Published as a conference paper at ICLR 2026
Patches
Specialized Transformer block
Layer Norm
QKV proj.
Layer
Layer Norm
Layer Scale
Linear
ML.
Scale
Attention
:
Layer Norm
QKV
Layer
Layer
Layer Scale
Linear
MLP
proj.
Scale
Norm
[CLS]
sharable weights
Figure 5: Architecture specialization. We investigate how [CLS] and patch tokens can be processed through specialized layers, while preserving their interactions within the attention mechanism.
meaningful features. We argue that disentangling their treatments would facilitate the model in learning better representation, as discussed in next section.
4 [CLS] PATCHES SPECIALIZATION: ANALYSIS
In this section, we first define our proposed layer specialization in Sec. 4.1 and set the experimental setting in Sec, 4.2. In Sec. 4.3, we discuss the benefits of splitting normalizations for [CLS] and patch tokens. We also investigate which part of the model needs specialization in Sec. 4.4, and more specifically which layers in Sec. 4.5.
4.1 OUR PROPOSAL: LAYER SPECIALIZATION
Based on observations made in the previous section, we explore disentangling the computation of global and local representations in ViTs. Taking inspiration from the success of doublc-stream ar- chitectures handle different modalities (Esser et al., 2024), we explore a similar approach for the [CLS] and patch tokens. More specifically, inside classic transformer block, [CLS] and patch to- kens go through several layers: projections, some normalizations and a MLP. We propose to decou- ple the [CLS] and patch tokens by processing them with different weights for certain layers. Indeed, instead of using a single layer to process both token types, we introduce two distinct layers- each with its set of specialized for either [CLS] or patch tokens. This allows each layer to better capture the unique characteristics of its respective token type. However, the tokens continue to interact through the attention mechanism usual, ensuring information flow between [CLS] and patch tokens is preserved. An illustration of this specialized architecture is provided in Fig. 5. While this approach introduces some additional memory overhead, our experiments show that the increase in model size remains mppll-approximatei 8% layer specialization is applied selectively achieve optimal performance. More importantly, layer specialization does not increase inference FLOPs, as the model continues to perform the same computational operations during inference. This ensures that the efficiency of the model maintained, even as we enhance representational capacity through targeted specialization.
4.2 EXPERIMENTAL SETTING: TRAINING AND EVALUATION
Training We investigate layer specialization with different pre-training paradigms including the popular self-supervised strategy DINOv2 (Oquab et al., 2023) and the fully-supervised DeiT-III (Touvron et al., 2022). We also investigate different model sizes (ViT-B, L, H). Unless specified otherwise, we produce results with a ViT-L. model trained following DINOv2 recipe. Following et al. (2025), we integrate the attention bias strategy, which mitigates high-norm anomalies (Darcet et al., 2023) without introducing additional tokens, in all models and attention operations. More dis- cussion can be found in Sec. A. 1. For DINOv2, we train models ImageNet-22K (Ridnik et al., 2021) dataset for 600k training steps. For DeiT-III. we train our models on ImageNet-1K (Deng et al., 2009) for respectively 400 epochs VIT-B and 800 epochs on ViT-L. For both training paradigms, we pre-train treitrain using the first pre-training phase, and drop the high-resolution fine-tuning step. We report more details in Sec. A.3.
5

---

Published as a conference paper at ICLR 2026
coslux sinslaitt
0.4
baseline specialionl
0.4
Spec.
Lincar Acc.
Avg Seg.
Avg Depth 1
0.0
0.0
85,4
64.5
1.232 1.178
norms
85.1
65.6
attn.
MLP
adtn.
MLP
(a) [CLS] -pttches cosine sim.
(b) Patches- Patches-patches cosine sim.
(c) Norm specialization
Figure 6: Normalization specialization. Mean and standard deviation of the cosine similarity com- puted between (a) [CLS] and all patches and (b) all to all patches. We compare post-normalization statistics between the baseline architecture (DINOv2 ViT-L with attn. bias) and when specializing
the normalization layers for [CL.]] and patch token (specialized norms), ). (c) Quantitative results
with specialized normalization layers. 'LN' stands for LayerNorm and 'LSS' for LayerScale.
Evaluation Following Oquab et al. (2014), we assess model representations via linear probing on global, with ImageNet-Ik (Deng et al., 2009), and dense prediction tasks. For semantic seg- mentation, we use ADE20K (Zhou et al., 2017), Cityscapes (Cordts et al., 2016) and PASCAL VOC (Everingham et al., 2010), reporting mloU. For depth estimation, we use KITTI (Geiger et al., 2013), NYU Depth v2 (Nathan Silberman & Fergus, 2012) and SUN RGB-D (Song et al., 2015), reporting RMSE. For detection, use COCO (Lin et al., 2014), reporting AP. Some tables show average segmentation and depth scores across corresponding benchmarks. More details in Sec. A.3.
4.3 SPECIALIZING NORMALIZATION LAYERS
As discussed in Sec. 3, ViTs attempt to separate the [CLS] and patch tokens with the LayerNorm applied prior to the attention operation. Building on this observation, our initial experiment focuses on specializing the normalization layers (LayerNorms and Layer Scales) within the model, with the aim of further supporting the model's 's inherent tendency to separate these feature types.
We specialize the normalization layers in all blocks of the model. This lightweight modification introduces only 0.05% additional parameters, yet significantly alters the feature distributions. In Fig. 6a, we report the mean and standard deviation of the cosine similarity between the [CLS] and patch tokens, computed after each normalization layer. We compare our variant with specialized normalization weights to the baseline. Conversely, Fig. 6b shows the corresponding statistics when using all patches instead of the [CLS]. We observe that specializing the normalization layers further amplifies the disentanglement of the [CLS] and patch tokens, resulting in a more distinct separation of their embeddings after each normalization step.
The impact of these specialized normalizations is quantified in Fig. 6c. The specialization leads to significant improvements on dense prediction tasks, yielding an average increase of +1.1 mloU points on segmentation benchmarks and an improvement of -0.054m results highlight that a better specialization of the token types benefits the patches representations. On the other side, global results are slightly degrading. We however show in the next section that this loss can be mitigated. Unless otherwise specified, in the remainder of the paper, we apply specialized normalization layers to all transformer blocks.
depth estimation. These
4.4 BLOCK-LEVEL TARGETED SPECIALIZATION
While normalization layers in ViTs show overall [CLS] -patch separation effect, we have ob- served that the extent of their impact is not uniform across all blocks. It can be seen from Fig. 7, each block,
which depicts [CLS] -patches cosine similarity before and after the first LayerNorm that the separation effect of the normalization varies depending on its position within the model. Indeed, blocks at the beginning and near the end see the most impact. We hypothesize that the importance of separation in the early blocks stems from their proximity to the different inputs. Al- though the [CLS] token is trained to summarize information from the patches, is initialized as a learned parameter and thus has an input distribution very distinct from that of the patch tokens. Later in the model, separation becomes important again as tokens are closer to the final representations and the training objectives. The observations above suggest that can benefit from more targeted

---

Published as a conference paper at ICLR 2026
mioU
mmse
Specialized blocks
Linear Acc.
Avg
Avg
1.25
Seg.
Depth $
66
Ø
85.4
64.5
1.232
1.2
1st half
85.1
65.7
1.191
65
2nd half
85.2
64.5
1.236
64
block
1.15
block
All
85.0
65.9
1.191
10
(a) Which model's part to specialize
(b) Avg. segmentation perf.
(c) Avg. depth perf. +
Figure 8: Block specialization. (a) Performance metrics when specializing specific parts of the
model (b) Average segmentation scores and (c) average depth rmse (L) vs number of specialized blocks at the beginning of the model. Normalization layers are specialized in all blocks. The base model is a DINOv2 ViT-L with attention bias.
specialization within the model. We study next which blocks should be specialized to optimize the model's performance.
We first quantitatively compare the impact of spe- cializing different sections of the model. as shown
0.2
in Fig. 8a. To this end, we train DINOv2 while
before LN after TNN
specializing either the first half, the second or all
cosine similarity
of the transformer blocks, on top of specializing all
0.1
normalization layers. Within a block, all layers are
specialized. Our findings indicate that the best per- formance is achieved when specializing the early layers, which closest to the input. Specifically, specializing the first half of the layers improves the segmentation results by average of 1.2 mIoU linear
0.0
10
20
points, with only a negligible decrease accuracy. In contrast, specializing the late layers yields no improvement compared to the baseline. We attribute this the fact that [CLS] and patch to- kens share the representation space in the first part; once this interaction is established, further special- ization has limited effect. Finally, while specializing all layers produces the highest segmentation performance, comes with a higher memory cost and larger drop in linear accuracy.
block index
Figure 7:
Mean cosine similarity between [CLS] and all patches before and after the first LayerNorm (LN) of each block.
We further analyze how the number of specialized blocks, starting from the first, affects performance (Fig. 8b and 8c). We vary the number of blocks specialized from to 24 (total number blocks in ViT-L) in steps of 4, and observe that specializing the first third of the model yields the best results, while specializing later layers degrades performance. Notably, the optimal point at one third of the model's 's depth coincides with marked shift in the statisties of similarity scores shown in Fig. 7, which might explain the effectiveness of specializing the early layers.
4.5 TARGETED SPECIALIZATION WITHIN TRANSFORMERS BLOCKS
The previous section has shown that careful selection of transformer blocks for specialization is important for optimizing the performance. We now explore whether further improvements can be achieved with a targeted selection of specific layers to specialize within the transformer blocks. In the following experiments, we specialize different layers of blocks in the first third of the model while also applying specialization to the normalization layers in all blocks.
Table shows model performance on global and dense prediction tasks when specializing different layers (QKV projection, Linear and MLP, see Fig. 5). We observe that the performance on global task remain largely stable independently of the selected layers. In contrast, results on dense segmen- tation tasks get further improvements beyond what is achieved with normalization specialization alone. Interestingly, the gains do not increase monotonically with the number of specialized lay- additional parameters, might be expected from typical scaling laws (Touvron et al., 2021). Specializing either or both QKV and post-attention projections consistently yields improvements. In particular, the greatest performance gains are achieved by specializing the QKV projection, which
7

---

Published as a conference paper at ICLR 2026
Table 1: Layer specialization ablation. Performance and increase in parameter count of models trained with different layer specialization strategies, applied to the first third of the transformer blocks. In all cases, the normalization specialization described Sec. 4.3 is applied in all blocks. The base model is DINOv2 ViT-L with attention bias.
QKV
Parameter Increase (%)
Linear Accuracy
Linear
MLP
Avg. Seg.
Avg. Depth 4
proj.
0.05
85.1
65.6
1.178
85.3
66.1
1.191
85.2
66.6
1.165
85.1
1.180
22
85.1
65.2
1.189
25
85.1
65.9
1.163
30
85.3
65.6
1.185
85.3
1.174
baseline
with specialization
accuracy
mIoU
mloU
48
84
46
66
82
64
iter.
42
iter.
iter.
200k 400k
600k
200k 400k
600k
200k 400k
600k
(a) INIK
(b) ADE20K
(c) Cityscapes
mloU
84
0.4
82
2.9
0.38
80
2.8
iter.
iter.
200k
400k
600k
200k 400k
600k
200k 400k
600k
(d) VOC
(c) KITTI $
(f) NYU
Figure 9: Performances and training dynamics. Performance vs. training iterations on global task ImageNet classification (INIk) and dense tasks seliessttttttaioonn (ADE20K, Cityscapes, VOC) and depth (KITTI, NYU)-will witth linear probing. We compare baseline DINOv2 ViT-L with attn. bias and when specializing QKV projection in the first 1/3 of the model and all normalizations.
introduces only 8% additional parameters while delivering an average increase of +1 mloU point normalizations alone. In contrast, specializing the post-attention projection does not offer fur- ther benefits, and specialization of the MLP layer either has no effect or negatively impacts perfor- overhead with Low Rank Adaptation used
mance. Note that could ease this 8% memory when specializing the QKV projection. We produce encouraging preliminary results with different ranks Sec. A.5 and leave further investigation future work.
Our overall results show that increasing the disentanglement between [CLS] and patch tokens be- fore the attention mechanism (with separated normalizations and projection) contributes improved dense prediction performance. We hypothesize that encouraging the [CLS] and patch tokens to as- sume more distinct roles in the attention mechanism enhances their interactions, ultimately improv- ing overall model effectiveness. We also report results in Sec. A.6 for the setting where only the QKV projections are specialized, while the normalization layers remain shared. In this configura- tion, performance is comparable to the baseline, indicating that the specialization of normalization layers is critical to achieve improvements, as shown in Sec. 4.3.
In Fig. 9, compare performance dynamics of normalization and QKV projection specialization against baseline DINOv2 with attention bias. Across all dense benchmarks both segmentation and depth estimation specialization consistently enhances results. These improvements evident from early stages of training and continue to increase over time. This trend suggests that employing specialization not only boosts performance but also contributes more stable training dynamics.
8

---

Published as a conference paper at ICLR 2026
Table 2: Generalizability of the specialization on (a) DINOv2 when using different high-norm handling strategies (4 registers (Darcet et al., 2023), attention bias (An et al., 2025) ("attn. bias") or none (0)), (b) different ViT sizes on DINOv2 with attention bias framework and (c) a supervised framework: DeI Relative difference between baseline and our specialization ('+ours") is shown green if improvement and red otherwise.
Classif.
Segmentation
Depth
Detection
Method
Size
ImNet
ADE
City
VOC
KITTI
NYU
SUN
COCO
DINOv2 With high-norm handling strategies
Ø
85.3
45.7
64.2
82,1
2.868
0.389
0.410
45.6
+owrs
85.3+0.0%
47.3
66.613.7%
83.7.1.9%
2.787 -.8%
0.369.5.1%
0.390
46.8 v2.6%
4 registers
85.3
45.6
64,9
82.2
2.893
0.372
0.411
46.4
+ours
85,3+0.0%
47.5+4.2%
65.9+15%
83.6+1.7%
2.906+0.4%
0.367-1.3%
0.395
46.8 +0.9%
Attn. bias
85.4
46.2
65.2
82.2
2.917
0.373
0.406
46.0
85.2-0.2%
48.4+1.8%
67.4+3.1%
84.0+2.2%
2.739 6.19%
0.362-2.9%
0.393-3.2%
48.2
DINOv2 Other sizes
Attn. bias
80.4
38.3
58.4
76.6
3,250
0.462
0.464
39.6
+ours
80.6+0.2%
38.5+0.5%
60.3+3.2%
7..5-0.1%
3.236.0.4 0.4%%
0.448 3.0%
0.470+1.3%
39.8 +0.8%
Attn. bias
86,2
48.1
67.0
83.1
2.717
0.359
0.387
49.9
+ours
86.1
49.2+2.3%
67.1+0.1%
83.5+0.5%
2.752+1.3
0.344-12%
0.386-033
49,5 -0.8%
DeiT-II1
Attm. bias
81.8
25.4
61.7
48.9
5.040
0.747
0.823
32.8
+ours
81.7 -0.1%
26.3+3.5%
62.7+1.6%
50.7+3.7%
4.900 -.8%%
0.732-2.0%
0.809
-17%
32.5
-0.9%
4.6 GENERALIZATION RESULTS
We investigate the generalizability of our specialization approach across different variants of DI- NOv2, as presented the upper part of Table 2. Specifically, we train models using the DINOv2 recipe with two high-norm handler strategies registers (Darcet et al., 2023) ("4 registers") and at- tention bias (An et al., 2025) ("attn. bias") as well as without any handler. all cases, we observe that specialization consistently boosts dense prediction results by up to 4.8% on ADE20K, while hav- ing negligible effect on classification performance (no decrease greater than 0.2%). This shows that better separating the treatment of [CLS] and patch tokens is complementary to both high-norm handling strategies to improve dense features. We also investigate the specialization on DINOv2 VIT-B and ViT-H models and present results in the middle section of Table 2. It can be seen that our proposed specialization leads to improvements on most benchmarks, confirming its generalizability across different ViT model sizes.
We further explore the fully-supervised training setting by applying specialization to ViT-B trained with DEIT-III (Touvron et al., 2022) strategy. We observe consistent improvements in dense predic- tion tasks, with gains reaching to 3.7% on VOC. For ViT-L, specialization does not yield benefits, likely due to the training dynamics related the absence of a local loss guide dense feature learn- ing which causes the dense performance to degrade over time (we provide more details in Sec. A.7). These results suggest that the effectiveness of the specialization may depend on training objectives, highlighting promising directions for future research.
Finally, we visualize the learned patch representations using PCA in Fig. for DINOv2 models trained with either registers or attention bias. In both settings, incorporating our specialization strat- egy produces cleaner and more semantically meaningful patch representations. Specifically, this approach reduces artifacts in textures and uniform regions, resulting in more accurate object seg- mentation. More visualizations can be found in A.8.
4.7 MORE COMPARISON
We compare our specialization approach to the aaass-tttention mechanism from CaiT (Touvron et al., 2021) in Table 3. We train both models following the DINOv2 framework, with and without atten- tion bias. In CaiT architecture, patch tokens are processed through the transformer blocks, then the [CL.S] is appended and updated via 2 class-attention layers aggregate information from the patch

---

Published as a conference paper at ICLR 2026
Table 3: CaïT specialization results. ViT-L models trained with DINOv2 framework. We com- pare DINOv2 baseline (Ø) with class-attention (CaiT) and specialization (ours) architectures. Re- sults without and with attention bias (Attn. blas). Best result in bold.
Classif.
Segmentation
Depth +
Detection
Method
ImNet
ADE
City
VOC
KITTI
NYU
SUN
COCO
Ø CaiT Ours
85.3
45.7
64.2
82.1
2.868
0.389
0,410
45.6
84.0
43.5
63.7
80.8
2.968
0.397
0.421
43.1
85.3
47.3
66.6
83.7
2.787
0.369
0.390
46.8
Alln. bias
85.4
46.2
65.2
82.2
2.917
0.373
0.406
46.0
Attn. bias + CaiT
85.2
45.3
65.7
82.0
2.882
0.374
0.400
46.8
Attn. bias ours
85.2
48.4
67.4
84.0
2.739
0.362
0.393
48.2
tokens of the last block. We report results in Table 3, we observe that our specialization consistently outperforms the class-attention mechanism across all downstream tasks.
CONCLUSION
In this work, we investigate the disentanglement of [CLS] and patches computations in Vision Trans- formers, focusing on their distinct roles and interactions. Through a comprehensive analysis, we demonstrate that disentangling their processing pathways and selectively specializing architectural layers leads to significant improvements in dense prediction tasks, including segmentation and depth estimation, while maintaining strong global performance. Our approach achieves these gains with- out increasing computational overhead, with minimal additional parameter cost, and generalizes multiple ViT architectures and frameworks. These findings highlight the importance of tai- lored architectural designs and suggest promising directions for future research, including further exploration of efficient specialization strategies and applications to broader modalities and tasks.
ACKNOWLEDGGENNTS
We thank Francisco Massa and Maximilian Seitzer for their insightful discussions and help during the course of this work.
REFERENCES
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang, Systematic outliers in large language models. ICLR, 2025.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In ICLR, 2023.
Daniel Bolya, Yao Huang, Peize Sun. Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi. Jathushan Rajasegaran, Hanoona Rasheed, al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504. 13181, 2025.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2026
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. 166888 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition at scale. In ICLR, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. Forty-first international conference on machine learning, 2024.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor Guilherme Turrisi da Costa, Louis Béthune, Zhe Gan, Alexander Toshev, Marcin Eichner. Moin Nabi, Yinfei Yang, Joshua M. Susskind, and Alaaeldin El-Nouby. Multi- modal autoregressive pre-training of large vision encoders, 2024.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The international journal of robotics research, 32(1 22111112311127, 2013.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Álvaro Bernardo, Bilal Pires, Zhaohan Guo, al. Bootstrap your own latent-a new approach to self- learning. In Advances in Neural Information Pro- cessing Systems, volume 33, pp. 21271-21284, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (22)3,2022
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yutong Lin, Yuhui Yuan, Zheng Zhang, Chen Li, Nanning Zheng, and Han Hu. Detr does not need multi-scale or locality design. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5545-6554, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu. Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012-10022 October 2021.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level im- age representations using convolutional neural networks. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1717-1724, 2014.
11

---

Published as a conference paper at ICLR 2026
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. In International Conference on Machine Learning, pp- 27408-27438. PMLR, 2023.
Valentinos Pariza, Mohammadreza Salehi, Gertjan Burghouts, Francesco Locatello, and Yuki M Asano. Near, far: Patch-ordering enhances vision foundation models' scene understanding. arXiv preprint arXiv:2408. /054, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21K pretraining for the masses, arXiv preprint 1rXiv:2104.1097 2021.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, al. Dinov3. arXiv preprint arXiv:2508. 10120000,0000,,,. 2025.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 567-576, 2015.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7262-7272, October 2021.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17700 2024.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 2020.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, pp. 42-42,2210 2021.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In ECCV, pp. 516- 533. Springer, 2022.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502. 14786, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of attention masks and layer norm in transformers. Advances in Neural Information Processing Systems, 37: 14774-14809, 2024.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi- Zi-Hang Jiang, Francis E.H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-toker vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralb. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2017.
12

---

Published as a conference paper at ICLR 2026
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arx1:22111.0782,, 2021.
A
APPENDIX
A.I ADDRESSING TOKEN INTERACTION ANOMALIES
In this work, we examine how the distinct roles of [CLS] and patch tokens affect their interactions within the model. Previous works (Darcet al., 2023; Sun et al., 2024) show that despite sharing computational pathways, these different token types develop inter-dependencies that can lead to token anomalies, manifested high-norm outliers in the patch features space. These anomalies suggest an underlying tension in how information flows between global and local representations.
Registers. order to mitigate such artifacts, observed when using different pre-training strate- gies (Oquab et al., 2023; Touvron et al., 2022; Radford et al., 2021), Darcet et al. (2023) propose to add learnable register tokens to the input sequence, whose roles are to replace the high-norm patches in the internal communication between patches and the [CLS] token. Doing mitigates the appearance of such artifacts and boost overall results.
Attention bias. The recent study on artifacts Large Language Models by An et al. (2025) in- vestigates the systematic appearance of outliers which they link to the attention mechanism. They propose solution consisting in adding learnable biases to the keys and values each attention head. They analyze the equivalence of their solution compared to registers.
Table The impact of norm handling strategies on DINOv2 results.
Norm. method
IN
ADE
City.
NYU
Ø
85.3
45.7
64.2
0.389
registers
85.3
45.6
64.9
0.372
attn. bias
85.4
46.2
65.2
0.373
In our experiments, we observe that both strategies have a similar impact on high-norm artifacts and seen in Table best overall performance achieved when using the attention bias ('attn. bias') strategy, with a significant improvement on segmentation benchmarks (e.g. ADE20k and Cityscapes). To minimize confounding factors that could affect the interaction between the [CLS] and patch tokens, we adopt the attention bias strategy, which mitigates high-norm anomalies without introducing additional tokens.
A.2 EFFECT OF OTHER LAYERS ON [CLS] -PATCHES SIMILARITIES
We report in Fig. 10, the effect on [CL.5] -patches similarity of the ML. and post-MLP LayerScale layers within transformer blocks. The MLP layer, similar to the self-attention layer, increases the similarity between [CLS] and patches as it aligns the features. The post-MLP LayerScale, similar to other normalization layers, shows stronger disentangling effect.
A.3 TRAINING AND EVALUATION DETAILS
Throughout this work, follow the experimental protocol of Oquab et al. (2023) and evaluate the performance of the trained models on a set of global and dense task benchmarks.
Classification For the global task, we perform linear probing on ImageNet classifi- cation (Deng et al., 2009). We train linear layer with SGD for 12500 itera- tions, using random-resized-crop data augmentation, and the [CLS] token as input for the linear layer. We also perform the following grid search 51. 2.0e -5 0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1)
learning rate
We then report the highest accuracy value obtained the validation set as is common practice.
13

---

Published as a conference paper at ICLR 2026
befor after
cosine simllrity
0.2
0.2
0.0
0.0
-0.2
0.2
[CLSS-patehes pulches-palcies
[CLS)-patches palches-patcins
(a) MLP
(b) post-MLP LayerScale
Figure 10: Effect of the MLP and post-MLP LayerScale layers [CLS] -patches similarity in vanilla DINOv2 pre-trained model. We show mean and standard deviation of the cosine similarity between [CLS] and all patches ('CI.S-patches', and between patches ("patches-patches) before and after the considered layers.
Segmentation For semantic segmentation, we use ADE20K (Zhou et al., 2017), Cityscapes (Cordts et al., 2016), and VOC (Everingham et al., 2010), and report the mean Intersection over Union (mloU) scores for each. When we report the average performance segmentation tasks, we average the scores across these 3 datasets. We train linear classifier the training set of each benchmark for 40000 iterations with a learning rate of Te-3 This linear layer is applied on top of the patch output features (after the last layer normalization) of the frozen backhone, with the features further normalized using a trained batch normalization layer.
Depth estimation For depth estimation, we evaluate on KITTI (Geiger et al., 2013), NYU Depth v2 (Nathan Silberman & Fergus, 2012), and SUN RGB-D (Song et al., 2015), reporting the average Root Mean Squared Error (rmse) scores. When we report the average performance of dense tasks, average the scores across these datasets. We train linear classifier benchmark for 38400 iterations with a learning rate of le -3, For the input of this linear layer, we take patch and [CLS] output features from four evenly spaced layers of the backbone, not applying the last layer normalization.
the training set of each
Detection For detection, we evaluate on COCO (Lin et al., 2014), reporting the average precision (AP) score. We train upon the Plain-DETR implementation (Lin al., 2023) using the configuration provided in the official repository. More specifically, we train RPE DETR model during 12 epochs with a linear rate of 0.0002 and 1000 steps warmup. Compared to the default configuration, and following Siméoni et al. (2025) evaluation framework, we keep the VIT encoder frozen.
When training with DINOv2 and DeiT-III models, we follow the default configurations provided in the official repository, modified to add biases in the attention and to specialize layers or blocks.
When training with CaiT, we follow the default architecture provided in Touvron et (2021) and train the models using the self-supervised framework of DINOv2. We use the same hyperparameters as DINOv2 experiments.
A.4 SPECIALIZATION OF NORMALIZATION LAYERS IN DEIT-III
We report in Fig. 11 the impact of the specialization of the normalization layers when using DeiT-III pre-training strategy. Similar to the case of DINOv2, the average [CLS] -patch cosine similarity significantly reduces when employing the specialized normalization, showing the disentanglement effect.
A.5 LORA APPROXIMATION
As the parameters increase can bottleneck for training efficiency, we explore the use of Low- Rank Adaptation (LoRA) (Hu et al., 2022) techniques to reduce the number of trainable parameters while maintaining performance. Additionally, we hypothetise that [CLS] and patches representa- tions share common features. Hence consider [CL.S] stream specialization of patches stream
14

---

Published as a conference paper at ICLR 2026
cosine sinilarity
0.2
baseline specialized
0.2
0.0
0.0
LN
LN
LN
MLP
nrtn.
MLP
(a) [CLS] -patch cosine similarity
(b) all-all patch cosine similarity
Figure 11: Specialization of normalization layers. Mean and standard deviation of the cosine sim- ilarity computed between (a) the [CLS] and all patches and (b) all patch to all patches. The average is computed over 1 images and all model blocks. We compare post-normalization statistics between the standard architecture ("Baseline") ") and the model after normalization specialization ("Specialized norms"), ). 'LN' stands for LayerNorm and 'LS' for LayerScale.
instead of complete different stream. Then, for a layer / that we choose specialize, we compute the operation on the class token Fets as the sum of the patches layer Spatch and a low-rank adaptation (LoRA) decomposition p(() of rank
Patches
QKV proj.
Param. Incr. (%)
Linear Acc.
Avg Seg.
Avg Depth 1
Specialization
Attention
85,4
64.5
1.232
norms +KKV
0.05
85.1
65.6
1.178
8.3
85.2
66.6
1.165
LoRA
+L.RA QKV r=16
0.2
85.3
65.8
1.188
CLS
+LoRA QKV r= 128
1.4
85.2
65.9
1.193
(a) LoRA design
(b) Results with LoRA
Figure 12: LoRA impact. (a) Visualization of LoRA design: [CLS] as an approximation of patches. (b) Performance metrics and parameter increase for different LoRA configurations (rank 16 and 128) during first third of the model. In all cases, the normalization specialization described in Sec. 4.3 is applied, corresponding to 'norms' row.
We conduct experiments in which we specialize normalization layers and the QKV projections with LoRA approximations of ranks 16 and 128 (over embedding dimension of 1024). The results presented in Fig. 12b shows improvements (+-0.2 and +0.3 in segmentation tasks) over specializing only the normalization layers, while adding limited number of parameters (+0..15% and +1.35% respectively). We leave further investigations as future work.
A.6 NORMALIZATIONS ARE NEEDED
Additionally to the specialization experiments we produced in Sec. 4.5, we also conduct an exper- iment specializing QKV projection during the first third of the model, but not the normalization layers. We plot the results of this experiment in Table 5 compared to the baseline and to our best model specializing normalization layers and QKV projection during third of the model. We observe that specializing only QKV projections brings little improvement over the baseline, e.g. +0.2 mloU pt in average on segmentation tasks. This shows that specializing the normalization layers is crucial for best performance.
A.7 ADDITIONAL RESULTS DEIT-III
We report in Fig. 13 the performance curves VOC segmentation task during the training of VIT-B and ViT-L models when following DIIT-III. We observe that the performance reaches its peak in the middle of the pre-training, then drops significantly towards the end. We attribute this behavior
15

---

Published as a conference paper at ICLR 2026
Table 5: Importance of specializing norms. Performance of for different layer specialization (Spec.) strategies applied on the first third of the transformer blocks). Normalization layers are specialized in all blocks. Baseline is ViT-L DINOv2 with attention bias.
Linear Acc.
Model
Avg. Seg,
Avg. Depth J
Baseline
85,4
64.5
1.232
Specialized norms
85.1
65.6
1.178
Specialized norms & QKV proj. Specialized QKV proj.
85.2
66.6
1.165
85.4
64.7
1.211
baseline
with specialization
mloU
mloU
66
64
65
62
iter.
60
iter.
100k
200k
300k
400k
200k
400k
600k
800k
(a) ViT-B
(b) ViT-L
Figure 13: DeiT-III training evolution. We visualize VOC segmentation performance (mloU) throughout training for (a) ViT-B and (b) ViT-L pre-trained with DeiT-III ('baseline") and when adding our layer specialization.
to the lack of local loss to drive dense performances. We observe a significant gain with our specialization in the first half of the training, but the gains are then diluted in the drop, particularly in the case of ViT-L.
A.8 OTHER QUALITATIVE RESULTS
We produce in Fig. 14, and 16 more qualitative results when pre-training the model following DINOv2 with the vanilla architecture, four registers attention bias and when integrating our spe- cialization. Each figure shows the first three components, computed with the patch features, and mapped RGB. In all cases, we observe that the specialization helps to produce more precise patch features with less artifact. For instance, we invite the reader to pay attention to the back of the dog (first row), where the artifacts visible in the original pre-training are notably reduce with our specialization.
A.9 WRITING DETAILS
We have used Large Language Models (LLMs) to help write and proofread this paper. More specifi- cally, they have helped to rephrase some parts of the text, propose synonyms, and check the grammar. We have carefully checked all the outputs the LLMs to ensure that they are accurate and faithful our work.
16

---

Published as a conference paper at ICLR 2026
Original image
(a) Vanilla DINOv2
(a) + ours
Figure 14: First PCA components of model outputs in RGB. Specialization of normalizations and QKV projections is made during 1/3 of the model. ViT-L with vanilla DINOv2.
17

---

Published as a conference paper at ICLR 2026
Original image
(a) DINOv2 w/ registers
(a) + ours
Figure 15: First PCA components of model outputs in RGB. Specialization of normalizations and QKV projections is made during 1/3 of the model. ViT-L DINOv2 with four registers.
18

---

Published as a conference paper at ICLR 2026
Original image
(a) DINOv2 w/ attn. bias
(a) + ours
Figure 16: First PCA components of model outputs in RGB. Specialization of normalizations and QKV projections is made during 1/3 of the model. VIT-L DINOv2 with attention bias.
19


Reviews:
[{"review_id": "MCvjomO4kx", "reviewer": "Reviewer_rfYY", "summary": "The paper analyzes how Vision Transformers (ViTs) process the global class token [CLS] and local patch tokens and argues that treating them identically causes friction for dense prediction tasks. Building on observations about token dynamics and normalization, the authors propose layer specialization: in selected blocks they use separate parameters for [CLS] vs. patch tokens (e.g., distinct LayerNorms and Q/K/V projections) while keeping attention shared, so information can still flow. On ImageNet pretraining (supervised DeiT-III and self-supervised DINOv2), they report essentially unchanged ImageNet classification accuracy but around +2 mIoU improvements on semantic segmentation and better depth RMSE, with only a small (~8%) parameter increase. The approach relates to earlier ViT foundations (ViT, DeiT-III, DINO/DINOv2) and to fixes for high-norm/outlier tokens (e.g., Registers) while differing from CaiT (which adds dedicated class-attention layers) by specializing within standard blocks rather than inserting new class-attention stages.\n\nSummary of the review: The paper presents a clear and well-motivated analysis of why identical processing of global and local tokens can limit dense prediction, proposing a simple yet effective solution via token-type-specific normalization and projections. The method is elegant, lightweight, and yields consistent improvements (~+2 mIoU) across models and pretraining setups. However, the novelty is somewhat incremental given overlap with CaiT’s class-attention, and broader validation (e.g., detection, in-context tasks) plus efficiency reporting would strengthen the case. Overall, it’s a technically solid and well-executed contribution with moderate originality, justifying a rating of 6 for clear insight and strong empirical support.", "questions": "1. CaiT vs. this work: On equal pretraining and decoder setups, how does your method compare to CaiT backbones for dense tasks? Any synergy if both are combined? \n\n2. Registers vs. specialization: With identical training, which helps dense tasks more on its own—Registers or your specialization? Any redundancy when combined? \n\n3. Detection results: Do COCO detection/instance-seg metrics show similar benefits? \n\n4. In-Context Visual Understanding: Does the model gain any benefits on the recent proposed Hummingbird evaluation [1] ( implemented openly by [2])?\n\n5. Stability & hyperparams: Does specialization require different learning-rate/weight-decay schedules to avoid the minor classification drops you mention in fully-specialized variants?\n\n[1] Balažević, I., Steiner, D., Parthasarathy, N., Arandjelović, R., & Hénaff, O. J. (2023). Towards In-context Scene Understanding. arXiv [Cs.CV]. http://arxiv.org/abs/2306.01667\n[2] https://github.com/vpariza/open-hummingbird-eval", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) Clear problem framing: Why identical processing of global/local tokens can hinder dense tasks is well-motivated and consistent with literature on token dynamics and normalization. \n\n2) Simple, effective change: Token-type-specific LayerNorm/QKV yields consistent gains without changing attention/FLOPs; minimal engineering overhead.\n\n3) Robust evaluation: Results across pretraining regimes (DINOv2, DeiT-III) and model sizes; improvement magnitude (~+2 mIoU) is meaningful for segmentation.", "weaknesses": "1. Novelty overlap: Since CaiT[1] already introduces class-attention (explicitly specializing how [CLS] aggregates information), authors should include a head-to-head comparison or discussion clarifying differences/advantages (e.g., same backbone size/recipe; dense-task transfer). \n\n2. Limited Task coverage: Add a few more tasks, like the hummingbird evaluation for in-context vision understanding, object detection/instance segmentation (e.g., COCO) to show benefits generalize beyond conventional segmentation/depth.\n\n3. Training efficiency: Report training memory/time impact of the ~8% parameter increase (batch size, wall-clock), not just inference parity.\n\n[1] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., & Jégou, H. (2021). Going deeper with Image Transformers. arXiv [Cs.CV].  http://arxiv.org/abs/2103.17239", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hks0R3mome", "reviewer": "Reviewer_HUNB", "summary": "This paper proposes an architecture specialization method to treat the CLS token and patch tokens distinctively in the ViT models, resulting in a disentangled distribution of two types of tokens. The sufficient and reasonable experiments support the methods and hyper-parameter configurations. The visualizations also coincide with the effectiveness in dense prediction tasks.", "questions": "The questions have been listed in the weakness part. I would consider further raising the score if the authors could address my concerns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The method section, along with comprehensive experiments, is impressive. The analyses are comprehensive with both qualitative and quantitative results. The statements are validated by the experiments and clearly support the proposed Specialization for the CLS token and patch tokens.\n\n2. The proposed method is compatible with existing methods (register tokens and attention bias), further improving the overall results of ViT across multiple benchmarks, as shown in Table 2. And it can also stabilize downstream tasks training, as shown in Figure 9. The above experiments stand for the practical application of the proposed method in the general vision area.\n\n3. Table 1 demonstrates that the performance gain does not simply come from the increasing parameters, but rather an intricate design through complete analyses.", "weaknesses": "1. The motivation is unclear at the beginning, such as in the Abstract, Introduction, and Methods. The motivation starts with rethinking the model architecture design, while the specific target is to improve the downstream performance. I would recommend starting with the tasks and the subsequent modifications that could leverage the disentangled distribution to improve the performance.\n\n2. The analysis and explanation of \"Dimension separation\" from L206 is not clear enough. For example, how to better understand Figure 4?  Is the other axis the \"token index\"? More explanation and a clear figure are required. \n\n3. What is the training strategy for ViT-H? This detail is missing.\n\n4. How to explain the visualization impact? For example, Figure 1 demonstrates the two distinctive results by register token and attn bias. However, when applying the proposed specialization method, the visualization becomes similar in general. Can it be explained that the specialization is more fundamental and effective in changing the feature distribution? In addition, from Figures 14-16, the specialization method produces more ambiguous and blurred visual details. Why could these features contribute to better dense prediction tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zLOEmy3Hqj", "reviewer": "Reviewer_3jym", "summary": "This paper investigates the interaction dynamics between the [CLS] token and patch tokens in Vision Transformers (ViTs), particularly under different pre-training strategies. The authors identify that standard ViT architectures treat both token types identically, despite their distinct roles—[CLS] for global representation and patches for local features. Through empirical analysis, they reveal that normalization layers implicitly differentiate these tokens. Building on this insight, they propose architectural modifications that explicitly specialize the processing paths for [CLS] and patch tokens, especially in normalization and QKV projection layers. Their approach improves dense prediction tasks (e.g., segmentation and depth estimation) by up to 2 mIoU points, with minimal parameter overhead and no increase in computational cost.", "questions": "* Why the proposed specialization strategy hurts the performance of image classification (Table 6(c) and Figure 9(a)) but benefits dense prediction tasks? Especially, the [CLS] token, which your method focus on, seems to be more important for image classifiction rather than dense prediction tasks.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed specialization strategy is simple yet impactful, improving dense prediction performance with only ~8% increase in parameters and no additional FLOPs.\n\n* The authors conduct extensive evaluations across model scales (ViT-B, L, H), training paradigms (DINOv2, DeiT-III), and tasks (segmentation, depth estimation, classification).", "weaknesses": "* Lack of theoretical justification. The paper is empirically strong but could benefit from deeper theoretical grounding to explain why specialization improves representation quality.\n\n* To highlight the effectiveness of the proposed layer specialization, while PCA visualizations are provided (Figure 1), more quantitative metrics on feature quality (e.g., clustering scores) could strengthen the argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)