Title: Revisiting [CLS] and Patch Token Interaction in  Vision Transformers
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: representation, vision, transformer, SSL, attention, specialization, architecture, interpretability, DINO, DINOv2, CLIP, DEIT

Abstract: Vision Transformers have emerged as powerful, scalable and versatile representation learners. To capture both global and local features, a learnable [CLS] class token is typically prepended to the input sequence of patch tokens. Despite their distinct nature, both token types are processed identically throughout the model.
In this work, we investigate the friction between global and local feature learning under different pre-training strategies by analyzing the interactions between class and patch tokens.
Our analysis reveals that standard normalization layers introduce an implicit differentiation between these token types. Building on this insight, we propose specialized processing paths that selectively disentangle the computational flow of class and patch tokens, particularly within normalization layers and early query-key-value projections.
This targeted specialization leads to significantly improved patch representation quality for dense prediction tasks. Our experiments demonstrate segmentation performance gains of over 2 mIoU points on standard benchmarks, while maintaining strong classification accuracy. The proposed modifications introduce only an 8\% increase in parameters, with no additional computational overhead.
Through comprehensive ablations, we provide insights into which architectural components benefit most from specialization and how our approach generalizes across model scales and learning frameworks.

TL;DR: We propose and analyze a new architecture to specialize CLS and patch tokens processing in ViTs, enhancing dense tasks performances.