Vision Transformers Need Registers
Authors: Timothée Darcet
Maxime Oquab
Julien Mairal
Piotr Bojanowski
Venue: ICLR 2024 oral (2024)
Keywords: representation
vision
transformer
register
SSL
CLIP
attention
attention map
interpretability
DINO
DINOv2
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We find artifacts in ViT features. We add new tokens (“registers”) that fix this issue.

Abstract: Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing.

Published conference paper at ICLR 2024
VISION TRANSFORMERS NEED REGISTERS
Timothée Darcet
Maxime Oquab!
Julien Mairal
Piotr Bojanowski
FAIR, Meta
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP. LJK, 38000 Grenoble, France
ABSTRACT
Transformers have recently emerged powerful tool for learning visual rep- resentations. In this paper. we identify and characterize artifacts in feature maps of both supervised and self-supervised VIT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative back- ground areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input se- quence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervieed models, sets new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads smoother feature maps and attention maps for downstream visual processing.
Without registers OpenCLIP
With registers OpenCLIP
Input
DeiT-IIII
DINOv2
DeiT-III
DINOv2
Figure Register tokens enable interpretable attention maps in all vision transformers, similar the original DINO method (Caron et al. 202T). Attention maps are calculated in high resolution for better visualisation. More qualitative results are available in appendix H
INTRODUCTION
Embedding images into generic features that can serve multiple purposes in computer vision has been long-standing problem. First methods relied on handcrafted principles, such as SIFT (Lowe) 2004), before the scale of data and deep learning techniques allowed for end-to-end training. Pur- suing generic feature embeddings is still relevant today. as collecting valuable annotated data for many specific tasks remains difficult. This difficulty arises because of the required expertise (e.g., medical data, or remote sensing) or the cost scale. Today, is common to pretrain a model for a task for which plenty of data is available and extract subset of the model to use as feature extractor. Multiple approaches offer this possibility; supervised methods, building classification
Conresoondecce mda @meta.ccomm com

---

Published conference paper at ICLR 2024
Input
DeiT-IIIBB
DeiT-II-LL
OpenCLIP-B
OpenCI.IP-1.
DINO-B
DINOv2-8
Figure 2: Illustration of artifacts observed in the attention maps of modern vision transformers. We consider ViTs trained with label supervision (DeiT-III), text-supervision (OpenCLIP) or self- supervision (DINO and DINOv2). Interestingly, all models but DINO exhibit peaky outlier values in the attention maps. The goal of this work is to understand and mitigate this phenomenon.
or text-image alignment, allow training strong feature models unlock downstream tasks. Alterna- tively, self-supervised methods building on the Transformer architecture have attracted significant attention due to their high prediction performance downstream tasks and the intriguing ability of some models to provide unsupervised segmentations (Caron et 2021)
particular, the DINO algorithm is shown to produce models that contain explicit information about the semantic layout of an image. Indeed, qualitative results show that the last attention layer naturally focuses semantically consistent parts of images and often produces interpretable attention maps. Exploiting these properties, object discovery algorithms such as LOST (Siméoni et al.) (2021) ) build on top of DINO. Such algorithus can detect objects without supervision by gathering information attention maps. They effectively unlocking new frontier computer vision.
DINOv2 (Oquab al. 2023). follow-up DINO, provides features that allow tackling dense prediction tasks. DINOv2 features lead to successful monocular depth estimation and semantic seg- mentation with frozen backbone and linear models. Despite the strong performance on dense tasks, we observed that DINOv2 surprisingly incompatible with LOST. When used extract features. it delivers disappointing performance, only on par with supervised alternative backbones this sce- nario. This suggests that DINOv2 behaves differently than DINO. The investigation described in this work notably exposes the presence of artefacts in the feature maps of DINOv2 that were not present in the first version of this model. These are observable qualitatively using straightforward methods. Also surprisingly, applying the same observations to supervised vision transformers ex- poses similar artifacts, shown in Fig. 2 This suggests that DINO is, in fact, exception, while DINOv2 models match the baseline behavior of vision transformers.
In this work, we set out to better understand this phenomenon and develop methods to detect these artifacts. We observe that they tokens with roughly 10x higher norm the output and correspond a small fraction of the total sequence (around 2%).. also show that these tokens appear around the middle layers of the vision transformer, and that they only appear after a sufficiently long training of sufficiently big transformer. In particular, show that these outlier tokens appear in patches similar to their neighbors, meaning patches that convey little additional information.
As part of our investigation, we evaluate the outlier tokens with simple linear models to under- stand the information they contain. We observe that, compared to non-outlier tokens, they hold less information about their original position in the image the original pixels in their patch. This ob-

---

Published conference paper at ICLR 2024
input image
DINO norms
DINOv2 norms
DINO norms
DINOv2 norms
80
10
10
20
Lg nom
La norm
Figure 3: Comparison of local feature norms for DINO ViT-B/16 and DINOv2 WTT-g/14. We ob- serve that DINOv2 has few outlier patches, whereas DINO does not present these artifacts. For DINOv2, although most patch tokens have norm between and 100, small proportion of tokens have very high norm. We measure the proportion of tokens with norm larger than 150 at 2.37%.
servation suggests that the model discards the local information contained in these patches during inference. On the other hand, learning an image classifier on outlier patches yields significantly stronger accuracy than doing so the other patches, suggesting that they contain global informa- tion about the image. We propose the following interpretation these elements: the model learns to recognize patches containing little useful information, and recycle the corresponding tokens to aggregate global image information while discarding spatial information.
This interpretation is consistent with inner mechanism in transformer models that allows per- forming computations within restricted set of tokens. In order test this hypothesis, append additional tokens - that we call registers to the token sequence, independent of the input image. We train several models with and without this modification and observe that the outlier tokens disappear from the sequenee entirely. As result, the performance of the models increases in dense prediction tasks, and the resulting feature maps are significantly smoother. These smooth feature maps enable object discovery methods like LOST mentioned above with the updated models.
2 PROBLEM FORMULATION
As shown in Fig. most modern vision transformers exhibit artifacts in the attention maps. The unsupervised DINO backbone (Caron et al. 2021) has been previously praised for the quality of local features and interpretability of attention maps. Surprisingly, the outputs of the subsequent DINOv2 models have been shown hold good local information but exhibit undesirable artifacts attention maps. In this section, we propose to study why and when these artifacts appear. While this work focuses alleviating artefacts in all vision transformers, we focus our analysis DINOv2.
2.1 ARTIFACTS
LOCAL FEATURES DINOv2
Artifacts are high-norm outlier tokens. We want to find a quantitative way of characterizing arteffacs that appear in the local features. We observe that important difference between "artifact" patches and other patches is the norm of their token embedding at the output of the model. Fig. 3 (left), we compare the norm of local features for a DINO and DINOv2 model given a reference image. We clearly see that the norm of artifact patches is much higher than the norm of other patches. We also plot the distribution of feature norms over small dataset of images in Fig. B (right), which is clearly bimodal, allowing us to choose a simple criterion for the rest of this section: tokens with norm higher than 150 will be considered as "high-norm" tokens, and we will study their properties relative regular tokens. This hand-picked cutoff value can vary across models. In the rest of this work, we use "high-norm" and "outlier" interchangeably.
Outliers appear during the training of large models. We make several additional observations about the conditions in which these outlier patches appear during the training of DINOv2. This analysis is illustrated in Fig. 4 First, these high-norm patches seem to differentiate themselves from other patches around layer 15 of this 40-layer ViT (Fig. 4a Second, when looking the distribution of norms along training of DINOv2, we see that these outliers only appear after one third training (Fig. [b)). (b)..[4b). Finally, when analyzing more closely models of different size (Tiny, Small, Base, Large, Huge and giant), that only the three largest models exhibit outliers (Fig. 4ch

---

Published conference paper at ICLR 2024
10-1
300
10
10
300
mmm
30
10
200 200
proportoon
nom
30
10
10
proparion
10
10
112K 312K 512K
Inyer
pretrain
(a) Norms along layers.
(b) Norms along iterations.
(C) Norms across model size.
Figure 4: Illustration of several properties of outlier tokens in the 40-layer DINOv2 ViT-g model. (a): Distribution output token norms along layers. (b): Distribution norms along training iterations. (c): Distribution of norms for different model sizes. The outliers appear around the middle of the model during training; they appear with models larger than and including ViT-Large.
20
normal patches
density
artifact patches
position prediction
reconstruction
10
top-1 acc
avg. distance |
L2 error J
normal outlier
41.7
0.79
18.38
0
22.8
5.09
25.23
(a) Cosine similarity to neighbors.
(b) Linear probing for local information.
Figure (a): Distribution of cosine similarity hetween input patches and their neighbors. We plot separately artifact patches (norm of the outpur token over 150) and normal patches. (b): Local information probing normal and outlier patch tokens. We train two models: one for predicting position, and one for reconstructing the input patch. Outlier tokens have much lower scores than the other tokens, suggesting they are storing less local patch information.
High-norm tokens appear where patch information is redundant. To verify this, we measure the cosine similarity between high-norm tokens and their 4 neighbors right after the patch em- bedding layer (at the beginning of the vision transformer). We illustrate the density plot in Fig. 5a We observe that high-norm tokens appear on patches that are very similar their neighbors. This suggests that these patches contrain redundant information and that the model could discard their information without hurting the quality of the image representation. This matches qualitative observations (see Fig, 2 that they often appear in uniform, background areas.
High-norm tokens hold little local information. order to better understand nature these tokens, we propose to probe the patch embeddings for different types of information. For that we consider two different tasks: position prediction and pixel reconstruction. For each of these tasks, we train linear model on top of the patch embeddings, and measure the performance of this model. We compare the performance achieved with high-norm tokens and with other tokens, to see if high-norm tokens contain different information than "normal" tokens.
Position prediction. We train a linear model to predict the position of each patch token in the image, and measure its accuracy. We note that this position information was injected in the tokens before the first ViT layer in the form of absolute position embeddings. We observe that high-norm tokens have much lower accuracy than the other tokens (Fig. 55). suggesting they contain less information about their position in the image.
Pixel reconstruction. We train linear model to predict pixel values the image from the patch emheddings, and measure the accuracy of this model. We observe again that high-norm tokens achieve much lower accuracy than other tokens (Fig- (5b). This suggests that high-norm tokens contain less information to reconstruct the image than the others.
Artifacts hold global information. In order to evaluate how much global information is gathered the high-norm tokens, we propose evaluate them standard image representation learning

---

Published conference paper at ICLR 2024
INIk
P205
Airc.
CFIO
CF1(0)
CUB
Cal
Cars
DTD
Flow.
Food Pets
SUN
VOC
[CLS]
86.0
66.4
87.3
99.4
94.5
91.3
96.9
91.5
85.2
99.7
94.7
96.9
78.6
89.1
normal
65.8
53.1
17.1
97.1
81.3
18.6
73.2
10.8
63.1
59.5
74.2
47.8
37.7
70.8
outlier
69.0
55.1
79.1
99.3
93.7
84.9
97.6
85.2
84.9
99.6
93.5
94.1
78.5
89.7
Table 1: Image classification via linear probing on normal and outlier patch tokens. We also report the accuracy of classifiers learnt on the class token. We see that outlier tokens have a much higher accuracy than regular ones, suggesting they are effectively storing global image information.
output
Transformer Model
[CLS]
[REG1]
[REON]
[
input patches
Figure 6: Illustration of the proposed remediation and resulting model. We add N additional learn- able input tokens (depicted in yellow), that the model can use as registers. At the output of the model, only the patch tokens and [CLS] tokens are used, both during training and inference.
benchmarks. For each image in classification dataset, forward through DINOv2-g and tract the patch embeddings. From those, we choose single token at random, either high-norm or normal. This token is then considered as the image representation. We then train a logistic regres- sion classifier to predict the image class from this representation, and measure the accuracy. We observe that the high-norm tokens have a much higher accuracy than the other tokens (Table [ This suggests that outlier tokens contain more global information than other patch tokens.
2.2 HYPOTIESIS AND REMEDIATION
Having made these observations, we make the following hypothesis: large, sufficiently trained mod- els learn to recognize redundant tokens, and to use them as places to store, process and retrieve global information. Furthermore, posit that while this behavior is not bad in itself, the fact that happens inside the patch tokens undesirable. Indeed, leads the model discard local patch information (Tab, (b), possibly incurring decreased performance on dense prediction tasks.
We therefore propose a simple fix to this issue: we explicitly add new tokens to the sequence, that the model can learn to as registers. We add these tokens after the patch embedding layer, with vision transformer, these tokens are discarded, and the [CLS] ] token and patch tokens are used as image representations, as usual. This mechanism first proposed in Memory Transformers (Burtsev al. 2020), improving translation tasks in NLP. Interestingly, y, we show here that this mechanism admits a natural justification for vision transformers, fixing an interpretability and performance issue that was present otherwise.
learnable value, similarly to [CTS]] token. the
We note that we have not been able to fully determine which aspects of the training led to the appear- ance of artifacts in different models. The pretraining paradigm seems play role, OpenCIII and DeiT-Ill exhibit outliers both at size B and (Fig. 22). However, the model size and training length also play important parts, observed in Fig.
3 EXPERIMENTS
this section, we validate the proposed solution by training vision transformers with additional [reg] register tokens. We evaluate the effectiveness of our approach by quantitative and quali- tative analysis. We then ablate the number of registers used for training, check that they do not

---

Published conference paper at ICLR 2024
200
1500
300
norm
200
1000
100
500
DINOV2
DINOv2
OpenCLIP OpenCLIP
DeiT-ill
DefT-lllreg
Figure 7: Effect of register tokens the distribution of output norms DINOv2, OpenCLIP and DeiT-III. Using register tokens effectively removes norm outliers that were present previously.
cause performance regression, evaluate an unsupervised object discovery method atop our features and finally provide qualitative analysis of the patterns learnt by the registers.
3.1 TRAINING ALGORITHMS AND DATA
As the proposed solution is simple architectural change, can easily apply it any training pro- cedure. We try it on three different state-of-the of-tth-arrtttt training methods for supervised, text-supervised. and unsupervised learning, shortly described below.
DEIT-III (Touvron et al. 2022) is simple and robust supervised training recipe for classification with VIT' on ImageNet- 1k and ImageNet-22K, We choose this method an example label- supervised training as it is simple, uses the base ViT architecture, achieves strong classification results, and is easy to reproduce and modify with our improvements. We this method the ImageNet-22K dataset, using the ViT-B settings, as provided in the official repository
OpenCLIP [Tharco et al. 2021) is a strong training method for producing text-image aligned mod- els, following the original CLIP work. We chose this method as an example of text- sute--uperiied training because is open-source, uses the base ViT architecture, and easy reproduce and mod- ify with our improvements. Shutterstock that includes only licensed image and text data. We use VIT-B/16 image encoder, as proposed in the official repository
run the OpenCLIP method on a sottimage--alinee corpus based on
DINOv2 (Oquab et al. (023) is self-supervised method for learning visual features, following the DINO work. We apply our changes to this method as it is the main focus of our study. We run this method on ImageNet-22k with the VIT-L configuration. We the official repository
3.2 EVALUATION OF THE PROPOSED SOLUTION
As shown in Fig. get rid of the artifacts by training models with additional register tokens. the appendix, we provide additional qualitative results for more images in Fig. [9 quantitatively measure this effect, for each model, we probe the norm of features at the output of the model. We report these norms for all three algorithms with and without registers Fig. 7 We see that when training with registers, models do not exhibit prgeenorm tokens the output, which confirms the initial qualitative assessment.
order
Performance regression. In the previous section, we have shown that the proposed approach re- moves artifacts from local feature maps. In this experiment, we want check that the use of register tokens does not affect the representation quality of those features. We run linear probing on Im- ageNet classification, ADE20k Segmentation, and NYUd monocular depth estimation. We follow the experimental protocol outlined in Oquab et al. (202)). We summarize the performance of the models described in Sec. [3. with and without register tokens in Table 2a We see that when us- ing registers, models do not lose performance and sometimes even work better. For completeness, we also provided the zero-shot classification performance ImageNet for OpenCLIP (Table (26). which remains unchanged. Please note that the absolute performance of our OpenCIIP reproduction is lower due to the data source we used.
clip

---

Published conference paper at ICLR 2024
ImageNet Top-1
ADE20k
NYUd
mloU
rmse
DeiT-III
84.7
38.9
0.511
DeiT-III+reg
84.7
39.1
0.512
OpenCLII
78.2
26.6
0.702
ImageNet Top-1
OpenCLIP+reg
78.1
26.7
0.661
DINOv2
84.3
46.6
0.378
OpenCIIP OpenCLIP+reg 60.1
59.9
DINOv2+reg
84.8
47.9
0.366
(a) Linear evaluation frozen features.
(b) Zero-shot classification.
Table 2: Evaluation of downstream performance of the models that we trained, with and without registers. We consider linear probing of frozen features for all three models, and zero-shot evaluation for the OpenCLIP model. We that using register not only does not degrade performance, but even improves it by a slight margin some cases.
Input
0 [reg]
[reg]
[reg]
4 [reg]
8 [reg]
16 [reg]
ImageNet
Average of segmentation tasks
Average of depth tasks
84.8
2.85
66.8
2.82
66.6
84.6
2.79
66.4
84.5
66.2
2.76
84.4
66.0
2.73
12 number of [reg] tokens
4
12
16
number of [reg] tokens
number of [reg] tokens
Figure 8: Ablation of the the number of register tokens used with a DINOv2 model. (top): qualita- tive visualization of artifacts appearing function of number of registers. (bottom): performance on three tasks (ImageNet, ADE-20k and NYUd) as a function number registers used. While one register is sufficient to remove artefacts, using more leads to improved downstream performance.
we propose alleviating the feature maps' artifacts by adding register tokens. In this experiment, we study the influence of the number of such tokens local features and downstream performance. We train DINOv2 VIT-L/14 models with 0. 4, 8 or registers. Fig. we report the results this analysis. In Fig- ttop). we qualitatively study the attention maps and observe that the visible artifacts disappear when adding at least one register. We then examine in Fig. (8(ott)mm performance
Number register tokens. described in Sec. 2.2
downstream evaluation benchmarks, following the protocol from Oquab et al. (2023). There seems to be an optimal number registers for dense tasks, and adding one brings most of the benefit. This optimum is likely explained by the disappearance of artifacts, leading better local features. On ImageNet, however, performance improves when using more registers. In our experiments, we kept register tokens.
3.3 OBJECT DISCOVERY
Recent unsupervised object discovery methods rely the quality and smoothness of local feature maps (Siméoni et al.] 2021] Wang et al. 2023). By leveraging DINO Caron et al. (2021). these methods have significantly surpassed the previous state of the art. However, the algorithm leads poor performance when applied to modern backbones such as DINOv2 Oquab al. (223)
supervised ones Touvron et (2022). We posit that this can be alleviated by the method proposed in this work. We run LOST (Siméoni et al. (021) on features extracted from backbones trained using the algorithms described in Sec with and without registers. We object discovery PASCAL

---

Published conference paper at ICLR 2024
VOC 2007
VOC 2012
COCO 20k
DeiT-III
11.7
13.1
10.7
DeiT-IlI+reg
27.1
32.7
25.1
OpenCLIP
38.8
44.3
31.0
OpenCLIP+reg
37.1
42.0
27.9
DINOv2
35.3
40.2
26.9
DINOv2+reg
55.4
60.0
42.0
Table Unsupervised Object Discovery using LOST (Siméoni et al. 2021) on models with without registers. We evaluated three types of models trained with various amounts of supervision VOC 2007, 2012 and COCO. We measure performance using corlec. We observe that adding register tokens makes all models significantly more viable for usage object discovery.
Input
[CLS]
[reqp]
[rege] ]
[regs]
[reqı2]
Figure 9: Comparison of the attention maps of the [CLS] and register tokens. Register tokens sometimes attend to different parts of the feature map, similarly slot attention (Locatello et al. 2020 This behaviour was never required from the model, and emerged naturally from training.
VOC 2007 and 2012 and COCO 20k. We use values for DeiT and OpenCLIP, and for DINOv2, we keys. Because the output features may have different conditioning, manually add bias the gram matrix of features. The results of this experiment are presented in Table 3 For DINOv2 and DeIT-III, adding registers significantly improves the discovery performance. For OpenCLIP. the performance slighty worse with registers (see Sec. C for analysis). The performance of DINOy2 on VOC2007 still does not match that of DINO as reported by Siméoni al. (202T) (61.9 corloc). However, the model with registers gets improvement of 20.1 corloc (55.4 versus 35.3).
3.4 QUALITATIVE EVALUATION OF REGISTERS
this final experiment, qualitatively probe for the behavior of register tokens. We want verify they all exhibit similar attention patterns or whether differentiation automatically emerges. To this end, we plot the attention maps of the class and register tokens to patch tokens. The result of this visualization shown in Fig. D We see that registers do not have completely aligned behavior. Some selected registers exhibit interesting attention patterns, attending to the different objects in the scene. While nothing enforced this behavior, their activations had some natural diversity. We leave the study of the regularization of registers for future work.
4 RELATED WORK
Feature extraction with pretrained models. Using pretrained neural network models for extracting visual features has stood the test of time since the AlexNet (Krihevsky al. 2012) CNN model pretrained on ImageNet- Ik (Russakovsky et al. (2015). More recent models have upgraded the same setup with modern architectures, such as ResNets (used in, e.g., DETR, Carion et al., 2020) or even Vision Transformers. As Transformers
easily able to handle different modalities during training, thetthe-sheff backbones are now commonly trained on label supervision (e.g.. DeiT-III on ImageNet- 22k. Touvron et al. 2022) or text supervision (e.g., CLIP (Radford et al. (021)), providing strong visual foundation models, scaling well with model sizes, and enabling excellent performance on a variety tasks including detection (Carion et al. (2000) and segmentation (Zheng et al.) 2021: Kirillov et al. 2023). In this context, supervision relies on annotations in the form of labels or text well characterized, yet they drive
alignment; the dataset biases (Torralba & Efros 201

---

Published a conference paper at ICLR 2024
learning and shape the learned models. An alternative approach consists of not using supervision and letting the models learn from the data via a pretext task that is designed to require understanding the content of images (Doersch et al, 2015). This self-supervised learning paradigm was explored in
multiple methods using Vision Transformers: MAE (He al. 2022) trains a model at reconstructing pixel values of hidden areas of an image and then applies fine- tueittnnng to address a new task. With a different approach, the self-distillation family of methods (He et al., 2020; Caron et al. 2021 Zhou ctt al. 2022) showcase strong performance using frozen backbones, allowing for more robustness domain shifts for task-specific downstream models. In this work, we focused the analysis on self- supervised learning, and more specifically on the DINOv2 approach (Oquab et al.) 2023), which has shown to be particularly effective for learning local features. We showed that despite excellent benchmark scores, DINOv2 features exhibit undesirable artifacts and that correcting these artifacts in the learning process allows for further improvements in the benchmark performances. These phenomenon is even more surprising as DINOv2 builds upon DINO (Caron al. 2021). which does not show signs of artifacts. We then further showed that the correction techniques hold for supervised paradigms by testing on DeiT-III and OpenCI.IP.
Additional tokens in transformers. Extending the transformer sequence with special tokens popularized BERT (Devlin et al., 2019). However, most approaches add new tokens either to pro- vide the network with new information as for example [SEP] tokens in BERT, provide opportunity spend more computation on the input
with the tape tokens in AdaTape (Xue al. 2023).
or to gather information in these tokens, and use their output value as an output of the model: for classification, as [CLS] ] tokens in BERT and ViT (Dosovitskiy ct al. 202T); for generative learning, [MASK] in BERT and BEiT (Bao et al. 202T); for detection, as object queries in DETR (Carion) jtt al. 2020), detection tokens in YOLOS Fang et al. 2021), and ViDT (Song et al. 2021 ) or for accumulating information from possibly multiple modalities before decoding, as latent token arrays in Perceivers (Jaegle et al., [2021] 2022). Different to these works, the tokens we add the sequence add no information, and their output value is not used for any purpose. They are simply registers where the model can learn to store and retrieve information during the forward pass. The Mem- ory Transformer (Burtsev al., 2020). closer to work. presents simple approach to improve transformer models using memory tokens added to the token sequence, improving translation perfor- mance. In follow-up work, [Bulatov et al. (2022) address complex copy-repeat-revers tasks, [Sandler et al. (2022) extend this line to the vision domain for fine-tuning but observe that such tokens do not transfer well across tasks. In contrast, we do not perform fine-tuning and employ additional tokens during pretraining to improve the features obtained for all tasks downstream. More importantly, our study contributes the following new insight in Sec. 2 the mechanism implemented through memory tokens already appears naturally Vision Transformers; our study shows that such tokens allow us not to create but to isolate this existing behavior, and thus avoid collateral side-effects.
Attention maps of vision transformers. Visualising the attention map from [CLS] token to patch tokens was popularized in DINO (Caron et al., 2021). was shown there that the attention maps of DINO were clean of artifacts, as opposed to the attention maps of previous vision transformers. Other works have since reported interesting attention maps using various techniques: by modifying the optimisation procedure (Chen et al. 2022). by steering the attention scores towards useful image parts (Shi et al., 2023), by modifying the architecture of the transformer layers (Yu et al. 2024), or by introducing Icarnable pooling to produce the [CLS] ] token (Psomas et al. (223).
5 CONCLUSION
In this work, we exposed artifacts in the feature maps of DINOv2 models, and found this phe- nomenon to be present in multiple existing popular models. We have described a simple method to detect these artifacts by observing that they correspond to tokens with an outlier norm value at the output of the Transformer model. Studying their location, we have proposed interpretation that models naturally recycle tokens from low- low-informative areas and repurpose them into a different role for inference. Following this interpretation, we have proposed a simple fix. consisting of appending additional tokens to the input sequence that are not used as outputs, and have found that this entirely removes the artifacts, improving the performance in dense prediction and object discovery. More- over, we have shown that the proposed solution also removes the same artifacts present in supervised models such DeiT-III and OpenCLIP, confirming the generality of our solution.

---

Published conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank Hu Xu, Oriane Siméoni, Mido Assran and Armand Joulin for their insightful discussions and help during course of work. We thank Pyrrhus for posing fig X Julien Mairal was supported by ANR 31A MIAI@Grenoble Alpes AANR-19-P3IA-00 and by ERC grant number 101087696 (APHELEIA project).
REFERENCES
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021.
Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recurrent memory transformer. In NewrIPS, 2022.
Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprini arXiv:2006. 1527, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong- When vision transformers outperform resnets without pre-training strong data augmentations. ICLR, 2022.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth
words: Transformers for image recognition scale. ICLR, 2021.
Yuxin Fang, Bencheng Liao, Xinggang Wang. Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. In NewrlPS, 2021.
Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. 2021.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In ICML., 2021.
Andrew Jaegle. Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Andrew Brock, Evan Shelhamer, Olivier H'enaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver io: general ar- chitecture for structured inputs & outputs. In ICLR, 2022.
Alexander Kirillov, Eric Mintum, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.0064444 2023.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NewrIPS, 2012.
10

---

Published conference paper at ICLR 2024
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold. Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. In NewrIPS, 2020.
David G Lowe. Distinctive image features from scale-invariant keypoints. IJV, 2004.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0711111 2023.
Bill Psomas, Ioannis Kakogeorgiou, Konstantinos Karantzalos, and Yannis Avrithis. Keep it sim- pool: Who said supervised transformers suffer from attention deficit? In ICCV, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
Mark Sandler, Andrey Zhmoginov, Max Vladymyrow, and Andrew Jackson. Fine-tuning image transformers using learnable memory. In CVPR, 2022.
Baifeng Shi, Siyu Gai, Trevor Darrell, and Xin Wang. Toast: Transfer learning via attention steering, 2023.
Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self superviee transformers and no labels. In BMVC, 2021.
Hwanjun Song, Deqing Sun, Sanghyuk Chun, Varun Jampani, Dongyoon Han, Byeongho Heo, Wonjae Kim, and Ming-Hsuan Yang. Vidt: An efficient and effective fully transformer-based object detector. In ICLR, 2021.
Antonio Torralba and Alexci A. Efros. Unbiased look at dataset bias. In CVPR, 2011.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. ECCV, 2022.
Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In CVPR, 2023.
Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, and Yang You. Adaptive computation with clastic input sequence. In ICML, 2023.
Yaodong Yu, Tianzhe Chu, Shengbang Tong, Ziyang Wu, Druv Pai, Sam Buchanan, and Ma.
Emergence of segmentation with minimalistic white-bor box transformers. In CPAL, 2024.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang. Philip HS Torr. et al. Rethinking semantic segmentation from sequuence-o--sequeneee perspective with transformers. In CVPR, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022.

---

Published conference paper at ICLR 2024
0.4
0.2
0.2
Figure 10: Feature norms along locations: proportion of tokens with norm larger than the cutoff value at given location. Left: official DINOv2 model (no antialiasing). right: models (with antialiasing). some positions, more than 20% of tokens have high norm.
025
-0.73
-1.00
Figure 11: Propagating unit gradients through bicubic interpolation (16 16 x without
antialiasing. We observe striping pattern similar to the one of Fig. 10 (left).
A INTERPOLATION ARTIFACTS AND OUTLIER POSITION DISTRIBUTION
We plot in Figure 10 (left) the proportion of outlier tokens, characterized by norm larger than the cutoff value defined manually, following the distribution of norms shown in Fig. g (main text). We make two observations:
First, the distribution has a vertical-stripe strripd pattern. We investigate this phenomenon and notice that in the original DINOv2 implementation, during training the position embeddings are interpolated from 16 16 map into x 7 map, without antialiasing. Propagating unit gradients through such an interpolation function (bicubic resize) leads to the following gradients, shown Fig. In this work, when producing results with DINOv2 (especially for the results in Tables 2a we always apply antialiasing the interpolation operator, removing the striping pattern, which gives updated distribution outlier positions as shown in Fig. (rihht)
Second, the outliers tend to appear in areas closer to the border the feature map rather than in the center. Our interpretation is that the base model tends to recycle tokens in low-informative areas use registers: pictures produced by people tend to be object-centric, and in this case the border areas often correspond to background, which contains less information than the center.
B COMPLEXITY ANALYSIS
Since our proposed fix introduces new tokens, it also increases the number of learnable parameters and the FLOP count of the model. We show Fig. the relationship between number registers and increase model FLOP count and parameter count. We observe that adding registers induces a negligible change in number of parameters, and slight change in FLOP count. Still, for = 4 registers, the increase in FL.OPA stays below 2%.
12

---

Published conference paper at ICLR 2024
6%
#param
Increase
4%
FLOP
2%
0%
12
number [reg]
Figure 12: Increase in model parameter and FLOP count when adding different numbers of registers. Adding registers can increase model FLOP count up 6% for 16 registers. However, in the more common case of using registers, that we use in most of our experiments, this increase is below 2%. In all cases, the increase in model parameters is negligible.
DeiT-III
OpenCLIP
DINOv2
w/o REG
w/ REG
w/o REG
REG
w/o REG
w/ REG
LOST NORE
des prod. w soed
seed expansion
Illustration of the intermediate computations in the LOST algorithm for all models. Adding registers drastically improves the look of all intermediate steps for DeiT-IlI and DINOv2. The difference is less striking for the OpenCLIP model.
Figure 13:
C ANALYSIS OF LOST PERFORMANCE
The results presented in Sec. 3.3 show that adding registers allows us to obtain better object dis- covery performance with DINOv2 models. The conclusions for the two other models studied in qualitatively study the
this work could be more crisp. In order understand why this is so, impact of removing artifacts on the intermediate computations in the LOST algorithm. We show the intermediate outputs of LOST for all models given input image in Fig. 13
Adding registers improves the scores and the resulting seed expansion for DeiT-III and DINOv2. This observation coherent with the improved numbers reported in Table 3 For OpenCI.IP how- ever, the LOST algorithm seems robust to the type of outliers observed in the local features. Adding registers does remove artifacts (as clearly shown in Fig. 20) but does not have much impact the LOST score. also worth noting that OpenCLIP. with without registers, provides comparable performance to DINOv2 without registers and DeiT-III with registers. The qualitative assessment is coherent with the numbers reported in Table
surprising observation that despite the existence of high-norm patches in the output of Open- CLIP models without registers (as seen Fig. [7), the seed expansion score Fig. [3 looks smooth. In the LOST experiment with OpenCLIP models, we do not use the features directly, but the values from the computation of attention maps. In Fig. [4] show the seed expansion CLIP models with and without registers for keys, queries and values. We see that artifacts are clearly visible spots in the background for keys and queries, for the model without registers, As soon as registers are used, the LOST score focusing on the object, with smoother score for values. We qualitatively observe that for the OpenCLIP model, the value projection filters out the outliers even without registers. This means that the outliers appear to live in the null space of the value projection layer; the investigation for this phenomenon left for future work.
for Open-

---

Published conference paper at ICLR 2024
keys
queries
values
w/o REG
w/ REG
Figure 14: Illustration of the seed expansion score in LOST for an OpenCLIP model with and without registers for the three types of features considered: keys, queries, and values. The score is qualitatively improved across all features, with fewer artifacts appearing. Interestingly, the seed expansion map computed using values does not exhibit artifacts with without registers.
D BEHAVIOR OF MODELS TRAINED WITH REGISTERS
In order to better understand the phenomenon at hand, we examine the question of to what extent did the register tokens "replace" the high-norm tokens and took the same role.
D.I NORMS
200
200
150
150
nom
morm
100
100
50
50
CLS
patch
CLS
reg_0 0
reg 1
reg_2
reg_3
patch
type
type
(a) DINOv2 register
(b) DINOv2 4 registers
Figure 15: Distribution of token norms for DINOv2 model without (left) and with (right) 4 regis- ters. Introducing registers entirely negates the high- norm outliers among the patch tokens.
In Fig. 1.5 we compare the distribution of token norms for a model with or without registers. This figure similar Fig. but with finer granularity, we also plot the norm distribution of individual register tokens and [CLS] tokens. We observe the following: with registers, the norms of patch tokens do not contain outliers anymore, and the high-norm tokens are entirely contained the set of registers. As result, we conclude that the behavior leading to high-norm outliers in the model effectively absorbed in the registers.
An additional interesting observation is that the norms of the registers appear to be quantized, com- pared to the previous outliers; we leave the investigation of this phenomenon for future work.
D.2 INFORMATION HELD BY TOKENS
We report on table 4 the linear probing performance of models trained with and without registers, when using different tokens as representations. We evaluate on the aircrafts dataset, showed clear conclusions in the similar table [ We observe that adding a register does not significantly modify the scores obtained with the [CLS] or patch tokens. However, the outlier patches are removed, and their behavior is transferred to the newly added register.
14

---

Published conference paper at ICLR 2024
top-1 accuracy
#registers
[CLS]
normal patch outlier patch
register
0
84.6
15.5
73.3
N/A
85.2
14.5
N/A
71.1
Table 4:
Linear probing of models with and without registers on the Aircraft dataset, using various tokens as representation. We ohserve that the behavior of the outlier tokens, aggregating global information, is absorbed into the register.
We further conduct an evaluation of the local information contained in the patch tokens of model trained with and without registers (table We observe that the non-outliers patches, both cases, hold similar local information, confirming that the registers only remove the outlier behavior, with- significantly modifying the information held by the other patches.
position prediction top-1 acc
reconstruction error 4
#registers patches considered
non-outliers non-outliers (ic all)
66.3
15.9
65.8
16.0
Table 5: Linear probing for local information on the patch tokens of models trained without or with registers. We only consider patches considered "normal", i.c. not the high-norm outliers. Wc observe that adding registers does not significantly modify the scores of these patches.
D.3 POSITIONAL FOCUS
(a) [CLS]
(b)
(c)
(d) rega
(c)
(f) putch
Figure 16: Average attention map of registers and [CLS] token. There variability observed, with register of this model focusing more on border areas. We also include the average attention map of patch for comparison. The patch has much more focused average attention.
In Fig. 16 display the positional focus for the class token and the 4 registers of a DINOv2+reg
model. We produce these plots by running the model random subset of ImageNet-22k, and av- eraging the attention maps for the corresponding tokens at the last layer. We note that ImageNet-22 contains mostly object-centric images rather than scenes, which explains why the average attention maps correspond to centered blobs.
make several observations. First, the attention maps registers can different of each other; for example, register tends to focus on border areas, while the other registers tend to focus on more centered arcas. Register 2 tends to focus slightly more the upper areas of images that others. This consistent with Fig. D where we show registers focusing on different large areas of the image, suggesting some level of specialization.
Second. by comparing the register maps to the [CLS] token map and to a patch token map, we observe that registers produce maps with large support area, very similarly the [CLS] token, and very different of typical patch token which is more localized. As the [CLS] token is known to carry global information (as proven by the linear probing classification performance): this suggests that registers also carry global information.
15

---

Published conference paper at ICLR 2024
E MASKED AUTOENCODERS
Masked Autoencoding (He et al. 2022) is another common way pretraining self-supervised mod- els. We observe in Fig. T7 that there are no artifacts in the maps produced by MAE: our hypothesis that the absence of artifacts is due the training procedure using only local loss the patch tokens, rather than an objective involving global aggregation information. However, we also note that the performance of MAE models is very low for self- representation learning (75% linear probing performance ImageNet classification for ViT--Lrgee,, preventing it from being used as is, and making fine-tuning a requirement.
Figure 17: First three principal components of the output feature map of ViT-Large Masked Au- toencoder.
F BEHAVTOR PER ATTENTION HEAD
this section, we investigate whether artifacts appear only on attention maps for specific heads of the last vision transformer block, or for all of them. We show in Fig. [8] the input image along with the attention maps for different heads. We observe that the artifacts appear for all atten- tion heads, despite heads focusing on different areas of the object. We still observe that some heads focus more on artifacts than others.
G VARIANCE ON TOKEN INFORMATION PROBING
The results presented in table obtained by taking random patch token, either normal outlier. However, the choice of this token adds significant source of variance the evaluation. For thoroughness, we report in table O the standard deviation of the scores obtained relative to this choice.
16

---

Published conference paper at ICLR 2024
input
Figure 18: Attention maps of the [CLS] token to the patch tokens, shown here separately per attention head. We produce these maps with DINOv2-L model trained without registers.
dataset token
Aire.
CF10
CF100
CUB
Cal101
Cars
DTD
normal
17.1 19.
97.1
81.3s0a
18.6
73.2
10.8.n.
63. 63.1
outlier
99.3.a0
93.7
84.9.21
97.6-a7
85.2
84.9
[CLS]
87.3
99.4
94.5
91.3
96.9
91.5
85.2
dataset token
Flow.
Food
IN1k
P205
Pets
SUN
VOC
normal
59.5
74.2
65.8
53.1
47.8
37.7
70.8
outlier
99.6.no
93.5
69.0san
55.1.10
94.1s02
78.5
89.7.m.
[CLS]
99.7
94.7
86.0
66.4
96.9
78.6
89.1
Table 6: Image classification via linear probing on normal and outlier patch tokens. As we select the patch tokens randomly among the set of eligible tokens, this adds source of variability. We report the standard deviation this variability grey along with scores. This table a detailed view of table I
H QUALITATIVE RESULTS
We trained three popular models: DeiT-III. OpenCLIP, DINOv2 with and without the introduction of register tokens. We observe in Fig. 19 the attention maps in the last layer of the Vision Trans- former, for all three cases. We see that our approach provides much cleaner attention maps, with considerably fewer artifacts, explaining the improvement on the downstream object discovery task mentioned in Sec. 3.3 The feature maps also visibly improved, as shown in Fig. 20 Finally,
17

---

Published conference paper at ICLR 2024
we also show the norm of the patch tokens Fig. 21 and confirm that in all three models, artifact patches correspond to norm outliers.
18

---

Published conference paper at ICLR 2024
Without registers OpenCLIP
With registers OpenCLIP
Input
DeiT-III
DINOv2
DeiT-IlI
DINOv2
Figure 19: Attention maps of models trained without and with registers on various images.
19

---

Published conference paper at ICLR 2024
Without registers OpenCIIIP
With registers OpenCIIIP
Input
DeñT-IIII
DINOv2
DeñT-III
DINOv2
DO NHA WAKES Y
HAPPY
Figure 20: First principal component feature maps output by models trained without with registers on various images. The components are whitened and the colormap covers the range
[=30, +30).
20

---

Published conference paper at ICLR 2024
Without registers OpenCLIP
With registers OpenCLIP
Input
DeiT-III
DINOv2
DeiT-III
DINOv2
DO WHA
HAPPY
Figure 21: Maps of token norms for models trained without and with registers various images. The norm outliers are very visible for models trained without registers.


Reviews:
[{"review_id": "gJU95oFfG2", "reviewer": "Reviewer_eoPK", "summary": "The paper discusses the discovery of artifacts in the feature maps of Vision Transformer (ViT) networks, both supervised and self-supervised. These artifacts appear as high-norm tokens during inference, typically in less informative background areas of images, and are utilized for internal computations by the network. To address this, the authors introduce a novel and straightforward method involving the addition of extra tokens to the ViT's input sequence. This technique effectively resolves the artifact issue for both types of models. It not only sets new performance benchmarks for self-supervised visual models on dense prediction tasks but also enhances object discovery with larger models. Crucially, the approach results in smoother feature and attention maps that benefit subsequent visual processing tasks.", "questions": "In table 3, OpenCLIP+reg is not better than OpenCLIP which is contrary to other two models that have significant improvement. Any further explaination could be helpful since it doesn’t support the claim that for all models, adding registers would improves the results.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies an interesting phenomenon observed in the popular transformer models (DINO). By removing this artifact, the authors demonstrates the improved models have clear attention maps that could be used for downstream analysis such as object localization. \n- The step-by-step investigation is solid and compelling. \n- The method of providing a junkyard to remove the artifact is novel and effective. \n- The experiments are convincing and comprehensive.", "weaknesses": "- The norm shows significant reduction for OpenCLIP in Figure 7, yet in Table 3, it doesn’t show significant improvement for object localization which is the main benefit of using register. Further explaination / exploration the reason behind it should be helpful for wide adoptation.  \n- Minor: it should be OpenCLIP instead of CLIP in Figure7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fhACYgf6aO", "reviewer": "Reviewer_GGy7", "summary": "The paper identifies the problem of artifactual areas of feature maps in vision transformers. On further analysis, these artifacts correspond to high norm tokens in the ViT coming from background regions in the image, tend to hold more global information and lack spatial information. This leads to the conclusion that tokens from these low-information regions are being repurposed by the model to hold global information for internal computations. Moreover, this issue afflicts most ViTs with DINO v1 being the exception. This problem was previously discussed in Memory transformer (Burtsev et al.) in the context of NLP datasets. Following Memory transformer's recommendation, the paper adds new tokens called registers at the to remediate this issue and show that the register-trained ViTs have better spatial feature maps, thus better downstream performance for object discovery tasks. Various ablations and experiments are also shown to shed light on the behavior of registers and why this problem occurs with DINOv2 models in the first place.", "questions": "1. I'm curious if other simple solutions like penalizing these high norms could work as well and if yes, would they be preferable? Would love to hear from the authors on other ways to address this issue.\n2. Any reason why adding more registers hurts performance for NYU depth dataset?\n3. What happens to downstream performance when registers are added to DINO models which do not seem to need it? Does the nature of what registers learn differ from the case of DINOv2?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The paper identifies an important problem of heatmaps lacking spatial resolution and accuracy in DINOv2 and other ViTs which leads to suboptimal downstream performance on object discovery and localization tasks. The fact that this is not a problem for DINOv1 is pretty surprising and the experiments done on the changes in token norm across model size will be a useful start to understand this better. The discovery of these high-norm tokens, experiments using the linear models and classifiers to characterize what these tokens contain, and then forming the hypothesis of how these tokens are getting repurposed for holding global information, all presents a coherent story of these misused outlier tokens.\n2. The solution of adding new tokens (memory or registers) is not a new one, but is shown to be very effective in removing these tokens with high norms as well as bringing back spatial interpretability into the feature maps. Furthermore, the downstream performance on image-level tasks stain consistent with ViTs without registers while the performance on object discovery tasks goes up in DINOv2 and DeiT-III after addition of registers. These results indicate that adding these new tokens does resolve the spatial issue with these ViTs. A huge plus of this approach is the simplicity of it.", "weaknesses": "1. The removal of these artifacts does come at the cost of new tokens, hence additional compute. The paper reports a 2-6% increase when adding 4-16 new register tokens.\n2. One very interesting observation was how the different register tokens end up focussing on the different areas of interest on the object. If there are spatially discrete areas of focus for the registers, does this undermine the argument that we need them for storing global information which was earlier being done using redundant patches?\n3. Not a weakness, but would be nice to see some norm-related metrics and/or visualizations for the outlier tokens across different heads. Do all the heads from these tokens end up getting these tokens repurposed? What does that variance across heads look like?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zu9LsoB33J", "reviewer": "Reviewer_KSLu", "summary": "This paper identifies and characterizes artifacts in the feature maps of vision transformer (ViT) models trained with supervision or self-supervision. In particular, the authors observe high-norm \"outlier\" tokens with high redundancy in the output features of several ViT models, and show that they hold less local patch information but more global image information compared to normal tokens. This suggests the model is repurposing redundant patches to store global information. They propose appending dedicated \"register\" tokens to the input sequence, which removes the artifacts and improves performance on downstream dense prediction tasks.", "questions": "See weaknesses above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The investigation is quite original; the use of memory/registers in transformers is not necessarily a new idea, but motivating them through removing redundancy and reducing attention artifacts is both novel and interesting.\n\n2. Experiments and analysis are mostly convincing (see questions below).\n\n3. I enjoyed the narrative exposition: the problem setting is clear, the motivation for registers is clear, and their utility is well-demonstrated via experiments.", "weaknesses": "1. While adding additional token (registers) seems like a simple and efficacious approach, I'm wondering if it's the only possible solution for reducing patch level redundancy. Did the authors observe similar effects across other self-supervised models, like MAE, where nominally the patch-level reconstruction should also alleviate representational redundancy?\n\n2. In demonstrating that the artifacts hold global information, the authors \"choose a single token at random, either high-norm or normal,\" and then \"train a logistic regression classifier to predict the image class from this representation, and measure the accuracy.\" Why choose this token at random? Why not use all the high-norm and normal tokens, or some projected and pooled version over all of them in order to regress to the class? In experiments we have conducted, this almost always outperforms using single tokens (cls or otherwise), and it may be the case that the conclusion that the high-norm tokens outperform the \"normal\" tokens is not so clear when this is done.\n\n2. There seems to be a conflict between the fact that \"high-norm tokens appear on patches that are very similar to their neighbors,\"  \"often appear[ing] in uniform, background areas,\" and the fact that performance on ImageNet classification improves almost monotonically with more registers, but not dense tasks like segmentation or depth estimation. In particular, I would expect that if reappropriating the \"redundant\" local patches helps on object-centric classification, it should help much more substantially on tasks that are even more reliant on good (non-redundant?) local information (i.e. segmentation or depth estimation).  Can the authors comment on this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ojGA6WMjKm", "reviewer": "Reviewer_mTMB", "summary": "This paper identifies an interesting phenomena in large scale transformers where some redundant tokens are repurposed for internal computation. The paper shows how the feature normal can be used to identify such tokens and how such tokens appear to capture global, rather than local, information compared to other tokens. Furthermore, such tokens make the attention maps less interpretable. The paper proposes to augment ViTs with register tokens which similar to CLS tokens are separate from the image patch tokens, but are not used used directly in any loss computations unlike CLS tokens. The proposed augmentation removes the normal outliers, results in small improvements on standard evaluation tasks, while improving the unsupervised object discovery performance of most methods.\n\n**Update (11/20):** I have updated my scores after reading the points made by other reviewers. Specifically, the point regarding the impact of the tokens on dense vs. image-level tasks, specifically weakness 3 raised by reviewer KSLu. It would be great if the authors could engage with the concerns raised by the reviewers.", "questions": "- Do the registers inherit the behavior exhbited by the outlier tokens? Specifically, are they good predictors of global image information as shown in Table 1? \n- Could you clarify on the discrepancy in OpenCLIP performance in Table 3 vs Sec 3.3? I wasn't sure if it's a missed negative result or a typo in the table.\n- Do the image tokens revert back to being more local in nature with the addition of tokens? How do they perform on the tasks exhibited in Table 1 and Figure 5?\n- How does the norm of the CLS token compare to the outliers before and after the addition of register tokens? It would be interesting to see if it matches the outlier tokens across conditions as shown in Figure 4, although simply reporting it on the final model would provide some insight into the internal mechanisms of ViTs. \n- Could you please comment on why you think LOST with DINO still performs better that with DINOv2? Is it the data (ImageNet vs LVD), newer training objective, some other factor? \n- It seems suprising that DINOv2 exhibits this behavior while using a dense mask-image-modeling objective. I was curious if you had any thoughts on why the masked image objective did not discourage such behavior despite requiring the patch features to retain the information that you show is lost in Fig 5b\n- I found the first line in page 9 a bit confusing. As I understand it, Torralba and Efros (2011) were arguing that datasets themselves were biased, not that the specific labels were. Such concerns still apply whether or not the data is labels or the training paradigm used for training as the bias arises from the data source and sampling process. While the samping process would be affected by the target labels, one can still get a biased dataset based solely on the data source (eg, instagram vs. inaturalist). Could you please elaborate on your statement and what you meant? \n- (Suggestion) The paper suggests that this outlier token is exhibited by larger models in Fig 4/Sec 2.2, yet the base model of CLIP/DEIT is used in Sec 3.1. While Fig 7 still shows that the base models of those models exhibit outlier tokens, it would be nice for the authors to add some commentary on this at the end of sec 2.2.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The analysis of the outlier tokens is very nice and thorough. I found the graphs and explanations very insightful, and the experiments very comprehensive (especially the experiments in Tab 1 and Fig 5). \n- The proposed inclusion of a register token is very simple and elegant and provides more interpretable attention masks.\n- I appreciated the limitations statement at the end of Sec 2.2. \n- The paper was very easy to follow and the visualizations were helpful to provide the reader intuition for what is going on.", "weaknesses": "- While the paper did a great job at analyzing the behavior of outlier tokens in previous models in Sec 2.1, the paper does not have experiments showing that such behavior is eliminated by adding the register tokens. It would have been interesting to see if the behaviors ascribed to normal/outlier tokens in Fig 5 and table 1 are now transferred to image/register tokens in the proposed model. \n- The discussion around the performance of models on unsupervised object discovery is fairly limited and does not match the resuls. \n    - The paper is strongly motivated by the difference in attention maps compared to DINO and the limited performance of DINOv2 on LOST. While the gains of DINOv2+reg are impressive, it is still very surprising that it doesn't match DINO. It would be great if there was more discussions or some qualitative examples of that to explain why. \n    - The paper states that \"for all models on all datasets, adding registers for training improves the unsupervised object discovery performance.\" However, the results indicate that registers harm the performance on OpenCLIP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)