Kati8/ICLR_2023_2025 huggingface dataset:
authors: Timothée Darcet
authors: Maxime Oquab
authors: Julien Mairal
authors: Piotr Bojanowski
reviews: 
Comments: This paper identifies an interesting phenomena in large scale transformers where some redundant tokens are repurposed for internal computation. The paper shows how the feature normal can be used to identify such tokens and how such tokens appear to capture global, rather than local, information compared to other tokens. Furthermore, such tokens make the attention maps less interpretable. The paper proposes to augment ViTs with register tokens which similar to CLS tokens are separate from the image patch tokens, but are not used used directly in any loss computations unlike CLS tokens. The proposed augmentation removes the normal outliers, results in small improvements on standard evaluation tasks, while improving the unsupervised object discovery performance of most methods.

**Update (11/20):** I have updated my scores after reading the points made by other reviewers. Specifically, the point regarding the impact of the tokens on dense vs. image-level tasks, specifically weakness 3 raised by reviewer KSLu. It would be great if the authors could engage with the concerns raised by the reviewers.
Strengths: - The analysis of the outlier tokens is very nice and thorough. I found the graphs and explanations very insightful, and the experiments very comprehensive (especially the experiments in Tab 1 and Fig 5). 
- The proposed inclusion of a register token is very simple and elegant and provides more interpretable attention masks.
- I appreciated the limitations statement at the end of Sec 2.2. 
- The paper was very easy to follow and the visualizations were helpful to provide the reader intuition for what is going on.
Weaknesses: - While the paper did a great job at analyzing the behavior of outlier tokens in previous models in Sec 2.1, the paper does not have experiments showing that such behavior is eliminated by adding the register tokens. It would have been interesting to see if the behaviors ascribed to normal/outlier tokens in Fig 5 and table 1 are now transferred to image/register tokens in the proposed model. 
- The discussion around the performance of models on unsupervised object discovery is fairly limited and does not match the resuls. 
    - The paper is strongly motivated by the difference in attention maps compared to DINO and the limited performance of DINOv2 on LOST. While the gains of DINOv2+reg are impressive, it is still very surprising that it doesn't match DINO. It would be great if there was more discussions or some qualitative examples of that to explain why. 
    - The paper states that "for all models on all datasets, adding registers for training improves the unsupervised object discovery performance." However, the results indicate that registers harm the performance on OpenCLIP.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 4 excellent
reviews: 
Comments: This paper identifies and characterizes artifacts in the feature maps of vision transformer (ViT) models trained with supervision or self-supervision. In particular, the authors observe high-norm "outlier" tokens with high redundancy in the output features of several ViT models, and show that they hold less local patch information but more global image information compared to normal tokens. This suggests the model is repurposing redundant patches to store global information. They propose appending dedicated "register" tokens to the input sequence, which removes the artifacts and improves performance on downstream dense prediction tasks.
Strengths: 1. The investigation is quite original; the use of memory/registers in transformers is not necessarily a new idea, but motivating them through removing redundancy and reducing attention artifacts is both novel and interesting.

2. Experiments and analysis are mostly convincing (see questions below).

3. I enjoyed the narrative exposition: the problem setting is clear, the motivation for registers is clear, and their utility is well-demonstrated via experiments.
Weaknesses: 1. While adding additional token (registers) seems like a simple and efficacious approach, I'm wondering if it's the only possible solution for reducing patch level redundancy. Did the authors observe similar effects across other self-supervised models, like MAE, where nominally the patch-level reconstruction should also alleviate representational redundancy?

2. In demonstrating that the artifacts hold global information, the authors "choose a single token at random, either high-norm or normal," and then "train a logistic regression classifier to predict the image class from this representation, and measure the accuracy." Why choose this token at random? Why not use all the high-norm and normal tokens, or some projected and pooled version over all of them in order to regress to the class? In experiments we have conducted, this almost always outperforms using single tokens (cls or otherwise), and it may be the case that the conclusion that the high-norm tokens outperform the "normal" tokens is not so clear when this is done.

2. There seems to be a conflict between the fact that "high-norm tokens appear on patches that are very similar to their neighbors,"  "often appear[ing] in uniform, background areas," and the fact that performance on ImageNet classification improves almost monotonically with more registers, but not dense tasks like segmentation or depth estimation. In particular, I would expect that if reappropriating the "redundant" local patches helps on object-centric classification, it should help much more substantially on tasks that are even more reliant on good (non-redundant?) local information (i.e. segmentation or depth estimation).  Can the authors comment on this?
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 4 excellent
reviews: 
Comments: The paper discusses the discovery of artifacts in the feature maps of Vision Transformer (ViT) networks, both supervised and self-supervised. These artifacts appear as high-norm tokens during inference, typically in less informative background areas of images, and are utilized for internal computations by the network. To address this, the authors introduce a novel and straightforward method involving the addition of extra tokens to the ViT's input sequence. This technique effectively resolves the artifact issue for both types of models. It not only sets new performance benchmarks for self-supervised visual models on dense prediction tasks but also enhances object discovery with larger models. Crucially, the approach results in smoother feature and attention maps that benefit subsequent visual processing tasks.
Strengths: - The paper identifies an interesting phenomenon observed in the popular transformer models (DINO). By removing this artifact, the authors demonstrates the improved models have clear attention maps that could be used for downstream analysis such as object localization. 
- The step-by-step investigation is solid and compelling. 
- The method of providing a junkyard to remove the artifact is novel and effective. 
- The experiments are convincing and comprehensive.
Weaknesses: - The norm shows significant reduction for OpenCLIP in Figure 7, yet in Table 3, it doesn’t show significant improvement for object localization which is the main benefit of using register. Further explaination / exploration the reason behind it should be helpful for wide adoptation.  
- Minor: it should be OpenCLIP instead of CLIP in Figure7.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good