reviews: 
Comments: The paper identifies the problem of artifactual areas of feature maps in vision transformers. On further analysis, these artifacts correspond to high norm tokens in the ViT coming from background regions in the image, tend to hold more global information and lack spatial information. This leads to the conclusion that tokens from these low-information regions are being repurposed by the model to hold global information for internal computations. Moreover, this issue afflicts most ViTs with DINO v1 being the exception. This problem was previously discussed in Memory transformer (Burtsev et al.) in the context of NLP datasets. Following Memory transformer's recommendation, the paper adds new tokens called registers at the to remediate this issue and show that the register-trained ViTs have better spatial feature maps, thus better downstream performance for object discovery tasks. Various ablations and experiments are also shown to shed light on the behavior of registers and why this problem occurs with DINOv2 models in the first place.
Strengths: 1. The paper identifies an important problem of heatmaps lacking spatial resolution and accuracy in DINOv2 and other ViTs which leads to suboptimal downstream performance on object discovery and localization tasks. The fact that this is not a problem for DINOv1 is pretty surprising and the experiments done on the changes in token norm across model size will be a useful start to understand this better. The discovery of these high-norm tokens, experiments using the linear models and classifiers to characterize what these tokens contain, and then forming the hypothesis of how these tokens are getting repurposed for holding global information, all presents a coherent story of these misused outlier tokens.
2. The solution of adding new tokens (memory or registers) is not a new one, but is shown to be very effective in removing these tokens with high norms as well as bringing back spatial interpretability into the feature maps. Furthermore, the downstream performance on image-level tasks stain consistent with ViTs without registers while the performance on object discovery tasks goes up in DINOv2 and DeiT-III after addition of registers. These results indicate that adding these new tokens does resolve the spatial issue with these ViTs. A huge plus of this approach is the simplicity of it.
Weaknesses: 1. The removal of these artifacts does come at the cost of new tokens, hence additional compute. The paper reports a 2-6% increase when adding 4-16 new register tokens.
2. One very interesting observation was how the different register tokens end up focussing on the different areas of interest on the object. If there are spatially discrete areas of focus for the registers, does this undermine the argument that we need them for storing global information which was earlier being done using redundant patches?
3. Not a weakness, but would be nice to see some norm-related metrics and/or visualizations for the outlier tokens across different heads. Do all the heads from these tokens end up getting these tokens repurposed? What does that variance across heads look like?
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 4 excellent