Title: Vision Transformers Don't Need Trained Registers
Venue: NeurIPS 2025
Accepted: yes
Keywords: interpretability

Vision Transformer

neurons

attention sink

CLIP

DINO
Abstract: We investigate the mechanism underlying a previously identified phenomenon in Vision Transformers -- the emergence of high-norm tokens that lead to noisy attention maps (Darcet et al., 2024). We observe that in multiple models (e.g., CLIP, DINOv2), a sparse set of neurons is responsible for concentrating high-norm activations on outlier tokens, leading to irregular attention patterns and degrading downstream visual processing. While the existing solution for removing these outliers involves retraining models from scratch with additional learned $\textit{register tokens}$, we use our findings to create a training-free approach to mitigate these artifacts. By shifting the high-norm activations from our discovered $\textit{register neurons}$ into an additional untrained token, we can mimic the effect of register tokens on a model already trained without registers. We demonstrate that our method produces cleaner attention and feature maps, enhances performance over base models across multiple downstream visual tasks, and achieves results comparable to models explicitly trained with register tokens. We then extend test-time registers to off-the-shelf vision-language models, yielding cleaner attention-based, text-to-image attribution. Finally, we outline a simple mathematical model that reflects the observed behavior of register neurons and high norm tokens. Our results suggest that test-time registers effectively take on the role of register tokens at test-time, offering a training-free solution for any pre-trained model released without them.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a method to move high-norm activations of \"outlier tokens\" that occur in Vision Transformers (ViTs) to register tokens. In contrast to previous methods dealing with this phenomenon, the authors' method does not require retraining the ViT. First, the authors identify the core mechanisms causing the occurrence of outlier tokens. Second, they use their findings to demonstrate that the mechanism can be employed to relocate outlier tokens to arbitrary positions within the image. Third, they utilize the mechanism to relocate outlier token activations to register tokens outside the image. The authors demonstrate that applying their method improves the model's performance on a range of downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is well-written and leads the reader through the discovery process of the mechanism behind outlier tokens and how to move them to register tokens at test-time.\n- The advancement of moving outlier tokens to register without re-training the model is meaningful.\n- The authors conduct apt experiments.\n\n**Weaknesses**\n\n- The method performs on average slightly worse than adding registers by re-training the model.\n- The pracitical relevance of the typographic attack mitigation using outlier tokens seems limited, because masking out the corresponding tokens in the input space works about equally well as moving high-norm tokens to the respective locations.\n\nQUESTIONS:\n- How does the number of test-time register tokens influence the performance on the downstream task(s) (There is some information regarding multiple registers in Appendix A.1, but I didn't find metrics on downstream tasks for this ablation)?\n- How does the experiment on typographic attacks support the author's claims?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the issue of high-norm outlier tokens in Vision Transformers (ViTs), which can lead to noisy attention maps. Prior solutions required retraining models with additional learned \"register tokens.\" In contrast, the proposed method, test-time registers, offers a training-free alternative. It identifies a sparse set of \"register neurons\" responsible for generating high-norm activations and shifts these activations into additional, untrained tokens outside the image area. This redirection of outlier information results in cleaner attention maps, improved performance on downstream visual tasks, and increased robustness to typographic attacks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The proposed method is simple yet effective.\n2. The paper is clearly written and easy to follow.\n3. The effectiveness of the proposed method is analyzed across various downstream visual tasks (such as classification, dense prediction, zero-shot segmentation, and unsupervised object discovery).\n4. The demonstrated use case showing how this approach can mitigate typographic attacks is also interesting and represents the usefulness of the proposed approach.\n\n**Weaknesses**\n\n1. Darcet et al. (2024) have shown that these outlier tokens are present in larger model sizes (Large, Huge, and Giant). In the experimental results (e.g. Table 2), the effectiveness of the proposed method is not analyzed across different model sizes. \n2. In Table 5, the reported results for models with trained registers differ from those in Table 3 of  (Darcet et al. (2024)). Could you clarify the reason for this discrepancy? If the models used in their experiments differ from those in yours, why did you choose not to use the same models for a direct comparison?\n3. In Table 4, I think \"CLIP\" should be replaced with \"OpenCLIP\" to avoid confusion.\n\nQUESTIONS:\n>  As no checkpoints with trained registers are available for OpenCLIP, we only compare our approach to the standard models.\n\n1. Could you please clarify why you have not directly compared with the numbers reported in (Darcet et al. (2024))?\n\n2. Please refer to the first and second points in **Weaknesses**.\n\nLIMITATIONS:\nSome of the limitations are discussed in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
This work investigates the emergence of high-norm tokens caused by a sparse set of neurons focusing strong activations on outlier tokens. This phenomenon, observed in models like CLIP and DINOv2, degrades downstream performance. While prior solutions require retraining with learned register tokens to suppress these effects, the authors propose a training-free alternative: redirecting high-norm activations into a new, untrained token during inference. This method cleans attention maps, improves task performance, and matches the effectiveness of retrained models with register tokens, offering a practical fix for existing pre-trained models.

STRENGTHS AND WEAKNESSES:
Strengths: a thorough analysis of the mechanism behind the creation of high-norm tokens in ViT, proposing a training-free method to solve the problem, showing that it is comparable in quality to alternatives based on retraining. 

Weaknesses: it would have been interesting to have more theory on "register neurons" either from simple models, or simple arguments.

QUESTIONS:
Can you develop a theoretical framework based on simple models or simple arguments for register neurons?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work follows previous works that introduced register tokens when training vision transformers to prevent the appearance of high norm tokens artifacts during inference. While register tokens are effective, they need to be trained alongside a model, making them impractical to include for already trained models.\nIn this work, the authors find that a small set of specific neurons can be identified that consistently cause these high activations over various images. Their activations on patches with artifacts can be shifted to a register added at test-time to fix the artifacts while preserving their global information. This approach allows to address the same issue as trained register tokens, but without the need for any training. The results show that these test-time registers maintain or improve performance on various tasks. The improvements are particularly strong on tasks that rely on having smooth feature maps, such as unsupervised object discovery, although it does not reach the level of trained register tokens.\n\nSTRENGTHS AND WEAKNESSES:\n(+) = strength, (-) = weakness\n\n**Quality**:\n1. (+) The paper is well motivated and proposes a new solution to a real observed known issue.\n2. (+) The study of the mechanisms leading to the observed artifacts produces multiple valuable insights, leading to the design of a good solution.\n3. (+) The performance of the proposed test-time register token that gathers activations from register neurons is good, improving or maintaining performance of models. The value of this approach is strongly validated by all the results.\n4. (+) The design choices (number of register tokens, initialisation) are motivated with experiments in the appendix.\n5. (+) Limitations are clearly acknowledged and discussed. \n6. (-) The results do not seem to be from multiple runs for statistical significance (based on the Checklist answer 7), even though linear probes are trained. The consistency would be valuable to validate considering how close some of the results are (e.g., Table 2). \n7. (-) The approach requires setting some hyperparameters, and it is unclear how involved this might be for new models.\n8. (-) The paper does not discuss why we would expect that shifting activations to a new token would still be valuable to the model without any re-training. While the experiments validate that the test-time register keeps global information, they do not validate that it still has a role in the model's predictions, and that improvements in the results come from more than disabling the artifacts from register neurons.\n\n**Clarity**: \n1. (+) The paper is well structured. I find the flow to support a good understanding of the work, with a strong study of the mechanisms before introducing the proposed solution and then the results.\n2. (-) I find that the full practical approach would benefit from being described in more detail and in one place, either through an algorithm, figure, or mathematical formulation. Algorithm 1 helps make the register neuron identification process very clear, but I believe that how we get to a register token at test time is not as clear due to the information to understand the whole process being split over multiple sections. I believe that it would be helpful to have a clear location in the paper that would make this whole process very clear, so that the reader clearly understands that register neurons are fixed for new images, but outlier patch locations have to be identified, etc.\n\n**Significance**:\n1. (+) This paper has strong significance. It proposes a method that can potentially improve any off-the-shelf vision transformers without retraining, and presents valuable new insights about the functioning of the models studied.\n\n**Originality**: \n1. (+) The proposed solutions are grounded in original findings about the behaviour of specific neurons and lead to a novel algorithm to manage them, making this work original.\n\nQUESTIONS:\n1. Can the authors define earlier what they mean by \"neurons\"? Understanding what a neuron consists of is key to this work, but a brief technical definition only appears in the related works (line 79). While this definition is complete, I think it is really critical to ensure that the reader grasps and visualises exactly what is meant by \"neuron\" as early as possible, potentially using a more extensive explanation. One potential way to do so would be to visualise attention maps stacked from different neurons.\n2. In the checklist, the authors claim that the full algorithm is described in Algorithm 1. However, this only details the process to identify register neurons, not the steps to move them to test-time registers. Can the authors please include a full description of the algorithm that includes register token creation at test time (see my comment on clarity)?\n3. Could the authors please clarify how Figure 3 (right) highlights that the small subset of <10 neurons are consistent across images? The fact that the neurons are consistent across images is very valuable, but I am currently failing to understand how that is visible from the results shown in the paper.\n4. Do the authors have any insights or results on why it is useful to shift register neuron activations to a new token at test-time, and if the model does indeed rely on attention to the added register to make predictions? It would seem like trained models could have learned to rely on positional encodings to affect the attention process, including to use the global information stored in register neurons. Is there a reason that we can expect the model to still be able to use the global information from the register neurons, even after shifting their activations to the register token? One additional test to validate that the model does use the test-time register could be to remove activations from register neurons (as is done when shifting them to the register token), but not include the register token.\n5. It would appear to me that the most critical aspect of preventing typographic attacks (Section 5.4) might be the modification of activations located in patches corresponding to text. Could the authors discuss more if it is critical that register neurons’ activations are used for this rather than simply setting zeros or random values in the same activations?\n\nLIMITATIONS:\nI find the limitations discussed in Section 6 sufficient.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}