Registers in Small Vision Transformers: A Reproducibility Study of Vision Transformers Need Registers
Authors: Linus Ruben Bach
Emma Bakker
Rénan van Dijk
Jip de Vries
Konrad Szewczyk
Venue: Accepted by TMLR ()

Abstract: Recent work has shown that Vision Transformers (ViTs) can produce “high-norm” artifact tokens in attention maps. These artifacts disproportionately accumulate global information, can degrade performance, and reduce interpretability in these models. Darcet et al. (2024) proposed registers—auxiliary learnable tokens—to mitigate these artifacts. In this reproducibility study, we verify whether these improvements extend to smaller ViTs. Specifically, we examine whether high-norm tokens appear in a DeiT-III Small model, whether registers reduce these artifacts, and how registers influence local and global feature representation. Our results confirm that smaller ViTs also exhibit high-norm tokens and registers partially alleviate them, improving interpretability. Although the overall performance gains are modest, these findings reinforce the utility of registers in enhancing ViTs while highlighting open questions about their varying effectiveness across different inputs and tasks. Our code is available at https://github.com/SnorrenanxD/regs-small-vits.

Published in Transactions on Machine Learning Research (02/2025)
Registers in Small Vision Transformers: A Reproducibility Study of Vision Transformers Need Registers
Linus Bach University of Amsteriam
www.sbacco@studen...o uvo.
Emma Bakker" University of Amstendam
wwme.bbakerr3@sttudend..uuuu.
Rénan van Dijk!
wew.n.ww..iikkkBssuudet.....u.
University of Amstendam
Jip de Vries University of Amstendam
jjp.dewrriss@tuudent.ue.ccmm
Konrad Szewezyk
konszewcaykkouutook.com
University of Amsteriam
These authors contributed equally to this work.
Reviewed on OpenReview: https: // openreview. net/ forum? 5Jf
Abstract
Recent work has shown that Vision Transformers (ViTs) can produce "high-norm" artifact tokens attention maps. These artifacts disproportionately accumulate global information, can degrade performance, and reduce interpretability in these models. Darcet al. (2024) proposed registers auxiliary learnable tokens to mitigate these artifacts. In this repro- ducibility study, we verify whether these improvements extend to smaller ViTs. Specifically, we examine whether high-norm tokens appear in DeiT-111 Small model, whether registers reduce these artifacts, and how registers influence local and global feature representation. Our results confirm that smaller ViTs also exhibit high-norm tokens and registers partially alleviate them, improving interpretability. Although the overall performance gains are mod- est, these findings reinforce the utility of registers in enhancing ViTs while highlighting open questions about their varying effectiveness across different inputs and tasks. Our code is available at https::/giithbb.. http:////giihuu..oooooooeereeeee http://hww....//oorrraaa///raaaa
registers-auxillaryy
1 Introduction
Vision transformers (ViTs) have emerged as a powerful alternative to convolutional neural networks (CNNs) in computer vision, offering flexible mochanism for modeling spatial relationships in images. Originally introduced by Dosoviiskiy al. (2021), ViTs self-attention mechanisms capture both local and global interactions within images, enabling strong performance across tasks such image classification, segmentation, and detection.
Building on the foundational VIT architecture, advancements like DeiT Touvron et al. 2021 DeiT-III [Touvron et al. 2022), and the DINOv2 model (Oquab et al.) 2023 have significantly enhanced the effi- ciency and effectiveness of ViTs. DeiT and DeiT-III introduced refined training methodologics including new augmentation techniques and optimized regularization strategies, making ViTs more data-efficient and practical for datasets like ImageNet- 1k and ImageNet-21k (Deng al.] 2009). Concurrently, DINOv2 rein-

---

Published in Transactions on Machine Learning Research (02/2025)
forced the role of self-supervised learning in ViTs by generating robust, general-purpose visual features from large-scale curated datasets, enabling superior performance in both image-level and pixel-level tasks.
Despite these advancements, ViTs, including enhanced versions like DeïT' and DINOv2, face significant challenges. Darcet et al. (2024) identified the presence of artifacts in the self-attention maps of transformer blocks, which encapsulate global information and inadvertently lead to the loss of crucial local details. The authors argue that traditional ViTs face two significant limitations: (1) heavy reliance on transformer layers to encode all image-level information, which can lead to inefficiencies, and (2) the lack of explicit mechanisms for maintaining state information across layers, potentially affecting long-term coherence in representations.
To address these limitations, Darcet et al. [024] introduced the concept of registers special tokens inspired by computer architecture that. accumulate and carry information between layers as an enhancement to the ViT framework. By functioning as memory units within the transformer sequence, registers provide a struc- tured mechanism for managing both local and global information more effectively. This approach roduces reliance on transformer layers to encode all image-level information and offers explicit stale maintenance across layers. Integrating registers has been shown to improve interpretability and feature smoothness by mitigating inconsistencies in feature maps and balancing local and global contexts more effectively.
The introduction of additional tokens for the purpose of retaining abstract information to transformer ar- chitectures has been proven effective in the field of NLP. Most relevant for our investigation is the work of Burtsev et al. (2020), who showed that concatenating memory tokens to a series of token embeddings improves translation quality in transformer-based machine translation models. The authors observe a similar issue to the artifacts in ViTs when mixing local and global information within token embeddings, inevitably leading to inefficiencics. While the authors extend their analysis to separate memory controllers, their results indicate that simply enabling the model to isolate global sequence representations improves performance and interpretability.
While prior work on registers has focused on large-scale Vision Transformers (ViT-L and above). recent studies show that high-norm artifact tokens are not exclusive these architectures. Darcel et al. (2024) found these artifacts to be especially prominent in large models (see their Figure 4c), but follow-up work has observed similar behavior in smaller ViTs, albeit typically less frequently less severely (Nakamura et al., 2024 Shang et al. 2024 This suggests that such artifacts represent broader phenomenon in ViTs. Since smaller models are widely used in real-time and resource-constrained settings, mitigating artifacts in this regime is both theoretically and practically relevant. If registers improve local feature fidelity and attention consistency in small models, they offer a lightweight enhancement worth exploring. Conversely, artifact severity scales with model size, this helps define when registers are most beneficial.
In this reproducibility study, we examine the main claims from Darcet et al. (2024) and extend their ex- periments to smaller-scale models. Specifically, we explore whether (1) high-norm artifact tokens also present in smaller ViT, (2) registers encoding reported for larger models also generalize to smaller architectures. By replicating and expanding these experiments, we aim to clarify the broader relevance of registers in stabilizing and refining Vision Transformer representations.
mitigate them effectively and (3) the improvements in local-global
2

---

Published in Transactions on Machine Learning Research (02/2025)
DINDO dame Without registrr
DIM022 Gaant V0tH requtars
DiMD2 Small PUSSAUE regulars
2/b Small registrr
DeiTall Smatt negistera
DenTil Smatt With regusarr
puuut Image
Figure A comparison of attention maps from the DINOv2 giant model with and without register tokens, the DINOv2 small model with and without register tokens and our DeiT-IIII small models with and without register tokens. Register tokens enable interpretable attention maps for most images of the DINOv2 model, while they do not necessarily improve the attention maps of the DeiT-Ill model.
1.1 Key Concepts and Terminology
We briefly define key terms used throughout this study to ensure clarity.
High-norm tokens (or artifact tokens) are token embeddings with unusually large norms, often encoding global information at the expense of local detail. In attention heatmaps, they appear as bright (yellow or green) patches, as seen in the first column of Figure
Low-norm tokens have more typical norms and retain more local features. They correspond darker (blue purple) regions in the maps.
Registers are additional learnable tokens inserted into the input sequence. They help carry global context across layers and reduce the burden on patch tokens. This mitigates high-norm artifacts and results in more eveuly distributed attention, as shown in the most colunns of Figure
2 Scope of Reproducibility
In their study, Darcel el al. (2024) use ground-up-trained DeiT-III base, OpenCLIP, and DINOv2 large models. We replicate these original experiments on larger pretrained VYT models and extend the investigation to smaller DciT-III Small model, trained from scratch on ImageNet-1k. Our focus is determining whether high-norm tokens persist across different architectures and scales and whether registers consistently alleviate these anomalies while improving learned representations. By exploring smaller ViTs, we assess whether the benefits observed in previous work hold in more constrained architectures.
2.1 Model Architecture and Registers Implementation
Artifact tokens, visible as strong poaks self-attention maps Figure 1), indicate where the model allocates disproportionate altention, potentially destabilizing its representation. Additionally, these outliers can be identified as having significantly raised token norm. Throughout this work, will use the terms high-norm tokens and artifacts synonymously.
Darcet et al. (202)) addressed this by introducing registers additional learnable tokens that act as inter- mediate memory, reducing the transformer's reliance on attention layers alone for global context. In that work, registers were added a DINOv2-based ViT (sce Figure 2 of Darcet et al.) (024)), where they join the input sequence alongside the [CLS] and patch tokens. This approach mitigated high-norm artifacts and improved representations, particularly in dense tasks where patch tokens retained.
3

---

Published in Transactions on Machine Learning Research (02/2025)
output
Transformer Model
[REC1]
[R8G2]
[REGN]
Input patches
Figure 2: Illustration of the proposed remediation and resulting model. We add N additional learnable input tokens (depicted in yellow), that the model can use as registers. At the output of the model, only the patch tokens and [CLS] tokens are used, both during training and inference. Adapted from Darcet al. (2024)
In our study, we adopt the same core idea the original paper by adding registers to DeiT-III models. Specifically, we investigate whether DefT-III Small architecture benefits similarly from these additional tokens. We focus confirming the presence of high-norm tokens in smaller models and evaluating whether registers effectively resolve reduce these artifacts.
2.2 Main Claims
The original paper by Darect et al. 2024) highlights three key points that we soek to reproduce and validate in this study:
1. High-Norm Tokens (Artifacts) in Larger Models: High-norm "artifact" tokens consistently appear in large-scale ViIs.
Register-Based Mitigation: Introducing registers-additioonn additional learnable tokens acting as inter- mediate memory reduces or eliminates these high-norm artifacts, leading to more stable and inter- pretable attention distributions.
3. Enhanced Local-Global Encoding: High-norm tokens reflect an imbalance in how local and global information is captured. By segregating global information into registers, ViT's retain a more effective balance local and global features, resulting in more coherent and expressive intermediate representations.
Together, these claims propose that registers offer a robust architectural enhancement for ViTs, improving both interprotability and represcntation quality and potentially generalizing across different model scales and training paradigns.
2.3 Broader Context
The incorporation of registers in ViTs makes these models more interpretable and efficient. By reducing inconsistencies in feature and attention maps, the registers clarify the model's internal processes, enhancing trust and understanding. They also strcamline information flow, potentially lowering computational overhead and improving scalability, which is a crucial advantage in resource-constraines settings. By incorporating smaller-scale DeiT--III models into this study, we broaden the scope of Darcet et al. (2024) to examine whether their proposed register mechanism also benefits ViTs with fewer parameters. confirmed, these findings suggest that registers can offer a robust architectural enhancement to wider array of ViT variants, potentially leading to more interpretable and efficient models across different scales.

---

Published in Transactions on Machine Learning Research (02/2025)
3 Experiments
To address the first two claims outlined above, we conducted experiments on pretrained DINOv2 models well as DeiT-III models trained from scratch, with and without registers. Our primary analyses include examining attention maps, token norms and final performance metrics on ImageNet-lk.
To evaluate claim 3 concerning the models' handling of global information we first analyze the spatial token embeddings for global content and then apply the LOST method (Siméoni et al. 2021 to assess object- level understanding. LOST (Localizing Objects with Self-supervised Transformers) is a sclf-supervised object discovery method that uses patch token similarity to identify salient regions an image without any labeled data. It relies on local relationships between patch representations and provides insight into the emergence of object-level semantics in transformer models. We apply LOST to DeiT-III model with register tokens to assess whether the inclusion of registers indirectly improves the model's ability to localize objects by influencing the quality of patch-level representations.
3.1 Model descriptions
In this study, evaluated the impact of incorporating registers in ViTs using both pretrained DINOv2 models as well as DeiT-III models that were trained from scratch.
Pretrained DINOv2 Models: For the reproducibility part of our study, we analyzed the pretrained DINOv2 Large model (300M parameters), obtained from the official repository DINOv2 employs a vanilla VIT backbone and a dual-network self-distillation setup, which one network generates target embeddings that the other learns. We evaluated the large model with and without registers to replicate key results from Darcet al. (2024). We also performed diagnostic checks on the smaller DINOv2 variant (21M parameters) to assess whether high-norm artifact tokens appear at reduced scale.
DeiT-III Models: We also trained DeiT-IIII Small models (22M paramcters) from scratch to directly measure the impact of register lokens on performance and representation. All models were trained for 800 epochs on ImageNet- 1k using 16 x 16 patches and standardized with ImageNet mean-std normalization, following the official DeiT-III repository by Touvron et al.) (sce Appendix A for dotails). To evaluate the effect of register count, we trained models with 0, 2, 4, 8, and 16 register tokens, implemented as additional CLS-style tokens following the DINOv2 setup. Unless otherwise stated, we the 4-register configuration for qualitative analyses, to remain consistent with the setting used by Darcet et al. (2024).
3.2 Datasets
This study primarily uses ImageNet-1k (Deng al. 2009) for training and evaluation Image 1k contains 1.28 million training images and 50,000 validation images across 1,000 classes. We use the official dataset split, which resizes images to 224 x 224 pixels and applies ImageNet mean-std normalization.
For the LOST experiment, we use the VOCO7 dataset (Everingham et al., 2010). Additionally, we employ Imagenctte [Torii] 2020) and Caltech101 Li et al. 2022 for further evaluation of token representations in specific experiments.
3.3 Experimental setup and code
We intograted the register tokens into the DeiT-III codebase by inserting additional learnable tokens alongside the [CLS] and patch tokens, as shown in Figure 3 The figure illustrates the configuration with four registers, following Darcet et al. (2024), but in our experiments we evaluated models with 2, 4, 8, and 16 register tokens. All registers were randomly initialized from a normal distribution with low variance, following the same approach as for the [CLS] token. Unlike SANOv2-basch architectures, our DeiT-III setup discards all patch tokens at the outpul stage, relying solely on the [CLS] loken for classification.
5

---

Published in Transactions on Machine Learning Research (02/202))
output
Transformer Model
11022
input patches
Figure 3: Modified ViT architecture with registers. The 4 additional learnable input tokens (highlighted yellow) act registers, while only the [CLS] token outputted during training and inference.
4 Results
This section examines our results in relation to the three main claims from Section 2 (1) the presence of high-norm tokens, (2) the effectiveness of registers in reducing artifacts, and (3) their impact on local- global feature balance. We structure our findings as follows: Section 4.1 evaluates classification performance, Section 4.2 analyzes local features by measuring high-norm tokens in DINOv2 and DeiT-III, and Section 4.3 investigates global features, using LOST for object-level understanding and assessing spatial patch token representations.
4.1 Classification Performance
We begin by evaluating our models on the ImageNet--lk (Deng et al. 2009 validation set. The baseline DeiT- IIT Small model (without registers) achieves a top-1 accuracy 81.05 $1.05%. When two register tokens are added, accuracy slightly decreases 80.96%, and with four registers it further drops to 80.81%. However, when using eight and sixteen registers, accuracy recovers to 81.04% and 81.07%, respectively. These differences are small, indicating that register count has a minimal elfect classification accuracy. For the rest of the analyses in this paper, we use the four-register configuration to enable direct comparison with the DINOv2- based setup in Darcet et al. (2024).
4.2
Local Features
4.2.1
Local information of high-norm tokens
To test whether high-norm tokens primarily carry detailed local information or global context, we train a small Multi-Layer Perceptron (MLP) to reconstruct the original 16 16 x image patch from single token embedding. The MLP has an input size of 384 (matching our DeiT-IIT hidden dimension), single hidden layer of size 1536, and an output layer of 768 (for the flattened 16 16 x patch). We train this MLP for five epochs on high- and low-norm tokens drawn from our DeiT-III model with no register tokens.
We find that high-norm tokens exhibit a substantially larger MSE of 176.54 compared to 120.02 for low- norm tokens, representing a 147.1% incrcase in error. This indicates that high-norm tokens lose a significant amount of local detail, supporting the original claim by Darcet et al. (2024) that high-norm "artifact" tokens predominantly capture global information and thus encode fewer spatial details.
4.2.2 Artifacts in DINOv2
Unless otherwise stated, all token norm and cosine similarity analyses in this section are computed over the full Imagenette validation set (3925 images). Using this data, we observe how token norms evolve in the pretrained DINOv2 giant model. In Figure [aj the distribution of token norms for the DINOv2 giant model without registers displays heavy tail toward large values. These "outlier" tokens, which have notably higher norms than the rest, coincide with the "artifacts" identified by Darcet. et. al. (2024). indicating

---

Published in Transactions on Machine Learning Research (02/2025)
accumulation of excessive global information in just few tokens. Such imbalanced allocation of global context may adversely alfect the encoding of local features.
When registers are introduced (Figure 4b the long tail of the distribution shrinks significantly, signaling that fewer tokens are carrying disproportionately large norms. This aligns with the hypothesis that registers may help absorb or redistribute global context that would otherwise concentrate certain patch tokens, although our results do not directly establish causality. Consequently, no single token disproportionately dominates the self-attention mechanism and the overall norm distribution becomes more balanced.
Denstty
20-1
Dersty
30-
304
204
Nor
(a) DINOv2
(b) DINOv2 (with registers)
(c) DeiT-IIII
(d) DeiT-III (with registers)
Figure 4: Histograms of token norm distributions for DINOv2 giant (subfigurcs and DeiT-III Small
(subfigures
comparing models without registers (left panels) and with registers (right panels).
Looking more closely at how these norms progress across the network, Figure shows layer-wise norm distributions for DINOv2 giant. Without registers (left), the deeper layers produce outlier tokens that far exceed the typical norm range, indicating over-accumulation of global context. With registers (right), the distribution becomes smoother and high-norm tokens effectively vanish, reflecting more balanced distribution of information.
Qualitative attention maps (Figure I corroborate these findings. Without registers, the model's attention is sharply focused on a few regions, reinforcing the notion that artifact tokens become dominant. In contrast, the inclusion registers yields more uniformly distributed and interpretable attention. Collectively, these observations validate that registers effectively mitigate high-norm artifacts in the DINOv2 giant architecture, ensuring healthier balance between local and global feature representation.
Distribus
10r1
50
Lepa
(a) DINOv2
(b) DINOv2 with registers
Figure Distribution norm values across layers models. Registers eliminate high-norm outliers in deeper layers.
Finally, we replicate the cosine similarity analysis for DINOv2 giant. In Figure ba we measure the cosine similarity of the initial patch encodings of neighboring tokens. Here, we set the threshold for high-norm (artifact) tokens at the 98th percentile norm value, following Dareet et al. who found approximately 2.37% of tokens be outliers. Consistent with Darcet et al. (2024). we observe that these high-norm tokens tend be more internally consistent. with some of their neighbors (i.e., close to cosine similarity of 1.0), suggesting

---

Published in Transactions on Machine Learning Research (02/2025)
strong clustering of attention in those areas. The introduction of registers helps redistribute this global context more evenly, diminishing the outlier effect and producing more stable representations.
DINOv2 (ViT-g14)) Cos Similarity
DeiT-Ill: Cos Similarity
Normal patches
Normal patches
Artifact patches
15
Artifact patches
Density
6
Density
2-
0-
0.2
1.0
0.0
0.4
0.6
Cosine Similarity
Cosine Similarity
(a) DINOv2 Cosine Similarity
(b) DefT--1I1 Cosine Similarity
Figure Cosine similarity analysis for DINOv2 and DeiT-III models.
Figure 1 shows that the small DINOv2 models do not demonstrate artifacts in the classification token's 's attention maps. This supports the original paper's claim that these artifacts only appear in larger models. We have noted that this does not hold for the DeiT-III model, where artifacts do arise in the small version of the model.
4.2.3 Artifacts in DeiT-III
Although the addition of registers has limited effect the accuracy of classification, it mitigales the existence of artifacts to some extent, improving the interpretability of the model. Figure illustrates that registers can cause artifacts to disappear, although this is not always the casc.
Iopur Image
18 aithoution nog strve
With me regichere
With 4 regidtes
58th & iegidwes
With 16 regiteee
Figure 7: Attention maps of our retrained DeiT-III model using 0, 2, 4, 8 and 16 registers
Unless otherwise specified, all DeiT-III models discussed in this section employ four register tokens, consis- tent with the configuration used for DINOv2 by DDrcet et al.] In the histograms of token norm distributions, visualized in figures 4c and Ad the outliers in the non-register version (4ch are less present than in DINOv2 giant (4a), yet they still manifest as a high-norm tail. Adding registers in DeiT-III compresses the dis- tribution, indicating a partial absorption of the global information that these outlier tokens had originally carried.
As shown in Figure Sa high-norm tokens also appear in DeiT-III, especially in deeper layers where certain tokens accumulate excessive information, mirroring DINOv2 giant. Introducing registers reduces both the frequency and intensity of these outliers (Figure Sht. resulting in more controlled distribution and fewer

---

Published in Transactions on Machine Learning Research (02/2025)
norm spikes. However, high-norm artifacts still occur, suggesting that while registers help balance attention allocation, their effectiveness may vary in smaller models like DeiT-III Small compared to larger architectures.
400
400
10
300
10
300
Nom
300
Nom
300
10
Proportion
30
Proportion
200
10-45
10
10
10
3
100
10-4
10
10-4
10
10
500
500
Layer
Layer
(a) DelT-III (layers)
(b) DeIT-III with registers
(c) DelT-III (iterations)
(d) DeiT-III with registers (it- erations)
(layers)
Figure 8: Distribution of norm values for De1T-111 ((a) and (c)) and DeiT-III with registers ((b) and (d)).
The left two plots show norms across layers and the right two plots show norms across training iterations.
To explore how these artifacts relate to the broader token space, we replicate the cosine similarity analysis on DeiT-III (Figure 6b), flagging "artifact" tokens as the top 2% highest norms. Like in the larger model, these high-norm patches appear in uniform, low-texture regions. While outliers capture global cues, they show weaker local aligmment with neighboring patches.
This effect is further illustrated in the attention maps of Figure L Unlike in DINOv2 giant, where registers consistently improved interpretability, the effect in DeIT-III is more mixed. Some attention maps become more structured with registers, while others retain scaltered and concentrated activations. This variabilily indicates that while registers can mitigate artifacts, their effectiveness may vary depending on specific input images and training dynamics.
spoch
80
160
240
320
400
480
560
640
720
800
REGO
REG1
RBG2
REG3
Figure 9: Attention maps of the CLS and register tokens across training epochs where artifacts are success- fully removed.
In figure we further analyze the norm distribution of the last layer throughout the training process. We compare models trained without registers (Figure Sch and with registers (Figure 8d]. Although the distri- bution of norm values appears visually similar for both models, DeiT-III with registers shows visually more compact distributions in the last quarter of training iterations. We did not compute statistical dispersion metrics for this observation, so it should be interpreted shows significant outlicrs in norm values in the first half, indicating that the registers are limited in their ability to combat. high-norm tokens.
qualitative trend. However, the model still
Figure 9 tracks the evolution of attention maps across training epochs. Here, can be seen that the model alternates between high-norm outliers and wellsstructured attention maps. This pattern indicates that attention dynamics are not entirely stable throughout training, yet the registers are often able to mitigate persistent artifacts.

---

Published in Transactions on Machine Learning Research (02/202))
CLS Taken
Token
Token
Token
Figure 10: Averaged attention maps for DefT-III model with registers on the Imagenette dataset. The visualization shows the positional focus of the [CLS] token and register tokens.
Figure 10 shows that in the DefT-III model with registers, attention remains centered around the [CLS] and register lokens, helping stabilize focus but not completely eliminating local artifacts. We further measured the cosine similarity of attention maps for each register token (Figure [1] and found that, while initially overlapping, these maps diverge slightly over time. However, this divergence is modest, and the attention patlerns of the register tokens remain broadly similar. This suggests that, in our DeiT-III Small model, the register tokens do not exhibit clear specialization in their spatial focus.
Casine Similarity of Registar Tokens Over Epachs
1.00
0.98
Cousre Siitlerty
2.30
0.9
0:52
0.90
100
200
300
400
500
600
700
800
Epoch
Figure 11: Mean cosine similarity between all register token attention map pairs over epochs. Lower similarity over time indicates increasing specialization among registers.
Overall, these results highlight that while registers help reduce high-norm artifacts in DeiT-III, their impact is more inconsistent. than DINOv2 giant. Some improvements in attention interpretability are evident, while certain cases still exhibit instability, particularly in deeper layers and across training epochs.
4.3
Global Features
4.3.1
Object Detection with LOST
In this section, we evaluated ViTs with registers using the LOST {imééni et al. 2021 method on the VOC2007 dataset [Everingham et al. 2010). Table 1 shows the CorLoc values for several ViT architectures, including the large and small DINOv2 models and DeiT-III model with two, four and six registers. Results for DINO (Darcet al. 2024 are also presented for reference.
For DINOv2, we observe moderate improvement in the large model, where CorLoc increases from 33.77% to 35.24% with registers. This trend is consistent with the findings in Darcet et al. (024]), though observed gain is smaller. In contrast, the small DINOv2 variant sees drop from 37.48% 48% to 32.93%, suggesting that limited capacity may hinder the effective use of register tokens for object localization.
For the DIl configurations, all models share the same baseline without registers (15.13%). As shown in Table 1 the CorLoe score improves progressively with more registers, reaching a maximum of 19.01% with sixteen tokens. These results suggest that adding more registers can enhance localization performance in small ViTs, although the gains modest and may plateau beyond certain point.
10

---

Published in Transactions on Machine Learning Research (02/2025)
Table CorLoc values of LOST on different ViT models with and without registers.
Model
Without Registers (%)
With Registers (%)
A (%)
DINOv2 (large)
33.77
35.24
+1.47
DINOv2 (small)
37.48
32.93
-4.5
DINO
61.48
DeIT-11I (2 regs)
15.13
15.49
+0.36
DefT-III (4 regs)
15.13
17.92
+2.79
DeiT-III (8 regs)
15.13
18.03
+2.90
DeiT-Ill (16 regs)
15.13
19.01
+3.88
We note that our LOST implementation is fully deterministic: uses fixed resizing and center cropping, and contains no stochastic components such as dropout or random sampling. As a result, CorLoc scores remain exactly reproducible across runs, even when changing the random seed (o =
4.3.2 Estimation of global information in spatial patch tokens
Another important aspoct of the artifacts described by Darcet et al. 20244) their potential to encode global information. In their study, the authors theorized that high-norm tokens might carry broad, high-level information erucial for global computations. To investigate this hypothesis, they trained a logistic regressor to classify images based on individual patch tokens. Notably, high-norm tokens consistently outperformed low-norm tokens, indicating that these "artifact" tokens capture more global concepts of the image.
Table Classification accuracy (+ std) for our DeiT III model, across three runs.
Caltech101
CIFAR10
no Registers
w/ Registers
no Registers
w/ Registers
High Norm
.8540 + .0078
.8153 + .0446
.7853 + .0067
.7841 + .0074
Low Norm
.7161 + .0062
.7911 + .0558
.7534 + 00033
.7727 + .0030
[CLS]
.9988 + .0002
.9990 + .0004
.9251 + .0017
.9285 + .0034
To replicate this, we conducted a similar experiment on our DeiT-III model. In cach batch, we identified the top 2% of tokens by norms as high-norm outliers and the rest as low-norm tokens. We then selected one high-norm token, one low-norm token, and the CLS token to train logistic regressor to predict the image class. Table 2 shows that high-norm tokens constantly achieved higher accuracy than low-norm tokens in models without register, but much more similar accuracics in models with registers This supports the idea that, without registers, artifact tokens encode more global information, and by introducing registers, we can distribute the information more evenly across tokens all.
5 Discussion
(0024) hold true for smaller-scale Vision Transformers, specifically a DeiT-III Small model trained on Image Net- 1k. This involved addressing three key questions: (1) Do high-norm artifact tokens also emerge in smaller ViTs? (2) Can registers effec- tively mitigate these artifacts in low-parameter regime? and (3) Do the local-global encoding improvements observed in larger ViTs generalize to smaller architectures? Below, contextualize our observations and connect them back to these questions.
Our primary aim was to examine whether the main findings of Darcet et al.
High-Norm Artifacts in Smaller Models: Our experiments confirm that, while artifacts do not appear frequently enough in DINOy2 small models, high-norm artifact tokens do arise in DeiT-IIII Small, albeit less
FFnnd additional datasets and results of DINOv2 small in the Appendix
11

---

Published in Transactions on Machine Learning Research (02/2025)
frequently and less pronounced than in larger models. This finding refines the conclusion from Darcet et al. (2024), whose Figure 4c suggests that only ViT-L, and above develop high-norm artifacts. While it is clear that increasing capacity amplifies the effect in DINOv2, our results underscore that smaller architectures are not entirely immune. In other words, although the smaller DeiT-III model cxhibits fewer "extreme" outlier tokens, we still observe instances of disproportionately high norms, indicating that artifact formation can emerge even outside the large-model regime.
Impact of Registers in Mitigating Artifacts: We found that adding registers generally reduced the prevalence and intensity of these outliers in DeiT-Ill, echoing the trends observed in DINOv2 giant. While the improvement was not as pronounced as in the original paper, our results do show that registers are beneficial for smaller models as well. Specifically, both norm distributions and qualitative attention maps indicated fewer extreme spikes, aligning with the second key question about the role of registers in stabilizing smaller ViTs.
Local-Global Encoding and Downstream Performance: A key question was whether the enhanced local-global encoding capacities reported in Darcet et al. (2024) would extend to smaller models. Using LOST (Siméoni et al. 2021 as proxy for object-level understanding, we observed moderate CorLoc gains when registers were added to both DINOv2 large and DeiT-IIII small, while DINOv2 small exhibited a decline in CorLoc performance. Although these gains are more modest than the improvements documented by Darcet et al. (2024) for larger ViT variants, our findings suggest that the effectiveness of register tokens is not determined by model scale alone. Instead, the improvements appear linked to the presence high- norm artifacts in the attention maps: both DINOv2 Large and DeiT-IIII Small, which exhibit such artifacts, benefited from registers, whereas DINOv2 Small, which does not, experienced degraded performance.
5.1 Limitations
While our reproducibility study confirms that register tokens can mitigate some high-norm artifacts in smaller ViTs, several caveats temper the generality of our conclusions.
First, restricted ourselves to focusing on a single archtecture-dataset pair (DeiT-I11-S on ImageNet-1k) 1k) and a fixed set of training hyperparameters. Although some experiments were also performed using DINOv2 small, these experiments were done using the pretrained models which were made available by the original authors. The observed behavior may not generalize to other backbone sizes, patch resolutions, optimization strategies, or data regimes (e.g., domain-shifted or multimodal corpora).
Second, we explored only one register initialization scheme and narrow range of register counts 16).). We did not perform a systematic search for optimal configurations examine interactions with other architec- tural components such as positional encodings token pruning.
Third, our evaluation focuses on four axes classiisiiiatnn accuracy, token-norm statistics, atlention visual- izations, and LOST CorLoc-omitting dense prediction tasks (e.g., segmentation, depth) where the original Darcet et al. (0244) reported the largest gains.
Fourth, the observed gains on small models were modest and sometimes inconsistent across images and metrics. This suggests that the register mechanism may be sensitive to training stochasticity or image complexity.
Taken together, these limitations suggest that our results should be viewed as an initial, rather than com- prehensive, validation of the rogister hypothesis at smaller scales.
5.2 Future work
While registers reduce high-norm artifacts in attention maps, their effect is not uniform across all images. A key avenue for future research is understanding why some inputs still exhibit artifact-like behavior while oth- ers benefit more substantially, Analyzing the relationship between image complexity, token representation, and attention dynamics could offer deeper insights into when and how registers succeed or fail. Ultimately, this could lead to architectural or training refinements that further enhance the robustness of ViTs.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Acknowledgments
We would like to thank Jax for posing in figure
References
Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory Sapunov, Memory transformer. 2020.
URL httpss//arxxxx.oor/////////0// 11527
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 16588
2024. URL
Jía Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Lí Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 248 255. IEEE, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL
http://aaaxii..o/aaa/////// wrgg/ba/////oo.o.. 11929
Mark Everingham, LuB Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge 2007 (voc2007). In 2007 IEEE International Conference on Computer Vision (ICCV), pp. 1429-2432 IEEE, 2010.
Fei-Fei Li, Marco Andreeto, Aurelio Ranzato, and Pietro Perona. Caltech 101, apr 2022.
Kazumoto Nakamura, Yuji Nozawa, Yu-Chieh Lin, Kengo Nakata, and Youyang Ng. Improving image clustering with artifacts attenuation via inference-time attention engineering, 2024. URL https://arxiv.
org/abs/ aog//bb//410... 04801
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. 2023. URL
wrwpv/..rrg/bab/234...o 07193
David Patterson, Joseph Gonzalez, Quoe Le, Chiyuan Liang, Lluis Munguia, David Rothchild, Daiyi So, Mae Texier, and Jeff Dean. Carbon emissions and large neural network training. https::/dooooor/////// 48550/arxiv.214 2104. 10350, 2021.
Jinghuan Shang, Karl Schmeckpeper, Brandon B. May, Maria Vittoria Minniti, Tarik Kelestemur, David Watkins, and Laura Herlant. Theia: Distilling diverse vision foundation models for robot learning, 2024.
URL https://arxiv hwwploorggggaoe wwgg/ba///eeeeeoooo wtw.a////20..... 20179
Oriane Siméoni, Gilles Puy, Huy V. Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. 2021.
URL htttg://gstuubcccm//aattt/nagggt 2020. Accessed: 2025-01-23.
14279
Shintaro Torii. Imagenette and imagewoof: A curated subset of imagenet for rapid prototyping with deep learning.
Hugo Touvron, Matthieu Cord, Maithijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. arXio (Cornell University). 2021. URL http::::/aaxxv..og/aaaa///////// 2
Hugo Touvron, Matthieu Cord, and Herve Jegou. Deit il:: Revenge of the vit. 2022. URL https: /tt:///rxxx org/ ha////bb/222.2...
13

---

Published in Transactions on Machine Learning Research (02/2025)
A
Model Parameters
Table 3: Hyperparameters used for the DeiT--IIT Small Model ImageNet-Ik 1k
Hyperparameter
Value/Description
Datasct
Datasct Input. Size
ImageNet-1k (ILSVRC 2012-2017)
224
Number of Classes
1000
Model Architecture
Model
deit_ Seit_small, fett_small_patchh6.61 LS
Drop Path Rate
0.05
Training Scttings
Batch Size
256
Epochs
800
Optimizer
fusedlamb
Learning Rate
4x10-3
Warmup Learning Rate
1x10-6
Weight Decay
0.05
Scheduler
cosine
Warmup Epochs
5
Unscale Learning Rate
True
Data Augmentation
Mixup
0.8
CutMix
1.0
Color Jitter
0.3
Repeated Augmentation
True
Augeent
True
Regularization
Reprob
0.0
Label Smoothing
0.0
Drop
0.0
Drop Path
0.05
Loss Function
BCE Loss
True
Miscellancous
Evaluation Crop Ratio
1.0
Seed
0
In Section 8 described primary experimental setup for evaluating registers in Vision Transformers. Here, provide detailed overview of the training pipeline and hyperparameters used for DeiT--111 Small experiments on ImageNet-1k, closely following the official DeiT-III repository. Table B outlines cach component of the setup including data augmentation, optimization and regularization ensuring trans- parency and reproducibility. We highlight key choices, such as employing register tokens (when applicable) and adopting an 800-cpoch training schodule. These settings closoly replicate the strong basclines established by Touvron al. (2022) while allowing us to isolate and evaluate the impact of registers under consistent conditions.
14

---

Published in Transactions on Machine Learning Research (02/2025)
B Extended Analysis of Persistent Attention Artifacts
In the main text (Figure we illustrated scenario where the model's attention maps, although prone to transient high-norm outliers, often return to well-structured patterns due to the presence of registers. However, Figure 12 provides a contrasting case. Here, the model remains stuck outlier states for pro- longed periods, leading persistent artifacts. Despite registers being designed to mitigate such issues, these results emphasize that their effectiveness is not guaranteed under all conditions. Factors such initializa- tion, optimization strategy dataset characteristics undermine the stabilizing role of registers. These observations motivale further exploration into methods for reliably stabilizing attention dynamics in more challenging training scenarios.
CLS
Image
REG1
RB02
Figure 12: Attention maps the CLS and register tokens across training epochs in a case where artifacts persist.
C Complete Results of Global Classification from Tokens
Accuracy results (mean + std) for High Norm, Low Norm, and CLS tokens of our DeiT III model across more datasots. Configurations were run three times on different initial seeds.
Table 4:
DeiT III
Imagenette w/ Registers
Caltech
no Registers
no Registers
w/ Registers
High Norm
.9568 + .076
.9759 + 00100
.8400 + .0078
.8153 + -0046
Low Norm
.9462 + .0199
.981 + .0036
.7161 + .0062
.7941 + .0058
[CLS]
1.0 = 0.0
1.0 - 0.0
.9988 + .0002
.9990 + .0004
Flower 102
CIFAR10
no Registers
w/ Registers
no Registers
w/ Registers
High Norm
.5608 + .0319
.2427 .0153
.7853 + .0067
.7841 0074
Low Norm
.1966 + .0087
.3880 .0217
.7534 .0103
.7727 .0030
(CLS)
1.0 + 0.0
1.0 + 0.0
.9251 + .0017
.9285 + .0034
15

---

Published in Transactions on Machine Learning Research (02/2025)
Table 5: Accuracy results (mean + std) for High Norm, Low Norm, and CLS tokens of the pretrained Configurations were run three times on different initial seeds.
DINOv2 small model across multiple datasets.
DINOv2 Small
Imagenette
Caltech101
no Registers
w/ Registers
no Registers
w/ Registers
High Norm
.9724 + .0025
.9859 + .0011
.9439 + .0048
.9602 + .0027
Low Norm
.9632 + .0015
.9757 + .0029
.9123 + .0081
.9442 + .0075
[CLS]
1.0 = 0.0
1.0 0.0
.9994 + .0004
.9992 - .0008
Flowers109 02
CIFAR10
no Registers
w/ Registers
no Registers
w/ Registers
High Norm
.9000 + .0027
.9468 + .0180
.9399 + .0034
9485 + .0010
Low Norm
.7255 = 00121
.8720 = 02366
.9222 = .0038
.934 + .0022
(CLS)
1.0 + 0.0
1.0 + 0.0
.9754 + .0026
.9344 + .0014
Positional Focus of CLS and Register Tokens on different datasets
Our attention-may visualizations in Section 4 focus primarily Imagenette. In Figure I extend this analysis to additional datasets CIFAR10, M.tasets-CIFARRO,, Caltech101 and Flowers102 to illustrate how registers influence CLS and patch-token attention across diverse visual domains. The CLS token consistently focuses on a central region, rellecting its role in aggregating global information. The register tokens exhibit distinct yet overlapping focus regions. The presence of registers leads to a more distributed attention pattern, suggesting that they share the burden of global information aggregation. While all registers maintain broad attention spread, subtle variations emerge, indicating potential specialization in capturing different aspects of the image.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Token
REG Token
REG Token
REG Token
REG Token
(a) Imagenette
CLS Token
REG Token
REG Token
REG Token
REG Token
(b) CIFAR10
CLS Token
REG Token
REEG Token
REG Token
REG Token
(c) Caltech101
(d) Flowers102
Figure 13: Comparison of positional focus maps for DeiT-III models on different datasets. (a) Imagenette,
(b) CIFARIO, (c) Caltech101, (d) Flowers102.
E
Computational requirements and estimated carbon footprint
For every number of registers tested (0, 4, 8 and 16) the model had be retrained. Each model was trained single node with 4 NVIDIA A100 GPUs and 72 Intel Xeon CPU cores, taking about 66 hours per model (330 hours total). Table summarizes the key parameters used to estim the carbon footprint. The Power Usage Effectiveness (PUE) is taken from the Snellius documentation while the average Carbon Intensity Factor reflects the Netherlands in 2024. In the table, we list only the approximate per-GIU power draw; CPU usage is not separately itemized. Using these values, we arrive at a total carbon footprint of approximately 256.4 kg CO2eq for training both models (following Patterson al. 2021 ))
httpp::/serrviceeees. wut..a///hhh///appp/////////aaa
17

---

Published in Transactions on Machine Learning Research (02/2025)
Table 6: Summary of carbon footprint for reproducing our model training, following Oquab et al. (2023
Here, GPU-hoours the product of the number of GPUs (4) and total hours (330). The power consumption (400 W per GPU) is an approximate average draw, excluding CPU usage.
GPU Type
GPU Power
GPU-hours
PUE
Total Power Carbon Emitted
(W)
(MWh)
(tCOgeq)
NVIDIA A100
400
1320
1.2
0.58
0.26
18


Reviews:
[{"review_id": "55cBeyEKma", "reviewer": "Reviewer_RC8a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**   \n1. The paper addresses an important aspect of model scalability, evaluating whether the advantages of registers in large ViTs apply to smaller models.\n2.  The authors provide access to their codebase, facilitating reproducibility and further research.\n3. The paper is well-organized and easy to follow.\n\n**Weakness**\n1. The study’s conclusions are based solely on the DeiT-III Small model. To generalize the findings, it would be beneficial to include a broader range of small ViT architectures, such as Compact Convolutional Transformers [1] and EfficientFormers [2], and other specifically designed Transformers for efficiency and performance in resource-constrained environments.\n\n2. The experimental results are reported without measures of variability (e.g., mean ± standard deviation). It would be better to conduct experiments multiple times with different random seeds and report the mean and standard deviation to evaluate the robustness of the findings.\n\n3. According to the paper, the benefit of registers in DeiT-III Small is much smaller compared to DINOv2. However, the paper does not discuss the reasons behind this difference in detail. Rather than attributing it simply to model size differences, it would be beneficial to conduct ablation studies on depth, dimension, and other architectural aspects.\n\n4. Some additional ablation studies would strengthen the work. For example, the study fixes the number of register tokens without exploring the sensitivity of results to this choice.\n\nReference:  \n[1] Hassani, Ali, et al. \"Escaping the big data paradigm with compact transformers.\" arXiv, 2021.  \n[2] Li, Yanyu, et al. \"EfficientFormer: Vision transformers at MobileNet speed.\" NeurIPS, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a reproducibility study of \"Vision Transformers Need Registers.\" The study aims to extend the findings regarding high-norm artifact tokens in Vision Transformers and the use of register to small transformers. The authors conducted experiments on the DeiT-III Small model trained on ImageNet-1k to evaluate whether the benefits of registers observed in larger models persist in a smaller context. The authors report that the benefits of registers in small ViTs are smaller than in larger ViTs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Include additional small ViT architectures in the study, such as CCT [1], EfficientFormer [2] and other small ViTs, to assess whether the benefits of registers generalize across different model designs.  \n2. Perform multiple runs with different random seeds for each experiment and report the results with statistical measures (e.g., mean ± standard deviation).  \n3. Given the different impacts of registers, provide a more in-depth comparison between DeiT-III Small and DINOv2 by analyzing architectural components such as depth, width, and embedding dimensions.  \n4. Conduct ablation studies on the number of register tokens to investigate the sensitivity of the results.", "broader_impact_concerns": "It would be beneficial for the authors to include a Broader Impact Statement discussing the potential implications of their work, such as the application of improved small ViTs in resource-constrained or sensitive area.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jDqNRrYBYI", "reviewer": "Reviewer_dYQ4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The investigation into the registers in Vision Transformers (ViTs) is of interest and lays the ground for improving the ViTs performance.\n\n2. The experiments are conducted thoroughly with sufficient details.\n\n3. The paper is well-written and easy to follow. The problem of interest is clearly stated. \n\n**Weaknessess:**\n\n1. While I agree that the investigation into the registers in Vision Transformers (ViTs) is of interest, I am not sure if the study of registers in small-scale ViTs (rather than large-scale ViTs) is of high importance. \n\n2. The authors should include a preliminary section to introduce the definition of some specific terms like high-norm artifacts, registers as well as provide corresponding instances so that readers from other background can understand.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The goal of this paper is to investigate the Vision Transformers (ViTs) in a small-scale setting in terms of three following questions: 1) whether high-norm artifact tokens emerge in small ViTs?; 2) whether registers effectively address these artifacts in a low-parameter regime?; 3) whether the local-global encoding improvements in large ViTs can be generalized to small ViTs? To answer these questions, the authors perform several experiments with clear description and reports.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors should include a paragraph in Section 5 to acknowledge the limitations of their work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4j4S6sOK3o", "reviewer": "Reviewer_BWaK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThis paper examines the phenomenon of artifact tokens in vision transformers and if registers can alleviate them, as described in Darcet et al 2024, for smaller ViTs, and also reproduces some of the results of this paper. The strengths of this paper include the partial reproduction of Darcet et al 2024, tackling the interesting and important problem of whether registers help for small ViTs, and that the paper is clear and well-written. \n\nWeaknesses:\n\nFigure 1 could benefit from being computed in higher resolution (as in the original paper Darcet et al 2024) and perhaps with a color scale.\n\nLOST is introduced on page 4 but not explained and no paper is cited. \n\nIn general, it would be better if “high-norm” and “low-norm” was defined early in the paper to allow a better evaluation of claims about high- and low- norm tokens. This would be helpful for understanding the “local information of high-norm tokens” experiment (Section 4.2.1).\n\nI also think the claim about global information accumulation in non-register models and global context redistribution should be moderated, as I don’t think you can conclude this on the basis of Figure 4 alone. I think this claim could be supported quantitatively by repeating the experiment in 4.3.2 with the DeiT-III-register model and showing that classification accuracy for the low-norm tokens increases and the classification accuracy for high-norm tokens decreases. \n\nThis claim should be quantified: “Although the distribution of norm values appear visually similar for both models, DeiT-III with registers shows more compact distributions in the last quarter of training iterations”, especially as it’s hard to tell visually that this is the case.\n\nThe suggestion that register tokens in DeiT-III have specialized is not very well-supported, relying only on subtle differences in the attention map.\n\nThe claim that the effect of registers is image-dependent should be quantified.\n\nFor Table 2, since the values can be very close, especially on the Imagenette dataset, I think the claims would be more strongly supported with the inclusion of error bars. The conclusion that “high-norm tokens consistently outperformed low-norm tokens” seems too strong for the evidence presented, which indicates that high- and low-norm tokens perform comparably on Imagenette. It would also be nice to see error bars for Table 1 as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper seeks to reproduce the paper “Vision Transformers Need Registers” (Darcet et al 2024) as well as instead the investigation of this paper to smaller ViTs. In particular, this paper seeks to reproduce the following phenomena from the original paper:  (1) the existence of artifacts (high-norm tokens) in larger models, (2) that adding registers (additional learnable tokens) mitigates these artifacts, and (3) that ViTs with registers have better intermediate features.\n\nThis paper runs experiments using pre-trained DINOv2 models (with 300M parameters) as well as DeiT-III models (with 22M parameters). The pre-trained DINOv2 models included versions without registers and with registers. The DeiT-III models were trained from scratch on ImageNet-1K with both vanilla and register versions. The register versions had four register tokens. \n\nDeiT-III models were examined across several experiments: (1) classification accuracy, comparing models without register to models without registers, where register models had slightly lower but comparable accuracy, (2) local reconstruction using low- and high- norm tokens, (3) object detection with LOST, and (4) image classification on high-norm tokens. They were also analyzed on the distribution of norms with and without registers (Figure 4), distribution of norm layers (Figure 7ab) and across iterations (Figure 7cd), and cosine similarity to neighboring patches for top 2% tokens (Figure 6).\n\nDINOv2 models were examined only on object detection with LOST, and were also analyzed in the same ways as the DeiT-III models, with the exception of the distribution of norms across training iterations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Main changes:\n1. Define high- and low- norm tokens early in the paper\n2. Moderate the claim that using registers distributes the accumulated global information of high-norm tokens or provide additional evidence for this claim (such as the experiment suggested)\n3. Quantify the claim that Deit-III with registers shows a more compact distribution in the last quarter of training iterations by providing statistics on the distribution\n4. Remove the claim that register tokens in DeiT-III with registers have specialized or provide additional evidence for it\n5. Quantify the claim that the effect of registers is image-dependent with the percentage of images where artifacts are removed for DINOv2/DeiT-III\n6. Provide errors bars (over multiple runs) for Tables 1, 2", "broader_impact_concerns": "I don't believe this paper needs a broader impact statement", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SnOBU4mDbL", "reviewer": "Reviewer_JFSv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- Well structured and written\n- Comprehensive visualization of attention maps and token norm distributions\n- Confirmation that high-norm artifacts do exist in smaller ViTs, though to a varying degree\n\n**Weaknesses:**\n\nAll weaknesses are summarized at a higher level. Specific suggestions are given in the below “Requested Changes” section.\n\n- The motivation for studying registers specifically in smaller Transformers is unclear. The paper doesn’t mention whether we should expect different behavior in smaller models.\n- Quantitative results are relatively lacking, and the family of models studied is limited to supervised VITs.\n- Limited model family limits scope of insights derived from the work.\n- Limited insights and quantitative results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Inspired by Darcet et al. (2024), this reproducibility study investigates whether smaller ViT architectures also develop high-norm artifacts in their attention maps, and whether adding learnable “register” tokens can mitigate this problem. The authors focus primarily on DeiT-III Small, examining whether:\n\n1. High-norm artifact tokens occur in smaller models\n2. Registers can effectively mitigate these artifacts\n3. Local-global encoding improvements observed in larger models generalize to smaller architectures\n\nThe paper provides qualitative analysis through attention map visualizations and limited quantitative evaluation through classification performance and LOST object discovery benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "At the moment, I think the work needs further improvement to meet TMLR’s bar for acceptance.\n\n1. Paper motivation\n    - Clarify why studying high-norm attention artifacts in smaller Transformers is important. Should we expect smaller ViTs to be more or less prone to this issue?\n    - Discuss your results in relation to the findings from Fig 4c. from [1], which suggest that for DINOv2, only ViT-L and above develop high-norm artifacts.\n2. Section 4.2.2: Please clarify what data the token norm is computed on.\n3. Experiments\n    - Add self-supervised VITs for a different perspective. E.g. is it possible to also train smaller DINO or MAE models? This paper focuses on only DEIT-III small, which is a relatively limited class of models.\n    - Quantitative results for more image classification datasets (e.g. IN1K) would provide a better picture of model performance. It is still unclear if adding registers improves results overall.\n    - In general, there are no insights about how to best add registers for smaller vision transformers. E.g. what is the optimal amount of registers and how does this change with model size? Is there a saturation point beyond 4 registers?\n4. Computational analysis is missing, even if there are only 4 additional tokens.\n\n[1] Vision Transformers Need Registers. Darcet et al. ICLR 2024.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)