Vision Transformers with Self-Distilled Registers
Authors: Zipeng Yan
Yinjie Chen
Chong Zhou
Bo Dai
Andrew Luo
Venue: NeurIPS 2025 spotlight ()
Keywords: Vision Transformers
Dense Features
Self Distillation
Primary area: deep_learning
TL;DR: Our approach improves Vision Transformer Dense Representations via Self-Distillation

Abstract: Vision Transformers (ViTs) have emerged as the dominant architecture for visual processing tasks, demonstrating excellent scalability with increased training data and model size. However, recent work has identified the emergence of artifact tokens in ViTs that are incongruous with local semantics. These anomalous tokens degrade ViT performance in tasks that require fine-grained localization or structural coherence. An effective mitigation of this issue is the addition of register tokens to ViTs, which implicitly ''absorb'' the artifact term during training. Given the availability of existing large-scale pre-trained ViTs, in this paper we seek to add register tokens to existing models without retraining the models from scratch, which is infeasible considering their size. Specifically, we propose Post Hoc Registers (**PH-Reg**), an efficient self-distillation method that integrates registers into an existing ViT without requiring additional labeled data and full retraining. PH-Reg initializes both teacher and student networks from the same pre-trained ViT. The teacher remains frozen and unmodified, while the student is augmented with randomly initialized register tokens. By applying test-time augmentation to the teacher’s inputs, we generate denoised dense embeddings free of artifacts, which are then used to optimize only a small subset of unlocked student weights. We show that our approach can effectively reduce the number of artifact tokens, improving the segmentation and depth prediction of the student ViT under zero-shot and linear probing.

Vision Transformers with Self-Distilled Registers
Zipeng Yan*!
Yinjie Chen
Chong Zhou?
Bo Dai
Andrew F. Luo" 11
University of Hong Kong
2 Zhejiang University
Nanyang Technological University
Equal contribution - Co-First authors
maxvellcaffrrer:155@gmml.. com, kelvinyzp@gmail.comm com
"Corresponding author: aluo@hku. hk
Abstract
Vision Transformers (ViTs) have emerged as the dominant architecture for visual processing tasks, demonstrating excellent scalability with increased training data and model size. However, recent work has identified the emergence of artifact tokens in VIT's that are incongruous with local semanties. These anomalous tokens degrade ViT performance in tasks that require fine-grained localization or structural coherence. An effective mitigation of this issue is the addition register tokens to VīTs, which implicitly "absorb" the artifact term during training. Given the availability of existing large-scale pre-trained ViTs, this paper we seek to add register tokens to existing models without retraining the models from scratch, which is infeasible considering their size. Specifically, we propose Post Hoc Registers (PH-Reg), efficient self-distillation method that integrates registers into existing VIT without requiring additional labeled data and full retraining. PH- Reg initializes both teacher and student networks from the same pre-trained VIT. The teacher remains frozen and unmodified, while the student is augmented with randomly initialized register tokens. By applying test-time augmentation the teacher's inputs, we generate denoised dense embeddings free of artifacts, which are then used optimize only a small subset of unlocked student weights. We show that our approach can effectively reduce the number of artifact tokens, improving the segmentation and depth prediction of the student ViT under zero-shot and linear probing. Our code is publicly available at this repository.
Introduction
Vision Transformers (ViTs) now the dominant architecture in visual modeling, delivering strong performance across classification, detection, and segmentation. Unlike convolutional networks with their built-in locality inductive bias, ViTs process images by spatially splitting them into patches and applying self-attention to enable global feature interactions. This architectural design leads superior scalability, particularly with contrastive self-supervised pre-training objectives, and facilítates more flexible representation learning, as is less constrained by the translation invariance assumptions inherent in CNNs. This flexibility enables remarkable emergent capabilities. Models like CLIP, trained solely on image-text alignment, achieve competitive open-vocabulary segmentation through zero-shot dense queries; while self-supervised approaches learn semantically rich features directly from unlabeled images.
However, the same data-driven attention mechanisms that enable ViT's representation power also lead to the emergence artifact tokens. These are outlier features often discordant with local image semantics, meaning they fail to correspond locally meaningful image structures. The propensity for ViT's to generate such tokens is exacerbated by their lack of strong, built-in spatial priors, which result in inconsistent dense representations. Ultimately, the presence of these artifact tokens
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Image
MaskCLIP
SCLIP
NACLIP
Ours (PH-Reg)
Figure 1: Effect of PH-Reg Open-vocabulary Segmentation. For each image, we compare four methods: MaskCLIP which directly takes the value features from the last attention layer; SCLIP which adds correlative self-attention;; NACLIP which further enforces locality bias; and our PH-Reg method with self-distilled registers. We utilize the same OpenAI CLIP ViT-B/16 weights for all three methods. For each method, we visualize the UMAP of the dense features and heatmap of one text query. Our method yields noticeably cleaner dense features and high quality localizations, and requires only small set of additional register parameters compared to the original network.
disrupts fine-grained localization, critical capability for tasks demanding high spatial precision, such as detailed semantic segmentation or part identification.
Recent work has sought to mitigate artifact tokens via architectural modifications, where register tokens are added to the network. These register tokens are randomly initialized, with learnable parameters that participate in the self-attention process similar to the [CLS] token, but are not otherwise used during the output. Although these register tokens are not explicitly supervised during training, they effectively "absorb" the artifact term and learn to attend to global objects. While effective, introducing register tokens constitutes a fundamental architectural modification that requires training from scratch-a time-consuming and computationally demanding process. This significantly limits their applicability, especially given the vast ecosystem of existing, high-performin pre-trained vision models.
We present solution this issue with Post Hoc Registers (PH-Reg), an efficient self-distillation framework that requires no labeled data or full retraining, We illustrate OpenAI CLIP with PH-Reg in Figure 1. In PH-Reg, both teacher and student networks are initialized from the same pre-traines model weights. And the only extra parameters are the register tokens added to the student network. Our proposed framework freezes the teacher during training. Images provided the teacher undergo test-time augmentation (e.g. random offsets and horizontal flips). This augmentation strategy effectively denoises the teacher's dense features without requiring gradient- teacher itself, yielding stable dense targets. The denoised dense features are used as distillation target for the student network, where only a small set of parameters are optimized. This entire process requires only modest set of unlabeled images, enabling significant enhancements pre- trained models with minimal computational overhead. Concretely our contributions are as follows: 1. We propose test-time augmentation scheme that effectively denoise dense features in vision transformers. Our denoiser does not require costly neural fields and does not require gradient based optimization. 2. We elucidate the underlying components in student model that contribute to learning a clean dense feature map. We show that by finetuning select weights, we are able to achieve clean dense features with minimal additional parameters. 3. We demonstrate that PH-Reg effectively improves the consistency of dense feature representations in ViT',, leading to quantifiable improvements downstream tasks that rely on fine-grained spatial understanding (e.g.. semantic segmentation or depth prediction). Our method preserves the original utility of dense features without inducing unwanted distribution shift, and functions well with zero-shot language-basee dense queries.
updates on the

---

(a)
(b)
X-coord
0;-(K,y2
Frozen
Denoise
05
1.0
C;
y-cgord
T[I
Teacher
Repeat F1...n
1.0
Initialized from same weights
Clean features
Coordinates C
C2
Distill
Student
Image I
Original with artifacts
Image I
I
Register Tokens
Low
High
(c)
RGB image
MaskCLIP
SCLIP
NACLIP
(PH-Reg)
Figure 2: Learning Framework of PH-Reg. (a) Our framework begins by creating two networks from the same set of weights. In the teacher, the weights are frozen and unmodified. In the student, only additional parameters are learnable register tokens. The teacher creates a learning target using denoised representations. (b) An image I undergoes augmentation by function T with random augmentation parameters consisting of random offsets and horizontal flips.(c) Given an RGB image, utilize UMAP visualize the features, and heatmap using CLIP text query. Our method can produce significantly cleaner dense representations with minimal additional inference cost.
2 Related Work
Transformers Visual Learning. Building upon the success of self-attention language mod- eling, architectures that leverage transformer based token-mixine mixing have been proposed for visual generation [1, 2, 3] and recognition tasks [4, 5], cumulating in the VIT architecture which relies very few locality biases [6]. In the years since, many improvements and variants have been proposed [7, 8, 9, 10, 11, 12]. The improvements have largely focused on data [13] and compute effi- ciency [14, 15, 16, 17, 18, 19, 20, 21, 22]. In general, vision transformers tokenize image into of patches, where each patch is first processed using an MLP or convolution block [23, 24, 25, 26]. the patches further processed with self-attention which enables global token interactions beyond those in convolutional networks. As self-attention is permutation invariant, positional information is typi- cally injected using learnable positional embeddings or relative positions [27, 28, 29, 16, 30, 31, 32]. Positional embeddings have been suggested to play a role in the emergence of dense VIT artifacts, as networks with positional embeddings removed have smooth feature maps [33].
Representation Learning with Vision Transformers. The lack of restrictive local inductive biases in Vision Transformers enables strong scaling behavior across a diverse set of tasks. Beyond traditional supervised learning on categorical datasets such ImageNet, methods have been proposed to learn on large scale datasets by leveraging language contrastive objectives [34, 35], or self- supervised image-level objectives [36, 37, 38] and patch-level objectives [39, 40, 41, 42, 43, 44, 45, 46]. While the training objectives are very different, these methods enable strong zero-shot and linear-probe performance across diverse of tasks, suggesting that these methods effectively learn the underlying statistics of visual input. While these methods lack explicit dense supervision, the dense features from these models have been shown have strong zero-shot emergent behavior with baguugge-bsed based segmentation [47], object part correspondence [48], and structural understanding [49].
language-be
Artifacts in Vision Transformer Dense Features. Recent work on DINOv2 [46] has found that Vision Transformers can have artifacts in their dense features. It has been proposed that artifacts can be mitigated with "register" tokens [50, 51, 52, 53]. These register tokens are effectively randomly initialized embeddings that are analogous to the [CLS] token. While registers participate in the self-attention process, they are discarded during output. This approach requires a model to be trained from scratch. The nature and the mechanisms that cause the emergence of artifact tokens are unclear, and there exists conflicting results on what information (global or no information) these artifact tokens contain [50, 54]. Recent work has further investigated the mechanism of register

---

Image
OpenAl CLIP
OpenCLIP
DFN-CLIP
DINOv2
Raw
Denoised
Raw
Denoised
Raw
Denoised
Raw
Denoised
Figure 3: Denoising Teacher Representations with Augmentations. For each model, we visualize the UMAP of dense features before and after applying test-time augmentation. The results show that our proposed method produces noticeably cleaner dense feature representations without requiring gradient-based learning. Please zoom in for details.
tokens [55]. Our own results have found that artifact tokens are not necessarily high-norm, and can be low-norm as well. Unlike the observation by [33], we find that positional embeddings alone cannot account fully for the artifacts. Regardless of "why" artifact tokens emerge, removing these artifacts is an active area of research, with proposals based on registers [50], magnitude smoothness priors [54], and the foundational work leveraging neural fields denoise VIT's with static artifact component [33, 56, 57]. A concurrent line of work has sought to remove artifacts for open-vocabulary segmentation with training-free attention modifications [47, 58, 59, 60, 61, 62]. Our framework can be applied to existing pretrained networks, introduces minimal additional parameters, can be applied tasks beyond open-vocabulary segmentation, and makes no assumptions on the magnitude static nature of the artifacts.
3 Methods
In this section, we will describe the PH-Reg framework, which we illustrate in Figure 2. This framework enables existing pretrained ViTs to benefit from register tokens, yielding significantly cleaner dense representations. During training, PH-Reg requires only unlabeled images for the self-distillation process. In section 3.1, we will first describe the denoising process of teacher network outputs. Unlike prior work that rely on a neural field/hash-grid,, this method denoises dense features without the use of expensive gradient- based learning. In section 3.2, we will describe how we initialize and modify the student architecture. This approach only introduces a small set of additional parameters to the network. Finally in section 3.3 we will describe our distillation process.
3.1 Efficient Denoising of Teacher Representations Our denoising process starts from the observation that artifact tokens are not static relative to image content. Put another way, if an image is shifted by certain amount (with the gaps padded with whitespace), the artifacts do not shift by the same amount. As shown in Figure 2, given an RGB image E # Wxx x3 we randomly sample n random aug- mentation parameters (01, 02, .... Qn), where each 8, defines a horizontal/vertica. offset (Iz, yı) and boolean flip, E 1) defined horizontally. For each image, we also compute the image space coor- dinate grid C (x-coords, y-coords). Where (Ir, v) are respectively in range [0, 1]: x defines the left- right axis, and y defines the top-down axis. The coordinates C' help us keep track of the original lo- cation of an image region after augmentation. In practice, as we are working with a VIT model with patch size k x k where the tokenization process H W. grid of image tokens, we define our parameters using offsets that are integer multiples of l to facilitate efficient indexing. Together, the image I, the coordinates C, and augmentation parameter 0g are provided to transform function to yield an augmented image I, and new coordinates C, (I,C, = (I)..).
Algorithm Denoising Process
Input: Image I E xxx33. Image space coordinates C E [0, 10 (0, 1}; Augmentation parameters 0,, 02 .2., 0,:
Augmentation function T;
ViT teacher model fteacheri
1. Zero init clean feature tensor Q
2. Zero init count tensor K
3. For i in {1, ..., n):
0, = (I ys. flip,)
5.
(II, T(I,C,
6.
Dense feature F3 = fteacher (L)
7.
(Fyalid Cyalid) IF- Ci, 0s)
Q(crathin QCraalid) + Fyalid
8.
9.
K K(Cyalidj = K (Cralid +1
10. return Q/K
yields a k

---

Image
UMAP
Croissant
Jam
Image
UMAP
Palm
Sky
Sea
MaskCLIP
SCLIP
NACLIP
Ours (PH-Reg)
Figure Visualization of Open-vocabulary Semantic Segmentation. We compare against MaskCLIP, SCLIP, NACLIP, and find that our method yields clean feature maps free of artifacts.
teacher model ftencher based on a frozen set original network weights, without any additional parameters. Given augmented image Z, this network outputs feature F,. We restore the features to the original location within an image using the inverse transform function T (F;, C.). The restored features additively accumulated across different augmentation parameters, while keeping track of the number occurrences for each location. At the end, the dimension wise sample mean is computed for the accumulated features. We present our full denoising process in Algorithm 1. The patch-wise expected value of this representation is the same as the optimal value when optimizing a discrete grid of representations to minimize mean squared error (as used in DVT [33] and traditional neural field based methods). However, as we do not require gradients, this denoising process can be done in less than 200ms, roughly two magnitudes faster than neural field based denoising DVT. The comparison between raw and denoised dense feature visualizations is shown in Figure 3.
3.2 Design of the Student Network
Our objective preserve maximimal computational efficiency of the student model, while leverag- ing the knowledge of the pre-trained weights. For this purpose we introduce m number of register tokens. providing a minimally invasive enhancement to the base architecture. After the addition of register tokens, a total of m + H x E. tokens participate in the self-attention process. Unlike prior work that trains registers from scratch [50], this approach updates only these registers and selectively unfreezes specific components during distillation, preserving the majority of the ViT's pretrained weights. Through ablation studies, we identify optimal unfreezing strategies, such as adjusting convolution layers, positional embeddings, or the last transformer block.
3.3 Learning and Optimization the Student
We employ a multi-objective distillation strategy, combining cosine similarity and mean squared error losses to ensure both directional and magnitude alignment between teacher representations F Freacher and student representations F Fstudent- Our final loss is: Loss = cossim(targel, predicted) + MSE(target, predicted).
Experiments
In this section, we comprehensively evaluate the performance of PH-Reg on diverse set of dense tasks, first using a zero-shot setup for open-vocabulary segmentation in section 4.1, followed by linear probe based segmentation and depth tasks in section 4.2. Finally we perform ablation studies to explore design decisions and investigate the nature of artifacts across different models in section 4.3. All implementation details are provided in the appendix.
4.1 Open-vocabulary Semantic Segmentation Evaluation
Datasets. In this section, we follow prior works [59, 62, 61] to evaluate our approach on six semantic segmentation datasets, with their names abbreviated (in parentheses) conserve table space: PASCAL
5

---

Table 1: Open-vocabulary Semantic Segmentation Quantitative Evaluation Results 8 Bench- marks. While the first 3 benchmarks (VOC21, PC60 and Obejct) include background class, the remaining benchmarks do not. We report the mean Intersection over Union (mloU, %) metric, where higher values indicate better performance, for our method and all baseline models. The best result for each dataset is highlighted in bolded. Additional results are provided in the supplementary material.
Method
VOC21
PC60
Object
VOC20
PC59
Stuff
City
ADE
Avg.
CLIP [34]
18.60
7.84
6.50
49.05
11.17
7.19
6.65
3.16
13.77
MaskCLIP [47]
49.27
25,46
26.94
66.56
28.62
18.80
28.33
13.70
32.21
SCLIP [59]
59.62
31.74
33.52
81.53
34,46
22.65
32.34
16.45
40.08
ClearCI.IP
59.76
32.56
32.77
84.56
35.91
23.89
30.04
16.65
39.52
NACLIP [62]
58.88
32.20
33.15
79.70
35.16
23.30
35.48
17.42
39.41
MaskCLIP DVT [47, 33]
44.29
25.08
20.89
65.88
29.50
17.10
30.89
14.06
30.96
NACLIP + DVT [62, 33]
60.25
32.73
32.89
80.26
35,91
23.41
36.31
17:54
39.91
Ours (PH--Reg)
63.01
34.52
35.27
83.05
37.88
24.66
37.17
19.22
41.85
VOC 2012 (VOC21) [63], PASCAL Context (PC 60) [64], COCO-Object (Object) [65], COCO- Stuff (Stuff) [66], Cityscape (City) [67], ADE20K-150 (ADE) [68]. In addition to these standard benchmarks, we also evaluate on two commonly used variants, PASCAL VOC 2012 (VOC20) and PASCAL Context (PC 59), in which the background class is excluded from the evaluation. For all experiments, we utilize the same evaluation harness for all methods, and apply sliding window inference strategy for non-square images. We also resize input images such that the shorter side is fixed to specific resolutions, accommodating the varying original image sizes across datasets.
Baselines, We compare our method against several relevant approaches in open open-vocabulary semantic segmentation, including MaskCLIP [47], SCLIP [59], ClearCLIP [61], and NACLIP [62]. We also include vanilla CLIP as baseline in our comparison, as it can be adapted for semantic segmentation. Unless otherwise specified, all visual encoders use the widely adopted pretrained VIT backbone with the same OpenAI CLIP ViT-B/16 weights to ensure a fair comparison. We also include denoised versions of MaskCLIP and NACLIP produced by DVT, as DVT represents closely related method to our approach. Unless otherwise noted, for CLIP we adopt the most basic MaskCLIP framework (direct output without any attention modifications) as our student model. Notably, we re-implemented all baselines using the same prompt templates in [59, 62]. All reported results obtained without any post-processing refinement.
Quantitative Results. Table 1 summarizes the quantitative comparison results of various open- vocabulary semantic segmentation models. We observe that PH-Reg CLIP consistently outperforms all compared methods on 7 out of 8 evaluated benchmarks, with particularly strong results in VOC21 (63.01%) and COCO Object (35.27 (7.).. Moreover, PH-Reg CLIP surpasses the denoised versions of MaskCLIP and NACLIP, where DVT fails to yield significant performance gains. We believe this is caused by the residual estimator in DVT, which assumes stationary artifacts - an assumption that does not hold consistently for training-fres free open vocabulary segmentation methods based CL.IP. We note that ClearCLIP slightly outperforms our method on the VOC20 dataset. This may be attributed to its use of correlative self-attention, i.e. q-q attention, which incorporates feature localization cues. This explanation is plausible, as SCLIP, which also employs q-q attention, similarly outperforms NACLIP on the VOC20 dataset.
Qualitative Results. Figure 4 presents a qualitative comparison between our PH-Reg CLIP and three baseline models: MaskCLIP, SCLIP, and NACLIP. We visualize the UMAP of the dense features produced by each model, as well as the corresponding heatmaps generated from different text queries. Our qualitative observations are as follows: 1. Artifact tokens are frequently observed in the UMAP visualizations of MaskCIII, SCLIP, and NACLIP. While some artifact tokens are reduced in the heatmap of MaskCI.I, they remain prevalent in the heatmaps of both SCLIP and NACLIP. 2. The presence of artifact tokens hinders the models' ability to maintain fine- fine-grained spatial alignment with the text queries, leading to suboptimal localization. 3. In contrast, PH-Reg CLIP consistently produces cleaner UMAPs and more fine-grained, semantically aligned heatmaps, which correspond well to meaningful local image structures. These visualizations demonstrate that method effectively preserves the consistency of dense feature representations and enhances semantic alignment between visual and textual modalities.

---

Table 2: Linear Probe Based Evaluation Results Segmentation and Depth. PH-Reg improves pretrained VIT backbones across various dense prediction tasks. For semantic segmentation, we report the mean Intersection over Union (mloU, %) metric and mean accuracy (mAcc, %). For monocular depth estimation, we report Root Mean Squared Error (RMSE), absolute relative error (Abs Rel), and accuracy under threshold 61. The best result for each dataset is highlighted in bold.
VOC21
ADE
NYUd
Method
(l)
mAcc(t)
mloU(1)
mAce t)
RMSE(L)
Abs Rel(.)
(1())
CLIP [34]
73.88
83.37
35.78
47.3
0.6843
0.2115
64.93
CLIP + DVT [34,3,
74.74
84.33
36.39
48.14
0.6800
0.2089
65.07
NACLIP [62]
74.01
83.16
37,06
48.33
0.6852
0.2082
64.52
NACLIP + DVT [3]
74.47
82.98
36,91
48.56
0,6845
0.2122
65,11
MaskCLIP [47]
70.28
79.06
34.43
44.74
0.6645
0,2030
67.71
MaskCLIP + DVT [3]
71.38
80.49
34.43
44.86
0,6792
0.2091
64.96
Ours (PH-Reg)
75.32
84.96
38.07
49.58
0.6746
0,1995
68.17
OpenCIIIP [70]
71.31
80.64
37.68
49.8
0.6853
0.2113
64.86
OpenCLIP + DVT [70, 33)
72.58
83.42
38.30
50.91
0.6811
0,2159
64.73
OpenCLIP Ours
73.25
83.99
39.32
51.24
0.6784
0,2019
65.32
DFN-CLIP [71]
71.98
82.07
36.81
47.83
0.6860
0.2118
64.50
DEN-CLIP [71, 33]
73.09
83.52
37.73
49.39
0.6852
0.2092
64.65
DFN-CLIP Ours
72.97
82,48
39.15
50.61
0.6768
0.2052
65,26
DINOv2 [46]
84.13
92.00
47.82
60.50
0.4566
0,1391
82.92
DINOv2 DVT [46, 331
85.43
93.37
48.86
61.61
0.4329
0.1289
85,23
DINOv2 Ours
84.85
92.46
48.66
61.57
0.4306
0.1216
86.35
4.2 Linear Probe Based Segmentation & Depth Evaluation
Datasets & Baselines. In this section, we evaluate our approach in two semantic segmentation datasets: PASCAL VOC 2012 (VOC21) [63] and ADE20K-155 -50 (ADE) [68] and one monocular depth estimation dataset: NYUv2-Depth dataset (NYUd) [69]. Since DVT [33] targets a similar objective with our approach, we include both the vanilla models and their denoised versions produced by DVT as comparison baselines. We adopt the same linear probe experimental setup as [33], and train a linear layer integrated into the backbones, depth logits from patch tokens. Table 2 summarizes the main experiment results.
decode head to predict pixel-wise segmentation or
Semantic Segmentation Results. As shown in Table 2, We observe significant and consistent improvements, outperforming at least 4 out of denoised ViT backbones across the evaluated datasets. While DVT consistently enhances the performance of DINOv2 and vanilla CLIP, it provides only limited improvements for other ViT backbones derived from other CLIP models. In contrast, our approach yields substantial performance boosts across these backbones, especially a notable +5.04 +5.04% mloU on VOC21 and +3.64% mloU on ADE20K. These results demonstrate that our method can be robustly adopted to enhance the performance of diverse VIT backbones in semantic segmentation.
Notably, DVT relies on neural fields and requires gradient-based optimization, making the iterative denoising process applied each image individually highly time-consuming. Our method leverages test-time augmentation for denoising, enabling the generation of cleaner dense feature representations without incurring excessive computational overhead. However, our results also show that the residual estimator as introduced in DVT may be beneficial to some model types (DINOv2, DFN-CLIP) more so than others. These results highlight that PH-Reg achieves superior performance in suppressing artifact tokens through a more robust and efficient design.
Depth Estimation Results. Following prior work [46, 33] we adopt AdaBins [72] for monocular depth evaluation. As shown in Table 2, our method consistently improves the performance of pretrained VIT backbones whereas the DVT assumption of stationary artifacts mostly hold true for DINOv2. Additionally, DVT achieves performance gains using an additional transformer block with 0.08x the parameters of the base models [33], our method achieves superior results with only negligible increase in parameter count introduced by the register tokens. These results demonstrate the efficiency of our approach, yielding noticeable performance gains with minimal model overhead.
7

---

4.3 Ablation Studies and Investigation of Artifacts
In this section, we conduct ablation studies on OpenAI CLIP ViT-B/16 to investigate various architectural and training components, focusing on both model performance and training feasibility.
1.00
0.50
1.00
0.96
0.95
0:57
0.90
0.00
0.90
50%
70% Most Different
Host Difherent
90% Most Differet
0.84
0.85
95% Most Diffcrent
0.82
9IF Most Difierent
Most Offfeeet
0.80
2/45 m10
2.100
1.50
2.10
M55
come get
-18
copr pos aateiiie vatoniio
0.80
10
Number of ouggeennters
(a) Registers' behavior. This plot illustrates adding registers improve PH-Reg teacher performance. In the blue settings, only registers are unfreczed. The green settings represent the improve- ments when positional embeddings are unlocked additionally. The red settings represent performance of unlocking more layers.
(b) Augmentations improves cosine simi- larity. The plot illustrates how increasing the number of augmentations improves the alignment of the model's predictions with the target of 200 augmentations' features, as measured by cosine similarity.
Figure 5: Ablation number of registers and augmentations
The Number of Register Tokens. We evaluate the influence of the number of register tokens on the cosine similarity between the student model's outputs and the target values using the COCO Caption dataset [73]. Specifically, we distill the student model with 0, 1, 2, 4, 8, or 16 register tokens.
As illustrated in Figure 5a, the cosine similarity increases as the number of register tokens grows, indicating improved alignment with the target representations. However, the performance gain becomes marginal when increasing the number of registers from 4 to 8 and from 8 to 16. Based on this observation, we use 16 register tokens in all subsequent experiments.
Distillation Architectural Settings. We further evaluate the impact of architectural configurations the COCO Caption dataset. In this setting, we vary the number of register tokens from 0 to 16 while allowing the positional embeddings to be updated during training. As shown in Figure 5a, the improvement in the cosine similarity from unlocking the position embedding becomes less pronounced as the number of register tokens increases. Nonetheless, unlocking positional embeddings continues to provide positive effect on alignment. This result suggests that in contrast to DVT, the positional embedding itself is unlikely to fully explain the artifact tokens.
during distillation by analyzing cosine similarity
Next, we fix the number of register tokens to 16 and evaluate the effect of unlocking additional layers, including the convolutional patch embedding layer and the later attention layers. For all experiments, we report 50th, 70th, 90th, 95th, and 99th percentiles (of cosine similarity to capture the distribution of the most dissimilar features.) Our analysis reveals that incorporating even single register leads to substantial improvements. In particular, the 99th percentile of feature cosine similarity in the 1 I-register configuration exceeds the 50th percentile (median) of the raw case without registers. This indicates that registers significantly enhance the quality of feature representations across the distribution,
only in extreme cases.
As suggested by [54, 33], attention layers close to the output also play an important role, which we confirm in our experiments. As shown in Figure 5a, unlocking the last attention layer significantly increases the cosine similarity between the student model's outputs and the target values. While unlocking the convolutional patch embedding layer alone slightly reduces cosine similarity value, the overall value improves when both the convolutional patch embedding and later attention layers are unlocked, compared to the baseline with only unlocked the position embeddings and later attention layers with 16 registers. Therefore, we unlock the positional embeddings, the convolutional patch embedding layer, and the final attention layer during distillation.
The Number of Augmentations in the Denoising Process. We evaluate our approach using cosine similarity on the COCO Caption dataset to investigate the effect of the number of augmentations. The
8

---

Image
OpenAl CLIP
OpenCLIP
(b)
Image
DFN-CLIP
DINOv2
Original
w/
Norm diff
Original
w/ PH-Reg
Norm diff
PH-Reg
Figure 6: Comparison of Original and PH-Reg Features and Norms. While prior work has noted artifact tokens in DINOv2 as having higher norm than other tokens, we observe this is not the case for all models. Some models have artifact tokens with lower magnitude.
Table 3: Distillation Approaches. This table reports the contributions of different components in our distillation framework. Vanilla refers to the student MaskCLIP, Denoising Only refers MaskCLIP with 10x averaging, the other rows refer to using NACLIP as teacher and MaskCLIP as student.
Approach
VOC21
PC60
Object
VOC20
PC59
Stuff
City
ADE
Avg.
Vanilla
49.27
25.46
26.94
66.56
28
18.80
28.33
13.70
32.21
Denoising only (10x aug)
51.41
28.13
29.00
69.58
31.03
20.25
31.82
15.20
34.55
Distill, no reg, denoise
61.16
33.51
34.51
81.51
36.70
23.96
35.74
18.34
40.68
Distill, with reg, no denoise
61.27
33.52
34.39
81.52
36.74
23.92
35.55
18.38
40.66
Distill, no reg, with denoise
62.48
34.28
35.00
82.27
37.62
24.46
36.83
18.92
41.48
Full Pipeline
63.01
34.52
35.27
83.05
37.88
24.66
37.17
19.22
41.85
student model is distilled with augmentations, where one of them is always an unmodified image. As shown in Figure 5b, high convergence threshold is observed at the 99th percentile where even the most dissimilar cases exhibit cosine similarity values above 0.95, indicating a substantial reduction in feature space outliers. As result, employ 10 augmentations generate high-quality features efficiently, thereby reducing computational overhead.
Distillation Approaches. We evaluate the contribution of different components in our distillation approach by conducting open- upen-vocabulary semantic segmentation task on benchmarks. In this setting, the student model is distilled by sequentially removing each component from our approach. These components include denoising process with augmentations, 16 additional register tokens, and self-distillation As shown in Table 3, each component contributes to improving the student model's performance. By comparing our full pipeline, where student model is distilled with denoising process and registers, find that approximately half of the improvement comes from registers, and the other half results from denoising process applied to the teacher model.
The Ratio of Shifting in the Denoising Process. To investigate the impact of different shifting ratios in our denoising process, we conduct open- vocabulary semantic segmentation task on 8 benchmarks. We gradually increase the shifting ratio from 0% to 30%, while fixing the number of augmentation at 10. As shown Table shifting ratio of 10% demonstrates the best performance across datasets, while a shifting ratio of 15% achieves the best performance across 3 datasets and provides optimal average performance across all datasets. Therefore, we adopt a shifting raio of 15% denoising process when applied to the teacher model.

---

Table 4: Effect of Shifting Ratio. This table reports the impact of different shifting ratios used in the denoising process, as defined in 3.1. Throughout this experiment, we use OpenAI CLIP ViT-B/16 as the backbone for the model and apply 10 augmentations in the denosing process.
Ratio
VOC21
PC60
Object
VOC20
PC59
Stuff
City
ADE
Avg.
0%
58.88
32.20
33.15
79.70
35.16
23.30
35.48
17.42
39.41
10%
60.49
32.91
33.73
80.47
35.94
23.86
36.60
17.94
40.24
15%
60.47
32.89
33.61
80.36
35,94
23.89
36.87
18.10
40.27
20%
60.39
32.82
33.46
79.93
35.85
23.86
36.98
18.07
40.17
25%
60.26
32.75
33.26
79.75
35.75
23.78
37.02
18.00
40.07
30%
60.14
32.76
33.25
79.39
35.77
23.77
37.07
18.10
40.03
12
16
200
11:
175
Nomm
10
14
150
125
12
100
10
75
Original OpenAl CLIP w/H--Re Original OpenCLIP w/PH-Reg
Original DINOV2
w/P H-Reg
Figure 7: Patch Norms. This figure illustrate norms of patch tokens of different backbones. Our method effectively reduces the variance of token norms and reduces the outliers, regardless if the artifacts are lower/higher norm.
4.4 Investigation of Artifacts and Registers
While prior work has noted that artifact tokens are high magnitude DINOv2 [50], in Figure 6 we find that this is not always the case. In OpenAl CLIP and OpenCLIP, the artifacts are generally lower norm than their surrounding patches. In contrast, DFN-CLIP and DINOv2, the artifacts are higher norm. This illustrates that there may be elements of the training dynamic at play, as the artifact norms can differ even when the training objective is very similar. In Figure we visualize the norms of the original network and those with registers added, find that our method effectively reduces the variance of the patch norms.
5 Discussion
Limitations and Future Work In this work we proposed PH-Reg, a method to reduce the artifact tokens existing pre-trained vision transformers. We show that PH-Reg can eliminate artifact tokens in ViT's effectively and generate clean dense feature maps, enhancing the performance in downstream dense prediction tasks. This approach relies on test time augmentation denoise dense feature presentations in the teacher model. While our method generally outperforms DVT on CLIP based models, we sometimes underperform when using the DINOv2 backbone. We believe this due to the static artifact estimator present in DVT. The assumption of static artifacts holds true for some models (DINOv2), but not others (CLIP). potential avenue for additional investigation is how to dynamically determine the artifacts without strong stationary assumptions.
Conclusion We introduce a novel post-training method PH-Reg, for learning clean dense feature representations ViTs through an efficient self-distillation framework that does not require additional labeled data. Our approach leverages test-time augmentation to denoise the teacher model, and guide the student model to optimize the dense feature representations. This enables us to eliminate artifact tokens effectively by integrating learnalle registers into existing pretrained models, without the need for training from scratch. We demonstrate that the distilled ViTs generate fine-grained dense feature maps, enhancing the consistency of feature representations in ViTs. We further show that cleaner dense feature maps ViTs leads to quantifiable improvements on dense prediction tasks. Finally, we illustrate that the distilled ViT's can accurately capture meaningful semantic structures in images, as shown by heatmaps generated from CLIP text queries. We validate our conclusions with extensive evaluations across multiple dense prediction benchmarks.
10

---

References
[1] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz, Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference machine learning, pages 4055-4064, PMLR, 2018.
[2] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904. 2019.
[3] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315-11325 2022.
[4] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. Advances in neural information processing systems, 32, 2019.
[5] Jean-Baph -aanttap Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. arXiv preprint arXiv:1911.03335. 2019.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / 1929, 2020.
[7] Tobias Christian Nauen, Sebastian Palacio, Federico Raue, and Andreas Dengel. Which transformer to favor: A comparative analysis of efficiency in vision transformers. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages IEEE, 2025.
[8] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12894-12904, 2021.
[9] Nya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34: 24261-24272, 2021.
[10] Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08602, 2021.
[11] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. International conference on machine learning, pages 10183-10192. PMLR, 2021.
[12] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Go- ing deeper with image transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 32 22-42,22 2021.
[13] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning. volume 139, pages 10347-10357, July 2021.
[14] Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International conference on machine learning, pages 9438-9447. PMLR, 2020.
[15] Krzysztof Choromanski, Valeri Likhosherstov, David Dohan, Xingyou Song, Andreca Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[16] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
11

---

[17] Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross- covariance image transformers. Advances in neural information processing systems, 34:20014- 20027, 2021.
[18] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.
[19] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.
[20] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov, A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 10809-1088 10818, 2022.
[21] Ting Yao, Yingwei Pan, Yehao Li, Chong- Chong-Wahha Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. In European conference on computer vision, pages
Springer, 2022.
[22] Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202. 07800, 2022.
[23] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pages 22-31, 2021.
[24] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi- Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, pages 558-567, 2021.
[25] Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The friendl transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 589-598, 2021.
[26] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi- scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 357-366, 2021.
[27] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International conference on machine learning, pages 1243- 1252. PMLR, 2017.
[28] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568:127063, 2024.
[29] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149: 105171, 2024.
[30] Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.123376 2024.
[31] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. Proceedings of the IEEE/CVF Conference on Computer 2024.
Vision and Pattern Recognition, pages
[32] Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for open- vocabulary object detection with vision transformers. In Proceedings of the IEEEF/CV confer- ence on computer vision and pattern recognition, pages 1114411155, 2023.
12

---

[33] Jiawei Yang, Katie Z Luo, Jiefeng Li, Congyue Deng, Leonidas Guibas, Dilip Krishnan, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denoising vision transformers. In European Conference on Computer Vision, pages 453-469 Springer, 2024.
[34] Alec Radford, Jong Wook Kim. Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[35] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaha Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL http:///pppeeeeeeeee/eeeeee/e//e Ee277P3AYC.
[36] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self- vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[37] Xinlei Chen Saining Xie*, and Kaiming He. An empirical study of training self- vision transformers. arXiv preprint WiXiv:2104.02057 2021.
[38] Chunyuan Li, Jianwei Yang. Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-super vision transformers for representation learning. International Conference on Learning Representations (ICLR), 2022.
[39] Trieu H. Trinh, Minh-Thang Luong, and Quoc V. Le. Selfie: Self-supervised pretraining for image embedding, 2019, URL https: //arxivv////iiv/////////a httg//b//19900 02940.
[40] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 1691-1703, PMLR, 13-18 Jul 2020. URL https: pettps://procee /ttp///ppoceeen./eee/ee/e/e/s.ee mlr. www..rresssvlee v1/1/111////// 11/1199//ae
chwne0senn.ccmeeeeeeee chhh///220ss html.
[41] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[42] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377 2021.
[43] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022.
[44] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. International Conference on Learning Representations, 2022. URL https: //openreview. net/forumminiiee ddpp- BhZS25904.
[45] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint- embedding predictive architecture. arXiv preprint arXiv:2301 082443. 2023.
[46] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[47] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference Computer Vision, pages 696-712. Springer, 2022.
[48] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. ECCVW What is Motion For?, 2022.
13

---

[49] Mike Ranzinger. Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomera- tive vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12490-12500, June 2024.
[50] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprini urXiv:2309. 16588, 2023.
[51] Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006. / 1527, 2020.
[52] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint 23ti::2009//455. 2023.
[53] Nick Jiang, Amil Dravid, Alexei Efros, and Yossi Gandelsman. Vision transformers don't need trained registers. arXiv preprint arXiv:2506.08010, 2025.
[54] Haoqi Wang, Tong Zhang, and Mathieu Salzmann. Sinder: Repairing the singular defects of dinov2. In European Conference on Computer Vision, pages 20-35. Springer, 2024.
[55] Alexander Lappe and Martin A. Giese. Register and cls tokens yield a decoupling of local and global features in large vits, 2025. URL https httpt::tt///attt //axxiv.../ii/////a//////ar org/ abb//505....... 05892.
[56) Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al. Emernerf: Emergent spatial-temporal scene decomposition via sclf-supervision. arXiv preprint arXiv:2311.02077 2023.
[57] Andrew F Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M Henderson, Leila Wehbe, and Michael Tarr. Brain mapping with dense features: Grounding cortical semantic selectivity in natural images with vision transformers. arXiv preprint arXiv:2410.08. 05.65666, 2024.
[58] Yi Li, Hualiang Wang, Yiqun Duan, Jiheng Zhang, and Xiaomeng Li. A closer look the explainability of contrastive language-image pre-training. Puttern Recognition, 162: 1409, 2025. ISSN 0031-3203. doi: /ttp://cwc//aattaao////////////p
1409. URL https:
//www sci www.ssciencediree.. com/
[59] Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision- language inference, 2024. URL https://
abs 2312. 01597.
[60] Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding everything: Emerging localization properties in dision-languag transformers, 2023. URL https: //axxiv. org/abs/22 22tb.////3a...e 231220078..
[61] Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense vision-language inference. In European Conference on Computer Vision, pages 143-160. Springer, 2024.
[62] Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2025.
[63] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 1 11::981136 2015.
[64] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam- Cho, Scong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 891-898, 2014.
[65] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6- 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
14

---

[66] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 1209-1218, 8, 2018.
[67] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3213-3223, 2016.
[68] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127:302-321, 2019.
[69] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746-760. Springer, 2012.
[70] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.
[71] Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. / 77225, 2023.
[72] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4008-4011 2021, doi: 10.1009/CVRR4443722011.
[73] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1.54 00325, 2015.
15

---

A Technical Appendices and Supplementary Material
Sections
1. Implementation Details for Self-Distillation (section B)
2. Implementation Details for Quantitative Evaluation (section C)
3. Additional Efficiency Analysis of PH-Reg Compared to DVT (section D)
4. Additional Qualitative Examples for Segmentation (section E)
5. Additional Qualitative Heatmaps for PH-Reg Zero-Shot (section F)
6. Proof of Test Time Augmentation (section G)
16

---

B Implementation Details for Self-Distilattio
In this section, we provide detailed overview of how implement self-distillation in PH-Reg. Our self-distillation framework consists of one teacher model and one student model. While both the teacher and student model are initialized from the same weights, the teacher is frozen, while additional register parameters are added to the student network.
Our codes and weights are avaible in http://giilbbcoom//rric////PRRe
B.1 Model Architectures
Teacher Model Architecture. For CLIP based models, since we focus on zero-shot open open-vocabulary segmentation, utilize the NACLIP modification to the final layer. This modification does not introduce any additional weights to the teacher network, and is training-free. Our empirical analysis in Section B.4 shows that NACLIP's neighborhood attention mechanism improves feature consistency. For DINOv2, we directly use the final output layer, without any modification to the teacher network.
Student Model Architecture. Based the results from the ablation studies integrate 16 register tokens into the student model. For the CLIP based student, to ensure representational alignment we directly take the head from the output layer (the MaskCLIP output). For DINO based students, do not apply such modifications. We bicubicly upsample the positional embedding so matches the input image. Unless otherwise specified, this modification is applied consistently, while all other layers remain unchanged.
B.2 Model Implementation
Table S.1: Model Implementation Libraries and Weights. We compare models trained using different datasets and objectives.
Model
Library
Weight
CLIP
clip (OpenAI)
ViT-B-16
OpenCLIP
open_clip
DFN-CLIP
open_clip
jf-nubappeeDDNNBB-CCPPPVVVBLL facebook/dinov2--base
DINOv2
transformers (Hugging Face)
We provide the model weights and the corresponding implementation libraries in Table S.I.
B.3 Optimization
In the distillation process, the shorter side of each input image is resized using bicubic interpolation to 448 for CLIP-based models and 518 for DINOv2. The resized image is then randomly cropped into square of size (448, 448) or (518, 518), respectively. For each input image, we generate N 10 augmentations using random shifts and horizontal flips. Assuming an image length 1, we uniformly sample the shift for both the horizontal and vertical axes from [-0.15, 0.15]. While the horizontal flip sampled with probability 0.5. To ensure each patch covered, we do not apply any augmentation to the first image of the 10. All shifted images are concatenated and fed into the teacher model, while the original (unshifted) images are used as input to the student model. The target feature is computed as the average of these 10 augmentations. To accommodate the resized input images for both the teacher and student models, we consistently resize the positional embeddings using bicubic interpolation. During training, the weights of the teacher model are frozen. In the student model, we allow updates to registers, the positional embeddings, the convolutional patch embedding layer, and the final transformer layer containing the self-attention mechanism.
The distillation framework is implemented in PyTorch, with distributed training managed via PyTorch Accelerate. Training is conducted on 4 NVIDIA Ada 6000 GPUs, with mixed-precision optimization to balance computational efficiency and numerical stability. Detailed training configurations are provided in Table S.2 and Table S.3.
17

---

Table S.2: Configs for CLIP-basee models.
Table S.3: Configs for DINOv2.
Config
Value
Config
Value
optimizer
AdamW
optimizer
AdamW
initial learning rate
3e-4
initial learning rate
lc-4
final learning rate
Ie-5
final learning rale
Se-6
weight decay
le-2
weight decay
le-2
optimizer momentum (51, (a)
(0.9, 0.999)
optimizer momentum (51, (a))
(0.9, 0.999)
learning rate scheduler batch size training epochs
Exponential Scheduler
learning rate scheduler batch size training epochs
Exponential Scheduler
16
8
100
100
augmentation
RandomSquareCrep
augmentation
RandomSquareCrop
B.4
Pearson Analysis of Open-vocabulary Segmentation
Table S.4: Open-vocabulary Semantic Segmentation Quantitative Comparison 7 datasets. We report the Pearson correlation coefficient for the zero-shot query against the one-hot ground truth labels. The results averaged within each image, then averaged across images. Compared mloU, pearson does not require knowledge of all of the categories present an image (via softmax). The value ranges from -1 where 1 perfect positive correlation, perfect negative correlation, and 0 no linear correlation. The best result for each dataset is highlighted in bolded.
Method
VOC21
PC60
VOC20
PC59
Stuff
City
ADE20k
Avg.
SCLIP
-0.005
0.349
0.409
0.443
0.323
0.291
0.308
0.303
ClearCLIP
0.012
0.428
0.489
0.543
0.393
0.336
0.418
0.374
NACLIP
0.011
0.422
0.470
0.543
0.392
0.363
0.425
0.375
NACLIP+DVT
0.003
0.438
0.487
0.551
0.395
0.367
0.427
0.381
Ours (PH-Reg)
0.013
0.468
0.494
0.590
0.424
0.381
0.461
0.404
In this section present additional evaluation results open-vocabulary semantic segmentation via the pearson metric. Results are illustrated Table S.4. Overall, PH-Reg CLIP significantly outperforms the baseline models on datasets. Even in the absence of prior category knowledge, PH-Reg CLIP achieves an average performance of 0.404, representing a clear improvement over the second-best method, DVT enhanced NACLIP, with an average performance of 0.381. These results highlight that our approach improves the consistency of dense feature representations by reducing artifact tokens, thereby offering robust and generalizable enhancement over existing methods.
We further observe that both ClearCLIP and NACLIP achieve competitive results; however, NACLIP significantly outperforms ClearCLIP on ADE20K and Cityscapes. The former requires the model to handle a large number of categories, while the latter demands fine-grained localization of small objects. Based this observation, choose NACLIP our primary teacher model, leveraging its neighbor attention mechanism to enhance the student model's 's performance on these challenging tasks.
18

---

Table S.5: Dataset Specific Details for Open-vocabulary Semantic Segmentation. We list the per-dataset resolution, crop size, and stride used for each dataset. We maintain the same settings for all methods within given dataset.
Dataset
VOC21
PC 60
Object
VOC20
PC 59
Stuff
City
ADE
Resize resolution
448
448
336
336
448
448
560
448
Crop size Stride
336
336
336
336
336
336
224
336
112
112
112
112
112
112
112
112
Table S.6: Training Time Efficiency Analysis. We report the training time of our PH-Reg method compared to DVT. For fairness, restrict all stages of our model to single GPU and evaluate the same model (DINOv2 ViT-B).
Method
Stage
Extraction
Stage Distillation
Stage 2 Training
Total
DVT
2998 min
18340 min
570 min
21908 min (365.1 h)
PH-Reg
9000 min (150 h)
C
Implementation Details for Quantitative Evaluation
In this section, provide detailed implementation information for our quantitative evaluation experiments. In section C.I, we present the evaluation details for open-vocabulary semantic segmen- tation (OVSS). In section C.2, we describe the evaluation details for linear probe based semantic segmentation and monocular depth estimation.
C.1 Implementation Details of Open-vocabulary Semantic Segmentation.
We follow SCLIP and NACLIP the setup for the open-vocabulary semantic segmentation evaluation. For fairness, utilize the same parameters for all models. We resize input images such that the shorter side is scaled to specific resolution, while maintaining the original aspect ratio for the longer side. Additionally, set fixed crop sizes and strides during evaluation. All evaluation parameters are summarized in Table S.5, while all other settings follow their default configurations.
C.2 Implementation Details of Linear Probe Based Evalution.
Our linear probe evaluation follows prior works (Vision Transformers Need Registers, Denoising Vision Transformers), where linear layer is trained decoding head to predict pixel-wise segmentation or depth logits.
Semantic Segmentation. We extract the final output features from the frozen backbone and, if applicable, pass them through the denoiser (for the DVT baseline). A single learnable linear layer then trained to predict the segmentation logits. For CLIP-based models, both training and testing images are resized to (448, 448), while for DINOv2, the images are resized to (518, 518).
Monocular Depth Estimation. Similar to semantic segmentation, we extract features from the backbone, and pass them through the denoiser if applicable. Following the method in DVT and DINOv2, we then append the [CLS] token to each patch token to enrich the feature representations for all methods. linear layer trained using SigLoss and gradient loss (scaled by a factor of 0.5) to predict depth values into 256 uniformly distributed bins. We adopt DVT's learning rate of 5e-3 for all experiments.
D Additional Efficiency Analysis of PH-Reg Compared to DVT
In this section, we provide a detailed efficiency analysis of PH-Reg compared to DVT, focusing on three aspects: training time, space usage, and inference cost.
Training Time. When evaluating training time, we utilize the official code provided in the DVT repository without any modification. The original DVT code specifies single GPU for feature
19

---

Table S.7: Inference Efficiency Analysis. We report the inference results for all models, evaluating efficiency based on model size and GFLOPs.
Method
GFLOPs
Params (M)
MaskCLIP
62.89
86.19
NACLIP
64.76
86.19
NACLIP w/ denoising (10x)
647.6
86.19
NACLIP DVT
70.32
94.07
CLIP + PH-Reg (Ours)
64.16
86.66
extraction and learning - for fairness we also limit all stages our own model to a single GPU here. For DVT and PH-Reg. evaluate the same model (DINOv2 ViT-B). Results illustrated in Table S.6. Our method has the advantage of not utilizing gradient- based neural field learning as done in DVT. Therefore, time compared to DVT.
method trains the student model in single stage, saving over 58.9% of the
Space Usage. A further advantage is the space utilization during training. DVT requires saving 1.4 terabytes of intermediate neural fields in the form of serialized Instant-NGP files. Our method computes all distillation targets on the fly, and requires additional space.
Inference Cost. When evaluating testing cost for all models, we cast parameters fp32 dtype. and use eager attention implementation for all models. For our method, we include the positional embeddings adapted for 448 resolution. For MaskCLIP and NACLIP, current official implementations have the same number of parameters as the original CLIP, although small reduction can be achieve in MaskCLIP by discarding the last q. k head. For DVT, we evaluate their stage 2 model, corresponding to the transformer block denoiser coupled to original vision transformer. Results are illustrated in Table S.7. Our method utilizes approximately 10% fewer FLOPs and 10% fewer parameters during inference compared to DVT.
20

---

E
Additional Qualitative Examples
Image
CLIP
MaskCLIP
SCIP
ClearCLIP
NACUP
Ours (Pit-Reg)
Ground Truth
Figure S.1: Open-vocabulary semantic segmantation qualitative comparision between different baseline models on ADE20K.
Image
CUP
MaskCLIP
SCLIP
ClearCLIP
NACUP
Ours (PH-Reg)
Ground Truth
Figure S.2: Open-vocabulary semantic segmantation qualitative comparision between different baseline models on Pascal Context59.
21

---

Image
CLIP
MaskcLIP
SCLIP
ClearCLIP
NACUP
Ours (PH-Rag)
Ground Truth
Figure S.3: Open-vocabulary semantic segmantation qualitative comparision between different
baseline models on COCO Obejct.
22

---

Additional Qualitative Heatmaps for PH-Reg Zero-Shot
Image
UMAP
Cat
Hand
Stone
MaskCLIP
SCLIP
NACLIP
Ours
Figure S.4: Zero-shot Heatmap Results. Our results have fewer artifacts than other methods.
Image
UMAP
Sofa
Table
Window
MaskCLIP
SCLIP
NACLIP
Ours
Figure S.5: Zero-shot Heatmap Results. Our results have fewer artifacts than other methods.
23

---

Image
UMAP
Fork
Steak
MaskCLIP
+|s
SCLIP
NACLIP
Ours
Figure S.6: Zero-shot Heatmap Results. Our results have fewer artifacts than other methods.
Image
UMAP
Path
Tractor
MaskCLIP
SCLIP
NACLIP
Ours
Figure S.7: Zero-shot Heatmap Results. Our results have fewer artifacts than other methods.
24

---

G Optimal Feature Aggregation
Let f1...., fa E Rd be feature vectors of a single patch from different transformations of input image I. We seek the optimal aggregated feature f that minimizes the total squared error:
f" arg min
1lf -0--1 fl2
(1)
f
i=1
Expanding the objective gives us:
arg min
Σ
(2)
Dropping constant terms that do not change the optimum:
nf f
(3)
Dividing and multiplying the right side by n
=ff 2n
(4)
Dividing the equation by n whole shows that we need to minimize:
f
(5)
So can be derived that the mean of the feature vectors is the minimizer under MSE loss:
(6)
12
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We justify each claim with extensive experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have a limitations section and discuss potential future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
This paper does not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide extensive details on our ablation studies. Additional details can be found in the supplemental.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification: Code will be included in the supplemental.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaeccccccccccccccat / ssssssolicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: A high-level outline is included in the main paper. Implementation details will be incldued in the supplemental.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Due to cost of running experiments, we only perform one run.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see the supplemental for computational details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp://neeisspe http://rrrrpp..cpppiii/.//////// cttpppplllttttccccceeeeeeeeeeeee ctt.iieisseillenneccc ines?
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
This is discussed in the supplemental.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper has no risk related to safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets that we public, and properly cited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
30

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code will be included in the supplemental
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not collect any human data.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not collect any human data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
31

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy thttps://a 220 for what should or should not be described.
32


Reviews:
[{"review_id": "diIfWpng0u", "reviewer": "Reviewer_H7Es", "summary": "The paper introduces Post Hoc Registers (PH-Reg), an efficient self-distillation framework to mitigate artifact tokens in pre-trained ViTs without full retraining or labeled data. By initializing teacher and student networks from the same pre-trained ViT, freezing the teacher, and augmenting its inputs(e.g.. random offsets, flips), PH-Reg generates denoised dense embeddings to guide student optimization, only updating a small subset of weights including randomly initialized register tokens. This approach effectively reduces artifact tokens, improving zero-shot and linear-probe performance in segmentation and depth prediction.Key contributions include an efficient test-time augmentation denoising scheme, selective finetuning of student components for clean feature learning, and demonstrated enhancements in dense representation consistency for fine-grained spatial tasks, as validated across multiple benchmarks.", "questions": "1.\tThe denoise step needs to ablate the shift ratio. And i'm wondering why not vertical fllp? \n\n2.\tIt seems all the experiments done with salient object. What about the results on more complex scenarios like coco and lvis?\n\nSee weaknesses for other questions.\n\nIf the author can address the above issues and provide a clearer theoretical analysis to demonstrate the effectiveness and robustness of the proposed method, I would be willing to raise the score.", "limitations": "It would be better to discuss the failure cases of Dino to analyze the potential shortcomings.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tPH-Reg introduces a novel self-distillation framework that integrates register tokens into pre-trained ViTs without full retraining, leveraging test-time augmentation to denoise dense features as the teacher.This approach avoids the computational cost of training from scratch while effectively reducing artifact tokens that degrade fine-grained localization tasks.\n\n2.\tPH-Reg demonstrates consistent improvements across diverse dense prediction tasks (e.g., semantic segmentation, depth estimation) under zero-shot and linear probing setups, outperforming state-of- the-art baselines on multiple benchmarks.\n\nWeaknesses:\n\n1.\tDenoising dense predictions in a self-supervised manner is a common strategy in image processing works. The paper simply adopts the idea and finds that it also helps with the denoise at feature level. However, it lacks formal theoretical justification for the effectiveness of register tokens and test-time augmentation in artifact mitigation, focusing primarily on empirical validation.\n\n2.\tWhile PH-Reg excels on CLIP-based models, it underperforms DVT on DINOv2 due to DVT's static artifact estimator, indicating sensitivity to model-specific artifact characteristics.\n\n3.\tThe denoising efficacy relies on carefully hand-designed augmentations (e.g,random offsets), which may require task-specific tuning for optimal results. It's also needed to ablate the shift ratio impact.", "was_revised": true, "final_justification": "The author's answers have addressed most of my concerns. Meanwhile, considering other reviewers' comments, I lean to accept the paper.", "rebuttal": "We appreciate your excellent suggestions! Below are our responses to specific questions. We look forward to further discussion, and welcome additional comments.\n\n> **Q1) Why not vertical flips as augmentation?** \n\nWe avoid vertical flips in our augmentation pipeline because this violates real-world physical constraints. Gravity imposes a consistent vertical orientation in natural scenes (e.g., sky above ground, human faces upright), making vertically flipped images unrealistic. In contrast, horizontal flips preserve vertical orientation while simulating naturally occurring viewpoint variations (e.g., an object facing left/right). Augmentations like horizontal flips and shifts expand the training distribution with plausible variations.\n\nThe mmseg package [1] commonly used for dense vision evaluation, similarly defaults to `horizontal` for `RandomFlip`. Generally other works in computer vision also avoid using vertical flips as augmentation. For example, the ResNet paper [2] and DeiT III [3] training recipe (utilized by DINOv2 for ImageNet) adopt horizontal flips without vertical flipping in their augmentation pipeline. \n\nWe will update our revision to include a paragraph clarifying why we do not utilize vertical flips in our augmentation function.\n\n[1] https://mmcv.readthedocs.io/en/2.x/api/generated/mmcv.transforms.RandomFlip.html\n\n[2] `Deep Residual Learning for Image Recognition (CVPR 2016)`\n\n[3] `DeiT III: Revenge of the ViT (ECCV 2022)`\n\n> **Q2) Ablation over shift ratios**\n\nThis is a great question. The optimal shifting ratio of 15% was determined using extensive experiments. Here we provide a table where additional shift ratios are explored using NACLIP teacher features (note that only the student network has registers).\n\nWe find that a shifting ratio of 15% provides the optimal performance (mIoU, higher is better):\n\n| ratio |   VOC21   | Context60 | COCO object |   VOC20   | Context59 | COCO stuff | Cityscape |  ADE20k  |    Avg    |\n|-------|:---------:|:---------:|:-----------:|:---------:|:---------:|:----------:|:---------:|:--------:|:---------:|\n| 0%    |   58.88   |    32.20   |    33.15    |    79.70   |   35.16   |    23.30    |   35.48   |   17.42  |   39.41   |\n| 10%   | **60.49** | **32.91** |  **33.73**  | **80.47** | **35.94** |    23.86   |    36.60   |   17.94  |   40.24   |\n| 15%   |   60.47   |   32.89   |    33.61    |   80.36   | **35.94** |  **23.89** |   36.87   | **18.10** | **40.27** |\n| 20%   |   60.39   |   32.82   |    33.46    |   79.93   |   35.85   |    23.86   |   36.98   |   18.07  |   40.17   |\n| 25%   |   60.26   |   32.75   |    33.26    |   79.75   |   35.75   |    23.78   |   37.02   |    18.00    |   40.07   |\n| 30%   |   60.14   |   32.76   |    33.25    |   79.39   |   35.77   |    23.77   | **37.07** | **18.10**   |   40.03   |\n\n***We will add this table to the paper in an upcoming revision.***\n\n> **Q3) Number of registers**\n\nWe find that the rate of improvement decreases as additional registers are added. This trend is observed over all models. We provide additional results here (median cosine similarity, higher is better), and will include these results in an upcoming revision:\n\n| Number of Registers  | 1      | 2      | 4      | 8      | 16     |\n|----------|--------|--------|--------|--------|--------|\n| OpenCLIP | 0.7852 | 0.7858 | 0.7985 | 0.8025 | **0.8139**|\n| DFNCLIP  | 0.7553 | 0.7581 | 0.7687 | 0.7751 | **0.7854** |\n| DINOv2   | 0.9265 | 0.9260 | 0.9327 | 0.9346 | **0.9350** |\n\n> **Q4) Proof of convergence**\n\nWe would like to clarify, that a key advantage of our method over DVT is that we avoid expensive gradient-based neural field optimization. On `page 12` we include a proof showing that we obtain the same optimal result in expectation as the neural field stage of DVT. \n\nLet $f_1, ..., f_n \\in R^d$ be feature vectors of a single patch from $n$ transformations of an input image. DVT optimizes a neural field using MSE loss gradients, while we take the unweighted mean. We summarize the proof here:\n\nLet $f^{*} = \\text{argmin}_f \\sum_i^n ||f_i-f||^2_2$\n\n`Expanding the objective gives us`: $\\text{argmin}_f \\sum_i^n (f_i^{T}f_i - 2f_i^{T}f + f^{T}f)$\n\n`Dropping constant terms that do not change the optimum`: $n f^{T}f - 2(\\sum_i^nf_{i}^{T})f$\n\n`Dividing and multiplying the right side by n`: $n f^{T}f - 2n(\\sum_i^n\\frac{1}{n}f_{i}^{T})f$\n\n`Dividing the equation by constant n shows that we obtain the same optimum as DVT`: $||f - \\frac{1}{n}\\sum_i^nf_i||^2_2$ so $f^* = \\frac{1}{n}\\sum_i^nf_i$\n\nDue to this, our unweighted mean yields the same optimum as DVT. We will highlight this proof in the upcoming revision.\n\n> **Q5) Evaluation of non-salient objects**\n\nWe agree that the main paper could benefit from additional visualizations. To clarify, `Figure 4` on page 5 of the main paper provides examples of open-vocabulary queries using the CLIP text branch for ***different objects within the same image***. \n\nWe indeed evaluate our model on COCO benchmarks in `Table 1` on Page 5 of the main paper (COCO-Objects is listed as `Object`, COCO-Stuff is listed as `Stuff`). We also evaluate on other widely used segmentation benchmarks like PASCAL VOC, PASCAL Context, CityScape, and ADE20k. We reproduce the results for COCO here (mIoU, higher is better), and will update the naming in an upcoming revision: \n\n| Dataset     | CLIP | MaskCLIP | SCLIP | ClearCLIP | NACLIP | MaskCLIP + DVT | NACLIP + DVT | Ours (PH-Reg) |\n|-------------|:----:|:--------:|:-----:|:---------:|:------:|:--------------:|:------------:|:-------------:|\n| COCO Object | 6.50 |   26.94  | 33.52 |   32.77   |  33.15 |      20.89     |     32.89    |   **35.27**   |\n| COCO Stuff  | 7.19 |   18.8   | 22.65 |   23.89   |  23.30 |      17.10     |     23.41    |   **24.66**   |\n\nAdditional segmentation visualization results are provided in `Figure 1` to `Figure 3` of the **supplemental PDF**, with heatmap visualizations of multiple objects within the same image provided from `Figure 4` to `Figure 7` of the supplemental. \n\nOur results demonstrate PH-Reg yields significant qualitative improvements in non-salient object localization.\n\nWe appreciate this suggestion. We will update the revision to include additional visualizations in the main paper.\n\n> **Q6) DINOv2 performance compared to DVT**\n\nWe would clarify that our PH-Reg method applied to DINOv2 ***obtains consistent performance improvements*** compared the unmodified DINOv2 as shown in `Table 2` on page 7 of the main paper. We also outperform the DVT denoiser on depth prediction tasks. Our method also has the advantage of not utilizing gradient-based neural field learning as done in DVT, offering ***significant training time and space usage*** improvements compared to DVT. \n\nHere we provide training time evaluations on a single GPU for DVT (***single GPU*** is specified in DVT's official code for all steps of feature extraction and neural field learning) and for fairness we also use a ***single GPU*** for our method's training. We follow all their official settings and use identical resolution and measure training time for the same ViT model. \n\n| Method | Stage 1 Extraction | Stage 1 Distillation | Stage 2 Training  | Total               |\n|--------|--------------------|----------------------|-------------------|---------------------|\n| DVT    | 2998 min           | 18340 min            |  570 min            | 21908 min (365.1 h) |\n| PH-REg | -                  | -                    |        -           | 9000 min (150 h)    |\n\nIn practice, we parallelize training across 4 GPUs, and complete PH-Reg in roughly 1.5 days. \n\nA further advantage is the ***space utilization*** during training. DVT requires saving 1.4 terabytes of intermediate neural fields in the form of serialized Instant-NGP files. Our method computes all distillation targets on the fly, and requires no additional space.\n\nIn summary, our method is significantly faster and space efficient compared to DVT -- while holding an advantage in depth prediction. We agree that future work should investigate how to further boost the performance in semantic segmentation when using PH-Reg.\n\n\n> **Conclusion**\n\nWe thank the reviewer for the suggestions, and have added additional quantitative ablations over the shifting ratio and number of registers. Following your suggestion, we have also highlighted the optimality proof we previously included in the supplemental. We will also include additional visualizations in the revision showing the localization quality of different objects within the same image, and clarify the datasets that we evaluate on. \n\nWe're grateful for your clear and helpful comments. In light of our response and the positive evaluation from other reviewers, ***we hope you might view our work in a more favorable light***. Please feel free to let us know if you have additional questions!"}, {"review_id": "5wArmPBywX", "reviewer": "Reviewer_cHXN", "summary": "Recent research shows that big Vision Transformer models such as DINOv2 or OpenCLIP contain artifact tokens that encode undesired information. While this information could be relevant, it does not provide the local semantics that should represent the patch token, degrading the performance of the ViT on tasks that require fine-grained patch information. Authors propose Post Hoc Registers (PH-Reg), an elegant solution to the problem that, contrary to the current SoTA, does not require full re-training or additional labelled data. Main contributions can be summarised in their simple but effective self-distillation training and the artifact denoising strategy that enables the whole training by acting as a training target.", "questions": "From the weaknesses explained before:\n1. Can authors provide more in-depth analysis regarding the performance gain decay as the number of registers increases?\n2. Is the proposed method more efficient than DVT? By how much in terms of memory and time?\n3. Improvements seem to be higher on VLMs, can the authors provide more in-depth insights about this behaviour?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\nTechnically sound method that successfully addresses the identified problem.\n\nPH-Reg stands as a model \"corrector\", automatically improving any ViT-based SoTA vision model that suffers from explained artifacts.\n\nProvided quantitative and qualitative analysis clearly display the benefits of the proposed method, highlighting especially the qualitative results.\n\nI find the denoising process intriguing. While a corrected method would always be more efficient, the described denoising strategy already manages to remove most of the artifacts without any specific training.\n\nThe authors yield several insights about the artifacts that may help the rest of the community working in the same field.\n\nThe paper is well-written and correctly organised.\n\n\n\n**Weaknesses:**\n\nThe authors claim that performance gain greatly decays after increasing the number of registers past 4/8. I would like to see more analysis on this, does this behaviour depend on the pretrained model? If not, what are the insights that could justify this general behaviour?\n\nEfficiency analysis regarding required memory (which I assume is low) and time between DVT and proposed approach would be interesting and could further prove the superiority of PH-Reg if they end up being much more efficient than DVT.\n\nProposed method seems to be much better when applied to VLMs, more in-depth insights about this behaviour would be appreciated.\n\nBeing a SoTA model \"corrector\", I believe the proposed approach has a strong short-term applicability. However, Registers are being widely applied already, and, at some point, models won't require a \"corrector\" anymore, dramatically decreasing the applicability of the proposed work.", "was_revised": true, "final_justification": "The author's answers have addressed my concerns; therefore, I keep my rating and recommend the acceptance of this paper.", "rebuttal": "Thank you for your detailed suggestions! We will incorporate all of your feedback into our paper.\n\n\n> **Q1) Investigation of register counts across models**\n\nThis is an interesting question. In the original paper we evaluate the number of registers only on the OpenAI CLIP ViT model. Based on your suggestion, we further evaluate OpenCLIP, DFN-CLIP, and DINOv2 -- three models that were trained using different datasets (in the case of OpenCLIP and DFN-CLIP) and different objectives (DINOv2). In all cases we observe decreased rates of performance gain when additional registers are added, across median cosine similarity (higher is better):\n\n| Number of Registers |    1   |    2   |    4   |    8   |     16     |\n|---------------------|:------:|:------:|:------:|:------:|:----------:|\n| OpenCLIP            | 0.7852 | 0.7858 | 0.7985 | 0.8025 | **0.8139** |\n| DFNCLIP             | 0.7553 | 0.7581 | 0.7687 | 0.7751 | **0.7854** |\n| DINOv2              | 0.9265 | 0.9260 | 0.9327 | 0.9346 | **0.9350** |\n\nWe are not the first to observe this, as similar trends are observed in the saturation of dense task performance in the original \"Vision Transformers Need Registers\" [1] paper. Both our own visualizations (will be added to upcoming revision), and those in [1] suggest that registers learn to focus on semantic objects within the image. We speculate that the saturation in performance is due to the limited number of objects in common images. \n\nOne potential follow-up experiment could be based on synthetic datasets like CLEVR, where the number of objects within each image is known. These datasets could be used to investigate if the performance benefit of register tokens correlates with the average number objects across a dataset.\n\n**We will include the table for different models in an upcoming revision of our paper.**\n\n> **Q2) Efficiency of PH-Reg compared to DVT**\n\nWe agree that additional quantification regarding both training and inference cost would be useful. Due to the potential influence of custom kernels on both performance and model memory usage, here we evaluate model size, FLOPs, and wall clock training time. When evaluating training time, we utilize the official code provided in the DVT repository without any modification. The original DVT code specifies a single GPU for feature extraction and learning -- for fairness we also limit all stages of our own model to a single GPU here. For DVT and PH-Reg, we evaluate the same model (DINOv2 ViT-B) at the same image resolution.\n\nTraining wall-clock time: \n\n| Method | Stage 1 Extraction | Stage 1 Distillation | Stage 2 Training  | Total               |\n|--------|--------------------|----------------------|-------------------|---------------------|\n| DVT    | 2998 min           | 18340 min            |  570 min            | 21908 min (365.1 h) |\n| PH-REg | -                  | -                    |        -           | 9000 min (150 h)    |\n\nDuring training, DVT also requires storage of the serialized Instant-NGP features. Which **requires approximately 1.4TB**, while our method requires no additional storage cost during training.\n\nWhen evaluating testing cost for all models, we cast parameters to `fp32` dtype, and use `eager` attention implementation for all models. For our method, we include the positional embeddings adapted for 448 resolution. For MaskCLIP and NACLIP, current official implementations have the same number of parameters as the original CLIP, although a small reduction can be achieve in MaskCLIP by discarding the last q,k head. \n\nFor DVT, we evaluate their stage 2 model, corresponding to the transformer block denoiser coupled to a original vision transformer. \n\n| Method                                | GFLOPs | Params (M) |\n|---------------------------------------|:------:|:----------:|\n| MaskCLIP                              |  62.89 |    86.19   |\n| NACLIP                                |  64.76 |    86.19   |\n| NACLIP + test time augmentation (10x) |  647.6 |    86.19   |\n| NACLIP + DVT                            |  70.32 |    94.07   |\n| CLIP + PH-Reg (Ours w/ 16 registers)                  |  64.16 |    86.66   |\n\nOur method is significantly faster to train (2x speed), and utilizes significantly less storage during training compared to DVT (0 vs 1.4TB). Our method also utilizes approximately 10% fewer FLOPs and 10% fewer parameters during inference compared to DVT. **We will include these details in an upcoming revision of our paper.**\n\n> **Q3) Why so well on Vision Language Models?**\n\nIndeed, our performance gains are the most striking on models trained with an image-language contrastive loss, where OpenAI CLIP, OpenCLIP, and DFN-CLIP were trained on different datasets. While we observe performance improvements on DINOv2 compared to the version without PH-Reg, the improvements are not as striking. \n\nWe show in Figure 6, for CLIP family models the artifacts primarily manifest as lower magnitude tokens when utilized for open-vocabulary segmentation. While in DINOv2 the artifacts are high magnitude -- confirming the observation in \"Vision Transformers Need Registers\" [1]. \n\nWe are unsure what causes the difference in magnitudes of artifact tokens, and our offline discussions with DINOv2 authors suggests that they are also unsure why artifacts appear in DINOv2 but not DINOv1. One co-author of DINOv2 suggested that the increase in single-color background images (collected from online retail), and the decrease in natural photographs in the training dataset was a likely reason in the artifact tokens.\n\nRegardless of the cause, if artifact tokens are low magnitude, then their influence decreases after averaging. We believe this is the reason of the increased performance of PH-Reg in CLIP family models. For future work, it may make sense to decouple patch direction and magnitude prior to distillation.\n\nWe will include a discussion of this in the revision.\n\n`[1] Vision Transformers Need Registers (ICLR 2024)`\n\n> **Conclusion**\n\nWe thank you for providing detailed and thoughtful feedback. Following your suggestions, we have run an evaluation to measure the impact of register counts in different models, the training efficiency of our method versus DVT, and the inference computational cost. We hope our answers have helped clarify your questions, and look forward to any additional discussion."}, {"review_id": "uv4GtXaNUU", "reviewer": "Reviewer_MibD", "summary": "The paper continues the recent area of research that indicates the presence of artifacts in vision transformers, and the introduction of register tokens to remedy this. The initial idea introduced and trained register tokens in the main architecture and discarded them for the loss and gradient computation. This paper introduces an important modification from there --  self-distilled registers which can be added to a pretrained model, and leads to good results at minimal retraining cost.", "questions": "Please see the strengths and weaknesses section above.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper presents a strong idea that significantly reduces the training costs, while also solving the issue of artifacts using the same registers idea.\n2. The experiment set and ablations are quite detailed and cover a variety of tasks, including semantic segmentation, depth estimation. The ablations cover various scenarios, like varying the number of register tokens, unlocking various layers, as well as augmentation settings during test-time adaptation.\n\nWeaknesses and Questions:\n\n1. As the authors point out, the method currently lacks the ability to determine artifacts in a dynamic manner. While this would require further research, a small discussion on potential future directions would be helpful.\n2. The paper \"Vision Transformers Need Registers\" suggested that the artifact tokens are high norm while this paper suggests that this isnt always the case. Can this be inspected further to understand if this inconsistency is the result of small architectural modifications? If tried out already, do the authors find a model trained along with registers from scratch to be consistent to the mentioned paper? Another recent paper \"Vision Transformers Don't Need Trained Registers\" uses the high norm claim to build a method (this is purely a mention and does not affect my rating as it is concurrent work)\n3. The \"Vision Transformers Need Registers\" paper suggests that the artifact tokens tend to contain global rather than local context, typically class information, etc. Do the authors find this observation while distilling register tokens as well?", "was_revised": true, "final_justification": "The presented work makes a significant stride on the concept of artifacts in vision transformers and how to mitigate them. Based on my earlier review, as well as the rebuttal, I believe this work would be relevant for further research directions, and thus I will continue to maintain my score (Accept).", "rebuttal": "We are deeply grateful for your helpful comments! We will address specific questions below, and will include additional details in a revision.\n\n> **Q1) Dynamic determination of artifact tokens** \n\nThis is a very interesting idea. Our current framework only removes the artifact tokens, but identifying the artifact tokens dynamically could be a promising direction. In principle, a \"corrector\" network could  be learned by estimating the difference between the original \"raw\" dense features and denoised teacher features. This network could be conditioned on the original dense features and learn the artifacts. \n\nAfter identification, the artifacts could be either removed or shifted to a register using ideas from recent work.\n\n**We will include a discussion on this in the upcoming revision.**\n\n> **Q2) Magnitude of Artifact Tokens** \n\nWe were not previously aware of `Vision Transformers Don't Need Trained Registers`. After a careful reading, we agree it is highly relevant and will include a reference to this work in our revision. It is striking that artifact tokens could be removed using heuristics alone. \n\nWe would like to clarify that for CLIP models, we focus on the zero-shot open-vocabulary segmentation task. In this case, using the `value features` (or some modified convex combination of the value features across the image) of the last attention block are the preferred approach [1, 2, 3, 4]. \n\nAs we show in `Figure 6`, we find when value features (or convex combination of value features) from the last layer are examined for CLIP models, indeed the artifact tokens are primarily low magnitude, rather than high magnitude in OpenAI's CLIP and OpenCLIP. However in DFN-CLIP, which shares the same architecture and training objective as the OpenCLIP model we evaluate -- the artifact magnitudes are more mixed. The primary difference between DFN-CLIP and OpenCLIP are the training images. This suggests that training data may play an important role in the emergence of artifact tokens. \n\nDiscussions with some DINOv2 authors offline, also suggests data may be a factor. When discussing why artifacts appear in DINOv2 but not DINOv1, one DINOv2 author suggests that increase proportion of images with white backgrounds (collected from online retail) in the DINOv2 training set may be the cause, whereas DINOv1 was trained with a higher ratio of naturalistic photographs. While this is plausible, we do not believe it is a complete explanation.\n\nWe will include additional discussion and a citation to the `Don't Need Trained Registers` work in our paper revision.\n\n[1] `Extract Free Dense Labels from CLIP (ECCV 2022)`\n\n[2] `SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference (ECCV 2024)`\n\n[3] `ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference (ECCV 2024)`\n\n[4] `Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation (WACV 2025)`\n\n> **Q3) Global information in artifact and register tokens**\n\nIt is fascinating that there seems to be a diverse set of views on the nature of artifacts -- [5] argues they include global information, while [6] argues they are image independent, with very recent work by `Lappe and Giese` [7] finding that some dimensions of artifact tokens contain global information -- with results varying due to model size and layernorm. \n\nSimilarly, there are disagreements about what causes artifact tokens to occur. DVT suggests that the positional embeddings are primarily responsible for artifacts. However in `section 4.3`, we find that artifacts cannot be fully resolved by finetuning the positional embeddings alone.\n\nWe have performed a visualization of the register tokens learned using PH-Reg -- which we will include in an upcoming revision. We find that in our current setup, the attention maps for the registers do indeed correspond to semantic objects within the image. However qualitatively the effect seems weaker than the models trained from scratch. One potential reason is that we do not constrain the `[CLS]` token of the student to decode global image information, and instead focus on the quality of dense representations.\n\nWe will include this attention map visualization in the upcoming revision, and discuss why our results may differ from models trained with `[CLS]` token constraints.\n\n[5] `Vision Transformers Need Registers (ICLR 2024)`\n\n[6] `SINDER: Repairing the Singular Defects of DINOv2 (ECCV 2024)`\n\n[7] `Register and CLS tokens yield a decoupling of local and global features in large ViTs (Arxiv)`\n\n> **Conclusion**\n\nWe thank you for your comments, and we hope that this clarifies our results! We will update the paper to reflect your suggestions."}, {"review_id": "n4GrreMvvT", "reviewer": "Reviewer_EcZk", "summary": "This paper introduces an interesting approach for learning registers in Vision Transformers (ViTs), building upon recent findings that additional registers help remove artifacts captured by the class token. The authors propose a novel method to clean artifact-contaminated feature maps using image augmentation, then leverage knowledge distillation to learn additional registers from the cleaned feature maps without requiring full training from scratch. Experiments on multiple segmentation tasks demonstrate the efficiency of the proposed method.", "questions": "All baseline comparisons appear to use models fine-tuned from pre-trained CLIP. How does the proposed method compare against the original register-ViT trained from scratch?\n\nWould it be possible to include experiments comparing against models trained from scratch without registers? This would help isolate the contribution of the distillation approach versus the underlying register mechanism.\n\nGiven that data augmentation eliminates artifacts from feature maps during training, does this suggest that test-time data augmentation would also improve task performance? It would be valuable to design experiments that isolate and quantify the specific contribution of the data augmentation component. For instance, could the authors compare: (1) models trained with their full pipeline, (2) models trained with distillation but without the augmentation-based artifact cleaning, and (3) models that apply similar augmentation strategies at test time? Such ablations would help disentangle whether performance gains stem primarily from the cleaner training signal or from the augmentation strategy itself.", "limitations": "The limitation has been well discussed.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The motivation is well-founded and addresses a relevant problem in the recent popular finding on ViT with learnable registers. The paper seeks to improve upon recent advances in register tokens by developing an efficient approach to learn these registers without costly full training or fine-tuning procedures.\n\n- The proposed artifact elimination method using feature map augmentation is simple, novel, and efficient. The approach offers a practical solution to a known issue in ViT architectures.\n\n- The distillation-based learning framework is well-motivated and demonstrates strong empirical performance according to the experimental results, providing an effective alternative to more expensive training procedures.\n\n\n\nWeaknesses:\n\nI didn't find major weakness. Here I list some flaws that should further improve the paper. \n\n- The methodology section would benefit from more detailed mathematical formulation. The authors should include explicit equations describing how the final cleaned feature maps are computed.\n\n- Section 3.3 lacks sufficient detail regarding the distillation methodology. The paper should provide clearer justification for the chosen loss function and include ablation studies demonstrating why this particular loss formulation was selected over alternatives.\n\n- While computational efficiency is claimed as a key advantage, the paper lacks quantitative comparisons of training time, memory usage, and computational overhead between the proposed method and full training/fine-tuning approaches with registers.\n\nThere are also some questions that I would like to discuss with authors. Please refer the below section.", "was_revised": true, "final_justification": "Following the rebuttal, the authors have demonstrated a clear commitment to addressing the raised concerns and have outlined concrete plans for revision. \n\nBased on these improvements and the authors' commitment to implementing the promised changes, I have revised my evaluation score upward. I trust the authors will follow through with their proposed revisions in the final manuscript.", "rebuttal": "We appreciate your detailed and concrete suggestions! We will incorporate all of your feedback into our paper.\n\n> **Q1) Detailed method description**\n\nWe agree that the paper could be improved with additional details about our distillation procedure. We have already incorporated your suggestions into our internal revision. Due to NeurIPS OpenReview policy this year, we cannot upload a revision. \n\nWe provide a condensed pseudocode here, where we assume we are targeting a vision transformer with patch size $k \\times k$, an image $\\mathcal{I}$ which is an integer multiple of the patch size. Augmentation parameters $(\\theta_1, \\theta_2, ..., \\theta_n)$ for up to $n$ augmentations are generated randomly from a constrained distribution, and each augmentation consists of a binary horizontal flip parameter $\\text{flip}_i \\in \\{0,1\\}$, and a vertical/horizontal offset tuple $(x_i, y_i)$. We also have a tuple grid of 2D image coordinates $C$, which helps us index the locations in an image. An augmentation function takes as input $(\\mathcal{I}, C, \\theta_i)$ and generates the corresponding shifted/flipped image and coordinates.\n\nPseudo code:\n> Inputs: Image $\\mathcal{I}\\in\\mathbb{R}^{H\\times W \\times 3}$; Image space coordinates $C\\in[0,1]\\times[0,1]$; Augmentation parameters $\\theta_{1}, \\theta_2, ..., \\theta_n$; Augmentation function $T$; ViT teacher model $f_\\text{teacher}$; Zero-init feature accumulator tensor $Q$; Zero-init count occurrence tensor $K$\n\n> For i in ${1,...,n}$\n>> $\\theta_i = (x_i, y_i, \\text{flip}_i)$\n>> \n>> ($\\mathcal{I}_i, C_i$) = $T(\\mathcal{I}, C, \\theta_i)$ # Augment image, and generate the image-coordinates of the shifted/flipped image\n>> \n>> Dense Feature $F_i = f_\\text{teacher}(\\mathcal{I}_i)$\n>>\n>>$(F^\\text{valid}_i , C^\\text{valid}_i) = T^{-1}(F_i , C_i, \\theta_i)$ # Given the coordinates and the feature map of the augmented image, we map the feature back to the position in the original image\n>>\n>> $Q[C^\\text{valid}_i] = Q[C^\\text{valid}_i] + F^\\text{valid}_i$\n>> \n>> $K[C^\\text{valid}_i] =K[C^\\text{valid}_i] + 1$\n>> \n>> return $Q/K$ \n\nWe include a proof on page 12 of the supplemental showing that this achieves the same expected value, as gradient-based MSE optimization used in DVT neural field learning.\n\n**We will include additional details in our updated revision.**\n\n> **Q2) Ablation study of the distillation loss**\n\nThis is a good suggestion. Here we perform an ablation study using MSE, cosine similarity, and both combined (used in our work, and also used in DVT) -- in this table higher is better:\n\n\n| Loss                     |   cos sim   |   MSE   | MSE & cos sim |\n|--------------------------|:------:|:------:|:------------:|\n| Median Cosine Similarity | 0.9652 | 0.9673 |  **0.9720**  |\n\nNote that cosine similarity alone does not preserve magnitude, and occasionally yields training instability when used alone.\n\n**We will include this result in the revision.**\n\n> **Q3) Computational efficiency**\n\nWe agree that additional quantification of training and inference costs is beneficial. Given that custom kernels can significantly influence both performance and memory usage, our evaluation focuses on model size, FLOPs, and measured wall-clock training time. To ensure a consistent baseline, training time was measured using the official DVT repository code without modifications. Since the original DVT implementation specifies a single GPU for its feature extraction and learning phases, we similarly constrained all stages of our model to a single GPU for this comparison. The evaluated models (DVT and PH-Reg) are based on the same architecture: DINOv2 ViT-B.\n\nTraining wall-clock time: \n\n| Method | Stage 1 Extraction | Stage 1 Distillation | Stage 2 Training  | Total               |\n|--------|--------------------|----------------------|-------------------|---------------------|\n| DVT    | 2998 min           | 18340 min            |  570 min            | 21908 min (365.1 h) |\n| PH-REg | -                  | -                    |        -           | 9000 min (150 h)    |\n\nDuring training, DVT must store serialized Instant-NGP features, requiring **approximately 1.4TB of storage**. In contrast, our method imposes no supplementary storage requirements.\n\nTo assess testing costs fairly, all models were evaluated with parameters cast to `fp32` and using an `eager` attention implementation. We account for positional embeddings adjusted for 448x448 resolution into the PH-Reg parameter count. Regarding MaskCLIP and NACLIP, their official implementations currently retain the same parameter count as the original CLIP model, though MaskCLIP can achieve a minor reduction by removing the final query and key head.\n\nIn the case of DVT, we assessed their second-stage model, which pairs a transformer block denoiser with an original vision transformer.\n\n| Method                                | GFLOPs | Params (M) |\n|---------------------------------------|:------:|:----------:|\n| MaskCLIP                              |  62.89 |    86.19   |\n| NACLIP                                |  64.76 |    86.19   |\n| NACLIP + test time augmentation (10x) |  647.6 |    86.19   |\n| NACLIP + DVT                            |  70.32 |    94.07   |\n| CLIP + PH-Reg (Ours)                  |  64.16 |    86.66   |\n\nWe achieve notable efficiency gains over DVT: a 2x reduction in training time, complete elimination of training storage costs (0 TB vs. 1.4 TB), and ~10% savings in both inference FLOPs and model parameters. **These metrics will be added in the upcoming revision of our paper.**\n\n\n> **Q4) Comparison against the official DINOv2 + registers**\n\nHere we perform a comparison against DINOv2 + registers, using the official weights provided by facebook research:\n\n\n|         Method         |   VOC21   |   VOC21   |   ADE20k  |   ADE20k  |    NYUv2   |    NYUv2   |     NYUv2     |\n|:----------------------:|:---------:|:---------:|:---------:|:---------:|:----------:|:----------:|:-------------:|\n|                        |  mIoU(↑)  |  mAcc(↑)  |  mIoU(↑)  |  mAcc(↑)  |   RMSE(↓)  | Abs Rel(↓) | $\\delta_1$(↑) |\n|         DINOv2         |   84.13   |     92    |   47.82   |    60.5   |   0.4566   |   0.1391   |     82.92     |\n|   DINOv2 + registers   |    83.9   |   91.04   | **48.75** | **62.48** |   0.4382   |   0.1295   |     85.13     |\n| DINOv2 + PH-Reg (Ours) | **84.85** | **92.46** |   48.66   |   61.57   | **0.4306** | **0.1216** |   **86.35**   |\n\nWe find that our method achieves consistent improvements all on metrics compared to the original DINOv2 (without registers). Our method has better performance on VOC21 segmentation and NYUv2 depth prediction compared to `DINOv2 + registers`, while having slightly inferior performance on ADE20k. Note that the VOC21 mAcc degradation of `DINOv2 + registers` compared to `DINOv2` was also observed by DVT. For VOC21 mIoU, DVT observes a very small improvement, while we observe a very small degradation. For fairness, we ran our own evaluation using the same resolution for all methods -- and the evaluation settings are slightly different than those used DVT in their paper. \n\nGiven that our method can be completed using around 1.5 days on 4 Nvidia Ada GPUs, we believe our framework offers a very attractive alternative to full model retraining (estimated at several GPU-months if referencing OpenCLIP ViT-B numbers). **We will include this comparison in the revised paper.**\n\n> **Q5) Ablation studies of distillation approach**\n\nHere we evaluate the contribution of each component in OpenAI CLIP + PH-Reg  (mIoU, higher is better):\n\n| Approach                     |   VOC21   | Context60 | COCO object |   VOC20   | Context59 | COCO stuff | Cityscape |   ADE20k  |    Avg    |\n|------------------------------|:---------:|:---------:|:-----------:|:---------:|:---------:|:----------:|:---------:|:---------:|:---------:|\n| Vanilla                      |   49.27   |   25.46   |    26.94    |   66.56   |   28.62   |    18.80   |   28.33   |   13.70   |   32.21   |\n| TTA only (10x augmentation)                          |   51.41   |   28.13   |    29.00    |   69.58   |   31.03   |    20.25   |   31.82   |   15.20   |   34.55   |\n| Distill from teacher w/o registers w/o TTA              |   61.16   |   33.51   |    34.51    |   81.51   |   36.70   |    23.96   |   35.74   |   18.34   |   40.68   |\n| Distill from teacher w/ registers  w/o TTA                |   61.27   |   33.52   |    34.39    |   81.52   |   36.74   |    23.92   |   35.55   |   18.38   |   40.66   |\n| Distill from teacher w/o reg + TTA                |   62.48   |   34.28   |    35.00    |   82.27   |   37.62   |    24.46   |   36.83   |   18.92   |   41.48   |\n| Full Pipeline (w/ reg + TTA) | **63.01** | **34.52** |  **35.27**  | **83.05** | **37.88** |  **24.66** | **37.17** | **19.22** | **41.85** |\n\nBy comparing our full pipeline with the `Distill from teacher w/o registers w/o TTA` baseline, we find that approximately half the improvement comes from registers, and half comes from test-time augmentation of the teacher network.\n\n**We will include these results in the revision.**\n\n> **Conclusion**\n\nWe are very grateful for the highly detailed and actionable suggestions. We have included additional details for our method, additional experiments on distillation loss, method efficiency, comparison against models with registers trained from scratch, and ablation studies of all components of our pipeline. \n\nGiven our experiments and clarifications, we hope we have resolved your questions! We look forward to any additional suggestions you may have."}]

Author rebuttal:

Meta-review: This paper introduces PH-Reg, a post-hoc self-distillation framework designed to mitigate artifact tokens in pre-trained ViTs without requiring full retraining or labeled data. By leveraging frozen teachers, lightweight student updates, and augmentation-driven denoising, the approach enables efficient feature correction while maintaining generality.

All reviewers agree that the paper addresses a timely and relevant problem, providing a practical and efficient solution for enhancing pre-trained ViTs without costly retraining. The authors are highly encouraged to compare their method with the reference [52], particularly in the semantic segmentation setting, to better contextualize the contribution. Although [52] only treats high-norm tokens as defects, it similarly avoids expensive retraining, making a direct comparison valuable for deeper insights.

Overall, reviewers are generally positive, with most leaning toward acceptance after rebuttal clarifications.
Decision: Accept (spotlight)