Title: Vision Transformers with Self-Distilled Registers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Vision Transformers

Dense Features

Self Distillation
Abstract: Vision Transformers (ViTs) have emerged as the dominant architecture for visual processing tasks, demonstrating excellent scalability with increased training data and model size. However, recent work has identified the emergence of artifact tokens in ViTs that are incongruous with local semantics. These anomalous tokens degrade ViT performance in tasks that require fine-grained localization or structural coherence. An effective mitigation of this issue is the addition of register tokens to ViTs, which implicitly ''absorb'' the artifact term during training. Given the availability of existing large-scale pre-trained ViTs, in this paper we seek to add register tokens to existing models without retraining the models from scratch, which is infeasible considering their size. Specifically, we propose Post Hoc Registers (**PH-Reg**), an efficient self-distillation method that integrates registers into an existing ViT without requiring additional labeled data and full retraining. PH-Reg initializes both teacher and student networks from the same pre-trained ViT. The teacher remains frozen and unmodified, while the student is augmented with randomly initialized register tokens. By applying test-time augmentation to the teacher’s inputs, we generate denoised dense embeddings free of artifacts, which are then used to optimize only a small subset of unlocked student weights. We show that our approach can effectively reduce the number of artifact tokens, improving the segmentation and depth prediction of the student ViT under zero-shot and linear probing.

Vision Transformers with Self-Distilled Registers

Abstract

Vision Transformers (ViTs) have emerged as the dominant architecture for visual processing tasks, demonstrating excellent scalability with increased training data and model size. However, recent work has identified the emergence of artifact tokens in ViTs that are incongruous with local semantics. These anomalous tokens degrade ViT performance in tasks that require fine-grained localization or structural coherence. An effective mitigation of this issue is the addition of register tokens to ViTs, which implicitly 'absorb' the artifact term during training. Given the availability of existing large-scale pre-trained ViTs, in this paper we seek to add register tokens to existing models without retraining the models from scratch, which is infeasible considering their size. Specifically, we propose Post Hoc Registers ( PH-Reg ), an efficient self-distillation method that integrates registers into an existing ViT without requiring additional labeled data and full retraining. PHReg initializes both teacher and student networks from the same pre-trained ViT. The teacher remains frozen and unmodified, while the student is augmented with randomly initialized register tokens. By applying test-time augmentation to the teacher's inputs, we generate denoised dense embeddings free of artifacts, which are then used to optimize only a small subset of unlocked student weights. We show that our approach can effectively reduce the number of artifact tokens, improving the segmentation and depth prediction of the student ViT under zero-shot and linear probing. Our code is publicly available at this repository.

1 Introduction

Vision Transformers (ViTs) are now the dominant architecture in visual modeling, delivering strong performance across classification, detection, and segmentation. Unlike convolutional networks with their built-in locality inductive bias, ViTs process images by spatially splitting them into patches and applying self-attention to enable global feature interactions. This architectural design leads to superior scalability, particularly with contrastive or self-supervised pre-training objectives, and facilitates more flexible representation learning, as it is less constrained by the translation invariance assumptions inherent in CNNs. This flexibility enables remarkable emergent capabilities. Models like CLIP, trained solely on image-text alignment, achieve competitive open-vocabulary segmentation through zero-shot dense queries; while self-supervised approaches learn semantically rich features directly from unlabeled images.

However, the same data-driven attention mechanisms that enable ViT's representation power can also lead to the emergence of artifact tokens . These are outlier features often discordant with local image semantics, meaning they fail to correspond to locally meaningful image structures. The propensity for ViTs to generate such tokens is exacerbated by their lack of strong, built-in spatial priors, which can result in inconsistent dense representations. Ultimately, the presence of these artifact tokens

Image

MaskCLIP

SCLIP

NACLIP

Ours (PH-Reg)

disrupts fine-grained localization, a critical capability for tasks demanding high spatial precision, such as detailed semantic segmentation or part identification.

Recent work has sought to mitigate artifact tokens via architectural modifications, where register tokens are added to the network. These register tokens are randomly initialized, with learnable parameters that participate in the self-attention process similar to the [CLS] token, but are not otherwise used during the output. Although these register tokens are not explicitly supervised during training, they effectively 'absorb' the artifact term and learn to attend to global objects. While effective, introducing register tokens constitutes a fundamental architectural modification that requires training from scratch-a time-consuming and computationally demanding process. This significantly limits their applicability, especially given the vast ecosystem of existing, high-performing pre-trained vision models.

We present a solution to this issue with P ost H oc Reg isters ( PH-Reg ), an efficient self-distillation framework that requires no labeled data or full retraining. We illustrate OpenAI CLIP with PH-Reg in Figure 1. In PH-Reg, both teacher and student networks are initialized from the same pre-trained model weights. And the only extra parameters are the register tokens added to the student network. Our proposed framework freezes the teacher during training. Images provided to the teacher undergo test-time augmentation (e.g. random offsets and horizontal flips). This augmentation strategy effectively denoises the teacher's dense features without requiring gradient-based updates on the teacher itself, yielding stable dense targets. The denoised dense features are used as a distillation target for the student network, where only a small set of parameters are optimized. This entire process requires only a modest set of unlabeled images, enabling significant enhancements to pretrained models with minimal computational overhead. Concretely our contributions are as follows: 1. We propose a test-time augmentation scheme that can effectively denoise dense features in vision transformers. Our denoiser does not require costly neural fields and does not require gradient based optimization. 2. We elucidate the underlying components in a student model that contribute to learning a clean dense feature map. We show that by finetuning select weights, we are able to achieve clean dense features with minimal additional parameters. 3. We demonstrate that PH-Reg effectively improves the consistency of dense feature representations in ViTs, leading to quantifiable improvements on downstream tasks that rely on fine-grained spatial understanding (e.g., semantic segmentation or depth prediction). Our method preserves the original utility of dense features without inducing unwanted distribution shift, and functions well with zero-shot language-based dense queries.

(a)

Teacher

Repeat

i

=1…n

Denoise

C

i

Distill

Initialized from

same weights

Frozen

Student

···

Register Tokens

Image

I

(b)

I

2

Image

I

I

1

Low

High

(c)

Ours (PH-Reg)

NACLIP

SCLIP

MaskCLIP

RGB image

0.0

1.0

0.5

1.0

0.5

x

-coord

y

-coord

Coordinates

C

θ1

=(x

1

,y

1

,flip

1

)

C

1

=(x

2

,y

2

,flip

2

)

C

2

T

(

I

,

C

,θ

i

)

I

i

…

Original with

artifacts

Clean features

2 Related Work

Transformers in Visual Learning. Building upon the success of self-attention in language modeling, architectures that leverage transformer based token-mixing have been proposed for visual generation [1, 2, 3] and recognition tasks [4, 5], cumulating in the ViT architecture which relies on very few locality biases [6]. In the years since, many improvements and variants have been proposed [7, 8, 9, 10, 11, 12]. The improvements have largely focused on data [13] and compute efficiency [14, 15, 16, 17, 18, 19, 20, 21, 22]. In general, vision transformers tokenize an image into a set of patches, where each patch is first processed using an MLP or convolution block [23, 24, 25, 26], the patches are further processed with self-attention which enables global token interactions beyond those in convolutional networks. As self-attention is permutation invariant, positional information is typically injected using learnable positional embeddings or relative positions [27, 28, 29, 16, 30, 31, 32]. Positional embeddings have been suggested to play a role in the emergence of dense ViT artifacts, as networks with positional embeddings removed have smooth feature maps [33].

Representation Learning with Vision Transformers. The lack of restrictive local inductive biases in Vision Transformers enables strong scaling behavior across a diverse set of tasks. Beyond traditional supervised learning on categorical datasets such as ImageNet, methods have been proposed to learn on large scale datasets by leveraging language contrastive objectives [34, 35], or selfsupervised image-level objectives [36, 37, 38] and patch-level objectives [39, 40, 41, 42, 43, 44, 45, 46]. While the training objectives are very different, these methods enable strong zero-shot and linear-probe performance across a diverse set of tasks, suggesting that these methods effectively learn the underlying statistics of visual input. While these methods lack explicit dense supervision, the dense features from these models have been shown to have strong zero-shot emergent behavior with language-based segmentation [47], object part correspondence [48], and structural understanding [49].

Artifacts in Vision Transformer Dense Features. Recent work on DINOv2 [46] has found that Vision Transformers can have artifacts in their dense features. It has been proposed that artifacts can be mitigated with 'register' tokens [50, 51, 52, 53]. These register tokens are effectively randomly initialized embeddings that are analogous to the [CLS] token. While registers participate in the self-attention process, they are discarded during output. This approach requires a model to be trained from scratch. The nature and the mechanisms that cause the emergence of artifact tokens are unclear, and there exists conflicting results on what information (global or no information) these artifact tokens contain [50, 54]. Recent work has further investigated the mechanism of register

DINOv2

DFN-CLIP

OpenCLIP

OpenAI CLIP

Image

Raw

Denoised

Raw

Denoised

Raw

Denoised

Raw

Denoised

tokens [55]. Our own results have found that artifact tokens are not necessarily high-norm, and can be low-norm as well. Unlike the observation by [33], we find that positional embeddings alone cannot account fully for the artifacts. Regardless of 'why' artifact tokens emerge, removing these artifacts is an active area of research, with proposals based on registers [50], magnitude smoothness priors [54], and the foundational work on leveraging neural fields to denoise ViTs with a static artifact component [33, 56, 57]. A concurrent line of work has sought to remove artifacts for open-vocabulary segmentation with training-free attention modifications [47, 58, 59, 60, 61, 62]. Our framework can be applied to existing pretrained networks, introduces minimal additional parameters, can be applied to tasks beyond open-vocabulary segmentation, and makes no assumptions on the magnitude or static nature of the artifacts.

3 Methods

In this section, we will describe the PH-Reg framework, which we illustrate in Figure 2. This framework enables existing pretrained ViTs to benefit from register tokens, yielding significantly cleaner dense representations. During training, PH-Reg requires only unlabeled images for the self-distillation process. In section 3.1, we will first describe the denoising process of teacher network outputs. Unlike prior work that rely on a neural field/hash-grid, this method denoises dense features without the use of expensive gradient-based learning. In section 3.2, we will describe how we initialize and modify the student architecture. This approach only introduces a small set of additional parameters to the network. Finally in section 3.3 we will describe our distillation process.

3.1 Efficient Denoising of Teacher Representations

Our denoising process starts from the observation that artifact tokens are not static relative to image content. Put another way, if an image is shifted by a certain amount (with the gaps padded with whitespace), the artifacts do not shift by the same amount. As shown in Figure 2, given an RGB image I ∈ R H × W × 3 , we randomly sample n random augmentation parameters ( θ 1 , θ 2 , ..., θ n ) , where each θ i defines a horizontal/vertical offset ( x i , y i ) and boolean flip i ∈ { 0 , 1 } defined horizontally. For each image, we also compute the image space coordinate grid C = ( x-coords , y-coords ) . Where ( x, y ) are respectively in range [0 , 1] : x defines the leftright axis, and y defines the top-down axis. The coordinates C help us keep track of the original location of an image region after augmentation. In practice, as we are working with a ViT model with patch size k × k , where the tokenization process yields a ( H k , W k ) grid of image tokens, we define our parameters using offsets that are integer multiples

Algorithm 1 Denoising Process

of k to facilitate efficient indexing. Together, the image I , the coordinates C , and augmentation parameter θ i are provided to transform function to yield an augmented image I i and new coordinates C i : T ( I , C , θ i ) ⇒ ( I i , C i ) .

Jam

Croissant

UMAP

Image

Sea

Sky

Palm

UMAP

MaskCLIP

SCLIP

NACLIP

Ours

(PH-Reg)

Image

A teacher model f teacher is based on a frozen set of original network weights, without any additional parameters. Given an augmented image I i , this network outputs feature F i . We restore the features to the original location within an image using the inverse transform function T -1 ( F i , C i ) . The restored features are additively accumulated across different augmentation parameters, while keeping track of the number of occurrences for each location. At the end, the dimension-wise sample mean is computed for the accumulated features. We present our full denoising process in Algorithm 1. The patch-wise expected value of this representation is the same as the optimal value when optimizing a discrete grid of representations to minimize mean squared error (as used in DVT [33] and traditional neural field based methods). However, as we do not require gradients, this denoising process can be done in less than 200ms , roughly two magnitudes faster than neural field based denoising in DVT. The comparison between raw and denoised dense feature visualizations is shown in Figure 3.

3.2 Design of the Student Network

Our objective is to preserve maximimal computational efficiency of the student model, while leveraging the knowledge of the pre-trained weights. For this purpose we introduce m number of register tokens, providing a minimally invasive enhancement to the base architecture. After the addition of register tokens, a total of m +1+ H k × W k tokens participate in the self-attention process. Unlike prior work that trains registers from scratch [50], this approach updates only these registers and selectively unfreezes specific components during distillation, preserving the majority of the ViT's pretrained weights. Through ablation studies, we identify optimal unfreezing strategies, such as adjusting convolution layers, positional embeddings, or the last transformer block.

3.3 Learning and Optimization of the Student

We employ a multi-objective distillation strategy, combining cosine similarity and mean squared error losses to ensure both directional and magnitude alignment between teacher representations F teacher and student representations F student . Our final loss is: Losstotal = 1 -cossim ( target , predicted ) + MSE ( target , predicted ) .

4 Experiments

In this section, we comprehensively evaluate the performance of PH-Reg on a diverse set of dense tasks, first using a zero-shot setup for open-vocabulary segmentation in section 4.1, followed by linear probe based segmentation and depth tasks in section 4.2. Finally we perform ablation studies to explore design decisions and investigate the nature of artifacts across different models in section 4.3. All implementation details are provided in the appendix.

4.1 Open-vocabulary Semantic Segmentation Evaluation

Datasets. In this section, we follow prior works [59, 62, 61] to evaluate our approach on six semantic segmentation datasets, with their names abbreviated (in parentheses) to conserve table space: PASCAL

VOC 2012 (VOC21) [63], PASCAL Context (PC 60) [64], COCO-Object (Object) [65], COCOStuff (Stuff) [66], Cityscape (City) [67], ADE20K-150 (ADE) [68]. In addition to these standard benchmarks, we also evaluate on two commonly used variants, PASCAL VOC 2012 (VOC20) and PASCAL Context (PC 59), in which the background class is excluded from the evaluation. For all experiments, we utilize the same evaluation harness for all methods, and apply a sliding window inference strategy for non-square images. We also resize input images such that the shorter side is fixed to specific resolutions, accommodating the varying original image sizes across datasets.

Baselines. We compare our method against several relevant approaches in open-vocabulary semantic segmentation, including MaskCLIP [47], SCLIP [59], ClearCLIP [61], and NACLIP [62]. We also include vanilla CLIP as a baseline in our comparison, as it can be adapted for semantic segmentation. Unless otherwise specified, all visual encoders use the widely adopted pretrained ViT backbone with the same OpenAI CLIP ViT-B/16 weights to ensure a fair comparison. We also include denoised versions of MaskCLIP and NACLIP produced by DVT, as DVT represents a closely related method to our approach. Unless otherwise noted, for CLIP we adopt the most basic MaskCLIP framework (direct v output without any attention modifications) as our student model. Notably, we re-implemented all baselines using the same prompt templates as in [59, 62]. All reported results are obtained without any post-processing refinement.

Quantitative Results. Table 1 summarizes the quantitative comparison results of various openvocabulary semantic segmentation models. We observe that PH-Reg CLIP consistently outperforms all compared methods on 7 out of 8 evaluated benchmarks, with particularly strong results in VOC21 (63.01%) and COCO Object (35.27%). Moreover, PH-Reg CLIP surpasses the denoised versions of MaskCLIP and NACLIP, where DVT fails to yield significant performance gains. We believe this is caused by the residual estimator in DVT, which assumes stationary artifacts - an assumption that does not hold consistently for training-free open vocabulary segmentation methods based CLIP. We note that ClearCLIP slightly outperforms our method on the VOC20 dataset. This may be attributed to its use of correlative self-attention, i.e. q-q attention, which incorporates feature localization cues. This explanation is plausible, as SCLIP, which also employs q-q attention, similarly outperforms NACLIP on the VOC20 dataset.

Qualitative Results. Figure 4 presents a qualitative comparison between our PH-Reg CLIP and three baseline models: MaskCLIP, SCLIP, and NACLIP. We visualize the UMAP of the dense features produced by each model, as well as the corresponding heatmaps generated from different text queries. Our qualitative observations are as follows: 1. Artifact tokens are frequently observed in the UMAP visualizations of MaskCLIP, SCLIP, and NACLIP. While some artifact tokens are reduced in the heatmap of MaskCLIP, they remain prevalent in the heatmaps of both SCLIP and NACLIP. 2. The presence of artifact tokens hinders the models' ability to maintain fine-grained spatial alignment with the text queries, leading to suboptimal localization. 3. In contrast, PH-Reg CLIP consistently produces cleaner UMAPs and more fine-grained, semantically aligned heatmaps, which correspond well to meaningful local image structures. These visualizations demonstrate that our method effectively preserves the consistency of dense feature representations and enhances semantic alignment between visual and textual modalities.

4.2 Linear Probe Based Segmentation & Depth Evaluation

Datasets & Baselines. In this section, we evaluate our approach in two semantic segmentation datasets: PASCAL VOC 2012 (VOC21) [63] and ADE20K-150 (ADE) [68] and one monocular depth estimation dataset: NYUv2-Depth dataset (NYUd) [69]. Since DVT [33] targets a similar objective with our approach, we include both the vanilla models and their denoised versions produced by DVT as comparison baselines. We adopt the same linear probe experimental setup as in [33], and train a linear layer integrated into the backbones, as a decode head to predict pixel-wise segmentation or depth logits from patch tokens. Table 2 summarizes the main experiment results.

Semantic Segmentation Results. As shown in Table 2, We observe significant and consistent improvements, outperforming at least 4 out of 6 denoised ViT backbones across the evaluated datasets. While DVT consistently enhances the performance of DINOv2 and vanilla CLIP, it provides only limited improvements for other ViT backbones derived from other CLIP models. In contrast, our approach yields substantial performance boosts across these backbones, especially a notable +5.04% mIoU on VOC21 and +3.64% mIoU on ADE20k. These results demonstrate that our method can be robustly adopted to enhance the performance of diverse ViT backbones in semantic segmentation.

Notably, DVT relies on neural fields and requires gradient-based optimization, making the iterative denoising process applied to each image individually highly time-consuming. Our method leverages test-time augmentation for denoising, enabling the generation of cleaner dense feature representations without incurring excessive computational overhead. However, our results also show that the residual estimator as introduced in DVT may be beneficial to some model types (DINOv2, DFN-CLIP) more so than others. These results highlight that PH-Reg achieves superior performance in suppressing artifact tokens through a more robust and efficient design.

Depth Estimation Results. Following prior work [46, 33] we adopt AdaBins [72] for monocular depth evaluation. As shown in Table 2, our method consistently improves the performance of pretrained ViT backbones whereas the DVT assumption of stationary artifacts mostly hold true for DINOv2. Additionally, DVT achieves performance gains using an additional transformer block with 0.08× the parameters of the base models [33], our method achieves superior results with only a negligible increase in parameter count introduced by the register tokens. These results demonstrate the efficiency of our approach, yielding noticeable performance gains with minimal model overhead.

4.3 Ablation Studies and Investigation of Artifacts

In this section, we conduct ablation studies on OpenAI CLIP ViT-B/16 to investigate various architectural and training components, focusing on both model performance and training feasibility.

Cosine Similarity

1.00

50%

registers

0.98

70%

+pos

0.96

95%

90%

+additional

99%

0.92

0.88

0.86

0.84

0.82

0.80

0reg

1reg

2reg

4 reg

8reg

16reg

conv +pos

+16regpos+16reg

+attention

(a) Registers' behavior. This plot illustrates adding registers improve PH-Reg teacher performance. In the blue settings, only registers are unfreezed. The green settings represent the improvements when positional embeddings are unlocked additionally. The red settings represent performance of unlocking more layers.

Cosine Similarity

Number of augmentations

1.00-

0.95

0.90-

50% Most Different

70% Most Different

90% Most Different

0.85

95% Most Different

98% Most Different

99% Most Different

0.80

2

4

6

8

10

The Number of Register Tokens. We evaluate the influence of the number of register tokens on the cosine similarity between the student model's outputs and the target values using the COCO Caption dataset [73]. Specifically, we distill the student model with 0, 1, 2, 4, 8, or 16 register tokens.

As illustrated in Figure 5a, the cosine similarity increases as the number of register tokens grows, indicating improved alignment with the target representations. However, the performance gain becomes marginal when increasing the number of registers from 4 to 8 and from 8 to 16. Based on this observation, we use 16 register tokens in all subsequent experiments.

Distillation Architectural Settings. We further evaluate the impact of architectural configurations during distillation by analyzing cosine similarity on the COCO Caption dataset. In this setting, we vary the number of register tokens from 0 to 16 while allowing the positional embeddings to be updated during training. As shown in Figure 5a, the improvement in the cosine similarity from unlocking the position embedding becomes less pronounced as the number of register tokens increases. Nonetheless, unlocking positional embeddings continues to provide a positive effect on alignment. This result suggests that in contrast to DVT, the positional embedding itself is unlikely to fully explain the artifact tokens.

Next, we fix the number of register tokens to 16 and evaluate the effect of unlocking additional layers, including the convolutional patch embedding layer and the later attention layers. For all experiments, we report 50th , 70th , 90th , 95th , and 99th percentiles (of cosine similarity to capture the distribution of the most dissimilar features.) Our analysis reveals that incorporating even a single register leads to substantial improvements. In particular, the 99th percentile of feature cosine similarity in the 1-register configuration exceeds the 50th percentile (median) of the raw case without registers. This indicates that registers significantly enhance the quality of feature representations across the distribution, not only in extreme cases.

As suggested by [54, 33], attention layers close to the output also play an important role, which we confirm in our experiments. As shown in Figure 5a, unlocking the last attention layer significantly increases the cosine similarity between the student model's outputs and the target values. While unlocking the convolutional patch embedding layer alone slightly reduces cosine similarity value, the overall value improves when both the convolutional patch embedding and later attention layers are unlocked, compared to the baseline with only unlocked the position embeddings and later attention layers with 16 registers. Therefore, we unlock the positional embeddings, the convolutional patch embedding layer, and the final attention layer during distillation.

The Number of Augmentations in the Denoising Process. We evaluate our approach using cosine similarity on the COCO Caption dataset to investigate the effect of the number of augmentations. The

OpenCLIP

OpenAI CLIP

Image

DINOv2

DFN-CLIP

Image

(a)

(b)

Original

w/

PH-Reg

Norm diff

Original

w/

PH-Reg

Norm diff

student model is distilled with 1 to 10 augmentations, where one of them is always an unmodified image. As shown in Figure 5b, a high convergence threshold is observed at the 99th percentile where even the most dissimilar cases exhibit cosine similarity values above 0.95, indicating a substantial reduction in feature space outliers. As a result, we employ 10 augmentations to generate high-quality features efficiently, thereby reducing computational overhead.

Distillation Approaches. We evaluate the contribution of different components in our distillation approach by conducting open-vocabulary semantic segmentation task on 8 benchmarks. In this setting, the student model is distilled by sequentially removing each component from our approach. These components include the denoising process with 10 augmentations, 16 additional register tokens, and self-distillation. As shown in Table 3, each component contributes to improving the student model's performance. By comparing our full pipeline, where student model is distilled with denoising process and registers, we find that approximately half of the improvement comes from registers, and the other half results from denoising process applied to the teacher model.

The Ratio of Shifting in the Denoising Process. To investigate the impact of different shifting ratios in our denoising process, we conduct open-vocabulary semantic segmentation task on 8 benchmarks. We gradually increase the shifting ratio from 0% to 30%, while fixing the number of augmentation at 10. As shown in Table 4, a shifting ratio of 10% demonstrates the best performance across 4 datasets, while a shifting ratio of 15% achieves the best performance across 3 datasets and provides optimal average performance across all 8 datasets. Therefore, we adopt a shifting raio of 15% in denoising process when applied to the teacher model.

Norm

Original OpenAI CLIP

w/PH-Reg

Original OpenCLIP

w/PH-Reg

Original DINOv2

w/PH-Reg

12

16

200

11

175

10

14

150

125

9

12

100

8

10

75

7

50

6

8

4.4 Investigation of Artifacts and Registers

While prior work has noted that artifact tokens are high magnitude in DINOv2 [50], in Figure 6 we find that this is not always the case. In OpenAI CLIP and OpenCLIP, the artifacts are generally lower norm than their surrounding patches. In contrast, in DFN-CLIP and DINOv2, the artifacts are higher norm. This illustrates that there may be elements of the training dynamic at play, as the artifact norms can differ even when the training objective is very similar. In Figure 7 we visualize the norms of the original network and those with registers added, we find that our method effectively reduces the variance of the patch norms.

5 Discussion

Limitations and Future Work In this work we proposed PH-Reg, a method to reduce the artifact tokens in existing pre-trained vision transformers. We show that PH-Reg can eliminate artifact tokens in ViTs effectively and generate clean dense feature maps, enhancing the performance in downstream dense prediction tasks. This approach relies on test time augmentation to denoise dense feature presentations in the teacher model. While our method generally outperforms DVT on CLIP based models, we sometimes underperform when using the DINOv2 backbone. We believe this is due to the static artifact estimator present in DVT. The assumption of static artifacts holds true for some models (DINOv2), but not for others (CLIP). A potential avenue for additional investigation is how to dynamically determine the artifacts without strong stationary assumptions.

Conclusion We introduce a novel post-training method PH-Reg, for learning clean dense feature representations in ViTs through an efficient self-distillation framework that does not require additional labeled data. Our approach leverages test-time augmentation to denoise the teacher model, and guide the student model to optimize the dense feature representations. This enables us to eliminate artifact tokens effectively by integrating learnable registers into existing pretrained models, without the need for training from scratch. We demonstrate that the distilled ViTs generate fine-grained dense feature maps, enhancing the consistency of feature representations in ViTs. We further show that cleaner dense feature maps in ViTs leads to quantifiable improvements on dense prediction tasks. Finally, we illustrate that the distilled ViTs can accurately capture meaningful semantic structures in images, as shown by heatmaps generated from CLIP text queries. We validate our conclusions with extensive evaluations across multiple dense prediction benchmarks.

References

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning , pages 4055-4064. PMLR, 2018.

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019.

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11315-11325, 2022.

Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. Advances in neural information processing systems , 32, 2019.

Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. arXiv preprint arXiv:1911.03584 , 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Tobias Christian Nauen, Sebastian Palacio, Federico Raue, and Andreas Dengel. Which transformer to favor: A comparative analysis of efficiency in vision transformers. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 6955-6966. IEEE, 2025.

Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12894-12904, 2021.

Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems , 34: 24261-24272, 2021.

Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08602 , 2021.

Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention for transformer models. In International conference on machine learning , pages 10183-10192. PMLR, 2021.

Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 32-42, 2021.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning , volume 139, pages 10347-10357, July 2021.

Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International conference on machine learning , pages 9438-9447. PMLR, 2020.

Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794 , 2020.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Crosscovariance image transformers. Advances in neural information processing systems , 34:2001420027, 2021.

Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics , 9:53-68, 2021.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10809-10818, 2022.

Ting Yao, Yingwei Pan, Yehao Li, Chong-Wah Ngo, and Tao Mei. Wave-vit: Unifying wavelet and transformers for visual representation learning. In European conference on computer vision , pages 328-345. Springer, 2022.

Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. arXiv preprint arXiv:2202.07800 , 2022.

Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 22-31, 2021.

Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision , pages 558-567, 2021.

Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The vision-friendly transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 589-598, 2021.

Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multiscale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision , pages 357-366, 2021.

Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International conference on machine learning , pages 12431252. PMLR, 2017.

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024.

Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing , 149:105171, 2024.

Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.12376 , 2024.

Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26439-26455, 2024.

Dahun Kim, Anelia Angelova, and Weicheng Kuo. Region-aware pretraining for openvocabulary object detection with vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11144-11154, 2023.

Jiawei Yang, Katie Z Luo, Jiefeng Li, Congyue Deng, Leonidas Guibas, Dilip Krishnan, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denoising vision transformers. In European Conference on Computer Vision , pages 453-469. Springer, 2024.

Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. URL https://openreview.net/forum?id= Ee277P3AYC .

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Xinlei Chen*, Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02057 , 2021.

Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. International Conference on Learning Representations (ICLR) , 2022.

Trieu H. Trinh, Minh-Thang Luong, and Quoc V. Le. Selfie: Self-supervised pretraining for image embedding, 2019. URL https://arxiv.org/abs/1906.02940 .

Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1691-1703. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/chen20s.html .

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In International Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377 , 2021.

Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR) , 2022.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=p-BhZSz59o4 .

Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a jointembedding predictive architecture. arXiv preprint arXiv:2301.08243 , 2023.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision , pages 696-712. Springer, 2022.

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. ECCVW What is Motion For? , 2022.

Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12490-12500, June 2024.

Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588 , 2023.

Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. Memory transformer. arXiv preprint arXiv:2006.11527 , 2020.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 , 2023.

Nick Jiang, Amil Dravid, Alexei Efros, and Yossi Gandelsman. Vision transformers don't need trained registers. arXiv preprint arXiv:2506.08010 , 2025.

Haoqi Wang, Tong Zhang, and Mathieu Salzmann. Sinder: Repairing the singular defects of dinov2. In European Conference on Computer Vision , pages 20-35. Springer, 2024.

Alexander Lappe and Martin A. Giese. Register and cls tokens yield a decoupling of local and global features in large vits, 2025. URL https://arxiv.org/abs/2505.05892 .

Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al. Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. arXiv preprint arXiv:2311.02077 , 2023.

Andrew F Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M Henderson, Leila Wehbe, and Michael J Tarr. Brain mapping with dense features: Grounding cortical semantic selectivity in natural images with vision transformers. arXiv preprint arXiv:2410.05266 , 2024.

Yi Li, Hualiang Wang, Yiqun Duan, Jiheng Zhang, and Xiaomeng Li. A closer look at the explainability of contrastive language-image pre-training. Pattern Recognition , 162:111409, 2025. ISSN 0031-3203. doi: https://doi.org/10.1016/j.patcog.2025.111409. URL https: //www.sciencedirect.com/science/article/pii/S003132032500069X .

Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense visionlanguage inference, 2024. URL https://arxiv.org/abs/2312.01597 .

Walid Bousselham, Felix Petersen, Vittorio Ferrari, and Hilde Kuehne. Grounding everything: Emerging localization properties in vision-language transformers, 2023. URL https://arxiv. org/abs/2312.00878 .

Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Clearclip: Decomposing clip representations for dense vision-language inference. In European Conference on Computer Vision , pages 143-160. Springer, 2024.

Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2025.

Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision , 111:98-136, 2015.

Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 891-898, 2014.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13 , pages 740-755. Springer, 2014.

Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1209-1218, 2018.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision , 127:302-321, 2019.

Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12 , pages 746-760. Springer, 2012.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.

Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425 , 2023.

Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4008-4017, 2021. doi: 10.1109/CVPR46437.2021.00400.

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 , 2015.

A Technical Appendices and Supplementary Material

Sections

Implementation Details for Self-Distillation (section B)

Implementation Details for Quantitative Evaluation (section C)

Additional Efficiency Analysis of PH-Reg Compared to DVT (section D)

Additional Qualitative Examples for Segmentation (section E)

Additional Qualitative Heatmaps for PH-Reg Zero-Shot (section F)

Proof of Test Time Augmentation (section G)

B Implementation Details for Self-Distillation

In this section, we provide a detailed overview of how we implement self-distillation in PH-Reg. Our self-distillation framework consists of one teacher model and one student model. While both the teacher and student model are initialized from the same weights, the teacher is frozen, while additional register parameters are added to the student network.

Our codes and weights are avaible in https://github.com/0raiser0/PH-Reg.git.

B.1 Model Architectures

Teacher Model Architecture. For CLIP based models, since we focus on zero-shot open-vocabulary segmentation, we utilize the NACLIP modification to the final layer. This modification does not introduce any additional weights to the teacher network, and is training-free. Our empirical analysis in Section B.4 shows that NACLIP's neighborhood attention mechanism improves feature consistency. For DINOv2, we directly use the final output layer, without any modification to the teacher network.

Student Model Architecture. Based on the results from the ablation studies we integrate 16 register tokens into the student model. For the CLIP based student, to ensure representational alignment we directly take the v head from the output layer (the MaskCLIP output). For DINO based students, we do not apply such modifications. We bicubicly upsample the positional embedding so it matches the input image. Unless otherwise specified, this modification is applied consistently, while all other layers remain unchanged.

B.2 Model Implementation

We provide the model weights and the corresponding implementation libraries in Table S.1.

B.3 Optimization

In the distillation process, the shorter side of each input image is resized using bicubic interpolation to 448 for CLIP-based models and 518 for DINOv2. The resized image is then randomly cropped into a square of size (448, 448) or (518, 518), respectively. For each input image, we generate N = 10 augmentations using random shifts and horizontal flips. Assuming an image length of 1, we uniformly sample the shift for both the horizontal and vertical axes from [ -0 . 15 , 0 . 15] . While the horizontal flip is sampled with probability 0.5. To ensure each patch is covered, we do not apply any augmentation to the first image of the 10. All shifted images are concatenated and fed into the teacher model, while the original (unshifted) images are used as input to the student model. The target feature is computed as the average of these 10 augmentations. To accommodate the resized input images for both the teacher and student models, we consistently resize the positional embeddings using bicubic interpolation. During training, the weights of the teacher model are frozen. In the student model, we allow updates to registers, the positional embeddings, the convolutional patch embedding layer, and the final transformer layer containing the self-attention mechanism.

The distillation framework is implemented in PyTorch, with distributed training managed via PyTorch Accelerate. Training is conducted on 4 NVIDIA Ada 6000 GPUs, with mixed-precision optimization to balance computational efficiency and numerical stability. Detailed training configurations are provided in Table S.2 and Table S.3.

B.4 Pearson Analysis of Open-vocabulary Segmentation

Table S.4: Open-vocabulary Semantic Segmentation Quantitative Comparison on 7 datasets. We report the Pearson correlation coefficient for the zero-shot query against the one-hot ground truth labels. The results are averaged within each image, then averaged across images. Compared to mIoU, pearson does not require knowledge of all of the categories present an image (via softmax). The value ranges from -1 to 1, where 1 = perfect positive correlation, -1 = perfect negative correlation, and 0 = no linear correlation. The best result for each dataset is highlighted in bolded .

In this section we present additional evaluation results on open-vocabulary semantic segmentation via the pearson metric. Results are illustrated in Table S.4. Overall, PH-Reg CLIP significantly outperforms the baseline models on 7 datasets. Even in the absence of prior category knowledge, PH-Reg CLIP achieves an average performance of 0.404, representing a clear improvement over the second-best method, DVT enhanced NACLIP, with an average performance of 0.381. These results highlight that our approach improves the consistency of dense feature representations by reducing artifact tokens, thereby offering a robust and generalizable enhancement over existing methods.

We further observe that both ClearCLIP and NACLIP achieve competitive results; however, NACLIP significantly outperforms ClearCLIP on ADE20K and Cityscapes. The former requires the model to handle a large number of categories, while the latter demands fine-grained localization of small objects. Based on this observation, we choose NACLIP as our primary teacher model, leveraging its neighbor attention mechanism to enhance the student model's performance on these challenging tasks.

C Implementation Details for Quantitative Evaluation

In this section, we provide detailed implementation information for our quantitative evaluation experiments. In section C.1, we present the evaluation details for open-vocabulary semantic segmentation (OVSS). In section C.2, we describe the evaluation details for linear probe based semantic segmentation and monocular depth estimation.

C.1 Implementation Details of Open-vocabulary Semantic Segmentation.

Wefollow SCLIP and NACLIP in the setup for the open-vocabulary semantic segmentation evaluation. For fairness, we utilize the same parameters for all models. We resize input images such that the shorter side is scaled to a specific resolution, while maintaining the original aspect ratio for the longer side. Additionally, we set fixed crop sizes and strides during evaluation. All evaluation parameters are summarized in Table S.5, while all other settings follow their default configurations.

C.2 Implementation Details of Linear Probe Based Evalution.

Our linear probe evaluation follows prior works (Vision Transformers Need Registers, Denoising Vision Transformers), where a linear layer is trained as a decoding head to predict pixel-wise segmentation or depth logits.

Semantic Segmentation. We extract the final output features from the frozen backbone and, if applicable, pass them through the denoiser (for the DVT baseline). A single learnable linear layer is then trained to predict the segmentation logits. For CLIP-based models, both training and testing images are resized to (448, 448), while for DINOv2, the images are resized to (518, 518).

Monocular Depth Estimation. Similar to semantic segmentation, we extract features from the backbone, and pass them through the denoiser if applicable. Following the method in DVT and DINOv2, we then append the [CLS] token to each patch token to enrich the feature representations for all methods. A linear layer is trained using SigLoss and gradient loss (scaled by a factor of 0.5) to predict depth values into 256 uniformly distributed bins. We adopt DVT's learning rate of 5e-3 for all experiments.

D Additional Efficiency Analysis of PH-Reg Compared to DVT

In this section, we provide a detailed efficiency analysis of PH-Reg compared to DVT, focusing on three aspects: training time, space usage, and inference cost.

Training Time. When evaluating training time, we utilize the official code provided in the DVT repository without any modification. The original DVT code specifies a single GPU for feature

extraction and learning - for fairness we also limit all stages of our own model to a single GPU here. For DVT and PH-Reg, we evaluate the same model (DINOv2 ViT-B). Results are illustrated in Table S.6. Our method has the advantage of not utilizing gradient-based neural field learning as done in DVT. Therefore, our method trains the student model in a single stage, saving over 58.9% of the time compared to DVT.

Space Usage. A further advantage is the space utilization during training. DVT requires saving 1.4 terabytes of intermediate neural fields in the form of serialized Instant-NGP files. Our method computes all distillation targets on the fly, and requires no additional space.

Inference Cost. When evaluating testing cost for all models, we cast parameters to fp32 dtype, and use eager attention implementation for all models. For our method, we include the positional embeddings adapted for 448 resolution. For MaskCLIP and NACLIP, current official implementations have the same number of parameters as the original CLIP, although a small reduction can be achieve in MaskCLIP by discarding the last q, k head. For DVT, we evaluate their stage 2 model, corresponding to the transformer block denoiser coupled to a original vision transformer. Results are illustrated in Table S.7. Our method utilizes approximately 10% fewer FLOPs and 10% fewer parameters during inference compared to DVT.

E Additional Qualitative Examples

MaskCLIP

Ground Truth

Image

Ours (PH-Reg)

ClearCLIP

NACLIP

SCLIP

CLIP

Ground Truth

MaskCLIP

ClearCLIP

NACLIP

SCLIP

Image

CLIP

Ours (PH-Reg)

Ground Truth

Ours (PH-Reg)

NACLIP

ClearCLIP

SCLIP

Image

CLIP

MaskCLIP

F Additional Qualitative Heatmaps for PH-Reg Zero-Shot

Image

Stone

Hand

Cat

UMAP

MaskCLIP

SCLIP

NACLIP

Ours

Image

Window

Table

Sofa

UMAP

MaskCLIP

SCLIP

NACLIP

Ours

口

Image

Steak

Fork

UMAP

MaskCLIP

SCLIP

NACLIP

Ours

Image

Tractor

Path

UMAP

MaskCLIP

SCLIP

NACLIP

Ours

G Optimal Feature Aggregation

Let f 1 , . . . , f n ∈ R d be feature vectors of a single patch from n different transformations of an input image I . We seek the optimal aggregated feature f ∗ that minimizes the total squared error:

Expanding the objective gives us:

Dropping constant terms that do not change the optimum:

Dividing and multiplying the right side by n :

Dividing the equation by n as whole shows us that we need to minimize:

So it can be derived that the mean of the feature vectors is the minimizer under MSE loss:

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We justify each claim with extensive experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have a limitations section and discuss potential future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not have theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide extensive details on our ablation studies. Additional details can be found in the supplemental.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code will be included in the supplemental.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: A high-level outline is included in the main paper. Implementation details will be incldued in the supplemental.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to cost of running experiments, we only perform one run.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see the supplemental for computational details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This is discussed in the supplemental.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper has no risk related to safeguards.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets that we use are public, and are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Code will be included in the supplemental.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not collect any human data.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We do not collect any human data.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: This research does not involve LLMs.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an interesting approach for learning registers in Vision Transformers (ViTs), building upon recent findings that additional registers help remove artifacts captured by the class token. The authors propose a novel method to clean artifact-contaminated feature maps using image augmentation, then leverage knowledge distillation to learn additional registers from the cleaned feature maps without requiring full training from scratch. Experiments on multiple segmentation tasks demonstrate the efficiency of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The motivation is well-founded and addresses a relevant problem in the recent popular finding on ViT with learnable registers. The paper seeks to improve upon recent advances in register tokens by developing an efficient approach to learn these registers without costly full training or fine-tuning procedures.\n\n- The proposed artifact elimination method using feature map augmentation is simple, novel, and efficient. The approach offers a practical solution to a known issue in ViT architectures.\n\n- The distillation-based learning framework is well-motivated and demonstrates strong empirical performance according to the experimental results, providing an effective alternative to more expensive training procedures.\n\n\n\nWeaknesses:\n\nI didn't find major weakness. Here I list some flaws that should further improve the paper. \n\n- The methodology section would benefit from more detailed mathematical formulation. The authors should include explicit equations describing how the final cleaned feature maps are computed.\n\n- Section 3.3 lacks sufficient detail regarding the distillation methodology. The paper should provide clearer justification for the chosen loss function and include ablation studies demonstrating why this particular loss formulation was selected over alternatives.\n\n- While computational efficiency is claimed as a key advantage, the paper lacks quantitative comparisons of training time, memory usage, and computational overhead between the proposed method and full training/fine-tuning approaches with registers.\n\nThere are also some questions that I would like to discuss with authors. Please refer the below section.\n\nQUESTIONS:\nAll baseline comparisons appear to use models fine-tuned from pre-trained CLIP. How does the proposed method compare against the original register-ViT trained from scratch?\n\nWould it be possible to include experiments comparing against models trained from scratch without registers? This would help isolate the contribution of the distillation approach versus the underlying register mechanism.\n\nGiven that data augmentation eliminates artifacts from feature maps during training, does this suggest that test-time data augmentation would also improve task performance? It would be valuable to design experiments that isolate and quantify the specific contribution of the data augmentation component. For instance, could the authors compare: (1) models trained with their full pipeline, (2) models trained with distillation but without the augmentation-based artifact cleaning, and (3) models that apply similar augmentation strategies at test time? Such ablations would help disentangle whether performance gains stem primarily from the cleaner training signal or from the augmentation strategy itself.\n\nLIMITATIONS:\nThe limitation has been well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper continues the recent area of research that indicates the presence of artifacts in vision transformers, and the introduction of register tokens to remedy this. The initial idea introduced and trained register tokens in the main architecture and discarded them for the loss and gradient computation. This paper introduces an important modification from there --  self-distilled registers which can be added to a pretrained model, and leads to good results at minimal retraining cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper presents a strong idea that significantly reduces the training costs, while also solving the issue of artifacts using the same registers idea.\n2. The experiment set and ablations are quite detailed and cover a variety of tasks, including semantic segmentation, depth estimation. The ablations cover various scenarios, like varying the number of register tokens, unlocking various layers, as well as augmentation settings during test-time adaptation.\n\nWeaknesses and Questions:\n\n1. As the authors point out, the method currently lacks the ability to determine artifacts in a dynamic manner. While this would require further research, a small discussion on potential future directions would be helpful.\n2. The paper \"Vision Transformers Need Registers\" suggested that the artifact tokens are high norm while this paper suggests that this isnt always the case. Can this be inspected further to understand if this inconsistency is the result of small architectural modifications? If tried out already, do the authors find a model trained along with registers from scratch to be consistent to the mentioned paper? Another recent paper \"Vision Transformers Don't Need Trained Registers\" uses the high norm claim to build a method (this is purely a mention and does not affect my rating as it is concurrent work)\n3. The \"Vision Transformers Need Registers\" paper suggests that the artifact tokens tend to contain global rather than local context, typically class information, etc. Do the authors find this observation while distilling register tokens as well?\n\nQUESTIONS:\nPlease see the strengths and weaknesses section above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecent research shows that big Vision Transformer models such as DINOv2 or OpenCLIP contain artifact tokens that encode undesired information. While this information could be relevant, it does not provide the local semantics that should represent the patch token, degrading the performance of the ViT on tasks that require fine-grained patch information. Authors propose Post Hoc Registers (PH-Reg), an elegant solution to the problem that, contrary to the current SoTA, does not require full re-training or additional labelled data. Main contributions can be summarised in their simple but effective self-distillation training and the artifact denoising strategy that enables the whole training by acting as a training target.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nTechnically sound method that successfully addresses the identified problem.\n\nPH-Reg stands as a model \"corrector\", automatically improving any ViT-based SoTA vision model that suffers from explained artifacts.\n\nProvided quantitative and qualitative analysis clearly display the benefits of the proposed method, highlighting especially the qualitative results.\n\nI find the denoising process intriguing. While a corrected method would always be more efficient, the described denoising strategy already manages to remove most of the artifacts without any specific training.\n\nThe authors yield several insights about the artifacts that may help the rest of the community working in the same field.\n\nThe paper is well-written and correctly organised.\n\n\n\n**Weaknesses:**\n\nThe authors claim that performance gain greatly decays after increasing the number of registers past 4/8. I would like to see more analysis on this, does this behaviour depend on the pretrained model? If not, what are the insights that could justify this general behaviour?\n\nEfficiency analysis regarding required memory (which I assume is low) and time between DVT and proposed approach would be interesting and could further prove the superiority of PH-Reg if they end up being much more efficient than DVT.\n\nProposed method seems to be much better when applied to VLMs, more in-depth insights about this behaviour would be appreciated.\n\nBeing a SoTA model \"corrector\", I believe the proposed approach has a strong short-term applicability. However, Registers are being widely applied already, and, at some point, models won't require a \"corrector\" anymore, dramatically decreasing the applicability of the proposed work.\n\nQUESTIONS:\nFrom the weaknesses explained before:\n1. Can authors provide more in-depth analysis regarding the performance gain decay as the number of registers increases?\n2. Is the proposed method more efficient than DVT? By how much in terms of memory and time?\n3. Improvements seem to be higher on VLMs, can the authors provide more in-depth insights about this behaviour?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Post Hoc Registers (PH-Reg), an efficient self-distillation framework to mitigate artifact tokens in pre-trained ViTs without full retraining or labeled data. By initializing teacher and student networks from the same pre-trained ViT, freezing the teacher, and augmenting its inputs(e.g.. random offsets, flips), PH-Reg generates denoised dense embeddings to guide student optimization, only updating a small subset of weights including randomly initialized register tokens. This approach effectively reduces artifact tokens, improving zero-shot and linear-probe performance in segmentation and depth prediction.Key contributions include an efficient test-time augmentation denoising scheme, selective finetuning of student components for clean feature learning, and demonstrated enhancements in dense representation consistency for fine-grained spatial tasks, as validated across multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tPH-Reg introduces a novel self-distillation framework that integrates register tokens into pre-trained ViTs without full retraining, leveraging test-time augmentation to denoise dense features as the teacher.This approach avoids the computational cost of training from scratch while effectively reducing artifact tokens that degrade fine-grained localization tasks.\n\n2.\tPH-Reg demonstrates consistent improvements across diverse dense prediction tasks (e.g., semantic segmentation, depth estimation) under zero-shot and linear probing setups, outperforming state-of- the-art baselines on multiple benchmarks.\n\nWeaknesses:\n\n1.\tDenoising dense predictions in a self-supervised manner is a common strategy in image processing works. The paper simply adopts the idea and finds that it also helps with the denoise at feature level. However, it lacks formal theoretical justification for the effectiveness of register tokens and test-time augmentation in artifact mitigation, focusing primarily on empirical validation.\n\n2.\tWhile PH-Reg excels on CLIP-based models, it underperforms DVT on DINOv2 due to DVT's static artifact estimator, indicating sensitivity to model-specific artifact characteristics.\n\n3.\tThe denoising efficacy relies on carefully hand-designed augmentations (e.g,random offsets), which may require task-specific tuning for optimal results. It's also needed to ablate the shift ratio impact.\n\nQUESTIONS:\n1.\tThe denoise step needs to ablate the shift ratio. And i'm wondering why not vertical fllp? \n\n2.\tIt seems all the experiments done with salient object. What about the results on more complex scenarios like coco and lvis?\n\nSee weaknesses for other questions.\n\nIf the author can address the above issues and provide a clearer theoretical analysis to demonstrate the effectiveness and robustness of the proposed method, I would be willing to raise the score.\n\nLIMITATIONS:\nIt would be better to discuss the failure cases of Dino to analyze the potential shortcomings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}