SiNGER: A Clearer Voice Distills Vision Transformers Further

## Abstract

Abstract Vision Transformers are widely adopted as the backbone of vision foundation models, but they are known to produce high-norm artifacts that degrade representation quality.
When knowledge distillation transfers these features to students, high-norm artifacts dominate the objective, so students overfit to artifacts and underweight informative signals, diminishing the gains from larger models. Prior work attempted to remove artifacts but encountered an inherent trade-off between artifact suppression and preserving informative signals from teachers. To address this, we introduce Si ngular N ullspace- G uided E nergy R eallocation (SiNGER), a novel distillation framework that suppresses artifacts while preserving informative signals.
The key idea is principled teacher feature refinement: during refinement, we leverage the nullspace-guided perturbation to preserve information while suppressing artifacts.
Then, the refined teacher’s features are distilled to a student.
We implement this perturbation efficiently with a LoRA-based adapter that requires minimal structural modification.
Extensive experiments show that SiNGER consistently improves student models, achieving state-of-the-art performance in multiple downstream tasks and producing clearer and more interpretable representations.

## 1 Introduction

Transformers have become the de facto standard architecture in both research and industry due to their scalability and effectiveness (dinov2; clip).
Their token-based self-attention mechanism is broadly applicable with minimal inductive bias (transformer-inductive-bias), and has enabled significant advances in computer vision and machine learning (openvla; unic).
Vision Transformers (ViTs, vit) extend this paradigm to visual data and form the backbone of Vision Foundation Models (VFMs).
Compared to convolutional networks, ViTs rely less on spatial inductive biases and instead exploit scale to achieve high performance.
ViTs are also highly scalable since supervised or self-supervised training produces increasingly generalizable representations (dinov2; deit3).

Figure: Figure 2: Qualitative analysis. Row 1: KD method comparison. Left: distilled feature map colored by patch norm, Right: patch-wise cosine similarity to the teacher. Row 2: Input image, two pretrained ViTs, and three ViT-L ${\to}$ ViT-T distilled variants. Each panel shows similarity from the $\times$-marked patch. SiNGER most closely preserves teacher semantics, showing the most coherent teacher-consistent similarity patterns.
Refer to caption: x3.png

However, the quadratic complexity of self-attention severely limits the practicality of scaling ViTs.
This tension between accuracy and efficiency motivates the study of compression.
Pruning (vit-pruning-01) and quantization (vit-quant-01) have been explored, but pruning often fails to deliver practical speedup due to structural rigidity (pruning-not-flexible), and quantization can induce numerical instability (quant-not-general-01; quant-not-general-02).
Knowledge distillation (KD, kd-hinton) has emerged as the most reliable solution for transferring knowledge from large ViTs to smaller students.
KD methods span diverse targets and frameworks (kd-response-01; kd-feature-01; kd-multi-teacher-01), consistently yielding structurally and numerically stable compact models.

Nevertheless, KD for ViTs suffers from subtle but critical limitations in their representation space.
vit-needs-registers revealed that ViT token representations contain high-norm artifacts.
gandikota2023sinder argue that these artifacts are singular defects induced by power-iteration-like accumulation across residual blocks, whereby tokens align with the leading left singular vector of the pre-trained weights.
These artifacts interact poorly with the standard feature mean squared error objective in KD: when the teacher and student are matched, gradients concentrate on the few high-norm tokens, producing an outlier-driven optimization bias that obscures informative signals in the inlier structure.
Therefore, suppressing outlier norms in teacher features is essential for KD in ViTs as the scale grows.
Prior work mitigated this issue via random masking of teacher features (kd-vitkd); however, this inevitably removes informative signals.
Therefore, a key challenge is to mitigate these artifacts without losing valuable information, a fundamental trade-off that requires a principled approach.

To resolve this trade-off, we introduce a nullspace-guided suppression: we modify only the nullspace component in the teacher features, mathematically, the subspace orthogonal to the downstream space.
This yields student-optimal supervision by suppressing artifacts without sacrificing informative signals.
Based on this insight, we propose Singular Nullspace-Guided Energy Reallocation (SiNGER), a framework that addresses this trade-off in ViTs distillation, illustrated in Figure [1(a)](https://arxiv.org/html/2509.20986v2#S0.F1.sf1).
To minimize the modification to the teacher’s signal, we attach a lightweight LoRA-based adapter (hu2021lora) to the KD architecture, which refines the teacher features.
The adapter produces a minimal perturbation guided toward the left-nullspace of the next block, suppressing high-norm outliers while leaving the next block output unchanged.
Our method achieves superior performance compared to baselines across multiple downstream tasks (Figure [1(b)](https://arxiv.org/html/2509.20986v2#S0.F1.sf2)).
It also produces more structured and interpretable feature maps (Figure [2](https://arxiv.org/html/2509.20986v2#S1.F2)). Our contributions are summarized as follows:

- •
We propose a novel distillation framework (SiNGER) that refines teacher signals via the LoRA-based adapter with nullspace initialization to guide effective perturbations.
- •
We analyze a fundamental limitation of naïve ViT distillation, showing degraded transfer on downstream benchmarks along with qualitative evidence.
- •
We provide extensive ablation studies to analyze the contribution of each component in SiNGER and validate the robustness of our framework.
- •
We demonstrate through extensive experiments that our method exceeds baseline performance across tasks and produces more interpretable feature maps.

## 2 Related Works

Vision Transformers.
ViTs (vit) underpin many VFMs and have become the representative architecture for large-scale visual learning.
Unlike convolutional networks (resnet; mobilenet), ViTs rely on self-attention with fully connected layers.
Since ViTs form the architectural core of most VFMs, studying them provides representative insights that generalize broadly.
Similarly, parameter-efficient tuning methods such as LoRA (hu2021lora) highlight how minimal perturbations can effectively adapt VFMs, a perspective that motivates our artifact-suppressing perturbations.
However, the quadratic complexity of the ViT architecture limits the practicality of large models despite their advanced representation.

Knowledge Distillation.
KD compresses models by training a smaller student to mimic a larger teacher (kd-hinton).
Among various approaches, FitNet-style methods (kd-feature-01) that align intermediate features are especially influential, as they encourage the student to learn useful representations beyond logits (kd-response-01).
Later extensions incorporated relational structures (kd-relation-01) or multi-teacher settings (kd-multi-teacher-01), while adaptations for ViTs (deit3) aimed to respect their architectural characteristics.
Despite these advances, distillation applied to VFMs often inherits undesirable properties from teachers, revealing the need for methods that improve not only compression but also the quality of transferred representations.

Artifacts in Transformers.
Artifacts are a recurring issue in transformer models, degrading the representation quality.
vit-needs-registers demonstrated that ViTs produce high-norm artifacts, particularly in background regions, harming interpretability and dense prediction.
Practical suppression strategies include register tokens (vit-needs-registers), and recent work argues these artifacts arise from power-method-like accumulation across residual layers, aligning tokens with the leading left singular vector  (gandikota2023sinder). In the knowledge distillation domain, ViTKD (kd-vitkd) randomly masks teacher features to reduce the mimicking of high-norm artifacts. However, such indiscriminate masking also removes informative inlier signals, motivating artifact-aware KD that suppresses high-norm artifacts while preserving inlier structure.

## 3 Method

### 3.1 Problem Formulation

#### 3.1.1 High-Norm Outliers in Vision Transformers

In large ViTs, a non-negligible fraction of patch features in

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. Clear and well motivated problem formulation. The paper starts from a real and now increasingly reported issue in ViTs: high-norm background/low-semantics tokens that get amplified across residual blocks and hurt both interpretability and distillation. The link they make (artifact dominance -> gradient bias in feature KD) is very sensible and well motivated. 

2. Principled mechanism. Instead of just masking teacher features (like ViTKD), they try to change the teacher supervision itself in a way that is theoretically tied to the next layer: perturb only in (approximate) nullspace so the next block’s output is preserved. This is a neat way to reconcile "suppress artifacts" vs "don't destroy useful signals".

3. Low-intrusion design. Using a LoRA-based adapter to implement the refinement is pragmatic: it's lightweight, it doesn't require re-training the whole teacher, and it aligns with current practice on parameter-efficient tuning.

4. Good ablations. The nullspace initialization vs random init, the presence/absence of the information-preservation loss, and the small hyperparameter sweep on rank r and percentile a all support the central claim.

5. Multi-task evaluation. Even though the numbers in Table 1 are a bit hard to read, the intent is good: they try to show that better, less-artifact supervision gives students that transfer better to unrelated tasks (segmentation, depth), not just ImageNet linear eval.
summary: The paper tackles a well-observed problem in Vision Transformer (ViT) distillation: high-norm "artifact" tokens in large ViTs dominate feature-level KD losses, so the student ends up overfitting to these outliers instead of learning the informative inlier structure. To address this, the authors propose SiNGER (Singular Nullspace-Guided Energy Reallocation), a distillation framework that first refines the teacher’s features and only then distills them. The key idea is to add a LoRA-style low-rank adapter to the teacher and to constrain its perturbations to (an approximation of) the left nullspace of the next block, so that: (i) high-norm outliers are suppressed, and (ii) downstream information is preserved because the perturbation lies in a direction that doesn't change the next layer's output. They couple this with two auxiliary losses: an outlier suppression loss (to explicitly shrink high-percentile tokens) and an information-preservation / Gram-matching loss (to keep patchwise relations intact). Experiments on several downstream tasks (ImageNet-1K, ADE-20K, NYUd-v2, domain shift benchmarks, fine-grained datasets) show that SiNGER improves over FitNet and ViTKD when distilling a ViT-Large to a ViT-Tiny, and produces visually "cleaner" feature maps with more uniform patch norms.
weaknesses: 1. Central assumption relies on a rough linearization. The core trick (perturb in the left nullspace of the next block so the next block's output is unchanged) is only feasible because they linearize the non-linear transformer block using the FFN weights and then pick the smallest singular vectors. But real blocks have attention, non-linearities, layernorm, residual couplings, and cross-token interactions. The paper cites an appendix showing FFN is the dominant source of norm amplification, but in the main text this is still an approximation, not a guarantee. A reviewer could fairly ask: how often does the adapter actually stay in the true (nonlinear) null directions during training? The ablation in Table 4a helps, but it’s still empirical. 

2. Evaluation scope is narrow. All main results are ViT-Large to ViT-Tiny, canonical ViT, every-second-layer alignment. That is a single, quite specific distillation path. There is no evidence it works for: (i) teacher and student of different families (e.g., DINOv2 to DeiT-Small), (ii) CNN to ViT, or (iii) same-size or near-size distillation where artifacts might behave differently. This makes the generality claim ("artifact-aware KD for VFMs") weaker than it sounds.

3. Choice of baselines. They compare to FitNet and ViTKD, but they do not compare to the most natural artifact-aware teacher-side fixes like register tokens (Darcet et al., 2024), or to SINDER / DINOv3-style cleaned teachers as sources for distillation--both of which are already in their related work. If the claim is "we are the principled way to keep info while suppressing artifacts", then we need to see: what if you just gave the student a cleaner teacher in the first place? Right now that experiment is missing. 

4. Table 1 is confusing / formatting issues. The version here has lines that appear interleaved or misaligned (e.g., "SiNGER 70.59 8.16 21.76 3.03 ..."), which makes it hard to confirm the exact gains on each benchmark. For a distillation paper, exact numbers matter. This hurts Presentation. 

5. Cost and practicality not fully discussed. Attaching adapters to multiple teacher layers and running SVD-style initialization on (linearized) FFNs is not free. The paper doesn’t quantify the extra compute / memory during distillation vs. plain feature KD. This is important for practitioners.

6. Long-tail result potentially goes in the wrong direction. The iNat2019 result is weaker and is brushed off as “long-tail is hard,” but that’s precisely a setting where better supervision and less artifact noise should help the student pick up minority classes. A short analysis or per-class breakdown would strengthen the story. 

7. Causality of the improvement is still a bit entangled. Three things change at once: teacher refinement, outlier loss, and Gram matching. We do see an ablation on Linfo, but we don’t see, for example, "plain LoRA adapter without nullspace init" vs "nullspace init" vs "only percentile shrinkage" on all tasks, just on a subset. It’s still plausible that a well-tuned teacher-side LoRA denoiser would get close.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: 1. Nullspace Verification: 
2. Can you scale to Larger Models?
3. Ablation on Linearization: What happens if you use the full nonlinear block's Jacobian
soundness: 2
strengths: 1. Well-Motivated Problem: The paper identifies a concrete limitation in ViT distillation—high-norm artifacts dominate gradient flow and harm student learning.
2. Comprehensive Evaluation: The multi-task evaluation spanning 10 benchmarks is thorough. Consistent gains on ImageNet and ADE-20K
summary: This paper addresses the problem of high-norm artifacts in ViTs that degrade knowledge distillation effectiveness. The authors propose SiNGER, a distillation framework that refines teacher features by applying perturbations guided toward the nullspace of subsequent layers. The key innovation is using LoRA-based adapters initialized with approximate nullspace vectors to suppress artifacts while preserving informative signals. Experiments demonstrate improvements across multiple downstream tasks including classification, segmentation, and depth estimation.
weaknesses: 1. Approximation Gap: The core theoretical claim assumes linear blocks, but ViT blocks are highly nonlinear.
2. Incomplete Comparison with ViTKD

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: How do you select the nullspace dimension or perturbation strength in practice? Are there hyperparameters, and how sensitive is performance to them (e.g., ablation results)?
soundness: 3
strengths: Principled Solution to a Core Problem: Identifies a fundamental limitation in ViT KD (artifact dominance) and resolves the trade-off between suppression and information preservation via nullspace guidance, leading to more effective knowledge transfer.

Efficient and Minimalist Design: The LoRA adapter requires little structural change, making it easy to integrate into existing KD pipelines while maintaining computational efficiency.
summary: The paper introduces SiNGER (Singular Nullspace-Guided Energy Reallocation), a knowledge distillation (KD) framework designed to improve the compression of Vision Transformers (ViTs) by addressing high-norm artifacts in teacher representations. These artifacts, arising from power-iteration-like accumulation in ViT layers, dominate the distillation objective, causing students to overfit to noise and underutilize informative signals. SiNGER refines teacher features using a lightweight LoRA-based adapter that applies nullspace-guided perturbations to suppress artifacts while preserving downstream-relevant information. The refined features are then distilled to smaller student models.
weaknesses: ViT-Specific Focus: The method is tailored to ViT artifacts (e.g., from self-attention scaling); its generalizability to other architectures like CNNs or hybrids isn't explored.

Dependency on Teacher Quality: Assumes well-pre-trained ViT teachers; if artifacts are severe or pre-training is suboptimal, refinement might not fully compensate.

Scope Limitations: Primarily evaluated on downstream vision tasks; extensions to multimodal or non-vision transformers aren't discussed.


Decision: Accept (Poster)