SiNGER: A Clearer Voice Distills Vision Transformers Further
Authors: Geunhyeok Yu
Sunjae Jeong
Yoonyoung Choi
Jaeseung Kim
Hyoseok Hwang
Venue: ICLR 2026 Poster (2026)
Keywords: Vision foundation models
model compression
knowledge distillation
representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose SiNGER, a nullspace-guided LoRA framework that suppresses artifacts in Vision Transformer distillation while preserving informative representations, achieving state-of-the-art student performance.

Abstract: Vision Transformers are widely adopted as the backbone of vision foundation models, but they are known to produce high-norm artifacts that degrade representation quality.
When knowledge distillation transfers these features to students, high-norm artifacts dominate the objective, so students overfit to artifacts and underweight informative signals, diminishing the gains from larger models. Prior work attempted to remove artifacts but encountered an inherent trade-off between artifact suppression and preserving informative signals from teachers. To address this, we introduce Singular Nullspace-Guided Energy Reallocation (SiNGER), a novel distillation framework that suppresses artifacts while preserving informative signals.
The key idea is principled teacher feature refinement: during refinement, we leverage the nullspace-guided perturbation to preserve information while suppressing artifacts.
Then, the refined teacher's features are distilled to a student. 
We implement this perturbation efficiently with a LoRA-based adapter that requires minimal structural modification.
Extensive experiments show that SiNGER consistently improves student models, achieving state-of-the-art performance in multiple downstream tasks and producing clearer and more interpretable representations.

Published as a conference paper at ICLR 2026
SINGER: A CLEARER VOICE DISTILLS VISION TRANSFORMERS FURTHER
Geunhyeok Yu! Sunjae Jeong!22*
Yoonyoung Choi'
Jaeseung Kim? Hyoseok Hwangh
Kyung Hee University MOBILTECH CO., LTD
fgeunhyeck, choiyy0313, sunj, byoseok ack ac. jason. him.momimmmbbbtccch...c
ABSTRACT
Vision Transformers are widely adopted as the backbone of vision foundation models, but they are known to produce high-norm artifacts that degrade represen- tation quality, When knowledge distillation transfers these features to students, high-norm artifacts dominate the objective, so students overfit to artifacts and un- derweight informative signals, diminishing the gains from larger models. Prior work attempted to remove artifacts but encountered inherent trade-off between artifact suppression and preserving informative signals from teachers. address this, introduce Singular Nullspace-Guided Guided Energy Reallocation (SiNGER), novel distillation framework that suppresses artifacts while preserving informative signals. The key idea is principled teacher feature refinement: during refinement, we leverage the nullspace-guu guided perturbation to preserve information while sup- pressing artifacts. Then, the refined teacher's features distilled student. We implement this perturbation efficiently with a LoRA-based adapter that requires minimal structural modification. Extensive experiments show that SiNGER con- sistently improves student models, achieving state-of-thee ofethhrart performance in mul- tiple downstream tasks and producing clearer and more interpretable representa- tions. Code is available at https: //////gii thub. com Com/AAILABKku NGER.
ImageNes-1K
Irput
+440
gramed 10.5%
Segmerration
Festures
SENCEE
+4.0%5
+8.7%
ImageeNeei
Towches
Student
Toacher
Rofiemment
Bawelines
Long
SINGER
(a) Overview of SiNGER distillation.
(b) Performance gains using SiNGER.
Figure 1: SiNGER suppresses artifacts and enhances transfer. (a) Feature visualizations highlight clearer and more interpretable representations. (b) Radar chart shows consistent multi-task gains.
INTRODUCTION
Transformers have become the de facto standard architecture in both research and industry due to
their scalability and effectiveness (Oquab et al., 2024; Radford et al. 202I). Their token-based self-attention mechanism is broadly applicable with minimal inductive bias (Lu et al. 2022). and
has enabled significant advances in computer vision and machine learning (Kim et al. 2024] Sariy- iidiz et al. 2024). Vision Transformers (ViTs, Dosovitskiy al. (2021) extend this paradigm to visual data and form the backbone of Vision Foundation Models (VFMs). Compared to convolu- tional networks, VITs rely less spatial inductive biases and instead exploit scale to achieve high
"Equally contributed. Corresponding author.

---

Published as a conference paper at ICLR 2026
160
180
35
140
175
170
120
165
25
100
160
135
20
FitNet (Romero et al.
2015
VITKD Yang et al. 2024
SINGER (ours)
Input
ViT-Large
ViT-Tiny
FitNet
VITKD
SINGER
Figure 2: Qualitative analysis. Row 1: KD method comparison. Left: distilled feature map col- ored by patch norm, Right: patch-wise cosine similarity to the teacher. Row 2: Input image, two pretrained ViTs, and three VIT-L VIT-T distilled variants. Each panel shows similarity from the x-marked patch. SiNGER most closely preserves teacher semantics, showing the most coherent teacher-consistent similarity patterns.
performance. ViTs are also highly scalable since supervised or self-supervised training produces increasingly generalizable representations (Oquab et al.] 2024 Touvron et al.) 2022). However, the quadratic complexity of self-attention severely limits the practicality of scaling ViTs. This tension between accuracy and efficiency motivates the study of compression. Pruning (Yang et al. 2023)
and quantization (Liu et al. 2021) have been explored. but pruning often fails to deliver practical speedup due to structural rigidity (Aghli & Ribeiro! 202T), and quantization can induce numerical instability (Jiang et al. 2024 laved et al. (024). Knowledge distillation (KD, Hinton et al. (2015)
has emerged as the most reliable solution for transferring knowledge from large ViT' smaller
students. KD methods span diverse targets and frameworks (Sun al. 2024 Romero et al. 2015
Ranzinger et al. 2024), consistently yielding structurally and numerically y stable compact models.
Nevertheless, KD for ViTs suffers from subtle but critical limitations in their representation space. Darcet et al. (2024) revealed that ViT token representations contain high-norm artifacts. WWan et al. (2025) argue that these artifacts singular defects induced by power-iteration-lie like accumulation across residual blocks, whereby tokens align with the leading left singular vector of the pre-trained weights. These artifacts interact poorly with the standard feature mean squared error objective in KD: when the teacher and student are matched, gradients concentrate on the few high-norm tokens, producing outlier-driven optimization bias that obscures informative signals in the inlier structure. Therefore, suppressing outlier norms in teacher features is essential for KD in ViTs as the scale grows. Prior work mitigated this issue via random masking of teacher features Yang et al.) 2024); however, this inevitably removes informative signals. Therefore, key challenge is to mitigate these artifacts without losing valuable information, fundamental trade-off that requires a principled approach.
To resolve this trade-off, we introduce a nullspace-guiee suppression: we modify only the nullspace component in the teacher features, mathematically, the subspace orthogonal to the downstream space. This yields student-optimal supervision by suppressing artifacts without sacrificing infor- mative signals. Based this insight, propose Singular Sallspace-Guided Energy Reallocation (SiNGER), a framework that addresses this trade-off in ViTs distillation, illustrated in Figure Ial To minimize the modification the teacher's signal, attach lightweight LoRA-basee adapter (Hu et al. 2022) to the KD architecture, which refines the teacher features. The adapter produces a min- imal perturbation guided toward the left-nullspace of the next block, suppressing high-norm outliers while leaving the next block output unchanged. Our method achieves superior performance com- pared to baselines across multiple downstream tasks (Figure lb). It also produces more structured and interpretable feature maps (Figure 2 Our contributions are summarized as follows:
We propose a novel distillation framework (SiNGER) that refines teacher signals via the LoRA- A-baeed adapter with nullspace initialization guide effective perturbations.
2

---

Published as a conference paper at ICLR 2026
We analyze fundamental limitation of naive VIT distillation, showing degraded transfer on downstream benchmarks along with qualitative evidence.
We provide extensive ablation studies to analyze the contribution of each component in SiNGER and validate the robustness of our framework.
We demonstrate through extensive experiments that our method exceeds baseline perfor- mance across tasks and produces more interpretable feature maps.
RELATED WORKS
Vision Transformers. Visual transformers (Dosovitskiy et al. 202 Touvron et 2022) underpin many VFMs and have become the representative architecture for large-scale visual learning. Unlike convolutional networks (He et al.] 2016) Howard al., 2017). ViTs rely on self-attention with fully connected layers. Since ViT' form the architectural core of most VFMs, studying them provides representative insights that generalize broadly. Similarly, parameter-efficient tuning methods such as LoRA (Hu et al., 2022) highlight how minimal perturbations can effectively adapt VFMs, per- spective that motivates our -upprettsuppressing perturbations. However, the quadratic complexity of the VIT architecture limits the practicality of large models despite their advanced representation.
Knowledge Distillation. KD compresses models by training a smaller student to mimic a larger teacher (Hinton al., 2015). Among various approaches, FitNet-style methods (Romero et al. 2013) that align intermediate features are especially influential, as they encourage the student to learn useful representations beyond logits (Sun et al., 2024). Later extensions incorporated rela- tional structures (Park et al., 2019) or multi-teacher settings (Ranzinger et al.] (024)) while adap- tations for VITs (Touvron et (022)) aimed respect their architectural characteristics. Despite these advances, distillation applied to VFMs often inherits undesirable properties from teachers, re- vealing the need for methods that improve not only compression but also the quality of transferred representations.
Artifacts in Transformers. Artifacts are recurring issue in visual transformer models, degrading the representation quality. Darcet et al. (2024) demonstrated that ViTs produce high-norm artifacts, particularly in background regions, harming interpretability and dense prediction. Practical suppres- sion strategies include register tokens (Darcet et al.] 2024), and recent work argues these artifacts accumulation across residual layers, aligning tokens with the leading left singular vector (Wang et al., 2025). In the knowledge distillation domain, VITKD (Yang al. 2024) randomly masks teacher features to reduce the mimicking of high-norm artifacts. However, such indiscriminate masking also removes informative inlier signals, motivating artifact-aware KD that suppresses high-norm artifacts while preserving inlier structure.
arise from nower-method--
3 METHOD
3.1 PROBLEM FORMULATION
3.1.1
HIGH-NORM OUTLIERS IN VISION TRANSFORMERS
In large ViT', non-negligible fraction of patch features in FF exhibit high-norm artifacts (outliers). Their prevalence and magnitude increase with model capacity, as |Dacet et al. (2024) reported. This particularly consequential for distillation from larger teacher a smaller student: artifact-prone teacher features introduce a systematic imbalance at the feature level and can obscure informative signals.
3.1.2 KNOWLEDGE DISTILLATION OBJECTIVE
Let FT E Ra and FS e x denote teacher (T) and student (S) features at layer We align dimensions d + d' with trainable projection P1 Id Bd and define the feature-level KD loss as follow:
1 Σ
LKD.1
FT
P.
(1)
n
t=1
3

---

Published as a conference paper at ICLR 2026
PT
P
Linto
W!
& &a
AFT
W
[+1
FT
Teacher flow
PI
LKD Loutlier
SINGER flow
Student flow
W
FPS
PP
Adapters
Figure 3: The overall pipeline of knowledge distillation with the SiNGER adapter at Ith layer..
where FT, and Fi, denote the i-th patch feature and is the number of patches and 2--nrm.
means the
3.1.3 OUTLIER DOMINANCE AND GRADIENT BIAS
Partition the patch indices into an outlier set O2 and an inlier set Iz to obtain
LKD =
Σ IED2
PTT P
(2)
JET.
Outlier Term
Inlier Term
By construction, for i E O we have [
| for j E Z, Hence, when the residual
magnitudes are of similar order across patches, the outlier term dominates both the objective and its gradients. In particular,
V PICET, CKD. 2 (P((F))))) - P(.)
(3)
T.
outliers induce proportionally larger updates. Optimization is therefore biased toward mimicking a few high-norm outliers, rather than consolidating the majority inlier structure that carries most of the informative signals. This gradient bias disrupts the learning of the dominant inlier representation and leads to suboptimal transfer. We therefore seek to refine the teacher features F at layer / before distillation, so that they are more conducive transferring informative signals the student.
3.2 SINGULAR NULLSPACE-GUINI ENERGY REALLOCATION
We consider KD at layer 1, where high-norm outliers in FT induce gradient bias toward few tokens. To prevent this, the outlier term in Equation (2) must be weakened, which practice means reducing the norm of outlier patches in F/T However, naive shrinkage erodes information carried by the larger teacher and can nullify the benefits distillation.
3.2.1 PERTURBATION ON NULLSPACE
Let F R and define refined feature map F/T F. AFF. Our two objectives
1. Suppress Outlier Norms. Reduce the norm of high-norm patches in F.T (Figure 4a
2. Preserve Information. Ensure that when the modified features are fed into the next teacher block, the conveyed information not altered (Figure 4b).
Consider the next block at layer I+1 with transformation W4 E xd' next-block output if and only if
xd" Then FT preserves the
(FF AFF W41
M. Wi+
AF W41 0.
4

---

Published as a conference paper at ICLR 2026
A perturbation AFT that satisfies the above is obtained by restricting it to the left-nullspace of the next block Wi+1. Let N++1 Null((W(+1)) denote this left-nullspace. Then the requirement as follows:
row AFF) C N1+1-
Consequently, to allow effective distillation, refine the features of the teacher FJ to Ff by perturbation guided to the left-nullspace N141.
Input
|
|
||
Input
F4+1
(a) Outlier suppression with the proposed adapter.
Information preservation after the (+ layer.
Figure 4: Two objectives of SiNGER; (a) outlier suppression and (b) information preservation.
| in (a) signed with the cosine-similarity between ArT and F/F In (b), cosine similarity between x x-marked patch and every patch of another feature map is visualized.
3.2.2 ADAPTER-BASER FEATURE REFINEMENT
We refine FT by adding a low-rank perturbation produced by a MoRR-bbsee adapter while freezing all teacher weights.
FT FFF AFF (Fl (down,)) Pup.is
where @down.! E Rdd x uup. BRexd" and rdd
To bias AFT toward the left-nullspace N4+1 of W1+1, set the initial weights of adapter,
@down. == N411 @up.1 := N
(7)
This initialization guides the optimization to remain near N4+1 and find solutions that satisfy the two objectives. Because the next block is nonlinear, its exact nullspace cannot be obtained via SVD. We adopt practical linearization of the next block, W4+1 s WH41, and define N as the left singular vectors associated with the smallest singular values of W411 By construction, N441 collects the left singular vectors corresponding to the smallest singular values of Wi+1, so [| W++ll 0d-r+1- Moreover, Appendix A provides detailed spectral analysis (sublayer perturbations, singular value diagnostics, and s-null bounds) showing that the same approximate- null relation holds for the nonlinear block. Consequently,
Ñu Dup. N
downn,
3.3 KNOWLEDGE DISTILLATION WITH SINGER
Figure summarizes the pipeline: SiNGER refines teacher features at selected layers before feature matching. Let D limer U (mima)} denote the distillation layers, where limer is set of intermediate layers and Linal the final layer. For each I D, an adapter di transforms FT into ET Training is guided by three losses, aggregated over D.
Knowledge-Distilatton Loss. The student is trained to mimic the refined teacher with CKD-
LKD Σ MME (FT,, P((FF)).
Outlier Suppression Loss. Adapters are explicitly encouraged suppress high-norm artifacts. For each I, let O be the indices of patches in F." whose norms exceed the a-percentile Ja.1 as Equation T0
Loutlier
Σ IED
(I (o..)
(10)
Oi
IEO,
5

---

Published as a conference paper at ICLR 2026
Information Preservation Loss. To retain informative signals while suppressing norms, align feature directions via Gram matching. Define
MSE (G(F)4)) G(F LE linters
Linfo.1
(11)
MSE (G(FF), G(FF)). 1 limal-
where G(FF) denotes the Gram matrix of F. This preserves the directional structure passed to the next block for intermediate layers, and preserves the final-layer structure at Limat- Consequently, the total information term is Linte EIED Linto,!-
Training Objective. We jointly optimize the student parameters 0s, projection parameters Op = (PiheD, and SiNGER adapter parameters 0s (9down,ts Pup, heD with single weighted sum loss:
Limal LKD + Aoutier Loutler Xinto Lintor
(12)
where Aomtler and Xinta balance artifact suppression and information alignment. This objective en- courages effective transfer while controlling high-norm artifacts in teacher features.
EXPERIMENTS AND ANALYSIS
4.1 DETAILS
Downstream Tasks. To evaluate LINGER-distilae VIT as VFM, we adopt the student network to a diverse set of downstream tasks. Specifically, we consider six representative benchmarks: ImageNet-IK validation set for large-scale classification (Deng et al. 2009), AOK-20K for semantic segmentation (Zhou et al. 2019). NYUd-v2 for depth estimation (Ignatov et al. 2024), iNaturalist- 2019 for long-tail classification (Vn Horn et al. 20188), ImageNet-R and imageNet-ve for domain shift robustness (Hendrycks et al. |021 Recht et al. (019), and four fine- fine-grain grained classification datasets (Maji al. 2013 Parkhi al. 2012 Bossard et al. 2014 Nilsback & Zisserman 2006).
Distillation Setup. We evaluate SiNGER on multiple teacher- configurations spanning both the canonical ViT (Dosovitskiy et al. (02T) and the modern DeiT-IIII (Touvron et al., (0022) covering a range of model scales. The detailed architectural specifications are summarized in Appendix B Student layers are aligned with every second teacher layer. The official implementation of FitNet and VITKD employs task-specifee loss objectives, such as cross-entropy minimization for classification. In contrast, we target VFM distillation and therefore exclude task specifie losses, distilling the last hidden layer's representation.
Rationale. We aim to probe pre-training agnostic mechanisms of artifact formation and suppres- sion. To this end, we conduct the ablation suite on canonical ViT's, whose transparent design full and widely adopted training recipe allow tighter control, clearer causal attribution, and more repro- ducible analysis.
4.2 MULTI-TASK EVALUATION
[abble[ssummrizze multi-task linear evaluation results across ten benchmarks. The teacher (Large) achieves strong performance, while the Tiny baseline shows significant degradation, particularly on dense prediction tasks; ADE-20K and NYUd-v2. FitNet improves over the Tiny baseline by transfer- ring intermediate features, but still inherits artifacts from the teacher, limiting overall gains. ViTKD performs poorly across all tasks in most cases, as its random masking strategy often collapses fea- ture representations and prevents effective learning. Distillation across diverse teacher-student pairs is discussed in Appendix C and additional discussion on VITKD is detailed in Appendix D
By contrast, SiNGER demonstrates consistent improvements over FitNet and ViTKD on most benchmarks. On IN-val, ADE-20K, NYUd-v2, DS, and FG, SiNGER yields large gains, approach- ing teacher performance despite the smaller capacity. The only exception iNat2019, where per- formance slightly drops compared to the non-distilled student-size models, which we attribute to the long-tail nature the dataset, as ZZhan et al. (0023) pointed out. We report an entropy-driven analysis 1Nat2019 in Appendix J Overall, these results confirm that suppressing artifacts dur- ing distillation produces student models that are both more accurate and more generalizable across diverse tasks.

---

Published as a conference paper at ICLR 2026
IN-val lop-1
ADE-20K
NYUd-12
INa22019
DS
FG
Model
Distillation
mleU (1)
RMSE (4)
top-1(()
top
top (t)
VIFL VIT-S VIFT
79.58
26.57
0,9157
71.42
54.20
82.27
(Non-distille))
76.05
19.57
1,1065
65.28
44.48
77.39
38.03
14.20
1.1807
43.95
28.75
62.02
VIT-L VITS
FitNer
72.50
25.15
0.9903
52.12
40.21
71.13
SINGER
79.13
30.06
0.90226
57.21
48.91
77.59
+6.63
+4.91
+0.0877
+5.09
+8.70
+6.46
VIT-L VIFT
FüNer
62.43
40,02
VITKD
5.07
11.92
1.1903
23.69
2,08
33.52
+8.16
+3.03
+0.0687
+1.09
+6.55
+2.13
DeiT-III-L
84.10
26.11
1,2311
57.59
56,95
75.35
DeiT-IIIBB
(Non distilled)
82.50
25.14
1.1781
56.41
54,73
74.65
DeiT-III-S
78.90
20.28
1.2961
45.39
49.50
67.84
DeiT-III-L
VITKD
66.54
19.58
1.2525
30.78
35.77
58.36
SINGER
79.37
29.47
1.1514
53.50
49.14
75.41
+12.83
+2.68
+0.0111
+3.46
+13.37
+0.63
DeiT-IlI-S DeiT-III-T
FitNtt
$1.41
16:06
1,2920
32.81
23.74
58.53
VITKD
35.56
8.71
1.4487
19.95
15.31
45.13
37.68
+12.09
+4.61
+0.3093
+4.87
+8.54
+5.79
Table 1: Multi-task linear evaluation results. ImageNet-1K validation (IN-val) for large-scale classi- fication, ADE-20K for semantic segmentation, NYUd-v2 for monocular depth estimation, iNatural- ist2019 (iNat2019) for long-tail learning, ImageNet-R and ImageNet-v2 for domain shift (DS), and four fine-grained classification (FG) benchmarks: FGVC-Aircraft, Oxford-IIIT Pet, Food-101, and Flowers-1022 tested. A rows indicate the performance gains of SiNGER, computed against the best-performing baseline among the distilled students (underlined).
4.3 REPRESENTATION QUALITY
We assess the quality and interpretability of distilled representations by comparing the feature maps and their Gram matrices. Figure depicts the Gram matrices of the feature maps. Quantitatively, SiNGER's Gram matrix is the most similar one to the teacher's Gram matrix. Gram Distance (GD), defined & distance between the Gram matrices, confirms this trend Figure 2 This shows that when artifacts are distilled, it disrupts the transfer of patch-wise relation, resulting in degraded stu- dent representation. Centered Kernel Analysis (CKA, Kornblith al. (2019)) measures the linear correlation between two feature maps. FitNet and ViTKD achieve higher similarity by following the teacher too closely, but this reflects replication of artifacts rather than useful knowledge trans- fer. By contrast, SiNGER learns structurally consistent yet information information-preservvd preserved representations, balancing similarity with the teacher.
depicts
Metric
FitNet
VITKD
SINGER
GD
0.237
0,520
0.130
Gram matrices of the patches. The input, the teacher, and distilled features; FitNet, ViTKD, and SiNGER in order.
Figure 5:
Table The teacher-student representa- tion's 's similarity in terms of &g distance be- tween the Gram Distance (GD) and CKA.
4.4 ADAPTER OPERATION
We empirically analyze how the optimized adapter operates on ImageNet-1K. KK. To probe the coupling with the next layer, we evaluate at an intermediate layer = 17.
Patch-Norm Distribution Between F and É.. visualize the distribution patch lo norms for F, and F, with side-by-side box plots (Figure 6). The teacher produces high-norm artifacts that are distinctly gathered group. We observed that SiNGER effectively draws such artifacts into the

---

Published as a conference paper at ICLR 2026
normal-patch range while preserving informative features. This results in stabilized gradient flow through the normal patches.
Cosine Similarity Between Fi+1 and Fi+1. To assess information preservation, we compute patch- wise cosine similarities for both Fj vs. Fj and Fz+1 vs. Pi+1, aggregating per image across the dataset (see Table 3 The 17, 18-th layers yield cosine similarity of 0.9566 and 0.9731 with negli- gible variance, respectively, which are clearly very similar.
250
300
350
400
450
Pair
fa
median
F17 F17
0.9566
0.0038
0.9569
||A||
F18 Fis
0.9731
0.0021
0.9732
50
200
400
22-m0rm
Figure 6: Patch-norm distributions of Fi and Pt. Artifacts are scale-colored separately with inferno.
Patch-wise cosine similarity (per- image mean over patches) on ImageNet-1K at 1=17 and 1=1-18.
Table 3:
4.5 ABLATION STUDY
We report four ablations, focusing on initialization, losses, hyperparameters, and distillation layers. Note that all experiments reported Table 4 were conducted using a subset of ImageNet- 1K.
Initialization Method. We validate whether nullspace initialization truly guides the adapter to induce perturbations along the nullspace dur- ing optimization by comparing nullspace-biased (SiNGER) and random initializations.
NYUd-v2
Initialization
Couati
Latk
top-1 (?)
(4)
Random
10.83
4.35
1.3048
Nullspace
10.57
4.32
1.1943
Let the next block layer 1+1 be linearized to W++1 DDD, From W1+1, define rank-r bases: the principal basis Pt+1 Dxr (largest singular directions) and the null basis Ni-1 E DDx (smallest singular directions). We quantify alignment with the normalized Frobenius norm
Nullspace
Nullspace
11.06
4.38
1.1870
Nullspace
11.57
4.99
1.1690
Table 4: Full ablation study the nullspace ini- tialization and loss terms.
16 Pitills
| N141ll
Peprob (+)
Fanfo(()
V& {ouppt (down.)
(13)
|
|0l1,
larger Frobenius norm indicates stronger alignment of the trained adapter matrix (dup. Pdown.)) with the corresponding subspace; in our design, the primary goal is to increase Egate (alignment to Ni+1).
In Table 5a initialization markedly increases alignment to N141 Esate reaches 0.83/0.76 for Φup, 1-17233, and 0.55/0.58 for @sown, Both are under 0.27 for random initialization. This provides strong evidence that the initialization guides optimization into the nullspace, yielding substantially higher Exate across layers and for both @up.1 and downn, which indicates successful guidance toward the nullspace directions. Mcanwhile, Eprob remains lower comparable under SiNGER, but objective is not to minimize Eprob per se; rather, to ensure that the learned parameters predominantly occupy N nullspace initialization stabilizes the refinement direction and avoids perturbations that conflict with the subsequent block, it does not lead to measurable performance improvements (Table 4
suppress high-norm amplification while preserving useful directions. Although
Loss Term. We ablate the loss design to verify the role of information preservation. Our full ob- jective uses both outlier suppression Loulter and information preservation Linfo whereas the ablated
8

---

Published as a conference paper at ICLR 2026
Layer
Matrix
Init
Eprab
Buata
Pus.
Random
0.2565
0.2532
Pair
Custter
Cut
sid
median
17
SINGER
0.1479
0.8337
Pup,
F 44 f.
14.22 1: 1.45
14.28
23
Pus.i
SINGER
0.1833
0.7589
p + f.
7.25 +: 0.84
7.19
(towa,
0.3100
F F.
/
72.36 = 7.61
71.85
SINGER
0.2847
0.5485
p 44
p.
41.71 1: 7.01
40.89
23
(dows,
Random
0.3025
0.2641
23
(tows.
SINGER
0.2746
0.5774
(a) Initialization methods.
(b) Information preservation term.
Table 5: Ablation studies on the initialization method and the information preservation loss.
variant uses Loutteer only. (Using Linto alone admits the trivial solution |||-00 and yields no up- dates.)
To assess preservation of teacher information, measure the Gram distance between FT and FT. Additionally, to evaluate the final effect on distillation, we measure how well the student features 1 preserve teacher relations by comparing Ss against FT, Distances are computed per image and summarized over ImageNet- IK. For feature map F, let G(X) FF and define
Da FJ) = G[)) G I|,
In Table 5b lower De indicates better preservation of pairwise feature relations. Compared to Coutier alone, adding Linto nearly halves the DG distance (14.22 + 7.25) and substantially improves teacher-studen alignment (72.36 41.71). Thus, the information preservation term prevents de- generate updates and maintains the relational geometry that is crucial for effective transfer. Table 4 shows that adding Loutier yields the largest improvement both ImageNet- I and ADE-20K, directly mitigates the dominant artifact tokens that bias distillation. Litto provides additional gains by enforeing information consistency between the teacher and student. When all components are combined, the student reaches its best performance across all tasks, demonstrating that SiNGER functions most effectively as an integrated framework rather than as a set of independent mecha- nisms. These results highlight the individual contribution of each component.
Hyperparameter Sensitivity. Two hyperparame- ters are required SiNGER: a and r. a deter- mines the strictness of artifact filtering by setting the percentile threshold based on the Gaussian- like distribution of the patch norms |[ across
RMSE (1)
61.25 (1)
8
1.4555
33.97
16
33.79
32
1.4907
33.07
64
1.6485
28.91
r controls the capacity of the perturbation ATT applied to F". A larger allows the adapter to explore a wider subspace, but may distort the se- mantic structure of the features. Conversely, if too small, the limited degrees of freedom restrict the adapter from effectively suppressing artifacts, while also risking the loss of informative compo- nents.
(a) Rank sweep with 0.95.
RMSE (L)
51.25 (t)
0.90
1.5989
29.78
0.95
1.4395
33.79
0.97
1.4748
33.66
0.99
1.5321
30.80
Table (6 reports the sensitivity of a and on NYUd-v2. We observe that performance degrades when is too small or too large, confirming the need for balanced capacity. Similarly. extreme values of either under-filter artifacts discard informative signals. The observed trends match theoretical intuition: performance improves
(b) Quantile threshold sweep with 16.
Table 6: Rank and quantile threshold sweeps on NYUd-v2. We conduct grid search over candidate values and select the configuration that yields the best performance.
when artifact suppression and information preservation are balanced, but deteriorates when either dominates. At the same time, the results show robustness robustness-perfommncce does not collapse outside the optimal point, indicating stability of the framework. Finally, the chosen hyperparameters (r 16 and a = 0.95) generalize well across other tasks and datasets, and we adopt them as the default configuration.

---

Published as a conference paper at ICLR 2026
Distillation Layers. Selecting the distillation layers is criti- cal because we aim to distill artifact-pront prone features. To en- sure gradients traverse the entire backbone, we always distill the last layer (1 = 23 in ViT-L). Beyond this, we select an additional intermediate layer by inspecting teacher feature trends (see Appendix H). Since our method is an artifacts- aware approach, we first pinpointed the location where ar- tifacts occur. For ViT-L, we observed that artifacts appear after 11. We additionally
Layers 17
NYUd-v2
23
RMSE (L)
0.9554
/
0,9406
0.9624
Table 7: Distillation layer selection.
select the intermediate layer at = 17. Across three variants, the / = 17, 23 configuration performs best, shown in Table 7
DISCUSSION
Approximation Gap. While we verified in Table 5a that our adapter maintains high alignment with the approxi- mated nullspace, this internal consistency holds little value if the approximation itself fails to reflect the true non non-linear nullspace. Since the ground truth nullspace is analytically intractable, we assessed the validity of our proxy by com- paring our FFN-centric linearization against a full Jacobian baseline using subset of ImageNet-IK. Specifically, we perturbed input images along the null directions computed by each method and measured the deviation between the bl-
Metric
SINGER
Jacobian
La (
0.169
0.191
Cosine simm (f)
0.9787
0.9564
CKA
0.9975
0.9947
Table 8: The output deviation of the non-linear' block when inputs are per- turbed along null directions computed by ours versus the full Jacobian.
ock's original output and the output produced from these perturbed inputs.
The results in Table&iindicat indicate that our method induces minimal output deviations across all metrics (L2 difference, Cosine similarity, and CKA), showing consistency comparable the full Jacobian baseline. This empirically suggests that our approximation serves as a valid and robust proxy for the non-linear block.
Complexity, We also analyzed the computational overhead of SiNGER. The one-time SVD initial- ization negligible (< 0.2s), and the lightweight adapters 1.2% additional parameters (65K) to ViT-T model. Regarding training compute, we observed an approximate 10% increase in time per epoch. themselves are computationally efficient, this overhead primarily stems from the extra forward pass through the teacher's subsequent block required for Linfo-
16, d' 1024) introduce only
worth noting that while the adapter operations
CONCLUSION
In this work, we investigated the challenge of artifact transfer knowledge distillation for ViTs. We showed that high-norm artifacts in teacher representations degrade interpretability and are naïvely inherited by student models, limiting the effectiveness and benefits from scaling conventional dis- tillation approaches. To address this issue, we proposed a distillation framework, namely SiNGER, and callspace-guidee adapter that introduces minimal perturbations to suppress artifacts while pre- serving informative representations. Our framework demonstrates consistent improvements over isting methods across diverse set of downstream tasks, yielding both higher accuracy and more in- terpretable features. We believe this perspective opens new directions for artifact-robust distillation and provides insights into the broader problem of transferring knowledge from wer-parameterized models.
Limitations and Future Work. Nevertheless, our method has limitations. It suppresses artifacts rather than fully eliminating their sources. Since the goal is retain as much teacher information as possible, the root causes of representation degradation remain. Future work will extend our approach to a wider range of foundation models and multi-modal settings, exploring whether nullspace-guides perturbations can serve general mechanism for reliable model compression and adaptation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) under Grant No. RS-2025-00564133 and in part by Convergence security core talent training business support program under Grant -TT--2023-RS-2222-0266666
REFERENCES
Nima Aghli and Eraldo Ribeiro. Combining weight pruning and knowledge distillation for com- pression. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recogni- fion (CVPR) Workshops, pp. 3191-3198, June 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In European conference on computer vision, pp. 446-461. Springer, 2014.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL
ftttp:://openreet
net mett:/orrumm1d2220003i.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: 7 //operevieew net wtt.:/eeeeeettooommmm//oiooo.aaa cNNTTy
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattem recognition, pp- 770-778, 2016.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A criti- cal analysis of out-of-distribttin generalization. Proceedings of the IEEE/CVF international conference on computer vision, pp. $340-8349) 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. httppp//t///
org/ h/05//b//ee 1503. 02531
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
Edward Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Dmitry Ignatov, Andrey Ignatov, and Radu Timofte. Virtually enriched nyu depth v2 dataset for monocular depth estimation: Do we need artificial augmentation? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6177-6186, 2024.
Saqib Javed, Hieu Le, and Mathieu Salzmann. Qt-dog: Quantization--wware training for domain 0602
generalization, 2024. URL
Jiacheng Jiang, Yuan Meng, Chen Tang, Han Yu. Qun Li, Zhi Wang, and Wenwu Zhu. Gagat: gradient-adaptive quattization-awwrr training for domain generalization, 2024. URL https:
w/a.xii.orr/aa////22///.. 05551
11

---

Published as a conference paper at ICLR 2026
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong. Thomas Kollar, Ben- jamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-tanguage-actin model, 2024. URL httlhtss////axiv.oo abs /20/24466.00 09246
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neu- ral network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Interaaionaal Conference Machine Learning, volume 97 of Proceed- ings of Machine Learning Research, pp. 3519-3522 PMLR, 09-15 Jun 2019. URL https:
7 p/proceedings.o mwr. www.ssmml..ppeeee v97 wwt.////okobtthhhooommm 9a html
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Pro- cessing Systems, volume 34, pp. 28092-28103, Curran Associates, Inc., 2021. URL https::/pproceedingg neur ips wwwpaper.pec.....p http://aper_fleeeppaeeeee Paperr. /tlpss/paper//222//11e
Paper.
Zhiying Lu, Hongtao Xie, Chuanbin Liu, and Yongdong Zhang. Bridging the gap between vision transformers and convolutional neural networks small datasets. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151,, 2013.
Maria-Elemt Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In IEEE Conference Computer Vision and Pattern Recognition, volume 2, pp. 1447-1454, 2006.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang- Beang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features with- out supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL Featured Certification.
net forum? adwoummidda6SSttt662
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), June 2019.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, 18-24 Jul 2021. URL https: html
mlr prw.s / 199 https////13////cffoo2222 radford2la.
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12490-12500, June 2024.
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5400. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2026
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, 2015. URL bttts::/arxiv..ot abs/
2412.6551
Mert Bulent Sariyildiz, Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, and Yannis Kalantidis. Unic: Universal classification models via multi-teacher distillation. In European Conference on Computer Vision (ECCV). 2024.
Oriane Siméoni, Huy Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15731-15740, June 2024.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, pp. 316-533, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031- 20053-3.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7769-8778, 2018.
Haoqi Wang, Tong Zhang, and Mathieu Salzmann. Sinder: Repairing the singular defects of dinov2. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024, pp. 20-35, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72667-55
Huanrui Yang. Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hewre saliency. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 18547-18557 June 2023.
Zhendong Yang. Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Feature-based knowledge distillation for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 1379-1388, June 2024.
Shaoyu Zhang, Chen Chen, Xiyuan Hu, and Silong Peng. Balanced knowledge distillation for long- tailed learning. Neurocomputing, 527:36-46, 2023.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):302-32) 2019.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
SINGER: A CLEARER VOICE DISTILLS VISION TRANSFORMERS FURTHER
A TRICKS FOR CALCULATING NULLSPACE
A.1 WEIGHTS LINEARIZATION
We generally compute a nullspace via the Singular Value Decomposition (SVD) a linear oper- ator M Raxd obtaining left-nullspace .V Null(MT). This procedure presumes that the target M is linear. However, a transformer block Wi at layer 7 is inherently non- non-linear due to atten- tion, activation, and residual pathways, the SVD-based nullspace of the block Wi not directly defined.
Let 1xd denote the row-vector feature. A standard Pre-LN transformer block at layer I can be written as
y1 = zz + (HA(IN(z)).
T[+1 + FFN (w)))
where both MHA(-) (.) and FFN(-) include non-linear operations (softmax attention, elementwise activations) and the residual additions further couple the sub-layers. Consequently, there is single linear matrix M that exactly represents Wi for SVD, motivating a linearization that we introduce next.
To compute a nullspace for non-linear block Wi, we first replace it with a linear surrogate W,. Our key design choice to linearize only the FFN sub-layer, motivated by an empirical study showing that the FFN induces larger relative feature changes than self-attention (SA).
We measure sub-layer-wise changes on a ViT teacher by sampling N=5000 random ImageNet train- ing images (uniform over class folders), resizing to 224x224, normalizing, and running forward pass to obtain per-layer tokens. For each layer I and each block, we then reapply the block with instrumented intermediates:
Fin E R Bx(LPP)
2SA = Iin + MHA LN(x:)).
hn LN(SSA),
21 h1 W1 + b2,
01 GELU
22 01 W2 + b2.
Fout JSA + 22,
where W1, W2 are the FFN weights cexpand-then-proiee in reporting expectations. We exclude the [CLS] token and compute patch-wise nornos..
and we ignore the stochastic drop-path
For each image and layer, we aggregate over patches using the mean and record four quantities:
|
Xinll2
||Tout
ISA ||2
Asa == meath
AFFN meaDpaath
1in 2|
| ||2
122 ||
GfEN1 =
mean
GEENE == mcangatch
| 1|
|
14

---

Published as a conference paper at ICLR 2026
1-17
1-23
18-
08
SAA
FFFN
GEEN1
GFFN2
ASA
ΑFF
GFFNI
GFFN
Metric
mean
median
p95
Metric
mean
median
p95
GypNi
0.0719
0.0714
0.0844
GEENI
0.3251
0.3231
0.3773
GPeNN
0.7988
0.7945
0.8571
Gren2
1.0226
1.0194
1.1754
APPN 0.1871
0.1852
0.2210
APFN
0.1543
0.1537
0.2001
ASA 0.1417
0.1415
0.1611
ASA
0.2960
0.2965
0.3468
Figure 7: Sub-layer change analysis at two depths. Top: box-plots of relative changes/gains in each layer. Bottom: summary statistics in each layer.
From Figure 7 the dominant amplification occurs in the second FFN stage (GFFN2 is largest at both depths), and the net FFN residual AFFN comparable or larger than the SA residual depending on the layer. This indicates that the principal source of norm inflation lies within the FFN pathway, especially its projection stage.
Guided by this analysis, we exclude the non-linear SA pathway when constructing a linear operator for SVD and focus on the FFN inside W, Since the non-linearity enters the FFN only via the GELU between two linear maps, removing GELU (and biases) yields a linear surrogate:
FFN(h) = à WWFFNIWFEN = h.
(14)
with row-vector features and right multiplication (the column-vector convention uses W
WT WFFNe WT WF We refer to W2 as the linearized weights of block I.
A.2 NULLSPACE OF LINEARIZED WEIGHTTS
Now, we can compute the SVD of the linearized FFN matrix Wi Rxd, We compute its SVD
Wi Up V.T,
Σ, =,=diag(a; 2 2 a 0)-
(15)
left-nullspace basis of dimension r is obtained by selecting the r left singular vectors associated with the smallest singular values:
N1 Tadxr, N T,, cols(N, U(-d-r+1:d)
(16)
For any row vector TT span(Ni) we then have the approximate-nin condition
TT Wi U. x 0,
(17)
since the selected modes correspond to the smallest singular values.
common concern that W; could be numerically full rank, making the exact nullspace trivial. We therefore quantify practical e-nullspace using two diagnostics defined below, and visualize them in Figure
Let the singular values of W, E Rxd be σi > 02 2 ...
2 2 0. Define the cumulative energy
k -=
E(k)
E 10,
-==1 o?
15

---

Published as a conference paper at ICLR 2026
layer 5
layer 13
101
101
100
100
10
10
10
10-2
singuler values
singuler values
10
99.9% energy 714
10
99.9% energy @ 847
10
200
400
600
800
1000
200
400
600
800
1000
index k
index
layer 17
layer 23
102
10a
101
100
10a
10-1
10
singular values
10-2
singular values
(abs)
(abs)
10-3
99.9% energy
t-threshold @ 960
10-3
t-rrehhlld @ 986
200
800
1000
200
400
1000
index
index
Figure 8: Singular-value spectra of W, for representative layers (1 = 5, 13, 17, 23) with green
horizontal line at the absolute threshold = 0.05 and red vertical lines marking Kenorgy (99.9% cumulative energy) and ke (first index with d₃ S E).
and for target level p (0,
Kenergy (p) == min k E {I....,,4} d} E(k) 2 p).
(18)
For an absolute tolerance >> define the first crossing index
kgg = min ls E (L....,d] d] ok SeJ.
Ta d kgg 1,
(19)
so that the e-tail has dimension rs and is spanned by the last r. left singular vectors.
As summarized by Figure we consider a VIT-L teacher with d-1024 and focus on an intermediate block (1=17). At this layer, the cumulative cunulative-energy index is Kenergy (0.999)=777, tail has size d - hinnery =251. With an absolute tolerance ==0.05, the first-crossing index k; =80, yielding e-tail of dimension r==d - k + 1-45. Consequently, the tail span forms high-quality approximate nullspace: for any this subspace,
the low-energy
|| Willa s s =||l||2,
which justifies cicisspcceeuuideee updates that suppress outlier energy while preserving informative structure.
B ARCHITECTURAL SPECIFICATIONS
This section summarizes the architectural specifications of all teacher and student models used in our experiments. Table lists the depth, embedding size, number of attention heads, the number of register tokens, the number of parameters, and GFLOPs for each architecture.
16

---

Published as a conference paper at ICLR 2026
Model
Depth
Embedding
Heads
Registers
(M)
GFLOPs
Vire
32
1280
16
0
630.92
254.63
ViT-Larg
24
1024
16
0
304.33
123.11
ViT-Base
12
768
12
0
86.57
35.13
VIT-Small
12
384
6
0
22.05
9.20
VIT-Tiny
12
192
3
0
5.72
2,51
DelT: III Large
24
1024
16
0
304.37
123.11
DelT-III- Bose
12
768
12
0
86.59
35.13
DelT-Ill-Saan
12
576
0
22.06
9.20
DeiT-I1l- Tiny
12
384
0
5.72
2.51
DINOv2-Large
24
16
0
123.11
DINov2-reg-Large
24
1024
16
4
303.35
125.68
DINgee-Large
125.68
Table 9: Architectural specifications of all models used in the experiments.
All models follow their official architectures. DINOv2-reg (Dareet et al 2024) incorporates register tokens, whereas DINOv2 (Oquab et al., 2024), DINOv2-reg. and DINOv3 (Siméoni al. 2025) use ViT-style backbones with the same depth, hidden size, and number of heads as ViT-Large but differ in training strategy and design details.
Across all configurations, student layers are aligned with every second teacher layer during distil- lation. Other architectural components (patch size, MLP ratio, and tokenization) follow the official implementations of each model family.
C VARIOUS BACCHER-STUDENT PAIRS
This section provides additional analysis of SiNGER under teacher configurations that differ from those used in the main paper. We evaluate three scenarios: distillation from cleaner teachers with minimal artifacts, cross-family distillation between heterogeneous architectures, and scaling the teacher from Large Huge capacity (Table TOp. All experiments were conducted on a small subset of ImageNet- KK due to computational constraints, but the training protocol was kept identical across methods for fair comparison.
Distil
IK
ADE
NYT/22
Distil
IN-IK
ADE
NYUv2
Distill
IN-IK
ADE
NYUv2
L- VIFT
DINOv2-L.
DeiT-III-T
VIT-L 11.57
VIT-T
PiNNet
6.97
3.06
1.3711
FitNet
2.90
2.86
1.2328
SINGER
4,99
L.1690
VITKO
1.81
1.57
1,3499
VILKD
SINGER
6.10
2,78
1,4058
SINGER
3.40
2.91
1.2222
VIT-H
VIFT
FiNNe SINGER
9.78
4.75
1.3006
DINOv3-L.
VIT-T
14.94
6.82
1.2438
FitNet
1.84
1.16
1.3868
VITKD
0.23
0.42
1.3157
SINGER
1.19
1.06
1,4550
(a) Cleaner teachers.
(b) Heterogencous families.
(c) Larger teacher.
Table 10: Distillation across diverse teacher-student pairs.
Cleaner Teachers. To examine the behavior of SiNGER when high-norm artifacts are reduced at the teacher level, we distilled from two cleaner teacher baselines: DINOv2 with register tokens (DINOv2-reg, Darcet al. (2024)) and DINOv3 (Siméoni et al. 2025) (Table [Oa). Both exhibit substantially lower artifact magnitude in their intermediate representations.
Across these settings, SiNGER performs competitively but does not consistently surpass the strongest baseline in absolute accuracy. For DINOv2-reg ViT-T, SiNGER improves segmen- tation accuracy over FitNet but falls slightly behind ViTKD on ImageNet- IK and NYUd-v2. A similar pattern is observed in DINOv3 ViT-T, where SiNGER achieves stable but modest results while outperforming FitNet on certain tasks.
These outcomes align with the design premise of SiNGER: the method targets scenarios in which the teacher contains high-norm artifact tokens that dominate the refinement direction and bias the student. When artifacts already minimal, the suppression loss may attenuate useful high-norm
17

---

Published as a conference paper at ICLR 2026
channels, producing minor reductions in absolute performance. Nonetheless. SINGER remains more stable than ViTKD, whose random masking strategy can behave inconsistently when artifact struc- ture is weak absent. This indicates that SiNGER provides a more principled refinement mecha- nism even unfavorable teacher conditions.
Heterogeneous Families. To assess generality beyond same-family ViTs, conducted cross- family distillation from DINOv2-L to DeiT-III-T (Table Tob). Owing to the reduced dataset size, absolute performance is lower than full-scale training; however, the relative gains are significant. SiNGER improves over FitNet by 17.2% on ImageNet-IK validation, 1.75% mloU on ADE-20K, and achieves a 0.86% reduction in RMSE on NYUd-v2. These results demonstrate that SiNGER effectively transfers information even when teacher and student architectures differ substantially, suggesting broad applicability the refinement strategy.
Scaling the Teacher. We further evaluated whether SiNGER maintains its benefit when increasing teacher capacity. Using ViT-H (huge) as the teacher and ViT-T as the student, SiNGER improves top-1 accuracy by 5.16%p, increases mloU by 2.07%p, and reduces RMSE compared to FitNet. Similar gains appear when distilling from ViT-L. These results confirm that the nullspace-guided refinement process remains effective for high-capacity teachers, even under constrained training budgets.
D FAILURE OF VITKD
In this section, we discuss the failure of ViTKD (Yang et al.) 2024) in learning the teacher rep- resentation. The core strategy behind VITKD masking and generation. Different from SiNGER, ViTKD does not adaptively detect artifacts and randomly discards patches regardless of their seman- tic validity, and generates through convolution, utilizing learnable generative tokens. This ensures students learn artifact-free representation. However, it also results in blurred representation, which one of the expected trivial solutions to minimize the mean squared error the generated features, resulting in significantly degraded representation.
As depicted in Figure 2 ViTKD successfully mimics the teacher's representation terms of cosine similarity, but fails in building the informatively structured feature map. This structural degradation makes the representation blurry, resulting in poor downstream task adaptation.
E VISUALIZING DISTILLATION OUTPUTS ON A SINGLE IMAGE
We visualize a single sample at token resolution 14x16 14 (patches only: [CLS] excluded). For the teacher(ViT-large), we show odd-numbered blocks / E {1,3, , .3)). For the sVuden((ViT--iny)), we show layers E {0.... .0..... [0.....11] aligned with the teacher columns. In this sample, the SiNGER adapter is applied only I [17,23) 23).
As shown in Figure 9 the first row F. exhibits artifacts: high-norm becomes more pronounced
at deeper blocks. After applying SiNGER adapter, the second row E attenuates these artifacts while preserving the informational structure, producing more waasfer-friendee teacher target. The third row visualizes the residual AFT F FT confirming that SiNGER removes a small set of outlier's magnitudes. Finally, the fourth row F/ aligns more closely with F? than with ET, indicating that the student learns the SiNGER-refined stracture-preseeving representation rather than the original outiier-domintted one.
F VISUALIZATION OF OUTLIER SUPPRESSION
This appendix illustrates, on a single sample, how outlier suppression operates numerically. As shown in Figure 10 in the last layer, the patchwise
map [IF contains outlier patch with
maximum norm of 638. At the same spatial location, the suppressed map [|F]]] drops to 54.1. Finally, the distribution of [[ appears much more uniform across patches, indicating that extremely high- norm outliers have been attenuated while the overall scale has been regularized.
18

---

Published as a conference paper at ICLR 2026
G VISUALIZATION OF INFORMATION PRESERVATION
We confirm whether the information is actually preserved right after the / + 1-th layer. The cross- similarity map between F and F are visualized in Figure 11 one shows F - F similarity map, and the other one shows the contrary. As shown, in both directions, the similarity maps are almost identical. This implies the information is actually preserved even after passing the next layer, which the source nullspace used for initializing the proposed adapter.
For each 2 cells, the left
H STUDENT VISUALIZATION
We evaluate the quality of our SiNGER- feature by visualizing the similarity map (Fig- ure T2). For each 2 x 2 cell, the top row is the student, the bottom row is the teacher. The left column is the norm map of the feature map, and the right column the similarity map the "x" marked patch. The norms of the teacher's feature maps are artifact-prone, but still produce the similarity map, which semantically makes sense. The student produces artifact-suppressed feature maps while maintaining the semantic relation among the patches. This emphasizes that SiNGER effectively optimizes two objective functions, distills high-quality feature representation.
I CORRELATION OF OUTLIER WITH PRINCIPAL AND NULL BASES
We conduct this experiment to validate the core design choice behind our method: we perturb fea- tures along a nullspace preserve information, but such perturbation is only meaningful if outliers do not primarily reside in the nullspace. Otherwise, nullspace-diresd press outliers. To test this, build Appendix [A.] and Appendix [A.2] from the linearized FFN matrix Wi, we take the left singular vectors with the largest singular values the principal basis, and the with the smallest singular values as the null basis.
updates would fail to sup-
Fix a layer and an image, and let X E (1+ be the teacher tokens (CLS excluded below). Compute
Wi = -U-V, U- [......, ual, Σg diag(o) 2 ... 2 dd 2 0).
Define the two r-dimensional bases
Uprin [u1, , ur]s Unall [id fud-rt1,.... ud]
Let Up E 1xd be the p-th patch feature with norm ny = | For a chosen r-dimensional orthonormal basis = ......... ut.] E Rdxr (e.g., columns selected from the left singular vectors of Wi), define the normalized subspace energy.
po Uin )
Ep(p)
M 11 +
In words, Ep(p) is the fraction of the patch's 's total energy captured by the subspace U-i.e., a norm- invariant measure of how strongly up aligns with U. In our analysis, we instantiate U by either of the above bases, i.e.,
E Uprin
Unall
In Figure 13 across layers we observe distinct behaviors as the rank increases. At the intermediate layer (1-17), outlier patches (high ( exhibit growing values in Ev,a as increases, while EU over those same patches does not grow accordingly. Conversely, at the last layer (I=23), outlier patches show increasing Eu,,. with whereas Eu.,. over outliers does not increase in the same manner. We quantify these patterns Figure 13 (m). For 1=17, the patch norm cor- relates positively with the null subspace energy (corr( F/ [|,, Fe > and negatively with
19

---

Published as a conference paper at ICLR 2026
the principal subspace energy (corr( [F Eum))) corr(1) (oFF(1) Fu,.)) O and corr(1 F/ [l2 Fujm)) >
0). In contrast, for 1=23 the signs flip:
These results indicate that at intermediate depth (e.g., 1=17) the high-norm (outlier) content lies relatively closer the null subspace, whereas at the final depth (e.g., 1-23) it aligns more with the principal subspace. Accordingly, initializing updates along the nullspace at intermediate layers achieves information preservation (by construction) while still enabling outlier suppression, consis- tent with our design objective.
J DISTILLED MODELS IN LONG-TAIL LEARNING
To diagnose the source of performance degradation on INaturalist0019 we examined the entropy of the teacher's s logits across the full set, the majority classes, and the long-tail classes. The results are summarized in Table
Full (53,649)
Major (1,873)
Long tail (1,819)
Model
Ent.
Acc.
Ent.
Acc.
Ent.
Acc.
VIT-
6.6336
71.41
6.6290
74.21
6.6526
61.24
VII-I
6.6435
44.71
6.6421
48.10
6.6548
32.22
SINGER
6.6940
41.08
6.6933
48.37
6.6989
24.96
FitNet
6.7097
39.98
6.7078
44.69
6.7135
26.50
Table 11: Entropy analysis in long-tail learning. The numbers in the parenthesis refer to the number of samples. Ent. and Acc. means entropy and accuracy, respectively.
A consistent pattern emerges: the teacher (ViT-L) exhibits the largest entropy increase specifically in long-tail classes, indicating substantially lower confidence and greater prediction ambiguity for minority categories. Because knowledge distillation transfers the teacher's 's class-level certainty, this elevated uncertainty is directly inherited by the student.
Crucially, SiNGER focuses on suppressing spatial artifact tokens but does not alter the teacher's class logits. Therefore, when the teacher already provides ambiguous supervision for rare classes, the long-tail accuracy becomes limited not by artifact noise but by the teacher's intrinsic uncer- tainty. This explains why the accuracy- than VIT-T trained without distillation, despite improving performance in other scenar- ios.
student shows smaller gains-and occasionally lower
Still, SINGER demonstrates lower loig-tail entropy than FitNet (6.6989 6.7135), suggesting that stabilizing the refinement direction yields more reliable supervision even when the teacher's uncertainty dominates.
K IMPLEMENTATION DETAILS
In this section, we report our implementation details for reproducibility. We trained our model on Ubuntu 22.04.5 LTS with CUDA v12.6.85 using eight NVIDIA GeForce RTX 3090 GPUs. Mixed precision training (FP16) was enabled reduce memory consumption and accelerate computation. All experiments were conducted with global batch size of 512, distributed evenly across GPUs using PyTorch's DistributedDataPrralae (DDP).
For distillation, the 8th layer and 17th layer were selected as the distillation layers for Tiny ViT and Large ViT, respectively. We tuned the weights for the losses equally to 1.0. The model and adapters were optimized using the AdamW optimizer with a cosine annealing learning rate scheduler of a single cycle. The learning rate was initialized at 10-4 and decayed to 10-8 over 100 epochs. Weight decay was set to 0.05, and gradient clipping with max norm of 1.0 was applied to stabilize training. Data augmentation followed common practice for ImageNet training, including random resized cropping and horizontal flipping. Input images were resized to 224 224 unless otherwise specified. During evaluation, a center crop was used. All hyperparameters and code will be released upon publication.
20

---

Published as a conference paper at ICLR 2026
L THE USE OF LARGE LANGUAGE MODELS
As per the ICLR 2026 guidelines on the use of Large Language Models (LLMs), we disclose that an LLM was used for minor grammar corrections and polishing of the text enhance readability, as well as for searching related research to broaden the scope of the literature review. The LLM did not contribute the research ideation, methodology, core findings of the paper.
21

---

Published as a conference paper at ICLR 2026
Figure 9: Distillation visualization across stages using complementary views. Each panel renders patch features either directional view (PCA with 3 components) or magnitude view (patch- wise norm). Within each panel, rows depict (top bottom) F.. FT. AFT. AFFFF..
22

---

Published as a conference paper at ICLR 2026
|
||FFI
Figure 10: Left: input image. Right: patchwise visualizations. Row 1: FFll and || [TT Row 2:
[[FF]] ||, F7T (signed map), and ||] with AFT FF R..
Figure 11: The cross-similarity map between ET and FTr 'x" mark is visualized.
23

---

Published as a conference paper at ICLR 2026
Figure 12: The similarity map to 'x" mark is visualized.
24

---

Published as a conference paper at ICLR 2026
@r==11=11
(b) r = 4,1=2 23
rr)= 8,1
(d) 8,7
(c) 16,7 16,1117 17
(f) 16,1 23
32,=2 17
(h) r 32, 23
() = 64,7= 17
( r 64,7 = 23
(k) r -= 128, 1 17
(I) r 128, / 23
0.3
pixcpal
Ful
0.2
0.1
corretzion
0.0
-0.1
..1
-033
peictppll
-0.6
20
120
60
120
adepter
adapuer
(m) correlation with basis at layer 17
(n) correlation with basis at layer
Figure 13: panels (a)-() show, for each setting, four views: the input image, the patchwise norm | 2. and the subspace energies Eu,,. and Ev..- Panels (m)-(n) (n) show correlation with each basis.
25


Reviews:
[{"review_id": "xujCeNOe6Q", "reviewer": "Reviewer_YFDi", "summary": "The paper tackles a well-observed problem in Vision Transformer (ViT) distillation: high-norm \"artifact\" tokens in large ViTs dominate feature-level KD losses, so the student ends up overfitting to these outliers instead of learning the informative inlier structure. To address this, the authors propose SiNGER (Singular Nullspace-Guided Energy Reallocation), a distillation framework that first refines the teacher’s features and only then distills them. The key idea is to add a LoRA-style low-rank adapter to the teacher and to constrain its perturbations to (an approximation of) the left nullspace of the next block, so that: (i) high-norm outliers are suppressed, and (ii) downstream information is preserved because the perturbation lies in a direction that doesn't change the next layer's output. They couple this with two auxiliary losses: an outlier suppression loss (to explicitly shrink high-percentile tokens) and an information-preservation / Gram-matching loss (to keep patchwise relations intact). Experiments on several downstream tasks (ImageNet-1K, ADE-20K, NYUd-v2, domain shift benchmarks, fine-grained datasets) show that SiNGER improves over FitNet and ViTKD when distilling a ViT-Large to a ViT-Tiny, and produces visually \"cleaner\" feature maps with more uniform patch norms.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear and well motivated problem formulation. The paper starts from a real and now increasingly reported issue in ViTs: high-norm background/low-semantics tokens that get amplified across residual blocks and hurt both interpretability and distillation. The link they make (artifact dominance -> gradient bias in feature KD) is very sensible and well motivated. \n\n2. Principled mechanism. Instead of just masking teacher features (like ViTKD), they try to change the teacher supervision itself in a way that is theoretically tied to the next layer: perturb only in (approximate) nullspace so the next block’s output is preserved. This is a neat way to reconcile \"suppress artifacts\" vs \"don't destroy useful signals\".\n\n3. Low-intrusion design. Using a LoRA-based adapter to implement the refinement is pragmatic: it's lightweight, it doesn't require re-training the whole teacher, and it aligns with current practice on parameter-efficient tuning.\n\n4. Good ablations. The nullspace initialization vs random init, the presence/absence of the information-preservation loss, and the small hyperparameter sweep on rank r and percentile a all support the central claim.\n\n5. Multi-task evaluation. Even though the numbers in Table 1 are a bit hard to read, the intent is good: they try to show that better, less-artifact supervision gives students that transfer better to unrelated tasks (segmentation, depth), not just ImageNet linear eval.", "weaknesses": "1. Central assumption relies on a rough linearization. The core trick (perturb in the left nullspace of the next block so the next block's output is unchanged) is only feasible because they linearize the non-linear transformer block using the FFN weights and then pick the smallest singular vectors. But real blocks have attention, non-linearities, layernorm, residual couplings, and cross-token interactions. The paper cites an appendix showing FFN is the dominant source of norm amplification, but in the main text this is still an approximation, not a guarantee. A reviewer could fairly ask: how often does the adapter actually stay in the true (nonlinear) null directions during training? The ablation in Table 4a helps, but it’s still empirical. \n\n2. Evaluation scope is narrow. All main results are ViT-Large to ViT-Tiny, canonical ViT, every-second-layer alignment. That is a single, quite specific distillation path. There is no evidence it works for: (i) teacher and student of different families (e.g., DINOv2 to DeiT-Small), (ii) CNN to ViT, or (iii) same-size or near-size distillation where artifacts might behave differently. This makes the generality claim (\"artifact-aware KD for VFMs\") weaker than it sounds.\n\n3. Choice of baselines. They compare to FitNet and ViTKD, but they do not compare to the most natural artifact-aware teacher-side fixes like register tokens (Darcet et al., 2024), or to SINDER / DINOv3-style cleaned teachers as sources for distillation--both of which are already in their related work. If the claim is \"we are the principled way to keep info while suppressing artifacts\", then we need to see: what if you just gave the student a cleaner teacher in the first place? Right now that experiment is missing. \n\n4. Table 1 is confusing / formatting issues. The version here has lines that appear interleaved or misaligned (e.g., \"SiNGER 70.59 8.16 21.76 3.03 ...\"), which makes it hard to confirm the exact gains on each benchmark. For a distillation paper, exact numbers matter. This hurts Presentation. \n\n5. Cost and practicality not fully discussed. Attaching adapters to multiple teacher layers and running SVD-style initialization on (linearized) FFNs is not free. The paper doesn’t quantify the extra compute / memory during distillation vs. plain feature KD. This is important for practitioners.\n\n6. Long-tail result potentially goes in the wrong direction. The iNat2019 result is weaker and is brushed off as “long-tail is hard,” but that’s precisely a setting where better supervision and less artifact noise should help the student pick up minority classes. A short analysis or per-class breakdown would strengthen the story. \n\n7. Causality of the improvement is still a bit entangled. Three things change at once: teacher refinement, outlier loss, and Gram matching. We do see an ablation on Linfo, but we don’t see, for example, \"plain LoRA adapter without nullspace init\" vs \"nullspace init\" vs \"only percentile shrinkage\" on all tasks, just on a subset. It’s still plausible that a well-tuned teacher-side LoRA denoiser would get close.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "otzP1mfTGD", "reviewer": "Reviewer_CqPa", "summary": "This paper addresses the problem of high-norm artifacts in ViTs that degrade knowledge distillation effectiveness. The authors propose SiNGER, a distillation framework that refines teacher features by applying perturbations guided toward the nullspace of subsequent layers. The key innovation is using LoRA-based adapters initialized with approximate nullspace vectors to suppress artifacts while preserving informative signals. Experiments demonstrate improvements across multiple downstream tasks including classification, segmentation, and depth estimation.", "questions": "1. Nullspace Verification: \n2. Can you scale to Larger Models?\n3. Ablation on Linearization: What happens if you use the full nonlinear block's Jacobian", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Well-Motivated Problem: The paper identifies a concrete limitation in ViT distillation—high-norm artifacts dominate gradient flow and harm student learning.\n2. Comprehensive Evaluation: The multi-task evaluation spanning 10 benchmarks is thorough. Consistent gains on ImageNet and ADE-20K", "weaknesses": "1. Approximation Gap: The core theoretical claim assumes linear blocks, but ViT blocks are highly nonlinear.\n2. Incomplete Comparison with ViTKD", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CtPQadMcxf", "reviewer": "Reviewer_Mz5w", "summary": "The paper introduces SiNGER (Singular Nullspace-Guided Energy Reallocation), a knowledge distillation (KD) framework designed to improve the compression of Vision Transformers (ViTs) by addressing high-norm artifacts in teacher representations. These artifacts, arising from power-iteration-like accumulation in ViT layers, dominate the distillation objective, causing students to overfit to noise and underutilize informative signals. SiNGER refines teacher features using a lightweight LoRA-based adapter that applies nullspace-guided perturbations to suppress artifacts while preserving downstream-relevant information. The refined features are then distilled to smaller student models.", "questions": "How do you select the nullspace dimension or perturbation strength in practice? Are there hyperparameters, and how sensitive is performance to them (e.g., ablation results)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Principled Solution to a Core Problem: Identifies a fundamental limitation in ViT KD (artifact dominance) and resolves the trade-off between suppression and information preservation via nullspace guidance, leading to more effective knowledge transfer.\n\nEfficient and Minimalist Design: The LoRA adapter requires little structural change, making it easy to integrate into existing KD pipelines while maintaining computational efficiency.", "weaknesses": "ViT-Specific Focus: The method is tailored to ViT artifacts (e.g., from self-attention scaling); its generalizability to other architectures like CNNs or hybrids isn't explored.\n\nDependency on Teacher Quality: Assumes well-pre-trained ViT teachers; if artifacts are severe or pre-training is suboptimal, refinement might not fully compensate.\n\nScope Limitations: Primarily evaluated on downstream vision tasks; extensions to multimodal or non-vision transformers aren't discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)