DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer
Authors: Haiduo Huang
Jiangcheng Song
Yadong Zhang
Pengju Ren
Venue: NeurIPS 2025 poster ()
Keywords: Knowledge Distillation
Gradient Momentum Decoupling
Dynamic Top-K Mask
Signal-to-Noise Ratio
Primary area: deep_learning

Abstract: Recent advances in knowledge distillation have emphasized the importance of decoupling different knowledge components. While existing methods utilize momentum mechanisms to separate task-oriented and distillation gradients, they overlook the inherent conflict between target-class and non-target-class knowledge flows. Furthermore, low-confidence dark knowledge in non-target classes introduces noisy signals that hinder effective knowledge transfer. To address these limitations, we propose DeepKD, a novel training framework that integrates dual-level decoupling with adaptive denoising. First, through theoretical analysis of gradient signal-to-noise ratio (GSNR) characteristics in task-oriented and non-task-oriented knowledge distillation, we design independent momentum updaters for each component to prevent mutual interference. We observe that the optimal momentum coefficients for task-oriented gradient (TOG), target-class gradient (TCG), and non-target-class gradient (NCG) should be positively related to their GSNR. Second, we introduce a dynamic top-k mask (DTM) mechanism that gradually increases K from a small initial value to incorporate more non-target classes as training progresses, following curriculum learning principles. The DTM jointly filters low-confidence logits from both teacher and student models, effectively purifying dark knowledge during early training. Extensive experiments on CIFAR-100, ImageNet, and MS-COCO demonstrate DeepKD's effectiveness.

DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer
Haiduo Huang", Jiangeheng Song", Yadong Zhang, Pengju Ren Institute of Artificial Intelligence and Robotics, Xi an Jiaotong University (huanghd, enone, 3488928833 xjtu. edu. en, pengjuren@xj tu. edu, en
Abstract
Recent advances in knowledge distillation have emphasized the importance of decoupling different knowledge components. While existing methods utilize mo- mentum mechanisms to separate task-oriented and distillation gradients, they overlook the inherent conflict between target-class and non-target-class knowl- edge flows. Furthermore, low-confidence dark knowledge in non-target classes introduces noisy signals that hinder effective knowledge transfer. To address these limitations, propose DeepKD, novel training framework that integrates dual- level decoupling with adaptive denoising. First, through theoretical analysis of gradient signal-to-noise ratio (GSNR) characteristics in task-oriented and non-task- oriented knowledge distillation, we design independent momentum updaters for each component to prevent mutual interference. We observe that the optimal mo- mentum coefficients for task-oriented gradient (TOG). target-class gradient (TCG), and non-target-class gradient (NCG) should be positively related to their GSNR. Second, we introduce dynamic top-k mask (DTM) mechanism that gradually increases K from small initial value to incorporate more non-target classes as training progresses, following curriculum learning principles. The DTM jointly filters low-confidence logits from both teacher and student models, effectively puri- fying dark knowledge during early training. Extensive experiments on CIFAR-100, ImageNet, and MS-COCO demonstrate DeepKD's effectiveness.
Introduction
Knowledge distillation (KD) has emerged a powerful paradigm for model compression since its introduction by Hinton et al. [I], finding widespread adoption across computer vision 2 | and NLP (5] domains. By transferring dark knowledge from large teacher models to compact student networks, KD addresses the critical challenge of deploying high sigh-performance models on resource-constrained devices fundamental requirement for emerging applications like autonomous
driving [6] and embodied AI systems 7 g.
Recent advances in KD methodologies have primarily focused on three directions: (1) Multi-teacher ensemble distillation [9] [0] to enhance information transfer, (2) Intermediate feature distillation [I]]
through sophisticated alignment mechanisms, and (3) Input-space augmentation [12 13 output-
space manipulation through noise injection 14 [5] and regularization [6 17]. However, these approaches lack systematic analysis two fundamental questions: Which components of knowledge transfer contribute to student performance? and How should different knowledge components be optimally coordinated during optimization? ? While previous works have made significant progress DKD 18 decouples KD loss into target class knowledge distillation (TCKD) and non con-target class leowledge distillation (NCKD) components through loss reparameterization, revealing NCKD's
Equal Contributions
Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

erucial role in dark knowledge transfer, and DOT [99 introduces gradient momentum decoupling between task and distillation losses- critical limitations persist. First, existing methods fail to address the joint optimization of decoupled losses and their corresponding gradient momenta. Second, the theoretical foundation for momentum allocation lacks rigorous justification, relying instead on empirical observations of loss landscape.
To address these limitations, we present DeepKD, as illustrated in Figure knowledge distillation framework with theoretically grounded features, Our investigation begins with a comprehensive anal- ysis of loss components and their corresponding optimization parameters in knowledge distillation. Through rigorous stochastic optimization analysis [20], we find that optimal momentum coefficients for -aikntrieeted gradient (TOG), target-class gradient (TCG), and non-targe con-target-class gradient (NCG) components should be positively related to their gradient signal-to-non
ratio (GSNR) [2T] in stochas- 22 This enables deep decoupling of optimization
tic gradient descent optimizer with momentum dynamics across different knowledge types.
0.31
10
Easy
Aare hard Aru
110
Interved
Vanila
DKD
DCT
156
Sie
Deep
55
330
Eyeck
150
250
240
(a) GSNR during model training
(b) Loss landscape visualization
(c) Dynamic top-k masking process
Figure 1: Analysis of optimization dynamics and knowledge transfer of ResNet3224/Ressll on CIFAR- (a) Gradient Signal-to-Noise Ratio (GSNR) comparison across different knowledge distillation methods, (b) Loss landscape visualization (23 showing the flatness of minima, and (c)
x4
Dynamic top-k masking process for dark knowledge denoising aligns with curriculum learning.
As shown in Figure Ta),, we visualize the GSNR of vanilla KD M. DKD 18 DOT 19] and our proposed DeepKD throughout the training process (with gradient sampling at 200-iteration intervals). The results demonstrate that DeepKD further decouples KD gradients into TCG and NCG, achieving higher overall GSNR. This enhancement directly contributes to improved model generalization [21 24). Moreover, Figure reveals that DeepKD exhibits a flatter loss landscape compared to other methods. This observation aligns with established findings that flatter minima in the loss landscape generally correlate with improved model generalization [25] 26 a critical factor for effective knowledge distillation. Remarkably, through our GSNR- deep gradient decoupling with momentum mechanisms alone, DeepKD achieves tftte-of-thhaan performance across multiple benchmark datasets, as detailed in the experiments section.
Additionally, while prior works I I8 19 emphasize the importance of teacher logits' dark knowl- edge, they typically process all non-target class logits uniformly. We challenge this convention through two key insights: (1) Only non-target classes semantically adjacent to the target class pro- vide meaningful and veritable dark knowledge. (2) Low-confidence logits may introduce optimization noise that outweighs their informational value. To address these issues, we introduce a dynamic top-k mask (DTM) mechanism that progressively filters low-confidence logits (i.e., potential noise sources) from both teacher and student outputs, implemented as a curriculum learning process (27), as shown in Figure (c). Unlike CTKD (28) which modulates task difficulty via learnable temperature param- eter, our method dynamically adjusts k from 5% of classes to full class count, balancing carly-stage stability and late-stage refinement. Notably, while ReKD (29 applies top-k selection to target-similar classes but retains other non-targe classes, we only preserve the top-k largest non- non-target-class logits based on the teacher and dynamically discard the remainder.
Comprehensive experiments across diverse model architectures and multiple benchmark datasets validate DeepKD's 's effectiveness. The framework demonstrates remarkable versatility by seamlessly integrating with existing logit-based distillation approaches, consistently achieving state-of-the-art performance across all evaluated scenarios.
2

---

2 Related Work
Knowledge distillation has evolved along two main directions: feature-based and logit-based approaches. Feature-based methods transfer intermediate representa- tions, starting with FitNets [30] using regression losses for hidden layer activations. This evolved through attention transfer [31] and relational distillation [32 to capture structural knowledge, culmi- nating in multi-level alignment techniques like Chen el al. 's
Knowledge Distillation Paradigms:
3 multi-stage knowledge review and
USKD's [4 normalized feature matching. While methods like FRS (35] and MDR (36] address teacher-student discrepancy through spherical normalization and adaptive stage selection. they often require complex feature transformations and overlook gradient-level interference. Logit-based distil- lation, pioneered by Hinton al. [,) focuses on transferring dark knowledge through softened logits. Recent advances like DKD [8] decouple KD loss into target-class, (TCKD) and non-target-class, (NCKD) components, revealing NCKD's crucial role. Extensions including NTCE-KD [37 and MDR 36 enhance non-target class utilization but neglect gradient-level optimization dynamics.
Theoretical Foundations and Methodological Advances: Recent advances in knowledge distillation have explored both optimization strategies and theoretical foundations. On the optimization front,
DOT 19 employs momentum mechanisms for gradient decoupling, CTKD [28 uses curriculum
temperature scheduling, and ReKD (29) implements static top-k filtering, though these approaches often rely on empirical heuristics. Dark knowledge purification has been addressed through various strategies: TLLM (38 identifies undistillable classes via mutual information analysis, RLD [39] proposes logit standardization, and TALD-KD [4] combines target augmentation with logit distortion. The theoretical underpinnings of model generalization have been extensively studied through loss with Jelassi er al. [2 analyzing momentum's 's role in generalization.
landscape geometry 25 23
Recent methodological advances have further enriched the knowledge distillation landscape. Niu et al. [40] propose respecting transfer gaps in knowledge distillation, while Huang et al. [41 intro- duce knowledge diffusion mechanisms for improved distillation. Li el al. 42 explore curriculum temperature scheduling for knowledge distillation, and Saidutta et al. formation flow approaches. Huang et al. [44] propose DIST+ with stronger adaptive teachers for enhanced knowledge transfer. Our work extends these principles by establishing the first theoretical connection between gradient signal-to-non noise ratio (GSNR) and momentum allocation in KD, bridging optimization dynamics with knowledge transfer efficiency. Unlike DKD's loss-level decoupling or DOT's empirical momentum separation, we provide GSNR-driven theoretical guarantees for joint loss-grad poss-gradient optimization. Compared to CTKD's 's temperature-cent filtering, our dynamic top-k masking offers principled noise suppression while preserving semantic relevance, addressing the limitation of uniform processing of non-target logits previous approaches.
present controlled in-
curriculum or ReKD's 's static
3 Methodology
3.1
Preliminaries
Vanilla KD: Given teacher model T and student model S, knowledge distillation transfers
knowledge from TI to S while maintaining performance. Let X' be the input space and y be the label space. For input x E X, the models produce logits x = T(x) and x = S(x). The standard knowledge distillation 1 loss combines:
(I) where a is the softmax function, Log the cross-entropy loss with hard labels y E V, LxL is the KL divergence between softened logits p" and ps, i.e., p" === o(2" /r) and p o(2 /T), 7 is the temperature hyperparameter that controls the softness of the distribution, and a balances the losses.
LKD = =Lcx(a(x")))))) ),y) + ),)()(())))+((((()))
DKD: Decoupled Knowledge Distillation (DKD) [88] splits the vanilla KD loss into target-class Knowledge Distillation (TCKD) and non- tan-teaget-ssass Knowledge Distillation (NCKD) components:
LDKD (p .y) + ( (rckD ), bp(p)) )) + BaCNcKD(PTT P()))
(2)
where bp(.) is the binary probabilities function of the target class p! (py¹ ), and all the other non-target classes (.) and p7: = a(2,,//) and & = c LTeKD transfers target class knowledge and LNCKD captures relationships between non-targe! classes, revealing NCKD's importance in KD.
DOT: Distillation Oriinnted Trainer (DOT) 19 maintains separate momentum buffers for cross- entropy and distillation loss gradients. For each mini-batch, DOT computes gradients Joe and gua from
3

---

LCE and LKD respectively, then updates momentum buffers and Vsa with different coefficients:
Ui 9 + (14 A)Ue; via aıa (p A)ww
(3)
where ja is the base momentum and A is a hyperparameter controlling the momentum difference. By applying larger momentum to distillation loss gradients, DOT enhances knowledge transfer while mitigating optimization trade-offs between task and distillation objectives. However, DOT has two key limitations: (1) it fails address the inherent conflict between target-class and non-target-class knowledge flows, which can lead to suboptimal optimization trajectories; and (2) it lacks systematic analysis of the optimization dynamics across different loss components and their corresponding gradient momenta, particularly in handling low- confideieecce dark knowledge that introduces noisy signals and impedes effective knowledge transfer.
3.2 GSNR-Driven Momentum Allocation
Unlike previous DOT [19] that only decouples task and distillation gradients at a single level, our approach introduces dual-level decoupling strategy further decomposing the gradient of the student model's 's training loss into three components: task-oriented gradient (TOG), target-class gradient (TCG), and non-target-class gradient (NCG). We define its gradient hignal-to-noiss ratio (GSNR) as:
| Var VL
|اوواع FiglT
1 + 9 + -
GSNR-
(4)
Flllg
where T is the sampling interval step size (default: 200), and g, denotes the gradient at step / including TOG VL., TCg = VTeCD, and NCG VLxCKD. For target class t and any class i, the gradients can be expressed as:
p 1, pp.
i t, t, TCG, = pr. pf clse
fi t, NCC, else
0, p
ii= t, clse
TOG,
-pp (pf p.))
p
(5)
where p and p. denote the softmax probabilities of the student and teacher models respectively. The detailed mathematical derivation this process provided in Appendix A.2
During stochastic optimization in deep neural networks, gradients computed at the end of each forward pass are used for backward propagation. These gradients form a sequence of stochastic vectors, where the statistical expectation and variance of the gradient can be estimated using the short-term sample mean within temporal window (20). Empirically, we find that gradient sampling at intervals of 200 iterations yields better performance. This estimation serves as the foundation for calculating the GSNR (45). Specifically, the statistical expectation represents the true gradient direction, while the variance quantifies noise introduced by stochastic sampling.
TOG
TOG
TOG
TDG
022
020
DENR
BSNN
6.10
000
0.00
0.0
220
0.00
255
422
422
Internal
(a) KD-GSNR
(b) DeepKD-GSNN
(c) KD-BSNR
(d) DeepKD-BSNR
Figure 2: Comparison of gradient and buffer SNR between vanilla KD and DeepKD: (a) KD GSNR with less component separation, (b) DeepKD GSNR with better component distinction, (c) KD BSNR with limited separation, and (d) DeepKD BSNR with enhanced component differentiation.
To better understand the optimization dynamics, we conduct empirical analysis of GSNR curves of decoupled vanilla KD throughout the training process. As shown in Figure 2 a). vanilla KD exhibits consistently high signal-to-noise ratios (SNR) for non-target-class gradients (NCG) under identical momentum coefficients, indicating inherent difficulties in transferring "dark knowledge" to the student model. This persistent gradient divergence suggests unresolved conflicts between distillation objectives and target tasks, ultimately causing SNR instability in the gradient accumulation buffer (BSNR) (Figure Notably, gradient divergence reflects suboptimal convergence since well-converged models typically exhibit near-zero gradients, leading to smoothly decaying SNR

---

trajectories. We hypothesize that gradient components with higher SNR should be prioritized with heuristic weighting to help optimization. Figure b demonstrates that our DeepKD framework achieves accelerated and better absorption of dark knowledge from NCG while maintaining equilib- rium among all gradient components. The resultant SNR trajectories exhibit smooth uniformity the buffer (Figure ( d)). validating our theoretical proposition. Empirical validation further corroborates that momentum coefficients for gradient components positively correlate with their respective SNRs during KD optimization. Crucially, experimental results reveal robustness to specific coefficient values, aligning with observations in prior work (DOT 19
Let's first examine the standard optimization formulation of SGD with momentum 46
V++1 = 9, μrf;; 0++1 0, -yy+
(6)
where vs and 0g represent the momentum buffer and the model's trainable parameters at time step t, 9: is the current gradient, μ is the base momentum coefficient, and y is the learning rate. Through analysis of the GSNR in Figure[[a) a), we observe that NCG and TOG maintain higher GSNR compared to TCG. This key observation motivates our adaptive momentum allocation strategy:
P1OG TOg (a + A) Proo; VICO TCg (μ A)rreoi UNCG NCC (μ AJUNCO
(7)
where A is a hyperparameter controlling the momentum difference. As shown in Figure 2(b) & (d), our DeepKD with different momentum coefficients achieves significantly improved GSNR in both gradient buffers and raw gradients, further validating the necessity of our deep momentum decoupling approach for gradient components. Our GSNR- driven approach ensures each knowledge component follows its optimal optimization path while maintaining component independence, leading more effective knowledge transfer. Note that our method is equally applicable to the Adam optimizer 47 by modifying only its first-order momentum, as validated on DeiT 48
(see Table
3.3 Dynamic Top-K Masking
While existing advanced approaches 18 19
typically process non-targ non-target class logits through identify two critical limitations in these conventional approaches: (1) Teacher models demonstrate extreme confidence in target class (softmax probabilities >0.99 for more than 92% of samples), while non-target classes collectively exhibit low confidence yet contain valuable dark knowledge, as evidenced in Figure (2) The dark know Indge from non-target classes exhibits varying degrees of assimilability classes semantically similar the target (e.g., "tiger" for target "cat") provide beneficial dark knowledge, while semantically distant classes (e.g.. "airplane") introduce noise and learning difficulties.
either uniform treatment or weighted separation
29
50000
40000
21.0
30000
2000
3.6
Phuze
Thrrtlrr Prese
Hed Thoc
10000
75.5
25.6
Probabllity
Escch
330
(a) Confidence target class
(h) Accuracy static top-k masking (c) Training process of the best top-k
Figure 3: Analysis of top-k masking strategy. (a) Distribution of teacher model's confidence target classes. (b) Accuracy comparison of different static top-k values for knowledge distillation. (c) Learning curve divided into distinct training phases with the optimal top-k masking approach.
To address these limitations, we first develop a static top-k masking approach that permanently filters classes with extreme semantic dissimilarity through fixed k-value masking, yielding baseline improvements as shown bb. Building upon this, we propose a more sophisticated dynamic top-k masking mechanism that implements phase-wis k-value scheduling inspired by curriculum learning [27]. This mechanism operates in three distinct phases via accuracy curves (Figure (c)):
Easy Learning Phase: K increases linearly from 5% of the total number of classes to the optimal static value
Transition Phase: Maintains the optimal static K value
Hard Learning Phase: Expands K linearly to encompass the full class count
5

---

The optimal static K value is determined through ablation studies or by using 20% of training data to reduce training cost. The complete process of dynamic top-k masking learning is illustrated in Figure c)..
c). For each training iteration i, we compute the mask M, as:
M, [(rank(z K:)
where rank(.) represents the rank of logits in ascending order, and K; gradually increases from 5% of classes to the total number of classes. The masked distillation loss is formulated
CDTM = (NCKD(o(M,, @ / (M, @ <x(/T))
(9)
where @ denotes element-wise multiplication. This mechanism effectively suppresses noise while preserving semantically relevant dark knowledge.
3.4 DeepKD Framework
(ur 4jp
TASK
TOG
Ground Truth
Task CE Loss
Trie-ere
Task Momentum
hard labal
PC
mask label
larget
Pm g (t d)p
non-larget
TOKD
TOG
Target Class KL Loes
Target class
Target Momentum buffer
d yu Update
Teacher
Pi
Decoupling
PE
- (it A0p
NCKD
NDG
cat
Non-larget Clsss Loss
Momentum buffer
Mon-larget
Student
dass Gredient
Dynamic Top-k Mask
Figure 4: Detailed architecture of our DeepKD framework. Input images flow through teacher and student networks, producing target (yellow) and non-target (green) logits. The framework uses three independent gradient paths (task-oriented, target-class, and non-targed non-target-clas)) with separate momentum buffers. Dynamic Top-k Mask filters low-confidence non-target logits (gray cells).
Building upon theoretical analysis of GSNR, we propose DeepKD, comprehensive framework that introduces deeply decoupled optimization with adaptive denoising for knowledge distillation. As illustrated in Figure 4 DeepKD decomposes the knowledge transfer process into three parallel gradient flows: task-oriented gradient (TOG), target-class gradient (TCG), and non-target-class gradient (NCG). Each gradient flow is managed independently with its own momentum buffer and optimized based their distinet GSNR properties. This decoupled architecture enables more effective knowledge transfer by allowing each component to be optimized independently. The complete loss function of DeepKD (see Algorithm| Tin the Appendix for detailed implementation):
DeepXD = aLCE (p", y) + 7 (BICCCK
bp(p))) + 32CMrM)
(10)
where LcE represents the standard cross-entropy loss, and 3; are fixed coefficients that balance the contribution of each loss component, LreK is the target class loss, and CD is the dynamic top-k masking loss of the non-target classes. This formulation enables the framework effectively combine task-specific learning with knowledge distillation while maintaining computational efficiency.
Experiments
4.1 Datasets and Implementation
We conduct comprehensive evaluations on three widely-used benchmarks: CIFAR-I00 [50] (100 classes, 50k training/ 10k validation 32 x 32 images), ImageNet- IK [5I] (1,000 classes, 1.28M/50K images cropped 224 224), and MS-COCO [52] (80-class detection, 118k training/Sk validation images). For implementation, we follow standard practices using SGD optimizer with momentum 0.9 and weight decay x 10-4 (CIFAR) or x 10-4 (ImageNet). The training schedule varies by dataset: CIFAR uses 240 epochs with batch size 64 and initial learning rate 0.01-0.05, while ImageNet uses 100 epochs with batch size 512 and learning rate 0.2. All experiments were conducted on system equipped with an Nvidia RTX 4090 GPU and AMD 64-Core Processor CPU.

---

Table Top-1 Accuracy (%) on CIFAR- 100 validation set. Results show homogeneous distillation (same architecture, different capacity) across caature--bsetd and logit-based methods. Performance gains from our DeepKD framework are highlighted in blue and red.
ResNet32x4
VGG11
WRN-40-2
40-2
RexNet56
ResNet ResNet110
Teacher
79.42
74.64
75.61
75.61
72.34
74.31
74.31
Type
R4
VGG8
WRN-40-1
WRN-16-2
ResNet20
ResNet32
ReaNet20
Student
72.50
70.36
71.98
73.26
69.06
71.14
69.06
FilNet
30
73,50
71.02
72.24
73.58
69.21
71.06
68.99
Feature
SimKD CAT-KD
78.08
74.89
74.53
75.53
S4
76.91
74.65
74,82
75.60
71.62
73.62
71,37
KD 11
73.33
72.98
73.54
74.92
70.66
73.08
70.67
KD+DOT
74.98
73.77
73.87
75.43
70,97
KD+LSKD 49
76.62
74_36
74.37
76.11
71.43
74.17
71.48
KD+Ours (w/o lop-k)
76.69 3.30
74.96 11.98
74.80 1120
76.14 11:22
11.13
11.12
(0.92
KD+Ours (w. kop-k)
77.03
75.12
75.05
76.45,1155
71,90,1.24
74.35
71,82
DKD [8]
76.32
74.68
74,81
76.24
71.97
74.11
70.99
DKD+DOT
76.03
84.86
74,49
75.42
71.12
71.58
DKD+LSKD 49
77.01
74.81
74.89
76..39
72.32
74.29
71.48
DKD+Ours (w/o top
77.25 0.93
75.09 +0.41
75.24 .0.24
yo.2 76.72.
72.86 0.99
10.21 74.48
1:007
Logit
DKD+Ours (w. top-k)
77.54
75.19
75.42
73.05
72.28
MI MI.KD [55
77.08
75.18
75.35
76.63
72.19
74.11
71.89
MLT 19
76.06
74.96
74.38
75.72
71.41
73.83
71.65
MLKD+LSKD 49
78.28
75.22
75,56
76.95
72.33
74.32
72.27
MLKD++Ous lop-k)
+1771
76.21+1 oa
+7.45
78.15 1.55
73.75
75.88
73.03
MI. MI.KD+Ours (w. tnp-k)
79.15
76.45
77,82
78.49
74.12
76.15
73.28
CRIDD 13
77.60
75.27
75.58
76.45
72.10
74.42
72.03
CRLD+DOT 19
76.54
74.34
74.75
75.57
71.11
73.91
70.67
CRLD-LSKD 49
78.
74.74
76.28
76.92
72.09
75.16
72.26
CRLD=Ouss top-k)
78.90
76.29 02
76.98-1144
77.99.1.54
73.29
76.03
73.07
CRID-Ours (w. kop-k)
79.25
76.58
77.35
78.42,+1.9
73.85
76.48
73.52
4.2 Image Classification
Results on CIFAR-100. Our DeepKD framework shows consistent improvements in both ho- mogencous and heterogeneous distillation settings. On homogeneous architectures (Table T DeepKD achieves accuracy gains of +0.61 % without top-k masking and +0.67%+43.70% with masking, outperforming feature-based methods by 1.2-4.8%. The top-k variant further im- proves performance by up to +1.86% % with MLKD+Ours reaching 79.15 % accuracy. In het- erogeneous scenarios (Table [ DeepKD shows strong generalization: CRLD+Ours achieves 72.85% for ResNer32 x MeoM-MMMbbeeee-e (+2.48%), while KD+Ours attains 77.15% for WRN-40- 2-t (+3.18% % Performance remains stable (variance 50.5%) under hyperparameter variations, confirming DeepKD's s effectiveness in handling conflicting distillation signals.
Results ImageNet-1K. As shown in Table 3 DeepKD achieves significant improvements across diverse teacher-student pairs. For ResNet50 -MobileNet-VI. KD+Ours (w. top-k) boosts top-1 accuracy by +4.15% (74.6%% vs. 70.50%). the largest gain among all configurations. CRLD+Ours (w. top-k) establishes new state-of-the-arr results: 73.34% % MeNNet34->ReNNet +0.97% and 75.75% % (RegNetY-16GF-DDe performance, contributing additional gains of +0.30 %-40.84% in top-5 accuracy. Notably, our framework demonstrates strong scalability: (1) For lightweight students MoobileNet-V/DDiii---- improvements reach +3.82 13.82%+44.114 : (2) With large teachers (RegNerY- top-5 accuracy ex- ceeds 93.85% % (CRLD+Ours), surpassing all feature-based methods. These results validate DeepKD's effectiveness in large-scale distillation scenarios.
+1.89 )).. The dynamic top-k masking consistently enhances
4.3 Object Detection on MS-COCO
DeepKD shows strong performance on object detection (see Table 4). With dynamic top-k masking (t). our method improves baseline KD by +1.93% AP (32.16% vs. 30.13%) and exceeds ReviewKD's 33.71% AP using only logit distillation. DKD+Ourst achieves 34.20% AP, the best among all KD variants, with +1.86 % gain over vanilla DKD. The dynamic top-k mechanism provides additional improvements of 10.05%--03449 AP, with the largest boost in AP75 (+2.64% for KDf). Our approach demonstrates better localization than feature-based LSKD,, reaching 36.59% AP75 (vs. LSKD'ss 36.34%) for DKD These results confirm DeepKD's effectiveness for dense prediction tasks.
7

---

Table Top-1 Accuracy (%) on CIFAR- 100 validation set with heterogeneous reacher-studen pairs.
Methods are grouped by type Sreature/logit-baseee Performance gains are shown in blue and red.
R4sNet32x4
ResNer32 x4
ResNet32x4
WRN-40-2
WRN-40-2
VGG13
ReaNet50
Teacher
79.42
79.42
75.61
Type
SHN-V2
WRN-16-2
WRN-40-2
ResNet8x4
MN.V2
MN.V2
MN.V2
Student
71.82
73.26
75.61
72.50
64.60
Review SimKD
77.78
76.11
78.96
74.34
71.28
Feature
53
78.39
77.17
79.29
75.29
70.10
69.44
69.97
CAT-KD
78.41
76.97
KD 0
74.45
74.90
77.70
73.97
68.36
67.37
67.35
KD+DOT 19
73.35
75.04
77.34
75.96
68.36
68.15
68.46
KD+LSKD 49
75.56
75.26
77.92
69.23
68.61
KD+Ours (wlo top-k)
76. 1.000
75.88.-0.8
78,38
76.69
69.39
69 (9.36,,120
69.13
(w. top
76.45, 2,00
76.12;11.22
0,000
77.15 $3.18
:1.45
|2.55
DKD [88]
77.07
75,70
78.46
75.56
69.28
69.71
70.35
DKD+DOT 19
77.41
75.69
78.42
75.71
62.32
68.89
70.12
DKD+LSKD (
77.37
76.19
78.95
76.75
70.01
69 69.98
70.45
DKD+Ours (W7o top-k)
27.68-0.61
76.61
79.57
76.8641.30
70.29
20.04
70.48 40.00
Logit
DKD+Ours (w. lop-k)
77.95-0..8
76.89.1.19
79.82
-9.82-1130
76.90,,1.4
70.65
70.38
70.72.;0.77
MLKD [S
78.44
76.52
79.26
77.33
70,78
70.57
71.04
ML.KDDDOT 19
78.53
75.82
79.01
76.53
69.15
68.26
67.73
MI.KDeISK SKD
78.76
77.53
79.66
77.68 77.88
71.61
70.94
71.19
MLKD+Ours (w/o top-k)
80.55 -0.35
28.28....76
21.40
78.3 40.se
72.17
73.04 1
MLKD+Ours top-k)
80.92.-2.46
78.65
81.78
78.49
72.53
72.82 72.82.2.25
73.40
CRLD 133
78.27
76.92
80,21
77.28
70.37
70.39
71.36
CRLD+DOT 19
78.33
75.97
79.41
76.41
64.36
61.35
69.96
CRID-LSKD 49
78.61
77.37
80.58
78.03
71,52
70.48
71.43
CRI.D-Ouss (w/ top-k)
79.72 L:46
78.79 1.87
81.82,,1411
78 28.62,.5 1.84
72.09 11.72
71.99
72.01 :0.66
CRLD=Ours (w. top-k)
80.15 1.88
79.25
29.25..2.35
82.35 L.29
79.18
72.85 #2445
72.65+2.0
72.78
Table 3: Accuracy (%) ImageNet-IK IK validation set. N/A indicates that the data is not available.
Teacher/Student
ResNer34/ReNet18
ResNet50/MNVV
RegNe 16CF/Deit-/iigg
Type
Accuracy
top-1
top-5
top-5
top-1
top-5
Teacher
73.31
91.42
76.16
92.86
82,89
96.33
Student
69.75
89.07
68.87
88.76
72.20
91.10
SimKD
71.59
90.48
72.25
90.86
N/A
N/A
Feature
CAT-KD
71.26
90.45
72.24
91.13
N/A
N/A
1
71.03
90.05
70.50
89.80
73.15
91.85
KD+DOT 9
71.72
90,30
73.09
91.11
73.42
92,10
KD+LSKD
71.42
90.29
72.18
90.80
73.27
91.95
KD+Ours (wlo top-k)
72.41
91.0551..0
74.3245.82
91. 9..94.2..11
74.36.1..11
92.85
KD+Ours (w. top-k)
72,85
91.35
74.65
92.25.42.40
7..83..
93.15..1..0
DKD 18
71.70
90.41
72.05
91.05
73.35
92.05
DKD+DOT
72.03
90.50
73.33
91.22
73.66
92.25
Logit
DKD+L.SKD 49
71.88
90,58
72.85
91.23
73.48
92,15
DKD+Ours (w/l lop-k)
72.78
90.96
74.41
92.08 4.08
74.57
93.07 ..02
DKD+Ours top-k)
73.15
91.25 0.84
74.43 2..8
91. 0..90
74.95 1.60
93.36 1.31
MLKD S5
71.90
90.55
73.01
91.42
73.54
92.25
MLKD+DOT 19
70.94
90.15
71.65
90.28
73.25
91.95
MLKD+LSKD
90.74
73.22
91.59
73.78
92.45
MLKD+Ours (w/o top-k)
73.18
91.23
74.77
92.35
75.15
93.48
MLKD+Ours (w. top-k)
73.31
91.39
74.85
92.45
26.446,...0
93.73
CRLD 13]
72.37
90.76
73.53
91.43
73.82
92.55
CRLD+DOT
71.76
90.00
72.38
90.37
73.37
92.05
CRLD+LSKD 49
72.39
90.87
73.74
91.61
73.95
92.65
CRLD+OOrs (w/o top-k)
73.18-00.1
91.23.40.47
14.10.40
91.49440.06
75.35
93.35
CRLD+Ours (w. top-k)
73.34
91.38
74,85
92.45
75.75
93.85
Ablation Study
Momentum Coefficients and Loss Functions. Based on gradient signal-to-noiss ratio (GSNR) analysis, we decompose the gradient momentum in DeepKD into Target-Class Gradient (TCG) and Non-Target-Clan Gradient (NCG). To validate this decoupling strategy, we conduct comprehensive ablation studies on CIFAR-100 using ResNer32x4(teache Table 5). The results demonstrate the effectiveness of our approach and highlight the importance of each component. Notably, DeepKD introduces only one parammer A, which proves
and ResNet8 44(tudent) pairs (see

---

Table Results on MS (S-CC000vval2017) based on RCNN--FNNSN1 Teacher-student pair is ResNet50 & MobileNet-V2 V2. The values of columns with & based on dynamic top-k masking.
Metric
Teacher
Student
ReviewKD
KD
KD-LSKD
KD+Ours
KD+Ouns
DKD
DKD-LSKD
DKD+Ours
DKD-Ours
42.04
29.47
33.71
30.13
31.71
-2.0
32.16..00
32.34
33.98
33,99.1.03
3..20-1.00
52.77
52.88-2
53.77
55.11-134
-..31
APT 43.81
30.90
36.13
31.35
33.40
33.65.29)
33.99
34.01
36.34
36.35 42.5
36.59.200
to be robust across different datasets. Following DOT 19], set A=0.075 for KD++eepKD CIFAR-100, and A=0.05 for DKD+DeepKD, MLKD+DeepKD, and CRLD+DeepKD, For ImageNet, where teacher knowledge more reliable, we use A=0.05 for all variants. Our ablation studies on loss functions reveal that each component contributes significantly to the overall performance, with NCKD being the dominant factor-consst consistent with findings in DKD [8] These results validate our decoupling strategy and demonstrate its effectiveness in improving model performance.
Table 5: Results of using our DeepKD+KD for Resnet32 x 4 Resnet8 x (studeett CIFAR- 100. Left: impact of momentum coefficients (A); Middle: effectiveness of different loss combinations; Right: performance with dynamic top-k masking and curriculum learning.
Momentam Coefficients
Loss Functions
Dynamic top-k with curriculum learning
Aroc
Areg
Axco
top-1
tep-5
TASK
TCKD
NCKD
top-1
top-5
k-value
Phasel
Phuse2
top- 1
top-5
170
91.62
0.075
0.00
0.00
74.89
91.27
/
X
X
73.28
92.89
60
170
77.03
92.07
0.00
0.075
0.00
74.58
93.52
/
X
74.58
93.52
55
60
160
77.31
93.38
0.00
0.00
0.075
75.25
93.61
X
/
75.25
93.61
60
180
77.13
92.28
0.075
0.075
0.00
75.41
93.71
/
/
X
71.50
93.11
60
40
170
77.20
92.51
/
170
0.075
0.00
0.075
76.19
93.97
X
V
76.06
94.12
60
60
160
77.29
93.12
0,075
0.075
0.075
76,69
94.21
V
V
76.69
94.21
60
180
77.21
93.32
Dynamic Top-k Masking, To discard the impact of dynamic top-k masking, we conduct the above ablation studies on momentum coefficients and loss functions without this strategy. For the dynamic top-k masking configuration, we empirically set the parameters k-value = 55, Phasel = 60, and Phase2 170 (see T). The experimental results demonstrate that even simple hyperparameter tuning can further improve model performance, suggesting that integrating a dynamic top-k masking mechanism into KD holds great potential and warrants further exploration in future work.
Table 6: Results of using our DeepKD with different distillation methods on CIFAR-100. We evaluate the impact of decoupled gradients (Decoupled) and dynamic top-k masking (DTM) strategies.
Method
KD
DKD LB
ML.D
S
CRLD [k]
Decoupled
DTM
x
73.33
76.69
74.89
77,03
76.32
77.25
76.58
77.54
77.08
78.81
77.41
79.15
77.60
78.90
77.93
79.25
Furthermore, we conduct ablation studies on both gradient decoupling and dynamic top-k masking strategies. The results demonstrate that each component individually enhances performance compared to the original distillation methods, while their combination yields further improvements. Note that when DTM is enabled individually, the mechanism operates on all logits including the target class.
Discussion
6.1
Computational Complexity Analysis
To address concerns about computational overhead, we provide comprehensive analysis of DeepKD's training efficiency and resource requirements. Table 7 shows the memory consump- tion and training time overhead for different teacher-student pairs on CIFAR-100 and ImageNet- 1K DeepKD introduces minimal memory overhead, with increases of less than 1% on CIFAR-100 and less than 0.2% on ImageNet- This modest increase primarily stems from storing student gradients during distillation, the teacher model remains frozen. The efficient CUDA memory allocator fur- ther mitigates additional memory footprint, confirming DeepKD's suitability for large-scale training scenarios.

---

Table Computational overhead analysis of DeepKD: All experiments use a single 2080Ti GPU for CIFAR- 1-0 and two RTX 4090 GPUs for training on ImageNet- IK.
Memory (MB)
Training Time (Hours)
Dataset
Method
Baseline
DeepKD
Baseline
DeepKD
KD
799
805 (+0.75%)
1.6
2.6 (+62%)
DKD
799
805 (+0.75%)
1.7
2.7 (+60%)
CIFAR-100
MLKD
983
987 (+0.41%)
9.2
12.3 (433%)
CRLD
981
985 (+0.41%)
2.9
4.5 (+52%)
KD
21344
21370 (+0.12%)
20.0
29.4 (+47%)
DKD
21350
21370 (+0.12%)
20.1
29.8 (+48%)
ImageNet-IK
MLKD
34910
34960 (+0.14%)
34.2
53.1 (+55%)
CRLD 34862
34909 (+0.13%%
32.2
51.2 (459%)
6.2 Training Time and Convergence Analysis
Table 8: Comparing training epochs and final accuracy. All baseline methods are trained for the standard number of epochs (240 for CIFAR-IOO 100, 480 for MLKD, 100 for ImageNet-1K)
CIFAR- 100
ImageNet-IK
Dataset
Method
Epochs
Top-1 1 (%)
Epochs
Top-1 (%)
240
73.33
100
71.03
DKD
76.32
71.70
Baseline
MLKD
480
77.08
100
71.90
CRLD
240
77.60
100
72.37
KD+Ours
160
76.83
65
71.76
DeepKD
DKD
160
77.31
72.16
Ours
320
79.04
72.52
CRLD+Ours
160
78.87
65
72.98
While DeepKD introduces moderate per-epoch training time increases, it achieves superior accuracy with substantially fewer epochs than baseline methods, resulting in a favorable overall time-accuss acceaccuracy trade-off. Table 8 demonstrates that DeepKD enables faster convergence while maintaining higher final performance. The additional training cost is justified by substantial performance gains. Since knowledge distillation is typically a one-time training process, investing more resources achieve superior models standard practice the field. Importantly, this additional training cost does not affect the final inference speed of the student model, making DeepKD highly efficient route to setter-prffrmiig models.
Limitations and Future Work
While our current work focuses on logit-based distillation. the SNR-driven momentum decoupling mechanism naturally extends feature distillation scenarios. By treating feature alignment losses as additional optimization components, our framework can automatically handle multi-level knowledge transfer without manual weighting, complementing existing feature enhancement techniques like attention transfer [31 and contrastive distillation [58). Future work could explore the application of our framework to more complex scenarios, such as multi-teacher distillation and cross-modal knowledge transfer. Additionally, our dynamic top-k masking strategy shows promising results in improving distillation performance, suggesting potential for further refinement and adaptation to different model architectures and datasets.
8 Conclusion
This paper presents DeepKD, novel knowledge distillation framework that introduces SNR-driven momentum decoupling to address the gradient conflict between task learning and knowledge transfer. Our approach automatically allocates appropriate momentum coefficients based on gradient SNR characteristics, enabling effective optimization of both task-specific and knowledge distillation objec- tives. Through extensive experiments on multiple datasets and model architectures, we demonstrate that DeepKD consistently improves the performance of various SOTA distillation methods.
10

---

Acknowledgments
This work was supported in part by National Natural Science Foundation of China No. 62088100 and No. N0.62302381. 1. The Authors are with the National Key Laboratory of Human- Hybrid Augmented Intelligence, National Engineering Research Center of Visual Information and Applica- tions, and Institute of Artificial Intelligence and Robotics, Xi' an Jiaotong University, Xi an, Shaanxi, China.
References
G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in neural network," arXiv preprint arXiv:1503.022 2015.
[2] W. Cao, Y. Zhang, J. Gao, A. Cheng. K. Cheng, and Cheng, "Pkd: General distillation framework for object detectors via pearson correlation coefficient," Advances Neural Information Processing Systems, vol. 35, pp. 15 394-15 406, 2022.
[3] C. Yang, H. Zhou, Z. An, X. Jiang, Y. Xu. and Q. Zhang. "Cross-image image relational knowledge distillation semantic segmentation." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022. pp. 12319- 328.
[4] W. Feng, C. Yang, Z. An, L. Huang, B. Diao, F. Wang, and Y. Xu, "Relational diffusion distillation for efficient image generation," " Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 205 213.
[5] J. Gou, B. Yu, J. Maybank, and D. Tao. "Knowledge distillation: A survey," International Journal of Computer Vision, vol. 129, no. 6, pp. 1789-1819, 2021.
[6] P. Agand, "Knowledge distillation from single-task teachers to multi-task student for end-to-end au- tonomous driving." " in Proceedings of the AAAI Conference Artificial Intelligence, vol. 38, 21, 2024, pp. 23 375-23333776
[7] Z. Zhao, K. Ma, W. Chai, X. Wang, K. Chen, D. Guo, Y. Zhang, H. Wang, and G. Wang. "Do we really need complex agent system? distill embodied agent into single model," arXiv preprint arXiv:2404.04mm 04619, 2024.
[8] J. C.-Y. Chen, S. Saha, E. Stengel-Eskin and M. Bansal, "Magdi: Structured distillation of multi-agent agent interaction graphs improves reasoning smaller language models," arXiv preprins arXiv:2402.0162) 2024.
[9] H. Zhang, D. Chen, and C. Wang, "Adaptive multi-teacher knowledge distillation with meta-lcamingg" in 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2023, pp. 1943-1948.
[10] C. Yang, X. Yu. H. Yang, Z. An. C. Yu, L. Huang, and Y. Xu. "Multi-teacher knowledge distillation with reinforcement learning for visual recognition." in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 9, 2025, pp. 9148 1956....
[11] B. Heo, J. Kim. S. Yun. H. Park, N. Kwak. and J. Y. Choi, "A comprehensive overhaul of feature distillation." in Proceedings of the IEEE/CVF international conference on computer vision. 2019, pp. 1921-1930.
[12] H. Wang. S. Lohit, M. N. Jones, and Y. Fu, "What makes a" good" data augmentation in knowledge distillation-a statistical perspective," Advances in Neural Information Processing Systems, vol. 35, pp. 13 13445--34600 469, 2022.
[13] W. Zhang, D. Liu, W. Cai, and C. Ma, "Cross-view consistency regularisation for knowledge distillation," in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 2011-011100 1-202...
[14] M. I. Hossain, S. Akhter, N. I. Mahbub, C. S. Hong, and E.-N. Huh, "Why logit distillation works: novel knowledge distillation technique by deriving target augmentation and logits distortion," Information Processing & Management, vol. 62, no. 3. 104056, 2025.
[15] M. I. Hossain, S. Akhter, C. S. Hong. and E.-N. Huh, "Single teacher. multiple perspectives: Teacher knowledge augmentation for enhanced knowledge distillation," in The Thirteenth International Conference on Learning Representations, 2025.
[16] L. Yuan, F. E. Tay, G. Li, T. Wang, and J. Feng. "Revisiting knowledge distillation via label smoothing regularization." in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020, pp. 1903-3911.
11

---

[17] L. Wang, L. Xu. X. Yang. Z. Huang, and J. Cheng. "Debiased distillation for consistency regularization." in Proceedings of the AAAI Conference Artificial Intelligence, vol. 39, no. 8, 2025, pp. 7799-7807.
[18] B. Zhao, Q Cui, R. Song. Y. Qiu, and J. Liang, "Decoupled knowledge distillation." in Proceedings of the IEFF/CVF Conference computer vision and pattern recognition, 2022, pp. 11 953-11 962.
[19] B. Zhao, Q. Cui, R. Song, and J. Liang, "Dot: A distillation-orientedd trainer," in Proceedings of the IEEE/CVF International Conference Computer Vision, 2023, pp. 6189-6198.
[20] J. Medhi, Stochastic processes. New Age International 1994.
[21] J. Liu, G. Jiang, Y. Bai, T. Chen, and H. Wang, "Understanding why neural networks generalize well through gsnr parameters," " arXiv preprint arXiv:2001.077555 107384, 2020.
[22] S. Jelassi and Y. Li, "Towards understanding how momentum improves generalization in deep learning," in International Conference on Machine Learning. PMLR, 2022, pp. 9965-10040.
[23] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, "Visualizing the loss landscape of neural nets," Advances in neural information processing systems, vol. 31, 2018.
[24] T. Rainforth, A. Kosiorek, T. A. Le, C. Maddison, M. Igl, F. Wood, and Y. W. Teh, "Tighter variational bounds are not necessarily better," in International Conference Machine Learning. PMLR, 2018, pp- 4277-4285.
[25] N. S. Keskar, D. Mudigere. J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, "On large-batch batch training for deep learning: Generalization gap and sharp minima." arXiv preprint arXiv:1609.00 04836, 2016.
[26] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, "Averaging weights leads to wider optima and better generalization," arXiv preprint
2018
[27] Y. Bengio. J. Louradour, R. Collobert, and J. Weston, "Curriculum learning." in Proceedings of the 26th annual international conference on machine learning. 2009. pp. 41-48.
[28] Z. Li. X. L. Yang, B, Zhao, R. Song, L. Luo, J. Li, and J. Yang. "Curriculum temperature for knowledge distillation," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 2, 2023, pp. 1504-1512.
[29] L. Xu, J. Ren, Z. Huang, W. Zheng, and Y. Chen, "Improving knowledge distillation via head and tail categories," IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 5, pp. 3465 3480, 2023.
[30] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, "Fitnets: Hints for thin deep nets," " arXiv preprint 141 2014.
[31] S. Zagoruyko and N. Komodakis, "Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer." arXiv preprint arXiv:1612.039228 2016.
[32] W. Park, D. Kim, Y. Lu, and M. Cho, "Relational knowledge distillation." in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2019, pp.
[33] P. Chen, S. Liu, H. Zhao, and J. Jia, "Distilling knowledge via knowledge review," in Proceedings of the IEEE/CVF conference on compuder vision and pattern recognition, 2021. pp. 5008-5017.
[34] Z. Yang. A. Zeng, Z. Li, T. Zhang, C. Yuan, and Y. Li. "From knowledge distillation to self- koollknoweedge distillation: A unified approach with normalized loss and customized soft labels," in Proceedings of the IEFF/CVF International Conference Computer Vision, 2023, pp. 17 185-17 194.
[35] J. Guo, M. Chen, Y. Hu, C. Zhu, X. He, and D. Cai, "Reducing the teacher-student gap via spherical knowledge disitllation," arXiv preprint arXiv:2010.0440 2020.
[36] J. Wang. L. Lu, M. Chi, and J. Chen, "Mdr: Multi-stage decoupled relational knowledge distillation with adaptive stage selection," in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 2175 2183.
[37] C. Li, X. Teng, Y. Ding, and I.. Lan, "Ntce-kd: Non-targee--clas--nhanncce knowledge distillation," Sensory, vol. 24, no. 11, 2024. [Online]. Available:
[38] Y. Zhu. N. Liu, Z. Xu. X. Liu, W. Meng. L. Wang. Z. Ou, and J. Tang. "Teach less, learn more: On the undistillable classes in knowledge distillation," Advances in Neural Information Processing Systems. vol. 35, pp. 32011 10111-22222 024, 2022.
12

---

[39] W. Sun, D. Chen, S. Lyu, G. Chen, C. Chen, and C. Wang, "Knowledge distillation with refined logits." arXiv preprint arXiv:2408.00 077003 2024.
[40] Y. Niu, L. Chen, C. Zhou, and H. Zhang, "Respecting transfer gap in knowledge distillation," Advances in Neural Information Processing Systems, vol. 35, pp. 27 195-27 206, 2022.
[41] T. Huang, Y. Zhang, M. Zheng, S. You, F. Wang. C. Qian, and C. Xu, "Knowledge diffusion for distillation." Advances in Neural Information Processing Systems, vol. 36, 2023.
[42] Z. Li, X. L. Yang, B. Zhao, R. Song. L. Luo, J. Li, and J. Yang. "Curriculum temperature for knowledge distillation," in Proceedings of the AAAI Conference Artificial Intelligence, vol. 37, no. 2, 2023, pp. 1504 1512.
[43] Y. M. Saidutta, R. S. Srinivasa, J. Cho, C.-H. Lee, C. Yang, Shen, and H. Jin, "Cifd: Controlled information flow to enhance knowledge distillation," Advances in Neurol Information Processing Systems, vol. 37, 2024.
[44] T. Huang, S. You, F. Wang, C. Qian, and C. Xu, "Dist+: Knowledge distillation from a stronger adaptive teacher," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 7, pp. 1793- 1805, 2025.
[45] M. Michalkiewicz, M. Faraki, X. Yu, M. Chandraker, and M. Baktashmotlagh, "Domain generalization guided by gradient signal to noise ratio of parameters," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 6177-6188.
[46] I. Sutskever, J. Martens, G. Dahl, and G. Hinton, "On the importance of initialization and momentum in deep learning." in International conference on machine learning. PMLR, 2013, pp. 1139-1147.
[47] D. Kingma and J. Ba, "Adam: method for stochastic optimization." arXiv preprint arXiv: 1412.6980, 2014.
[48] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. "Training data-efficient image transformers & distillation through attention," " in International conference machine learning. PMLR, 2021. pp. 10 307-10357
[49] S. Sun, Ren, J. Li, R. Wang, and X. Cao, "Logit standardization in knowledge distillation," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 15 731- 15 740.
[50] A. Krizhevsky, G. Hinton et al., "Learning multiple layers of features from tiny images," " 2009.
[51] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang. A. Karpathy, A. Khosla, Bernstein et al., "ImageNet large scale visual recognition challenge," IJCV, IJCV,2015 2015.
[52] T.-Y. Lin, M. Maire, S. Belongie, Hays, Perona, D. Ramanan, Dollár, and C. L., Zitnick, "Microsoft coco: Common objects in context," in ECCV, 2014.
[53] D. Chen, J.-P. Mci, H. Zhang, C. Wang. Y. Feng, and C. Chen, "Knowledge distillation with the reused teacher classifier," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022, pp. 11 933-11 942.
[54] Z. Guo, H. Yan, Li, Lin, "Class attention transfer based knowledge distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 11 868-168111 11 877.
[55] Y. Jin, J. Wang, and D. Lin, "Multi-level logit distillation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023, pp. 24 276-24 285.
[56] P. Chen, S. Liu, H. Zhao, and J. Jia, "Distilling knowledge via knowledge review," in Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, 2021, pp. 5008-5017.
[57] S. Ren, K. He, R. Girshick, and Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks." in NeurlPS, 2015.
[58] Y. Tian, D. Krishnan, and P. Isola, "Contrastive representation distillation," arXiv preprint arXiv: 1910. 10699, 2019.
[59] L. Van der Maaten and G. Hinton, "Visualizing data using t-sne." JMLR, 2008
13

---

[60] Z. Liu, Y. Lin, Y. Cao. H. Hu, Y. Wei, Z. Zhang. S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," Proceedings of the IEEE/CVE international conference on computer vision, 2021, pp. 10012-10022.
[61] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly et al. "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint ar1iv:2010. 1929, 2020.
[62] Chang, Wang, H.-M. Xu, Chen, C. Yang, and F. Zhao, "Detrdistill: universal knowledge distillation framework for detr-families." in Proceedings of the IEEE/CVE international conference on computer vision, 2023, pp. 6898-6908.
[63] S. Zagoruyko and N. Komodakis, "Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer," arXiv preprint arXiv:16/2.03998 2016.
[64] Park, Kim, Lu, and Cho, "Relational knowledge distillation." Proceedings of IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3967-3976.
[65] Y. Tian, D. Krishnan, and P. Isola, "Contrastive representation distillation," " arXiv preprint arXiv: 1910. 10699, 2019.
[66] B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and Y. Choi, "A comprehensive overhaul of feature distillation," in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1921-19999911 1921-1930.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the Abstract Section and Section 1, where related material for the question be found.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the Limitation Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
14

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Refer to the qualitative analysis provided in the ablation study section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: This paper presents comprehensive experimental configurations and imple- mentation details. The open-source code and trained checkpoints will be made available to facilitate reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same
15

---

dataset, provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This paper presents comprehensive experimental configurations and imple- mentation details. The open-source code and trained checkpoints will be made available to facilitate reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guidest (oooli//guidde//dddetbbbasssssss "oodeSSumissiooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 7/nips. cc/ httcc/ubblccgguuuee gwtp://ggeeeeeeeeeeeeoeeee Sttpe//ooooSSosissiooeeaa
setpbbmmsssioooooooiy.o for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
16

---

Justification:
This paper presents comprehensive experimental configurations and imple- mentation details. The open-source code and trained checkpoints will be made available to facilitate reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please see the experimental section for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please see the experimental section for details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
17

---

Question:
Does the research conducted in the paper conform, in every respect, with the
NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have read the NeurIPS Code of Ethics and conform to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Due to space limitations, this social impact aspect is not discussed in the main paper. This paper doesn't 'tt' involve negative societal impacts, including potential malicious or unintended uses. Our proposed methods aim to achieve an efficient and effective model acceleration technique reduce computational requirements, which beneficial to social advancement.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper doesn't have any high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We conform to the CC-BY 4.0 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: This paper does not LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp:::///rrrippe //onfffrencee/22255555 LLLM for what should or should be described.
20

---

A Technical Appendices and Supplementary Material
A.1 Distillation fidelity and feature visualization.
To provide an intuitive understanding of our method's effectiveness, we visualize both the distillation fidelity and deep feature representations. Following DKD [T8], we calculate the absolute distance between correlation matrices of the teacher xResNNe33244) and student (ResNet8 x4) on CIFAR- 100. As shown Figure [5] DeepKD enables the student to produce logits more similar to the teacher compared to other methods. Additionally, our feature visualizations in Figure ddemonsttttstttttee that our pre-process enhances feature separability and discriminability across various distillation methods including KD, DKD, MLKD, and CRLD.
15-
10-
18
10-
(a) Vanilla
1
(h) DKD 18
(c) DOT
(d) KD+DeepKD (Ours)
Figure 5: Difference of student and teacher logits. DeepKD leads to significantly smaller difference (more similar prediction) than other KD methods.
Wo Ours
w/ Ours
w/O Ours
w/ Ours
w/o Ours
w/ Otrrs
DKD 18
(b) MLKD 55
(c) CRLD 13
Figure 6: The t-SNE
55 feature visualization of ResNet32 x4 and ResNet8 x4 on CIFAR-100.
A.2 Theoretical Analysis
Let us first define the key components of our knowledge distillation framework. Given teacher model and a student model S, we aim to transfer knowledge from to S while maintaining task performance. The overall loss function combines task-specific loss and knowledge distillation loss:
C = =Lclpp >D))+ + (1=o)_K((p" ,p"
(11)
where a a balancing parameter, pd and pT are the output probabilities student and teacher models respectively, and pd represents the ground truth labels. The knowledge distillation loss LKD can be elegantly decomposed into two components using KI. divergence:
LKD(DS pJ) = KL(pT | = KL (1 + + (L-DT)KL("IIP
(12)
Our DeepKD introduces dual-level decoupling strategy further decomposes the gradient of the student model's training loss into three components: task-oriented gradient (TOG), target-class gradient (TC9), and non-target-class gradient (NCg).
The probability computation:
Pi
(13)
Rk
21

---

where 2, is the logit of the i-th class, And its derivatives are
e Σ
e
e
Öp:
8
Zz
e )2
Σ c
(14)
e
Σ e
Σ
e
k
Pi
pi)
0 Σ
E
E
E
0p:
dzj
t )
e
e
(15)
E
Σ
e
PPj Vj ti
The task loss:
Ltask = CE(p', .p¹) = llg(pp))
Siip,,p"'
-
(16)
and its gradients are
TOg. = QLrak
Ozs
0Ltas
Op,
Opp
0z5
(17)
WE (1 - pt
pr. -1
@hrask d25
TOG;
OLrask
Opp
(18)
1
(-r v3))
pt
= ps. Vj #
The binary probability is constructed as
b = [pt, 1 - - T
(19)
The TCKD Loss:
KL(b [1bS) Pp. llog PT. + (1 po.) log
1 - p!.
pi.
=-p. logps (1 - po. -oog(1 p..) + p.. logpp + (1 - p.) -log(1 pt.)
=CF(b bs) H(b²)
(20)
22

---

and its derivatives are
TCg. OKT(b" (B6))
OKL(b' (
0p
dzh
(21)
(- Pf 1 p
(p) (1 p(
Pt 1 pd
-p/ (1 Pt (1 pt p.
14 pl
LKL 1
TCg,
0K L(b l16S)
Op.
Op.
0z5
-(- PK
+ 1 pt
(-p 15))
pt - pt
(pT 13)) ). (1 p.) + (1 - p.. )- (-p p55)
(22)
1 1-př pr.
pf 17 ve D
1 14
5
(p! p3) 1 pt.
p (pf - D.vi
The non-target class probability distribution is calculated as:
Pio =
[p) P2; Pt- Pt+1 PN]
(23)
and
p = Normalize(pp
= Pre
(24)
1 -
Specifically, its components are
D. Pi
1 Pt
Σ ek
Pi k
Pt
Σ
(25)
e
Vi At
Σ E
and its derivatives are
0p: 0, Vi #
(26)
Oz
0p.
P: (1 p.), #t
(27)
0z
0p.
(28)
02j
23

---

The non-target class loss:
P. log 1
KL
1100
>
N
N
(29)
p logp + Σ p.
logp! T
1=1 Eigt
CE p0) (p")
and its derivatives are:
NCg, OKL(p' T 11p²)
dz
N L
0p
Op,
Ozp
(30)
OKT( T 11p
0
Op.
NCg, TK (10²)
dzy
OKL l1BS)
Op.
025
OKL(p" [10²)
OKL(p" 7
[105)
Opp
Ops
0z5
Opp
0z5
(31)
(0) (1 130) + Σ
(-p.
13))
DJ. 609 1) +1 -1-
= p D7. #j
Suppose that the gradient g, at step is composed of signal s, and noise ny:
9: = 8, + n2c
(32)
And suppose the noise is zero-meant
E n..] 0, Vt
(33)
Estimate the signal 8, with the mean of gradient samples in short time centered at f:
8; μ
E(g,]
(34)
1
9.
2n +
f=t-n
The signal power
P Psignat ||Ra|12
(35)
24

---

The noise power is
++
Pucise
99: 2
(36)
2n
i=t-n
The signal-to-noise ratio is
P Piignal
SNR
(37)
Pnoise
A.3 Delta Parameter Stability Analysis
To address concerns about the robustness of momentum difference parameter A, conduct comprehensive ablation studies evaluating different positive values
on CIFAR-100 using ResNer32x4 (teacher) and ResNet8x4 (student) pairs. As shown Table 9 all settings with A consistently and significantly outperform the baseline with A 0. The performance remains stable and comparable across wide range of A values (0.05 0.08), confirming that method is robust and not sensitíve to the exact choice of A as long as it is positive and within a reasonable range (0, 0.1). This validates our theoretical analysis that the momentum coefficients should be positively related to their respective GSNR values.
Table 9:
Delta parameter stability analysis on CIFAR-100 using DeepKD+KD. All experiments base momentum 0.9.
Aroc
Area
ANce
Top-1 Acc (%)
Top-5 Ace (%)
0.00
0.00
0.00
74.13
92.82
0.05
0.05
0.05
76.11
94.02
0.06
0.06
0.06
76.25
94.04
0.075
0.075
0.075
76.69
94.21
0.08
0.08
0.08
76.32
94.17
A.4 Transformer Architecture Experiments
To demonstrate DeepKD's effectiveness on modern Transformer-basee architectures, we conduct additional experiments on Swin Transformer (60) and Vision Transformer (ViT) [6] distillation scenarios on ImageNet- As shown in Table TO DeepKD consistently outperforms standard KD baselines in Transformer-tor haw.foormee-t--TTaafoofoooo distillation scenarios. The method achieves notable improvements of +1.12% and +0.92% Top-1 accuracy on Swin Transformer and ViT architectures, respectively. These results demonstrate DeepKD's versatility and effectiveness across diverse modern architectures, validating our method's applicability beyond CNN-based networks.
Table 10: Results Transformer architectures for ImageNet-IK. DeepKD shows consistent improve- ments across different Transformer-to- Trans-ooroorrrrrmeer distillation scenarios.
Teacher & Student Models
Method
Top-1 Acc (%)
Top-5 Acc (%)
Teacher (Swin-L)
86.30
97.87
Student (Swin-T)
81.20
95.50
Lwin-Large Swin-Tiny
Baseline (KD)
81.59
95.96
Ours (DeepKD)
82.71
95.73
Teacher (VTT-L)
84.20
96.93
Student (VTT-B)
78.29
94.08
VIT-L/16 + ViT-B/32
Baseline (KD)
79.40
94.76
Ours (DeepKD)
80.32
95.01
A.5 DETR Object Detection Experiments
To further validate DeepKD's effectiveness on Transformer-basee architectures for object detection, we conduct experiments on the DETR (DEtection TRansformer) architecture using MS-COCO dataset.
25

---

Table 11: DETR distillation results on MS-COCO. DeepKD provides consistent improvements over
standard distillation methods on Transformer-basee object detection.
Method
AP
AP,
AP,
APt
Teacher (DETR-RI01)
43.6
25.4
46.8
60.7
Student (DETR-R50)
42.3
25.3
44.8
LD (Detrdistill [62]. ICCV 2023)
43.7
25.3
46.5
LD + Ours
44.7
25.3
46.5
FD (Detrdistill 62) ICCV 2023)
43.5
25.4
46.7
60.0
LD + FD Ours
45.3
25.8
47.0
61.0
As shown in 11 DeepKD achieves consistent improvements on DETR-based object detection, with up to +1.0 AP gain over standard distillation methods. The method demonstrates strong performance across different scales (AP., APm. APr). confirming DeepKD's effectiveness for Transformer-baser dense prediction tasks and addressing concerns about the method's applicability to stronger architectures.
A.6 Feature Distillation Experiments
To demonstrate DeepKD's versatility beyond logit-based distillation, we conduct experiments inte- grating DeepKD with feature-based distillation methods. Table 12 shows results combining DeepKD with FitNet and CRD on CIFAR-100.
Table 12: Feature distillation experiments CIFAR- 100. DeepKD provides substantial gains when combined with feature-basee methods, demonstrating its general applicability.
Method
VGGL3-VOGS WRN-4022WRNN-000 WRN-0-2-WRN-162
ReNNo56-RReNee20
HiNet
73.50
71.02
72.24
73.58
69.21
FilNet KD
75.19
72.61
72.68
74.32
70.09
DeepKD
Gain (2%)
+3.82
+1.65
+3.25
+2.97
+2.80
CRD CRD KD
75.51
73.94
74.14
75.48
71.16
75.46
74.35
64
71.63
CRD DeepKD
77.61
75.77
75,80
76.83
72.78
Gain (21%)
+2.15
+1.48
+1.42
+1.19
+1.15
The results demonstrate that DeepKD is not restricted to logit-based distillation. Our GSNR-driven optimization principles can be effectively combined with feature- methods to achieve state- of-the-art results. FitNet=DeepKD achieves gains of +3.82% on ResNet32x4 - while CRD+DeepKD provides consistent improvements across all teacher-studed pairs. This confirms DeepKD's ' general applicability and strength as a universal distillation optimizer.
A.7 Complete Results of Main Text
We provide the complete results corresponding to Table 1-3 in the main text, including all teacher- student pairs and methods evaluated. As shown in Table [3] Table 14 and Table [5] DeepKD consistently improves upon standard KD and its variants across all scenarios. The method achieves state-of-the-art performance in homogeneous distillation settings, demonstrating its effectiveness and versatility.
A.8 Algorithm (DeepKD)
In this section, we provide the pseudo code for our proposed DeepKD framework, which includes the main algorithm (Algorithm) 1 and the Dynamic Top-K Masking (DTM) strategy (Algorithm) 2
26

---

Table 13: The Top-1 Accuracy (%) different knowledge distillation methods on validation set of CIFAR-100. We evaluate homogeneous distillation scenarios where teacher and student share the same architecture but differ in model capacity. Methods are categorized by their distillation type (feature- ficature-based logit-based). Our DeepKD framework is applied to existing logit- methods with performance gains (blue and red) shown. Best results are highlighted bold.
ResNer32xx
VGG13
WRN-40-2
WRN-40-2
ResNet56
ResNet110
ResNet1
Teacher
79.42
74.64
75.61
75.61
72.34
74.31
74.31
Type
R4
VGGS
WRN 40-1
WRN-16-2
ResNet20
ResNet32
ResNet20
Student
72.50
70.36
71.98
73.26
69.06
71.14
69.06
FieNet 3D
73,50
71.02
72.24
73.58
69.21
71.06
68.99
63
73.44
71.43
72.77
74.08
72.31
RKD 64
71.90
71.48
72.22
73.35
69.61
71.82
69.25
CRD 65
75.51
73.94
74.14
75.48
71.16
73.48
71.46
Feature
OFD
66
74.95
73.95
74.33
75.24
70.98
73.23
71.29
ReviewKD
56
75.63
74.84
75.09
76.12
71.89
73.89
71.34
SimKD
78.08
74,89
74,53
75.53
71,05
71.06
CAT-KD
54
76.91
74.65
74.82
75.60
71.62
73.62
71.37
KD 1
73.33
72.98
73.54
74.92
70.66
73.08
70.67
KD+DOT
74.98
73.77
73.87
75.43
71.11
73.37
70,97
KD+LSKD :0
76.62
74.36
74.37
76.11
71.43
74.17
71.48
KD+Ours lop-k)
76.69.33.36
74.96.+1 98
74.80
76.14..1.2
71.79
+4.20
71.59
KD+Ours (w. k)
77.03
75.12
75.05
76.45
71,90,
74.35
71,82
DKD [8]
76.32
74.68
74.81
76.24
71.97
74.11
70.99
DKD&DOT 19
76.03
74, 86
74,49
75.42
71.12
73.57
71.58
DKD+LSKD 49
77.01
74.81
74.89
76.39
72.32
74.29
71.48
DKD+Ours top-k)
40.95
75.09
75.24..0:4
76.48..a
72.86+00.00
74.32+002
72.06-1.07
Logit
DKD+Ours (w. top-k)
77.54-1..91
75.19
75.42..0.0
76.72.
73.05
74.48
72.28
MLKD [5
77.08
75.18
75.35
76.63
72.19
74.11
71.89
MLKDDDDT 19
76.06
74.96
74.38
75.72
71,41
73.83
71.65
MI.KD+I SKD
78.28
75.22
75.56
76.95
72.33
74.32
72.27
ML MLKD++Ous (w/o top-k)
78.81
76.21
47.45
78. 15.15
73.75
75.88
73.03
MLKD+Ours (w. top-k)
79.15
76.45.1107
77.82./2417
78.49
74.12
76.15
73.28
CRLD 13]
77.60
75.27
75.58
76.45
72.10
74.42
72.03
CRLD-DOT 19
76.54
74.34
74.75
75.57
71.11
73.91
70.67
SKD 49
78.23
74.74
76.28
76.92
72.09
75.16
72.26
CRLD-Ours (wlo top-k)
78.90 L1.00
76.29 11.02
76.98
77.99 11.54
11.19
76.03 ;.66
73.07 1.004
CRLD-Ouss (w. top-k)
79.25
76.58..1.31
77.35
78.42....99
73.85
76.48
73.52
27

---

Table 14: The Top-1 Accuracy (%) of different knowledge distillation methods the validation of CIFAR-100. The teacher and student have distinct architectures. The KD methods are sorted by the types, i... feature-basse and logit-based. Our DeepKD framework is applied to existing logit-based methods with performance gains (blue and red) shown. Best results are highlighted in bold.
RcsNer32x4
ResNer32xx4
ResNe132x4
WRN-40-2
WRN-40-2
VGG13
ResNet50
Teacher
79.42
74.64
Type
SHN V2
WRN-16 16.2
WRN-402
ResNet8
MN V2
MN V2
MN. V2
Student
71.82
73.26
72.50
FilNet 3D
73.54
74.70
77.69
74.61
68.64
64.16
63.16
AT 63
72.73
73.91
77.43
74.11
60.78
59.40
58.58
RKD
73.21
74.86
77.82
75.26
Feature
CRD
65
75.65
75.65
78.15
73.24
70.28
69.73
69.11
ReviewKD KD
50
77.78
76.11
78. 78.96
74.34
71.28
70.37
69.89
SimKD
78.39
77.17
79.29
75.29
70.10
69.97
CAT-KD S4
78.41
76.97
78.59
75.38
70.24
69.13
71.36
KD 1
74.45
74,90
77.70
73.97
68.36
67.37
67.35
75.55
75.04
77.34
75,96
KD+LSKD 49
75.56
75,26
77.92
77.11
69.23
68.61
69.02
KD+Ours (w/o lop-k)
1..14_1..n
75.88.-0.15
20.08
76.69 4-2
69.39
69 (0.36,,1 1
69.13.,,.77
KD+Ours (w. top-k)
76.45;
76.12,
78.65 10.96
77.15
69.85 (1.49
69.92 (2.55
69.78 (2.4)
DKD [S
77.07
75.70
78.46
75.56
69.28
69.71
70.35
77.41
DKD+LSKD 49
77.37
76.19
78.95
76.75
70.01
69
70.45
DKD+Ours (w/o top-k)
27.68-0.01
76.61-0.81
29.57
76.86.-1.00
70,29
70.04.40.3
70.48
Logit
DKD+Ours top-k)
77.95
76.89..1.19
79.82
76.90
70.65
70.38
70.72..o.o7
MLKD 55]
78.44
76.52
79.26
77.33
70.78
70.57
71.04
19
75.82
79.01
MLK 49
78.76
77.53
79.66
77.68 68
71.61
70.94
71.19
MLKD+Ours (w/o top-k)
80.55
18.28
81.40.2.14
40.8
72.17.....0
12.4
73.04..2.00
MLKD+Ours (w. top-k)
80.92....18
78.65..2.15
81.78...5p
78.49
72.53
72.82
73.40
CRLD [13
78.27
76.92
80.21
77.28
70.37
70.39
71.36
CRLD-DOT
78.33
75.97
CRI.D-ISKI 49
78.61
77.37
80,58
78.03
71.52
70.48
71.43
CRID=Ouus (wlo top-k)
1..72.u1.45
78.79 1.87
81.822
4.9442.3..4
72.09 11.72
71.99 11.60
72.01 c.666
CRLD-Ous (w. top-k)
10.15
79.25
82.35
79.18
72.85..2.40
72.65
72.78..1.02
28

---

Table 15: The accuracy (%) on the ImageNet-IK IK validation set. Our DeepKD framework is applied existing logit-based methods, with performance gains (shown in blue and red). The best results emphasized in bold. N/A indicates that the data is not available.
Teacher/Student
ResNer34/ReNNe118
ResNe150MN-VI
RegNerY-J6GFPDDe 16GF//eit-Tiny
Type
Accuracy
top-1
top-5
top-5
top-1
top-5
Teacher Student
73.31
91.42
76.16
92.86
82.89
96.33
69.75
89.07
68.87
88.76
72.20
91.10
AT [63
70.69
90.01
69.56
89.33
N/A
N/A
OFD 6n
70.81
89.98
71.25
90.34
N/A
N/A
Feature
CRD 65
71.17
90.13
71.37
90.41
N/A
N/A
Review KD
50
71.61
90.51
72.56
91,00
N/A
N/A
SimKI)
71.59
90.48
72.25
90.86
N/A
N/A
CAT-KD
71.26
90.45
72.24
N/A
KD+DOT 19
71.72
90.30
73.09
91.11
73.42
92.10
KD+LSKD 40
71.42
90.29
72.18
90.80
73.27
91,95
(Wlo top-k)
+2.41
-1.00
74.32+3..2
91,94,..14
74.36 21
92.85 1.00
KD+Ours top-k)
72.85
91.35
74.65440.15
92.25
74.83
93.15441.00
DKD [8]
71.70
90.41
72.05
91.05
73.35
92.05
DKD+DOT 19
72.03
90.50
73.33
91.22
73.66
92,25
Logit
49
71.88
72.85
DKD+Ours (wle top-k)
72.78
90.96
74.4142.36
92.08
74.57
93.07 +1.07
DKD+Ours (w. top-k)
73.15
91.25..o.es
74.43
91.95
74.95
93,36,
MLKD (5S
71.90
90.55
73.01
91.42
73.54
92,25
MLKD+DOT
70.94
90.15
71.65
90.28
73.25
91.95
MLKD+LSKD 49
72.08
90.74
73.22
91.59
73.78
92.45
MLKD+Ours (w/o top-k)
73.18
91.23-1o.am
74.77
92.35
75.15
93,48
MLKD+Ours (w. top-k)
73.31:1.4
91.39 0.84
74.85 1.84
92.45 L.08
75.46 1.92
93.73 1.28
CRLD 13]
72.37
90.76
73.53
91.43
73.82
92.55
CRLD+DOT
71.76
90.00
72.38
90.37
73.37
92.05
CRLD 49
72.39
90.87
73,74
91.61
73.95
92.65
CRI.D+Ours (W/ top
73.18
91.23-0.47
74.10..0.7
91.49
75.35
93.35+00.00
CRLD+Ours top-k)
73.34 00.07
91.38 .0.02
74.85 11.12
92.45 L.02
75.75 1.59
93.85 1..40
29

---

Algorithm Pseudo code of DeepKD Gradient Decoupling PyTorch-like style.
1_stu: student logita, t_tea: teacher logits
# т: temperature, t: target class index # a, 81, $2: loss weights
ju: base momentum, A: momentum difference
v_tog, v_teg, V_neg: momentum buffers
Calculate probability
p_stu_task softmax(1_stu)
P_tea softmax(1_tea//e
P_stu
b_tea [p_tea[t], 1 [_]e
b_stu [p_stu[t], p_stu[t]]
p_hat_tea p_tea.clone( clone()
p_hat_ste clome)
del p_hat_tea[t]
del p_hat_stu[t]
topk ree__dyaaicc.KKeureen.eee
-pocas) See Algorithm
topk_indices argsort (p_hatttea_ [-topk:]
p_hat_tea- p_hat_tea _e.ba__teatook______ (ndpc_]niices]
p_hat_stu p_ __ [topk_indices;
P. peeat_tea /= cumpphat__ted
p_hat_stu /= sum(p_hat_stu)
# Calculate gradients
tog (CE st_ppp_suuu____
stu_tasstk task, y))
tcg B1 (ra b_stu)) T
nog B2 (KL (p__ht___te,,, p_hat_stu)) T-2
# Momentum updates v_tog tog (μ v_tog tcg (μ
v_tog Higher momentum
v_teg Lower momentum
v_ncg = nog (μ A) * v_ncg Higher momentum
Parameter update
params ir (v_tog v_teg v_neg)
Algorithm Pseudo code of Dynamic Top-K Masking (DTM) a PyTorch-like style.
k_init: initial value (5% classes)
# k_max: max & value (100% classes)
# k_opt: optimal k value phase: easy/ttramsition/harr learning phase
def get__dnamic__(eee (epoch, total_epochs):
epoch 0.3 total_epochs: # Easy phase return linear_ Sinear_interp(k_int k_opt, epoch)
elif epoch 0.7 total_epochs: # Transition return k_opt
else:
# Hard phase return Linear_interp(k.orr k_max, epoch)
30


Reviews:
[{"review_id": "XL9Q31uIhR", "reviewer": "Reviewer_ryqe", "summary": "This work focuses on improving logit-based knowledge distillation. The authors firstly observe two critical issues: the conflicts between target-class and non-target-class knowledge flow, and the noise introduced by low-confidence \"dark knowledge\" from non-target classes. To mitigate these challenges, the paper introduces DeepKD, which integrates a dual-level decoupling mechanism with an adaptive denoising strategy. In particular, the proposed DeepKD assigns three independent momentum updaters for different gradient components (task-oriented, target-class, and non-target-class) according to the theoretical analysis of Gradient Signal-to-Noise Ratio (GSNR). Moreover, in order to effectively reduce the noise of non-target classes, a Dynamic Top-K Mask (DTM) mechanism is introduced to progressively filter low-confidence logits from both the teacher and student models during training. Extensive experiments are conducted on CIFAR-100 and ImageNet-1K on image classification task, and MS-COCO on object detection task.", "questions": "Please see the weaknesses above.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-motivated. The proposed DeepKD is pioneering in establishing a direct connection between Gradient Signal-to-Noise Ratio (GSNR) and momentum allocation in the context of logit-based knowledge distillation. \n\n2. Independent momentum updaters derived from the GSNR analysis is proposed to prevent mutual interference among the different knowledge flows during training process.\n\n3. The Dynamic Top-K Mask (DTM) mechanism is introduced to adaptively filter low-confidence logits from both the teacher and student models by gradually increasing the value of 'K'.\n\n**Weaknesses**\n\n1. The computational complexity is increased. The proposed DeepKD framework involves three distinct momentum updaters and a dynamic top-K mask mechanism with three predefined learning phases (Easy, Transition, Hard), inherently adds significant complexity to the training pipeline. It may lead to longer training time and potentially higher memory consumption compared to other KD methods.\n\n2. The experimental settings are limited to CNN-based networks. The paper does not extensively demonstrate DeepKD's effectiveness or performance characteristics when applied to other dominant architectures, such Transformer-based networks. \n\n3. The independent effectiveness assessment of proposed DeepKD is missing. DeepKD is frequently applied in conjunction with other logit-based knowledge distillation methods (e.g., KD, DKD) in the reported tables. This makes it challenging to precisely isolate and quantify DeepKD independent performance. Further empirical evidence is needed to strengthen its contributions, specifically by demonstrating its standalone effectiveness and the performance gains achieved when incorporating feature distillation.", "was_revised": true, "final_justification": "Given that my main concerns resolved, I’ve adjusted my score to 4.", "rebuttal": "**W1: DeepKD introduces additional computational complexity, which may increase training time and memory usage compared to other KD methods.**\n\nThank you for raising this important and practical concern regarding computational complexity. We analyze DeepKD's overhead on CIFAR-100 and ImageNet-1K. Results are shown in the tables below.\n\n_Table 1: Overhead analysis. For CIFAR-100, we use ResNet32×4 as the teacher and ResNet8×4 as the student, training on a single 2080Ti GPU for 240 epochs (480 for MLKD). For ImageNet-1K, we use ResNet34 as the teacher and ResNet-18 as the student, training on two RTX4090 GPUs for 100 epochs._\n Method | CIFAR-100 Memory (M) | CIFAR-100 Time (Hours) | ImageNet-1K Memory (M) | ImageNet-1K Time (Hours)\n---|---|---|---|---\n KD | 799 | 1.6 | 21344 | 20.0\n KD + Ours | 805 (+0.75%) | 2.6 (+62%) | 21370 (+0.12%) | 29.4 (+47%)\n DKD | 799 | 1.7 | 21350 | 20.1\n DKD + Ours | 805 (+0.75%) | 2.7 (+60%) | 21370 (+0.12%) | 29.8 (+48%)\n MLKD | 983 | 9.2 | 34910 | 34.2\n MLKD + Ours | 987 (+0.41%) | 12.3 (+33%) | 34960 (+0.14%) | 53.1 (+55%)\n CRLD | 981 | 2.9 | 34862 | 32.2\n CRLD + Ours | 985 (+0.41%) | 4.5 (+52%) | 34909 (+0.13%) | 51.2 (+59%)\n\nAs shown, the practical overhead introduced by DeepKD is well-controlled and represents a favorable trade-off for the performance improvements achieved.\n\n**Memory Consumption is Minimal:**  \nTable 1 demonstrates that DeepKD incurs only a negligible increase in memory usage—less than 1% on CIFAR-100 and less than 0.2% (about 10–25MB per GPU) on ImageNet-1K. This minor increase is primarily due to the storage of student gradients during distillation, as the teacher model remains frozen. The efficient CUDA memory allocator further mitigates any additional memory footprint, confirming that DeepKD is suitable for large-scale training scenarios.\n\n**Training Time and Efficiency:**  \nAlthough DeepKD introduces a moderate increase in per-epoch training time, Table 2 demonstrates that it achieves superior accuracy with substantially fewer epochs than baseline methods. This results in a more favorable overall time-accuracy trade-off.  All DeepKD results are from **independent full training runs** at the specified epochs for fair comparison. \n\n_Table 2: DeepKD achieves higher accuracy with fewer training epochs. For fair comparison, all baseline methods are trained for 240 epochs on CIFAR-100 (480 for MLKD) and 100 epochs on ImageNet-1K, using ResNet32×4/ResNet8×4 and ResNet34/ResNet-18 as teacher/student pairs, respectively._\n Method | CIFAR-100 Epoch | CIFAR-100 Top1 (%) | ImageNet-1K Epoch | ImageNet-1K Top1 (%)\n---|---|---|---|---\n KD | 240 | 73.33 | 100 | 71.03\n KD+DOT | 240 | 74.98 | 100 | 71.72\n KD+LSKD | 240 | 76.62 | 100 | 71.42\n **KD+Ours** | **160** | **76.83** | **65** | **71.76**\n DKD | 240 | 76.32 | 100 | 71.70\n DKD+DOT | 240 | 76.03 | 100 | 72.03\n DKD+LSKD | 240 | 77.01 | 100 | 71.88\n **DKD+Ours** | **160** | **77.31** | **65** | **72.16**\n MLKD | 480 | 77.08 | 100 | 71.90\n MLKD+DOT | 480 | 76.06 | 100 | 70.94\n MLKD+LSKD | 480 | 78.28 | 100 | 72.08\n **MLKD+Ours** | **320** | **79.04** | **65** | **72.52**\n CRLD | 240 | 77.6 | 100 | 72.37\n CRLD+DOT | 240 | 76.54 | 100 | 71.76\n CRLD+LSKD | 240 | 78.23 | 100 | 72.39\n **CRLD+Ours** | **160** | **78.87** | **65** | **72.98**\n\nOur framework enables faster convergence and higher final accuracy than baseline methods, even with fewer training epochs. For example:\n- On CIFAR-100, **KD + Ours** achieves 76.83% accuracy in just 160 epochs, outperforming the 240-epoch KD baseline.\n- On ImageNet-1K, **CRLD + Ours** reaches 72.98% accuracy in only 65 epochs, surpassing the 100-epoch CRLD baseline.\n\n**Baselines Cannot Match DeepKD Even with Extended Training:**  \nTable 3 further demonstrates that simply increasing the training duration for baseline methods does not close the performance gap with DeepKD.\n\n_Table 3: Baseline performance after extended training. For fair comparison, all methods are trained for 480 epochs on CIFAR-100 (960 epochs for MLKD) and 200 epochs on ImageNet-1K, as reported in the main paper._\n Method | CIFAR-100 Epoch | CIFAR-100 Top1 (%) | ImageNet-1K Epoch | ImageNet-1K Top1 (%)\n---|---|---|---|---\n KD | 480 | 73.00 | 200 | 72.29\n KD+DOT | 480 | 74.72 | 200 | 72.25\n KD+LSKD | 480 | 76.51 | 200 | 71.99\n **KD+Ours** | **240** | **77.03** | **100** | **72.85**\n DKD | 480 | 75.89 | 200 | 72.10\n DKD+DOT | 480 | 76.28 | 200 | 72.41\n DKD+LSKD | 480 | 76.63 | 200 | 72.36\n **DKD+Ours** | **240** | **77.54** | **100** | **73.15**\n MLKD | 960 | 77.64 | 200 | 72.76\n MLKD+DOT | 960 | 76.21 | 200 | 71.47\n MLKD+LSKD | 960 | 78.87 | 200 | 72.96\n **MLKD+Ours** | **240** | **79.15** | **100** | **73.31**\n CRLD | 480 | 77.99 | 200 | 72.98\n CRLD+DOT | 480 | 76.34 | 200 | 71.83\n CRLD+LSKD | 480 | 78.61 | 200 | 73.13\n **CRLD+Ours** | **240** | **79.25** | **100** | **74.23**\n\nFor instance, even after 480 epochs, the CRLD baseline on CIFAR-100 only reaches 77.99%, which is still below our CRLD+Ours result of 79.25% achieved in half the epochs. The same trend is observed on ImageNet-1K.\n\n**The Training Cost is Justified by Substantial Gains:**  \nThe primary goal of knowledge distillation is to maximize the student model’s inference performance. In practice, it is standard and reasonable to invest more resources in a one-time training process if it results in a superior model, as inference will be performed many times over the model’s lifecycle. Importantly, this additional training cost does not affect the final inference speed of the student model. In summary, DeepKD provides a highly efficient route to a better-performing model.\n\n---\n\n**W2: The experiments mainly focus on CNNs. More evidence is needed to show DeepKD's effectiveness on Transformer-based architectures.**\n\nThank you for highlighting the importance of evaluating DeepKD on diverse architectures. We appreciate this point and have taken steps to provide more comprehensive evidence of DeepKD’s effectiveness on modern Transformer-based networks.\n\nTo clarify, our original submission already included experiments on Vision Transformers, where DeepKD improved the Top-1 accuracy of a DeiT-Tiny student by up to +1.90% (see Table 3 in the main paper).\n\nTo further address your concern, we conducted two additional sets of experiments, focusing on pure-Transformer architectures—Swin Transformer (ICCV 2021) and Vision Transformer (ViT, ICLR 2021)—on the ImageNet-1K dataset.\n\n_Table 4: Transformer Experiments on ImageNet-1K_\n Teacher & Student Models | Method | Top-1 Acc (%) | Top-5 Acc (%)\n---|---|---|---\n| Swin-Large → Swin-Tiny | Teacher (Swin-L) | 86.30 | 97.87\n|  | Student (Swin-T) | 81.20 | 95.50\n|  | Baseline (KD) | 81.59 | 95.96\n|  | **Ours (DeepKD)** | **82.71** | **96.23**\n| ViT-L/16 → ViT-B/32 | Teacher (ViT-L) | 84.20 | 96.93\n|  | Student (ViT-B) | 78.29 | 94.08\n|  | Baseline (KD) | 79.40 | 94.76\n|  | **Ours (DeepKD)** | **80.32** | **95.01**\n\nAs shown, DeepKD consistently and significantly outperforms the standard KD baseline in these challenging Transformer-to-Transformer distillation scenarios.\n \n---\n\n**W3: DeepKD is mainly evaluated together with other logit-based distillation methods, making it hard to isolate its individual effect. More evidence is needed to show its independent contribution and its impact when combined with feature distillation.**\n\nThank you for this insightful question regarding the independent assessment of DeepKD and its applicability beyond logit-based methods. We would like to clarify the nature of our framework and provide new experimental results to directly address your concerns.\n\n**First**, DeepKD is a training framework or optimization strategy, not a standalone loss function. This is a common characteristic among many state-of-the-art knowledge distillation methods, such as DKD (CVPR 2022), DOT (ICCV 2023), and LSKD (CVPR 2024). These methods are designed to enhance a primary distillation signal, either by reformulating the loss, modifying the optimization process, or pre-processing the logits. As such, none of these methods—including DeepKD—can be meaningfully evaluated in complete isolation from a base teacher-student setup.\n\n**Second**, the independent effectiveness of our two main contributions—Decoupled Gradients and Dynamic Top-k Masking (DTM)—is already quantified in Table 6 of our main paper, which reports the individual performance gains from each component across various logit-based baselines.\n\n**Finally**, to further demonstrate the versatility of DeepKD, we conducted new experiments by integrating DeepKD with two widely used feature-based distillation methods: FitNet (ICLR 2015) and CRD (ICLR 2020). The results, summarized in Table 5, show that DeepKD provides substantial additional gains over standard feature distillation and logit-based KD combinations.\n\n_Table 5: Feature Distillation Experiments on the CIFAR-100 validation._\n Method | ResNet32×4→ResNet8×4 | VGG13→VGG8 | WRN-40-2→WRN-40-1 | WRN-40-2→WRN-16-2 | ResNet56→ResNet20\n---|---|---|---|---|---\n FitNet | 73.50 | 71.02 | 72.24 | 73.58 | 69.21\n FitNet + KD | 75.19 | 72.61 | 72.68 | 74.32 | 70.09\n FitNet + DeepKD | 77.32 (+3.82%) | 75.67 (+4.65%) | 75.49 (+3.25%) | 76.55 (+2.97%) | 72.01 (+2.80%)\n CRD | 75.51 | 73.94 | 74.14 | 75.48 | 71.16\n CRD + KD | 75.46 | 74.29 | 74.38 | 75.64 | 71.63\n CRD + DeepKD | 77.61 (+2.10%) | 75.77 (+1.83%) | 75.80 (+1.66%) | 76.83 (+1.35%) | 72.78 (+1.62%)\n\nThese results provide strong evidence that DeepKD is not limited to logit-based distillation. Its GSNR-driven optimization principles can be effectively combined with feature-based methods, demonstrating the generality and strength of our framework. We will include these findings in the appendix of our revised manuscript.\n\nWe hope these clarifications and new results comprehensively address your concerns. We will revise our manuscript to make these points more explicit and appreciate your thoughtful feedback."}, {"review_id": "jKnQ4hUmch", "reviewer": "Reviewer_qHHw", "summary": "The paper proposes DeepKD, a novel knowledge distillation framework that integrates dual-level decoupling with adaptive denoising. The key contributions are: 1. Dual-Level Decoupling**: Decomposes gradients into task-oriented, target-class, and non-target-class components. Momentum coefficients are adaptively allocated based on gradient signal-to-noise ratio (GSNR), enabling independent optimization of different knowledge types. 2. Dynamic Top-k Masking: Gradually incorporates more non-target classes during training, filtering low-confidence logits to purify dark knowledge transfer. 3. Empirical Validation: Extensive experiments on CIFAR-100, ImageNet, and MS-COCO demonstrate state-of-the-art performance, showing DeepKD's effectiveness across diverse models and tasks.\nDeepKD enhances knowledge transfer efficiency and model generalization through its theoretically grounded approach.", "questions": "1. How does the dynamic top-k masking mechanism specifically determine the scheduling of the k value during different training phases, and what is the impact of this scheduling on the overall training efficiency and model performance?\n2. In terms of technical details, how does the dual-level decoupling strategy dynamically adjust the momentum coefficients for different knowledge components during the training process to ensure effective knowledge transfer and model optimization?", "limitations": "Yes, the paper does discuss its limitations. It notes that the current work focuses on logit-based distillation, but the SNR-driven momentum decoupling mechanism could be extended to feature distillation scenarios. Additionally, the dynamic top-k masking strategy shows promise for improving distillation performance and warrants further exploration for different model architectures and datasets.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths:\n1. theoretical contributions: The paper proposes a novel dual-level decoupling strategy based on gradient signal-to-noise ratio (GSNR) and a dynamic top-k masking mechanism, both of which are theoretically well-founded and address limitations in existing knowledge distillation methods.\n2. empirical validation: Extensive experiments on multiple benchmarks (CIFAR-100, ImageNet, MS-COCO) demonstrate the effectiveness of DeepKD. The method shows consistent improvements across various model architectures and scenarios, including image classification and object detection.\n3. innovation: the combination of GSNR-driven momentum allocation and dynamic top-k masking is original and provides a new perspective on optimizing knowledge distillation processes.\n4. versatility: deepKD can be seamlessly integrated with existing logit-based distillation approaches and consistently achieves state-of-the-art performance.\n\nWeaknesses:\n1. limited to Logit-Based Distillation: The current framework focuses on logit-based distillation. While the authors mention potential extensions to feature-based scenarios, the paper does not provide concrete results or methods for these cases.\n2. complexity: the introduction of multiple components (TOG, TCG, NCG) and the dynamic top-k masking mechanism may increase the complexity of the implementation and require careful tuning of hyperparameters.\n3. scalability concerns: Although the method shows strong performance on the datasets tested, its scalability to even larger models or more diverse tasks remains to be fully explored and validated.\n4.potential for overfitting: The detailed decoupling and masking strategies might lead to overfitting if not properly regularized, though the paper does not extensively discuss this aspect.", "was_revised": true, "final_justification": "Considering the effectiveness and novelty of the method proposed in this article, combined with the opinions of other reviewers, I give the final boarderline acceptance rating.", "rebuttal": "**W1: The framework mainly discusses logit-based distillation, and lacks concrete results or methods for feature-based scenarios.**\n\nThank you for highlighting the scope of our framework. To directly address this concern and further demonstrate the flexibility of DeepKD, we have conducted additional experiments on feature-based distillation.\n\nSpecifically, we integrated DeepKD with two widely adopted feature-based methods, FitNet (ICLR 2015) and CRD (ICLR 2020), as shown in Table 1. Our objective was to evaluate whether DeepKD’s GSNR-driven optimization could enhance a combined feature-and-logit distillation pipeline, beyond what is achievable with standard KD.\n\n_Table 1: Feature Distillation Experiments  on the CIFAR-100 validation_\n Method | ResNet32×4→ResNet8×4 | VGG13→VGG8 | WRN-40-2→WRN-40-1 | WRN-40-2→WRN-16-2 | ResNet56→ResNet20 \n---|---|---|---|---|---\n FitNet | 73.50 | 71.02 | 72.24 | 73.58 | 69.21 \n FitNet + KD | 75.19 | 72.61 | 72.68 | 74.32 | 70.09 \n FitNet + DeepKD | 77.32 (+3.82%) | 75.67 (+4.65%) | 75.49 (+3.25%) | 76.55 (+2.97%) | 72.01 (+2.80%) \n CRD | 75.51 | 73.94 | 74.14 | 75.48 | 71.16 \n CRD + KD | 75.46 | 74.29 | 74.38 | 75.64 | 71.63 \n CRD + DeepKD | 77.61 (+2.10%) | 75.77 (+1.83%) | 75.80 (+1.66%) | 76.83 (+1.35%) | 72.78 (+1.62%) \n\nThese results provide clear evidence that DeepKD is **not restricted to logit-based distillation**. Our GSNR-based optimization can be effectively combined with feature-based approaches, consistently yielding state-of-the-art results. This demonstrates the broad applicability and strength of DeepKD as a general-purpose distillation optimizer.\n\n---\n\n**W2: The framework introduces several components and a dynamic masking mechanism, which may raise concerns about implementation complexity and hyperparameter tuning.**\n\nThank you for raising concerns about implementation complexity and hyperparameter tuning. We address these aspects separately below:\n\n1. **Loss Weights (`β₁`, `β₂`):**  \n   These coefficients are **not tuned** in our framework. We follow the standard settings from DKD (CVPR 2022) to ensure fair and consistent comparisons, using their fixed values for TCKD and NCKD loss components.\n\n2. **Momentum Difference (`Δ`):**  \n   This parameter is **theoretically motivated and robust**. Our GSNR analysis justifies simply setting `Δ > 0`. As shown in our ablation (Table 5, Left), the final performance is not sensitive to the exact value of `Δ`—any positive value following the GSNR principle consistently outperforms the baseline (`Δ=0`).\n\n3. **DTM Schedule (`k-value`, `Phase-1`, `Phase-2`):**  \n   While the DTM schedule introduces new hyperparameters, our experiments on large-scale datasets (e.g., ImageNet-1K) show that the schedule is highly robust and practical. The DTM consistently improves performance across all tested teacher-student pairs, confirming its stability and utility.\n\nIn summary, DeepKD’s key hyperparameters are either fixed by standard practice, guided by strong theoretical principles, or shown to be robust in large-scale settings. This ensures the framework remains practical and easy to adopt.\n\n---\n\n**W3: Concerns about scalability and potential overfitting: The scalability of the method to larger models and more diverse tasks, as well as the risk of overfitting due to decoupling and masking strategies, require further validation.**\n\nWe appreciate your comments on scalability. To provide a comprehensive validation, we have conducted new experiments covering dataset scale, model scale, task diversity, and different distillation paradigms.\n\nBeyond classification and detection on **ImageNet-1K** and **MS-COCO** (see Tables 3 and 4 in our main paper), we further evaluated DeepKD on the challenging object detection task using the Transformer-based **DETR** architecture (see Table 2 below).\n\n_Table 2: DETR Distillation Results on MS-COCO_\n Method | AP | APs | APm | APl \n---|---|---|---|---\n Teacher (DETR-R101) | 43.6 | 25.4 | 46.8 | 60.7 \n Student (DETR-R50) | 42.3 | 25.3 | 44.8 | 58.2 \n LD (Detrdistill, ICCV 2023) | 43.7 | 25.3 | 46.5 | 60.7 \n **LD + Ours** | **44.7** | **25.3** | **46.5** | **60.7** \n FD (Detrdistill, ICCV 2023) | 43.5 | 25.4 | 46.7 | 60.0 \n **FD + Ours** | **45.3** | **25.8** | **47.0** | **61.0** \n\nWe also evaluated DeepKD with strong teachers (e.g., **RegNetY-16GF**) and a diverse set of modern Transformer architectures. For example, DeepKD achieves a +1.90% gain when combined with MLKD on DeiT. Further, our new experiments on Swin Transformer (ICCV 2021) and ViT-to-ViT (ICLR 2021) distillation show notable Top-1 accuracy improvements of **+1.12%** and **+0.92%**, respectively (see Table 3).\n\n_Table 3: Larger Transformer Experiments on ImageNet-1K_\n Teacher & Student Models | Method | Top1 Acc (%) | Top5 Acc (%) \n---|---|---|---\n| Swin-Large → Swin-Tiny | Teacher (Swin-L) | 86.30 | 97.87 \n|  | Student (Swin-T) | 81.20 | 95.50 \n|  | Baseline (KD) | 81.59 | 95.96 \n|  | **Ours (DeepKD)** | **82.71** | **96.23** \n| ViT-L/16 → ViT-B/32 | Teacher (ViT-L) | 84.20 | 96.93 \n|  | Student (ViT-B) | 78.29 | 94.08 \n|  | Baseline (KD) | 79.40 | 94.76 \n|  | **Ours (DeepKD)** | **80.32** | **95.01** \n\n\nAdditionally, we validated DeepKD on semantic segmentation, comparing with KA (Tong He, CVPR 2019), as shown below:\n\n_Table 4: Pascal VOC 2012 Semantic Segmentation (val set, trained on trainaug)_\n Method | mIOU (%) \n---|---\n Teacher: ResNet-50 | 76.21 \n Student: MobileNetV2 | 70.57 \n Student+KD | 71.32 \n Student+FitNet | 71.30 \n Student+KA | 72.50 \n Student+Ours | **73.41** \n\nIn summary, our extensive experiments—spanning large datasets, diverse tasks, strong teachers, a variety of modern Transformer architectures (DeiT, Swin, ViT, DETR), and multiple distillation paradigms—provide robust evidence of DeepKD’s scalability and generality.\n\n---\n\n**W4: Overfitting risk: Could the decoupling and masking strategies cause overfitting if not well regularized? The paper does not discuss this in detail.**\n\nThank you for raising the important issue of overfitting. We believe that the core mechanisms of DeepKD—GSNR-driven decoupling and Dynamic Top-k Masking (DTM)—are inherently regularizing and actually help mitigate overfitting.\n\n1. **GSNR-Driven Decoupling as Regularization:**  \n   Our GSNR-based momentum allocation stabilizes training by assigning higher momentum to more reliable gradient signals (TOG and NCG). This leads to convergence at **flatter minima** in the loss landscape, as shown in our loss landscape analysis (Figure 1(b)). Flatter minima are well-known to correlate with better generalization and reduced overfitting (Hao Li, NIPS 2018).\n\n2. **DTM as Curriculum-Based Regularization:**  \n   DTM filters out low-confidence, potentially noisy non-target logits, especially early in training. This prevents the student from overfitting to unreliable aspects of the teacher’s dark knowledge. By gradually expanding the knowledge set, DTM provides a curriculum learning effect, further enhancing generalization.\n\nTogether, these mechanisms ensure that the student learns robust, generalizable features, actively guarding against overfitting.\n\n---\n\n**Q1: How is the k value scheduled in DTM, and how does this affect training efficiency and model performance?**\n\nThank you for your question regarding the scheduling of the k value in DTM.\n\n**1. DTM Scheduling Strategy:**  \nOur DTM schedule is inspired by **curriculum learning** and is implemented in three phases using a linear growth heuristic:\n\n- **Phase 1 (Easy Learning):** `K` increases linearly from a small value (e.g., 5% of total classes) to an optimal static value (`k-value`). This allows the student to focus on the most confident, semantically relevant non-target classes at the start.\n- **Phase 2 (Transition):** `K` is held constant at `k-value`, giving the model time to consolidate knowledge from this core set.\n- **Phase 3 (Hard Learning):** `K` increases linearly from `k-value` to the full class count, exposing the student to the complete distribution of dark knowledge for final refinement.\n\nEmpirically, we find it effective to end Phase 1 at ~25% of total epochs (e.g., epoch 60/240) and Phase 2 at ~70-75% (e.g., epoch 170-180/240).\n\n**2. Impact of DTM Scheduling:**  \n- **Training Efficiency:** The scheduling logic is a simple epoch-based `if/else` and adds negligible computational overhead.\n- **Model Performance:** The curriculum-based schedule consistently improves performance, as shown in all main paper tables. For example, in Table 1, DTM improves `DKD+Ours` from 77.25% to **77.54%**. The approach prevents the student from being overwhelmed by noisy logits early on, leading to more stable and effective learning.\n\n---\n\n**Q2: How does the dual-level decoupling strategy dynamically set the momentum coefficients for different knowledge components during training?**\n\nThank you for this technical question. To clarify, our dual-level decoupling strategy does **not** dynamically adjust momentum coefficients during training. Instead, our GSNR analysis provides a **one-time, principled basis** for a fixed momentum allocation.\n\n1. **Theoretical Basis:**  \n   As shown in Figure 2(a) of the main paper, throughout training, the gradients for TOG and NCG consistently have higher GSNR than TCG. This indicates that TOG and NCG provide more reliable learning signals.\n\n2. **Practical Mechanism:**  \n   Based on this observation, we assign a **fixed higher momentum (`μ + Δ`)** to TOG and NCG, and a **fixed lower momentum (`μ - Δ`)** to TCG, as described in Equation 7. The value of `Δ` is constant and, as shown in our ablation, the method is robust to its precise value.\n\nThis design achieves the benefits of GSNR-informed optimization without the complexity of dynamic adjustment, making DeepKD both effective and practical.\n\nWe thank the reviewer again for their thoughtful feedback. We will revise the manuscript to clarify these points and are confident that these explanations address the concerns raised."}, {"review_id": "OCjKiqG1ca", "reviewer": "Reviewer_jZPz", "summary": "This paper introduces DeepKD, a novel knowledge distillation method that combines principles from DOT and DKD. The core contribution is twofold. \n1. It builds upon DOT's use of separate momentum hyperparameters for the cross-entropy and KL-divergence losses by introducing a principled approach to determine the momentum. This innovation removes the need to tune one of DOT's key hyperparameters. It extends this to the decoupled loss of DKD (DKD decouples of target and non-target knowledge).\n2. The paper proposes Dynamic Top-K Masking (DTM), a technique designed to filter out noise from the probability distributions of semantically unrelated classes by masking out the low probability classes dynamically, in a scheduled manner.", "questions": "Question:\n- What are the phase-1 and phase-2 values for ImageNet?\n\n\nMinor Comment:\n- It would be good to include some of the newer works like [A-E] in the appendix for a more comprehensive comparison. [A] also has a nice list of prior works to contextualize improvement over time.\n\n[A] Yulei Niu, Long Chen, Chang Zhou, and Hanwang Zhang. \"Respecting transfer gap in knowledge distillation.\" Advances in Neural Information Processing Systems, 2022.\n\n[B] Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu, “Knowledge diffusion for distillation,” Advances in Neural Information Processing Systems, 2023.\n\n[C] Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang, “Curriculum temperature for knowledge distillation,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2023, vol. 37, pp. 1504–1512.\n\n[D] Yashas Malur Saidutta, Rakshith Sharma Srinivasa, Jaejin Cho, Ching-Hua Lee, Chouchang Yang, Yilin Shen, and Hongxia Jin, \"CIFD: Controlled Information Flow to Enhance Knowledge Distillation.\" Advances in Neural Information Processing Systems, 2024\n\n[E] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu, “DIST+: Knowledge Distillation From a Stronger Adaptive Teacher,” IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 47, NO. 7, July 2025", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper explores a novel technique of dynamic top-K masking (DTM) \n2. It explores a novel combination of DOT and DKD and proposes an automatic momentum hyper-param selection using GSNR.\n3. The paper shows compelling empirical results, promising performance gains across a variety of student-teacher architectures and on several benchmark datasets, including CIFAR-100, ImageNet, and MS-COCO.\n\n**Weaknesses**\n1. A significant concern is the number of new hyperparameters with respect to KD. In addition to the $\\Delta$ parameter for momentum, the method requires tuning $\\Beta_1$ and $\\Beta_2$ values (for TCKD and DTM, respectively) and a two-stage schedule (Phase-1 and Phase-2) for DTM.\n2. The sensitivity of these DTM scheduling parameters is concerning. The ablation studies show very high performance swings. For instance, with k=55 and Phase-2=170, a minor change in the Phase-1 end from 60 to 40 results in a performance drop of over 7% (to 69.98%). This is significantly worse than the 72.50% accuracy achieved by a standard student model trained without any knowledge distillation. This raises questions about the method's stability and practicality.", "was_revised": true, "final_justification": "Update after author discussion. My main concern was the instability of some of the results which the authors addressed by providing more experiments and rectifying the typo of results in the original submission.", "rebuttal": "**W1: The main concern is the introduction of new hyperparameters, including `Δ`, `β₁`, `β₂`, and the DTM schedule (Phase-1 and Phase-2).**\n\nThank you for raising this important point regarding the hyperparameters introduced by our method. We address each of the parameters as follows:\n\n1. **Loss Weights (`β₁`, `β₂`):**  \n   These are **not tuned hyperparameters** in our framework. To ensure fair and consistent comparisons, we directly adopt the fixed coefficients for TCKD and NCKD as proposed in the original DKD (CVPR 2022) method, following established best practices.\n\n2. **Momentum Difference (`Δ`):**  \n   This parameter is **principled and robust**. Our GSNR analysis provides strong theoretical motivation to simply set `Δ > 0`. As shown in our ablation study (**Table 5, Left**), the final performance is not sensitive to the exact value of `Δ` as long as it is positive; all settings with `Δ > 0` significantly and consistently outperform the baseline with `Δ=0`.\n\n3. **DTM Schedule (`k-value`, `Phase-1`, `Phase-2`):**  \n   We acknowledge that the DTM schedule introduces new hyperparameters, which is expected for a new mechanism. However, our empirical results show that the DTM schedule, implemented as a three-phase curriculum with a heuristic linear growth strategy inspired by **curriculum learning**, is highly robust and practical. This is validated by extensive experiments on the large-scale ImageNet-1K dataset, where the DTM component consistently improves performance across all tested teacher-student pairs, confirming both its stability and practical value.\n\nIn summary, the key hyperparameters in DeepKD are either fixed by standard practice, guided by strong theoretical principles and shown to be robust, or empirically robust on large-scale tasks. This ensures that our framework remains practical and easy to apply.\n\n---\n\n**W2: The ablation studies suggest DTM schedule parameters may cause large performance fluctuations. For example, changing Phase-1 from 60 to 40 (with k=55, Phase-2=170) leads to a >7% drop, even below the student-only baseline. This raises concerns about the method's stability and practicality.**\n\nWe sincerely appreciate your careful and rigorous review. Your observation prompted us to conduct a thorough audit of our experiment logs, during which we discovered a critical data compilation error in Table 5 of our manuscript.\n\nWe apologize for this oversight. The error was due to a manual mistake in transcribing results for two experimental settings. To ensure transparency and reproducibility, we will release our complete code and final model checkpoints soon.\n\nThe two erroneous data points that led to concerns about model stability have now been corrected (see Table 1). \n\n_Table 1: Correction of erroneous data._\n k-value | Phase1 | Phase2 | Top-1 | Top-5 \n---|---|---|---|---\n 40 | 55 | 170 | ~~69.98%~~ | 91.62% \n 40 | 55 | 170 | **76.98%** | 91.62% \n 60 | 60 | 170 | ~~70.19%~~ | 92.36% \n 60 | 60 | 170 | **77.19%** | 92.36% \n\nWith these corrections, all tested configurations in our ablation study now significantly outperform the student-only baseline. The results are now logically consistent and demonstrate that our DTM framework is highly robust across a wide range of parameterizations. The previously reported \"high performance swings\" and \"instability\" were artifacts of the data error and are now resolved. The corrected results provide clear and consistent support for our claims.\n\nThank you again for your diligence, which has been instrumental in identifying and correcting this issue, thereby strengthening the accuracy and reliability of our work.\n\n---\n\n**Q1: What are the phase-1 and phase-2 values for ImageNet?**\n\nThank you for your question regarding the DTM schedule parameters for our ImageNet experiments. Due to the high computational cost of exhaustive hyperparameter search on ImageNet, we adopted a principled two-step approach:\n\n1. **Principled Schedule Transfer from CIFAR-100:**  \n   We first transferred insights from our extensive CIFAR-100 ablations, which indicated that a well-proportioned three-phase curriculum is effective. Accordingly, for the 100-epoch ImageNet training, we set the phase transitions at the **25% mark (epoch 25)** and the **75% mark (epoch 75)**.\n\n2. **Experimental Verification on ImageNet:**  \n   We then conducted a targeted ablation study on ImageNet, exploring variations around these initial values. The results are summarized below:\n\n_Table 2: ImageNet DTM Schedule Verification (ResNet34 teacher, ResNet-18 student; static top-k: k=100 yields best performance)._\n k-value | Phase1 (epochs) | Phase2 (epochs) | Top-1 Acc (%) | Top-5 Acc (%) \n---|---|---|---|---\n 100 | 15 | 75 | 72.44 | 91.07 \n **100** | **25** | **75** | **72.85** | **91.35** \n 100 | 35 | 75 | 72.71 | 91.26\n 100 | 25 | 65 | 72.52 | 91.12 \n 100 | 25 | 85 | 72.65 | 91.18 \n\nThis ablation confirms that our initial, principled choice is indeed optimal. The best-performing schedule used in our main ImageNet experiments is: **`k-value=100`, `Phase1` ends at epoch 25, and `Phase2` ends at epoch 75**. Importantly, the results demonstrate the **high robustness** of our DTM schedule on ImageNet, as performance remains stable across all tested configurations. This confirms that our method is not sensitive to its hyperparameters on large-scale tasks.\n\n---\n\n**Minor Comments: Including recent works [A-E] in the appendix would provide a more complete comparison and better context for our improvements.**\n\nThank you for these valuable literature suggestions. We agree that including these recent works [A-E] will help contextualize our contributions within the rapidly evolving field of knowledge distillation.\n\nIn the revised manuscript, we will:\n\n- Update the **Related Work** section to thoroughly discuss these papers.\n- Expand the comparison tables in the **appendix** to include reported results from these methods where possible, providing a more comprehensive view of the state-of-the-art and better highlighting the improvements of our work over time.\n\nWe appreciate your suggestions and are confident these additions will further strengthen our paper."}, {"review_id": "Cx7ppFmpbk", "reviewer": "Reviewer_Untf", "summary": "This paper introduces DeepKD, a method that consists of two core components: a dual-level gradient decoupling mechanism based on GSNR analysis and dynamic top-k masking. The authors validate the effectiveness of DeepKD through experiments on CIFAR-100, ImageNet-1K, and MS-COCO, demonstrating performance improvements over baseline approaches.", "questions": "Provide justification for the chosen hyperparameters, such as using 200 iterations for GSNR computation - Include experiments with stronger teacher models, such as ViT or DETR-based architectures", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.The paper presents several straightforward yet effective ideas\n2. Evaluation on the object detection task is a valuable addition\n\nWeakness:\n\n1. The proposed method introduces additional hyperparameters that are difficult to apply in practice, as some lack theoretical justification. For instance, computing GSNR every 200 iterations is hardcoded based on empirical observations. However, this choice likely depends on factors such as the dataset, model architecture, batch size, learning rate, and optimizer, which the authors do not thoroughly analyze.\n2. Why are all experiments conducted using relatively weak teacher models, such as ResNet or R-CNN? How does the proposed method perform when applied to stronger architectures like ViT or DETR?", "was_revised": true, "final_justification": "Thank you to the authors for providing further explanations and conducting additional experiments that effectively addressed my main concerns. Based on these clarifications and new results, I have decided to raise my score. I hope that these insights and the outcomes of the new experiments will be incorporated into the final version of the paper, should it be accepted.", "rebuttal": "**W1: The method introduces extra hyperparameters, such as the 200-iteration interval for GSNR computation, which seems empirically chosen and may depend on various factors. The authors do not fully analyze or justify these choices.**\n\nThank you for raising this important point regarding hyperparameter justification. We would like to clarify that the GSNR computation interval (e.g., every 200 iterations) is not a hyperparameter involved in the actual training process, but rather a setting used exclusively for our offline theoretical analysis. The GSNR plots in our paper (e.g., Figure 1a, Figure 2) are generated to empirically validate our hypothesis about gradient signal quality, and the interval simply determines the sampling frequency for these visualizations. This is a standard approach for analyzing training dynamics and does not impact the method’s practical application.\n\nOur findings show that the theoretical conclusions drawn from the GSNR analysis are robust to the choice of sampling interval, as long as the interval is not extremely small (e.g., >50 iterations). A smaller interval may introduce more noise, while a larger interval improves statistical reliability at the cost of higher computational overhead. We chose 200 as a balanced value for clear visualization and efficient computation. Importantly, this choice does not affect the training or require tuning for different datasets, architectures, or optimization settings.\n\nIn summary, our method does not introduce any GSNR-related hyperparameter that needs to be tuned in practice. The 200-iteration interval is solely for offline analysis and does not influence the training process. The actual hyperparameters used in our method are principled, practical, and robust, as demonstrated in our experiments. We will further clarify this distinction in the revised manuscript.\n\n---\n\n**W2: Why not use stronger teacher models (e.g., ViT, DETR) in experiments? How does your method perform on these architectures?**\n\nThank you for your question regarding the diversity and strength of teacher models in our experiments. Our experimental design follows established practice in the knowledge distillation literature, using widely adopted baselines such as ResNet and Faster R-CNN to ensure fair and direct comparison with prior state-of-the-art methods. In addition, we have included strong teacher models such as ResNet50 and RegNetY-16GF on large-scale datasets like ImageNet.\n\nWe fully agree that evaluating on modern Transformer-based architectures is essential. In fact, our original submission already includes results on Vision Transformers: Table 3 demonstrates that DeepKD, when combined with MLKD, significantly improves the performance of a DeiT-Tiny student on ImageNet, with up to +1.90% Top-1 accuracy gain. We also evaluated strong teacher models, including RegNetY-16GF, across a range of modern architectures.\n\nTo further strengthen our validation, we have conducted additional experiments on Swin Transformer (ICCV 2021) and ViT-to-ViT (ICLR 2021) distillation. As shown in Table 1 below, DeepKD achieves notable Top-1 accuracy improvements of +1.12% and +0.92%, respectively, on ImageNet-1K.\n\n_Table 1: Results on Larger Transformer Architectures (ImageNet-1K)_\n\n| Teacher & Student Models      | Method            | Top1 Acc (%) | Top5 Acc (%) |\n| :----------------------------| :---------------- | :-----------:| :-----------:|\n| Swin-Large → Swin-Tiny       | Teacher (Swin-L)  |     86.30    |     97.87    |\n|                              | Student (Swin-T)  |     81.20    |     95.50    |\n|                              | Baseline (KD)     |     81.59    |     95.96    |\n|                              | **Ours (DeepKD)** |   **82.71**  |   **96.23**  |\n| ViT-L/16 → ViT-B/32          | Teacher (ViT-L)   |     84.20    |     96.93    |\n|                              | Student (ViT-B)   |     78.29    |     94.08    |\n|                              | Baseline (KD)     |     79.40    |     94.76    |\n|                              | **Ours (DeepKD)** |   **80.32**  |   **95.01**  |\n\nIn response to your specific question about DETR, we have also performed new experiments on the DETR architecture for object detection on MS-COCO. As shown in Table 2, DeepKD provides consistent improvements over standard distillation methods, achieving up to +1.0 AP gain on DETR-based models.\n\n_Table 2: DETR Distillation Results on MS-COCO_\n\n| Method                       |    AP    |   APs    |   APm    |   APl    |\n| :--------------------------- | :------: | :------: | :------: | :------: |\n| Teacher (DETR-R101)          |   43.6   |   25.4   |   46.8   |   60.7   |\n| Student (DETR-R50)           |   42.3   |   25.3   |   44.8   |   58.2   |\n| LD (Detrdistill, ICCV 2023)  |   43.7   |   25.3   |   46.5   |   60.7   |\n| **LD + Ours**                | **44.7** | **25.3** | **46.5** | **60.7** |\n| FD (Detrdistill, ICCV 2023)  |   43.5   |   25.4   |   46.7   |   60.0   |\n| **FD + Ours**           | **45.3** | **25.8** | **47.0** | **61.0** |\n\nThese results demonstrate that DeepKD generalizes well to both CNN-based and Transformer-based architectures, including challenging tasks such as object detection with DETR.\n\nWe believe these comprehensive experiments provide strong evidence of DeepKD’s scalability and effectiveness across a wide range of modern architectures. All new results will be included in our revised manuscript.\n\n---\n\n**Q1: Why was 200 chosen for GSNR computation, and are there results on stronger teacher models like ViT or DETR?**\n\nThank you for these questions. Please find a concise summary below, with detailed explanations provided in the main rebuttal text above:\n\n- **On Hyperparameter Justification:** As discussed in the section **\"On Hyperparameters and Theoretical Justification,\"** the GSNR computation interval is used only for offline analysis and does not affect training. Our method’s actual hyperparameters are principled, practical, and robust.\n\n- **On Experiments with Stronger Models:** As detailed in **\"On the Strength and Diversity of Teacher/Student Architectures,\"** we have conducted extensive experiments on modern and strong architectures, including DeiT, Swin Transformer, ViT, and DETR.\n\nWe hope these clarifications fully address your concerns, and we will incorporate all new experimental results into our revised draft."}]

Author rebuttal:

Meta-review: This submission introduces DeepKD, a novel approach to logit-based knowledge distillation, which leverages a dual-level decoupling mechanism and an adaptive denoising strategy to mitigate the observed issues of target-class and non-target-class knowledge conflict and the noise from low-confidence logits. The method has been empirically validated across several benchmarks, including CIFAR-100, ImageNet-1K, and MS-COCO, showing promising results.

Taking into consideration the reviews provided, the submission is positioned on the borderline between accept and reject. While the novelty and theoretical contributions of DeepKD are compelling, and its empirical validation demonstrates effectiveness, concerns around its computational complexity, the limited scope of architectures evaluated, and the absence of an independent effectiveness assessment of DeepKD temper its overall impact.

However, given the positive adjustments in response to reviewer concerns, especially the additional experiments and clarifications provided, it seems that the paper has a solid technical foundation and contributes positively to the advancement of knowledge distillation methods. Therefore, the decision leans towards a "Accept". It is recommended that the authors incorporate the insights and outcomes from the additional experiments into the final version of the paper. Moreover, addressing the highlighted weaknesses, particularly around computational complexity and expanding the scope of experimental validation to include other architectures, would substantially strengthen the paper.
Decision: Accept (poster)