Title: DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer
Venue: NeurIPS 2025
Accepted: yes
Keywords: Knowledge Distillation

Gradient Momentum Decoupling

Dynamic Top-K Mask

Signal-to-Noise Ratio
Abstract: Recent advances in knowledge distillation have emphasized the importance of decoupling different knowledge components. While existing methods utilize momentum mechanisms to separate task-oriented and distillation gradients, they overlook the inherent conflict between target-class and non-target-class knowledge flows. Furthermore, low-confidence dark knowledge in non-target classes introduces noisy signals that hinder effective knowledge transfer. To address these limitations, we propose DeepKD, a novel training framework that integrates dual-level decoupling with adaptive denoising. First, through theoretical analysis of gradient signal-to-noise ratio (GSNR) characteristics in task-oriented and non-task-oriented knowledge distillation, we design independent momentum updaters for each component to prevent mutual interference. We observe that the optimal momentum coefficients for task-oriented gradient (TOG), target-class gradient (TCG), and non-target-class gradient (NCG) should be positively related to their GSNR. Second, we introduce a dynamic top-k mask (DTM) mechanism that gradually increases K from a small initial value to incorporate more non-target classes as training progresses, following curriculum learning principles. The DTM jointly filters low-confidence logits from both teacher and student models, effectively purifying dark knowledge during early training. Extensive experiments on CIFAR-100, ImageNet, and MS-COCO demonstrate DeepKD's effectiveness.


Peer reviews:
comments: SUMMARY:
This paper introduces DeepKD, a method that consists of two core components: a dual-level gradient decoupling mechanism based on GSNR analysis and dynamic top-k masking. The authors validate the effectiveness of DeepKD through experiments on CIFAR-100, ImageNet-1K, and MS-COCO, demonstrating performance improvements over baseline approaches.

STRENGTHS AND WEAKNESSES:
Strengths:

1.The paper presents several straightforward yet effective ideas
2. Evaluation on the object detection task is a valuable addition

Weakness:

1. The proposed method introduces additional hyperparameters that are difficult to apply in practice, as some lack theoretical justification. For instance, computing GSNR every 200 iterations is hardcoded based on empirical observations. However, this choice likely depends on factors such as the dataset, model architecture, batch size, learning rate, and optimizer, which the authors do not thoroughly analyze.
2. Why are all experiments conducted using relatively weak teacher models, such as ResNet or R-CNN? How does the proposed method perform when applied to stronger architectures like ViT or DETR?

QUESTIONS:
Provide justification for the chosen hyperparameters, such as using 200 iterations for GSNR computation - Include experiments with stronger teacher models, such as ViT or DETR-based architectures

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DeepKD, a novel knowledge distillation method that combines principles from DOT and DKD. The core contribution is twofold. \n1. It builds upon DOT's use of separate momentum hyperparameters for the cross-entropy and KL-divergence losses by introducing a principled approach to determine the momentum. This innovation removes the need to tune one of DOT's key hyperparameters. It extends this to the decoupled loss of DKD (DKD decouples of target and non-target knowledge).\n2. The paper proposes Dynamic Top-K Masking (DTM), a technique designed to filter out noise from the probability distributions of semantically unrelated classes by masking out the low probability classes dynamically, in a scheduled manner.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper explores a novel technique of dynamic top-K masking (DTM) \n2. It explores a novel combination of DOT and DKD and proposes an automatic momentum hyper-param selection using GSNR.\n3. The paper shows compelling empirical results, promising performance gains across a variety of student-teacher architectures and on several benchmark datasets, including CIFAR-100, ImageNet, and MS-COCO.\n\n**Weaknesses**\n1. A significant concern is the number of new hyperparameters with respect to KD. In addition to the $\\Delta$ parameter for momentum, the method requires tuning $\\Beta_1$ and $\\Beta_2$ values (for TCKD and DTM, respectively) and a two-stage schedule (Phase-1 and Phase-2) for DTM.\n2. The sensitivity of these DTM scheduling parameters is concerning. The ablation studies show very high performance swings. For instance, with k=55 and Phase-2=170, a minor change in the Phase-1 end from 60 to 40 results in a performance drop of over 7% (to 69.98%). This is significantly worse than the 72.50% accuracy achieved by a standard student model trained without any knowledge distillation. This raises questions about the method's stability and practicality.\n\nQUESTIONS:\nQuestion:\n- What are the phase-1 and phase-2 values for ImageNet?\n\n\nMinor Comment:\n- It would be good to include some of the newer works like [A-E] in the appendix for a more comprehensive comparison. [A] also has a nice list of prior works to contextualize improvement over time.\n\n[A] Yulei Niu, Long Chen, Chang Zhou, and Hanwang Zhang. \"Respecting transfer gap in knowledge distillation.\" Advances in Neural Information Processing Systems, 2022.\n\n[B] Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu, “Knowledge diffusion for distillation,” Advances in Neural Information Processing Systems, 2023.\n\n[C] Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang, “Curriculum temperature for knowledge distillation,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2023, vol. 37, pp. 1504–1512.\n\n[D] Yashas Malur Saidutta, Rakshith Sharma Srinivasa, Jaejin Cho, Ching-Hua Lee, Chouchang Yang, Yilin Shen, and Hongxia Jin, \"CIFD: Controlled Information Flow to Enhance Knowledge Distillation.\" Advances in Neural Information Processing Systems, 2024\n\n[E] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu, “DIST+: Knowledge Distillation From a Stronger Adaptive Teacher,” IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 47, NO. 7, July 2025\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DeepKD, a novel knowledge distillation framework that integrates dual-level decoupling with adaptive denoising. The key contributions are: 1. Dual-Level Decoupling**: Decomposes gradients into task-oriented, target-class, and non-target-class components. Momentum coefficients are adaptively allocated based on gradient signal-to-noise ratio (GSNR), enabling independent optimization of different knowledge types. 2. Dynamic Top-k Masking: Gradually incorporates more non-target classes during training, filtering low-confidence logits to purify dark knowledge transfer. 3. Empirical Validation: Extensive experiments on CIFAR-100, ImageNet, and MS-COCO demonstrate state-of-the-art performance, showing DeepKD's effectiveness across diverse models and tasks.\nDeepKD enhances knowledge transfer efficiency and model generalization through its theoretically grounded approach.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths:\n1. theoretical contributions: The paper proposes a novel dual-level decoupling strategy based on gradient signal-to-noise ratio (GSNR) and a dynamic top-k masking mechanism, both of which are theoretically well-founded and address limitations in existing knowledge distillation methods.\n2. empirical validation: Extensive experiments on multiple benchmarks (CIFAR-100, ImageNet, MS-COCO) demonstrate the effectiveness of DeepKD. The method shows consistent improvements across various model architectures and scenarios, including image classification and object detection.\n3. innovation: the combination of GSNR-driven momentum allocation and dynamic top-k masking is original and provides a new perspective on optimizing knowledge distillation processes.\n4. versatility: deepKD can be seamlessly integrated with existing logit-based distillation approaches and consistently achieves state-of-the-art performance.\n\nWeaknesses:\n1. limited to Logit-Based Distillation: The current framework focuses on logit-based distillation. While the authors mention potential extensions to feature-based scenarios, the paper does not provide concrete results or methods for these cases.\n2. complexity: the introduction of multiple components (TOG, TCG, NCG) and the dynamic top-k masking mechanism may increase the complexity of the implementation and require careful tuning of hyperparameters.\n3. scalability concerns: Although the method shows strong performance on the datasets tested, its scalability to even larger models or more diverse tasks remains to be fully explored and validated.\n4.potential for overfitting: The detailed decoupling and masking strategies might lead to overfitting if not properly regularized, though the paper does not extensively discuss this aspect.\n\nQUESTIONS:\n1. How does the dynamic top-k masking mechanism specifically determine the scheduling of the k value during different training phases, and what is the impact of this scheduling on the overall training efficiency and model performance?\n2. In terms of technical details, how does the dual-level decoupling strategy dynamically adjust the momentum coefficients for different knowledge components during the training process to ensure effective knowledge transfer and model optimization?\n\nLIMITATIONS:\nYes, the paper does discuss its limitations. It notes that the current work focuses on logit-based distillation, but the SNR-driven momentum decoupling mechanism could be extended to feature distillation scenarios. Additionally, the dynamic top-k masking strategy shows promise for improving distillation performance and warrants further exploration for different model architectures and datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on improving logit-based knowledge distillation. The authors firstly observe two critical issues: the conflicts between target-class and non-target-class knowledge flow, and the noise introduced by low-confidence \"dark knowledge\" from non-target classes. To mitigate these challenges, the paper introduces DeepKD, which integrates a dual-level decoupling mechanism with an adaptive denoising strategy. In particular, the proposed DeepKD assigns three independent momentum updaters for different gradient components (task-oriented, target-class, and non-target-class) according to the theoretical analysis of Gradient Signal-to-Noise Ratio (GSNR). Moreover, in order to effectively reduce the noise of non-target classes, a Dynamic Top-K Mask (DTM) mechanism is introduced to progressively filter low-confidence logits from both the teacher and student models during training. Extensive experiments are conducted on CIFAR-100 and ImageNet-1K on image classification task, and MS-COCO on object detection task.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is well-motivated. The proposed DeepKD is pioneering in establishing a direct connection between Gradient Signal-to-Noise Ratio (GSNR) and momentum allocation in the context of logit-based knowledge distillation. \n\n2. Independent momentum updaters derived from the GSNR analysis is proposed to prevent mutual interference among the different knowledge flows during training process.\n\n3. The Dynamic Top-K Mask (DTM) mechanism is introduced to adaptively filter low-confidence logits from both the teacher and student models by gradually increasing the value of 'K'.\n\n**Weaknesses**\n\n1. The computational complexity is increased. The proposed DeepKD framework involves three distinct momentum updaters and a dynamic top-K mask mechanism with three predefined learning phases (Easy, Transition, Hard), inherently adds significant complexity to the training pipeline. It may lead to longer training time and potentially higher memory consumption compared to other KD methods.\n\n2. The experimental settings are limited to CNN-based networks. The paper does not extensively demonstrate DeepKD's effectiveness or performance characteristics when applied to other dominant architectures, such Transformer-based networks. \n\n3. The independent effectiveness assessment of proposed DeepKD is missing. DeepKD is frequently applied in conjunction with other logit-based knowledge distillation methods (e.g., KD, DKD) in the reported tables. This makes it challenging to precisely isolate and quantify DeepKD independent performance. Further empirical evidence is needed to strengthen its contributions, specifically by demonstrating its standalone effectiveness and the performance gains achieved when incorporating feature distillation.\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}