Title: SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Model compression

Self-knowledge Distillation

Proximal Gradient Optimization
Abstract: Self-knowledge distillation (SKD) enables single-model training by distilling knowledge from the model's own output, eliminating the need for a separate teacher network required in conventional distillation methods. However, current SKD methods focus mainly on replicating common features in the student model, neglecting the extraction of key features that significantly enhance student learning. Inspired by this, we devise a self-knowledge distillation framework entitled Self-Distillation training via Proximal Gradient Optimization or SDPGO, which utilizes gradient information to identify and assign greater weight to features that significantly impact classification performance, enabling the network to learn the most relevant features during training. Specifically, the proposed framework refines the gradient information into a dynamically changing weighting factor to evaluate the distillation knowledge via the dynamic weight adjustment scheme. 
Meanwhile, we devise the sequential iterative learning module to dynamically optimize knowledge transfer by leveraging historical predictions and real-time gradients, stabilizing training through mini-batch-based KL divergence refinement while adaptively prioritizing task-critical features for efficient self-distillation. Comprehensive experiments on image classification, object detection, and semantic segmentation demonstrate that our method consistently surpasses recent state-of-the-art knowledge distillation techniques. Code is
available at: https://github.com/nanxiaotong/SDGPO.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SDPGO, a self-knowledge distillation method that uses gradient magnitudes to dynamically weight feature importance. The approach is evaluated on standard and fine-grained classification benchmarks, showing incremental performance gains. However, the core idea of dynamic gradient-based weighting is not sufficiently novel, with similar strategies explored in recent work.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\na. Comprehensive evaluation on both standard benchmarks (CIFAR, ImageNet) and fine-grained tasks (CUB, Cars196), along with downstream tasks and ViT-style models.\\\nb. The results show that the proposed SDPGO performs consistently better across different datasets and architectures.\n\nWeaknesses\n\na. Limited Novelty: The core idea of dynamically weighting feature importance using gradient magnitude overlaps with several recent works that also explore adaptive or gradient-based knowledge distillation [1–5].\\\nb. Lack of Component Isolation: It remains unclear how much the Proximally Weight Assignment Module contributes to the performance gains, as it is not ablated or compared in isolation under the same training settings.\\\nc. Lack of Feature Space Analysis: Although the method emphasizes learning task-critical features via gradient weighting, the paper provides no investigation into what types of features are actually learned. Beyond score comparisons, there is no qualitative or quantitative analysis of feature representations (e.g., t-SNE plots, similarity metrics, layer-wise activations), which limits understanding of the method’s true impact on learned representations.\\\nd. Lack of study on the lambda hyperparameter, which controls feature sparsity and can be critical to performance.\n\n[1] Dynamic weighted knowledge distillation for brain tumor segmentation\\\n[2] Knowledge distillation with adapted weight\\\n[3] A General Dynamic Knowledge Distillation Method for Visual Analytics\\\n[4] Learning Efficient and Accurate Detectors With Dynamic Knowledge Distillation in Remote Sensing Imagery\\\n[5] Dynamic Feature Distillation\n\nQUESTIONS:\nPlease refer to the weaknesses part and address them accordingly.\n\nLIMITATIONS:\na. Lack of direct and isolated evaluation of the Proximally Weight Assignment Module. \\\nb. Lack of feature space analysis. \\\nc. Lack of study on the critical lambda hyperparameter.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SDPGO, a self-distillation framework that leverages proximal gradient optimization to dynamically assign importance to features based on their gradient magnitudes. By integrating a sequential iterative learning module, SDPGO refines soft labels using historical predictions and real-time gradients, enhancing training stability and efficiency. Extensive experiments on CIFAR, ImageNet, and fine-grained datasets demonstrate consistent performance improvements across various network architectures. The method also shows strong deployment potential with reduced inference time and memory usage on edge devices.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n* The method introduces dynamic gradient-based weighting to prioritize critical features, improving self-distillation effectiveness without relying on static heuristics.\n* Extensive experiments on diverse datasets and architectures show consistent improvements over strong baselines, validating the method’s generality and robustness.\n* The approach significantly reduces inference time and memory usage, making it suitable for deployment on resource-constrained edge devices.\n\nWeakness:\n* While the paper leverages gradient information to prioritize features, similar gradient-based sample selection strategies have been widely explored in prior works such as curriculum learning and long-tailed recognition. Beyond the introduction of a weighting scheme, the method lacks substantial innovation in supervision signal design or optimization strategy.\n* Although the proposed method achieves strong results across multiple datasets, the paper lacks a discussion on its compatibility with existing self-distillation techniques and whether SDPGO can be integrated or combined with other advanced SKD strategies.\n* The theoretical analysis, particularly on training stability, appears superficial; for example, the claimed benefit of the proximal term lacks justification compared to standard techniques like weight decay, and its specific advantages remain unclear.\n\nQUESTIONS:\nIt remains unclear where the performance gain truly comes from; unlike data augmentation or auxiliary architectures, the proposed method does not introduce additional information to consistently guide knowledge evolution during training.\n\nLIMITATIONS:\nThe paper lacks analysis of failure cases; it would be helpful to include examples where SDPGO underperforms, or investigate how sensitive existing methods are to suboptimal hyperparameter settings for a fairer comparison.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper determines the global importance of features for SKD (Student-Knowledge Distillation) based on layer-wise gradients. In the proposed method, to ensure the stability of gradient-based KD, the authors estimate feature importance using the variant of the gradient direction, leveraging a proximal approach and z-score. The proposed method achieved higher classification accuracy than existing methods across various datasets.\n\nSTRENGTHS AND WEAKNESSES:\n$\\textbf{Strengths}$\n1. The writing is clear and well-organized, making the paper easy to understand overall.\n2. The structural diagram of the proposed method is intuitive and easy to comprehend.\n3. The paper presents experimental results comparing the proposed method with other approaches across multiple datasets.\n4. The simplicity of the algorithm highlights the contribution of the proposed method.\n\n$\\textbf{Weaknesses}$\n1. The lambda value, which refines the key gradient components determining feature importance, is set manually.\n\nQUESTIONS:\n1. Can the proposed method still be effectively applied if the form of the $L_{SIL}$ loss function changes (e.g., to cross entropy, entropy, etc.)?\n2. If there are experimental results for applying the proposed method without determining the importance of samples (i.e., with $\\alpha=1$), it is recommended to provide them.\n3. Does the effectiveness of the proposed method change depending on the lambda?\n4. The lack of dimensionality representation makes it difficult to grasp the computational meaning. For example, in equation (7), all Z-scores of a layer are added together to form a scalar ($\\alpha$), but how is the dimensionality of $M^l$ defined?\n5. How does the proposed method behave for high and low values of $\\alpha$? An intuitive explanation needs to be added to the manuscript.\n6. Some notations are missing (e.g. $W_i$).\n7. What is the reason for splitting into mini-batches in line 126? The explanation of prior iteration and next iteration is not well connected to the proposed method.\n\nLIMITATIONS:\nIt is unclear whether the advantages in performance improvement can be transferred to attention-based models (such as ViT, as mentioned by the authors), which use alternative methods for determining feature importance.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces a self distillation optimization method called Proximal Gradient Optimization, which is a strategy that combines feature gradient amplitude to determine the importance of features and affect the optimization process. The author describes the shortcomings of traditional self distillation methods and provides solutions, verifying the effectiveness of the method in classification, detection, and segmentation tasks. The core contribution of this article lies in the proximal gradient optimization strategy, which is based on the assumption that features with larger gradient values are more critical in the optimization process. It dynamically assigns self distillation losses and weights to model features during the optimization process, solving the problem that distillation models under static weights cannot adjust knowledge transfer intensity based on the real-time learning status of the model. At the same time, this article proposes dividing batches to balance the model's learning of basic classification ability and distillation generalization ability, providing an adaptation method for proximal gradient optimization strategies.\n\nSTRENGTHS AND WEAKNESSES:\n### strengths\n1. Based on the assumption that the model obtained from the previous stage in the field of self distillation has sufficient generalization ability to guide the learning of student models in the new stage, experiments were conducted on datasets such as CIFAR-100, ImageNet, and CUB200 to verify the validity of this assumption under certain conditions.\n\n2. By dynamically adjusting the weight of self distillation loss through gradient amplitude, the assumption that features with larger gradient values are more important in the optimization process is given more important weights to features in different layers, breaking through the limitations of static distillation.\n\n3. The paper demonstrates in Figure 4 that the method achieves efficient inference delay reduction and memory usage compression on mobile devices, especially on lightweight models such as MobileNet-V2, which validates the practicality of the method on edge constrained devices.\n\n4. In the extension experiment of ViT class models, our method achieved a maximum accuracy improvement of 1.88%, proving that it can adapt to non CNN architectures.\n\n### weaknesses\n1. The experiment mainly focused on classification tasks and the evaluation metric was only accuracy. For segmentation and detection tasks, validation was only conducted on the COCO dataset, the experimental setup was relatively simple.\n\n2. The paper did not include stability testing of noisy labels and small sample scenarios in the experimental design. The experiment only used clean labeled data (such as CIFAR-10/100, ImageNet), without simulating label perturbations (such as injecting noisy labels into CIFAR-10) or controlling the training sample size to verify small sample generalization.\n\n3. The convergence proof of the paper (Section 5.1) requires the loss function to be convex and the gradient to satisfy L-smoothness, which is assumed to be ideal. However, the actual model's loss surface is highly non convex, which may lead to limitations in practical applications.\n\n4. It is not explained in the article whether assigning feature weights based on gradient amplitude will introduce a significant amount of training overhead.\n\n5. The hard threshold sparsity (Eq. 5) strategy lacks flexibility and the selection of thresholds lacks basis.\n\nQUESTIONS:\n1. Does assigning weights to each feature map introduce additional training overhead? What is the degree of impact?\n2. Is there a plan to introduce ablation experiments on sparsity thresholds?\n3. Whether the weight of key features is $\\^{w}_k^l$, could it be indicated in Figure 2?\n4. Is it suitable to verify generalization in tasks such as semantic segmentation (Cityscapes) or pose estimation (MPII)?\n5. Do you plan to conduct additional noise injection experiments (such as adding label perturbations to CIFAR-10)?\n6. Is there a plan to add other indicators (precision/recall/f1/IoU, etc.) to the main experiment (classification, detection, segmentation)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}