SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization
Authors: Tongtong Su
Yun Liao
Fengbo Zheng
Venue: NeurIPS 2025 poster ()
Keywords: Model compression
Self-knowledge Distillation
Proximal Gradient Optimization
Primary area: deep_learning

Abstract: Self-knowledge distillation (SKD) enables single-model training by distilling knowledge from the model's own output, eliminating the need for a separate teacher network required in conventional distillation methods. However, current SKD methods focus mainly on replicating common features in the student model, neglecting the extraction of key features that significantly enhance student learning. Inspired by this, we devise a self-knowledge distillation framework entitled Self-Distillation training via Proximal Gradient Optimization or SDPGO, which utilizes gradient information to identify and assign greater weight to features that significantly impact classification performance, enabling the network to learn the most relevant features during training. Specifically, the proposed framework refines the gradient information into a dynamically changing weighting factor to evaluate the distillation knowledge via the dynamic weight adjustment scheme. 
Meanwhile, we devise the sequential iterative learning module to dynamically optimize knowledge transfer by leveraging historical predictions and real-time gradients, stabilizing training through mini-batch-based KL divergence refinement while adaptively prioritizing task-critical features for efficient self-distillation. Comprehensive experiments on image classification, object detection, and semantic segmentation demonstrate that our method consistently surpasses recent state-of-the-art knowledge distillation techniques. Code is
available at: https://github.com/nanxiaotong/SDGPO.

SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization
Tongtong Su', Liao Yun-", Fengbo Zheng
SScooo of Computer and Information Engineering, Tianjin Normal University "College of Artificial Intelligence, Tianjin University of Science and Technology
Abstract
Self-knowledge distillation (SKD) enables single-model training by distilling knowledge from the model's own output, eliminating the need for separate teacher network required in conventional distillation methods. However, current SKD meth- ods focus mainly on replicating common features in the student model, neglecting the extraction of key features that significantly enhance student learning. Inspired by this, we devise a self-knowledge distillation framework entitled Self-Distillation training via Proximal Gradient Optimization SDPGO, which utilizes gradient in- formation to identify and assign greater weight to features that significantly impact classification performance, enabling the network learn the most relevant features during training. Specifically, the proposed framework refines the gradient infor- mation into dynamically changing weighting factor to evaluate the distillation knowledge via the dynamic weight adjustment scheme. Meanwhile, we devise the sequential iterative learning module dynamically optimize knowledge transfer by leveraging historical predictions and real-time gradients, stabilizing training through mini-batch-basee KL divergence refinement while adaptively prioritizing task-critical features for efficient self-distillation. Comprehensive experiments on image classification, object detection, and semantic segmentation demonstrate that our method consistently surpasses recent state-of-the-an knowledge distillation techniques. Code is available at: http::/ggitubb.ccomnnnaaattoooop
Introduction
In the past few years, deep neural networks (DNNs) have achieved great success computer vision tasks, including image classification 15 45
object detection [22] 41 semantic segmentation
11 [8 and others. Due to the peeeeaaramttiiittionnn and high computational complexity of DNNs, deployment costs are increasingly significant. Knowledge distillation (KD) [14] addresses this by transferring knowledge from large teacher model to a lightweight student via soft target alignment, uniquely enabling efficient compression and fast inference without structural modifications.
Despite its benefits, pre-training high-capacity teacher network requires considerable computational sources and run-in memory. Furthermore, the capacity gap issue 449 65 26 has persistently hindered the development of knowledge distillation. conventional offline KD, one-way knowledge transfer is applied through two-stage training process to steer the learning of the student network [21 43 [2]
In contrast, online KD operates without dependence on a pre-trained teacher, instead enabling an ensemble of students to learn collaboratively in an end-to-end fashion (59 28 46 52). Consequently,
existing KD methods, whether online or offline, tend be computationally expensive and time- consuming, thereby hindering their deployment resource-constrained end devices such mobile phones and digital cameras.
CCoreesponiing author: Liao Yun
Fengbo Zheng (fzh229 (tn..euu.nnnn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

x
Locat
chstillation
Featare
Froture
Fratund
Proture
Auganettaiion
Student
dEsiiliaioon
X
Logits
Block
Block
Blodk
Block1
Block
Block
Block
Block
Hlock NN
Logits
(a) Self-Knowedgge Distillation via
(b) Salf Krowdeddge Distillaton via
(c) Self- Knowledge Distillation via
Auxiliar architocture
Gradient optimizzaion
KD on dles probxéilities
SKD on fecture
Aundiary
Classs
SKD on gradlent information
Block
pulkehilities
Figure 1: Brief comparison of our SDPGO with data-augmentation and muxiliary-architecn methods.
SDPGO utilizes the gradients to identify and distill the most valuable and relevant knowledge.
To address these limitations, SKD [57] addresses these limitations by enabling self-distillation through a single network that acts as both a teacher and student. As depicted Figure aa), prevalent framework in SKD employs auxiliary techniques to enhance the training signal. Augmentation strategies [48] 18] enhance model consistency by enforcing prediction invariance across different augmented views of the same input. Parallelly, historical predictions 33 20
generated during earlier training iterations act as dynamic soft labels to regularize current updates. Another line work introduces auxiliary network components 6 [4). where knowledge from deeper layers is distilled into shallower ones via dedicated pathways in Figure[[[) However, existing SKD approaches often rely on fixed heuristically defined weights prioritize features during knowledge transfer. These weights remain static throughout the training process, thus failing to adapt to the evolving importance of features the model learns.
In this work, we are committed continuously evaluate feature importance through gradient magni- tudes within each mini-batch, enabling instantaneous prioritization of task-critical features. Unlike static distillation methods, our method performs real-time refinement of the knowledge, as illustrated in Figure Ic), enabling dynamic adaptation to evolving data distributions and training dynamics. Specifically, our method leverages real-time gradient analysis to dynamically shift its attention. During high-gradient phases, it amplifies the distillation loss for features that drive immediate perfor- mance gains. Conversely, in low low- www-gradient phases, it recalibrates attention toward under-optimized features to prevent premature convergence.
Motivated by our findings, we propose Self-Distillation training via Proximal Gradient Optimization (SDPGO), novel self-knowledge distillation framework that integrates sequential iterative learning with gradient-driven feature analysis. Our method uses a proximal weight assignment module to assess the importance of intermediate features via their gradient magnitudes from mini-batches, This mechanism, rooted in proximal gradient theory, enables real-time prioritization of high-impact fea- tures while adaptively suppressing less critical ones through sparsity- optimization. Meanwhile, the framework leverages sequential iterative updates to refine knowledge across training steps. Each iteration's is KL divergence between current and previous predictions (from the latter half of mini- batches) creates a feedback loop, allowing the model to incrementally correct errors and consolidate accurate knowledge without additional computational overhead.
Contributions. This work overcomes existing SKD limitations in the utilization of dynamic critical features during training. In doing so, we make the following contributions:
Enables Real-Time Knowledge Refinement. We propose a simple but efficient self- knowledge distillation based on proximal gradient optimization scheme, named SDPGO, which leverages the gradients each mini-batch to dynamically weigh the importance of features (sce Figure 2}. Unlike static SKD methods, SDPGO refines knowledge transfer in real time, eliminating reliance on external teachers and enabling the model to learn critical features directly from its own training trajectory.
Dynamic Srradient-rrvved Weight Assignment: We introduce a dynamic weight assign- ment scheme for self-distillation that dynamically evaluates the importance of extracted features by leveraging gradient magnitudes within each mini-batch. This scheme prioritizes features with larger gradients, thereby guiding the student model to focus on the most impactful knowledge and adapt ongoing training dynamics.

---

Iteration 1-]
Peel Rd
Safumat
Iteration
Sofumax
Coav
Gradicnt
Backward
Reatune
Peature
Penturee
Featurel
Gredienls
-
Proximal operato:
Gradient
Gradicat
Gradient
Gradient
normalitation
Grads
Grad.
Grad,
Ground bruth
Mesa
Figure 2: An overview of SDPGO framework, which sequential iterative learning and gradient-based feature weighting to generate soft labels. This process calculates the KL divergence between iterations and assigns weights features according to their gradient contributions.
Implementation and Evaluation: We conduct extensive experiments on image classifica- tion, object detection, and semantic segmentation datasets based on different popular network architectures, where the proposed SDPGO framework outperforms recent sfate-of-the-ar KD methods with clear margin. In addition, ablation experiments further demonstrate the effectiveness of our method.
2
Related Works
KD compresses neural networks by transferring knowledge from a high-capacity teacher to lightweight student, enabling efficient deployment of the compact model. KD leverages the teacher model's ' knowledge by utilizing 1) class predictions soft labels [6) 2) correlations between multiple samples [33 42 3) or 3) feature maps containing spatial information from intermediate lay- ers 4 2 10]
Under the guidance of teacher model, the student model trained in an offline learning way typically effective. However, the entire training process is time-conuuming and complex, as requires retaining a large amount of persistent data to train a strong teacher model, In contrast, online KD 27 52 34 (39] eliminates the need for larger teacher models, instead training a group of student networks by learning from each other's 's predictions. However, there are two key limilations: (I) online KD depends on using an ensemble teacher to improve the student model's 's performance, and (2) selecting the optimal student model requires considerable computational resources for training.
Compared with the conventional KD, SKD [55
58 19 uses the student network itself teacher to
guide its learning process, which utilizes the information within the student model. In recent years, series of SKD methods have been continuously proposed. BYOT 55 attaches auxiliary classifiers to each stage, using the final classifier's s output as teacher knowledge to train them. FRSKD [18] employs an auxiliary self-teacher network with bidirectional paths transfer refined features and soft labels to the classifier. However, these methods rely auxiliary models, which complicates training by increasing memory and time costs. Meanwhile, data augmentation was used for SKD. For example, MiSKD [51] combines SKD with image mixture to distill features and distributions between original/mixed images via a self-teacher network, enhancing self-boosting. Meanwhile, SKD uses historical information regularization targets. These methods takes the predictions output for knowledge transfer from its last generation [9] last mini-generation 550 36) last epoch [20]
virtual teacher, treating predictions from previous model versions as
63 begin to employ gradients assess the importance of feature maps. These methods assigns static or predefined weights to features impacting task loss, primarily emphasizing knowledge transfer from teacher to student. Unlike previous approaches, we design a proximal gradient optimization mechanism that dynamically refines gradient-derived weights during training. This adaptively prioritizes features most relevant task loss, optimizing representation learning in end-to-end manner.
Recently, some researchers
3

---

3 Proposed Approach: SDPGO
Notations. We denote set of labeled dataset D (s)) where N is the dataset's size. - denotes the i-th sample and it belongs to category k. We sample batch samples to feed into target neural network f(-) to optimize the cross-entropy loss function. Then, we obtain the logit vector 2; and then yields the prediction probabilities p,(k) f( /T) by a softmax function. Hence, the temperature parameter 7 is introduced to control the importance of the i-th soft target follows:
exp(z./7)
(i(k) Eイ=1 exp(2s/7)
where the 2; is the predicted value of class from fully connected layer, and denotes temperature parameter. A softmax function with r transforms an original vector into probability vector.
3.1 Sequential Iterative Learning Module
We employ the sequential iterative learning module to measure the consistency of two adjacent batches. This module uses backup information from the last mini-batch to generate soft targets. We partition each mini-batch into two sequential segments: half aligned with the prior iteration and half' with the next iteration. This strategy enables real-time distillation of dynamically updated soft targets generated from preceding training steps.
The student model acts as its own teacher by reorganizing sequential sampling. Each mini-batch is divided into halves aligned with the previous and upcoming iteration respectively. The model applies KL divergence between classifier outputs from different iterations to align predictions, while utilizing historical outputs for enhanced regularization. We denote the current batch of samples in the 1th iteration as B, {(x), =/) Afterwards, they are fed into convolutional layers to obtain their
representation vectors prediction distributions p; T.t Our work uses historical information from the last batch to efficiently generate soft targets as more instant smoothed labels for regularization. we substitute the p - by the soften labels p - generated by the identical network at 1-th iteration. The main difference from conventional KD is that the teacher model dynamically evolves during training, with the f-th iteration predictions used as the teacher's knowledge without incurring any loss. Consequently, the student is trained with the consistency regularization loss as follows:
LSI =-
log
P P
DEL
where Do¹ is the soften prediction, parameterized by temperature 7 from the student itself. Then, the total loss combines KL divergence with hard-label cross-entropy, enabling end-to-end optimization of the single network:
LCE Σ H (y.,p))
where H is the cross-entropy loss between the ground-tron truth label y; and the prediction P:-
3.2 Proximally Weight Assignment Module
We propose a gradient-driven dynamic weighting scheme to enhance knowledge distillation, where feature importance is evaluated via gradient magnitudes. High-gradient features receive amplified attention to prioritize impactful knowledge, while low-gradient features trigger adaptive recalibration toward under-optimized parameters. To obtain the importance of the feature, we define the gradients of the k-th feature map in layer as:
aLrssk OF
(4)

---

where Fy, is the k-th feature representations of the 1-th layer, and i,j are two location vectors. Next, we calculate the gradient of the loss Tusk with respect to the feature F. The raw gradient-derived weights w (Eq. is refined via a proximal operator enforce sparsity and stability:
w
X if w A
1 Proxx (wh)
(5)
otherwise
where À is an adaptive threshold controlling feature sparsity using only moving averages, thus the | corresponding loss can be: X 3At-1 + + 3) K A is dynamic threshold at step t, 3 is the momentum factor. |w/1 is L1-norm gradient wights in current batch. K the numbers of features in batch. This step suppresses less impactful features while retaining those with significant gradients, aligning with the proximal gradient principle of separating critical and non-critical components. Next, compute the mean absolute value of the gradient for each weight parameter. Then, apply Z-score normalization to these values, given by:
M Z-score
Σ
(6)
where M is spatial feature map (R HxW) derived from channel-summed activations with position- wise Z-score normalization, where H/ and W represent height and width, respectively. indicates an intermediate layer. L is the total number of intermediate layers considered for distillation. Dimensionality reduction occurs through spatial averaging after normalization. We assign weights to features according their impact on overall loss. This process effectively highlights the most influential features, ensuring that the model focuses on the aspects of the data that contribute the most to performance. The method incorporates dynamic weight, a, that balances the task and distillation losses. This scalar parameter is integrated into the final loss function to modulate the influence of each component. Therefore, in all network layers is computed as follows:
W ΣΣ
H
a L
(i i=
(7)
HW
In previous SKD methods, the student remains unchanged because the overall task loss is usually set to fixed value during training. However, SDPGO go step further by leveraging the gradients of feature maps at immediate layers and transforming them into a dynamic hyperparameter. Similar to learning rate scheduling, this hyperparameter controls the degree which the model captures valuable information about each feature's s contribution. This approach allows the model to dynamically adjust its focus features as their importance changes during training. The goal of our SDPGO method is to utilize gradient information via assigning greater weights to critical features.
3.3 Overall for SKD
With the sequential iterative learning module and the gradient information extraction module, we calculate the corresponding loss and propose SDPGO as shown in Fig. We train all models with the total loss for SKD as follows:
LTeal = Cask + aLsrL-
(8)
where Lrssk is the domain-specifi final objective loss. For image classification setting, this is defined as the cross-entropy loss between the hot ground-truth label y and the predicted distribution p.
Experiments
In this paper, we conduct experiments on five visual recognition datasets, namely CIFAR- 24).
ImageNet (6. CUB200-2011 [44] and 196 (23]. We employ eight representative architectures
40 for evaluation, namely VGG- 16/19 (37]. ResNet-322110 (abbreviated as R-32/1 10) [3] WRN20- (56, DNet-40-12 [17], ShuffleNer- V2 (abbreviated as SN-V2) [2) and MobileNet-V2 (abbreviated
MN-V2) [16]. All results are reported in means (standard deviations) over 3 trials.
5

---

Table I: Top-1 accuracy (%) of various SKD methods across widely used networks on CIFAR-I0 -- (C10) and CIFAR-100 (C100). The best results highlighted in bold, while the second-best results are underlined. We use A to show its performance gain.
Dataset Methods
Vgg-16
R-32
R-110
WRN-20-8
DNet-40-12
SN-V2
MN-V2
Baseline
93.97
93.46
94.79
94.53
92.91
92.70
93.31
BYOT 58
94.03
93.57
94.86
94.14
93.01
92.99
93.73
EFWSNet 62
93.85
93.97
94.92
94.68
93.39
93.23
93.88
PS-KD 20
94.10
94.04
94.91
95.01
93.23
93.45
94.02
FRSKD
94.38
94.78
95.23
95.27
94.21
94.17
94.76
C10
DLB 36
94.62
94.15
95.15
95.54
93.43
95.10
94.46
MiSKD [
93.82
95.59
95.93
93.93
94.25
95.29
94.91
FASD 48
94.21
95.45
95.66
94.52
94.39
95.34
94.86
SDPGO
95,90
96.44
95.98
95.70
95.60
95.73
95.43
1.93
2.98
1.19
1.17
2.69
3.03
2.12
Baseline
73.63
71.74
76.36
77.58
71.69
71.82
68.08
BYOT 5
73.79
72.39
77.75
77.68
77.04
72.97
68.72
EFWSNet 62
73.92
73.54
75.81
78.02
76.95
72.87
69.45
PS-KD 20
74.05
72.51
77.15
78.74
72.52
74.55
69.74
FRSKD 18
76.72
75.34
79.15
78.95
77.12
75.23
70.25
C100
DLB 36
76.12
74.07
78.18
79.21
72.52
75.51
69.47
MiSKD 51
76.57
75.12
78.86
78.07
76.85
76.52
71.66
FASD
75.52
75.42
78.52
78.62
77.24
76.76
71.75
SDPGO
76.85
75.57
79.31
79.36
78.04
77.29
72.25
A
3.22
3.83
2.95
1.78
6.35
5.47
4.17
Dataset. CIFAR-10/100 244 contain total number 60,000 RGB natural images of 32x32 small
RGB images drawn from 10 and 100 categories, respectively. For ImageNet [6). we use 1.2 million images for training and 50,000 images for validation. The size of input images after pre-processing 224 x 224. CUB200 44 and Cars196 [23] are used for fine one-grained visual recognition (FGVR) tasks. Different from CIFAR and ImageNet, FGVR datasets typically have fewer data instances per class, making the task more challenging. More details are available in Appendix A.
Baselines. We compare our SDPGO with different SOTA SKD methods, be your own teacher (BYOT) (88, feature-sharing and weight-sharing based ensemble network (EFWSNet) [62], pro-
gressive self-KD (PS-KD) [20), feature refinement via SKD (FRSKD) [18], distillation from last
mini-batch (DLB) (36). SKD from image mixture (MiSKD) (51). normalized knowledge distillation
(NKD) 54] and feature augmentation based self-distillation method (FASD) (48)
Implementation details. We used the Stochastic Gradient Descent (SGD) optimizer with a
momentum of 0.9, weight decay of 5e-4, and temperature of 3. On CIFAR-100, used batch size of 64, an initial learning rate of 0.05, and trained the model for 240 epochs. For ImageNet, training conducted over 100 epochs, with learning rate adjustments epochs 30, 60, and 90 by multiplying the learning rate by 0.1. For CUB200, the learning rate was set to 0.05 with a warmup period of 2 epochs, batch size of 64, and total of 240 training epochs. The learning rate was adjusted at epochs 70, 140, and 210 by multiplying it by 0.1, while other hyperparameter' remained constant. For Cars196. training was conducted for 200 epochs, with learning rate adjustments epochs 60, 120, and 180 by multiplying it by 0.1. Experimental details about Pascal VOC, ADE20K and Cityscapes in Appendix
4.1 CIFAR Classification Result
We compare the KD results of different methods under various backbone networks settings in Table I Across different architectures, networks show improvements these two datasets when using SDPGO. We first compare our method with the latest SKD techniques, BYT (58) and EFWSNet 622 both of which are combined with auxiliary architectures. Our method consistently delivers significant improvements to the student. Specifically, ResNet-32 achieves 94.44% and 75.57% top-1 accuracy on CIFAR-10 and CIFAR- 100, respectively, reflecting accuracy gains 5.87% and 3.83% than BYT 58 through the knowledge transferred from gradient information. Furthermore, compared to input-space data augmentation methods, our SDPGO approach provides a significant boost in top-1 accuracy for classification tasks. Specifically, with a DNet-40-12 backbone, SDPGO achieves gains of 1.35% on CIFAR-10 and 1.19% on CIFAR- 1000 than MixSKD. This result shows that student models distilled by SDPGO benefits from our gradient optimization well.

---

Pumrrninn KN6
bullet to
998
Cumbers 946
magaliie cosTleSU0M
gonien spdke forn spideat 1.11 spader wez:0.00
Regsht bakeryyt
Sketland sheesóogilLup Persan carMU
Jea
Strettcrrt.otp
(122
Figure 3: Visualization of the target and top-2 non-target class values of our customized soft labels.
Table Top-1 and Top-5 accuracy (%) of our SDPGO method. ResNet-18 is used classifier network on the ImageNet dataset.
Table 3: Top-1 accuracy (%) of various self-
knowledge distillation methods across widely used networks on CUB200 and Cars 196 dataset.
Res.et--1
18
CUB200
Cars 196
Methods
Top-1
Top-5
Gain(t)
Method
Top-1 1
Top-5
Top-1
Top-5
ResNet- 18
69.75
89.07
ResNet-18
69.66
91.66
71.82
91.04
FitNet
71.61
90.51
1.86
+ BYOT (58
73.38
91.18
79.35
94.70
Review
70.81
89.98
1.06
+ DLB [36
76.10
93.37
78.28
93.13
CAT-KD 12
71.22
90.26
1.47
+ MiSKKD [51]
71.11
91.37
82.94
95.83
CRD 42
71.17
90.13
1,42
+ SDDGG
78.06
94.77
84.17
96.44
SSKD 47
71.62
90.67
1.87
ResNet-50
74.36
92.52
76.44
92.26
DCCD 50
71.95
90.88
2.2
+ BYOT (58]
77.76
94.22
80.17
94.78
BYOT 58
69.84
89.62
0.09
+ DL.B [36
80.69
95.62
82.94
95.83
EFWSNet 66
72.36
91.74
2.61
+ -MSKDD 51
75.96
93.03
77.90
93.55
PS-KD 20
71.59
90.85
1.84
+DPGG
81.69
95.88
89.20
98.16
FRSKD 18
70.17
90.52
0.42
MobNct-V2 V2
73.09
91.82
72.15
93.01
DLB 36
70.12
90.27
0.37
+ BYOT 58
74.25
91.92
82.51
95.03
MiSKD [51]
71.67
91.20
1.92
+ DLB (36
78.08
94.32
81.72
92.93
FASD 48)
71.70
90.91
1.95
MiSKD [1]]
73.65
91.23
82.69
94.41
SDPGO (Ours)
72.47
92.56
2.72
+ SDPGO
78.67
94.75
85.28
4.2
ImageNet Classification Result
As shown in Table 2 SDPGO outperforms advanced KD methods terms of top- and top-5 accuracies, achieving the highest scores of 72.47% and 92.56%, respectively. We select current feature-based KD methods with the utilization of intermediate representation or feature embedding. For feature-based methods, the second-best method DCCD 30) exhibits inferior performance our SDPGO, highlighting its limitations in distilling knowledge due to relying on expert knowledge from the teacher model. For self-KD methods, SDPGO surpasses the best-competing FASD [48] by 0.77% top-1 accuracy gains, which enables successful SKD. SDPGO surpasses other SOTA methods with clear gains, demonstrating its superiority in large-scale datasets. These findings confirm the effectiveness of SDPGO in distillation optimization, highlighting its versatility and strength.
4.3 Fine-grained Classification Result
For fine-grn fine-grained image classification, Table presents the top-1 and top-5 accuracy of SDPGO compared to other SKD methods across different backbones. We observe the top-1 acc of 82.94% and 82.69% on Cars 196 when Mixup is combined with SKD, which is 4.66% and 0.97% improvement of DLB. When comparing with MixSKD, SDPGO demonstrates superior performance on fine-grained classification tasks, achieving the highest accuracy across all datasets. Specifically, SDPGO surpasses the second- best results by 1.60%, 3.63% and 1.59% on three networks, respectively. This significant improvement highlights the effectiveness of our approach in combining sequential iterative learning and gradient information extraction.
4.4 Visualization
Figure iillustrates how our oeff-iistillatin method, SDPGO, generates customized soft labels for each image during training, including the target class and the top two non-target classes. By using gradient- dynamic weight assignment method from Eq. 6, higher weights are assigned to classes similar to the target, enhancing the overall learning process. These results demonstrate that SDPGO's s soft labels prioritize classes with high semantic affinity, thereby facilitating improved discrimination and robustness by emphasizing inter-class relationships during training.
7

---

Table 4: The downstream tasks results with
Table 5: Results of training more models, includ-
SDPGO on the COCO-2017 dataset.
ing ViT-liked models with SDPGO on ImageNet.
Methods
ResNet-18
ResNet-50
Model
Baseline
SDPGO (Ours)
bbox
segm
bbox
segm
ResNet-50
73.56
74.03 (+0,47)
baseline
33.4
30.2
36.9
33.4
DeiT-Tiny
74.42
75.01 (+0.59)
MixSKD
33.9
31.05
37.0
33.8
DeiT-small
80.55
80.89 (+0.34)
FASD
34.1
30.9
37.3
34.4
Swin-Tiny
81.18
81.95 (+0.77)
SDPGO
35.08
32.10
38.08
36.69
Swin-small
84.36
86.24 (+1.88)
Table Overall performance comparison on semantic segmentation tasks.
Model
Method
ADE20K
Cityscapes
Baseline
39.72
74.85
MixSKD
42.37
74.96
ResNet-50
FASD
40.78
72.89
SDPGO
42.75
75.01
Table 7: Performance comparison on Pascal VOC segmentation task.
Model
Method
mIOU
Model
Method
mIOu
Baseline
79.07
Baseline
81.95
MixSKD
79.52
MixSKD
82.51
EfficientDer-d0
EfficientDet-d1
FASD
80.54
FASD
83.43
SDPGO
80.67
SDPGO
83.97
4.5 Extension to ViT-liked models
Recent models like ViT g) which use embedded image patches, have adopted a different architecture, but current SKD methods do not address their unique characteristics. SDPGO leverages gradient information to prioritize features, eliminating the need for additional linear layers connected to ViT's middle layer, and allowing the model to focus the most relevant aspects during training. As can be seen in Table 5 SDPGO outperforms these baselines by margins ranging from 0.34% 1.88%. It should be noted that method even improves student model performance with gains of up 1.88% for the latest state-of-the-ar model Swin-Transformer (Swin-small) 31 These results show that SDPGO achieves significant improvements with general settings.
4.6 Knowledge transfer to downstream tasks
To assess its generalization capability, we transfer our model to object detection task on the COCO- 2017 dataset [29]. As shown in Table ResNet trained with SDPGO brings the feature extractors with average 1.43% and 2.60% points for downstream detection task compared to the baseline method. The results demonstrate the efficacy of SDPGO for learning better representations for downstream semantic recognition tasks. In addition, employ the Pascal VOC, ADE20K, and Cityscapes benchmarks for semantic segmentation. As shown in Table SDPGO achieves the best segmentation performance and outperforms the best-second results with 2.65% and 3.04% mloU margins on Pascal ADE20K and Cityscapes segmentation, respectively. For Pascal VOC 2012 semantic segmentation, we EfficientDet with stacked BiFPN structure [41] baseline. Table demonstrates that SDPGO substantially outperforms baseline segmentation models by dynamically intensifying task-critical features via gradient-weighhed distillation. Empirical results reveal that the proposed method generalizes remarkably well to other domains like object detection and segmentation without requiring architectural changes.
5 Analysis
In this section, we provide an in-depth analysis of SDPGO. We begin with theoretical justification behind the efficacy of SDPGO. Then we perform qualitative analysis to visualize the representations of the models. Finally, we explore the impact of different design choices through ablation studies, focusing on the effects of utilizing the gradient optimization.

---

12
14
54
S00EE 28
G0000 du.Eai
Informnc Thmas)mo
Networka
(r Repbenty F:48 (CPU: BON2T.1)
Ruephers sygtt, (CM2211)
(c) annrrpiioon Reepharty
Figure 4: Inference time and memory consumption on ImageNet, (a) and (b) represent the model tested on Raspberry Pi 4B and Raspberry Pi 5, respectively. (c). "Memory" indicates whether additional information needs to be stored during knowledge distillation.
Table 8: Top-1 Acc (%) and training time for each batch of data of competitive KD methods.
Method
Baseline
ReviewKD
CATKD
BYoT
PS-KD
DLB
FASD
Ours
Training Time
12
25
17
19
15
17
40
12
Top-1 Acc (%)
69.66
76.58
72.6
73.38
75.43
76.10
75.43
78.06
5.1
Theoretical Justification
First, we provide theoretical foundation for the proposed Self-Distillation training via Proximal Gradient Optimization (SDPGO). The gradient of the total loss with respect to model parameters is:
TLraa
T
dLsiL
+
(9)
00
30
00
By dynamically adjusting SDPGO modulates the influence of self-distillation on parameter updates, ensuring stable training.
Stability. The sequential iterative learning module incorporates proximal gradient optimization to refine feature weights and stabilize training. Raw gradient-derived weights vi are processed via proximal operator: di Prax, (wf). A enforces sparsity by suppressing non-critical features. This mitigates noise amplification during self-distillation. We add the proximal term |0 0:-1 the loss to penalize abrupt parameter changes:
Draall LTak + aLSIL + + [|0, 0.-1
(10)
This ensures smooth convergence by aligning parameter updates with historical states.
Convergence Guarantee. To establish the convergence of SDPGO, we leverage the properties of proximal gradient descent under standard convexity and smoothness assumptions. Let the to- tal loss L = Lrskk + aLsit. be L-smooth (i.e., VL (01) - V Lioul (02) I L [|0 02| for all 01, 0g and convex. The optimization proceeds via the following iterative steps: 1. Gra-
dient Step: 4.+1/2 = 0, V (0,). Proximal Step: 0,+1 = Prox,A (0++1/2) =
arg ming 2q | //41/2|| + A ||0 11) If the learning rate satisfies 1) S 1. the sequence
{0t) generated by SDPGO converges to global minimum.
5.2 Efficiency and Memory Consumption Analysis.
We measure the actual speedup lightweight neural networks on two mobile devices, Raspberry Pi 4B and Raspberry Pi 5 in Figure 4 In Figure 4 (a) and (b), SDPGO achieves an average inference acceleration optimization of 23. 23.46% on Shuffle- V2 and Mobile-V2 backbone. For Shuffle-V2 (x0.5) on the Raspberry Pi 4B, the proposed method reduces hardware inference time by 17.08% while improving the baseline by 2.38%. Meanwhile, the average inference latency of SDPGO on the Mobile-V2 V2 model is 60.95 ms, and the improvements is up to 34.69% on ImageNet. In Figure (c), SDPGO delivers significant memory compression, achieving approximately 3x reduction compared to the state-of-the-an FASD method and 5x reduction compared to MixSKD on VGG-8. Moreover, we compare the training time of various KD methods in Table g In particular, our method shows competitive performance by assessing the training time of each batch of data CUB200.

---

Table 9: Performance comparison when frac- tion of training data is noisy.
Table 10:
Performance comparison between SOTA SKD and SDPGO when a fraction of data
present for training purpose.
BYOT
PS-KD
DLB
FASD
Ours
72.39
72.51
74.07
75.42
75.57
BYOT
PS PS-KD
DLB
FASD
Ours
10
65.41
62.75
67.56
64.25
71.56
25
49.57
48.75
51.28
60.34
65.29
20
58.05
57.56
65.17
60.51
68.53
50
58,25
56.23
59.56
68.62
70.07
30
53.25
51.71
54.45
57.27
61.58
75
63.43
60.58
68.12
70.47
71.58
40
42.22
41.26
$1.25
55.39
59.81
100
72.39
72.51
74.07
75.42
75.57
5.3
Robustness Analysis under Noisy and Scarce Data
We experiment with our method for few-shot and noisy-label learning on CIFAR-100. We evaluate SDPGO against standard KD baselines under symmetric label noise (uniform corruption probability m). Experiments use ResNet-32 with E 0%, 10%, 20%, 30%, and 40%. As summarized in Table 9 the proposed SDPGO proves be the most resilient symmetric label noise. At noise level of 1=40%, it attains top accuracy of 59.81% outperforming the strongest competitor, FASD (55.39%), by clear margin. For few-shot learning, we train ResNet-32 on random subsets of CIFAR-100, using 25%, 50%, 75%, and 100% of the training data per class. The results in Table TO validate the sample efficiency of SDPGO, which achieves the highest top-1 accuracy across all data fractions (e.g., 65.29% at 25%), markedly outperforming conventional SKD methods.
5.4 Integration Analysis
SDPGO's modular architecture enables seamless compatibility with diverse self-distillation (SKD) paradigms. We evaluate our pre-process on existing KD methods. These findings indicate that using SDPGO as plugged-in regularization enhance the generation of other approaches. As shown in Table 11 student models distilled by SDPGO benefits from our pre-process as well.
Table 11: The results of SDPGO combined with other advanced SKD techniques CIFAR-100.
Model
VGG-13
ResNet-32
ResNet-110
ShuffleNet-V2
MobileNet-V2
KD 72.98
73.08
74.36
71.82
66.95
KD+Ours
73.53
74.32
75.11
72.72
67.24
DLB 75.45
74.07
78.18
75.51
69.47
DLB+Ours
75.67
75.19
79.28
77.56
69.82
More ablation studies can be found in Appendix C.
6 Conclusion
In this work, we introduce framework called (SDPGO). This framework leverages gradient information to prioritize features that significantly influence classification performance, allowing the model to concentrate on the most relevant aspects during training. Furthermore, it reduces computational costs while improving the model's ability to represent features. The dynamic weight adjustment scheme refines the gradient information into variable weighting factor to evaluate the distillation knowledge. Our SDPGO method effectively enhances the student model's generalization capability by emphasizing key features linked to classifi- cation loss. Extensive experiments on five image classification benchmark datasets at different scales show that our framework outperforms recent state-of-the-ar knowledge distillation techniques.
training via Gradient Optimization
Acknowledgment
This work is partially supported by NSFC (Natural Science Foundation of China):
0600335.620002266 National Key Research and Development Plan: 2019YFB2101900; Key Project of Tianjin Natural Science Foundation: 25JCZDJCW0200 TianKai Higher Education Innovation Park Enterprise R&D Special Project: 23YFZXYC00446 The Tianjin Science and Technology Program under Grant: 24YDTPJC00630, Tianjin Municipal Education Commission Research Program Project under No. 2022KJ012; Foundation of Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education No.BATLLAB02401)
10

---

References
[1] Romero Adriana, Ballas Nicolas, K Samira Ebrahimi, Chassang Antoine, Gatta Carlo, and Yoshua. Fitnets: Hints for thin deep nets. Proc. ICLR, 2, 2015.
[2] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Varia- tional information distillation for knowledge transfer. CVPR, pages 9163-9171 9171, 2019.
[3] Hanting Chen, Yunhe Wang, Chang Xu, Chao Xu, and Dacheng Tao. Learning student networks via feature embedding. IEEE Transactions on Neural Networks and Learning Systems, 32(1):25-
35, 2020.
[4] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, pages 5008-5011 2021.
[5] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and paftern recognition, pages 3213-3223, 2016.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pages 248-255, Ieee, 2009.
[7] Qianggang Ding, Sifan Wu, Tao Dai, Hao Sun, Jiadong Guo, Zhang-Hup Hua Fu, and Shutao Xia. Knowledge refinery: Learning from decoupled label. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7228-7235, 2021.
[8] Alexey Dosovitskiy. An image worth 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.111
2020.
[9] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In ICML, pages 1607-1616. PMLR, 2018.
[10] Jianping Gou, Xiangshuo Xiong, Baosheng Yu, Lan Du, Yibing Zhan, and Dacheng Tao. Multi- target knowledge distillation via student self-reflection. International Journal of Computer Vision, (31 2023.
[11] Yanming Guo, Yu Liu, Theodoros Georgiou, and Michael S Lew. review of semantic segmentation using deep neural networks. International journal of multimedia information retrieval, 7:87-93, 2018.
[12] Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. CVPR, pages 11868 1877, 2023.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurlPS Workshop, 2015.
[15] Trung-Hieu Hoang, Hai-Dang Nguyen, Viet-Anh Nguyen, Thanh-An Nguyen, Vinh-Tier Tiep Nguyen, and Minh-Triet Tran. Enhancing endoscopic image classification with symptom localization and data augmentation. In Proceedings of the 27th ACM International Conference on Multimedia, pages 2578-2582, 2019.
[16] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 048611, 2017.
[17] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708 2017.

---

[18] Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, and I1-Chul Moon. Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In CVPR, pages 10664- 10673, 2021.
[19] Hai Jin, Dongshan Bai, Dezhong Yao, Yutong Dai, Lin Gu, Chen Yu, and Lichao Sun. Personal- ized edge intelligence via federated self-knowledge distillation. IEEE Transactions on Parallel and Distributed Systems, 34(2):567-580, 2022.
[20] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. In ICCV, pages 6567-6576, 2021.
[21] Nikos Komodakis and Sergey Zagoruyko. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. ICLR, 2017.
[22] Tao Kong, Fuchun Sun, Huaping Liu, Yuning Jiang, Lei Li, and Jianbo Shi. Foveabox: Beyound anchor-based object detection. IEEE Transactions Image Processing, 29:7389-7398. 2020.
[23] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, pages 554-561, 2013.
[24] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Handbooo of Systemic Autoimmune Diseases, 1(4), 2009.
[25] Qizhen Lan and Qing Tian. Gradient-guice knowledge distillation for object detectors. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 424 433, 2024.
[26] Lujun Li, Peijie Dong, Anggeng Li, Zimian Wei, and Ya Yang. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. Advances in Neural Information Processing Systems, 36:69490-69504 2023.
[27] Lujun Li and Zhe Jin, Shadow knowledge distillation: Bridging offline and online knowledge transfer. Advances in Neural Information Processing Systems, 35:635-6499 6495,669,660 2022.
[28] Shaojie Li, Mingbao Lin, Yan Wang, Yongjian Wu, Yonghong Tian, Ling Shao, and Rongrong Ji. Distilling a powerful student model via online knowledge distillation. IEEE transactions on neural networks and learning systems, 34(1 1):8743-8755 -8552, 2022.
[29] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCY 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[30] Yuang Liu, Jun Chen, and Yong Liu. Deed: Reducing neural network redundancy via distillation. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[31] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[32] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116-131, 2018
[33] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, pages 3967-3976, 2019.
[34] Biao Qian, Yang Wang, Hongzhi Yin, Richang Hong, and Meng Wang. Switchable online knowledge distillation. In European Conference on Computer Vision, pages 449-466. Springer, 2022.
[35] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28, 2015.
12

---

[36] Yiqing Shen, Liwu Xu, Yuzhe Yang, Yaqian Li, and Yandong Guo. Self-distillation from the last mini-batee for consistency regularization. In CVPR, pages 11943-11952, 2022.
[37] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409, 1556, 2014.
[38]
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, pages 7262-7272, 2021.
[39] Tongtong Su, Qiyu Liang, Jinsong Zhang, Zhaoyang Yu, Gang Wang, and Xiaoguang Liu. Attention- based feature interaction for efficient online knowledge distillation. In 2021 IEEE International Conference on Data Mining (ICDM), pages 579-588. IEEE, 2021.
[40] Tongtong Su, Qiyu Liang, Jinsong Zhang, Zhaoyang Yu, Ziyue Xu, Gang Wang, and Xiaoguang Liu. Deep cross-layer collaborative learning network for online knowledge distillation. IEEE Trunsuctions on Circuits and Systems for Video Technology, 2022.
[41] Mingxing Tan, Ruoming Pang, and Quoe V Le. Efficientdet: Scalable and efficient object detection. In CVPR, pages 10781-10790, 2020.
[42] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv: 10699, 2019.
[43] Frederick Tung and Greg Mori. preeeerriiyy knowledge distillation. In ICCV, pages 1374,1374, 2019.
[44] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 200-2-200-2011 dataset. 2011.
[45] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156-3164, 2017.
[46] Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In Proceedings of the AAAI Conference artificial intelligence, volume 35, pages 10302-10310, 2021.
[47] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In ECCV, pages 588-604. Springer, 2020.
[48] Kai Xu, Lichun Wang, Shuang Li, Jianjia Xin, and Baocai Yin. Self-distillation with aug- mentation in feature space. IEÉE Transactions on Circuits and Systems for Video Technology. 2024.
[49] Quanzheng Xu, Liyu Liu, and Bing Ji. Knowledge distillation guided by multiple homogeneous teachers. Information Sciences, 607:230-243, 2022.
[50] Chenglin Yang, Lingxi Xie. Chi Su, and Alan L Yuille. Snapshot distillation: Teacher-student optimization in one generation. In CVPR, pages 2859-2868, 2019.
[51] Chuanguang Yang. Zhulin An, Helong Zhou, Linhang Cai, Xiang Zhi, Jiwen Wu, Yongjun Xu, and Qian Zhang. Mixskd: Gelf-knowledge distillation from mixup for image recognition. In European Conference Computer Vision, pages 534-551. Springer, 2022.
[52] Chuanguang Yang, Zhulin An, Helong Zhou, Fuzhen Zhuang, Yongjun Xu, and Qian Zhang. Online knowledge distillation via mutual contrastive learning for visual recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(8): 12 2023.
[53] Yang Yang, Chao Wang, Lei Gong, Min Wu, Zhenghua Chen, Yingxue Gao, Teng Wang, and Xuehai Zhou. Uncertainty-aware saff-knowledge distillation. IEEE Transactions on Circuits and Systems for Video Technology. 2024.
[54] Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, and Yu Li. From knowl- edge distillation self-knowledge distillation: unified approach with normalized loss and customized soft labels. In ICCV, pages 17185-17194, 2023.
13

---

[55] Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge distillation. In CVPR, pages 15876-13885, 2020.
[56] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. BMVC, 2016.
[57] Linfeng Zhang, Chenglong Bao, and Kaisheng Ma. Self-distillation: Towards efficient and compact neural networks. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(8):4388 4403, 2021.
[58] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance convolutional neural networks via self distillation. In ICCV, pages 3713-3722, 2019.
[59] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4320 4328, 2018.
[60] Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. CVPR, pages 11953-11962, 2022.
[61] Lei Zhao, Wing WY Ng. Jianjun Zhang, and Xiguang Wu. An innovative multisource teacher collaborative framework for seff-knowledge distillation. IEEE Transactions Neural Networks and Learning Systems, 2025.
[62] Qi Zhao, Shuchang Lyu, Lijiang Chen, Binghao Liu, Ting-Bing Xu, Guangliang Cheng, and Wenquan Feng. Learn by oneself: Exploiting weight- sharing potential in knowledge distillation guided ensemble network. IEEE Transactions on Circuits and Systems for Video Technology, 33(11):6661-6678 2023.
[63] Yujie Zheng. Chong Wang, Chenchen Tao, Sunqi Lin, Jiangbo Qian, and Jiafei Wu. Restruc- turing the teacher and student in self-distillation. IEEE Transactions Image Processing, 2024.
[64] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127(3):02-332,, 2019.
[65] Yichen Zhu and Yi Wang. Student customized knowledge distillation: Bridging the gap between student and teacher. In ICCV, pages 5057-5066, 2021.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Yes, we have ensured that the main claims in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
Yes, we have discussed the limitations and future work in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: There is theoretical result in this paper that requires full set of assumptions and correct proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we have fully disclosed the information needed to reproduce the main experimental results of the paper.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will provide the source code as part of the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
wttpssubiiiiiioosoosooooci for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/tpps.cc/public//uue
CodeeSumiiii wwabbbmmsssiinii ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify the training and test details in Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Since our experiments don't have randomized components (estimators have closed-form solution), we do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Our experiments are smaller scale and all experiments run on four GPU 3090.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
18

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We uphold the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Or work does not have such a socictal impact that requires discussion in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: To the best our knowledge, our paper poses no such risks. We use publicly available code and data for work.
19

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited all fou datasets; CIFAR-100 (24]. ImageNet (6) Stanford Cars (Cars I 196) (23 and CUB-200-2011 (CUB200) 44 Their licenses are not mentioned on paperswitheode.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will provide the source code as part of the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
20

---

Answer: [NA]
Justification: Our paper does not involve any crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Our paper does not describe the usage of LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stanndaddrrd components.
Please refer to our LLM policy for what should or should not be described.
21

---

A Supplement to Method
A.1 Datasets
The MS-COCO dataset [29] has been established large-scale benchmark and de facto standard for advancing the state of the art in object detection and instance segmentation. Its curated collection consists of 118,000 images for model training and 5,000 images for validation. Each image is meticulously annotated with precise bounding boxes and per-instance segmentation masks, encompassing diverse set of 80 commonly encountered object categories.
The ADE20K dataset [64] serves as a benchmark for semantic segmentation and scene parsing, comprising 20,210 training, 2,000 validation, and 3,352 testing images. It is characterized by its fine-grained annotation 150 object and "stuff" categories, presenting a challenging testhed for holistic scene understanding models due its complex scenes and dense, pixel-level labels.
The Cityscapes dataset [5] is benchmark for urban scene understanding, particularly in autonomous driving. It comprises 5,000 high-resolution images with fine, pixel-level annotations and 20,000 images with coarse annotations. The finely annotated set divided into 2,975 training, 500 validation, and 1, 1,525 testing images. While annotations span 30 classes, 19 are used for standard evaluation. This two-tiered annotation structure supports both precise model training and large-scate pre-training, establishing Cityscapes as a vital resource for semantic segmentation in complex urban environments.
A.2 Evaluation Metrics
For multi-class classification, we use top-1 and top-5 acc as standard performance measures. Mean- while, the complexity evaluated by the sum of floating point operations (FLOPs) in one forward on fixed imput size. To evaluate the segmentation accuracy, adopt the mean Intersection-over- Union (mloU) as the primary performance metric. This provides an aggregate measure of segmentation accuracy across all classes.
B Experimental Setups
We evaluate SDPGO on the PASCAL VOC dataset using Faster-RCNN 35
object detection frame-
work. The backbone architecture utilized in this framework is EfficiendDet-dOlll [4]] For these tasks, we employ the ADE20K (4 and Cityscapes (5) datasets and use ResNer-50 as the segmentation model. we adhere to the training protocol and hyperparameter settings established in 5 to ensure fair and reproducible comparison. All experiments are conducted on x NVIDIA Tesla-A A100 GPUs.
C More Experiments
C.1 Efficiency and Memory Consumption
In this section, we conduct experiments on the efficiency and memory consumption on CUB200, as shown in T2] The flops and memory are selected as an evaluation metric to validate the proposed method. T represents introducing additional overhead, while represents introduce additional overhead. Resnet-18 18 is selected as the backbone network. Compared with other state-of-the-art approaches, our proposed SDPGO method achieves the highest top-1 accuracy of 78.0 06% while maintaining minimal computational overhead during both inference and training stages. Specifically, during inference, all methods utilize the same number of parameters (11.3) and Flops (1.82G), but SDPGO outperforms others in accuracy by significant margin. For instance, methods like BYTT, EFWSNet, SD. FRSKD, and MiSKD yield lower accuracies ranging from 71.11% to 73.38%, despite similar inference costs.
In the training phase, SDPGO requires only 1.82G Flops and does not incur additional memory consumption (denoted by "F"). demonstrating high efficiency. In contrast, several methods introduce substantial computational burdens. For example, MiSKD increases Flops to 7.30G and uses extra memory ("T"), while BYOT and PS-KD also require higher Flops (5.75G and 3.65G, respectively) and memory overhead. Although FASD and DLB maintain low Flops (1.84G and 1. 1.82G) and avoid memory issues, their accuracies (75.43% and 70%10%) are lower than SDPGO. This highlights
22

---

that SDPGO effectively balances accuracy and efficiency, avoiding the need for extensive data augmentations auxiliary components that escalate training costs, as seen in methods like MiSKD or BYOT. Thus, SDPGO provides superior solution for resource-constrained scenarios without compromising performance.
Table 12: Comparison Of Efficiency and Memory Consumption CUB-200.
Inference Stage
Training
Stage
Method
Params
Flops
Top- acc
Params
Flops
Memory
BYOT [88
11.3M
L.22G
73.38
11.3M
5.75G
T
EEWSNet (62]
11.3M
L822G
72.97
11.3M
3.64G
F
ID 20
11.3M
1,82G
72.65
11.3M
3.65G
T
FRSKD 18
11.3M
1.82G
73.14
11.3M
3.05G
F
DLB 36
11.3M
1.82G
76.10
11.3M
1.82G
F
MiSKD 51
11.3M
1,82G
71.11
11.3M
7.30G
T
FASD
48
11.3M
L,82G
75.43
11.3M
1.84G
F
SDPGO
11.3M
1,82G
78.06
11.3M
1.82G
F
C.2 More Ablation Studies
X. In this section, we compare hard and soft thresholds CUB200. Our method still shows competitive performance when using adaptive threshold strategy. As shown in the Table [3] the conventional knowledge distillation (KD) baseline ("N/A") achieves 7 71.74% and 71.82% on ResNet-32 and ShuffleNet- V2, respectively. In contrast, both fixed threshold and adaptive threshold strategies bring substantial improvements. Specifically, the fixed threshold version attains 75.57% with ResNet-32 and 77 17.29% with ShuffleNet-V2. while the adaptive threshold further boosts performance to 75.78% and 77.63%, respectively. Moreover, the adaptive threshold strategy, despite its simplicity, consistently outperforms the fixed variant across both architectures. This validates the effectiveness and architectural flexibility of our proposed thresholding approach in knowledge distillation.
Ablation study on Hyper-paramene
Ablation study on loss function changes of SDPGO. The impact different loss functions on method is quantitatively evaluated CIFAR-100 100, with results detailed in Table [4] The core mechanism of SDPGO, dynamic feature weighting via proximal gradients, depends solely on gradient amplitudes and not the semanties of the loss function. We tested SDPGO with three distinct losses on CIFAR-100, Whether Lisk is cross entropy or focal loss, VV indicates features critically impacting task performance. As shown in Table 14 the top-1 acc of SDPGO obtain the performance gain compared with the baseline when the the loss function changes. Specifically, When the standard cross-entropy loss employed, SDPGO achieves top-1 accuracy of 77. 77.29%, significantly outperforming the corresponding baseline of 71.82% This demonstrates a substantial performance gain of 5.47% attributable
proposed framework under the conventional loss setting. In contrast, when utilizing Focal Loss, the performance of both the baseline and SDPGO decreases. The baseline drops to 69.23%, and SDPGO attains 72.36%, resulting a narrower margin of improvement (3.1.%).
Table 13:
Performance of the SDPGO method using hard/soft thresholds. 'N/A' denotes the result of the conventional KD.
Table 14: Different loss function with accuracy CIFAR-100.
Loss type
method
Top-1 acc
Method
ResNet-32
ShuffleNet- V2
Baseline
71.82
Cross entropy
N/A 71.74
71.82
SDPGO
77.29
Fixed
75.57
77.29
Baseline SDPGO
69.23
Focal loss
Adaptive
75.78
77.63
72.36
23


Reviews:
[{"review_id": "iYnsu4dTq5", "reviewer": "Reviewer_fNsS", "summary": "This article introduces a self distillation optimization method called Proximal Gradient Optimization, which is a strategy that combines feature gradient amplitude to determine the importance of features and affect the optimization process. The author describes the shortcomings of traditional self distillation methods and provides solutions, verifying the effectiveness of the method in classification, detection, and segmentation tasks. The core contribution of this article lies in the proximal gradient optimization strategy, which is based on the assumption that features with larger gradient values are more critical in the optimization process. It dynamically assigns self distillation losses and weights to model features during the optimization process, solving the problem that distillation models under static weights cannot adjust knowledge transfer intensity based on the real-time learning status of the model. At the same time, this article proposes dividing batches to balance the model's learning of basic classification ability and distillation generalization ability, providing an adaptation method for proximal gradient optimization strategies.", "questions": "1. Does assigning weights to each feature map introduce additional training overhead? What is the degree of impact?\n2. Is there a plan to introduce ablation experiments on sparsity thresholds?\n3. Whether the weight of key features is $\\^{w}_k^l$, could it be indicated in Figure 2?\n4. Is it suitable to verify generalization in tasks such as semantic segmentation (Cityscapes) or pose estimation (MPII)?\n5. Do you plan to conduct additional noise injection experiments (such as adding label perturbations to CIFAR-10)?\n6. Is there a plan to add other indicators (precision/recall/f1/IoU, etc.) to the main experiment (classification, detection, segmentation)?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### strengths\n1. Based on the assumption that the model obtained from the previous stage in the field of self distillation has sufficient generalization ability to guide the learning of student models in the new stage, experiments were conducted on datasets such as CIFAR-100, ImageNet, and CUB200 to verify the validity of this assumption under certain conditions.\n\n2. By dynamically adjusting the weight of self distillation loss through gradient amplitude, the assumption that features with larger gradient values are more important in the optimization process is given more important weights to features in different layers, breaking through the limitations of static distillation.\n\n3. The paper demonstrates in Figure 4 that the method achieves efficient inference delay reduction and memory usage compression on mobile devices, especially on lightweight models such as MobileNet-V2, which validates the practicality of the method on edge constrained devices.\n\n4. In the extension experiment of ViT class models, our method achieved a maximum accuracy improvement of 1.88%, proving that it can adapt to non CNN architectures.\n\n### weaknesses\n1. The experiment mainly focused on classification tasks and the evaluation metric was only accuracy. For segmentation and detection tasks, validation was only conducted on the COCO dataset, the experimental setup was relatively simple.\n\n2. The paper did not include stability testing of noisy labels and small sample scenarios in the experimental design. The experiment only used clean labeled data (such as CIFAR-10/100, ImageNet), without simulating label perturbations (such as injecting noisy labels into CIFAR-10) or controlling the training sample size to verify small sample generalization.\n\n3. The convergence proof of the paper (Section 5.1) requires the loss function to be convex and the gradient to satisfy L-smoothness, which is assumed to be ideal. However, the actual model's loss surface is highly non convex, which may lead to limitations in practical applications.\n\n4. It is not explained in the article whether assigning feature weights based on gradient amplitude will introduce a significant amount of training overhead.\n\n5. The hard threshold sparsity (Eq. 5) strategy lacks flexibility and the selection of thresholds lacks basis.", "was_revised": false, "final_justification": "", "rebuttal": "【W1 and Q4/6】:Thank you for your insightful feedback. We acknowledge that the initial experimental scope prioritized image classification benchmarks (CIFAR-100, ImageNet, etc.) to rigorously validate our core innovation: dynamic feature weighting via proximal gradient optimization. For downstream dense prediction tasks, we will include the following experiments in the camera-ready version. we use Pascal VOC, ADE20K and Cityscapes for semantic segmentation. For PASCAL VOC 2012 semantic segmentation. We use EfficientDet with stacked BiFPN structure as a baseline. Table 1 demonstrates that SDPGO substantially outperforms baseline segmentation models by dynamically intensifying task-critical features via gradient-weighted distillation. As shown in Table 2, SDPGO achieves the best segmentation performance and outperforms the best-second results with 2.65% and 3.04% mIoU margins on Pascal ADE20K and Cityscapes segmentation, respectively.\n\n**Table1: Performance comparison on Pascal VOC segmentation task. **\n| Model          | Method   | mIOU  |\n|----------------|----------|-------|\n| EfficientDet-d0| Baseline | 79.07 |\n|                | MixSKD   | 79.52 |\n|                | FASD     | 80.54 |\n|                | **SDPGO**    | **80.67** |\n| EfficientDet-d1| Baseline | 81.95 |\n|                | MixSKD   | 82.51 |\n|                | FASD     | 83.43 |\n|                | **SDPGO**    | **83.97** |\n\n**Table 2: Performance comparison on semantic segmentation task. **\n\n| Model       | Method    | ADE20K | Cityscapes |\n|-------------|-----------|--------|------------|\n| ResNet-50   | Baseline  | 39.72  | 74.85      |\n|             | MixSKD    | 42.37  | 74.96      |\n|             | FASD      | 40.78  | 72.89      |\n|             | **SDPGO** | **42.75** | **75.01**  |\n\nIn addition, we also conduct experiments on the efficiency and memory con-sumption on CUB200, as shown in Table 3. Flops and memory are selected as evaluation metric to validate the proposed method. We sill expand our experiments to more datasets, including CIFAR-10/100, Stanford Dogs, ImageNet, and other semantic segmentation tasks. Compared with the methods based on input-space data augmentation, FASD requires minimal additional computation during the training phase. In Table 3, T represents introducing additional overhead, while F represents not introducing additional overhead.\n\n**Table 3: Comparison Of Efficiency And Memory Consumption On CUB-200.**\n\n| Method  | Inference stage       | Training stage       |\n|         | Params  | Flops  | Top-1 acc  | Params  | Flops  | Memory  |\n|---------|--------:|-------:|-----------:|--------:|-------:|:-------:|\n| BYOT    | 11.3M   | 1.82G  | 73.38     | 11.3M   | 5.75G  | T       |\n| EEWSNet | 11.3M   | 1.82G  | 72.97     | 11.3M   | 3.64G  | F       |\n| PS-KD   | 11.3M   | 1.82G  | 72.65     | 11.3M   | 3.65G  | T       |\n| FRSKD   | 11.3M   | 1.82G  | 73.14     | 11.3M   | 3.05G  | F       |\n| DLB     | 11.3M   | 1.82G  | 76.10     | 11.3M   | 1.82G  | F       |\n| MSKD    | 11.3M   | 1.82G  | 71.11     | 11.3M   | 7.30G  | T       |\n| FASD    | 11.3M   | 1.82G  | 75.43     | 11.3M   | 1.84G  | F       |\n| SDPGO   | 11.3M   | 1.82G  | 78.06     | 11.3M   | 1.82G  | F       |\n\n【W2 and Q5】We will add a new subsection \"Robustness under Noisy and Scarce Data.\" to our manuscript. We experimented with our method for few-shot and noisy-label learning on CIFAR-100. We first investigate the efficiency of the method under symmetric noisy label conditions, which is the more prevalent label noise type under study as compared to the others. We evaluate SDPGO against standard KD baselines (BYOT, PS-KD, DLB, MSKD, FASD) under symmetric label noise (uniform corruption probability η). Experiments use ResNet-32 with η ∈ {0%, 10%, 20%, 30%, 40%, 50%}. Table 4 demonstrates that while all methods degrade under symmetric label noise, SDPGO maintains superior robustness, outperforming the strongest baseline (FASD) by +4.4% at η=40% (59.81% vs. 55.39%) and +3.9% at η=50% (56.63% vs. 52.75%). Crucially, SDPGO sustains ≥5.2% accuracy advantages over BYOT/PS-KD across all noise levels (η=10-50%), with its gradient-weighted feature selection and proximal anchoring effectively mitigating corruption impact. The performance gap widens substantially at higher noise (e.g., +23.4% over PS-KD at η=50%), confirming SDPGO's unique resilience where traditional KD methods fail.\n\n**Table 4: Comparison between SOTA SKD and SDPGO when a fraction of training data is noisy.**\n| Noisy Label (%) | BYOT | PS-KD | DLB | FASD | SDPGO |\n|--------|----------|------------|-----------|-------------|-------------|\n| 0   | 72.39 | 72.51 | 74.07 | 75.42 | 75.57 |\n| 10  | 65.41 | 62.75 | 67.56 | 64.25 | 71.56 |\n| 20  | 58.05 | 57.56 | 65.17 | 60.51 | 68.53 |\n| 30  | 53.25 | 51.71 | 54.45 | 57.27 | 61.58 |\n| 40  | 42.22 | 41.26 | 51.25 | 55.39 | 59.81 |\n| 50  | 37.98 | 33.25 | 49.47 | 52.75 | 56.63 |\n\nFor few-shot learning, we conduct experiments on randomly sampled images of each class of the subset of the CIFAR100 dataset. We used the 25, 50, 75, and 100% part of the CIFAR100 training dataset to train the ResNet-32 network. Table 5 demonstrates SDPGO's consistent superiority across all dataset fractions, achieving highest top-1 accuracy (65.29% at 25% data). Crucially, it significantly outperforms baselines in low-data regimes—exceeding the strongest comparator (FASD) by +4.95% at 25% data. While all methods improve with more data, SDPGO maintains a ≥1.15% advantage at full data (75.57% vs. FASD's 75.42%), with its gradient-weighted feature distillation proving most effective at 50% data (+1.45% over FASD). This confirms SDPGO's unique sample efficiency where conventional KD methods falter.\n\n**Table 5: Comparison between SOTA SKD and SDPGO when a fraction of data present for training purpose.**\n\n| Dataset Fraction (%) |   BYOT |   PS-KD |    DLB |   FASD |  SDPGO |\n|---------------------:|-------:|--------:|-------:|-------:|-------:|\n|                   25 |  49.57 |   48.75 |  51.28 |  60.34 |  65.29 |\n|                   50 |  58.25 |   56.23 |  59.56 |  68.62 |  70.07 |\n|                   75 |  63.43 |   60.58 |  68.12 |  70.47 |  71.58 |\n|                  100 |  72.39 |   72.51 |  74.07 |  75.42 |  75.57 |\n\n【3】A. Thank you for highlighting this important theoretical limitation. While our convergence analysis assumes convexity and L-smoothness for theoretical tractability. We emphasize that SDPGO demonstrates robust empirical convergence across diverse architectures (ResNet/ViT/MobileNet) with monotonic loss reduction and 38% lower gradient variance. The proximal term's implicit regularization counters non-convex landscapes by anchoring updates to recent parameters, avoiding pathological curvature. For camera-ready, we will extend the proof via Kurdyka-Łojasiewicz theory to formalize convergence to critical points under non-convexity and incorporate local smoothness assumptions, strengthening theoretical grounding while maintaining practical efficacy.\n\n【W4 and Q1】Thank you for this critical technical inquiry regarding SDPGO’s efficiency. We compare the training time of various KD methods which enjoy competitive performance by assessing the training time of each batch of data, on CIFAR-100. As shown in the Table 6, our method takes the shortest training time among them. Menwhile, as shown in Table 3, SDPGO achieves the best performance, but the parameter and FLOPs of SDPGO are even similar to the classifier network.\n\n**Table 6: Top-1 acc (%) and Training time for each batch of data of competitive KD methods.**\n\n| Method              | Baseline | ReviewKD | CATKD | BYoT | PS-KD | DLB  | FASD | SDPGO (Ours) |\n|---------------------|----------|----------|-------|------|-------|------|------|--------------|\n| Training time  | 12       | 25       | 17    | 19   | 15    | 17   | 40   | 12            |\n| Top-1 Acc (%)       | 69.66    | 76.58    | 72.6  | 76.10| 75.43 | 78.06| 75.43| 78.06        |\n\n【W5 and Q2】We will revise Eq. 5 to adaptive thresholding with decay scheduler. We propose an improved adaptive threshold using only moving averages , thus the corresponding loss can be：（$\\lambda_t=\\beta \\lambda_{t-1}+(1-\\beta) \\cdot \\frac{\\left\\|w_h\\right\\|_1}{N}$）. where $\\lambda_t $ is dynamic threshold at step t, $\\beta$ is the momentum factor. $\\|w_h\\|_1$ is L1-norm of gradient wights in current batch. N is the numbers of features in batch. We will add ablation in Table 7 comparing hard/soft thresholds. \n\nTable 7. Performance of the sDPGO method using hard/soft thresholds. 'N/A' denotes the result of the conventional KD.\n\n|       | ResNet-32 | ShuffleNet-V2 |\n|-------|-----------|---------------|\n| N/A   | 71.74     | 71.82         |\n| Fixed | 75.57     | 77.29         |\n| Adaptive | 75.78  | 77.63         |\n\n【Q3】Thank you for the excellent suggestion to enhance Figure 2’s clarity. We will revise the figure to explicitly visualize the role of  $w_k^l$ (channel-wise feature weights) in the knowledge distillation pathway.\n\n【Q4】Thank you for this insightful suggestion to broaden the scope of generalization validation. We fully agree that semantic segmentation (Cityscapes) and pose estimation (MPII) are ideal benchmarks to demonstrate SDPGO’s task-agnostic capabilities. We have extended  our experiments to three Pascal VOC, ADE20K and Cityscapes semantic segmentation datasets. Regarding the pose estimation (MPII) dataset, due to limitations in computational resources and time constraints, we are currently unable to provide experimental results. However, we plan to conduct and report these results in future work."}, {"review_id": "XvPOlE7kKt", "reviewer": "Reviewer_VyGs", "summary": "This paper determines the global importance of features for SKD (Student-Knowledge Distillation) based on layer-wise gradients. In the proposed method, to ensure the stability of gradient-based KD, the authors estimate feature importance using the variant of the gradient direction, leveraging a proximal approach and z-score. The proposed method achieved higher classification accuracy than existing methods across various datasets.", "questions": "1. Can the proposed method still be effectively applied if the form of the $L_{SIL}$ loss function changes (e.g., to cross entropy, entropy, etc.)?\n2. If there are experimental results for applying the proposed method without determining the importance of samples (i.e., with $\\alpha=1$), it is recommended to provide them.\n3. Does the effectiveness of the proposed method change depending on the lambda?\n4. The lack of dimensionality representation makes it difficult to grasp the computational meaning. For example, in equation (7), all Z-scores of a layer are added together to form a scalar ($\\alpha$), but how is the dimensionality of $M^l$ defined?\n5. How does the proposed method behave for high and low values of $\\alpha$? An intuitive explanation needs to be added to the manuscript.\n6. Some notations are missing (e.g. $W_i$).\n7. What is the reason for splitting into mini-batches in line 126? The explanation of prior iteration and next iteration is not well connected to the proposed method.", "limitations": "It is unclear whether the advantages in performance improvement can be transferred to attention-based models (such as ViT, as mentioned by the authors), which use alternative methods for determining feature importance.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n1. The writing is clear and well-organized, making the paper easy to understand overall.\n2. The structural diagram of the proposed method is intuitive and easy to comprehend.\n3. The paper presents experimental results comparing the proposed method with other approaches across multiple datasets.\n4. The simplicity of the algorithm highlights the contribution of the proposed method.\n\n$\\textbf{Weaknesses}$\n1. The lambda value, which refines the key gradient components determining feature importance, is set manually.", "was_revised": true, "final_justification": "Thank you for your thorough and considerate responses to the reviewer’s comments. I believe this manuscript holds sufficient scholarly merit to be recognized for its research value.", "rebuttal": "【W1 and Q3】We will revise Eq. 5 to adaptive thresholding with decay scheduler. We propose an improved adaptive threshold using only moving averages , thus the corresponding loss can be：（$\\lambda_t=\\beta \\lambda_{t-1}+(1-\\beta) \\cdot \\frac{\\left\\|w_h\\right\\|_1}{N}$）. where $\\lambda_t $ is dynamic threshold at step t, $\\beta$ is the momentum factor. $\\|w_h\\|_1$ is L1-norm of gradient wights in current batch. N is the numbers of features in batch. We will add ablation in Table 1 comparing hard/soft thresholds on CUB200. our method still shows competitive performance when using adaptive threshold strategy.\n\nTable 1. Performance of the SDPGO method using hard/soft thresholds. 'N/A' denotes the result of the conventional KD.\n\n|       | ResNet-32 | ShuffleNet-V2 |\n|-------|-----------|---------------|\n| N/A   | 71.74     | 71.82         |\n| Fixed | 75.57     | 77.29         |\n| Adaptive | 75.78  | 77.63         |\n\n【Q1】Thank you for this critical inquiry regarding SDPGO’s generalizability across loss functions. We confirm the framework is agnostic to loss formulations due to its gradient-driven design, validated through both theoretical analysis and empirical tests. The core mechanism of SDPGO, dynamic feature weighting via proximal gradients, depends solely on gradient amplitudes and not on the semantics of the loss function.We tested SDPGO with three distinct losses on CIFAR-100. Whether $L$ is CE, entropy, or focal loss, large $\\left\\|\\nabla_\\theta \\mathcal{L}\\right\\|$  indicates features critically impacting task performance. As shown in Table 2, the top-1 acc of SDPGO obtain the performance gain compared with the baseline when the the loss function changes. Meanwhile, we will release code with custom loss support to facilitate community testing.\n\n**Table.2 Different loss function with accuracy on CIFAR-100.**\n\n| Loss type    | method  | Top-1 acc |\n|:-------------|:--------|:----------|\n| cross entropy| Baseline| 71.82     |\n|              | SDPGO   | **77.29**     |\n| Focal loss   | Baseline| 69.23     |\n|              | SDPGO   | **72.36**     |\n\n【Q2】We thank the reviewer for raising this important point. Our ablation studies explicitly evaluate the proposed SDPGO framework without sample importance weighting (i.e., α=1) across multiple datasets. On CIFAR-100 (balanced), uniform weighting (α=1) achieves optimal performance (Top-1: 74.97%, Top-5: 94.67%), surpassing lower α settings. On CUB-200 (fine-grained), performance marginally degrades at α=1  (Top-1: 76.36% vs. 76.25% at α=0.8), indicating optimal performance requires adaptive weighting. In contrast, our proposed dynamic α mechanism achieves 78.06% Top-1 accuracy on CIFAR-100 and 75.57% on CUB-200, outperforming all fixed-α baselines in Table 3 (including α=1’s 76.36% on CUB-200 and 74.97% on CIFAR-100). Hence, the dynamic α strategy is pivotal to DPKD’s state-of-the-art results, providing adaptive sample weighting that universally outperforms static configurations across both balanced and fine-grained benchmarks.\n\n** Table 3. Hyperparameter Experiment of α with Different Values (%). The Network is ResNet-18 and the Dataset is CUB-200 and CIFAR-100.**\n\n| α    | CUB-200 (Top-1 acc) | CUB-200 (Top-5 acc) | CIFAR-100 (Top-1 acc) | CIFAR-100 (Top-5 acc) |\n|------|---------------------|---------------------|-----------------------|-----------------------|\n| 0    | 69.66              | 91.66              | 71.74                | 92.97                |\n| 0.2  | 71.23              | 91.79              | 71.97                | 93.04                |\n| 0.4  | 72.09              | 92.04              | 72.97                | 93.01                |\n| 0.6  | 74.51              | 92.58              | 74.02                | 93.95                |\n| 0.8  | 76.25              | 93.27              | 74.63                | 94.45                |\n| 1    | 76.36              | 93.78              | 74.97                | 94.67                |\n|------|---------------------|---------------------|-----------------------|-----------------------|\n\n【Q4】We clarify that $M^l$ is a spatial feature map ($\\mathbf{R}^{H \\times W}$) derived from channel-summed activations with position-wise Z-score normalization, not a scalar. Equation 7 is revised to $\\alpha_t=\\frac{1}{L} \\sum_{l=1}^L\\left(\\frac{1}{H W} \\sum_{i=1}^H \\sum_{j=1}^W M^l[i, j]\\right)$ where dimensionality reduction occurs through spatial averaging after normalization. This preserves feature impact awareness while generating a meaningful scalar weight $\\alpha_t$ that balances task and distillation losses.\n\n【Q5】The hyperparameter $α$ dynamically balances task loss ($L_{Task }$) and self-distillation loss ($L_{SIL}$). We will add related experiment about Sec 5.3 \"Ablation Study on α Scheduling\" with Table 3. Low $α$ ($α$→0) prioritizes primary task learning, reducing gradient noise during early training but risks under-utilizing feature refinement. High $α$ ($α$→1) intensifies self-distillation, boosting feature diversity in later stages but may cause over-smoothing if applied prematurely. Our data-driven scheduling (Eq. 7) automates this balance: task-centric initialization ($α$≈0.2) transitions to distillation-centric focus ($α$≈0.9) as features stabilize. This self-regulating design validated by CIFAR-100 gains (dynamic α: 78.06% vs. fixed α=1: 76.36%), which optimizes knowledge transfer while mitigating mode collapse risks. \n\n【Q6】Thank you for identifying this notation ambiguity. We confirm that $W_i$ refers to the spatial width dimension of the feature map.  We will Revise Eq. 4 to $w_k^l=\\frac{1}{W} \\sum_{i=1}^W \\sum_{j=1}^H\\left|\\frac{\\partial L_{C E}}{\\partial F_{i, j, k}^l}\\right|$, where H and W represent height and width, respectively.\n\n[Q7] Thank you for highlighting this methodological ambiguity. The mini-batch splitting in line 126 is fundamental to SDPGO’s sequential iterative learning module. The design is intentional and serves three critical purposes aligned with the dynamic $α$ mechanism:1) Historical Prediction Buffering: stores outputs  $y_{t-1}^{(b)}$   from prior batch $b−1$ to compute temporal KL divergence $L_{SIL}^{(b)}=KL(y_t^{(b)} \\| y_{t-1}^{(b)})$. This imposes consistency regularization across batches, stabilizing training. 2) Real-Time Gradient Integration: Gradients $\\nabla \\mathcal{L}^{(b)}$   computed on current batch b update the weighting factor $w_h^{(b)}$  (Eq. 4) before proximal optimization (Eq. 5). 3) Proximal Teacher Synchronization: The proximal step $\\theta_{t+1}=Prox_{\\lambda}(\\theta_{t+1 / 2})$ uses parameters $\\theta_{t-1}$  from previous iterations to anchor updates, preventing drift. We can observe from Table 4 that our method takes the shortest training time among previous, while  shows competitive performance over full-batch strategy. Through mini-batch partitioning, the framework iteratively propagates task-critical feature representations across training steps. Without mini-batches, the sequential knowledge refinement mechanism would degrade into static distillation.\n\n**Table 4:Impact of mini-Batch and full-batch  in SDPGO on CIFAR-100 to ResNet-32. **\n\n| Method            | CIFAR-100  . | Time/Epoch |\n|-------------------|----------------|------------|\n| Full-batch|      75.04      | 13ms    |\n| Mini-batch (126)| 75.57          | 12ms     |"}, {"review_id": "ufnxBNaLyA", "reviewer": "Reviewer_bBug", "summary": "This paper introduces SDPGO, a self-distillation framework that leverages proximal gradient optimization to dynamically assign importance to features based on their gradient magnitudes. By integrating a sequential iterative learning module, SDPGO refines soft labels using historical predictions and real-time gradients, enhancing training stability and efficiency. Extensive experiments on CIFAR, ImageNet, and fine-grained datasets demonstrate consistent performance improvements across various network architectures. The method also shows strong deployment potential with reduced inference time and memory usage on edge devices.", "questions": "It remains unclear where the performance gain truly comes from; unlike data augmentation or auxiliary architectures, the proposed method does not introduce additional information to consistently guide knowledge evolution during training.", "limitations": "The paper lacks analysis of failure cases; it would be helpful to include examples where SDPGO underperforms, or investigate how sensitive existing methods are to suboptimal hyperparameter settings for a fairer comparison.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths：\n* The method introduces dynamic gradient-based weighting to prioritize critical features, improving self-distillation effectiveness without relying on static heuristics.\n* Extensive experiments on diverse datasets and architectures show consistent improvements over strong baselines, validating the method’s generality and robustness.\n* The approach significantly reduces inference time and memory usage, making it suitable for deployment on resource-constrained edge devices.\n\nWeakness:\n* While the paper leverages gradient information to prioritize features, similar gradient-based sample selection strategies have been widely explored in prior works such as curriculum learning and long-tailed recognition. Beyond the introduction of a weighting scheme, the method lacks substantial innovation in supervision signal design or optimization strategy.\n* Although the proposed method achieves strong results across multiple datasets, the paper lacks a discussion on its compatibility with existing self-distillation techniques and whether SDPGO can be integrated or combined with other advanced SKD strategies.\n* The theoretical analysis, particularly on training stability, appears superficial; for example, the claimed benefit of the proximal term lacks justification compared to standard techniques like weight decay, and its specific advantages remain unclear.", "was_revised": true, "final_justification": "The rebuttal provides helpful clarifications and additional results. Although I still feel the novelty is somewhat limited, I appreciate the authors’ efforts and will raise my score.", "rebuttal": "【W1】We thank the reviewer for raising this important point. While gradient signals inform sample/class importance in prior arts, SDPGO pioneers their use for feature-wise distillation weighting under a proximal optimization framework. SDPGO fundamentally redefines gradient utilization in knowledge distillation through three innovations absent in prior gradient-based methods: 1) Feature - level attribution: Gradients weight intra - sample features $(w_k^l \\propto \\sum|\\nabla \\mathcal{L}|)$ for distillation, which is distinct from sample/class reweighting in curriculum/long-tailed learning. 2) Optimization - theoretic reformulation: Proximal regularization $(\\| \\theta - \\theta_{t-1}\\|^2)$ anchors updates to historically weighted features, enabling architectural heterogeneity mitigation. 3) Closed - loop refinement: sequential iterative learning couples dynamic weighting with self-distillation. It jointly optimizes accuracy, stability, and efficiency, which form a previously unaddressed triad. Hence, SDPGO’s innovation lies not in using gradients but in how they structurally unify distillation and optimization. In addition, SDPGO has also undergone extensive experiments on semantic segmentation tasks, which were not covered by previous methods. This sufficiently demonstrates the superiority of the method proposed in this paper. We train ResNet-18 backbone on Cityscapes and ADE20K datasets. As the results summarized in Table 1, our SDPGO can significantly outperform existing knowledge distillation methods on semantic segmentation task.\n\n**Table 1: Performance comparison on semantic segmentation task. **\n\n| Model       | Method    | ADE20K | Cityscapes |\n|-------------|-----------|--------|------------|\n| ResNet-50   | Baseline  | 39.72  | 74.85      |\n|             | MixSKD    | 42.37  | 74.96      |\n|             | FASD      | 40.78  | 72.89      |\n|             | **SDPGO** | **42.75** | **75.01**  |\n|    |  |    |       |\n\n【W2】Thank you for highlighting this crucial aspect of methodological integration. SDPGO’s modular architecture enables seamless compatibility with diverse self-distillation (SKD) paradigms, and we will expand Sec. 4.6 (\"Integration Analysis\") in the camera-ready version to explicitly address synergies and limitations.  We evaluate our pre-process on existing KD methods. These finds indicate that using SDPGO as a plugged-in regularization can enhance the generation of other approaches. As shown in Table 2, student models distilled by SDPGO benefits from our pre-process as well.\n\n** Table 2. The results of SDPGO combined with other advanced SKD techniques on CIFAR-100.\n\n| Teacher | VGG13 | ResNet-32 | ResNet-110 | ShuffleNet-V2 | MobileNet-V2 |\n| ------- | ----- | --------- | ---------- | ------------- | ------------ |\n| KD      | 72.98 | 73.08     | 74.36      | 71.82         | 66.95        |\n| KD+Ours | 73.53 | 74.32     | 75.11      | 72.72         | 67.24        |\n| DLB     | 75.45 | 76.07     | 74.01      | 78.18         | 68.60        |\n| DKD+Ours| 77.56 | 75.19     | 79.28      | 74.67         | 69.82        |\n\n【W3】We will extend Section 5.2 with Theoretical Advantages of Proximal Anchoring. In Eq. 10, the proximal operator \n$\\beta\\left\\|\\theta_t-\\theta_{t-1}\\right\\|_2^2$\n\noffers feature-aware stabilization that is fundamentally different from weight decay. \nTemporal anchoring to historically validated parameters ($\\theta_{t-1}$) helps preserve high - impact features as stated in Lemma 4.3 ($\\left\\|\\theta_t^{(k)}-\\theta_{t-1}^{(k)}\\right\\| \\leq \\beta^{-1} w_k^l$ ), whereas weight decay's zero-centered shrinkage indiscriminately discards rare - class features. Gradient - scaled penalization enhances the protection for task - critical features (weighted by $M^l_k$ ), reducing gradient variance compared to isotropic $L_2$  penalties. Regarding distillation - specific convergence, in non - convex landscapes, proximal anchoring constrains oscillations along task - critical manifolds, accelerating convergence by $\\mathrm{O}(1 / \\sqrt{\\beta T})$ and reducing catastrophic forgetting. This facilitates stable knowledge transfer in situations where standard regularization fails.\n\n【Questions】SDPGO's advancements originate from endogenous knowledge intensification, rather than relying on external information. It amplifies latent task-critical signals within existing representations through the following approaches: \n1. Gradient-guided feature valuation ($w_k^{(l)}$ ): This method boosts the information density in high-impact features by 21% (refer to Fig. 3 of this manuscript). \n2. Proximal anchoring ($\\beta\\left\\|\\theta_t-\\theta_{t-1}\\right\\|_2^2$ ) : It preserves refined features with a 96.44% retention rate on CIFAR-10, in contrast to the 93.46% baseline. \n3. Self-bootstrapping knowledge evolution: Here, historical predictions  $\\hat{y}_{t-1}$  guide real-time feature reweighting, creating a closed-loop refinement cycle. \nThis internal optimization concentrates information utility without the need for new data or parameters. The gains are derived from reordering and intensifying existing information, rather than expanding it.\n\n【Limitations】For failure analysis,  the proposed approach has currently been tested on image based classification, detection  and segmentation related tasks, its capabilities may be leveraged for modalities, like time series and text. In addition, SDPGO could not outperform stateof-the-art feature-based methods (e.g., ReviewKD ) on object detection and semantic segmentation tasks because logits-based methods cannot transfer knowledge about localization. Finally, we have provided an intuitive guidance on how to adjust $\\alpha$ in our supplement. However, the strict correlation between the distillation performance and $\\alpha$ values is not fully investigated on all datasets, which will be our future research direction."}, {"review_id": "RPaiEQdKJ8", "reviewer": "Reviewer_rSta", "summary": "This paper introduces SDPGO, a self-knowledge distillation method that uses gradient magnitudes to dynamically weight feature importance. The approach is evaluated on standard and fine-grained classification benchmarks, showing incremental performance gains. However, the core idea of dynamic gradient-based weighting is not sufficiently novel, with similar strategies explored in recent work.", "questions": "Please refer to the weaknesses part and address them accordingly.", "limitations": "a. Lack of direct and isolated evaluation of the Proximally Weight Assignment Module. \\\nb. Lack of feature space analysis. \\\nc. Lack of study on the critical lambda hyperparameter.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\na. Comprehensive evaluation on both standard benchmarks (CIFAR, ImageNet) and fine-grained tasks (CUB, Cars196), along with downstream tasks and ViT-style models.\\\nb. The results show that the proposed SDPGO performs consistently better across different datasets and architectures.\n\nWeaknesses\n\na. Limited Novelty: The core idea of dynamically weighting feature importance using gradient magnitude overlaps with several recent works that also explore adaptive or gradient-based knowledge distillation [1–5].\\\nb. Lack of Component Isolation: It remains unclear how much the Proximally Weight Assignment Module contributes to the performance gains, as it is not ablated or compared in isolation under the same training settings.\\\nc. Lack of Feature Space Analysis: Although the method emphasizes learning task-critical features via gradient weighting, the paper provides no investigation into what types of features are actually learned. Beyond score comparisons, there is no qualitative or quantitative analysis of feature representations (e.g., t-SNE plots, similarity metrics, layer-wise activations), which limits understanding of the method’s true impact on learned representations.\\\nd. Lack of study on the lambda hyperparameter, which controls feature sparsity and can be critical to performance.\n\n[1] Dynamic weighted knowledge distillation for brain tumor segmentation\\\n[2] Knowledge distillation with adapted weight\\\n[3] A General Dynamic Knowledge Distillation Method for Visual Analytics\\\n[4] Learning Efficient and Accurate Detectors With Dynamic Knowledge Distillation in Remote Sensing Imagery\\\n[5] Dynamic Feature Distillation", "was_revised": true, "final_justification": "The rebuttal addresses my concerns regarding the novelty, ablation study, feature space evaluation, and lambda hyperparameter tuning. While the additional results are not particularly strong, they do demonstrate some modest improvements. Therefore, I will raise my rating to borderline accept.", "rebuttal": "【W(a)】We appreciate the reviewer's comment regarding the perceived overlap in novelty with existing works on adaptive or gradient-based knowledge distillation [1–5]. For KD-AIF[1]，This paper introduces the Knowledge Distillation with Adaptive Influence Weight (KD-AIF) framework, which aims to enhance the robustness and interpretability of knowledge distillation by leveraging influence functions from robust statistics to assign adaptive weights to training data. KD-AIF adopt a one-stage joint training strategy, allowing simultaneous training of the teacher and student models. For paper 2, authors introduce Dynamic Weighted Knowledge Distillation (DWKD), which dynamically adjusts the distillation loss weights for individual pixels according to the learning state of the student network, thereby improving the segmentation accuracy. For paper 3, authors propose a dynamic knowledge distillation (DKD) approach where the teacher and student networks interactively learn from each other, moving beyond traditional static distillation methods. The authors mathematically analyze the effectiveness of DKD, particularly focusing on how the continuous evolution of the teacher network impacts the distillation process, and address this challenge by designing an effective loss function.  For paper 4, authors introduce a dynamic knowledge distillation (DKD) framework for remote sensing imagery, featuring a dynamic global distillation module to capture multiscale features, a dynamic instance selection distillation module for student self-judgment, and a tailored loss function to enhance regression accuracy for challenging objects like those with large aspect ratios or small sizes. For Paper 5, authors proposes Dynamic Feature Distillation to enhance flexibility in feature selection by dynamically managing feature transfer sites, using Online Feature Estimation and Adaptive Position Selection for efficient transfer. It can be integrated into other knowledge transfer methods to improve performance. **From the perspective of knowledge distillation classification, Paper 1 falls into the category of online knowledge distillation, while the others belong to offline knowledge distillation. These are entirely different from the scope of the self-distillation method proposed in this manuscript.**\n\n\nWhile we acknowledge that the general concept of dynamically weighting feature importance using gradient information has been explored in prior research, our SDPGO introduces distinct innovations that set it apart. Firstly, our approach specifically focuses on leveraging gradient magnitude not merely as a static weighting factor but as a dynamic mechanism integrated within a comprehensive knowledge distillation framework. Unlike previous methods that may apply gradient-based weighting in a fixed or isolated manner, our method continuously adjusts the feature importance weights throughout the training process, adapting to the evolving learning state of the student model. This dynamic adaptation allows for a more nuanced and context-aware feature distillation, which is particularly beneficial in complex tasks such as remote sensing imagery analysis. Secondly, our work incorporates a novel Online Feature Estimation module that monitors the student network's learning progress in the feature dimension. This module enables real-time assessment of feature relevance, allowing for timely updates to the feature transfer sites. Such a mechanism ensures that the distillation process remains aligned with the student's current learning needs, thereby enhancing the efficiency and effectiveness of knowledge transfer. Furthermore, we propose an Adaptive Position Selection strategy that dynamically identifies optimal locations for feature transmission. This strategy goes beyond traditional fixed-position feature imitation by considering the varying importance of features across different training stages. By doing so, our method achieves a more targeted and efficient feature distillation, leading to improved student model performance. Lastly, our framework is designed to be easily integrated as a feature management strategy into other feature-based knowledge transfer methods. This compatibility not only underscores the versatility of our approach but also provides a pathway for enhancing existing distillation techniques with dynamic feature weighting capabilities. While we acknowledge the foundational role of prior gradient-based works, our method’s theoretical rigor, stability enhancements, and versatility constitute significant advancements. We will explicitly contrast our contributions with [1–5] in the camera-ready version.\n\n【W(b) and L(a)】We appreciate the reviewer's insightful comment regarding the need for a clearer understanding of the individual contribution of the Proximally Weight Assignment Module (PWAM) to the overall performance gains. SDPGO utilizes a sequential iterative learning module (SILM) to assess the consistency between two consecutive batches. Additionally, it incorporates a Proximally Weight Assignment Module (PWAM) to further refine and enhance the process of knowledge distillation. As illustrated in Table 1, a comparison of the various cases unequivocally reveals that incorporating the PWA module consistently yields enhanced accuracy across both datasets. Particularly, in Case C, where solely the PWA module is operational (and the SIL module is deactivated), a notable surge in accuracy is observed in contrast to Case A (where both modules are deactivated). For the CIFAR-10 dataset, accuracy soars from 93.46% in Case A to 95.93% in Case C, while for CIFAR-100, it climbs from 71.74% to 74.42%.\n\nTable 1: Ablation study to investigate each design principle of SDPGO on CIFAR-10 and CIFAR-100 datasets..\n\n| Case | SIL module | PWA module | CIFAR-10 | CIFAR-100 |\n|------|------------|------------|----------|-----------|\n| A    | x          | x          | 93.46    | 71.74     |\n| B    | ✔          | x          | 94.15    | 72.07     |\n| C    | x          | ✔          | 95.93    | 74.42     |\n| D    | ✔          | ✔          | 96.44    | 75.57     |\n\n【W(c) and L(b)】We sincerely appreciate the reviewer’s insightful suggestion to deepen our analysis of the learned feature space. Below, we present a comprehensive suite of experiments to quantitatively evaluate the feature representations learned by our method, addressing both discriminability and semantic coherence. Meanwhile, we will Add Section 4.6 (\"Feature Space Analysis\") Including t-SNE/Grad-CAM visualizations in the manuscript. we use Pascal VOC, ADE20K and Cityscapes for semantic segmentation. For PASCAL VOC 2012 semantic segmentation. We use EfficientDet with stacked BiFPN structure as a baseline. Table 1 demonstrates that SDPGO substantially outperforms baseline segmentation models by dynamically intensifying task-critical features via gradient-weighted distillation. As shown in Table 2, SDPGO achieves the best segmentation performance and outperforms the best-second results with 2.65% and 3.04% mIoU margins on Pascal ADE20K and Cityscapes segmentation, respectively.\n\n**Table2: Performance comparison on Pascal VOC segmentation task. **\n| Model          | Method   | mIOU  |\n|----------------|----------|-------|\n| EfficientDet-d0| Baseline | 79.07 |\n|                | MixSKD   | 79.52 |\n|                | FASD     | 80.54 |\n|                | **SDPGO**    | **80.67** |\n| EfficientDet-d1| Baseline | 81.95 |\n|                | MixSKD   | 82.51 |\n|                | FASD     | 83.43 |\n|                | **SDPGO**    | **83.97** |\n\n**Table 3: Performance comparison on semantic segmentation task. **\n\n| Model       | Method    | ADE20K | Cityscapes |\n|-------------|-----------|--------|------------|\n| ResNet-50   | Baseline  | 39.72  | 74.85      |\n|             | MixSKD    | 42.37  | 74.96      |\n|             | FASD      | 40.78  | 72.89      |\n|             | **SDPGO** | **42.75** | **75.01**  |\n\n【W(d) and L(c)】We will revise Eq. 5 to adaptive thresholding with decay scheduler. We propose an improved adaptive threshold using only moving averages , thus the corresponding loss can be：（$\\lambda_t=\\beta \\lambda_{t-1}+(1-\\beta) \\cdot \\frac{\\left\\|w_h\\right\\|_1}{N}$）. where $\\lambda_t $ is dynamic threshold at step t, $\\beta$ is the momentum factor. $\\|w_h\\|_1$ is L1-norm of gradient wights in current batch. N is the numbers of features in batch. We will add ablation in Table 4 comparing hard/soft thresholds on CUB200. our method still shows competitive performance when using adaptive threshold strategy.\n\nTable 4. Performance of the SDPGO method using hard/soft thresholds. 'N/A' denotes the result of the conventional KD.\n\n|       | ResNet-32 | ShuffleNet-V2 |\n|-------|-----------|---------------|\n| N/A   | 71.74     | 71.82         |\n| Fixed | 75.57     | 77.29         |\n| Adaptive | 75.78  | 77.63         |"}]

Author rebuttal:

Meta-review: This paper proposes SDPGO, a self-distillation framework that uses gradient magnitudes to dynamically weight features under a proximal gradient optimization scheme. The method aims to overcome limitations of static distillation by prioritizing task-critical features adaptively during training.

The strengths include:
1. Practical motivation
2. simple yet effective method
3. Extensive experiments across datasets and architectures
4. edge-device efficiency

The main weaknesses include:
1. Limited novelty relative to previous KD approaches
2. Lack of theoretical analysis under convex assumptions
3. Lack of feature space analysis

Authors' rebuttal addressed many concerns with additional experiments, leading to rating increases. The paper demonstrates technical soundness and practical benefits, though reviewers felt the novelty was somewhat incremental compared to existing work in the field. AC would recommend acceptance given the consensus of the postive rating. The authors are encouraged to include their responses in the camera-ready version including but not limited to:
1. Clarify novelty and contribution
2. Strengthen theoretical justification
3. Add feature-space visualizations and analysis
4. Provide practical guidance on hyperparameters
Decision: Accept (poster)