Decoupled Kullback-Leibler Divergence Loss
Authors: Jiequan Cui
Zhuotao Tian
Zhisheng Zhong
XIAOJUAN QI
Bei Yu
Hanwang Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Adversarial Training; Knowledge Distillation; Kullback-Leibler Divergence Loss; Long-tail Recognition
Primary area: machine_vision

Abstract: In this paper, we delve deeper into the Kullback–Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of 1) a weighted Mean Square Error ($\mathbf{w}$MSE) loss and 2) a Cross-Entropy loss incorporating soft labels. 
Thanks to the decomposed formulation of DKL loss, we have identified two areas for improvement. 
Firstly, we address the limitation of KL/DKL in scenarios like knowledge distillation by breaking its asymmetric optimization property. This modification ensures that the $\mathbf{w}$MSE component is always effective during training, providing extra constructive cues.
Secondly, we introduce class-wise global information into KL/DKL to mitigate bias from individual samples.
With these two enhancements, we derive the Improved Kullback–Leibler (IKL) Divergence loss and evaluate its effectiveness by conducting experiments on CIFAR-10/100 and ImageNet datasets, focusing on adversarial training, and knowledge distillation tasks. The proposed approach achieves new state-of-the-art adversarial robustness on the public leaderboard --- \textit{RobustBench} and competitive performance on knowledge distillation, demonstrating the substantial practical merits. Our code is available at https://github.com/jiequancui/DKL.

Decoupled Kullback-Leibler Divergence Loss
Jiequan Cui'
Zhuotao Tian*
Zhisheng Zhong²
Xiaojuan Qi Bei Yu?
Hanwang Zhang'
Nanyang Technological University*
The Chinese University of Hong Kong"
The University of Hong Kong
HIT(SZ)*
Abstract
In this paper, we delve deeper into the Bellback-Leibler (KL) Divergence loss and mathematically prove that it is equivalent to the Decoupled Kullback-Leibler (DKL) Divergence loss that consists of 1) weighted Mean Square Error (wMSE) loss and 2) a Cross-Entropyy loss incorporating soft labels. Thanks to the decomposed formulation of DKL loss, we have identified two areas for improvement. Firstly, we address the limitation of KL/DKL in scenarios like knowledge distillation by breaking its asymmetric optimization property. This modification ensures that the wMSE component is always effective during training, providing extra constructive cues. Secondly, introduce class-wise global information into KL/DKL to mitigate bias from individual samples. With these two enhancements, derive the Improved Kullback-Leible (IKL) Divergence loss and evaluate its effectiveness by conducting experiments on CIFAR-10/100 and ImageNet datasets, focusing on adversarial training, and knowledge distillation tasks. The proposed approach achieves new state-of-the-art adversarial robustness on the public leaderboard - RobustBench and competitive performance on knowledge distillation, demonstrating the substantial practical merits. Our code is available at https://gitthue wttpaawwwooo..euuuuu.ee.... cooo//iiuuaccc//KK
Introduction
Loss functions critical component of training deep models. Cross-Entropy loss is particularly
important image classification tasks [28] 55|
59
20
44
12), while Mean Square Error (MSE)
loss is commonly used in regression tasks [51 27
25 Contrastive loss 7 26
g 23 16 17
has emerged as a popular objective for representation learning. The selection of an appropriate loss function exert substantial influence
model's performance. Therefore, the development of
effective loss functions B
43 S
63 36 2 65
58
15] remains a critical research topic in the fields
of computer vision and machine learning.
Kullback-Leibles (KL) Divergence quantifies the degree of dissimilarity between a probability distribution and reference distribution. As one of the most frequently used loss functions, it finds application in various scenarios, such as adversarial training [71, 66 14 35). knowledge distillation (33 6 73). incremental learning [ 42). and robustness
out-of-distribution data [29]. Although many of these studies incorporate KL Divergence loss as part of their algorithms, they may not thoroughly investigate the underlying mechanisms of the loss function. To bridge this gap. paper aims to elucidate the working mechanism of KL Divergence regarding gradient optimization.
Our study focuses on the analysis of Kullback-Leiblee (KL) Divergence loss from the perspective of gradient optimization. For models with sofimax activation, we provide theoretical proof that is equiv- alent to the Decoupled Kullback-Lein Square Error (wMSE) loss and Entropy loss with soft labels. Figures [[a) and (b) reveal the equivalence between KI. and DKL. losses regarding gradient backpropagation. With the decomposed formulation, it becomes more convenient to analyze how the KL loss works in training optimization.
(DKL) Divergence loss which comprises a weighted Mean
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

KL Loss
wMSE CrossEntropy
BMSE + Crosspntropy
Qp
On
Om
On
Om
On
M
N
M
N
M
N
mm
2m
2m
(a) Loss
(b)
(c) IKL
Figure I: Comparisons of gradient backpropagation between KL, DKL, and IKL losses. DKL loss is equivalent to KL loss regarding backward optimization. .M and N can be the same (like in adversarial training) or two separate (like in knowledge distillation) models determined by application scenarios. Similarly, Im- Zn, X can also be the same (like in knowledge distillation) two different (like in adversarial training) images. 0m- On are logits output with which the probability vectors obtained when applying the softmax activation. Black arrows represent the forward process while colored arrows indicate the backward process driven by the corresponding loss functions the same color. "w MSE" is weighted Mean Square Error (MSE) loss. "WMSE" is incorporated with class-wise global information.
We have identified potential issues of KL loss with the newly derived DKL loss. Specifically, its gradient optimization asymmetric regarding the inputs. As illustrated in Fig- ure (b), the gradients Om and on asymmetric and driven by the WMSE and Cross-Entropy individually. This optimization asymmetry can lead the wMSE component being ignored in certain scenarios, such as knowledge dis- tillation where Om is the logits of the teacher model and detached from gradient backpropagation. Fortunately, it is convenient address this issue with the decoupled formu- lation of DKL loss by breaking the asymmetric optimization property. As evidenced by Figure [c)) enabling gradient on On from w MSE alleviates this problem.
Clean Ace (%)
70
65
60
55
25
30
35
40
AA (%)
Figure 2: We achieve SOTA robust- noss on R-FAR-100. "star" represents our method while "circle" denotes pre- vious methods. "Black" means adver- sarial training with image preprocess- ing only including random crop and flip, "Blue" is for methods with Au- toAug or CutMix, and "red" represents methods using synthesized data. AA is short for Auto-Attack [10]
Moreover, wMSE component is guided by sample-wise pre- dictions. Hard examples with incorrect prediction scores can lead to challenging optimization. We thus insert class-wise global information to regularize the training process. Inte- grating DKL with these two points, derive the Improved Bellbacck-Libble (IKL) Divergence loss.
To demonstrate the effectiveness of our proposed IKL loss, evaluate it with adversarial training and knowledge distillation tasks. Our experimental results on CIFAR-10/ 10/100 and ImageNet show that the IKI. loss achieves state-of-the-art robustness the public leaderboard of RobustBench Comparisons with previous methods on adversarial robustness are shown in Figure
In summary, the main contributions of our work are:
We reveal that the KL loss is mathematically equivalent to a composite of a weighted MSE (wMSE) loss and Cross-Entropy loss employing soft labels.
Based on our analysis, we propose two modifications for enhancement: breaking its asym- metric optimization and incorporating class-wise global information, deriving the Improved Kullback-Leibler (IKI.) loss.
With the proposed IKL loss, obtain the state-of-the-at adversarial robustness Robust- Bench and competitive knowledge distillation performance on 1IFAR-10/100 and ImageNet.
hhtps://oobuttrrcch.ggubbbb

---

2 Related Work
Adversarial Robustness. Since the identification of adversarial examples by Szegedy et al. [57] the security of deep neural networks (DNNs) has gained significant attention, and ensuring the reliability DNNS has become a prominent topic in the machine learning community. Adversarial training [46]. being the most effective method, stands out due to its consistently high performance.
Adversarial training incorporates adversarial examples into the training process. Madary et 46 propose the adoption of the universal first-order adversary, specifically the PGD attack, in adversarial training. Zhang al. [7] trade off the accuracy and robustness by the KL loss. Wu et al. [66] intro- duce adversarial weight perturbation to explicitly regulate the flatness of the weight loss landscape. Cui et al. [14 leverage guidance from naturally-trainet models to regularize the decision boundary in adversarial training. Additionally, various other techniques [35] focusing
optimization or training aspects have also been developed. Besides, recently, several works [22] 64 | have explored the use of data augmentation techniques to improve adversarial training. We have explored the mechanism of KL loss for adversarial robustness in this paper. The effectiveness of the proposed IKL loss is tested in both settings with and without synthesized data [38].
Knowledge Distillation. The concept Knowledge Distillation (KD) was first introduced by Hinton et al. [33]. It involves extracting "dark knowledge" from accurate teacher models to guide the learning process student models. This is achieved by utilizing the KI. loss to regularize probabilities of student models, aligning them with those of their teacher models when given the same inputs. This simple yet effective technique significantly improves the generalization ability of smaller models and finds extensive applications in various domains, Since the initial success of KD [33] several advanced methods, including logits-based
output
[9 21 67 72 73 34 and features-based 49 (50 68 have been introduced. This paper decouples the KL loss into a new formulation, i.e., DKL, and addresses the limitation of KL loss for application scenarios like knowledge distillation.
approaches [53 61 30 70 66
[3 [32 39
Other Applications of KL Divergence Loss. In semi-supervised learning, the KL loss acts
consistency loss between the outputs of weakly and strongly augmented images [56 60) In continual
learning, KL. loss helps retain previous knowledge by encouraging consistency between the outputs
of pre-trained and newly updated models [5] [22 Additionally, KL loss is also applied to enhance
model robustness to -uttt--distiibutiin data 29
76
Method
In this section, we begin by introducing the preliminary mathematical notations Section 3.1
Theoretical analysis of the equivalence between KL and DKL losses is presented in Section 3.2
Finally, we propose the IKI. loss to address potential limitations of KL/DKL Section 3.3 by case study with additional analysis in Section 3.4
followed
3.1 Preliminary
Definition of KL Divergence. Kullback-Leiller (KL) Divergence measures the differences between two probability distributions. For distributions P and Q of continuous random variable, It is defined to be the integral:
+00
p(x) log p(x) g(x))
(PI(()
dr.
where p and q denote the probability densities of P and Q.
The KL loss is one of the most widely used objectives in deep learning, applied across various contexts involving categorical distributions. This paper primarily examines its role in adversarial training and knowledge distillation tasks.
In adversarial training, the KL loss improves model robustness by aligning the output probability distribution adversarial examples with that of their corresponding clean images, thus minimizing output changes despite input perturbations. In knowledge distillation, the KL loss enables student model to mimic the behavior teacher model, facilitating knowledge transfer that enhances the student model's generalization performance.
3

---

Applications of KL Loss in Deep Learning. We consider image classification models that predict probability vectors using the softmax activation. Let o, E RC represent the logits output from a model given an input image E X, where C' denotes the number of classes. The predicted probability vector is S; E RRC computed as seftmax(o:). The values of and s correspond to the logits and probabilities for the j-th class, respectively. The KL loss is often used to encourage similarity between Sm, and S, in various scenarios, resulting in the following objective:
[Kr.(Em; Inn =
Σ
Sm * log
m
For example, in adversarial training, Im represents a clean image, while zn is its corresponding adversarial example. In knowledge distillation, 2im and F. the same image, but they are input separately to the teacher and student models. Notably, in the knowledge distillation process, Sm is detached from gradient backpropagation, the teacher model is pre-trained and fixed during training.
3.2 Decoupled Kullback-Leibler Divergence Loss
71 14 have incorporated the KI. loss into their algorithms without inves- tigating its underlying mechanism. This paper aims to uncover the driving force behind gradient optimization by analyzing the KL loss function. With the backpropagation rule in training optimiza- tion, the derivative gradients are as follows,
Previous works
3
ŌLKL dom
Σ (AAma Anj,)) (s s,,)),
(3)
k=1
0LKL S - s
(4)
doh
where Amy,k = o - o and Any,k = 0 o
Leveraging the antiderivative technique alongside the structured gradient information, we introduce a novel formulation called the Decoupled Kullback-Leibbee (DKL) Divergence loss, as presented in Theorem I The DKL loss is designed to be equivalent to the KL loss while offering a more analytically tractable alternative for further exploration and study.
Theorem From the perspective of gradient optimization, the Kullback-Leibler (KL) Divergence loss is equivalent to the following Decoupled Kullback-Leible (DKL) Divergence loss when a = 1 and B 1.
LDKL xm) =
a G I (mm- S(An)) -3 -S(s)))) log Sny
(5)
Cross-Entropy
weighted MSE (wMSE)
where S-)) represents stop gradients operation, s is transpose of Sm. in s/h * sm Amj.k
of %% and Anj,k of o Summation is used for the reduction of 1|2
Proof See Appendix A.I
Interpretation. With Theorem we know that KL loss is equivalent to DKL loss regarding gradient optimization, i.e., DKL loss produces the same gradients as KL loss given the same inputs. Therefore, KI. loss can be interpreted as a composition of a w MSE loss and a Cross-Entropy loss. This is the first work reveal the precise quantitative relationships between KL, Cross-Entrops and MSE losses. Upon examining this new formulation, we identify two potential issues with the KL loss.
Asymmetire Optimization. As shown in Eqs. and gradient optimization is asymmetric for o... and o,. The wMSE and Pross-Entropy losses in Theorem I are complementary and collaboratively work together make on, and o,, similar. Nevertheless, the asymmetric optimization cause the wMSE component to be neglected or overlooked when o, is detached from gradient backpropagation, which is the case for knowledge distillation, potentially leading to performance degradation.
Sample-wise Prediction Bias, As shown Eq. Wm WMSE component is conditioned on the prediction score of Tm. However, sample-wise wise predictions can be subject to significant variance.

---

Incorrect prediction of hard examples or outliers will mislead the optimization and result in unstable training. Our study in Sections 3.4 and 4.4 indicates that the choice of Wm significantly affects adversarial robustness.
3.3 Improved Kullback-Leible Divergence Loss
Based on the analysis in Section 3.2 we propose an Improved Kullback-Leible (IKL) Divergence loss. Distinguished from DKL in [ we make the following improvements: 1) breaking the asymmetric optimization property; 2) inserling class-wise global information to mitigate sample- wise bias. The details are presented as follows.
Breaking the Asymmetric Optimization Property. As shown in Eq. (5) the wMSE component encourages o, resemble On by capturing second-order information, specifically the differences between logits for each pair of classes. Each addend in wMSE only involves logits of two classes. We refer this property as locality. On the other hand, the Cross- Entropy loss ensures that s, and Sm produce similar predicted scores. Each addend in the Cross- Enooso- gathers all class logits. We refer this property as globality. Two loss terms collaboratively work together make o, and On similar in locality and globality. Discarding any one of them can lead to performance degradation.
However, because of the asymmetric optimization property of KL/DKL, the unexpected case can occur when 8m is detached from the gradient backpropagation (scenarios like knowledge distillation), in which the formulation will be:
LDKL-KD(Om In) a - S -(AA)))
-B-S(s,))) log
(6)
Cross-Entropy
weighted MSE (wMSE)
As indicated by Eq. (6) the wMSE component loss takes effect training optimization since all sub-components of wMSE are detached from gradient propagation, which can potentially hurt the model performance. Knowledge distillation exactly matches this case because the teacher model is fixed during knowledge distillation training. Thanks to the decomposition of DKL. formulation, we address this issue by breaking the asymmetric optimization property, ie., enabling the gradients of S( ((An) in Eq. (5. Then, the updated formulation of Eq. (6) becomes,
In) = a j ll S( S(wm))(m))))) (m:))((AAm) - An) -3 (S)
Cross-Entropy
weighted MSE (wMSE)
After enabling the gradients of S(An) in Eq. 5 wMSE will produce symmetric gradients on O, and Om. Regarding the knowledge distillation. wMSE can output gradient on, and promote the training optimization demonstrated by Eq. 7
Inserting Class-wise Global Information. Recall in Theorem
Wm in Eq. ( is calculated as:
w / S m
(8)
It indicates that Wm. depends on the sample-wist prediction scores. Nevertheless, the model cannot output correct predictions when dealing with outliers or hard examples training. In this case, wMSE will attach the most importance the predicted class ý arg max O. rather than the ground-truth class, which misleads the optimization and makes the training unstable.
We thus insert class-wise global information into wMSE component, replacing wm, with wy:
w.l.k A
5y g
(9)
where y is ground-truth label of 2m. By
XX, Σx,xx, 8;.
The class-wise global information injected by w, can act as regularization to enhance intra-class consistency and mitigate biases that may arise from sample noises. Especially, the late stage of training, Wy can always provide correct predictions, benefiting the optimization of wMSE component.
5

---

Table 1: Ablation study "GI" and "BA" with DKL loss. "Gl" represents "Inserting Global Information", and "BA" indicates "Breaking Asymmetric Optimization". "Clean" the test accuracy of clean images and "AA" is the robustness under Auto-Attack. CIFAR- 1-100 is used for the adversarial training task and ImageNet is adopted for the knowledge distillation task.
Index
GI
BA
Adversarial Training
Knowledge Distillation
Descriptions
Clean (%)
AA (%)
Top-1 (%)
Na
Na
62.87
30.29
71.03
baseline with KT. loss.
(b)
x
62.54
30.20
71.03
DKL, equivalent to KL loss. (b) with BA. (c) with GI, ie. IKL.
V
62.69
30.42
71.80
(d)
65.76
31.91
71.91
D.04
D:09
-003
(a) IKL-AT
(b) TRADES
(c) Margin differences
Figure 3: Visualization comparisons. (a) t-SNE visualization of the model trained by IKL-AT on
CIFAR-10M 100; (b) t-SNE visualization of the model trained by TRADES on CIFAR-100. (c) Class margin differences between models trained by IKI-AT and TRADES.
To this end, we derive the IKL loss in Eq. 10 by incorporating the two designs,
LIKI,(Zm, un)
i ill SlV An)ll -3 S(()))) log
(10)
Cross-Entropy
weighted MSE (WMSE)
where y is the ground-truth label for Feas Wy E RCC is the weights for class calculated with Eq. (9).
3.4 A Case Study and Analysis
Case Study. We empirically examine each component of IKL on CIFAR-100 with the adversarial training task and on ImageNet with the knowledge distillation task. Ablation experimental results and their setting descriptions are listed in Table [ In the implementation, for adversarial training, we improved TRADES [1]
baseline that combines with AWP (66) and uses increasing epsilon schedule 1 For knowledge distillation, we use the official code from DKD. The comparison between (a) and (b) shows that DKL can achieve comparable performance, confirming the equivalence to KL. The comparisons among (b), (c), and (d) confirm the effectiveness of the "GI" and "BA".
Analysis on Inserting Class-wise Global Information. As evidenced by Table I class-wise global information plays an important role in adversarial robustness. The mean probability vector s, of all samples in the class is more robust than the sample-wise probability vector. During training, once the model gives incorrect predictions for hard samples or outliers, Wm in Eq. (5) will wrongly guide the optimization. Adoption of Wy in Eq. 10)) mitigate the issue and meanwhile enhance intra-class consistency.
To visualize the effectiveness of inserting class-wise global information, we define the boundary margin for class y as:
Margin, 8y [y]
max 8,y [k].
(11)
k#y
We plot the margin differences between models trained by IKL-AT and TRADES on CIFAR-100. As shown in Figure 3c. almost all class margin differences are positive, demonstrating that there are larger decision boundary margins for the IKL-AT model. Such larger margins lead stronger robustness. This phenomenon is coherent with our experimental results in Section 4.1

---

Table 2: Test accuracy 1%) of clean images and robustness (%) under AutoAttack CIFAR-100.
All results are the average over three trials.
Dataset
Method
Architecture
Augmentation Type
Clean
AA
AWP
WRN-34-10
Basic
60.38
28.86
LBGAT
WRN-34-10
Basic
60.64
29.33
LAS-AT
WRN-34-10
Basic
64.89
30.77
ACAT
WRN-34-1
Basic
65.75
30.23
CIFAR-100
IKL-AT
WRN-34-10
Basic
65.76
31.91
(Leo, 8/255)
ACAT
WRN-34-10
AutoAug
68.74
31.30
IKL-AT
WRN-34-10
AutoAug
66.08
32.53
DM-AT 64
WRN-28-10
50M Generated Data
72.58 73.85
38.83
IKL-AT
WRN-28-10
50M Generated Data
39.18
Table 3: Test accuracy (%) of clean images and robustness (%) under AutoAttack on CIFAR-10. Average over three trials are listed.
Dataset
Method
Architecture
Augmentation Type
Clean
AA
Rice et al. [52
WRN-34-20
Basic
85.34
53.42
LBGAT
WRN-34-20
Basic
88.70
53.57
AWP
WRN-34-10
Basic
85.36
56.17
LAS-AT
WRN-34-10
Basic
87.74
55.52
CIFAR-10
ACAT
WRN-34-10
Basic
82,41
55.36
(oo. = 8/255)
IKL-AT
WRN-34-10
Basic
84.80
57.09
ACAT
WRN-34-10
AutoAug
88.64
57.05
IKL-AT
WRN-34-10
AutoAug
85.20
57.62
DM-AT (44
WRN-28-10
20M Generated Data
92.44
67.31
IKL-AT
WRN-28-10
20M Generated Data
92.16
67.75
We also randomly sample 20 classes in CIFAR-100 100 for t-SNE visualization. The numbers in the pictures class indexes. For each sampled class, collect the feature representation of natural images and adversarial examples with the validation set. The visualization by t-SNE shown in Figures 3b and 3a Compared with TRADES that trained with KL loss, features by IKL-AT models are more compact and separable.
4 Experiments
To verify the effectiveness our IKL loss, we conduct experiments on CIFAR-10, CIFAR100, and ImageNet for adversarial training (Section 4.1) and knowledge distillation (Sections and 4.3) More ablation studies are included in Section
4.1 Adversarial Robustness
baseline, which Experimental Settings. We use an improved version of TRADES 71
incorporates AWP (66) and adopts an increasing epsilon schedule. SGD optimizer with momentum of 0.9 is used. We the cosine learning rate strategy with initial learning rate of 0.2 and train models 200 epochs. The batch size is 128, the weight decay is 5e-4 and the perturbation size is set to 8/255. Following previous work [71 [14, standard data augmentation including random crops and random horizontal flip is performed for data preprocessing. Models are trained with 4 Nvidia GeForce 3090 GPUs.
Under the setting of training with synthesized data by generative models, we strictly follow the training configurations in DM-AT [64] for fair comparisons. Our implementations based their open-sourced code. We only replace the KL loss with our IKL loss.
Datasets and Evaluation. Following previous work 66 CIFAR-10 10 and CIFAR-100 are used for the adversarial training task. we report the clean accuracy on natural images and adversarial robustness under Auto-Attack [10] with epsilon 8/255.

---

Table 4: Top-1 accuracy (%) the ImageNet validation and training speed (sec/iteration) comparisons. Training speed is calculated on 4 Nvidia GeForce 3090 GPUs with a batch of 512 224x224 images. All results the average three trials.
ResNet34
ResNet50
Teacher
Distillation Manner
73.31
76. MobileNet
Extra Parameters
ResNet18
Student
69.75
68.87
AT
x
70.69
69.56
OFD
V
70.81
71.25
Features
CRD
V
71.17
71.37
ReviewKD
V
71.61
0.319 s/iter
72.56
0.526 s/iter
DKD
X 71.70
72.05
KD IKL-KD
X
71.03
70.50
Logits
X
71.91
0.197 s/iter
72.84
0.252 s/iter
Comparison Methods. To compare with previous methods, we categorize them into two groups according to the different types of data preprocessing:
Methods with basic augmentation, i.e., random crops and random horizontal flip.
Methods using augmentation with generative models or Auto-Aug [I], CutMix [69].
Comparisons with State-of-the-art on CIFAR-100. On CIFAR- 100, with the basic augmentations setting, we compare with AWP, LBGAT, LAS-AT, and ACAT. The experimental results are summa- rized in Table Our WRN-34-10 models trained with IKL. loss do a better trade-off between natural accuracy and adversarial robustness. With ] 5 and 3 5, the model achieves 65.76% % top-1 accuracy on natural images while 31.91% % adversarial robustness under Auto-Attack. An interesting phenomenon is that IKL-AT is complementary to data augmentation strategies, like AutoAug, without any specific designs, which is different from the previous observation that the data augmentation strategy hardly benefits adversarial training [66]. With AutoAug. obtain 32.53 % adversarial robustness, achieving new state-of of-the-rrt under the setting without extra real or generated data.
We follow DM-AT (64 to take advantage of synthesized images generated by the popular diffusion models 38 With 50M generated images, we create new state-of-the-arr with WideResNet-22. 10, achieving 73.85 % top-1 natural accuracy and 39.18% adversarial robustness under Auto-Attack.
Comparison with State-of-the-art on CIFAR-10 Experimental results on CIFAR-IO listed in 3 With the basic augmentation setting, our model achieves 84.80% top-1 accuracy on natural images and 57.09% robustness, outperforming AWP by 0.92% on robustness. With extra generated data, we improve the state-of of-the-art by 0.44%, achieving 67.75 % robustness.
4.2 Knowledge Distillation
Datasets and Evaluation. Following previous work we conduct experiments on CIFAR- 100 40 and ImageNet [54] to show the advantages of IKL on knowledge distillation. For evaluation, we report top-1 accuracy on CIFAR- and ImageNet validation. The training speed of different methods is also discussed.
Experimental Settings. We follow the experimental settings in DKD. Our implementation for knowledge distillation is based on their open-sourced code. Models are trained with and 8 Nvidia GeForce 3090 GPUs on CIFAR and ImageNet separately.
Specifically, on CIFAR-100, we train all models for 240 epochs with a learning rate that decayed by 0.1 the 150th, 180th, and 210th epoch. We initialize the learning rate to 0.01 for MobileNet and ShuffleNet, and 0.05 for other models. The batch size is 64 for all models. We train all models three times and report the mean accuracy. On ImageNet, for 100 epochs and decays the learning rate for every 30 epochs. We initialize the learning rate to 0.2 and set the batch size to 512.
the standard training that trains the model
For both CIFAR-100 and ImageNet, we consider the distillation among the architectures having the same unit structures, like ResNet56 and ResNet20, VGGNet13 and VGGNet8. On the other

---

Table 5: Peformance (%) on imbalanced data, i.e., the ImageNet-L.T
Method
Teacher
Student
Many(%)
Medium(%)
Few(%)
All(%)
Baseline
ResNet-18
63.16
33.47
5.88
41.15
Baseline
ResNet-50
67.25
38.56
8.21
45.47
Bascline
ResNet-101
68.91
42.32
11.24
48.33
KL-KD
ResNeXt-101
ResNet-11
64.6
37.88
9.53
44.32
KL-KD
ResNeXt-101
ResNet-55
68,83
42.31
11.37
48.31
IKL-KD
ResNeXt-101
ResNet-18
66.60
38.53
8.19
45,21
IKL-KD
ResNeXt-101
ResNet-50
70.06
43.47
10.99
49.29
hand, also explore the distillation among architectures made up of different unit structures, like WideResNet and ShuffleNet, VggNet and MobileNet-V2.
Comparison Methods. According to the information extracted from the teacher model in distillation training, knowledge distillation methods can be divided into two categories:
Features-basee methods [53 61 6 30) This kind of method makes use of features from different layers of the teacher model, which can need extra parameters and high training computational costs.
Logits- begeed-based methods [33 73 This kind method only makes use of the logits output of the teacher model, which does not require knowing the architectures of the teacher model and thus is more general practice.
Comparison with State-of-the-art on CIFAR-100. Experimental results on CIFAR-100 100 are summa- rized in Table and Table 14 (in Appendix). Table T3lists lists the comparisons with previous methods under the setting that the architectures of the teacher and student have the same unit structures. Models trained by IKL-KD can achieve comparable or better performance all considered settings. Specifically, achieve the best performance in 4 out of 6 training settings. Table [14] shows the comparisons with previous methods under the setting that the architectures of the teacher and student have different unit structures. We achieve the best performance in out of 5 training configurations.
Comparison with State-of-the-art on ImageNet. We empirically show the comparisons with other methods on ImageNet in Table With a ResNet34 teacher, our ResNet18 achieves 71.91 % top-1 accuracy. With a ResNet50 teacher, our MobileNet achieves 72.84% top-1 accuracy. Models trained by IKL-KD surpass all previous methods while saving 38% and 52% computation costs for ResNet34 ResNet18 and SesNee50--Moblees distillation training respectively when compared
with ReviewKD 6
4.3
Knowledge Distillation on Imbalanced Data
Data often follows long-tailed distribution. Tackling the long-tailed recognition problem is essential for real-world applications. Lots of research has contributed to algorithms and theories 3 19
37 47
17 [6 18
75 on the problem. In this work, we examine how the knowledge distillation with our IKL loss affects model performance on imbalanced data, i.e., ImageNet-LT [45). We train ResNets models 90 epochs with Residom-Resiee--Copp and horizontal flip image pre-processing. Following previous work [13], we report the top-1 accuracy on Many-shot, Meidum-shot, Few-shot, and All classes. As shown in Table IKL-KD consistently outperforms KL-KD on imbalanced data.
4.4 Ablation Studies
Ablation on a and 3 for Adversarial Robustness. Thanks to the decomposition of the DKL loss formulation, the two components (wMSE and Cross-Entropy) of IKL can be manipulated independently. We empirically study the effects of cyper-paraamterts of a and 3 on CIFAR-100 for adversarial robustness. Clean accuracy natural data and robustness under AA [10] reported in Table 7 and Table Reasonable a and 3 should be chosen for the best trade-off between natural accuracy and adversarial robustness.
Ablation on Temperature (r) Global Information. As discussed Section 3.3 the incorporated wise global information is proposed to promote intra-class consistency and mitigate the biases

---

Table 6: Ablation study hyper-parameters of IKL.
Clean
AA
B
Clean
AA
7
Clean
AA
3
67.52
31.29
66.13
30.95
1
59.99
31.35
4
66.26
31.33
8 = 66.31
31.33
7 = 2
63.77
31.88
5
65. 76
31.91
66.00
31.57
= 3 65.28
31.69
6
65.14 14
31.64
8 = 65.76
31.91
7 4
65.76
31.91
I
Table Effects of 4.
Table 8: Effects of B.
Table 9: Effects of T.
Table 10: Ablation study of
AA
Method
Clean
4
8
12
Avg.
255
255
255
255
255
255
TRADES
62.87
53.88
45.31
37.28
30.29
24.28
19.17
35.04
IKL-AT
63.40
55.31
46.76
38.98
31.91
25.33
19.98
36.38
Table 11:
Evaluation under PGD and CW attacks.
Method
Acc
PGD-10
PGD-20
CW-10
CW-20
Auto-Attack
Worst
KL(TRADES)
62.87
36.01
35.84
40.03
39.86
30.29
30.29
(Our))
63.40
36.78
36.55
40.72
40.47
31.91
31.92
IKLsOurs with autoaug)
65.93
38.15
37.75
41.10
40.86
32.53
32.52
IKurS with synthetic data)
73.85
44.43
44.12
47.59
47.53
39.18
39.18
When calculating the my and Sy. temperature could be applied before getting sample probability vectors. We summarize the experimental results in Table [ for ablation of Interestingly, we observe that models usually exhibit higher performance on clean images with higher 7. There are even 5.75% improvements of clear accuracy while keeping comparable robustness when changing = 1 to 4, which implies the vast importance of weights in wMSE component of DKL/KL. for adversarial robustness. To achieve the strongest robustness, we finally choose as illustrated by empirical study.
from sample noises.
Ablation Various Perturbation Size We evaluate model robustness with unknown perturbation size in training under Auto-Attack. The experimental results are summarized in Table TO As shown Table 10 model robustness decreases significantly as the increases for both the TRADES model and our model. Nevertheless, our model achieves stronger robustness than the TRADES model under all of e, outperforming TRADES by 1.34% on average robustness. The experimental results demonstrate the super advantages of models adversarially trained with our IKI. loss.
Robustness under Other Attacks. Auto-Attack is currently of the strongest attack methods. It ensembles several adversarial attack methods including APGD-CE, APGD-DLR, FAB, and Square Attack. To show the effectiveness of our IKL loss. also evaluate our models under PGD and CW attacks with 10 and 20 iterations. The perturbation size and step size are set to 8/255 and 2/255 respectively. As shown Table [1] with increasing iterations from 10 to 20, models show similar robustness, demonstrating that our models don't 't suffer from obfuscated gradients problem.
Conclusion and Limitation
In this paper, we have investigated the mechanism of Kullback-Leibler (KL) Divergence loss in terms of gradient optimization. Based on our analysis, we decouple the KL loss into weighted Mean Square Error (wMSE) loss and Cross-Entropy loss with soft labels. The new formulation is named Decoupled Kullback-Leibh (DKL) Divergence loss. To address the spotted issues of KL/DKL, we make two improvements that break the asymmetric optimization property and incorporate class-wise global information, deriving the Improved Kullback-Leible (IKL) Divergence loss. Experimental results on CIFAR- 10/100 and ImageNet show that we create new state-of sfate-of-the-ass tht-hh--r art adversarial robust- ness and competitive performance on knowledge distillation. This underscores the efficacy of our Innovative KI (IKL) loss technique. The KI. loss exhibits wide range of applications. As part of our future work, we aim explore and highlight the versatility of IKI. various other scenarios, data, and incremental learning.
like robustness on
10

---

References
[1] Sravanti Addepalli, Samyak Jain, and Venkatesh Babu Radhakrishnan. Efficient and effective augmentation strategy for adversarial training. In NeurlPS, 2022.
[2] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-uiiinn measure in neural networks. In CVPR, pages 4413-4421, 2018.
[3] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabel-distribution-aaaa margin loss. NewrIPS, 2019.
[4] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning visual features by contrasting cluster assignments. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-FForina Balcan, and Hsuan- Tien Lin, editors, NewrIPS, 2020.
[5] Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, pages 532-547, 2018.
[6] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In CVPR, 2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for contrastive learning of visual representations. In ICML, 2020.
[8] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, 2020.
[9] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, pages 4794-4802, 2019.
[10] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. JCML. PMLR, 2020.
[11] Ekin D. Cubuk, Barret Zoph. Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data, In CVPR, 2019.
[12] Jiequan Cui, Pengguang Chen, Ruiyu Li, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Fast and practical neural architecture search. In ICCV, pages 6509-669600 6518, 2019.
[13] Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia. Reslt: Residual learning for long-tailed recognition. IEFE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2022.
[14] Jiequan Cui, Shu Liu, Liwei Wang, and Jiaya Jia. Learnable boundary guided adversarial training. In ICCV, 2021.
[15] Jiequan Cui, Yuhui Yuan, Zhisheng Zhong, Zhuotao Tian, Han Hu. Stephen Lin, and Jiaya Jia. Region rebalance for long-tailed semantic segmentation. arXiv preprint arXiv:2204.01969, 2022.
[16] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In ICCV, pages 715-724, 2021.
[17] Jicquan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, and Jiaya Jia. Generalized parametric contrastive learning. arXiv preprint arXiv:2209. 12400, 2022.
[18] Jiequan Cui, Beier Zhu, Xin Wen, Xiaojuan Qi, Bei Yu, and Hanwang Zhang. Classes are not equal: An empirical study on image recognition faimess. In CVPR, pages 23283 23292, 2024.
[19] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss hased on effective number of samples. In CVPR, pages 9268-9277, 2019.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11929, 2020.
[21] Tommaso Furlanello, Zachary Lipton, Michael Tschannen. Laurent Itti, and Anima Anandkumar, Born again neural networks. In ICML, pages 1607-1616. PMLR, 2018.
[22] Sven Gowal, Sylvestre Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. 2021.

---

[23] Saan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya. Carl Doersch, Bemaardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self- supervised learning. In Hugo Larochelle. Mare" Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, NeurlPS. 2020.
[24] Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, and Yunhe Wang. Vanillakd: Revisit the power of vanilla knowledge distillation from small scale to large scale. arXiv preprint arXiv:2305.1778, 2023.
[25] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, pages 16000-16009, 2022.
[26] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning. In CVPR, 2020.
[27] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-can. In CVPR, pages 2961-2966 2017.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[29] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method improve robustness and uncertainty. arXiv preprint arXiv: 191..2791,,02781, 2019.
[30] Bycongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. comprehen- sive overhaul of feature distillation. In ICCV, 2019.
[31] Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A compre- hensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921-1930, 2019.
[32] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. AAAI, pages 9779-3787, 2019.
[33] Geolfrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. 2015.
[34] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. In NeurlPS, 2022.
[35] Xiaojun Jia, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Las-at: Adversarial training with learnable attack strategy. In CVPR, 2022.
[36] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, pages 694-711. Springer, 2016.
[37] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalan- tidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.0922 2019.
[38] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206.00364 2022.
[39] Jangho Kim. SeongUk Uk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. NeurIPS, 2018.
[40] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
[41] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[42] Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Woo Ha, and Byoung- Tak Zhang. Overcoming catas- trophic forgetting by incremental moment matching. NeurIPS, 2017.
[43] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022 2021.
12

---

[45] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVE conference on computer vision and puttern recognition, pages 2537-2546, 2019.
[46] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.060S: 2017.
[47] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:200..03314 2020.
[48] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, pages 5191-5198, 2020.
[49] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR. pages 3967-3976, 2019.
[50] Baoyun Peng. Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In ICCV, pages 5007-5016, 2019.
[51] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn; Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[52] Leslie Rice, Eric Wong, and Zico Kolter. Overlitting in adversarially robust deep learning. In ICML. PMI.R, 2020.
[53] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
[54] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bemstein. ImageNet large scale visual recognition challenge. IJCV, 2015.
[55] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR, 2015.
[56] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplitying semi semi-supervised learning with consistency and confidence. NeurIPS, 33:596-608, 2020.
[57] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint urXiv:1312.61999 2013.
[58] Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In CVPR, pages 11662-11671. 2020.
[59] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc
Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019.
[60] Anti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve supe-supe deep learning results. NeurlPS, 30, 2017.
[61] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020.
[62] Zhuotao Tian, Pengguang Chen, Xin Lai, Li Jiang, Shu Liu, Hengshuang Zhao, Bei Yu, Ming-Chang Yang. and Jiaya Jia. Adaptive perspective distillation for semantic segmentation. IEEE Tronsactions on Pattern Analysis and Machine Intelligence, 45(2):1372-1387) 2022.
[63] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[64] Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. arXiv preprint 23Xx:3332200633 2023.
[65] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In ECCV, pages 499-515. Springer, 2016.
[66] Dongxian Wu, Shu- Tao Xia, and Yisen Wang, Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958 2922929999,9 2020.
13

---

[67] Chenglin Yang, Lingxi Xie, Chi Su, and Alan L Yuille. Snapshot distillation: Teacher-student optimization in one generation. In CVPR, pages 2859-2868. 2019.
[68] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, pages 1133-4141, 2017.
[69] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
[70] Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
[71] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoreti- cally principled trade-off hetween robustness and accuracy. In JCML. PMLR, 2019.
[72] Ying Zhang. Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, pages 4320-4328, 2018.
[73] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, 2022.
[74] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings the IEEE/CVF International Conference Computer Vision, pages 15659- 15669, 2023.
[75] Beier Zhu, Yulei Niu, Xian-Sheng Hua, and Hanwang Zhang. Cross-domain empirical risk minimization unbiased long-tailed classification. Proceedings of the AAAI conference on artificial intelligence. volume 36, pages 3589-3597, 2022.
[76] Beier Zhu, Yulei Niu, Sacil Lee, Minhoe Hur, and Hanwang Zhang. Dehiased fine-tuning for vision- language models hy prompt regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3834-3333 3842, 2023.
[77] Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine- tuned models by removing label bias in foundation models. Advances in Neural Information Processing Systems, 36, 2024.
14

---

A Appendix
A.1 Proof to Theorem 1
To demonstrate that DKL in Eq. 5 is equivalent to KL. in Eq. (2) for training optimization, we prove that DKL and KL produce the same gradients when given the same inputs.
For KL loss, we have the following derivatives according to the chain rule:
as
Oo
=
S *
m
j!=
as
-8 -m s
d
0LkL
log sm
log si + 1,
Üs
0Lkt.
Sn
sm
(12)
0o
0LKL
LKI.
Us do
LKI
dsh
Oo
Osm
do
= (logs)n
logs + 1) m
m +
(logs' - llog s/ + +1 * (-s (-))( +
f!
C
Σ (logs/n log s/n) (log 8
llog s²,) )}>)))))))))))))))))) * (s/ s
=
d=
C
Σ
((0 )//) - -o,)) * (s/, * s
Amiyj An:,)) who
d=j
Σ (Ami. Ang wht
(13)
For DKL los, we expand the Eq.
as:
(DKI(Zm)
d S( S(An))l1 -B (.((s))))) log Sn
(14)
Cross-Entropy
weighted MSE (wMSE)
C
- S(An)a)) S(wth))) - Σ S(s log
-
1=1
1=1
weighted MSE (wMSE)
Cross-Entropy
According to the chain rule, we obtain the following equations:
θLDKL B (s), s,))
(15)
Oo'
ØLDKL
*2*
(Amja Ang + (-wlit) Σ (Amis ADE wht)
do
(Amaj Amij) w
(16)
15

---

Table 12: New state-of-the-art public leaderboard RobustBench [10].
Experimental Settings
augmentation strutegy
Clean
AA
Computation saving
Generated Data (Previous results)
Basic
62.99
31.20
Generated Data (Ours)
Basic
65.76(+2.67)
31.91(+0.71)
33.3%
Generated (Previous results)
Autoaug Autoaug
68.75
31.85
w/o Generated Data (Ours)
66.08
(2.53(+0)688)
33.3%
Generated Duta (Previous best results) w/ Generated Data (Ours)
Genreated data
72.58
38.83
Generated data
73.85(+1.27)
39.18(+00335)
0%
Table 13: Top-1 aceuracy (%) on the CIFAR-100 validation. Teachers and students are in same architectures. All results are the average over three trials.
ResNet56
ResNetl
ResNet32 x4
WRN-40-2
WRN-40-2
VGG13
Teacher
Distillation Manner
72.34
74.31
79.42
75.61
75.61
74.64
ResNet20
ResNet32
ResNet8x4
WRN-16-2
WRN-40-1
VGG8
Student
69.06
71.14
72.50
73.26
71.98
70.36
FitNet
69.21
71.06
73.50
73.58
72.24
71.02
RKD
69.61
71.82
71.90
73.35
72.22
71.48
Features
CRD
71.16
73.48
75.51
75.48
74.14
73.94
OFD
70.98
73.23
74.95
75.24
74.33
73.95
ReviewKD
71.89
73.89
75.63
76.12
75.09
74.84
DKD
71.97
74.11
76.32
76.24
74.81
74.68
KD
70.66
73.08
73.33
74.92
73.54
72.98
Logits
IKL-KD
71.44
74.26
76.59
76.45
74.98
74.98
Table 14: Top-1 accuracy ( (%) on the CIFAR-100 validation. Teachers and students are in different architectures. All results are the average over trials.
x4
WRN-40-2 75,61
VGG13 74.64
ResNet50
ResNet32
Teacher
Distillation Manner
79.42
79.34
79.42
ShufileNet-Vi 70.50
ShufflcNet-Vi
MobileNet-V2
MobilcNet-V2
ShuffleNet-V2
Student
70.50
64.60
64.60
71.82
FitNet
73.59
73.73
64.14
63.16
73.54
RKD
72.28
72.21
64.52
64.43
73.21
Features
CRD
75.11
69.73
69.11
75.65
OFD
75.98
75.85
69.04
76.82
ReviewKD
77.45
77.14
70.37
69.89
77.78
DKD
76.45
76.70
69.71
70.35
77.07
KD
74.07
74.83
67.37
67.35
74.45
Logits
IKL-KD
76.64 + 0.02
77.19 = 0.01
70.40 + 0.03
70.62 + 0.08
77.16 = 0.04
Comparing Eq. 12 and Eq. 15 Eq. 13 and Eq. (16), conclude that DKL loss and KI. loss have the same derivatives given the same inputs. Thus, DKL. loss is equivalent to KI. loss in terms of gradient optimization.
A.2 New state-of-the-art robustness on CIFAR-100/10
Robustbench is the most popular benchmark for adversarial robust models in the community. evaluates the performance of models by the Auto-Attack. Auto-Attack [10] is ensemble of different kinds attack methods and is considered the most effective method to test the robustness of models.
We achieve new state-of-tee tfate-of-the-an the-art- art robustness CIFAR- 10 and CIFAR- 100 under both settings w/ and w/o generated data. shown in Table 12 on CIFAR-100 without extra generated data, we achieve 32.53% robustness, outperforming the previous best result by 0.68% while saving 33.3 % computational cost. With generated data, our model boosts performance 73.85% natural accuracy, surpassing the previous best result by 1.27 % while maintaining the strongest robustness. More detailed comparisons can accessed on the public leaderboard io/
giithbb
16

---

Table 15: Comparisons with strong training settings ImageNet for knowledge distillation.
Method
KD
DKD
DIST
IKL-KD
Top-1 Accuracy (%)
80.89
80.77
80.70
80.98
A.3 Comparisons on CIFAR-100 for Knowledge Distillation
We experiment on CIFAR-100 with the following cases: 1) the teacher and student models have the same unit network architectures; 2) the teacher and student models have different unit network architectures. The results are listed in [3ble[33 and Table [4] We have achieved the best results in of and out of experimental settings respectively.
Moreover, we follow the concurrent work 24 and conduct experiments with BEiT-Large as the teacher and ResNet-50 the student under a strong training scheme, the experimental results summarized in Table [5] The model trained by IKL-KD shows slightly better results.
A.4 Other Applications with IKL
Semisupervised learning. We use the open-sourced code from https://github. com/ siicrosoft/Seii-uuperve serriiusuueiiiie-eeeaeee searneddneeenngnggee and conduct semi-supervised experiments on CIFAR- 100 with FixMatch and Mean-Teacher methods. Specifically, each class has 2 labeled images and 500 unlabeled images. All default training hyper-parameters are used for fair comparisons. We only replace the consistency loss with our IKL loss. As shown in Table I6. with our IKL loss, the Mean-Teacher Teacher method even surpasses the FixMatch.
Table 16: Semi-supervised Learning on CIFAR-100 with ViT-small backbone.
Method
Pscudo-label
Consistency Loss
Last epoch Top-1 Acc(%)
FixMatch
FixMatch
hard
Cross-entropy Loss
69.20
FixMatch
soft
Cross-entroy/KL Loss
69.09
FixMatch
soft
IKL Loss
70.00
Mean-Teacher
Mean-Teacher
soft
MSE Loss IKL Loss
67.38
Mean-Teacher
soft
70.05
Semantic segmentation distillation. We conduct ablation on the semantic segmentation distillation task. We use the APD [62] as our baseline for their open-sourced code. All default adopted. We only replace the original KL loss with our IKL loss. As shown in Table T7] achieve better performance with the IKL loss function, demonstrating that the IKL loss can be complementary other techniques in semantic segmentation distillation.
Table 17: Semantic segmentation distillation with APD on ADE20K.
Method
Teacher
Student
Teacher mloU
Student mloU
Baseline
ResNet-18
37.19
APD with KL loss APD with IKL loss
ResNet-101 101
ResNet-18
43.44 43.44
39.25
ResNet-101
ResNet-18
39.75
A.5 Complexity IKL.
Compared with the KI. divergence loss, IKL loss is required to update the global class-wise prediction scores W CxC where is the number of classes during training. This extra computational cost can be nearly ignored when compared with the model forward and backward. Algorithm
17

---

shows the implementation of our IKL loss in Pytorch style, On dense prediction tasks like semantic segmentation, A. and A,, can require large GPU Memory. Here, we also provide the memory- implementations for wMSE loss component, which is listed in Algorithm 2
Algorithm 1 Pseudo code for DKL/IKL loss in Pytorch style.
Input: logits,, logits; E R BBC one-hot label Y, E RCxC a, B.
class_ one-hot @ W:
Somple_weights- class cecosssccreeeviiii view(-1, C, 1) @ class_ class_score..ieex(11,, 1,
A, loggiss,.vew((......... C. logits,,.
C);
A logitss..view((11 C. 1)
1, C);
wMSE loss (torch. (torch.pow(A,: A.. 2) * heiiplee_wiighhssssmmii-----
score a F.soooaaa(loooiits.. dim=l).detach():
log score_ b Ftloollooooooooaaaeeeeeeeeee softmax(logits,, dim=-l);
CE los_osss=- = -(score____ a log_score. b)..bm)ssmmmmm b).sum()).mean(() ).mean();
return * CE loss + wMSE loss.
Algorithm 2 Memory efficient implementation for wMSE_loss in Pytorch style.
Input: logitssa, logitss BBxC oonehhot label Y, W RCXC;
class_ cecos_sccessse one-hot @ W:
loss (class_sceee sccosss_ceeeeee llgits. * llggits,..sum(di==)) * 2 ttoch.pow
2) 2;
loss b (class_scorf)
logitst * logitss)).um((din])
torch.pow((class_corre *
logits;) (uniits>).um(dimm1),, 2) 2;
loss (class_sooe (class_scores" logils, * (onnits)).uud(di==1)) (class_scores * (oggitss).uu((dim=))
(class_ scores logits;) (inrtsss)summdimm111 4; oclass_scores'
wMSE loss (loss loss b loss ex).mean())
return wMSE loss.
A.6
Connection between IKL and the Jensen-Shannon (JS) Divergence
With the following IS divergence loss,
JSD(PIIQ) 1 KL(PI))) 1 " KLQQIMM), = p 1.Q
(17)
We calculate its derivatives regarding on (the student logits),
@LJsD
Ai (Anay Am'
(18)
0o
1
Softmax (Om')
Sn +
2 8m
(19)
where Om is the logits from the teacher model, Omn' virtual logits satisfying Eq. 19 Sm Softmax(Om). Sn Softmax(o..). Am's, o o An,,; of o
Correspondingly, the derivatives of IKL loss regrading on (the student logits).
0LIKL 00
a
w Ai (Ani. Am;.;) + 3 s (s", - 1) + * (1 sin)
(20)
Effects of Cross-Entropy
Effects of wMSE.
Compared with IKL loss, the problem for ISD divergence in knowledge distillation that: The soft labels from the teacher models often embed dark knowledge and facilitate the optimization of the student models. However, there are no effects of the cross-ent cross-entropy loss with the soft labels from the teacher model, which can be the underlying reason that JSD is worse than KD and IKL-KD.
As shown in Table T8 we also empirically demonstrate that IKI. loss performs better than JID divergence on the knowledge distillation task.
18

---

Table 18: Comparisons between KI., IKI. and JSD ImageNet-L.T
Method
Student
Teacher
Teacher Acc(%)
Student Acc(%)
Self-distillation Imbalanced Data
ResNet-50
ResNet-50
45.47
47.04
JSD
ResNet-50
ResNet-50
45.47
46.64
Ours
ResNet-50
ResNet-50
45.47
47.50
Knowledge distillation on Imbalanced Data
KL
ResNet-50
ResNeXt-101
48.33
48.31
JSD
ResNet-50
ResNeXt-101
48.33
47.82
Ours
ResNet-50
ResNeXt-101
48.33
49.22
A.7
Licenses
All the datasets we considered are publicly available, we list their licenses and URLs as follows:
CIFAR-10 41
MIT License,
http://ww....oooooooe.//////aaaa html
CIFAR-100 41 MIT htttps://ww. cs. www.otoo.eu/okriiieo..... http://..t/t/////aoooooooooaaaar
html
ImageNet [54]
Non-commercial, http: htap:::/mmmgmeeett net.orgeetrrg
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We theoretically prove that the Kullback-Leibler (KL) Divergence loss is equivalent Decoupled Kullback-Leibler (DKL.) loss regarding gradient optimization. Based on this analysis, we improve KL/DKL loss on adversarial training and knowledge distillation tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The conclusion and limitation section is included.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: Our proof to Theorem D iss in Appendix A.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed experimental settings in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide a link for our code and models the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
pripp..cc/ubllicguuidee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experimental settings Section and ablations for hyper-parameters the Appendix 44.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are included in Table 14 of the Appendix. Due to the heavy computation, other tables don't provide error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: On adversarial training, Each run with basic augmentations takes around days using 4GPUs while days using GPUs for adversarial training with generated data. On knowledge distillation, 8 Nvidia GeForce 3090 GPUs are used on ImageNet. Each run takes about day for our method.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http://pp...pppulc/c//tccccaaaaa
Answer: [Yes]
Justification:
Our research conforms NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper is fundamental research in adversarial robustness and knowledge distillation and there is no obvious socictal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our models don't suffer from high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Proper citations added the paper. Licenses for used data is included in Appendix A.7
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "u0mDg0Sipf", "reviewer": "Reviewer_JUtg", "summary": "The paper presents an in-depth analysis of the Kullback-Leibler (KL) Divergence loss, a critical component in training deep learning models. The authors mathematically demonstrate that the KL loss is equivalent to a Decoupled Kullback-Leibler (DKL) loss, which is composed of a weighted Mean Square Error (wMSE) loss and a Cross-Entropy loss with soft labels. They identify limitations of the KL/DKL loss, particularly in scenarios like knowledge distillation, and propose improvements by introducing an Improved Kullback-Leibler (IKL) Divergence loss. This IKL loss breaks the asymmetric optimization property of the original KL loss and incorporates class-wise global information to mitigate biases from individual samples. The paper evaluates the IKL loss on CIFAR-10/100 and ImageNet datasets, focusing on adversarial training and knowledge distillation tasks. The results show that the proposed IKL loss achieves state-of-the-art adversarial robustness and competitive performance in knowledge distillation.", "questions": "1. How does the IKL loss perform in comparison to other contemporary loss functions that are designed for specific tasks, such as object detection or semantic segmentation?\n2. What are the implications of the IKL loss for models that are trained with limited data, and how does it handle class imbalances?\n3. Can the authors provide more insight into the selection of the hyperparameters α and β in the IKL loss, and how sensitive are the results to these values?\n4. The paper mentions the use of class-wise global information; how does this interact with models that are trained in a semi-supervised or unsupervised setting?\n5. How does the IKL loss handle adversarial examples that are crafted to be more sophisticated than those generated by the Auto-Attack method used in the paper?", "limitations": "see the sections of weaknesses and questions", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a novel theoretical perspective on the KL loss by proving its equivalence to the DKL loss, offering a new understanding of gradient optimization in deep learning.\n2. The IKL loss addresses specific limitations of the KL loss and demonstrates improved performance in adversarial training and knowledge distillation, which are significant for real-world applications.\n3. Comprehensive experiments on standard datasets like CIFAR-10/100 and ImageNet validate the effectiveness of the proposed IKL loss, enhancing the paper's credibility.", "weaknesses": "1. While the IKL loss shows promising results on the mentioned datasets, it is unclear how well these improvements would generalize to other types of data or tasks outside the scope of the paper.\n2. The introduction of the IKL loss adds complexity to the model training process, which might be a concern for practitioners looking for simpler solutions.\n3. The paper does not provide a detailed analysis of the computational cost associated with the IKL loss, which is important for resource-constrained environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and valuable comments. Here we provide our responses to address your concerns.\n\n*Q1: Besides the adversarial training and the knowledge distillation on balanced data, how does the IKL loss perform on other tasks, like semi-supervised learning, knowledge distillation on imbalanced data, and semantic segmentation knowledge distillation?*\n\nThanks for your suggestions. Besides the adversarial training task and knowledge distillation on balanced data, We conduct experiments on other tasks including semi-supervised learning, knowledge distillation on imbalanced data, and semantic segmentation knowledge distillation. Experimental results show that our IKL loss can significantly improve model performance when compared with KL loss. For details, please refer to **our global rebuttal Q1**.\n\n*Q2: Pseudo code of IKL loss and complexity analysis.*\n\nCompared with the KL loss, IKL loss is only required to update the global class-wise prediction scores $W \\in \\mathbb{R}^{C \\times C}$ where $C$ is the number of classes during training. \nThis extra computational cost can be nearly ignored when compared with the model forward and backward. Algorithm 1 shows the implementation of our IKL loss in Pytorch style. On dense prediction tasks like semantic segmentation, $\\Delta_{a}$ and $\\Delta_{b}$ can require large GPU Memory due to the large number of pixels. Here, we also provide the memory-efficient implementations for $w$MSE loss component, which is listed in Algorithm 2.\n\n#### *Algorithm 1: Pseudocode for DKL/IKL loss in Pytorch style*\n**Input:** $logits_{a}, logits_{b} \\in \\mathbb{R}^{B \\times C}$; one-hot label $Y$; $W \\in \\mathbb{R}^{C \\times C}$; $\\alpha$ and $\\beta$.\n\nclass_scores = $Y$ @ $W$;\n\nSample_weights = class_scores.view(-1, C, 1) @ class_scores.view(-1, 1, C);\n\n$\\Delta_a$ = $logits_a$.view(-1, C, 1) - $logits_a$.view(-1, 1, C);\n\n$\\Delta_b$ = $logits_b$.view(-1, C, 1) - $logits_b$.view(-1, 1, C);\n\n*wMSE\\_loss* = (torch.pow($\\Delta_{n}$ - $\\Delta_{a}$, 2) * Sample\\_weights).sum(dim=(1,2)).mean() * $\\frac{1}{4}$;\n\nscore\\_a = F.softmax($logits_a$, dim=1).detach();\n\nlog\\_score\\_b = F.log\\_softmax($logits_b$, dim=-1);\n\n*CE\\_loss* = -(score\\_a * log\\_score\\_b).sum(1).mean(); \n\n$\\textbf{return}$ $\\beta$ * CE\\_loss + $\\alpha$ * wMSE\\_loss.\n\n\n#### *Algorithm 2: Memory efficient implementation for wMSE in Pytorch style.*\n**Input:** $logits_{a}, logits_{b} \\in \\mathbb{R}^{B \\times C}$; one-hot label $Y$; $W \\in \\mathbb{R}^{C \\times C}$;\n\nclass\\_scores = $Y$ @ $W$;\n\nloss\\_a = (class\\_scores * $logits_a$ * $logits_a$).sum(dim=1) * 2 - torch.pow((class\\_scores * $logits_a$).sum(dim=1), 2) * 2;\n\nloss\\_b = (class\\_scores * $logits_b$ * $logits_b$).sum(dim=1) * 2 - torch.pow((class\\_scores * $logits_b$).sum(dim=1), 2) * 2;\n\nloss\\_ex = (class\\_scores * $logits_a$ * $logits_b$).sum(dim=1) * 4 - (class\\_scores * $logits_a$).sum(dim=1) * (class\\_scores * $logits_b$).sum(dim=1) * 4;\n\nwMSE\\_loss = $\\frac{1}{4}$ * (loss\\_a + loss\\_b - loss\\_ex).mean();\n\n$\\textbf{return}$ wMSE\\_loss.\n\n\n*Q3: Can the authors provide more insight into the selection of the hyperparameters $\\alpha$ and $\\beta$ in the IKL loss, and how sensitive are the results to these values?*\n\nFor tasks using KL divergence loss, like knowledge distillation, adversarial training, and semi-supervised learning, we have priors to set the loss weight $\\gamma$ for KL divergence loss. When replacing KL divergence loss with our IKL loss, we first set the $\\beta$ =$\\gamma$ for the cross-entropy component. Then, we perform the grid search $\\frac{\\alpha}{4}$ $\\in$ \\{1, 2,3, 4,5\\}. After determining the value of $\\frac{\\alpha}{4}$, we again adjust the $\\beta$ to achieve the best performance on validation data if needed. As shown in Table 15 of the Appendix, in a reasonable range $\\frac{\\alpha}{4} \\in$ \\{3,4,5,6\\} and $\\beta \\in$ \\{2,3,4,5\\}, increasing $\\frac{\\alpha}{4}$ and $\\beta$, the performance firstly increases steadily and then decreases.\n\n*Q4: How does the class-wise global information interact with models that are trained in a semi-supervised or unsupervised setting?*\n\nIn semi-supervised or unsupervised settings, we often can get the pseudo labels for the data. Equipping with a threshold $\\gamma$, we filter out the samples that the maximum confidence score is lower than $\\gamma$, like in FixMatch [ref2]. Then, the remaining samples can be used to update the global class-wise prediction scores.\n\n[ref2] FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. NeurIPS 2020.\n\n*Q5: How does the IKL loss handle adversarial examples that are crafted to be more sophisticated than those generated by the Auto-Attack method used in the paper?*\n\nThe auto-attack is one of the strongest attack methods. It is an ensemble of several adversarial attacks, including APGD-CE, APGD-DLR, FAB, and Square Attack. Here we also test the robustness of our models under PGD, and CW attacks with 10 and 20 iterations. \nThe perturbation size and step size are set to 8/255 and 2/255 respectively. As shown in the Table, with increasing iterations from 10 to 20, our models show similar robustness, demonstrating that our models don't suffer from the obfuscated gradients problem. \n\n| Method | Acc | PGD-10 | PGD-20 | CW-10 | CW-20 | Auto-Attack |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| KL (TRADES) | 62.87 | 36.01 | 35.84 | 40.03 | 39.86 | **30.29** |\n| IKL (Ours) | 63.40 | 36.78 | 36.55 | 40.72 | 40.47 | **31.92** | \n\nThe worst-case is in **bold**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZqOOwAXbdf", "reviewer": "Reviewer_nZvw", "summary": "This paper demonstrates that KL divergence can be decoupled into a weighted mean square error (wMSE) loss term and a cross-entropy term with soft labels. Based on this decoupling, the authors propose an improved version of the KL loss. In the context of knowledge distillation, the proposed method addresses issues with the student's wMSE loss. It also adjusts the weight term to avoid being determined by incorrect labels, instead using supervision. The proposed method is tested in two scenarios where KL divergence is used: knowledge distillation and adversarial training, achieving higher performance than existing methods in both cases.", "questions": "- The overall readability of the paper needs improvement. Figures should clearly illustrate the proposed method and its benefits. The proof method in Eq. 5 should be more understandable.\n\n- The paper should include comprehensive comparisons with all current state-of-the-art methods, especially showing higher performance than existing logit-based KD methods.\n\n- The authors should clearly explain the problems arising from the $\\Delta n$ term not receiving gradients in Eq. 7 and the benefits of it receiving gradients.", "limitations": "The authors have adequately addressed some limitations of their work. There are no  negative societal impacts.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The paper provides a theoretical proof that KL divergence can be decoupled into two terms and improve original loss, which is a significant contribution.\n- The proposed method achieves higher performance than most of the existing methods in both knowledge distillation and adversarial training scenarios.", "weaknesses": "- The paper lacks comparisons with the latest state-of-the-art methods, particularly in the knowledge distillation benchmark and RobustBench. The performance is also inferior compared to methods like [1].\n- References are not provided for the methods mentioned, making it difficult to identify the specific works being referred to, which significantly hampers readability.\n- The overall readability of the paper is low, with figures and text that are not self-explanatory. For example, Figure 1 is unclear without reading the full explanation, and terms like $o_m$ and $o_n$ are not adequately explained until later in the text. Additionally, the term \"Inserting Class-wise Global Information\" used for the weight term replacement in Eq.5 is misleading and lacks a clear explanation.\n- The proof approach for decoupling KL divergence is not straightforward. Instead of deriving DKL from KL divergence, the authors present DKL first and then show its equivalence through gradients, which obscures the understanding of the derivation process.\n- The benefits and reasons for why the $\\Delta n$ term in Eq.7 should receive gradients are not well explained. The phrase \"potentially hurt performance\" is vague and misleading.\n\n[1] Multi-level Logit Distillation, Jin et al., CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and valuable comments. Here we provide our responses to address your concerns.\n\n*Q1: We achieve significant improvements over the baseline and state-of-the-art adversarial robustness on RobustBench*\n\n*For adversarial robustness*\n\n**First**, with the same experimental settings and only replacing the KL loss with the IKL loss, we significantly outperform our baseline TRADES by **1.63%** robustness under auto-attack. \n\n**Second**, without extra techniques from previous algorithms, we achieve **the state-of-the-art adversarial robustness on the public leaderboard --- RobustBench under both settings**: _with the basic augmentation strategy including random crop and horizontal flip, with advanced data augmentation or synthesized data_. \n\nWe only replace the KL loss with our IKL loss using the TRADES pipeline and thus our method is much more computationally efficient than LBGAT and ACAT, saving **33.3%** training time. Our results are summarized in **Table 35 in the rebuttal_tables.pdf**\n\n*For knowledge distillation*\n\nOn the knowledge distillation, for the mentioned work \"multi-level logit distillation`` [ref3], we observe that it uses a weak augmentation (_RandomResizedCrop_ and random horizontal flip) data batch and a strong augmentation (Autoaug) data batch for each training iteration. For fair comparisons, based on their open-sourced code (https://github.com/Jin-Ying/Multi-Level-Logit-Distillation), we also apply the same pipeline to our IKL loss. The experimental results are summarized in **Table 36 in the rebuttal_tables.pdf**. Under the same experimental setting, we achieve better performance, outperforming the work [ref3] by **0.35% (ResNet-34--ResNet-18) and 0.33% (ResNet-50--MobileNet) on the ImageNet**.  \n\n[ref3] Multi-level Logit Distillation. CVPR 2023.\n\n*Q2: About the references, terms, and Figure 1.*\n\nThanks for your suggestions. We would add citations properly, revise Figure 1 for a clear explanation, and define the terms systematically in the new version.\n\n*Q3: The term \"Inserting Class-wise Global Information\" used for the weight term replacement in Eq.(5) is misleading and lacks a clear explanation.*\n\nIn Eq (5), we don't discuss the global class-wise information. We first propose Theorem 1 and identify that the DKL loss can suffer from sample-wise biases because the $w$MSE component depends on the sample-wise prediction scores. Especially in adversarial training, the natural accuracy is only around 60% on CIFAR-100. There can be many hard examples or outliers that cause sample biases. The KL loss is equivalent to DKL loss in terms of gradient optimization. Thus the KL loss also suffers from this problem. Then we address this potential issue in Section 3.3 by introducing the global class-wise information. For more details, please refer to our analysis in Section 3.4. \n\n*Q4: The proof approach for decoupling KL divergence is not straightforward. Instead of deriving DKL from KL divergence, the authors present DKL first and then show its equivalence through gradients, which obscures the understanding of the derivation process.*\n\nIn Section 3.2, we first analyze the gradient information of KL divergence loss regarding $o_{m}$ and $o_{n}$.\nThen based on the structured gradient information,  we construct the DKL formulation and propose Theorem 1. Finally, we give our proof for the equivalence between KL divergence loss and our DKL loss in the Appendix. \n\n*Q5: The benefits and reasons for why the term in Eq. (7) should receive gradients are not well explained.*\n\nPlease refer to **our global rebuttal Q3**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cc9Er154V1", "reviewer": "Reviewer_hZqZ", "summary": "In this paper, the authors analyzed the optimization gradient of the commonly used KL Divergence loss metric in adversarial training and distillation. KL loss can be reformulated as a Decoupled KL (DKL) Divergence loss term through antiderivative operations. The gradients of both terms are equivalent in the case where the parameters of the DKL are set to $\\alpha$ = 1 and $\\beta$ = 1. The DKL loss term consists of  1) a weighted MSE (wMSE) loss term, which encourages similarity between output logits, and 2) a CE loss term, which encourages accurate predictions. The authors highlighted two potential drawbacks of the KL/DKL formulations : 1) asymmetric optimization of both loss terms, 2) sample-wise conditioning term $w_m$ which induces strong variance into the optimization process. The authors proposed two modifications to the DKL loss term, namely, 1) Enabling the gradients of $\\Delta n$ instead of $\\Delta m$ in the formulation, thereby overcoming asymmetric optimization, 2) using global class-wise conditioning term $\\bar{w_y}$, conditioned on global ground truth, instead of $w_m$. The novel Improved KL Divergence Loss (IKL) is shown to be quantitatively and qualitatively superior to the classic and decoupled KL divergence terms. The authors also performed thorough adversarial robustness and distillation experiments, achieving improved performance compared to accuracy and robustness performance in various natural image datasets ( CIFAR-10, CIFAR-100, ImageNet ), augmentation strategies , and network backbones, outperforming multiple benchmarks.", "questions": "1. Does the assumption of DKL/KL equivalence in Theorem 1 still hold when the parameters $\\alpha$ and $\\beta$ are not set to 1? The authors have experimented with the values of these parameters while assuming that the DKL/IKL formulation still results in equivalent gradient formulations.\n2. While various recent works have highlighted the importance of solving the gradient conflict issue for accuracy/robustness tradeoff in various settings. Some works even proposed that anti-symmetric gradients are essential to solving the gradient conflicts. How can the authors justify their approach compared to these opposing approaches/perspectives?\n - Zhang, C., Zhang, K., Zhang, C., Pham, T. X., Yoo, C. D.,\nand Kweon, I. S. How does simsiam avoid collapse without negative samples? a unified understanding with selfsupervised contrastive learning. International conference on learning representations, 2022. https://arxiv.org/abs/2203.16262\n - Zhuo, Z., Wang, Y., Ma, J., and Wang, Y. Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. International conference on learning representations, 2023.\nhttps://arxiv.org/abs/2303.02387\n- Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824–5836, 2020.\nhttps://arxiv.org/abs/2001.06782\n- Wen, Z. and Li, Y. The mechanism of prediction head in noncontrastive self-supervised learning. Advances in Neural\nInformation Processing Systems, 35:24794–24809, 2022.\nhttps://arxiv.org/abs/2205.06226\n- Waseda, F. et al. Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off. https://arxiv.org/abs/2402.14648, 2024.", "limitations": "The authors have not addressed the limitations of their work. Please provide some potential limitations that can be improved on for future works.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Experiments : Thorough experiments were performed, incorporating various datasets, network backbones, and augmentation strategies.\n2. Clarity : The intuition behind the main concept is clear and straightforward. Readers can easily follow the thought process and justifications of the authors.\n3. Originality : The paper shows the promising potential of IKL in two different application domains, thus highlighting the IKL metric's relative flexibility. Furthermore, the hyperparameter ablations suggest that one can tune the predictive accuracy and adversarial robustness tradeoff. The competitive Robustbench results further substantiate the authors' claims.", "weaknesses": "1. Theoretical evaluation is not convincing enough. There are no further justifications for why the specific anti-derivative formulation expressed in Theorem 1 is chosen as the main decoupled KL (DKL) formulation, as there could be multiple anti-derivative formulations for the same function that also lead to equivalent derivative formulations. The mathematical proof provided in the appendix has merely shown that performing chain rules on KL/DKL loss terms justifies the equivalence of the derivatives. \n2. While the authors have intuitively highlighted the issues of the unbalanced and potentially unstable optimization issues with DKL, the authors did not delve deep into the aforementioned issues. Further theoretical proofs or gradient analysis of KL/DKL/IKL optimization would provide a stronger justification, especially where exactly the stop-gradient operations are crucial. The following papers have performed gradient analysis and a thorough study into stop-gradient mechanisms. \n - Zhang, C., Zhang, K., Zhang, C., Pham, T. X., Yoo, C. D.,\nand Kweon, I. S. How does simsiam avoid collapse without negative samples? a unified understanding with selfsupervised contrastive learning. International conference on learning representations, 2022. https://arxiv.org/abs/2203.16262\n - Zhuo, Z., Wang, Y., Ma, J., and Wang, Y. Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. International conference on learning representations, 2023.\nhttps://arxiv.org/abs/2303.02387\n3. The accuracy and robustness results are generally only marginally better than the benchmarks. t-SNE visualization of the purported benefits of the IKL formulation does not additionally convey a strong qualitative advantage. The authors highlighted the lower computational costs of the approach in some of their experiments. I suggest that the authors pivot more on this to demonstrate that the IKL is a competitive, relatively computationally lighter method.\n\nMinor : While the structure of the paper is clear, there are multiple grammatical and expression errors. I would recommend stronger proofreading and reformulation for improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and valuable comments. Here we provide our responses to address your concerns.\n\n*Q1: Theoretical evaluation is not convincing enough. There are no further justifications for why the specific anti-derivative formulation expressed in Theorem 1 is chosen as the main decoupled KL (DKL) formulation, as there could be multiple anti-derivative formulations for the same function that also lead to equivalent derivative formulations.*\n\nWe decouple the KL divergence loss into a $w$MSE component and a Cross-entropy component because MSE and Cross-entropy loss are the two most popular objectives for image recognition. There are lots of loss functions that can be used for image classification, like Mean Absolute Error and Hinge loss. However, Cross-entropy loss is the most effective one for gradient optimization. Similarly, MSE loss has also been validated as one of the most effective loss functions in many tasks like object detection and other regression tasks.  \n\nWith Theorem 1, we know that the KL divergence loss can be decoupled into a $w$MSE and a Cross-entropy loss. Then **we can examine how each component affects model performance, which cannot be done by directly operating on KL divergence loss.**\n\n*Q2:  While the authors have intuitively highlighted the issues of the unbalanced and potentially unstable optimization issues with DKL, the authors did not delve deep into the aforementioned issues. Further theoretical proofs or gradient analysis of KL/DKL/IKL optimization would provide a stronger justification, especially where exactly the stop-gradient operations are crucial.*\n\nThe **_stop-gradient_** operation is just used to ensure that our DKL loss (Theorem 1) is equivalent to the KL divergence loss from the perspective of gradient optimization, which implies that KL loss is a special case of our DKL/IKL loss.\n \nBy examining each component of DKL loss, we identify the potential issues of DKL loss, i.e., the asymmetric gradient property and possible sample-wise biases from hard examples or outliers. **Since the DKL loss is equivalent to the KL loss, we consider that the KL loss also suffers from the two problems.** Then we propose the IKL loss to address the potential issues.\n\n\n*Q5: We achieve significant improvements over the baseline and state-of-the-art adversarial robustness on public leaderboard --- RobustBench.*\n\n**First**, with the same experimental settings and only replacing the KL loss with the IKL loss, we significantly outperform our baseline TRADES by **1.63%** robustness under auto-attack.\n\n**Second**, without extra techniques from previous algorithms, we achieve **the state-of-the-art adversarial robustness on the public leaderboard --- RobustBench under both settings**: _with the basic augmentation strategy including random crop and horizontal flip, with advanced data augmentation or synthesized data_. \n\nWe only replace the KL loss with our IKL loss using the TRADES pipeline and thus our method is much more computationally efficient than LBGAT and ACAT, saving **33.3%** training time. Our results are summarized in **Table 35 in the rebuttal_tables.pdf**.\n\n*Q6: Does the assumption of DKL/KL equivalence in Theorem 1 still hold when the parameters $\\alpha=1$ and $\\beta=1$ are not set to 1?*\n\nDKL loss is equivalent to KL loss only when $\\alpha=1$ and $\\beta=1$.\nThe significance of our Theorem 1 is that: **the KL loss can be decoupled into a weighted MSE loss and a Cross-entropy loss. Then the two terms can be adjusted independently for efficient optimization, which can not be achieved by directly tuning hyper-parameters of KL loss**. \n\n*Q7: Various recent works have highlighted the importance of solving the gradient conflict issue for the accuracy/robustness trade-off in various settings. Some works even proposed that anti-symmetric gradients are essential to solving gradient conflicts. How can the authors justify their approach compared to these opposing approaches?*\n\nThat's an interesting question. \nFirst, **it is how the gradients affect training optimization matters rather than the property of symmetric or asymmetric.**\nIn our case, the asymmetric gradient property of KL divergence loss causes the $w$MSE loss not to work under scenarios like knowledge distillation, leading to optimization problems during training (please refer to **our global rebuttal Q3**). Thus, we break the asymmetric property and observe performance improvements on several tasks including KD, AT, and semi-supervised learning (please refer to **our global rebuttal Q1**).\n\nSecond, as mentioned by you, some work like [ref4] proposes that anti-symmetric gradients can solve gradient conflicts between invariance loss and classification loss, and promote model robustness. In the paper, the key is still to remove the harmful gradients. The asymmetric gradient optimization is just one of the ways to solve gradient conflicts in this case. Thus, we should always pay attention to how the gradients affect the optimization process rather than the symmetric or asymmetric properties of gradients. I think it is a promising direction for deep learning optimization not only in robustness. I will include the relevant discussion in the new revision. The mentioned works on _stop gradient_ for unsupervised representation learning will also be discussed.\n \n[ref4] Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off. arXiv:2402.14648. \n\n[ref5] How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning. ICLR 2022.\n\n[ref6] Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism. ICLR 2023.\n\n[ref7] The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning. NeurIPS 2022.\n\n[ref8] Gradient surgery for multi-task learning. NeurIPS 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "07Tghg5EM6", "reviewer": "Reviewer_wpCy", "summary": "The paper investigates the Kullback-Leibler (KL) Divergence loss and demonstrates its equivalence to the Decoupled Kullback-Leibler (DKL) Divergence loss in a limited setting, which consists of a weighted Mean Square Error (wMSE) and a Cross-Entropy loss with soft labels. By addressing the asymmetric optimization property of KL/DKL loss and incorporating class-wise global information, the authors propose the Improved Kullback-Leibler (IKL) Divergence loss. Experimental evaluations on CIFAR-10/100 and ImageNet datasets highlight the effectiveness of IKL loss in enhancing adversarial robustness and knowledge distillation tasks.", "questions": "1. Regarding the asymmetric problem, what is the relation between the proposed DKL and the Jensen-Shannon (JS) divergence?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The writing is clear and easy to understand.", "weaknesses": "1. It should be explicitly clarified that Theorem 1 and the subsequent analysis are solely based on the assumption that the probability distribution is a categorical distribution.\n2. Since the author claims that the proposed method mainly applies to adversarial training and knowledge distillation, a formal connection between these two methods and the KL divergence should be provided.\n3. The KL divergence can be applied to a much wider research area since it is a convenient measure between distributions. The author provides no relevant works regarding the KL divergence and other statistical distances, raising questions about whether the author has a comprehensive understanding of the KL divergence.\n4. Additionally, the related works section is perplexing. The author spends most of the paper describing the KL divergence without defining adversarial training and knowledge distillation, yet the related works section focuses entirely on these two areas.\n5. Lines 150-152 state, 'The asymmetric optimization can cause the wMSE component to be neglected or overlooked when $o_m$ is detached from gradient backpropagation, which is the case for knowledge distillation, potentially leading to performance degradation.' This should be a central investigation target of the paper. The author should provide either empirical or theoretical evidence to support this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and valuable comments. Here we provide our responses to address your concerns.\n\n*Q1: It should be explicitly clarified that Theorem 1 and the subsequent analysis are solely based on the assumption that the probability distribution is a categorical distribution.*\n\nThanks for your suggestions. We will explicitly clarify that our analysis is based on the assumption that the probability distribution is a categorical distribution in the new revision.\n\n*Q2: The connection between KL loss and related tasks.*\n\nThanks for the suggestion. We have described the connections between KL loss and the adversarial training and knowledge distillation in Sec. 3.1. For clarification, We will re-organize the section for related work and Section 3.1 and include other related applications of the KL divergence loss in the new revision.\n\nMoreover, our paper focuses on the applications of the KL divergence loss from the gradient optimization perspective rather than the measurement between distributions. We delve into how KL divergence loss optimizes models during training in terms of gradient optimization. \n\n*Q3: New applications of our IKL loss on semi-supervised learning,  knowledge distillation on imbalanced data, and semantic segmentation knowledge distillation.*\n\nBesides the adversarial training task and knowledge distillation on balanced data, We conduct experiments on other tasks including semi-supervised learning, knowledge distillation on imbalanced data, and semantic segmentation knowledge distillation. Experimental results show that our IKL loss can significantly improve model performance when compared with KL loss. **On semi-supervised learning**, replacing the consistency loss with our IKL loss in the Mean-Teacher algorithm, we achieve **2.67%** improvements. **For the knowledge distillation on imbalanced data**, replacing the KL loss with our IKL loss, our model outperforms the baseline model by **1.44%** top-1 accuracy. **For the semantic segmentation distillation**, replacing the KL loss with our IKL loss, we achieve **0.5 mIoU** gains on the ADE20K dataset. \n\nFor details, please refer to **our global rebuttal Q1**.\n\n*Q4: Why do we need to break the asymmetric property of KL loss in scenarios like knowledge distillation?*\n\nPlease refer to **our global rebuttal Q3**.\n\n*Q5: Regarding the asymmetric problem, what is the relation between the proposed DKL and the Jensen-Shannon (JS) divergence?*\n\nThat's a good question. With the following JS divergence loss,\n\n$\\quad \\quad \\quad JSD(P||Q) = \\frac{1}{2} KL(P|| M) + \\frac{1}{2} KL(Q||M), \\quad M=\\frac{1}{2} P + \\frac{1}{2} Q.$ \n\nSuppose $P$ and $Q$ are probability distributions of the teacher model and the student model respectively. We calculate its derivatives regarding $\\mathbf{o}_{n}$ (the student logits),\n\n$\\quad \\quad \\quad \\frac{\\partial \\mathcal{L}_{JSD}}{\\partial \\mathbf{o}\\_{n}^{i}}=\\sum\\_{j=1}^{C} \\mathbf{w}\\_{\\mathbf{n}}^{i,j}(\\Delta n\\_{i,j} - \\Delta \\mathbf{m}^{'}\\_{i,j})$, with the constraint $\\quad Softmax(\\mathbf{o}\\_{m^{'}}) = \\frac{1}{2} \\mathbf{s}\\_{n} + \\frac{1}{2} \\mathbf{s}\\_{m}$,\n\nwhere $\\mathbf{o}\\_{m}$ is the logits from the teacher model, $\\mathbf{o}\\_{m^{'}}$ is a virtual logits satisfying the constraint, $\\mathbf{s}\\_{m} = \\textit{Softmax}(\\mathbf{o}\\_{m})$, $\\mathbf{s}\\_{n}= \\textit{Softmax}(\\mathbf{o}\\_{n})$, \n$\\Delta \\mathbf{m}^{'}\\_{i,j} = \\mathbf{o}\\_{m^{'}}^{i} -  \\mathbf{o}\\_{m^{'}}^{j}$, $\\Delta \\mathbf{n}\\_{i,j} = \\mathbf{o}\\_{n}^{i} -  \\mathbf{o}\\_{n}^{j}$, $\\mathbf{w}\\_{n}^{i,j}=\\mathbf{s}\\_{n}^{i} * \\mathbf{s}\\_{n}^{j}$.\n\nCorrespondingly, the derivatives of IKL loss regarding $\\mathbf{o}_{n}$ (the student logits),\n\n$\\quad \\quad \\frac{\\partial \\mathcal{L}\\_{IKL}}{\\partial \\mathbf{o}\\_{n}^{i}} = \\underbrace{\\alpha \\sum\\_{j=1}^{C}  \\mathbf{\\bar w}\\_{m}^{i,j}(\\Delta \\mathbf{n}\\_{i,j} - \\Delta \\mathbf{m}\\_{i,j})}\\_{Effects \\ \\ of \\ \\ wMSE \\ \\ loss} + \\underbrace{\\beta \\left( \\mathbf{s}\\_{m}^{i} * (\\mathbf{s}\\_{n}^{i} - 1) + \\mathbf{s}\\_{n}^{i} * (1 - \\mathbf{s}\\_{m}^{i}) \\right)}\\_{Effects \\ \\ of \\ \\ Cross-Entropy \\ \\ loss}$\n\nAlthough JSD can produce gradients on $\\mathbf{o}\\_{n}$, it has lost the property of KL loss that forces $\\mathbf{o}\\_{n}$ and $\\mathbf{o}\\_{m}$ to be similar **_absolutely_** in the Euclidean space and **_relatively_** in the _Softmax_ space.\n\nFirst, the soft labels from the teacher models often embed dark knowledge and facilitate the optimization of the student models. However, there are no effects of the cross-entropy loss with the soft labels for JSD loss.\n\nSecond, there could be multiple virtual $\\mathbf{o}^{'}\\_{m}$ satisfying the above constraint ($Softmax(\\mathbf{o}\\_{m^{'}}) = \\frac{1}{2} \\mathbf{s}\\_{n} + \\frac{1}{2} \\mathbf{s}\\_{m}$). Thus, JSD loss is still to make $\\mathbf{o}\\_{m}$ and $\\mathbf{o}\\_{n}$ to be similar **_relatively_** in the softmax space and suffers from the same problem of KL divergence in scenarios like knowledge distillation. \n\nWe also empirically validate that IKL loss achieves better performance on the knowledge distillation task. As shown in **Table 33 in the rebuttal_tables.pdf**, the model trained with IKL loss surpasses the models trained by KL and JSD losses by **1.44%** and **2.03%** top-1 accuracy respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xWjQMNspsb", "reviewer": "Reviewer_RheX", "summary": "The paper proposed the Decoupled Kullback-Leibler Divergence Loss (DKL), mathematically equivalent to the KL divergence. To solve the asymmetries issue of KL/DKL, the authors introduce the Improved KL divergence, which consists of class-wise global information and cross-entropy loss for soft labels. The proposed approach achieves new state-of-the-art adversarial robustness on the public leaderboard.", "questions": "1. The first terms in Eq5, 6, and 7 are slightly different, but the names are the same; could you please clarify this?\n2. In Table 1, the GI seems more important than BA; could you please explain it and add the results for only the use of GI?\n3. The asymmetric issue can be simply solved by using symmetric KL divergence by 1/2(KL(p,q)+KL(q,p)) or Jensen–Shannon divergence. Could you explain the advantages of enabling the gradients proposed in IKL?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of this paper is clear.\n2. The results on adversarial robustness show the effectiveness of the proposed method.", "weaknesses": "The empirical study is not enough to show the effectiveness of the IKL compared with KL divergence. \n\n1. In the context of adversarial robustness, the author's approach of replacing KL in the DM-AT and comparing it with a few other baseline methods is insightful. However, to further validate the effectiveness of IKL compared with KL, it would be beneficial to see this approach applied to other methods such as TRADES or MART [1].\n2. The current adversarial attack experiments, which focus solely on the AA attack is not enough. It would be valuable to see more results with different attacks, such as PGD and CW, to better understand the robustness of the IKL.\n3. Regarding knowledge distillation, I wonder about its performance in the semi-supervised setting. Could you please add the experiments for comparing with Mean-Teacher [2] and provide a toy example to show the effectiveness of the IKL compared with KL? \n\n[1] Improving Adversarial Robustness Requires Revisiting Misclassified Examples.\n[2] Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions and valuable comments. Here we provide our responses to address your concerns.\n\n*Q1: Ablation with TRADES and MART.*\n\nThanks for your suggestion. TRADES is our baseline. We have already listed the results in Table 1 of the main paper. To highlight the significant improvements from our IKL loss, here we present the experimental results with TRADES and MART on CIFAR-100. Equipped with IKL loss, the robustness is enhanced for both methods. We transfer the hyper-parameters from TRADES to MART. Carefully tuning the hyper-parameters can potentially further improve performance. We would include the comparisons in the new revision.\n\n| Method | Acc | AA |\n| :---: | :---: | :---: |\n| KL (TRADES) | 62.87 | 30.29 |\n| IKL (Ours) | **63.40(+0.63)** | **31.92(+1.63)** |\n| IKL (Ours\\*)| **66.51(+3.64)** | **31.43(+1.14)** |\n| --- | --- | --- |\n| MART | 66.10 | 30.65 |\n| MART with IKL (Ours) | 64.44 | **31.22(+0.57)** |\n\n*Q2: Evaluation under PGD and CW attacks.*\n\nAs suggested, we also test the robustness of our models under PGD, and CW attacks with 10 and 20 iterations. The perturbation size and step size are set to 8/255 and 2/255 respectively. As shown in the Table, with increasing iterations from 10 to 20, our models show similar robustness, demonstrating that our models don't suffer from the obfuscated gradients problem.\n\n| Method | Acc | PGD-10 | PGD-20 | CW-10 | CW-20 | Auto-Attack |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| KL (TRADES) | 62.87 | 36.01 | 35.84 | 40.03 | 39.86 | **30.29** |\n| IKL (Ours) | 63.40 | 36.78 | 36.55 | 40.72 | 40.47 | **31.92** | \n\nThe worst-case is in **bold**.\n\n*Q3: Applications of IKL loss on semi-supervised learning.*\n\nPlease refer to **our global rebuttal Q1**.\n\n*Q4: The first terms in Eq5, 6, and 7 are slightly different, but the names are the same; could you please clarify this?*\n \nEq.(5) is equivalent to KL loss regarding gradient optimization.\nIn Eq. (6), the _stop gradient_ operation in $\\mathcal{S}(\\Delta \\mathbf{m})$ is applied because the teacher model is fixed and thus its outputs $o_{m}$ are detached from the gradient backpropagation during distillation.\nWith Eq. (6), we know that the $w$MSE component will have no effects on optimization in knowledge distillation. This issue is caused by the asymmetric property of KL loss. Thus we propose to break the asymmetric gradient property by enabling the gradients of $\\Delta \\mathbf{n}$ in Eq. (5) and Eq. (6). As a result, we derive the Eq. (7).\n\n\n*Q5: In Table 1, the GI seems more important than BA; could you please explain it and add the results for only the use of GI?*\n\nActually, GI and BA work together for adversarial training. Adversarially-trained models only achieve around 60% natural accuracy on CIFAR-100. The low performance can lead to heavy sample-wise biases for $w$MSE during training. GI can properly address this problem. Additionally, we observe that,  without the BA, the model with GI is easy to crush during adversarial training. BA mechanism can make the training stable and efficient. \n\n*Q6: Regarding the asymmetric issue, what are the advantages of IKL loss over JSD loss?*\n\nThat's a good question. With the following JS divergence loss,\n\n$\\quad \\quad \\quad JSD(P||Q) = \\frac{1}{2} KL(P|| M) + \\frac{1}{2} KL(Q||M), \\quad M=\\frac{1}{2} P + \\frac{1}{2} Q.$ \n\nSuppose $P$ and $Q$ are probability distributions of the teacher model and the student model respectively. We calculate its derivatives regarding $\\mathbf{o}_{n}$ (the student logits),\n\n$\\quad \\quad \\quad \\frac{\\partial \\mathcal{L}_{JSD}}{\\partial \\mathbf{o}\\_{n}^{i}}=\\sum\\_{j=1}^{C} \\mathbf{w}\\_{\\mathbf{n}}^{i,j}(\\Delta n\\_{i,j} - \\Delta \\mathbf{m}^{'}\\_{i,j})$, with the constraint $\\quad Softmax(\\mathbf{o}\\_{m^{'}}) = \\frac{1}{2} \\mathbf{s}\\_{n} + \\frac{1}{2} \\mathbf{s}\\_{m}$,\n\nwhere $\\mathbf{o}\\_{m}$ is the logits from the teacher model; $\\mathbf{o}\\_{m^{'}}$ is a virtual logits satisfying the above constraint; $\\mathbf{s}\\_{m} = \\textit{Softmax}(\\mathbf{o}\\_{m})$; $\\mathbf{s}\\_{n}= \\textit{Softmax}(\\mathbf{o}\\_{n})$; \n$\\Delta \\mathbf{m}^{'}\\_{i,j} = \\mathbf{o}\\_{m^{'}}^{i} -  \\mathbf{o}\\_{m^{'}}^{j}$; $\\Delta \\mathbf{n}\\_{i,j} = \\mathbf{o}\\_{n}^{i} -  \\mathbf{o}\\_{n}^{j}$; $\\mathbf{w}\\_{n}^{i,j}=\\mathbf{s}\\_{n}^{i} * \\mathbf{s}\\_{n}^{j}$.\n\nCorrespondingly, the derivatives of IKL loss regarding $\\mathbf{o}_{n}$ (the student logits),\n\n$\\quad \\quad \\frac{\\partial \\mathcal{L}\\_{IKL}}{\\partial \\mathbf{o}\\_{n}^{i}} = \\underbrace{\\alpha \\sum\\_{j=1}^{C}  \\mathbf{\\bar w}\\_{m}^{i,j}(\\Delta \\mathbf{n}\\_{i,j} - \\Delta \\mathbf{m}\\_{i,j})}\\_{Effects \\ \\ of \\ \\ wMSE \\ \\ loss} + \\underbrace{\\beta \\left( \\mathbf{s}\\_{m}^{i} * (\\mathbf{s}\\_{n}^{i} - 1) + \\mathbf{s}\\_{n}^{i} * (1 - \\mathbf{s}\\_{m}^{i}) \\right)}\\_{Effects \\ \\ of \\ \\ Cross-Entropy \\ \\ loss}$\n\nAlthough JSD can produce gradients on $\\mathbf{o}\\_{n}$, it has lost the property of KL loss that forces $\\mathbf{o}\\_{n}$ and $\\mathbf{o}\\_{m}$ to be similar **_absolutely_** in the Euclidean space and **_relatively_** in the _Softmax_ space.\n\nFirst, the soft labels from the teacher models often embed dark knowledge and facilitate the optimization of the student models. However, there are no effects of the cross-entropy loss with the soft labels for JSD loss.\n\nSecond, there could be multiple virtual $\\mathbf{o}^{'}\\_{m}$ satisfying the constraint ($Softmax(\\mathbf{o}\\_{m^{'}}) = \\frac{1}{2} \\mathbf{s}\\_{n} + \\frac{1}{2} \\mathbf{s}\\_{m}$). Thus, JSD loss is still to make $\\mathbf{o}\\_{m}$ and $\\mathbf{o}\\_{n}$ to be similar **_relatively_** in the softmax space and suffers from the same problem of KL divergence in scenarios like knowledge distillation. \n\nWe also empirically validate that IKL loss achieves better performance on the knowledge distillation task. As shown in **Table 33 in the rebuttal_tables.pdf**, the model trained with IKL loss surpasses the models trained by KL and JSD losses by **1.44%** and **2.03%** top-1 accuracy respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers' efforts and valuable comments on our paper. Here we respond to the most concerned questions from reviewers.

*Q1. New applications of our IKL loss in semi-supervised learning, knowledge distillation on imbalanced data, semantic segmentation knowledge distillation.*

Thanks for the suggestions. Besides the adversarial training task and knowledge distillation on balanced data, We conduct experiments on other tasks including semi-supervised learning, knowledge distillation on imbalanced data, and semantic segmentation knowledge distillation.

**Semi-supervised learning.** We use the open-sourced code from https://github.com/microsoft/Semi-supervised-learning and conduct semi-supervised experiments on CIFAR-100 with FixMatch and Mean-Teacher methods. Specifically, each class has 2 labeled images and 500 unlabeled images. All default training hyper-parameters are used for fair comparisons. We only replace the consistency loss with our IKL loss. As shown in **Table 32 in the rebuttal_tables.pdf**, with our IKL loss, the Mean-Teacher method even surpasses the FixMatch, outperforming the baseline by **2.67%**.

**Imbalanced data.** For the long-tailed recognition on ImageNet-LT, we train models 90 epochs with cross-entropy loss. We only preprocess images with _RandomResizedCrop_ and random horizontal flip.  All training settings are the same for fair comparisons. As shown in **Table 33 in the rebuttal_tables.pdf**, Models trained with our loss function achieve much better performance than the original KL loss, surpassing the baseline by **1.44\%**.


**Semantic segmentation distillation.**  As suggested, we conduct ablation on the semantic segmentation distillation task. We use the APD[ref1] as our baseline for their open-sourced code. All default hyper-parameters are adopted. We only replace the original KL loss with our IKL loss. As shown in **Table 34 in the rebuttal_tables.pdf**, we achieve **0.5 mIoU** performance gains with the IKL loss, demonstrating that the IKL loss can be complementary to other techniques in semantic segmentation distillation. 

[ref1] Adaptive Perspective Distillation for Semantic Segmentation. IEEE TPAMI 2022.

*Q2: Significant improvements from IKL loss and state-of-the-art robustness on the public leaderboard---RobustBench.*

**First**, with the same experimental settings and only replacing the KL loss with the IKL loss, we significantly outperform our baseline TRADES by **1.63\%** robustness under auto-attack. 

**Second**, without extra techniques from previous algorithms, we achieve **the state-of-the-art adversarial robustness on the public leaderboard --- _RobustBench_ under both settings**: _with the basic augmentation strategy including random crop and horizontal flip_, _with advanced data augmentation or synthesized data._ 

We only replace the KL loss with our IKL loss using the TRADES pipeline and thus our method is much more computationally efficient than LBGAT and ACAT, saving **33.3\%** training time. Our results are summarized in **Table 35 in the rebuttal_tables.pdf**

*Q3: Why do we need to break the asymmetric property of KL loss in scenarios like knowledge distillation?*

**_First_**, the cross-entropy loss is invariant to the mean value shift. Suppose $\mathbf{v} \in \mathbb{R}^{C}$ and $\gamma \in \mathbb{R}$, then the following equation is established:

$\quad \quad \mathcal{L}=-\log\frac{e^{v_{y}}}{\sum_{i=1}^{C} e^{v_{i}}} = -\log\frac{e^{v_{y}+\gamma}}{\sum_{i=1}^{C} e^{v_{i}+\gamma}} $,
where $\mathbf{v}_{y}$ is the $y$th value of $\mathbf{v}$.

**_Second_**,  for tasks like the adversarial training, KL loss forces two outputs to be similar **_absolutely and relatively_**.
Recall the derivatives of KL loss regarding $o_{m}$ and $o_{n}$ in Eq.(3) and (4) of main paper,

$\quad \frac{\partial \mathcal{L}\_{KL}}{\partial \mathbf{o}\_{m}^{j}} = \sum\_{k=1}^{C} ((\Delta \mathbf{m}\_{j,k} -\Delta \mathbf{n}\_{j,k}) * (\mathbf{s}\_{m}^{k} * \mathbf{s}\_{m}^{j}))  \quad Eq. (3) $,

$\quad \frac{\partial \mathcal{L}\_{KL}}{\partial \mathbf{o}\_{n}^{j}} = \mathbf{s}\_{m}^{j} * (\mathbf{s}\_{n}^{j} -1 ) + \mathbf{s}\_{n}^{j} * (1-\mathbf{s}\_{m}^{j}) \quad Eq. (4) $.

Eq.(3) indicates that the KL loss encourages $o_{m}$ to be similar to $o_{n}$ **_absolutely_** in the Euclidean space. 
Eq.(4) indicates that the KL loss encourages $o_{n}$ to be similar to $o_{m}$ **_relatively_** in the _Softmax_ space. The two items work collaboratively and drive the optimization together. 

**_Third_**, for the knowledge distillation, the teacher model is well-trained and fixed during the distillation process. Thus, $o_{m}$ will be detached from the gradient backpropagation, and Eq.(3) takes no effects during the training. As a result, the KL divergence loss only forces $o_{m}$ and $o_{n}$ to be similar **_relatively_** in _Softmax_ space.

This optimization can cause a problem: For a two-class classification task, after the knowledge distillation with an ideal teacher model, we derive the logits $a_{1}=[1,2]$, $a_{2}=[0.5,1]$, and $a_{3}=[0.3, 0.2]$ for inputs $x_{1}$ and $x_{2}$ and $x_{3}$ individually. With their logits, we known that $x_{1}$ and $x_{2}$ belong to class-2 while $x_{3}$ is in class-1. However, in Euclidean space, we observe that the distance between $a_{2}$ and $a_{3}$ is even smaller than the distance between $a_{1}$ and $a_{2}$, which contradicts our intuition that images in the same class should have smaller distance than images in different classes.

Our IKL loss can address this problem by enabling the gradient of $\Delta \mathbf{n}$ in DKL loss, thus forcing the output from the student model to be similar to the output from the teacher model **_absolutely_**. This operation breaks the asymmetric gradient property of KL loss and meanwhile keeps its good property that optimizes two outputs to be similar **_absolutely_** in the Euclidean space and **__relatively__** in the _Softmax_ space simultaneously.

Meta-review: Five experts in the field reviewed this paper. Their recommendations are 2 x Weak Accept, 2 x Borderline Accept, and a Borderline Reject. Overall, the reviewers appreciated the paper because it demonstrates that KL divergence loss can be decoupled into a weighted MSE term and a CE term with soft labels. The authors propose an Improved KL (IKL) based on this decoupling. The writing and motivation of this paper is clear. The paper also provides a comprehensive experimental validation on CIFAR-10/100 and ImageNet datasets, showing that the proposed IKL loss can provide state-of-the-art adversarial robustness and knowledge distillation. 

The reviewers initially raised significant concerns regarding the paper, but the author provided a detailed and insightful response that addressed most of these concerns. I recommend accepting this paper based on the reviewers’ feedback and the authors’ satisfactory rebuttal. Overall, the method is novel, and the experimental validation is convincing. However, the reviewers have raised some concerns that should be addressed in the final camera-ready version of the paper. In particular, the authors should improve their paper to: 
- Clarify the scope of their work. While the IKL loss shows promising empirical results on CIFAR-10/100 and ImageNet datasets, can these improvements generalize to other types of data or tasks?
- Explain the underlying principles that make the proposed approach effective in knowledge distillation.
- Clarify that Theorem 1 and the subsequent analysis are solely based on the assumption that the probability distribution is a categorical distribution. 
- Explain how the proposed method for improvement by aligning the student model with the teacher model absolutely in the Euclidean space differs from simply regressing the absolute logit values using L1/L2 loss for KD, and from: Sun et al., Logit Standardization in Knowledge Distillation, CVPR 2024.
- Provide an analysis of computational complexity associated with the IKL loss, which is important for resource-constrained environments.  

The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)