Enhancing Logits Distillation with Plug&Play Kendall's  $\tau$ Ranking Loss
Authors: Yuchen Guan
Runxi Cheng
Kang Liu
Chun Yuan
Venue: ICML 2025 poster (2025)
Keywords: Knowledge Distillation
Kendall's tau Coefficient
Ranking Loss
Primary area: general_machine_learning->supervised_learning
TL;DR: Plug & Play Ranking Loss for Logits Distillation Abstract:

Abstract: Knowledge distillation typically minimizes the Kullback–Leibler (KL) divergence between teacher and student logits. However, optimizing the KL divergence can be challenging for the student and often leads to sub-optimal solutions. We further show that gradients induced by KL divergence scale with the magnitude of the teacher logits, thereby diminishing updates on low-probability channels. This imbalance weakens the transfer of inter-class information and in turn limits the performance improvements achievable by the student. To mitigate this issue, we propose a plug-and-play auxiliary ranking loss based on Kendall’s $\tau$ coefficient that can be seamlessly integrated into any logit-based distillation framework. It supplies inter-class relational information while rebalancing gradients toward low-probability channels. We demonstrate that the proposed ranking loss is largely invariant to channel scaling and optimizes an objective aligned with that of KL divergence, making it a natural complement rather than a replacement. Extensive experiments on CIFAR-100, ImageNet, and COCO datasets, as well as various CNN and ViT teacher-student architecture combinations, demonstrate that our plug-and-play ranking loss consistently boosts the performance of multiple distillation baselines.

Enhancing Logits Distillation with Plug& Kendall's T Ranking Loss
Yuchen Guan
Runxi Cheng"
Kang Liu'
Chun Yuan'
Abstract
Wrong Prodiction: figs
Prediction: dog
Knowledge distillation typically minimizes the Kullback-Leibler (KI.) divergence between teacher and student logits. However, optimiz- ing the KL divergence can be challenging for the student and often leads to sub-optimal solu- tions. We further show that gradients induced by KL divergence scale with the magnitude of the teacher logits, thereby diminishing updates on low-probability channels. This imbalance weak- ens the transfer of inter-class information and in turn limits the performance improvements achiev- able by the student. To mitigate this issue, we propose a plug-and-play auxiliary ranking loss based on Kendall's coefficient that can be seam- lessly integrated into any logit-base distillation framework. supplies inter-class relational infor- mation while rebalancing gradients toward low- probability channels. We demonstrate that the proposed ranking loss is largely invariant to chan- nel scaling and optimizes an objective aligned with that KL divergence, making a natu- ral complement rather than a replacement. Ex- tensive experiments on CIFAR-100 -00, ImageNet. and COCO datasets, well as various CNN and ViT teacher-student architecture combina- tions, demonstrate that our plug-and-play rank- ing loss consistently boosts the performance of multiple distillation baselines. Code is available at bectps://githh: www/Oveerigghiieeeoeeeeeeee RankingLoss-KD
Soft Labet deg
Aux 0.1222
0,4S
9.6
6A4
8:36
86
4-g
will
Fri
61
wil
Ape
Agg
milf
Hir
Stoutent
Student
Figure 1. Suboptimal Case of the KI. Divergence.
storage costs associated with complex networks limite their applicability. To address this, knowledge distillation (KD) has been proposed obtain performant lightweight mod- els. Typically, knowledge distillation involves well-trained heavy teacher model and untrained lightweight student model. The same data is fed to both the teacher and the student, with the teacher's outputs serving as soft labels for training the student. By constraining the student to pro- duce predictions that match the soft labels, the knowledge in the teacher model is transferred the lightweight student model.
Most logit-based KD methods adhere to the paradigm in- troduced by Hinton (Hinton et al., 2015), employing the Kullback-Leible (KL) divergence (Kullback & Leibler, 1951) to align the logits output by the student and teacher models:
C log
LKI(g' (1
Σ
9
E q
(I)
1. Introduction
The recent advancements in deep neural networks (DNN) have significantly enhanced performance in the field of com- puter vision. However, the heightened computational and
Where ge, 4s E RIXC' the prediction vectors of teacher and student. The optimization goal of KL divergence is to achieve identical outputs between the student and teacher, thereby transferring much knowledge as possible from the teacher the student. However, the optimization process of KI. divergence is not easy, as it is prone to suboptimal points, which can hinder further improvement in student performance. As illustrated in Fig. 1, compared to Student 2, Student I exhibits a smaller KL divergence with the teacher; however, Student achieves the correct classification result
'Equal contribution 'Tsinghua Shenzhen International Grad- uate School, Tsinghua University, Shenzhen, China. Correspon- dence to: Kang Liu <liuk22@ meiemmmmillssimmi wsw.aii..siiguu............ Chun Yuan <yuanc sz.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
B3: Fixing relative channel order helps the model escape suboptimal solutions.
17500
Student
15000
Therefore, we propose to constrain the channel ranking similarity between student and teacher. We construct a plug-and-play ranking loss function based on Kendall's r Coefficient. This ranking loss can supplement the attention to low low-probability channels in the logits and provide inter- class relationship information. We demonstrate that the gradients provided by KL divergence are related to channel scale, whereas the proposed ranking loss is not. We also show that the optimization objective of the proposed ranking loss is consistent with that of KL divergence, and visualize the loss landscape to illustrate that the ranking loss helps avoid suboptimal points in the early stage and does not alter the optimization objective in the end. Extensive experiments CNN and ViT show that the proposed plug-and -nn--lly ranking loss can enhance the performance of various logit- based methods. In conclusion, our contributions are as follows:
12500
Number
10000
7500
5000
2500
0-
Logits
Figure 2. Most Channels in the Distribution are Low- Pro-arrbbbilityy Channels.
consistent with the teacher, while Student 1 does not. This indicates that the optimization direction of KI divergence sometimes diverges from the task objective, leading students to suboptimal points.
We introduce plug-and-pll and-play ranking loss for assisting knowledge distillation tasks, addressing the issues of KL divergence's 's neglect of low- www-robabilltty channels and its tendency to fall into suboptimal points, while also learning inter-class relationship information to further enhance student performance.
Intuitively, and as preliminary observation, Eq. I shows that in the KL divergence each channel is weighted by the predicted probability of the teacher. Channels with small of therefore contribute marginally to the loss, causing the student tend to largely ignore them (the formal derivation is provided in Appendix A.5). In practice, most channels in single prediction have lower probabilities, as illustrated in Fig. 2. Neglecting these channels hampers knowledge transfer for two reasons:
We demonstrate that the gradients provided by KL di- vergence exhibit sensitivity to channel scales, while the proposed ranking loss remains more robust to such variations. We also prove and visualize that the op- timization objective of the proposed ranking loss consistent with that of KL divergence and helps avoid the suboptimal points of KL divergence.
RI: Gradient attenuation. Because oww-roobability channels carry tiny weights, the corresponding gradients are strongly damped, which makes alignment with the teacher network difficult..
Extensive experiments are conducted on variety of CNN and ViT teacher-student architectures using the CIFAR-100, ImageNet, and MS-COCO datasets. Our findings confirm the widespread effectivvneess of the proposed ranking loss in various distillation tasks, and its role as a alg-and-plaa auxiliary function provides substantial support for the training of distillation tasks.
R2: Impaired capture of inter-class information. Low- probability channels retain rich cross-category correlation information in the teacher logits: overlooking them deprives the student of these semantic correlations. For example, in Fig. 1, Student may not learn the distinction between the cat and fox classes
Considering these challenges, we aim to find an auxiliary method that mitigates the issues caused by channel scale differences, learns inter-class relationship information, and avoids the suboptimal points that KL. divergence might en- counter while maintaining the optimization objective of KL divergence. To address this problem, we propose ranking- based plug-and- play auxiliary loss. The benefits of imposing ranking constraints are as follows:
2. Related Works
Knowledge Distillation. Knowledge distillation, initially proposed by Hinton (Hinton et al., 2015), serves as a method for model compression and acceleration, aiming to transfer knowledge from heavy teacher model lightweight student model. By feeding the same samples, the teacher can produce soft labels, and training the student with these soft labels allows the transfer of knowledge from the teacher model to the student. Knowledge distillation tasks can be mainly divided into two categories: feature-base based distil- lation (Chen et al., 2022; Zhang & Ma, 2021; Yang et al.,
B1: Gradients from the ranking term depend only weakly on the absolute channel scale. B2: Ordered channel scores make cross-categon relation- ships explicit.
2

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
2022c; Guo et al., 2023; Park et al., 2019; Yang et al., 2022b) and logit-based distillation (Hinton al., 2015; Li et al., 2023; Zhao et al., 2022; Jin et al., 2023; Sun et al., 2024; Chi et al., 2023; Wen et al., 2021; Yang et al., 2023b). Feature- based distillation additionally utilizes the model's 's intermedi- ate features, providing more information to the student and enabling the student to learn at the feature level from the teacher, often resulting in better performance. Considering safety and privacy, the intermediate outputs of the model are often not obtainable; hence, logit-based methods that only use the model outputs for distillation offer better versatility and robustness. Our proposed method can act as a plug- and-play module added to logit-based methods, offering higher flexibility and further enhancing the performance of logit-based methods.
allowing for distillation solely through outputs when the internal structures of the student and teacher are invisible. For given dataset D, assuming there are C' categories and N samples, we possess a teacher model fi and stu- dent model fs. For a given sample I E 'D, we can obtain the outputs of the teacher and student model, denoted as =' = fe(T), = = (Z) where 2², E RIIC", Through softmax function with temperature, the outputs are pro- cessed into the prediction vectors qt. R IRICC finally:
explet/T) of EJ=1 caples/TTT d = exple;/T)
(2)
EJ= caplez/TT
where T is temperature parameter, 2 represents the logit value of the i-th channel of the model output, qi represents the predicted probability for the target being the i-th class. The KL divergence loss function used to constrain the stu- dent and teacher logits is of the following form:
Recent knowledge distillation methods have found that overly strict constraints can sometimes hinder the student's s transfer of knowledge from the teacher's soft labels. For instance, (Cao et al., 2022) discovered that differences in feature sizes in feature-based methods could limit the stu- dent's learning; while (Sun et al., 2024) found that using the same temperature for both teacher and student in logit-based methods could affect further improvements in student per- formance. To reduce the learning difficulty for the student, (Sun et al., 2024; Cao et al., 2022) provided methods to ease the logit matching difficulty between student and teacher, yet we find that methods providing additional guidance to the student have not been fully explored.
Cxild'l0 Σ de log(
It can be observed that in Eq. 3, the importance of the match between the student and teacher at the i-th channel is influenced by the coefficient q'. This implies that channels with smaller logit values receive less attention, leading to the KI. divergence's 's disregard for smaller channels.
Ranking Loss in Knowledge Distillation. For knowledge distillation, ranking loss is relaxed constraint that can pro- vide rich inter-class information. It was first applied in the distillation of recommendation systems (Reddi et al., 2021; Tang & Wang, 2018; Choi et al., 2021; Qin et al., 2023; Yang et al., 2022a), (Li et al., 2022) explored the application of ranking loss in object detection tasks, and (Gao et al., 2020) discussed the role of ranking in the distillation of language model tasks. However, the exploration of ranking loss in logit-based image classification task distillation is not yet comprehensive. We find that the KI. divergence used for distillation in classification tasks tends to over- look information from low-probability channels and may lead to suboptimal results. Our proposed method leverages ranking loss to balance the model's 's attention between high- probability and low-pobbayylity channels, while also using inter-class relationships to help the model avoid suboptimal outcomes.
4. Ranking Loss Based on Kendall's T Coefficient
In this section, we introduce the plug-and-play ranking loss function designed to constrain the ranking of channels in the logits output by the student model. With the aid of the ranking loss, the knowledge distillation task can balance the overemphasis on larger logit values and the neglect of smaller ones as measured by the KL divergence. Addition- ally, the ranking loss imposes a constraint on the leading channel value, which helps to correct the optimization di- rection and avoid suboptimal solutions. In Sec. 4.1, we will introduce the ranking loss function and employ Kendall's 7 coefficient to compute the ordinal consistency between the teacher and student logits. In Sec. 4.2, we discuss how ranking loss benefits optimizing logits distillation with KL divergence. Furthermore, we discuss the differentiable form of Kendall's T coefficient and, based on this, design three distinct forms of ranking loss functions in Appendix A.3.
3. Preliminary
4.1. Differentiable Kendall's 7 Coefficient
Most logit-based distillation methods adhere to the origi- nal KD proposed by Hinton (Hinton et al., 2015), which transfers knowledge by matching the logit outputs of the student and teacher. This setting is more generalizable,
In order to make the loss function pay more attention to the information provided by low cow-robbyylyity channels as well as to help correct the suboptimal problem in Fig. I, we introduce the ranking loss based on Kendall's r coefficient.
3

---

Enhancing Logits Distillation with Plag&Play Kendall's 7 Ranking Loss
By pairing each of the C' channels, we can obtain 2
C(C-1)
301
pairs, whose Kendall's coefficient is expressed as follows:
16"
Pa-Pa Pa C((C-)
(4)
where Pa represents concordant pairs and Pa represents dis- cordant pairs. Kendall's T coefficient provides an expression of ordinal similarity. For the teacher and student logits, a channel pair (i is considered concordant if the signs of (of - =5) for the teacher and - 2ÿ) for the student are the same; otherwise, the pair is discordant. We substitute the logits of the teacher and student in the following manner.
16
Logts
Figure 3. The Gradient of Ranking Loss and KL Divergence for Diffcrent Values of Logits.
En Ejes sgn(z) - -2,) -ggle2 - 25)
ranking loss is written as follows. The specific calculation process of the gradient can be found in the appendix A.5:
7
-C(C-1)
where sgn() represents sign function. While we have quan- tified the ordinal similarity between the teacher and student logits, the aforementioned formula is non-smooth. To utilize the ordinal similarity for gradient computation, we approxi- mate the sign function with the tanh function to convert it into a differentiable form:
0LKD dzş
-T (gi - g.)
(9)
0CRK Oz7
k
ttanh? (k(2f - -=]))
C(C' 1)
jti
tanh (M:22
Ei Ejes tanh(k (2)- - -2]) tanh(k (2f =2])
(10)
Td
]0(C 1)
The gradient scale of the KI. divergence influenced by the scales of the teacher and student models. Consequently, for outputs from low-probability channels, when the scales of the teacher and student are similar, the KL divergence's 'ss gradient for these components becomes negligible, leading to the neglect of information from low-probability channels. Although temperature scaling is typically employed to ad- dress this issue, it uniformly amplifies the gradients of all logits, thereby continuing to overlook the outputs of low- probability channels. In contrast, the gradients produced by ranking loss will be less affected by the scale of the teacher's logits. The gradient of a logit channel primarily depends on the difference between its rank and the target rank, ef- fectively harnessing the knowledge from low-probability channels. Therefore, In the early stage of training, ranking loss helps the model quickly learn the overall ranking of the teacher model, which helps the model converge to better initial solution faster.
2
ΣΣ
2
C(C-
1+e 2
jei
1+ 2- -9)
(6) where l is parameter that controls the steepness of the function; larger k causes the tanh function to more closely approximate the sign function. By negating the similarity measure in Eq. 6, it can serve as a loss function (bounded within [-1, 11) to enforce the consistency of the logits order between the teacher and student:
LRK- = -Td
The overall loss function is formulated as follows:
L aLkL + +BCCE + +Lkk
(8)
where LKL denotes KL divergence loss, Leg denotes Cross EossroEy loss, LnK denotes our ranking loss, and a, 3,3,7 7 are hyper-parameters.
In our study, we conduct a visualization of the gradients associated with both the ranking loss and the KL divergence loss in Fig 3. The logits of the student model are randomly generated but maintain the same scale as those of the teacher model across each channel. Our findings indicate that the ranking loss consistently provides gradients of similar scale across varying logit values, whereas the gradients from the KI. loss are heavily dependent on the specific logit values. This suggests that the ranking loss offers more uniform attention distribution across different logits compared to the
4.2. How Ranking Loss Benefits Optimizing Logits Distillation with KL Divergence
4.2. 1. FROM THE PERSPECTIVE OF GRADIENT: RANKING Loss CARES ABOUT NOW-PROBABILITY CHANNELS
In this section, we analyze why ranking loss cares about low- probability channels. The gradient of KI. divergence and
KL loss. Notably, the ranking loss assigns low loo-probyy gradients to logits with larger values, as they are the clas-
4

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
Table CCFAR-100 Hcterogeneous Architecture Results. The Top-1 Accuracy (%) is reported as the evaluation metric. The teacher and student have heterogeneous architecturss. We incorporate the proposed ranking loss into the existing logit-based methods, with the performance gains indicated in parentheses. The best and second best results are emphasized in bold and underlined.
ResNet32x4 79.42
ResNet32
ResNet50
ResNet32 79.42
WRN-40-2
Teacher
79.42
79.34
75.61
KD
WRN-16-2
WRN-40-2
MN-V2
SHN-VI
SHN-VI
Student
73.26
75.61
64.60
70.50
70.50
FitNet (Adriana et al., 2015)
74.70
77.69
63.16
73.59
73.73
CRD (Tian et al., 2019)
75.65
78.15
69.11
75.11
76.05
ReviewKD (Chen al., 2021)
76.11
78.96
69.89
77.45
77.14
DIST (Huang al., 2022)
75.58
78.02
68.66
76.34
76.00
LSKD (Sun et al., 2024)
77.53
79.66
71.19
76.48
76.93
KD (Hinton et al., 2015)
74.9
77.7
67.35
74.07
74.83
KD+Ours
75.18(+0.28)
78,50(00.80)
70.45(+3.10)
75.98(+1.91)
76.13(+1.30)
CTKD (Li al., 2023)
74.57
77.66
68.67
74.48
75.61
CTKD+Ours
75.71(+1.14)
78.61(+0.95)
70. 10.15((1,51)
76.67(+2.19)
76.80(+1.19)
DKD (Zhao al., 2022)
75.7
78.46
70.35
76.35
76.33
DKD+Ours
75.99(+0.29)
78.75(+0.29)
70.90(+0.55)
77.36(+1001
76.43(+0.10)
MLKD (Jin et al., 2023)
76.52 76.83(+0.31)
79.26
71.04
77.18
77.44
MI.KD+Ours
79.86(+0.60)
71.66(+0.62)
77.63(+0.45)
77.87(+0.43)
sification targets and reach the correct rank. Consequently, these logits receive less attention from the ranking loss.
4.2.3. FROM THE PERSPECTIVE OF CLASSIFICATION: RANKING Loss HELPS STUDENT CLASSIFY CORRECTLY
4.2.2. FROM THE PERSPECTIVE OF OPTIMAL SOLUTION DOMAIN: RANKING LOss WON'T INTERFERE WITH KL DIVERGENCE AT THE END
As illustrated in Fig. 1, using KL loss may lead to smaller loss yet result in incorrect classification. The tendency to fall into local optima will hinder the optimization process in distillation. Although the use of cross-cntropy loss can mitigate this issue, it does not incorporate information from the teacher model. Consequently, in experiments, cross- entropy loss is often assigned very small weight, which also leads to small gradient. For instance, KD (Hinton et al., 2015) LKI. = 0.1 0.9; DKD (Zhao et al., 2022) LCE : LTCKL LNCKL = -1-
In this section, we analyze the optimal solution domain for Ballback-Leibler (KL. divergence and ranking loss to show how ranking loss affects the optimization process of KL divergence. For KL divergence, the optimal solution domain is achieved when the distribution of the student model aligns perfectly with that of the teacher model, which is equivalent to a linear mapping on the logits: Vi
8; MLKD (Jin et al., 2023) LCE LxL = 0.1 : 9; LSKD (Sun et al., 2024) LCR : LxI. = 0.1 9. As a result, cases with lower KL loss but incorrect classification still frequently occur. In contrast, by aligning the rank between student and teacher, ranking loss helps KL divergence avoid such suboptimal situations. Also, the ranking loss can incorporate informa- tion from the teacher model. In this way, ranking loss helps students classify correctly, avoiding the suboptimal case in logit distillation. Furthermore, a model with better general- ization should have a more reasonable rank of logits. For instance, recognizing that tiger more similar to a cat than to a fish. By learning such inter-class relationships, the student can improve classification performance and enhance its representational capacity.
df o
Vi
=
(11)
of and of respectively represent the probability outputs of the teacher and the student for the i-th class, and represents any constant number. For the ranking loss, the optimal solution domain is such that for any two indices, the order of the logits output by the teacher and the student is consistent:
sgn(z)- - =2))- sgn(z - 22) =1 Vi,j
- (=f ==)) (zf - =)) Vi,j
(12)
- = F(+)) Vi where F'(x)) 0
It implies that the optimal solution domain for ranking loss is quite lenient and easily attainable, encompassing the opti- mal solution domain for KL divergence. Therefore, in the later stage of training, the ranking loss will not hinder the optimization of KL divergence, which means our ranking loss can perfectly be used as an auxiliary loss.
5. Experiments
Datasets. In order validate the efficacy and robustness of our proposed method, we conduct extensive experiments
5

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranninn Loss
Table CIFAR-100 Homogenous Architecture Results. The Top-1 Accuracy (%) reported as the evaluation metric. The teacher and student have homogeneous architectures. We incorporate the proposed ranking loss into the existing logit-based methods, with the performance gains indicated in parentheses. The best and second best results are emphasized in bold and underlined.
ResNet32x4
VGG13
WRN-40-2
ResNetl 10
Teacher
79.42
74.64
75.61
74.31
KD
Student
ResNet8 x4
VGG8
WRN-40-1
ResNet20
72.50
70.36
71.98
69.06
FitNet (Adriana et al., 2015)
73.50
71.02
72.24
68.99
CRD (Tian al., 2019)
75.51
73.94
74.14
71.46
Re (Chen et al., 2021)
75.63
74.84
75.09
71.34
DIST (Huang et al., 2022)
76.31
73.80
74.73
71.40
LSKD (Sun et al., 2024)
78.28
75.22
75.56
72.27
KD (Hinton et al., 2015)
73.33
72.98
73.54
70.67
KD+Ours
74.74(+1.41)
74.14(+1.16)
74.49(+0.95)
71.09(+0.42)
CTKD (Li et al., 2023)
73.39
73.52
73.93
70.99
CTKD+Ours
75.59(42.2)
74.76(+1.24)
74.86(+0.93)
71.08 71.08(+0.09)
DKD (Zhao et al.. 2022)
76.32
74.68
74.81
71.06
DKD+Ours
76.61(+0.29)
75.1(+0.42)
74.94 (+0.13)
71.84(+0.78)
MLKD (Jin et al., 2023)
77.08
75.18
75.35
71.89
MLKD+Ours
77.25(+0.17)
75.35(+0.11))
76.08(+0.73)
72.35(+0.46)
on three datasets. 1) CIFAR- 100 (Krizhevsky et al., 2009) a significant dataset for image classification, comprising 100 categories, with 50,000 training images and 10,000 test images. 2) ImageNet (Russakovsky et al., 2015) is large- scale dataset utilized for image classification, comprising 1,000 categories, with approximately 1.28 million training images and 50,000 test images. 3) MS- AS-COCO (Lin al., 2014) is mainstream dataset for object detection compris- ing 80 categories, with 118,000 training images and 5,000 test images.
in Eq. are set to be the same the compared baselines to maintain fairness, and "y set equal to We utilize I NVIDIA GeForce RTX 4090 to train models on CIFAR- 100 and 4 NVIDIA GeForce RTX 4090 for training on ImageNet. The algorithm of our method can be found in Appendix A.4.
Other Settings. In order to prevent the occasional gradient explosions the start of optimization due to the too variant logits caused by random initialization of student, we add and only add normalization in ranking loss, which does not affect our faimesss as plug-and- and-play function. We conduct experiments about ablation of normalization in Appendix A.2, and the algorithm can be found in Appendix A.4.
Baselines. As a plug-and-plaa loss, we apply the proposed ranking loss to various logit- methods, including KD (Hinton et al., 2015), CTKD (Li et al., 2023), DKD (Zhao al., 2022). and MLKD (Jin et al.. 2023), to verify whether it can bring performance gains to knowledge distillation methods. We also compare it with various feature-basee methods, including FitNet (Adriana et al., 2015), CRD (Tian et al., 2019), and Re (Chen et al., 2021). Addition- ally, we compare it with other auxiliary losses and modi- fied KL divergence methods, including DIST (Huang et al., 2022) and LSKD (Sun al., 2024).
5.1. Main Result
CIFAR-100 Results. In our study, we conduct a compar- ative analysis of Knowledge Distillation (KD) outcomes across various teacher-student (He et al., 2016; Simonyan Zisserman, 2014; Zagoruyko & Komodakis, 2016; Zhang et al., 2018; Howard et al., 2017; Sandler al., 2018) config- urations. While Tab. presents cases where the teacher and student models share the heterogeneous architecture, Tab. 2 illustrates instances of homogenous structures. Furthermore, as a suug-andd-lay method, we applied ranking loss in multi- ple logit- logit-based distillation techniques. The incorporation of ranking loss resulted in an average improvement of 1.83% in vanilla KD. In the context of existing sate-of-theean (SOTA) logits- based methods, including DKD, CTKD, and MLKD, significant gains are made.
Implementation Details. To ensure the robustness of the proposed plug-and-play loss without introducing excessive configurations, we maintain the same experimental settings as the baselines used (KD+Ours and KD share the same experimental setups for example). We set the batch size to 64 for CIFAR-100, 100. 512 for ImageNet and for COCO. We employ SGD (Sutskever al., 2013) as the optimizer, with the number of epochs and learning rate settings consistent with the comparative baselines. The param--assmeters a. 3
ImageNet Results. The results on ImageNet of KD in terms of top-1 I and top-5 accuracy are compared in Tab. 3.
6

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 RRanking Loss
Table The top-1 and top-5 accuracy (%) the ImageNet vali- dation set. The teacher and student are ResNet50 and MN-VI
Table 5. Knowledge Distillation for Object Detection. All experi-
ments are conducted on COC02017 with the teacher as ResNet50 and the student as MobileNet-V2 V2.
AT
OFD
CRD
KD
KD+Ours
Method
AP
AP50
AP75
Top-1
69.56
71.25
71.37
70.50
71.54(+1.04)
Top-5
89.33
90.34
90.41
89.80
90.84(+1.04)
KD (Hinton ct al., 2015)
30.13
50.28
31.35
FitNet (Adriana et al., 2015)
30.20
49, 90.80
31.69
PGFI (Wang al., 2019)
31.16
50.68
32.92
CTKD (Li et al., 2023)
31.39
52.34
31.35
Table Knowledge Distillation for Transformer. The Top-I Accu- racy (%) on the validation set of CIFAR- 1000. The Teacher model
KD+Ours
31.96(+1.86)
53.80(+3.52) 33.37(+2.02)
ResNet56.
Ablation of Temperature. There has been extensive and detailed research on the temperature of the KL loss (e.g.. CTKD(Li et al., 2023), MLKD(Jin et al., 2023), LSKD(Sun et al., 2024)), achieving significant results. Unlike these studies, our approach aims to move away from focusing on the KI. divergence and instead explore plug-and-play aux- iliary losses to guide the KL divergence. Therefore, in our experiments, the temperature and other parameters of the KL divergence are kept the same those in the respective baselines (with dynamic/muultilevel temperatures used in CTKD+Ours and MLKD+Ours). The proposed ranking loss does not have temperature parameter because the tempera- ture does not affect the order of logits. Instead, the control over the distribution can be achieved by manipulating the steepness parameter of the sign function, and ablation ex- periments for k are shown in Tab. Meanwhile, although the temperature helps control the logit distribution, addi- tional measures are needed to prevent the KL divergence from falling into suboptimal. We conducted an ablation study on the temperature, as shown in Tab. 8. The results demonstrate that ranking loss as a plug-ande and-play loss can bring further improvements under multiple temperature set- tings. Knowledge Distillation for Object Detection. To further verify the generality of our proposed method in downstream tasks, we also conduct experiments under the setting of object detection. The results show that our proposed ranking loss can improve the performance of knowledge distillation method in object detection and achieve better performance than the same period of the feature- object detection method, which is shown in Tab. 5.
DeiT-Tiny 65.08
ViT- 69.37
PiT-Tiny 73.58
PVT-Tiny 69.22
Student
KD
71.11
71.72
74.03
72.46
KD+Ours 73.25(+2.14)
72.49(+0.77) 74.33(+0.30) 73.42(+0.96)
Our proposed method can achieve consistent improvement on the large-scale dataset as well. Additional ImageNet experiments are provided in Appendix A.9
5.2. Extensions
Knowledge Distillation for Transformer. To validate the effectiveness of our soug-and-play ranking loss on VIT and to assess its performance when facing larger teacher-student structural differences, we conduct experiments using ViT students. The experimental results indicate that the incorpo- ration of ranking loss yields significant improvements over the conventional knowledge distillation, as shown in Tab. 4. This denotes that our proposed method is also applicable to distillation challenges predicated on transformer archi- tectures. The implementation details of the Transformer experiments are attached in the Appendix.
Comparison between Ranking loss and KL Divergence. We separately evaluate distillation with KI. divergence alone and with the proposed ranking loss alone; the results are presented in Tab. 9. We observe that using the ranking loss alone achieves performance comparable to using KL divergence alone, and that combining the two objectives yields further gains. These findings substantiate the efficacy of the proposed ranking loss.
Ablation of Weight and Steepness Parameter. We conduct extensive ablation studies to investigate the effectiveness of ranking loss under various settings of k and coefficients. Tab. presents the distillation outcomes across different k configurations. is observed that a larger l value yields superior performance, suggesting that the differential form of ranking loss more closely approximates the Kendall coefficient, thereby imparting stronger ranking knowledge. Tab. 6 delincates the performance of ranking loss under varying coefficients. This setting effectively aligns the clas- sification outcomes of the teacher and student models, thus significantly enhancing the distillation effect.
More Experiments. Additional experiments and discus-
Table 6. Ablation of Weight. The Top-1 Accuracy (%) on the validation set of CIFAR-100.
ResNe132x4
WRN-40-2
ResNet32
ResNe150 79,34
Teacher
79.42
75.61
79.42
ResNet8 x4 72.50
WRN-40-1
SHN-VV
MN-V2
Student
71.98
71.28
64.60
KD(Baseline)
73.33
73.54
74.45
67.35
7 = 0.1
74.
74.15
75.52
69.25
7 0.5
74.84
74.07
76.34
69,81
0.9
74.74
74.49
76.58
70.45
= 2
74.62
74.65
77.07
69.97
7 =
75.07
73.60
77.05
70.59
7 =
74.78
73.09
76.90
69.58
7

---

Enhancing Logits Distillation with Plag&Play Kendall's 7 Ranking Loss
VGG13
ResNet50 MN-V2
ResNet32x4 ResNet8x4
VGG13 VGG8
MIN-V2
69.5
70.0
74.6
74.0
69.0
74.4-
73.0
Accirscy
Accurecy
24.2
Accuracy
74.0-
68.5-
73.0-
73.4
68.0
73.6-
732
67.5
Nin
Book
Top Min
3rp Wink
23.0
Kin
Percentage
Percentage
Percentage
Percenlagge
(a)
(b)
(c)
(d)
Figure 4. Ranking Loss using Top-K and Min-K channels. Top-1 Accuracy (9) of Top-k/Min-k Ranking Loss. 0 and 100 on the x-axis represent KD and KD+Rank, respectively.
Table Ablation The Top-1 Accuracy (%) on the validation set of CIFAR-100.
Table 9. Experiments on Combining DIST.
Teacher
ResNar32xc4 SHNN- V1
ResNetS2xx4 WRN -40-2
WRN-40-2 SIEN-VI V1
VGGL3 VOGS
WRN- 40-2
Stident
WRN- 40-1
R4
WRN-40-2
ResNet32Xx
ResNet50
Teacher
79.42
75.61
79.42
79,34
Only Ranking Loss
KD (Ouly KL.)
24,07
75,60
77.20
78.04
75.81
74.83
73:72
72.98
74.00
73.54
ResNet8 x4
WRN-40-1
SHN-V2
MN-V2
KL=Ranking Loss
75.98
78.50
76.13
74.14
74.49
Student
72.50
71.98
71.28
64.60
(Daalllline)
73.33
73.54
74.45
67.35
the model converge and achieve better generalization and performance.
k 0.1
73.13
73.75
75.47
68.9
k 0.5
74.36
74.17
75.73
68.87
& 1
74.74
74.49
76.58
70.45
k 2
74.79
74.87
77.06
70.53
16
5:-
k
75.56
74.48
77.21
70.81
00-1
00-0uu
k 6
75.74
74.11
76.11
70.32
4--
Table 8. Ablation of Temperature. All experiments are conducted on CIFAR-100 with the teacher ResNet32x4 and the student as
28
60
1/6
129
26h
255
106
200
55
124
129
249
250
ResNet8x4.
Epocc
Epooch
Epoch
Figure 5. Accuracy & Loss Curves with Ranking Loss. Left: Top- Test Accuracy (%) Curve. Middle: Loss Curve of Ranking Loss. Right: Loss Curve of KL Divergence.
Temperature
T
= 5
T 6
T = 10
KD
73.43 KD+Ours 73.56(+0.23) 73.49(+0.10) 74.36(+0.93)
73.33
73.39
73.55
74.04(+0.49)
Visualization of Loss Landscape. To further investigate the role of ranking loss in the distillation process, we visualized the loss landscapes (Li et al., 2018) of student models with and without the application of ranking loss during Knowl- edge Distillation (KD), as depicted in Fig. 4. It is evident that the student models distilled with ranking loss exhibit a markedly flatter loss landscape and fewer local optima com- pared those without it. We hypothesize that during the optimization process, ranking loss can filter out certain local optima that, despite presenting better overall loss perfor- mance, yield poorer classification outcomes. Consequently, ranking loss can effectively enhance the generalization per- formance of student models.
sions, including Derivation of the KL Loss and Ranking Loss, Ablation of Normalization, and Different Forms of Ranking Loss, are provided in the Appendix. Please refer to the Sec. A for further details.
5.3. Analysis
Accuracy & Loss Curves with Ranking Loss. The ac- curacy and loss curves of KD and KD+Ours, as shown in Fig. 5, demonstrate how ranking loss aids in optimization. The middle figure shows that the precise alignment of KL divergence also makes channel ranking more ordered, but adding ranking loss achieves more consistent ranking more quickly. The right figure shows that in the early stages, rank- ing loss accelerates the reduction of KL loss and reduces its oscillation in suboptimal regions. In the later stages, ranking loss does not interfere with KL divergence and ultimately reaches a better position. The left figure shows that with ranking loss, student achieves leading accuracy at all stages. This indicates that the addition of the ranking loss helps
Top-k & Min-k Ranking. To further validate the bene- ficial knowledge present in low cow-probayility channels, conducted comparative experiments using the top 10%, top 30%, top 50%, and min 10%, min 30%, min 50% channels. The experiments were performed across four combinations of homogeneous and heterogeneous teacher-student pairs, and the results are presented in Fig. We observed that
8

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
140
Sanalietyy a-yy
Senstaltyy
675
K75
500
980
859
08
815
M2
764
244
20
11
744
26F
40
10
PM
PM
2
742
0.5
M44
Figure 6. Loss Landscape. Left: The left landscape shows the suboptimal solutions the distillation task. Right: After adding our ranking loss, the suboptimal solution is significantly reduced, as shown on the right.
Figure 8. Sensitivity Analysis. Left: Sensitivity of a - Right: Sensitivity of B
using the min-k channels achieves results similar to those obtained with top-k channels (as shown in Fig. ((a),bb), and (c)). Additionally, in some cases, min-k channels provide even more beneficial information to aid student learning (as demonstrated in Fig. 4(d)). which indicates that the low- probability channels also contain rich knowledge.
6. Conclusion
In this paper, we investigate the optimization process of logit distillation and identify that the Kullback-Leibler di- vergence tends to overlook the knowledge embedded in the low-probability channels of the output. Moreover, KL divergence does not guarantee alignment between the clas- sification results of the student and teacher models. To ad- dress this issue, we introduce plug-and-play ranking loss based on Kendall's r Coefficient that encourages the student model to pay more attention to the knowledge contained within the low-probability channels, while also enforcing alignment with the teacher's 's predictive outcomes. We pro- vide a theoretical analysis demonstrating that the gradients of the ranking loss are less affected by channel scale and that its optimization objective is consistent with that of KL divergence, making it an effective auxiliary loss for distil- lation. Extensive experiments validate that our approach significantly enhances the distillation performance across various datasets and teacher-student architectures.
KD+nIRS
Clrl Stocher
Figure 7. Ranking Alignment Comparison. KD+Ours.
Left: KD. Right:
Further visualization of Ranking. We show in Fig. that ranking loss balances the focus on high-prooh bigh-probability and low- probability channels. To further illustrate the role of ranking loss, we present visualization of the ranking results gener- ated by KD and KD+Ours in Fig. 7. It can be seen that after adding ranking loss, the channels are more ordered overall, which is more obvious in the low-probability channel part. This is because KL divergence does not pay attention to sorting and neglects oow-roobaillity channels. Although the large channels in KD receive more attention, the alignment value alone cannot achieve order consistency, thus losing some knowledge. The results indicate that our approach consistently enhances channel alignment, underscoring the robustness and general applicability of our method, particu- larly in its focused attention on low-probayility channels.
Acknowledgements
This work is supported by the National Key R&D Pro- gram of China 4002 SSTIC Grant (KI21200209923111500002, AJZD2002309231449000 GJHZZ202402181 1330022440001111100000008000 Beijing Key Lab of Networked Multimedia.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Further Ablation Study of the Hyperparameters. To further substantiate the generalization capability and ro- bustness of our method, conducted comprehensive ab- lation studies using different combinations of coefficients. As illustrated in Fig. 8, our method consistently maintains high performance across various settings, underscoring the strong generalization ability and universal applicability of our method.
References
Adriana, R., Nicolas, B., Ebrahimi, K. S., Antoine, C., Carlo, G., and Yoshua, B. Fitnets: Hints for thin deep nets. Proc. ICLR, 2(3):1, 2015.

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
Cao, W., Zhang, Y., Gao, J., Cheng, A., Cheng, K., and Cheng, J. Pkd: General distillation framework for object detectors via pearson correlation coefficient. Advances in Neural Information Processing Systems, 35:15394- 15406, 2022.
Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang,
W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:/704.0486// 2017.
Huang, T., You, S., Wang, F., Qian, C.,, and Xu, Knowl- edge distillation from stronger teacher. Advances in Neu- ral Information Processing Systems, 35:33716-33722 2022.
Chen, D., Mei, J.-P., Zhang, H., Wang, C., Feng, Y., and Chen, C. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11933- 11942, 2022.
Jin, Y., Wang. J., and Lin, D. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24276-24285, 2023.
Chen, P., Liu, S., Zhao, H., and Jia, J. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5008-5017, 2021.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Kullback, S. and Leibler, R. A. On information and suf- ficiency. The annals of mathematical statisties, 22(1): 79-86, 1951.
Cheng, R., Xiong, F., Wei, Y., Zhu, W., and Yuan, C. Who- ever started the interference should end it: Guiding data- free model merging via task vectors. arXiv preprint arXiv:2503.08099 2025.
Li, G., Li, X., Wang, Y., Zhang, S., Wu, Y., and Liang, D. Knowledge distillation for object detection via rank mimicking and ceddiciionguuide feature imitation. In Proceedings of the AAAI conference on artificial intelli- gence, volume 36. pp. 1306-1313, 2022.
Chi, Z., Zheng, T., Li, H., Yang, Z., Wu, B., Lin, B., and Cai, D. Normkd: Normalized logits for knowledge distillation. arXiv preprint arXiv:2308.0055
2023.
Choi, J., Jung, E., Suh, J., and Rhee, W. Improving bi- encoder document ranking models with two rankers and multi-teacher distillation. In Proceedings of the 44th international ACM SIGIR conference research and development in information retrieval, pp. 2192-2196, 2021.
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 2018.
Li, Z., Li, X., Yang, L., Zhao, B., Song, R., Luo. L., Li, J., and Yang, J. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1504-1512, 2023.
Gao, L.,, Dai, Z., and Callan, J. Understanding bert rankers under distillation. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval, pp. 149-152, 2020.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCY 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Guo, Z., Yan, H., Li, H., and Lin, X. Class attention trans- fer based knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11 1868-11877 2023.
Park, W., Kim, D., Lu, Y., and Cho, M. Relational knowl- edge distillation. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 3967-3976, 2019.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Qin, 7., Jagerman, R., Pasumarthi, R. K., Zhuang, H., Zhang, H., Bai, A., Hui, K., Yan, L., and Wang, X. Rd- suite: A benchmark for ranking distillation. Advances in Neural Information Processing Systems, 36, 2023.
Heo, B., Kim, J., Yun, S., Park, H.. Kwak, N., and Choi, J. Y. A comprehensive overhaul of feature distillation. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 1921-1930, 2019.
Reddi, S., Pasumarthi, R. K., Menon, A., Rawat, A. S., Yu, F., Kim, S., Veit, A., and Kumnr, S. Rankdistil: Knowl- edge distillation for ranking. In International Conference on Artificial Intelligence and Statistics, pp. 2366-3776
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv: 025311 2015.
PMLR, 2021.
10

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z-,, Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211-252, 2015.
Yang, Z., Li, Z.,, Shao, M., Shi, D., Yuan, Z., and Yuan, C. Masked generative distillation. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI, pp. 53-69. Springer, 2022c.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In P Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Yang. Z., Zeng, A., Li, Z.., Zhang, T., Yuan, C., and Li, Y. From knowledge distillation to self-knowledge dis- tillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 17185-17194, October 2023a.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: /409: 1556, 2014.
Yang, Z., Zeng, A., Li, Z., Zhang, T., Yuan, C., and Li, Y. From knowledge distillation to self-knowledge dis- tillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17185 17194, 2023b.
Sun, S., Ren, W., Li, J., Wang, R., and Cao, X. Logit standardization in knowledge distillation. arXiv preprint arXiv:2403.01427, 2024.
Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learn- ing. In International conference on machine learning, pp. 1139-1147. PMLR, 2013.
Zagoruyko. S. and Komodakis, N. Wide residual networks arXiv preprint arXiv:1605.07146 2016.
Zagoruyko, S. and Komodakis, N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2017.
Tang, J. and Wang, K. Ranking distillation: Learning com- pact ranking models with high performance for recom- mender system. In Proceedings the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2289-22996 2018
Zhang, L. and Ma. K. Improve object detection with feature- based knowledge distillation: Towards accurate and effi- cient detectors. In International Conference on Learning Representations, 2021.
Tian, Y., Krishnan, D., and Isola, Contrastive representa- tion distillation, arXiv preprint arXiv: 10699, 2019.
Zhang, X., Zhou, X., Lin, M., and Sun, J. Shufflenet: An extremely efficient convolutional neural network for mo- bile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2018.
Wang, T., Yuan, L., Zhang, X., and Feng, J. Distilling object detectors with fine-grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4933-4942, 2019.
Zhao, B., Cui, Q., Song, R., Qiu, Y., and Liang, J. De- coupled knowledge distillation. In Proceedings of the IEEE/VVF Conference on computer vision and pattern recognition, pp. 11953-11962, 2022.
Wei, Y., Tang, A., Shen, L., Yuan, C., and Cao, X. Modeling multi-task model merging as adaptive projective gradient descent. arXiv preprint arXiv:2501.0222 2025.
Wen, T., Lai, S., and Qian, X. Preparing lessons: Improve knowledge distillation with better supervision. Neuro- computing, 454:25-33, 2021.
Zheng, K., Zhang, H., and Huang, W. Diffkendall: A novel approach for few-shot learning with differentiable kendall's rank correlation. Advances in Neural Informa- tion Processing Systems, 00:49403-49415, 2023.
Yang, S., Sanghavi. S., Rahmanian, H., Bakus, J., and SVN, V. Toward understanding privileged features distillation in learningtto-rank Advances in Neural Information Processing Systems, 35:26658-266 266700 2022a.
Yang, Z., Li, Z., Jiang, X., Gong, Y., Yuan, Z., Zhao, D., and Yuan, C. Focal and global knowledge distillation for detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4643- 4652, 2022b.

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
A. Appendix
A.I. Implementation Details for Transformer
We use the AdamW optimizer and train for 300 epochs with an initial learning rate of 5e-4 and a weight decay of 0.05. The minimum learning rate is 5e-6, and the patch size is 16. We a 1, 1, y 0.5, and batch size is 128. The value range of ranking loss [-1,]]
1] We use a single RTX4090 for CIFAR-100 and 4 RTX4090 for ImageNet.
A.2. Ablation of Normalization
Due to the random initialization of student, the output logits will be too variant at the start of optimization and occasionally lead to gradient explosions. Therefore, we add normalization to the ranking loss to stabilize the ranking loss optimization the very beginning. We also supplemented ranking performance did not come from the normalization added to the ranking loss, shown in Tab. 10 below:
of small-scale ablation experiments that showed that the improvement in
Table Ablation of Normalization.
ResNet32
WRN-40-2
Teacher
79.42
75.61
SHN-VI
WRN-40-1
Student
70.30
71.98
KD)
74.07
73.54
KD+Ours w/o Norm
76.38(+2.31)
74.07(+0.53)
KD+Ours w Norm
75.98(+1.91)
74.49(+0.95)
A.3. Different Forms of Ranking Loss
In the initial application of the ranking loss(Zheng et al., 2023), it is necessary to compute the gradients for two input vectors separately. However, In the scenario of distillation, the soft labels from the teacher model do not require gradients. Therefore, we propose three variants the diff- Kendall ranking loss suitable for distillation scenarios, aimed at further exploring the role of ranking loss in distillation. Since the sample pair (z) - 24) itself has sign., we can derive an equivalent form from Eq. 5:
Σ, EjE sgn((2) - 22) (2] 25))
(13)
_C(C -)
For Eq. 5 and Eq. 13, we can similarly transform them into differential forms of ranking loss by replacing sgn with tanh, and gradients are not computed for the output part corresponding to the teacher, which is:
ofrml EiEjer tanh(z) 25)detach tanh(2f - 23)
(14)
]C(CC- -1
L form2 Ejes (and((( 25)dctack (2f 29)]
(15)
]CCC
Where Qdetach means not participating in training. Further, since the gradient of the teacher's output is not required, we can directly use the sign function instead of the tanh function to obtain the ranking loss:
I form3 EeEcer - =z)aettcc lanh(= zz)
(16)
CCCC-1 11
Different Forms of Ranking Loss. In our investigation, examined the performance of the three forms of distillation presented in Sec. A.3, as depicted in Tab. 11. Forml exhibited the most superior performance, followed by Form2, with Form3 trailing yet still enhancing the efficacy the original knowledge distillation. This suggests that within the ranking loss, it is imperative to optimize considering the magnitude of the teacher's logits differences as a coefficient for the loss, rather than merely optimizing as sign function.
12

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
Table 11. Different Forms of Ranking Loss. The experiments conducted the CIFAR-100, with heterogeneous and homogenenus
architectures. The average Top-1 accuracy (%) is reported.
Loss Form
KD
KD+Forml
KD+Form2
KD+Form3
Similar Structure Different Structure
72.01
73.47(++144) 73.50(+0..6)
73,42(+1.41) 73.36(+0.62)
73.38(+1.37)
72.74
73.05(+0.31)
A.4. Algorithm
Algorithm Plug-and-Play Ranking Loss for Logit Distillation
Transfer set D with samples of image-label pair (Zn, 1. base temperature T', teacher ft, student f., knowledge distillation Loss LXD. ranking Loss Lax. the weight of ranking Loss
Input:
Output: Trained student model
for (Zn, 3n)
do
Get the logits of Teacher and student:>* - fa(xx)), 22" fx(xy)
Calculate the probability with lemperature: v² softmar + ) softmaae (
Get the normalized logits of Teacher and student: 8¹ 3'-3
stdlat)) 80 2²-20
ald(=")
Update fa towards minimizing: Lesal = LxD(q', ,²) + L
end
A.5. Derivation of the KL Loss and Ranking Loss
Derivation of the KL Divergence with Respect to Student Logits in Knowledge Distillation.
Denote the teacher's logits as z' 22, =t. Denote the student's logits z [ef, 22 .......... =gl-
Let T be the temperature scaling factor used in the softmax function.
Teacher probabilities can be calculated as:
exp
for i = 1,2,..., C.
(17)
Σ
exp
T
Student probabilities can be calculated as:
exp
T
for i 1,2....,C. C.
(18)
C
exp
T
j=l
The loss function used in knowledge distillation is the scaled Kullback-Leilerr (KL) divergence between the teacher and student probability distributions:
C
LKD = 2 KL (q' l1 7
Σ
log
q o
(19)
The derivative of the loss with respect to the student logits =f is:
0LKD
alog q
2
Σg
gk
(20)
02ş
0zg
Jan
13

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
Taking the natural logarithm:
=x
c
)
log q% log
Σ
oxp
(21)
T
T
1=1
Compute the partial derivative:
i log 9
2
log
Σ exp
(22)
0z
0z1
T
dzt
j=l
Compute each term separately:
0
2k
1
Siks
1,
if i k;
(23)
dz
T
T
0.
if i + k.
1
Σ exp
1
log
exp
(24)
dzd
T
C
2j
0z
Σ
exp
==1
T
1=1
The derivative inside the is:
c
exp
2j
2
1
exp
exp
(25)
dzf
T
T
T
Therefore, the second term becomes:
4
exp
T
9
(26)
2)
T
Σ
exp
T
1=1
Thus, the total derivative is:
log of
1
- Tin
7
7
(6sk - -().
(27)
Oz
T
i log o
We now substitute
back into the expression for the derivative of the loss:
02g
tLKD
2
(ōuu gi))
Oz5
qk
T
(28)
C
T
Σ
(0t - 9:).
1=1
Considering:
Jeói = 4/
(29)
R=
Σ
of Σ
gt of 1 qff
Σ o = 1.
(30)
k=1
k=1
k=1
Therefore, the loss derivative simplifies to:
14

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
L
- (gl q.)
(31)
dzp
Derivation of the Ranking Loss with Respect to Student Logits in Knowledge Distillation. The Rank loss is calculated as:
Σ, EiE tanh(k(2) -2,)) tanh(k(22 -2() -2)))
LRK
(32)
C(C-1)
The derivation is calculated as:
ŌLRK
1
Σ
θ 1
[tanh (k<22 25)) tanh (R:2f - 25))
(33)
Ozg
C(C-1)
0z7 2
0
Σ
[lanh (k(2) zz)) tanh (k(z) - =}))]
(34)
C(C
1)
dzg
j#i
Denote dij as:
i = tanh (k(2f 27)) tanh (M(2f - 2ÿ))
(35)
Its derivative w.r.t. 2 is:
0Φg
[1 tanh? (k(2) 25])] tanh (k(2) - -2)))
(36)
dzg
Finally, the gradient of LRK with respect to 2 is:
ÖLRK Ozf
Σ
[1
tanh? (k(2) =2))] tanh (k(2f z5))
(37)
C(C 1) j#t
A.6. Derivation of the Differential Ranking Loss
In this section, explain how we get the final form of Eq. 6. Noticed that:
-- 2x
tanh(x)
(38)
e0 - 22x +1
222 +
Then, we can get the final form of Eq. 6 by expanding the tanh function.
Pd Σx tanh(k:
(2f ----,pp --²))) Laan((k (zf 15---pp 2z))
(39)
CC(C-1)
-1-1-02--000 1
-)C-1)
ΣΣ jái
1
1 1 2+
( 2
(40)
A.7. Does Temperature Solve the Problem of Ignoring Low-Probability Channels
Some methods (Wei et al., 2025; Cheng et al., 2025) address the issue of neglecting low- low-probability channels by employing temperature scaling. We examined the probability distribution of LSKD (Sun et al., 2024), an approach using adaptive temperature that has shown promising results. As shown Fig .9, while LSKD somewhat alleviates the problem of www-probyyility channels, the transformed probabilities still predominantly occupy these low- wwwooabbbblltyy channels. Thus, using temperature scaling does not effectively resolve the issue of neglecting low-probability channels.
15

---

Enhancing Logits Distillation with Plug&Play Kendall's 7 Ranking Loss
Loght
10"
10000
Mambb
10-
0
0.2
04
0.7
0.2
044
65
68
Laghts
Probsbiitt
Proteblity
Figure 9. probability produced by KD and LSKD
A.8. Combining with Other Correlation Coefficients.
DIST (Huang et al., 2022) explores the use of the Pearson correlation coefficient for knowledge distillation. However, unlike our proposed method, the Pearson correlation coefficient measures the linear relationship between two variables. Our method focuses the ranking relationship of logits, which is not strictly linear relationship. Therefore, Kendall's correlation coefficient may be more suitable for capturing this non-linear ranking information. To further investigate the combined effect of using both linear and non-linear auxiliary functions, we incorporate ranking loss into DIST, which improves its performance. The results are shown in Tab. 12.
Table 12, Experiments on Combining DIST.
Teacher
ResNet32x4 WRN-16-2
ResNer32x4 WRN-40-2
WRN-40-2 SHN-V1
VGG13
WRN-40-2
Student
VGG8
WRN-40-1
DIST (Huang et al., 2022)
75.58
78.02
76,00
73.80
74.73
DIST+Ours
75.85(+0.27)
78.54(+0.52)
76.23(+0.23)
74.06(+0.26)
74.86(+0.13)
A.9. More ImageNet Results
We further augment CTKD (Li al., 2023) and NKD (Yang al., 2023a) with the proposed ranking loss and evaluate them on the ImageNet dataset; the results are reported in Tab. 13.
Table 13, The top-1 and top-5 accuracy (%) on the ImageNet validation set. The teacher and student are ResNet34 and ResNet18
Method
Top-1 I Acc.
Top-5 Acc.
(Zagoruyko Komodakis, 2017) OFD (Hco al., 2019)
70.69
90.01
70.81
99.98
CRD (Tian al., 2019)
71.17
90.13
CTKD (I.i et al., 2023)
71.32
90.27
CTKD+Ours
71.68(+0.36)
90.30(+0.03)
NKD (Yang al., 2023a)
71.96 72.06(+0.10)
90.27
NKD+Ours
90.49(+0.22)
16


Reviews:
[{"review_id": "YhEscCmEuA", "reviewer": "Reviewer_Za63", "summary": "This paper proposes an auxiliary ranking loss based on Kendall’s tau coefficient to improve knowledge distillation by addressing the limitations of KL divergence. Traditional KL-based distillation struggles with optimization challenges and tends to overlook low-probability channels, leading to suboptimal performance. The proposed ranking loss provides inter-class relationship information and balances attention across all channels while maintaining consistency with KL divergence. It is a plug-and-play module compatible with various distillation methods. Extensive experiments on CIFAR-100, ImageNet, and COCO datasets demonstrate its effectiveness across different CNN and ViT teacher-student architectures.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the in-depth feedback! We highly value each of your comments, and all your concerns are addressed point by point:\n\n---\n\n**Q1. Discuss More techniques of detection improvement:**\n\n- We sincerely appreciate your suggestion. Exploring the application of the proposed method in downstream tasks is indeed one of our future directions. However, our method is primarily designed to enhance knowledge distillation. We have compared our approach with other knowledge distillation-based object detection methods, and the results are presented in Table 5 of the paper and in Table T2.2 of our response to Reviewer vqaG. In future versions, we will include discussions and potential integrations with pure object detection methods.\n\n---\n\n**Q2. Comparison with Pearson Correlation:**\n\n- Thank you for your interest in our work. We also note the application of the Pearson correlation coefficient in distillation; however, unlike our proposed method, the Pearson correlation coefficient measures the linear relationship between two variables. Our method focuses on the ranking relationship of logits, which is not strictly a linear relationship. Therefore, the Kendall correlation coefficient may be more suitable for capturing this non-linear ranking information. \n- We have already compared the performance of DIST and our proposed method in the distillation tasks in **Table 1 and 2 of the paper**. To further investigate the combined effect of using both linear and non-linear auxiliary functions, we incorporated a ranking loss into DIST, which improved its performance. The results are shown in **Table T1.1**.\n\n  *Table T1.1: Experiments on Combining DIST.*\n  | Teacher -> Student | ResNet32×4 -> WRN-16-2 | ResNet32×4 -> WRN-40-2 | WRN-40-2 -> SHN-V1 | VGG13 -> VGG8 | WRN-40-2 -> WRN-40-1 |\n  | :---: | :---: | :---: | :---: | :---: | :---: |\n  | DIST[1]      | 75.58 | 78.02 | 76.00 | 73.80 | 74.73 |\n  | DIST[1]+Ours | 75.85(+0.27) | 78.54(+0.52) | 76.23(+0.23) | 74.06(+0.26) | 74.86(+0.13) |\n\n[1] Knowledge distillation from a stronger teacher, NeurIPS 2022\n\n---\n\n**Q3. Range of the Ranking Loss:**\n\n- According to **Equations 5 and 6 in the paper**, the proposed ranking loss has a value range of [-1, 1]. We set the weight of ranking loss to be the same as that of KL divergence. For baselines using different KL divergence weights, the value of the ranking loss may vary, but its scale generally remains within the range of [-1, 1]. Across all teacher-student combinations with the same baseline and dataset, the ranking loss maintains consistent weight and scale.\n\n---\n\n**Q4. Analysis of the Steepness Parameter:**\n\n- In Table 7 of our paper, we investigate the impact of the steepness parameter. We observe that although the optimal steepness parameter may vary across different teacher-student combinations, larger steepness parameters generally yield better results. This finding aligns with intuition: as the steepness parameter increases, the tanh function approximates the sign function more closely, making it more sensitive to ranking and less sensitive to scale differences.\n\n**Q5. Other Comments and Suggestions:**\n\n- Thank you very much for your feedback regarding the formatting of our paper. We will adjust these formulas in the future version of the manuscript. We appreciate your attention to detail.", "claims_and_evidence": "Yes, the claims have been supported by clear and convincing evidences.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense.", "theoretical_claims": "Yes, theoretical claim is proved properly.", "experimental_designs_or_analyses": "Experimental designs and analyses are valid.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "A contribution of the proposed method is to improve the object detection performance via distillation, and it is suggested to discuss several more techniques of detection improvement such as [1-2].\n\n[1] Dynamic head: Unifying object detection heads with attentions, CVPR 2021.\n[2] Rethinking image restoration for object detection, NeurIPS 2022.", "other_strengths_and_weaknesses": "Strengths: \n\n- The idea of emphasizing the ranking information of logits through a ranking loss is both innovative and impactful. By explicitly modeling the relative order of predictions, the proposed approach enhances the training process, leading to more refined and meaningful feature representations.  \n\n- The ablation studies presented in Tables 6 and 7 are thorough, covering various aspects of the proposed method. These studies systematically analyze the contribution of each component, reinforcing the effectiveness and necessity of the design choices.  \n\n- The method demonstrates a substantial performance improvement over existing baselines across multiple benchmarks. The quantitative results highlight its superiority in terms of both accuracy and robustness, confirming its effectiveness in handling divergent cases.  \n\n- The role of the ranking loss is comprehensively discussed, with detailed insights into how it influences the learning dynamics. The paper provides both empirical evidence and theoretical analysis showing how ranking-based supervision enhances model generalization and optimization.  \n\n- The proposed method also proves to be beneficial for distilling vision transformers, effectively transferring knowledge while preserving critical ranking information. This indicates its potential for improving the efficiency and performance of large-scale transformer-based architectures.\n\nWeaknesses:\n\n- One closely related work [1] incorporates intrinsic ranking information of logits using Pearson Correlation. Since both methods aim to enhance knowledge distillation by capturing ranking relationships, it would be valuable to compare the proposed ranking loss with this approach as an auxiliary loss with more discussions. This would provide deeper insights into their relative effectiveness and compatibility.  \n\n- A discussion on the numerical range of the ranking loss would be helpful, as it would provide better interpretability and understanding of its impact during training. Clarifying whether its scale remains stable across different datasets and architectures could further enhance reproducibility.  \n\n- The influence of the steepness parameter in Eq. 6 is not explicitly analyzed. Since this parameter likely affects the sensitivity of the ranking loss, a more detailed discussion would be beneficial to determine its optimal setting and impact on training stability and performance.\n\n[1] Knowledge distillation from a stronger teacher, NeurIPS 2022.", "other_comments_or_suggestions": "It is recommended to revise the functions and equations to ensure proper formatting, condensing them into a single line where possible or adjusting line breaks appropriately, particularly for Equations (6) and (10).", "questions_for_authors": "Please see weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Aq1u8kTsnD", "reviewer": "Reviewer_vqaG", "summary": "This paper points out two drawbacks of KL divergence in knowledge distillation: (1) it is often prone to suboptimal points; (2) it overlooks low-probability channels. The authors use Kendall’s $\\tau$ coefficient to mitigate these issues and better model inter-class relationships. Experiments on CIFAR-100, ImageNet, and COCO across various architectures demonstrate improved performance compared to baselines.\n\n## Update after rebuttal\n\nI have carefully read the authors' rebuttal as well as reviews provided by other reviewers.\n\n- Some of my concerns have been addressed. \n\nThe authors reasonably explain that KL tends to ignore low-probability channels due to their small gradients, but this point should be further elaborated in the main paper. Moreover, the authors have provided additional experimental results.\n\n- **Nonetheless, there remains a critical concern that has not been addressed.**\n\nThe motivation for introducing the ranking loss—enabling the student model to better capture inter-class relationships (IRs)—is highly related to existing methods (e.g., [WKD]). However, **the authors failed to discuss connections with and differences from WKD.**\n\nUnlike using logit ranking as qualitative information in this paper, WKD quantitatively models IRs based on feature-level statistics and employs the Wasserstein Distance (WD) to measure the distance between discrete probability distributions across all classes. *From a theoretical standpoint, WKD may offer a more principled approach than the proposed ranking loss.*\n\nTherefore, the authors are required to explicitly discuss methodological distinctions and offer theoretical insights in the main paper. Furthermore, it is important to empirically evaluate the individual performance of ranking loss and WKD and their potential complementarity. Such discussions would help further substantiate the soundness and effectiveness of the proposed approach.\n\n[WKD] Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation. NeurIPS, 2024.\n\n- **Due to the unresolved concern, my actual evaluation lies between \"weak reject\" and \"weak accept\". However, as no such score under the current review criteria, I select \"weak accept\".**", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the in-depth feedback! We highly value each of your comments, and all your concerns are addressed point by point:\n\n---\n\n**Q1. The scale of KL is influenced by both the teacher’s probability and the logarithmic term. Could the authors clarify the specific role of the logarithmic term in KL?**\n\n- The logarithmic term is derived from maximum likelihood, but the coefficient of the teacher is independent of the log term. As shown in the formula\n$$K L(P | Q)=\\sum p(x) \\log \\frac{p(x)}{q(x)}=\\underbrace{-\\sum p(x) \\log (q(x))}_1+\\underbrace{\\sum p(x) \\log (p(x))}_2$$\nthe first part is a constant, while the second part is the optimizable component. The teacher's probability, acting as the coefficient of the log term, leads to the optimization process ignoring channels where the teacher's output probability is small. The neglect of low-probability channels by the KL divergence can also be explained at the gradient level. We have provided a detailed explanation in Figure 3 of the paper and Section A.6 of the appendix to validate our observation that small channels often receive smaller gradients.\n\n---\n\n**Q2. The analysis of Eq.5 seems to contradict the claim in R1:**\n\n- In fact, lines 77-80 (R1) explain that when using KL divergence, low-probability channels receive smaller gradients during the optimization process. Equation 5 represents our proposed ranking loss, which we designed to be independent of channel values to ensure that low-probability channels are not overlooked. Therefore, the fact that the gradient of Equation 5 is independent of channel values does not conflict with R1; rather, this is precisely the intended goal of our design.\n\n---\n\n**Q3. The current experimental design on ImageNet and COCO could be further refined:**\n\n- We have individually integrated the proposed ranking loss into NKD [1], CTKD [2], and WTTM [3], and conducted comparative experiments on ImageNet. Notably, CTKD+Ours achieved results surpassing the baseline despite being trained for 23 fewer epochs than CTKD. The experiments for WTTM are still ongoing, and we will promptly update our response once the results are obtained. The results are presented in **Table T2.1**.\n\n*Table T2.1: Experiments on ImageNet.*\n  | ResNet34 -> ResNet18 | Acc@1 | Acc@5 |\n  | :---: | :---: | :---: |\n  | NKD[1]      | 71.96 | - |\n  | NKD[1]+Ours | 72.06 | 90.57 |\n  | CTKD[2] (120epochs)      | 71.32 | 90.27 |\n  | CTKD[2]+Ours(97epochs) | 71.43 | 90.29 |\n\n- For object detection on COCO, WTTM does not provide relevant experiments, and the NKD paper only includes experiments related to the self-supervised method USKD, which differs from our experimental setup. Therefore, we compared our method with CTKD, and the results are presented in T**able T2.2**.\n\n*Table T2.2: Experiments on COCO.*\n  | ResNet50 -> MN-V2 | AP | AP50 | AP75 |\n  | :---: | :---: | :---: | :---: |\n  | CTKD[2] | 31.39 | 52.34 | 31.35 |\n  | Ours    | 31.99 | 53.80 | 33.37 |\n\n---\n\n**Q4. The paper lacks a comparison with relevant SOTA methods:**\n\n- In fact, we have already compared our method with LSKD [5], a highlight method from CVPR 2024, in the paper. We further add a performance comparison with WKD[4] in Table T2.3.\n\n*Table T2.3: Comparison with WKD.*\n  | Teacher -> Student | WRN-40-2 -> WRN-40-1 | ResNet32×4 ->ResNet8×4 | VGG13 -> VGG8 | WRN-40-2 -> SHN-V1 | ResNet50 -> MN-V2 |\n  | :---: | :---: | :---: | :---: | :---: | :---: |\n  | WKD-L[4] | 74.84 | 76.53 | 75.09 | 76.72 | 71.10 |\n  | Ours     | 76.08(+1.24) | 77.25(+0.72) | 75.35(+0.26) | 77.87(+1.15) | 71.66(+0.56) |\n\n---\n\n**Q5. Typos:**\n\n- We are sorry that we could not locate any citations in Figure 5 or line 378, nor did we find any incorrect citations in Table 5. If there are any errors, we would greatly appreciate it if you could point them out to us.\n\n---\n\n**Q6. Why does KL overlook low-probability channels? Why low-probability channels will have smaller gradients?**\n\n- In Section A.6 of the appendix, we provide a detailed derivation of the gradients provided by the KL divergence. The gradient obtained by a specific channel is related to the difference between the student and teacher values in that channel. Since the values of the student and teacher in low-probability channels are on a smaller scale, the gradients obtained are also smaller. This is the underlying reason why KL divergence tends to overlook small channels. In Figure 3 of the paper, we illustrate the gradients provided by KL divergence for channels of different sizes, which also demonstrates that low-probability channels generally receive smaller gradients.\n\n---\n\n**Q7. Discussions on other ordinal data measures:**\n\n- We are currently experimenting with replacing the Kendall correlation coefficient with Spearman’s rank correlation coefficient and Goodman & Kruskal’s Gamma for ablation studies. The experiments are still ongoing, and we will promptly update our response with the results as soon as they are available.", "claims_and_evidence": "1.\tAccording to Eq.1, the authors claim that KL overlooks low-probability channels as the teacher's probability serves as a weighting factor. However, this claim is questionable. The scale of KL is influenced by both the teacher’s probability and the logarithmic term. Could the authors clarify the specific role of the logarithmic term in KL?\n\n2.\tIn Lines 77-80 (R1), the authors claim that low-probability channels will receive smaller gradients during optimization. However, according to Eq.5, the gradient appears to be influenced by the difference of bewteen teacher and student,  and independent of the channel values. The analysis of Eq.5 seems to contradict the claim in R1.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. The current experimental design on ImageNet and COCO could be further refined. The current baseline, KD, has a relatively weak performance in the context of knowledge distillation. To provide a more comprehensive evaluation of the proposed method, the authors should conduct experiments on stronger KL-based baselines and analyze performance changes, such as [NKD], [CTKD], and [WTTM].\n\n[NKD] From Knowledge Distillation to Self-Knowledge Distillation. ICCV, 2023.\n\n[CTKD] Curriculum Temperature for Knowledge Distillation. AAAI, 2023.\n\n[WTTM] Knowledge Distillation based on Transformed Teacher Matching. ICLR, 2024.\n\n2. The paper lacks a comparison with relevant SOTA methods. For instance, [WKD] also leverages inter-class relationships for distillation and employs Wasserstein Distance (WD) instead of KL as a measurement. A comparison with such methods would help better highlight the advantages of the proposed approach.\n\n[WKD] Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation. NeurIPS, 2024.", "relation_to_broader_scientific_literature": "It seems there is no broader scientific literature for this paper.", "essential_references_not_discussed": "The core contribution of this paper lies in utilizing rank loss to measure the inter-class relationships. In contrast, WKD employs WD to capture inter-class relationships (refer to “Experimental Designs or Analyses”). Rank loss focuses solely on class ranking, while WKD provides a quantitative measure of inter-class relationships. The authors should further discuss distinctions and connections with related works.", "other_strengths_and_weaknesses": "1. The idea of introducing rank loss in knowledge distillation to overcome the limitations of KL and provide inter-class relationships is interesting.\n\n2. The visualizations are well-executed, which is helpful to enhance the overall understanding of the paper.", "other_comments_or_suggestions": "There are some typos, including the incorrect citation of Fig. 5 in Line 378 (right column). The authors should carefully review and correct these issues.", "questions_for_authors": "1.\tWhy does KL overlook low-probability channels? Why low-probability channels will have smaller gradients? The current theoretical analysis presented by the authors is unconvincing. A more rigorous theoretical or experimental analysis should be provided.\n\n2.\tThe discussion on ranking loss is rather limited. Besides Kendall’s τ, other ordinal data measures, such as Spearman’s rank correlation coefficient and Goodman & Kruskal’s Gamma, could also be considered. It is important to analyze and compare the performance of these metrics.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QcIfetUNaX", "reviewer": "Reviewer_Zvoo", "summary": "This paper introduces a plug-and-play ranking loss based on Kendall’s τ to mitigate two drawbacks of KL-based logit distillation: the neglect of low-probability channels and getting stuck in suboptimal points. By aligning channel-wise rankings between teacher and student, it leverages inter-class relationships and balances attention across all channels. The loss is theoretically shown to be compatible with KL divergence—sharing the same optimal solution—yet more robust to scale. Experiments on CIFAR-100, ImageNet, and MS-COCO, using both CNNs and ViTs, confirm that adding this ranking loss consistently enhances performance over baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the in-depth feedback! We highly value each of your comments, and all your concerns are addressed point by point:\n\n---\n\n**Q1. Font Size in Figure 1:**\n\n- Thank you very much for your feedback regarding the formatting of our paper. We will will use appropriate font sizes in the future version of the manuscript. We appreciate your attention to detail.\n\n---\n\n**Q2. The Motivation Seems Not Reasonable: Intuitively, KL divergence seems to assign different learning weights to different class channels, with the target class having a higher weight, making it more important.**\n\n- In the distillation task, the cross-entropy loss computed with the one-hot hard labels of the target class has already emphasized its significance. The soft labels output by the teacher model differ from the one-hot hard labels, as they contain richer inter-class relationship knowledge. While the Kullback-Leibler (KL) divergence captures this information by constraining the distribution, it tends to overlook the low-probability channels. In contrast, the proposed ranking loss exhibits a stronger capability in capturing relational information among low-probability channels, thereby providing complementary support to the KL divergence.\n\n---\n\n**Q3. The Motivation Seems Not Reasonable: The phenomenon in Figure 1 does not occur frequently.**\n\n- Figure 1 presents a toy case designed to illustrate the potential suboptimal optimization behavior of the Kullback-Leibler (KL) divergence. A similar suboptimal scenario is also demonstrated in **Figure 2 of the LSKD [1] paper**. To further validate whether the proposed ranking loss can address the potential suboptimality in the KL divergence, we visualize the loss landscape in **Figure 6 of our manuscript**. We observe that when only the KL divergence is used, suboptimal points exist near the global optimum, while the introduction of the ranking loss effectively mitigates this issue.\n\n[1] Logit Standardization in Knowledge Distillation. CVPR 2024 Highlight\n\n---\n\n**Q4. Performance Comparison of using only Ranking Loss versus using only KL Divergence:**\n\n- We are currently conducting comparative experiments using only the ranking loss and only the KL divergence. We will promptly update our response once the results are obtained. \n- In fact, the proposed ranking loss serves as an auxiliary function to the KL divergence, designed to guide the KL divergence in avoiding suboptimal solutions and capturing inter-class relationship information. The ranking loss focuses solely on aligning the order of channels and imposes minimal constraints on channel values and distributions. Therefore, using only the ranking loss typically does not yield satisfactory distillation performance.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes. The authors compared their method with current state-of-the-art approaches on common visual distillation tasks using CIFAR-100, COCO, and ImageNet. I find the experiments to be quite comprehensive overall. A potential shortcoming maybe is that they did not evaluate the method’s effectiveness in the increasingly popular large-model distillation settings.", "relation_to_broader_scientific_literature": "This study analyzes the deficiencies in a series of previous logit-based methods based on KL divergence and proposes a novel auxiliary ranking loss based on Kendall’s τ Coefficient to mitigate the aforementioned issues.", "essential_references_not_discussed": "To the best of my knowledge, I think the authors have considered the relevant works related to their study.", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper is well-written and easy to follow.\n2. The experiments are extensive and conducted on CIFAR-100, ImageNet, and COCO datasets. Additionally, more visual analyses are provided to explain the effectiveness of the proposed method.\n\nWeaknesses：\n1. The font in Figure 1 appears to be very small.\n2. The proposed motivation does not seem very reasonable. I have a different opinion from the authors regarding why KL divergence would overlook the matching of low-probability channels. Intuitively, KL divergence seems to assign different learning weights to different class channels, with the target class having a higher weight, making it more important. This appears to be a reasonable arrangement. Although the authors provide an example in Figure 1, I do not believe this phenomenon occurs frequently.\n3. Following the previous question, if time permits, could the authors compare the performance of using only the proposed rank loss versus using only KL divergence? This would provide a more intuitive evaluation of the effectiveness of the proposed method.", "other_comments_or_suggestions": "See above", "questions_for_authors": "I don’t quite understand lines 183-185, which state: “The gradient of a logit channel primarily depends on the difference between its rank and the target rank, effectively harnessing the knowledge from smaller channels.” From Eq.9 and Eq.10, I don’t seem to arrive at this conclusion. Could the author provide further clarification?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zRKyd4eG85", "reviewer": "Reviewer_wbp7", "summary": "This paper presents a method to enhance logits distillation by incorporating a ranking loss based on Kendall’s τ coefficient. The main finding is that the conventional knowledge distillation approach, which relies heavily on KL divergence, often neglects smaller logit channels that contain valuable information. The authors introduce a ranking loss that ensures the order consistency between the teacher and student logits. The main algorithmic idea is to combine the traditional KL divergence loss with the proposed ranking loss, allowing the student model to not only align with the teacher's probability distribution but also maintain the ordinal relationship among logits.\n## update after rebuttal\nI have carefully read the rebuttal and the other comments. The rebuttal provide the new experimental results and the explanation of the motivation and clearly illustrate the scenario that low-probability channels receiving smaller gradients under the KL divergence. The rebuttal also provide the role of the ranking loss for the the gradients of low-probability labels. Some of my concerns are well solved. Therefore, I change my score. However, some issues like the motivation of using ranking loss, the innovation of the ranking loss, are not clealy discussed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the in-depth feedback! We highly value each of your comments, and all your concerns are addressed point by point:\n\n---\n\n**Q1: Claims And Evidence**\n1. **Small gradient**: We would like to clarify that while small gradient information does not necessarily imply ineffectiveness in the learning process, it is indeed prone to being overlooked. Our goal is for the model to learn more from the distribution of low-probability channels rather than ignoring them. Furthermore, we do not treat low-probability channels as dominant information; instead, we ensure that the model learns more consistently ranked logits under the constraint of KL divergence, thereby better capturing the knowledge hidden within low-probability channels.\n2. **Optimal solution space**: We would like to clarify that the optimal solution space primarily to illustrate the compatibility between ranking loss and KL, meaning that optimizing ranking loss does not significantly interfere KL. However, this is not the main reason why ranking loss improved distillation performance.\n\n---\n\n**Q2: Theoretical Claims**\nWe would like to clarify that correct ranking inherently encompasses the notion of correct classification. Notably, the teacher model generally achieves accurate classification on the training set; therefore, ensuring consistency in the ranking order between the student and the teacher, to some extent, also guarantees correct classification for the student. Consequently, the ranking loss serves to enhance the classification accuracy of the student model.\n\n---\n\n**Q3： The rationality of the motivation**\n\n1. **The rationality of introducing the ranking loss**: The motivation for introducing ranking loss is to help the student model better capture the inter-class information of small channels and help the student model classify accurately without affecting the optimization of KL divergence. These are the three main perspectives we discuss in the paper. Figure 1 is not a hypothetical scenario, it has also been mentioned in works such as LSKD [1].\n\n2. **Limited improvement**: Our average improvement is 0.85, which we believe is effective. Considering the capacity of the student model and its gap with the teacher model, we suggest that our method effectively improves the performance of distillation.\n\n3. **Time comsumption**: **Table T4.1** shows the distillation optimization time with and without ranking loss. It can be seen that ranking loss only brings a small extra time consumption.\n   \n**Table T4.1**: Additional Computation Evaluation of Ranking Loss. The full training time is reported as the evaluation metric\n  | Dataset | CIFAR-100|\n  |:---:|:---:|\n  | KD      | 0.93 Hours |\n  | KD+Ours | 1.06 Hours |\n\n---\n\n**Q4: How the ranking loss enhances the classification ability of the student network**\nIntuitively, a tiger should be more like a cat than a fish. This relationship between channels can help the model better capture generalization features. According to previous work [2], the essence of distillation is to help the student model to learn features from more views to improve generalization. Therefore, ranking loss can help the student model generalize more effectively. Ranking loss enhances the training classification accuracy of the student model and better understanding the knowledge of small channels without affecting the optimization of KL divergence, thereby improving the generalization ability of the student model.\n\n---\n\n**Q5: Other types of ranking losses.**\nGoodman Kruskal's Gamma can be understood as Kendall's with ties ignored, but these ties have no gradients when optimizing Kendall's taus, so they are essentially the same when used for optimization. Considering the form of Spearman’s rank correlation coefficient, it is difficult to convert it into a differentiable form. Considering the above reasons, using the Kendall coefficient tau as a ranking loss is a suitable choice.\n\n---\n\n**Q6: Derivation details**\nBased on equation 23, we can simply get equation 29. Considering that the sum of the probability output by the model is 1, we can get equation 30. Expanding equation 28 and substituting it into equations 29 and 30, we can get equation 31. We will further clarify the steps here in the revision.\n\n[1] Logit Standardization in Knowledge Distillation\n\n[2] Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning", "claims_and_evidence": "1.\tThe paper merely states that using KL divergence for distillation loss ignores the gradient information of low-probability labels, but this viewpoint lacks any theoretical support. In Equation 3, $q_i^t$ merely indicates that the gradient information for small labels will be small, which does not imply that they are ineffective during learning. Furthermore, low-probability labels themselves should not dominate the information in the distillation process.\n2.\tThe proof of consistency between the ranking loss and the KL divergence optimization objective has only been preliminarily explored from the perspective of the optimal solution space, without a thorough analysis of their relationship in the context of complex models and data. This insufficient analysis cannot fully explain why the ranking loss can effectively assist in knowledge distillation. The presentation merely elaborates on the advantages of the ranking loss, lacking more rigorous mathematical proof and theoretical support, and fails to provide an in-depth explanation of the internal mechanisms of the proposed method.", "methods_and_evaluation_criteria": "The evaluation criteria employed in the paper include accuracy metrics (top-1 and top-5) on benchmark datasets like CIFAR-100, ImageNet, and MS-COCO. These datasets and evaluation are widely used in KD.", "theoretical_claims": "The paper does not elaborate on any theory. In Section 4.2, it discusses the advantages of the ranking loss from three different perspectives. The proofs for the first two perspectives do not contain typos, but the analysis for the third perspective is clearly incorrect. Knowledge distillation aims to enhance the generalization of the model through the KL divergence and ensures the correctness of classification through cross-entropy loss. Assigning a larger weight to the distillation loss is precisely to ensure that the logits of the student network are closer to those of the teacher network. This analysis does not demonstrate that the ranking loss can help the student network achieve better classification. This part of the analysis should instead focus on how the introduction of the ranking loss can help the student network better ensure that the optimal label dominates, reducing the risk of low-probability labels exceeding the optimal label during the learning process.", "experimental_designs_or_analyses": "There are three issues with the experimental design section of the paper: \n1. The experiments conducted in the paper are insufficient, as many teacher-student combinations have not been covered, such as Res56/Res20, wrn40-2/wrn16-2, vgg13/mbv2, res32x4/shv1, etc. \n2. In the experiments on ImageNet, the top-5 accuracy results are missing, and the experimental results for DKD+, CTKD+, and MLKD+ are also lacking. \n3. The advantages compared to the logit-standard method are too insignificant, with differences of only 0.1-0.2.", "relation_to_broader_scientific_literature": "1. The paper proposes an auxiliary ranking loss based on Kendall's τ coefficient to mitigate the neglect of low-probability channels by the Kullback-Leibler (KL) divergence loss. This ranking loss is designed to focus on the alignment of logits between the teacher and student models, taking into account both high- and low-probability channels. This contribution builds upon and extends the existing knowledge distillation framework by incorporating a new loss function that addresses a specific limitation of the KL divergence.\n2. The paper demonstrates that the proposed ranking loss can be used as a plug-and-play auxiliary function in various distillation tasks, which is similar to the setting in Logit-standard KD.", "essential_references_not_discussed": "These two works should be selected as comparison methods in the experimental setup.\n[1] A unified approach with normalized loss and customized soft labels\n[2] DOT: A Distillation -Oriented Trainer", "other_strengths_and_weaknesses": "My major concern is the rationality of the motivation. \n1. Although the KL divergence has the issue of neglecting low-probability channels, the paper does not clearly elaborate on the rationality of introducing a ranking loss in knowledge distillation. Figure 1 seems more like a hypothetical scenario and lacks real-case analysis. \n2. Second, the differences in experimental results compared to other SOTA methods are too small to sufficiently demonstrate the effectiveness of the proposed method. \n3. Third, there is no detailed analysis of the impact on model training time and computational resource consumption after adding the ranking loss.", "other_comments_or_suggestions": "1. Conduct an in-depth analysis of the gradient characteristics of the ranking loss based on Kendall’s τ coefficient, for instance, by rigorously mathematically deriving and proving its stability with minimal influence from channel scale across different model structures and data distributions.\n2. Utilize visualization  (such as feature importance visualization, label information visualization, etc.) to conduct an in-depth analysis of the specific role that the inter-class relationship information provided by the ranking loss plays in knowledge distillation.", "questions_for_authors": "1. The paper mentions that the gradient of the ranking loss is less affected by channel scale and that its optimization objective is consistent with the KL divergence. Can we theoretically analyze how the ranking loss enhances the classification ability of the student network?\n2. There are various types of ranking losses. What are the reasons for choosing Kendall’s τ? Can we add comparisons with other ranking losses in the experimental section?\n3. In the appendix, could you provide some derivation details on how Eq. 28 is derived to Eq. 31?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Standard knowledge distillation often uses KL divergence to align student-teacher logits. This paper argues this can lead to suboptimal solutions and neglect low-probability channels, hindering learning and inter-class relationship transfer. It then introduces an auxiliary ranking loss, based on Kendall's tau, to mitigate these issues. By constraining logit rank similarity, the proposed loss provides scale-invariant gradients, better captures inter-class information, and helps avoid KL divergence's pitfalls without altering its main objective. Theoretical analysis and experiments across multiple datasets/architectures suggest that this ranking loss can enhance various logit-based distillation methods.

Reviewers generally acknowledged the innovative nature of leveraging ranking information in this context and recognized the potential benefits. The reviewers were also generally positive about baseline experiments across diverse datasets using both CNN and ViT architectures, ablation studies, and helpful visualizations. However, some concerns were raised around the paper's positioning within the current literature and the rigor of its claims. Reviewers (Za63, vqaG, wbp7) raised concern about missing comprehensive comparisons with SOTA distillation methods, specifically methods focusing on inter-class relationships like WKD and DIST, as well as strong KL-based techniques such as NKD, CTKD, WTTM, and LSKD. Additionally, reviewers (wbp7, Zvoo) expressed skepticism about whether KL divergence truly "neglects" low-probability channels detrimentally or if the provided examples were sufficiently representative. Less critical concerns were the missing analysis of computational/memory overhead (wbp7), clarification on numerical range of the loss and hyperparameter effects (Za63), the choice of Kendall's tau over other ranking metrics (vqaG, wbp7), and the theoretical justification for certain claims (wbp7, vqaG).

In the rebuttal, the authors presented results demonstrating that their auxiliary ranking loss consistently improved the performance of several requested SOTA methods (including NKD, CTKD, WTTM, LSKD, DIST, and WKD) when used as a plug-in. They also showed the additional computational time was minimal. They further addressed technical questions regarding the loss range, the choice of Kendall's tau, and provided clarifications on derivations and theoretical points. This thorough response successfully resolved most major issues raised by (Za63, Zvoo, wbp7) and they increased their initial score. (vqaG) acknowledged the new results but maintained a weak accept, primarily due to a perceived lack of deep methodological discussion comparing ranking loss and WKD, despite the authors providing performance comparisons. Overall, the rebuttal significantly strengthened the paper's standing, leading to a final consensus leaning towards acceptance. In agreement with the reviewers, I find the idea of using Kendall's tau for knowledge distillation a promising direction, and recommend accept.
Decision: Accept (poster)