PLD: A Choice-Theoretic List-Wise Knowledge Distillation
Authors: Ejafa Bassam
Dawei Zhu
Kaigui Bian
Venue: NeurIPS 2025 poster ()
Keywords: knowledge distillation
model compression
Plackett–Luce
list‐wise ranking
teacher–student networks
Primary area: deep_learning
TL;DR: A choice-theoretic loss that distills knowledge by teaching the student the teacher’s class ranking under the Plackett–Luce model, keeping the true label as the top choice and unifying distillation without an additional cross-entropy term.

Abstract: Knowledge distillation is a model compression technique in which a compact "student" network is trained to replicate the predictive behavior of a larger "teacher" network. In logit-based knowledge distillation, it has become the de facto approach to augment cross-entropy with a distillation term. Typically, this term is either a KL divergence that matches marginal probabilities or a correlation-based loss that captures intra- and inter-class relationships. In every case, it acts as an additional term to cross-entropy. This term has its own weight, which must be carefully tuned. In this paper, we adopt a choice-theoretic perspective and recast knowledge distillation under the Plackett–Luce model by interpreting teacher logits as "worth" scores. We introduce *Plackett-Luce Distillation (PLD)*, a weighted list-wise ranking loss. In PLD, the teacher model transfers knowledge of its full ranking of classes, weighting each ranked choice by its own confidence. PLD directly optimizes a single "teacher-optimal" ranking. The true label is placed first, followed by the remaining classes in descending teacher confidence. This process yields a convex and translation-invariant surrogate that subsumes weighted cross-entropy. Empirically, across CIFAR-100, ImageNet-1K, and MS-COCO, PLD achieves consistent gains across diverse architectures and distillation objectives, including divergence-based, correlation-based, and feature-based methods, in both homogeneous and heterogeneous teacher–student pairs.

PLD: A
List-Wiss Knowledge
Distillation
Ejafa Bassam" Dawei Zhuf Kaigui Bian' School of Computer Science, Peking University
Abstract
Knowledge distillation is model compression technique in which a compact "stu- dent" network is trained to replicate the predictive behavior of a larger "teacher" network. In logit-based knowledge distillation, has become the de facto ap- proach to augment cross-entropy with distillation term. Typically, this term is either a KL divergence that matches marginal probabilities or a correlation- based loss that captures intra- and inter-class relationships. In every case, it acts an additional term to cross-entropy. This term has its own weight, which must be carefully tuned. In this paper, adopt choice-theors knowledge distillation under the Plackett-Luce model by interpreting teacher logits "worth" scores. We introduce Plackett-Luce Distillation (PLD), a weighted list-wise ranking loss. In PLD, the teacher model transfers knowledge of its full ranking of classes, weighting each ranked choice by its own confidence. PLD directly optimizes a single seacher-optimal ranking. The true label placed first, followed by the remaining classes in descending teacher confidence. This process yields a convex and iranslationiinvariant surrogate that subsumes weighted cross- entropy. Empirically, across CIFAR-100, ImageNet- IK, and MS-COCO, PLD achieves consistent gains across diverse architectures and distillation objectives, including divergence-basee correlation-bass and feature-basse methods, in both homogeneous and heterogeneous teacher-studer pairs.
perspective and recast
1 Introduction
Deep neural networks (DNNs) have achieved remarkable success across a wide array of tasks-from
image classification and object detection to semantic segmentation and beyond [47] 41 Accuracy,
generalization, and robustness usually improve as models become deeper and larger 42 However, these gains require more computation and memory, which limits deployment on mobile or embedded
devices.
Knowledge distillation (KD) 12 addresses this by training a compact "student" network to mimic a large "teacher" network. Classically, KD minimizes weighted sum of cross-entropy on hard labels and lemperature-scaled Kullback-Leiler (KL) divergence between teacher and student logits. This simple framework enriches the student's learning signal and has seen widespread adoption for model
compression and robustness enhancement 9 34
Despite its popularity, recent studies reveal paradox. Distilling from larger more accurate teachers can sometimes degrade the student's performance, especially when there is a capacity mismatch Several methods have been proposed address this issue. These include teacher- assistant frameworks [24] 33), selective distillation [ (44, and auxiliary classifiers [18] However, these methods often add extra architectural components training stages. An alternative line of
24
13
"Email: ejafabassam @ stu, pku.edu.cn
Email: dwzhu@ @ Serrhh@@ku..duu. pku.edu.cn
1 'Corresponding author; Email: bkg@ pku.cdu.n
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

15.72
78-14
76.56
74888
7037
PLR 2mmm
(a) CF-weiigh sensitivity for KD and DIST.
(b) KD, DIST, and PLD after 100 vs. 300 epochs.
Figure 1: (a) Varying the CE mixing weight reveals that KD and DIST have different sensitivities- too much CE hurts both, while a sweet spot near a = 0.1 maximizes Top-1 accuracy. (b) Under extended training (100 300 epochs), PLD consistently outperforms both KD and DIST, demon- strating its sustained gains.
work, DIST 13
remains within the classical KD pipeline yet replaces the KL-based distillation term. DIST argues that matching only marginal probabilities via KL divergence fails to preserve the relational structure encoded in the teacher's outputs.
Instead, DIST employs Pearson correlation-base loss to preserve both inter-class correlations within each prediction and intra-class correlations across examples. Nevertheless, like other distillation methods, DIST still requires a separate cross-entropy term. As Figure [a shows, reducing the weight on this cross-entropy term can improve accuracy, but removing it entirely leads to performance drop. Therefore, a unified objective that combines cross-entropy and distillation is desirable. Such an objective should derive its weights from the teacher's confidence rather than manual tuning.
To overcome these limitations, propose list-wise, thoice-theoretic perspective distillation. We interpret logits as "worth" scores under the classical Plackett-Luce model [22] 288. We then derive a single teacher-optimal ranking, where the label comes first and the remaining classes follow descending teacher confidence. This ranking is imposed on the student using the unsoftened Plackett-Luce likelihood. The PL model based on Luce's Choice Axiom, which ensures that choice probabilities are invariant to irrelevant alternatives. Plackett's 's extension generalizes this to full rankings, forming coherent distribution over permutations through sequential removal. The first selection term of the PL model matches the cross-entropy on the true label. Thus, it naturally includes the standard KD cross-entropy loss. However, similar [31 16] we observe that proper weighting of each subsequent selection is crucial (see Sec. 3.. We weight each step k by the teacher's softmax mass ag. This leads to the Plackett- Distillation (PLD) Loss, a convex and translation invariant surrogate. PLD unifies cross-entropy, ListMLE (45), and P-ListMLE [6 as special cases. This choice-theoretic foundation captures the full ranking structure provided by the teacher and enables efficient, gradient-based optimization, eliminating the need for separate cross-entropy terms or manual weight adjustments.
Our main contributions are:
We introduce a novel list-wise distillation objective that enforces the teacher-optimal per- mutation π -the unique ordering that places the ground-truth label first and ranks all other classes by descending teacher logits-via the Plackett-Luce likelihood, with each selection step k weighted by the teacher's softmax ak, thereby eliminating the need for ad-hoc weight tuning.
We show PLD is convex, smoothly differentiable with closed-fomn gradients, and subsumes CE, ListMI.E, and P-ListMI.E. Like classical KD and DIST, PLD is simple and efficient, easy to implement (see Appendix C). and requires no architectural modifications.
We empirically demonstrate on CIFAR-100, ImageNet-Iee and MS-COCO that PLD achieves consistent gains across diverse architectures and distillation objectives, including divergence based, correlation- based, and feature- based methods, in both homogeneous and heterogeneous teacher-stude student pairs.
The rest of the paper is organized as follows. Section 2 reviews prior work on KD and ranking losses. Section presents necessary preliminaries. Section derives the PLD loss and analyzes its properties. Section reports empirical results. Finally, Section
6 concludes with discussion of future directions.

---

2 Related Work
2.1 Knowledge Distillation and Capacity-Mismatch Remedies
Knowledge distillation (KD) 12]. inspired by model-bootstrapping techniques 53 trains compact student to mimic a larger teacher. It minimizes a temperature-scaller KL divergence between their output logits (softened probabilities). This process transfers both accuracy and robustness [9] while enabling model compression [34]. Over time, this simple yet effective framework has become a cornerstone for compressing and enhancing deep neural networks. However, a range of studies have shown that students distilled from very large or highly accurate teachers can underperform, even This counterintuitive degradation is mainly ascribed to a capacity mismatch between teacher and student. To address this issue, several architecture-level methods introduce intermediate or auxiliary models. Teacher Assis- tant KD (TAKD) employs a mid-sized assistant to bridge the gap [4]. Densely Guided KD (DGKD) aggregates multiple assistants for richer supervision [33. Neighbor Self-KD (NSKD) adds auxiliary classifiers within student layers [18]. Student Customized KD (SCKD) adjusts the distillation loss based on gradient alignment with the student's objective [55]. Teacher- reacher-knowledge regularization techniques also refine the teacher's signal. Etrly-stopped teachers often outperform fully converged
under ad versariially robust settings 3 26 24
33
55
39
54
13
18
50 48
ones [3] CheckpointKD selects intermediate checkpoints to prevent over-specialization [39]. Some
methods exclude undistillable classes to focus supervision on reliable predictions [54]. Adapter
modules control label smoothness [30], and coudent-friendly KD (SKD) simplifies teacher outputs through a softening-simplifier pipeline (50).
2.2 Logit-Based Distillation Beyond KL Divergence
This trend is reflected in recent logit-based distillation methods that seek alternatives to KI. divergence. Relational KD (RKD) (66 transfers mutual relations among data examples via distance and angle- wise losses that penalize structural discrepancies. DIST [3] introduced Pearson correlation coefficient-based loss to explicitly capture the teacher's intrinsic inter and intra-class relations. Correlation Matching KD (CMKD) (25] reinterprets inter-class rankings in terms of the decision boundary and employs both Pearson and Spearman correlation losses, dynamically weighted by sample difficulty using differentiable sorting operation. It also observed (6) that better teacher calibration correlates with improved KD performance under standard methods. This finding motivated calibration inseessivvee metrics such ranking-basse based losses. These methods are computationally efficient, require no complex training pipelines or architectural overhead, yet achieve competitive performance. This observation motivates search for similarly efficient, calibration-insensttiee objective grounded in list-wise ranking theory.
2.3 Ranking-Basss and List Losses for Distillation
Early ranking methods focused on pairwise losses. In these methods, each pair of items classified as correctly or incorrectly ordered, as in the ranking SVM [4]. Listwise approaches, which operate over entire item lists, later gained traction in information retrieval and machine learning 2 (45. Many draw on classical statistical ranking models most notably Luce's 's choice model (22) and the Plackett-Luce permutation distribution (28). As the number of items increases, even stochastic top-k extensions such as ListNet 23 become impractical. Moreover, these losses designed for direct supervision and do not naturally extend to knowledge distillation.
More recent work applies ranking losses to classification and distillation. For multiclass tasks, listwise losses have been proposed [40, 71. while distillation under a position-aware binary ranking loss has been studied (37 8). The method in [37] removes items that the teacher ranks low. In contrast, [7 adapts ranking-bar based distillation to collaborative filtering. Similarly, RankDistil [T preserves the teacher's top-k ordering by matching the student's item order to the teacher's and penalizing lower-ranked items.
Building on ListMLE 45). we directly optimize the likelihood of a single teacher-optimal permutation under the Plackett-Luce model. Inspired by RankDistil (31) and posetion-aware ListMLE (P-ListMLE) [6]. we assign greater weight to top-ranked classes than to those ranked lower. Unlike RankDistil, which designed for general ranking tasks and optimizes top-k metrics, our method focuses on classification tasks. In this setting, top-1 accuracy is most important.
3

---

3 Preliminaries
We first review the multiclass classification and classical knowledge-distillttio framework, then introduce the Plackett-Luce permutation model for rankings.
3.1 Multiclass Classification and Knowledge Distillation
Let D= = ,(x(")))"))) ) be a training set of N examples, where each x( E Rd and y(n) E {1...., C}. neural network fo! I + EC maps an input æ to logit vector 8 = fo(z) = (81..... sc) E RC, which we refer to the network's logits & RCC These logits induce a softmax distribution over classes:
exp(si)
plV = s)
i= I....,C. C.
Σラ exp(8;)
The standard cross-ent coss-entropy loss is
CcE(s,y) logr(y (8) By log
which depends only the target logit sy and is therefore intransitive: any permutation of the other logits leaves LCE unchanged.
In knowledge distillation, pretrained teacher fr and student fs produce logits
(x), = fzx).
We soften these via temperature 7 > O: exp(tt/T) Σj= exp(tz/7))
o exp(s:/T)
of
E exp(8//77)
and measure their divergence via temperature-seales KL term:
log df
LKD(S, () =2 KL, 114( =
2m
The student minimizes the combined loss
L(s,y) = LCR(s, v) + (1 a) LsD(s,
where a E [o, 1] balances fitting the hard labels against matching the teacher's full output distribution.
Minimizing L encourages the student both to place the correct class first and to mirror the teacher's full distribution. In Section we extend this perspective by imposing a full Plackett-Luuce ranking over the logits.
3.2 Plackett-Luce (PL) Ranking Model
Let RC be the logit vector produced by the network. We write s; for its ith component and interpret i > - 8; > 8j as "class preferred to class j. A full ranking is a permutation (...... ..... πC)= E Sc. The Plackett-Luce model assigns each class strictly positive worth w; (8,) > O, where & : IR + Rs0 is strictly increasing (we take ((s) = e²). It defines a
probability over permutations by
(T ()
II k=
WT
II A=
exp(sT.)
Lizk exp(sn,)
Di=k
At each step k, the class ग& is chosen from the remaining items in proportion to its worth.
A key property is translation- inviriancee adding å to all logits (i.e. s; + 6) multiplies every w; by e², which cancels in the softmax-sty ratios. By contrast, scaling logits by a > 0 (i.e. 8; + a a;) changes the sharpness of the distribution.
The PL model factorizes the full C!-way ranking distribution into chain softmax terms. This
factorization provides an efficient likelihood for any fixed permutation 45 In Section 4 we fix "teacher-optimal" permutation x and minimize its negative log-likelihood under the student. This approach enforces both top-1 correctness and richer inter-class ordering information.

---

Method
We derive the PLD objective in stages. First, extract feacher-optimal permutation from the teacher's logits using the Plackett-Luce model (Sec. (41).. This step identifies the ranking target for distillation. Second, we introduce confidence-weighteed likelihood form the final Plackett-Luce Distillation (PLD) loss (Sec. 4.2
4.1 Teacher-Optimal Permutation under Plackett-I.uce
From the Plackett-Luce ranking viewpoint, a classifier's logits define sequential choice process among the C' classes. the first selection step, the model chooses one class out of all C in proportion to exp(s;), exactly matching the softmax probability used in standard cross-entropy. Consequently, minimizing cross-entropy enforces only the first-choice probability, i.c., the probability of selecting the correct class first. It remains agnostic to the ordering of the remaining C' 1 classes. In contrast, the full Plackett-Luce model assigns probabilities to all C! possible permutations. by chaining softmax-style selections at each step. Thus, cross-entropy can be seen a relaxed Plackett- loss that cares only about the top-1 selection.
does this
term, such as KL matching or correlation- If we augment cross-entropy with a ghowledge-distilaatee based loss, the objective still focuses mainly on the true class through the cross-entropy component. The distillation term only adjusts the remaining logits. Viewed through the Plackett-Luce perspective, this process enforces the first selection shapes later choices according to the teacher's preferences instead of matching the entire distri- bution. Concretely, since Plackett-Luce makes each selection independentyy-coosii the remaining classes in proportion to their worth-we fix the first pick to the true label and let the teacher's s descending-loog ordering guide all later picks. Denoting the teacher's logit vector by t == tc) RC, refer to the resulting fixed ordering
the ground-truth label, as in cross-entropy. It then
mi among
π² (y, argsort(() {v))
as the teacher-optimal permulation, and use x² our sole ranking target.
construct surrogate losses on single Plackett-Luce permutation. Examples include likelihood-basse (ListMLE), cosine-similarity, and cross-entropy variants, all suitable for gradient-basee optimization. Among these, the likelihood (ListMLE) surrogate stands out for its simplicity and for satisfying desirable properties such as consistency, soundness, continuity, differentiabilityy and convexity, while also exhibiting strong empirical performance. We adopt the ListMLE approach. We place an empirical one-hot target on the teacher-optimal permutation x' and define its negative log-like likelihood under the student's 's logits as the unweighted loss.
As shown in [45.
exp STZ
Lunweighted (s: = log PPL PrL (s) -
- log k=l
Σie exp(s>;)
This formulation naturally yields one-hot target over the C! Plackett- permutations cemmutations-assinnnng assigning probability one to πT" and zero to all others-and serves as the foundation for our weighted distillation objective.
4.2 Confidence- Likelihood Yielding PLD
In the standard PL model, each selection step is weighted equally-ie.., every position contributes identically to the likelihood. Position-aware ListMLE (P-ListMLE) [16] addresses this by introducing a fixed, strictly decreasing weight sequence, such as a3 =
2 C- 1.
This scheme penalizes enrors at top ranks more than those lower down. However, in multiclass classification-where only the top-1 decision ultimately matters- fixed, hand-crafted weight schedules are both ineepparamttersensiitie selection compared to subsequent ones. Observing that pretrained teacher naturally assigns far greater confidence to the correct class than to the others, we therefore parameterize the weight at step by the teacher's softmax probability: ag = 4"
and fail to reflect the inherently greater importance of the first
This choice ensures the loss automatically emphasizes the top-1 selection when the teacher is confident yet relaxes its focus when the teacher's output distribution is more uniform. Furthermore, data-
5

---

Method
78
77.3
76.6
75.98
76.33
DIST
(96
75.63
75.47
75.4
PLD
74.91
74.86
72
70.05 70.07
67.38
VIF-Small
ResNet-50 50
MiNet4:MMe dlum(Hybrid)
MMetV4:Meddlu(Coo
MNetV4:55mal (Conv)
Student Model
(a) Top-1 accuracy in the homogeneous distillation setting.
ResNet-50 Top-1 Accuracy (%)
Method
KD
DIST
PLD
77.3
77.34
76.85
76.86
76
75.53
75.4
75.05
75
74.27
74
73
ResNet-152
MN Meett/4 (Hyrii))
MNatVA:LargelCCnn VIT-8/16
VIT-L/26
Teacher Model
(b) Top-1 accuracy in the heterogeneous distillation setting.
Figure 2: (a) Homogeneous setting: larger teachers and smaller students within the same architecture family. (b) Heterogeneous setting: fixed ResNet-50 student distilled from diverse teacher architec- tures.
driven weights recover several known ranking surrogates as special cases. Setting 0,.0..... ... 0) gives the standard cross-entropy. A uniform a = (c...., d) produces ListMLE-like objective, and any fixed decreasing sequence yields the P-ListMLE surrogate.
The resulting Plackett-Luce Distillation (PLD) Loss is defined as
(PLD(s,t:))
Σ
S#z + log >e
thereby providing unified, saffidence-weighter ranking objective for knowledge distillation. Since softmax-weighted weighted variant ListMLE, inherits the same favorable
PLD can be viewed as a teacher softmax---f structural properties. In particular, each summand
S
log
Emk
is convex in the logits as it is the sum of an affine function and a log-sum-exp. Because nonnegative weighting by oi; and summation over k preserve convexity, the full PLD loss remains convex s. This convexity, together with smooth differentiability, ensures efficient, gradient- optimization. The gradient derivation is provided in Appendix A

---

Experiments
We evaluate PLD on three representative visual recognition datasets: CIFAR-100 [5] ImageNet- IK [. and MS-COCO [20]. These datasets cover small- and large-scale image classification and general object detection.
We first benchmark PLD on CIFAR-100 using traditional convolutional architectures and a unified training recipe. We then assess scalability ImageNet-Ike where we adopt stronger architectures and optimized training strategies. Finally, we test PLD on MS-COCO for object detection to examine generalization beyond classification.
5.1 CIFAR-100 Classification
We compare PLD with classical and feature-basse knowledge distillation methods on CIFAR-100 [5] The dataset contains 50,000 training images and 10.000 validation images across 100 categories at 32 x 32 resolution. All models are trained from scratch for 250 epochs. We use the AdamW optimizer with 31 =0.9 and =0.999.. The learning rate follows a aesine-annealing schedule, starting at 0.001. We set weight decay to 0.5 and the batch size to 128.
Table I: Evaluation results on the CIFAR-100 dataset. The upper and lower models denote the teacher and student, respectively.
Homogeneous setup
Heterogeneous setup
Method
WRN-40-2
ResNet-56 ResNet-8xx4 ResNet-20
ResNet-32xx4
ResNet-50 MobileNetV2
WRN-40-1
Student (CE)
70.26.10.24
67.57
71.56100.16
64.42.10.67
Feature-hased methods
AT [1
71.9910.34
68.42-10.16
72.5110.37
52.28110.84
FitNet
70.86=2.33 66.75=0.27
72.59+0030
62.77=0.005
PKT
27
71.43=0.116 68.66=+0.20
72.86+0.17
66.08++0.11
RKD
26
11.32+1.84 68.53.10.18
71.45-L0.41
65.16:10.21
SP 38
73.4910.20 69.36.10.27
72.4510.08
65.30.10.29
VID
70.61=0.39 68.52=0.12
71.70+0.32
64.14=0.82
Logits-based methods
KD 12
72.44=0.36 69.45-0..24
72.11+0.10
66.61#1.16
DIST 13
72.74=0033 69.66=00499
73.21+0.10
66.98:t0.74
DKD 522
73.22:10.38 67.80-10.03
73.08+10.27
69.51-10.42
PLD (ours)
73.50
70.28 73.97-10.05
68.16110.38
We apply standard data augmentations: random cropping, horizontal flipping, and per-channel normalization. We report the mean and standard deviation over three independent runs. For DIST [I3), we set 1=110 for the cross-entropy term, 3=2.0 for the inter-class relation loss, 7=2.0 for the intra- class relation loss, and use a temperature of 440.0 PLD uses the same temperature (r=4.0).
5.2 ImageNet-1K Classification
All experiments on ImageNet- IK 4 are conducted following the recipe [44 with minimal data augmentation. We train for 100 epochs with an effective batch size of 2048 images (256 per GPU across eight NVIDIA A100 SXM4 80 GB accelerators) using the LAMB 49 optimizer, initial learning rate of 5 x 10-3 decayed by a cosine schedule and linearly warmed up over the first epochs, and weight decay fixed 0.02. Under this setup, 100 epochs require approximately 6 hours for convolutional backbones and hours for Vision Transformers. To isolate the effect of our losses, use only random resized crops, horizontal flips, and per-channel normalization.
For validation we adopt the "A-recipe" from test crop ratio p 0.95, then resizemin
Specifically, we set the test resolution r = 224 and r/p ~ 236, apply a bicubic resize of the shorter side

---

to resizemin. followed by center crop of size r x r. We normalize by the same mean and standard deviation in training.
Table Top-1 accuracy (%) and student model size across student-teacher pairs on ImageNet- "Teacher Acc" is the teacher's standalone Top-1; ADIST and ÅKD are PLD's gains over DIST and KD.
Top-1 % Accuracy
Student
Teacher
Params(M)
Teacher
KD
DIST
PL.
ADIST
AKD
ViT-Small
331-La)ge(304.33M)
22.05
84.80
75.33
74.91
75.63
0.72
0.30
ResNet-50
ResNet- ResNet-152(60.195 152(60.19M)
25,56
79.61
76.80
76.60
77.30
0.70
0.50
MobileNet-v4 Hlybrid-Medinn
Large (conv)
11.07
80.83
75.47
75.98
76.33
0.35
0.86
Medium (conv)
(32.59M)
9.72
80.83
74.86
75.40
75.72
0.32
0.86
Small (conv)
3.77
80.83
67.38
70.05
70.07
0.02
2.69
Table Top-1 accuracy of ResNet-50 distilled from diverse teachers on ImageNet-IK "Params" is the teacher's parameter count in millions. DIIT and ÅKD report PLDD's gains over DIST and KD, respectively.
Top-1 % Accuracy
A
Teacher
Params (M)
Teacher
KD
DIST
PLD
ADIST
AKD
ResNet-152
60.19
79.61
76.80
76.60
77.30
0.70
0.50
MobileNet-v4 Hybrid Medium
11.07
78.66
76.85
77.00
77.34
0.34
0.49
MobileNet-v4 Conv Large
32.59
80.83
73.98
75.05
75.53
0,48
1.55
ViT-Base/16
86.57
82.07
74.27
75.40
75.80
0.40
1.53
ViT-Large/16
304.33
84.80
75.98
76.86
77.38
0.52
1.40
Table Top-1 accuracy (%) of the MobileNet-v4 Medium student distilled from a MobileNet-v4 Large teacher after 100 and 300 training epochs. The "A" row shows the improvement from 100 to 300 epochs.
Epochs
CE
KD
DIST
PL.D
ADIST
AKD
100 epochs
72.37
74,86
75.40
75.72
0.32
0.86
300 epochs
73.04
76.14
76.56
76.94
0.38
0.80
A (300-100)
0.67
1.28
1.16
1.22
5.2.1 Baseline Configuration and Ablation
We evaluate four losses under identical training: CE, KD 12.. DIST [3] and PLD. We use a ResNet- 50 student and ResNet- 152 teacher [0O]. Each objective replaces only the logit-based term and has comparable cost. Thus, KD (divergence-
and DIST (correlation-bar basedd) are natural baselines for our ranking-basse based PLD. We sweep the key hyperparameters for each method: a for KD, 8, for DIST, and Tr for PLD. Figure Ta shows the effect of the CE mixing weight. Reducing improves KD and DIST to a region near a ~ 0.1. Setting = 0 (no CE) degrades performance. Table 5 reports results with and without CE (see Table 8 for the full ablation). Table lists PLD-specific sweeps.
5.2.2 Ablation of PLD
(1)
As PLD weights each position by ag
Ei exp (/g /TT) we sweep TT E {0.5, 1.0, 2.0, 2.0,
to assess sensitivity: lower Ty sharpens the distribution, while higher 7y flattens the distribution. We also test special cases: Uniform (ListMLE): os 1/C. Position-aware (p-ListMLEF) Ok = u E3 oh.

---

As Table 6 shows, the unsoftened PLD (Tr = 1) consistently achieves the highest Top-1/Top-5 accuracies, with uniform ListMLE trailing by 2-3 pp. Overall, the best results occur 7r 1.0 (no softening) or with only slight perturbations (1.0 + e), demonstrating that PLD is robust to the exact choice of a distribution and does not require aggressive sharpening or over-softening.
Table 5: Top-1 and Top-5 accuracy for baseline methods (ResNet-50 student, ResNet- 152 teacher with Top-1 Acc. 79.61%) indicates not applicable. CE: standard cross-entropy. LS: [36] cross-entropy with label smoothing (e = 0.1). KD: vanilla distillation with mixing weight a and temperature DIST: relational distillation with inter-class weight 3. intra-class weight y, and temperature 1.
Method
Top-1 Acc. %
Top-5 Acc.%
Teacher (ResNct-152)
79.61
71.35
LS (c = 0.1)
73.92
KD
0.00
2.00
75.92
92.82
0.10
2.00
76.80
93.16
DIST
0.00
0.50
0.50
1.00
76.47
93.19
0.10
0.45
0.45
1.00
76.60
93.30
Table 6: Top-1 and Top-5 accuracy for PLD variants (ResNet-50 student, ResNet-152 teacher with Top-1 Ace. 79..1%). "_" indicates not applicable.
Variant
Top-1 Acc.%
Top-5 Acc.% %
PLD
0.50
76.06
92.39
1.00
77.30
93.28
1.50
77.17
92.74
2.00
76.46
92.34
4.00
75.22
91.34
HLD-ListMLA PLD -LL-ILiMMLE
74.11
90.60
76.60
93.23
5.2.3 Distillation Across Homogeneous Architectures
Figure () evaluates three backbone families in a homogeneous setting: ResNet [T0, MobileNNe-vv [29. and Vision Transformers (5). Across convolutional models, PLD outperforms KD by 0.5-2.7 pp and DIST by 0.2 0.7 pp. On ViT, PLD yields +0.72 pp over DIST and +0.30 pp over KD. Averaged over all pairs, PLD improves Top-1 accuracy by +0.42 versus DIST and +1.04 versus KD. Table reports full Top-1 results for KD, DIST, and PL.D.
5.2.4 Distillation Across Diverse Teachers
Logit-based distillation can use teachers from different architectures as long as the logits match the student's dimensionality. Figure b) shows the heterogeneous setting. We fix the student to ResNet-50 (= 25.56M parameters) and distill from range of convolutional and transformer teachers. Across all teachers, PL.D consistently outperforms KD and DIST. Its ĀDIST gains range from +0.34 pp (MobileNet-vv4 Hybrid) to +0.70 pp (ResNet-152),, averaging +0.48 pp. Its AKD gains range from +0.49 pp +1.55 pp. averaging +1.09 pp. teacher capacity increases, KD's benefit declines and DIST shows modest upward trend. PLD follows this trend and further widens the margin over DIST larger teachers. The largest gap between DIST and PLD appears
5osNet-50+-ReNeett pairing, Table 3 reports each teacher's Top-1 accuracy, parameter count, and the student's Top-1 under KD, DIST. and PLD, along with ADIST and AKD-
5.2.5 Consistency under Extended Training
Our homogeneous experiments (Sec. 5.2.3 show that KD sometimes beats DIST and vice versa. We whether PL.D consistently surpasses both under longer training. We train a MobileNet-vá Medium student from MobileNet-va Large teacher for 100 and 300 epochs with identical hyperparameters.

---

Table 7: Object detection performance MS-COCO. We distill Faster R-CNN detectors with FPN [19] using DKD [52] and PLD. All models are trained under identical schedules. Results are reported va12017.
Teacher + Student
Method
AP
APso
APrs
AP,
mm
APt
ResNet-50 - MobileNetV2-FPNN
DKD
29.24
50.42
29.87
16.15
31.11
38.07
ResNet-50 + MobileNetV2-FPN
PLD
28.91
49.63
29.88
16.05
30.41
38.59
ResNet- 101 ResNet-18-FPP
DKD
32.11
53.43
33.90
18.46
34.43
41.50
ResNet- 101 ResNet-18-FPN
PLD
32.47
53.83
34.17
18.50
34.97
42.12
ResNet- 101 ResNet-50-FPPN
DKD
36.54
58.57
39.46
21.74
39.80
47.31
ResNet- 101 ResNet-50-FPN
PLD
36.60
58.28
39.58
21.37
39.76
47.24
Table reports Top-1 on ImageNet-IK, and Figure 1b provides a detailed comparison. At 100 epochs, PLD achieves 75.72%. This is +0.32 pp over DIST and +0.86 pp over KD. At 300 epochs, PLD reaches 76.94% (+0.38 pp over DIST; +0.80 pp over KD). The PLD gains with longer training are +1.22 pp, which nearly twice the standard pretraining gains (+0.67 pp). This scaling is comparable across KD (+1.28 pp) and DIST (+1.16 pp). Thus, PLD preserves and slightly widens its advantage under extended training.
5.3 Object Detection MS-COCO
PL.D extends to tasks where the teacher and student share the same output logits, including object detection and semantic segmentation. We distill Faster R-CNN [9] detectors on MS-COCO (00 with standard FPNs. We evaluate two convolutional teacher-student pairs: ResNet-50 MobileNetV2- FPN and ResNet-101 + [ResNet-18-FP)) ResNet-50-FPN). All models use identical hyperparame- ters, and schedules. We compare PLD the strong DKD baseline [52]
Table shows that PLD achieves comparable or slightly better performance than DKD across multiple pairs. PLD improves AP and AP7s in most settings while maintaining similar AP, for small objects. These results indicate that PLD transfers structured knowledge for dense prediction without modifying the detector.
6 Conclusion and Limitation
In this work, we introduced Plackett-Luce Distillation (PLD), unified, choice-theoretic frame- work for logit-based knowledge distillation. Empirically, across CIFAR-100, ImageNet-I and MS-COCO, PLD achieves consistent gains across diverse architectures and distillation objectives, including divergence- based, correlation-baxe based, and fcature-based methods, in both homogeneous and heterogencous teacher-stude pairs. These results show that transferring the teacher's structured preferences yields stronger and more stable student performance than marginal- or correlation- objectives.
PLD applies broadly to any setting where the student and teacher share the same logit dimensionality. However, it assumes fully aligned class vocabularies and does not yet support mismatched label sets incremental class addition. PLD also requires sorting over the C output logits to extract the teacher-optimal permutation, which has OCC'log C') complexity per example. Although efficient in distributed implementations, this cost is higher than the O(C) complexity of standard KD or DIST. Empirical runtime analysis (Appendix[I II shows that the overhead is negligible for common architectures. As with other distillation methods, PLD's 's benefits depend on teacher confidence and may diminish when the teacher's softmax distribution is nearly uniform, illustrated in the loss landscape analysis (Appendix[]) J.
PLD's ahoice-theoreet foundation opens several directions for future work. First, adaptive or curriculum-driver weighting schemes could adjust the ranking loss based on sample difficulty. Second, extending PL.D to other domains, such as sequence modeling, reinforcement learning, or multitask learning, may offer similar gains. We hope this work encourages further exploration of ranking-based based objectives for principled and effective model compression.
10

---

Acknowledgments and Disclosure of Funding
This work is partially sponsored by National Key R&D Program of China under Grant 2022YFB3104200, NSFC U24A20235 and 62032003.
References
[1] S. Ahn, S. X. Hu. A. Damianou, N. D. Lawrence. and Z. Dai. Variational information distillation knowledge transfer. Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 9163-9171, 2019.
[2] Z. Cao. Qin, T.-Y. Liu, M.-F. Tsai, and Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pages 129-136, 2007.
[3] J. H. Cho and B. Hariharan, On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1794-4802. 2019.
[4] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248 255. leee. 2009.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, Heigold, Gelly, et image worth 16x16 words: Transformers for image recognition scale. arXiv preprint vrXiv:2010.1/922 2020.
[6] W.S. Fan, S. L.- Li, D.-C. Zhan, and L. Gan. Revisit the essence of distilling knowledge through calibration. In Forty-first International Conference on Machine Learning, 2024.
[7] A. Frydenlund, G. Singh, and Rudzicz. Language modelling via learning to rank. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 10636-10644, 2022.
[8] L. Gao. Z. Dai, and J. Callan. Understanding bert rankers under distillation. In Proceedings of the 2020 ACM SIGIR International Conference Theory of Information Retrieval, pages 149-152, 2020.
[9] J. Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129(6):1789 2021.
[10] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv preprint WiXiv:152..03385 2015.
[11] R. Herbrich, T. Graepel, Obermayer. Support vector learning ordinal regression. 1999 Ninth International Conference on Artificial Neural Networks ICANN 99.(Conf. Publ. No. 470), volume 1, pages 97-102. IET, 1999.
[12] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 15i.00550.0233,, 2015.
[13] T. Huang. S. You, F. Wang, C. Qian, and C. Xu. Knowledge distillation from stronger teacher. NeurIPS, 2022.
[14] T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133-142, 2002.
[15] A. Krizhevsky al. Learning multiple layers of features from tiny images. 2009.
[16] Y. Lan, Y. Zhu, J. Guo, S. Niu, and X. Cheng. Position-aware listmle: A sequential learning process for ranking, UAI, volume 14, pages 449-458. 2014.
[17] J.w. Lee, Choi, Lee, and Shim. Collaborative distillation for top-n recommendation. 2019 IEEE International Conference on Data Mining (ICDM), pages 369-378. TEEE, 2019.
[18] P. Liang, W. Zhang, J. Wang, and Guo. Neighbor self-knowledge distillation. Information Sciences. 154:1, 2024.
[19] T.-Y. Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.

---

[20] T.-Y. Lin. M. Maire, S. Belongie. J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects context. In European conference on computer vision, pages 740-755. Springer, 2014.
[21] Loshchilov and Hutter, Decoupled weight decay regularization. arxiv preprint arXiv:
2017.
[22] R. D. Luce, Individual choice behavior, volume 4. Wiley New York, 1959.
[23] T. Luo, D. Wang, R. Liu, and Y. Pan. Stochastic top-k listnet. arXiv preprint arXiv:1511.0022 2015.
[24] S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh, Improved knowl- edge distillation via teacher assistant. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5191-5198, 2020.
[25] W. Niu, Y. Wang. G. Cai, and H. Hou. Efficient and robust knowledge distillation from a stronger teacher based on correlation matching, arXiv preprint arXiv:2410.06561, 2024.
[26] W. Park, D. Kim, Y. Lu, and M. Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
[27] N. Passalis and A. Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European conference on computer vision (ECCV), pages 268-284, 2018.
[28] R. L.. Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193-25 202, 1975.
[29] D. Qin, C. Leichner, M. Delakis, M. Fornoni, S. Luo, F. Yang, W. Wang, C. Banbury, C. Ye, B. Akin, al. Mobilenety4: universal models for the mobile ecosystem. Eurppean Conference on Computer Vision, pages 78 78-96. Springer, 2024.
[30] J. Rao, X. Meng, L. Ding. S. Qi, X. Liu, M. Zhang, and D. Tao, Parameter-effiiicnt and student- friinnty knowledge distillation. IEEE Transactions on Multimedia, 2023.
[31] S. Reddi, R. K. Pasumarthi. A. Menon, A. S. Rawat, F. Yu, S. Kim, A. Veit, and S. Kumar. Rankdistil: Knowledge distillation for ranking. In International Conference Artificial Intelligence and Statistics. pages 2368 2376. PMLR, 2021.
[32] A. Romero. N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv: 1412.655, 2014.
[33] W. Son, J. Na, J. Choi, and W. Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9395- 9404, 2021.
[34] Song, Y. Chen, Ye, and M. Song. Spot-adaptive knowledge distillation. IEEE Transactions on Image Processing, 11:3359-3370 2022.
[35] S. Sun, W. Ren, J. Li, R. Wang, and X. Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 15731-15740, 2024.
[36] C. Szegedy, V. Vanhoucke, S. Toffe, J. Shlens, and 7. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[37] J. Tang and K. Wang. Ranking distillation: Learning compact ranking models with high performance for recommender system. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2289 2298, 2018
[38] F. Tung and G. Mori. Similarity-preseriing knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1365-1374, 2019.
[39] C. Wang, Q. Yang, R. Huang, S. Song, and G. Huang. Efficient knowledge distillation from model checkpoints. Advances in Neural Information Processing Systems, 35:607-619, 2022.
[40] N. Wang, Z. Qin, L. Yan. H. Zhuang, X. Wang, M. Bendersky, and M. Najork. Rank4class: a ranking formulation for multiclass classification. arXiv preprint 21X:02211.009777, 2021.
12

---

[41] Y. Wang. C. Yang. S. Lan, W. Fei, L. Wang, G. Q. Huang, and L. Zhu. Towards industrial foundation models: Framework, key issues and potential applications. In 2024 27th International Conference on Computer Supported Cooperatiee Work Design (CSCWD), pages 1-6. IEEE, 2024.
[42] Y. Wang, C. Yang, S. Lan. L. Zhu, and Y. Zhang. End-edge-cloud collaborative computing for deep learning: A comprehensive survey, IEEE Communications Surveys Tutorials, 2024.
[43] R. Wightman. Pytorch image models. http:://wthaa.cco///gggggggg/eea 2019.
[44] R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint urXiv:2110.00470 2021.
[45] F. Xia, T.-Y. Liu, J. Wang. W. Zhang, and H. Li. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pages 1192- 2008.
[46] X. Xie, Zhou, H. Li, Lin, and S. Yan. Adan: Adaptive nesterov momentum algorithm for faster optimizing deep models. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024.
[47] C. Yang, Y. Wang. S. Lan. L. Wang. W. Shen, G. Q. Huang. Cloud-edge-device collaboration mechanisms of deep learning models for smart robots in mass personalization. Robotics and Computer- 2022.
Integrated Manufacturing,
[48] S. Yin, Z. Xiao, M. Song, and J. Long. Adversarial distillation based on slack matching and attribution re- gion alignment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 24605-2488
2024.
[49] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song. J. Demmel, K. Keutzer, and C.-J. Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv: 190Kv:9044.000 1904.0092,, 2019.
[50] M. Yuan, B. Lang, and F. Quan. Student- knowledge distillation. Knowledge- Systems. 196:11 2024.
[51] S. Zagoruyko and N. Komodakis. Paying more attention attention: Improving the performance of convolutional neural networks attention transfer. arXiv preprint arXiv:1612.03928. 2016.
[52] B. Zhao, Q Cui, R. Song, Y. Qiu, and J. Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953-1992,, 2022.
[53] Z.-H. Zhou. Learnware: the future of machine learning. Frontiers Comput. Sci., 10(4):589-3)) 5900 2016.
[54] Y. Zhu, N. Liu, Z. Xu, X. Liu, W. Meng. L. Wang, Z. Ou, and J. Tang. Teach less, learn more: On the undistillable classes knowledge distillation. Advances Neural Information Processing Systems, 35:3201 1-32024, 2022.
[55] Y. Zhu and Y. Wang. Student customized knowledge distillation: Bridging the gap between student and teacher. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5057-5066. 2021.
[56] Zhuang, T. Tang, Y. Ding, S. C. Tatikonda, N. Dvornek, X. Papademetris, and J. Duncan, Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances neural information processing systems, 11876, 2020.
13

---

Due to the linearity of A Gradient Derivation for PLD Loss
A.1 Definitions
Let ($1...., sc) E RC be the student's logits. Let = (मi..... =t) be a given permutation of class indices, referred as the teacher-optimal permutation. Let ag for k 1,... C' be scalar weights, constant with respect to s.
The PLD loss function is defined as:
((s)
O - + log
exp(xx;) t=
k=1
We can write [(s) Ex=1 La (8), where
Lx(s) akk STz + d
and
(x) = log exp($>;)) t=k
Our goal is to compute the gradient v,C(s), whose i-th component is ds," differentiation,
0C
ac
0L
0s;
0s:
k=1
#Ls We will first compute Ds,
A.2 Derivative of Ls (s):
We have La(s) -0485; + (x)x(s). Let's differentiate each term with respect sq.
1. Derivative of the affine term: The first term is -x
i (-aks:;) = -Okk Os;
0s
Us;
Since s5z the component of at index -i. its derivative with respect to 8; if m and otherwise. This can be written using the indicator function 1 = m
(-ausx;)
-a,ffi mt).
ds:
2. Derivative of the log-sum-exp term: The second term is asdx(s), where (s(s) log Σ exp(s;;).
8 (ak (s) ak Os:
Joh
Os;
To compute ddc let Xs(s) Ee exp(s=;;) Then cbe(s) = log (x(s). Using the chain rule,
0dg
(x)
Now, we compute @xa(s).
(x(*) 0s,
(x+(s)
Σ exp($t;)
0s;
gs;
t=k
(exp(sx;))
Osy
t=k
Os:;
Σ
(exp(xx;)
Oz:
Σ
exp(sm; }l{i = 7
Em k
14

---

The term 1 Tτ is non zero (equal to 1) only if x.. This occurs if the index i is part of the set of indices {TT. k+++. Tel. If i in this set, then exactly one in the sum (from & to C') will satisfy मæ and for that specific E, the term becomes exp(s:). If is not in this set, all terms zero. Thus,
(x)(s)
(exp(s:) if ( m².
0s;
0
otherwise.
So,
Bok becomes:
0s:
aok
(exp(s:)
if otherwise
8s;
Li=x exp(sai)
00
Let us define (i) as:
exp(si)
if E (T m
(1) =
Dezk exp(87;)
otherwise.
Doh Then, Us:
= (i))). And the derivative of the second term of La(s) is:
(ax (s)) (1).
0s;
3. Combining derivatives for LA (s).
Now, we combine the derivatives of the two parts of (s):
OL. θ
(-asss;) + (a)))(s))
0s: 0s;
3s;
-a T + ou (i)
= ar (ox(i) 1 = m
A.3 Final Gradient V.C(s):
The i-th component of the gradient of the total loss [(s) is:
OC
aL
Σ
Σ ak (ox 1fi xf])
as;
We can separate this sum into two parts:
ac
(1) Σ ag .,}-
0s:
1=1
For the first part of this expression, E (i), the term (i) is non-zero and only if E exp(a;) ( ) Therefore, this sum can be written Ex Ex: a Σ exp(s.;)) For the
second part, De agi = = the indicator function 1fi = m is equal to and only if ग = i. Since nr" is permutation, for any given i, there is precisely value of k for which this condition met. Consequently, this sum simplifies to =g= agg, which effectively selects the single as corresponding to the position of i in the permutation T².
Combining these, the i-th component of the gradient is:
ac
(xp(s:)
Σ
Σ
OE
ag.
Os:
Pe=k exp( (8x))
k: (SL
A:
x
In vector form, let ex; denote the standard basis vector that is 1 at index Tे and 0 elsewhere. Let be a vector whose j-th component is (i). Then the gradient of Lx(s) is (sLe(s) = as(ox Exg)) The total gradient is:
C
v.C(s)
Σ
ank cmg)
k=1
15

---

A.4
Convexity and Smoothness:
Each term (x(s) = ai -sx; + log Ei exp(sa;) a sum of an affine function (-on and
non-negatively weighted log-sum-exp function (a (s). Affine functions
convex. The log-sum- exp function (s) is known to be convex. Assuming dę 2 0 (which is true if they are derived from softmax probabilities, as om;) the term 0x (s) also convex. The sum of convex functions is
convex, so each La (s) convex. The total loss (() D (x(s) is sum convex functions, and therefore, ((s) is convex in
Furthermore, affine functions are smooth (infinitely differentiable). The log- rog-sum-eey, function is also smooth. Since differentiaaion, non-negative weighting, and summation preserve smoothness (specifically, Lipschitz continuity the gradient on any bounded domain), the full PLD loss ((s) has Lipschitz-continuuus gradient on any bounded domain, which is beneficial for gradient-based optimization methods.
B Full Baseline Ablation
Table 8: Hyperparameter grid search on ImageNet-1K. Each configuration uses a ResNet-50 student distilled from ResNet-152 teacher (Top-1 Acc. 79.61%). "_" indicates not applicable. The best setting in each group is highlighted.
Hyperparameters
Accuracy (%)
Method
B
7
Top-1
Top-5
Teacher
79.61
CE
71.35
LS (e=0.1)
73.92
KD hyperpurumeter sweep
(r=2)
0.00
2.00
75.92
92.82
KD
0.10
2.00
76.80
93.16
0.20
2.00
76.65
93.11
0.30
2.00
76.30
93.02
0.40
2.00
76.33
93.10
0.50
2.00
76.12
93.12
0.60
2.00
75.94
92,79
0.70
2.00
75.85
92.71
0.80
2.00
75.21
92,39
0.90
2.00
74.28
91.59
DIST hyperparameter sweep
(T=1)
1.00 2.00 2.00 1.00
75.77
92.67
0.00 0.50 0.50 1.00
76.47
93.19
DIST
0.10 0.45 0.45
1.00
76.60
93.30
0.20 0.40
0.40
1.00
75.69
92.63
0.30 0.35
0.35
1.00
74,88
92.18
0.40 0.30
0.30
1.00
73.90
91.35
0.50 0.25 0.25
1.00
72.94
90.95
0.60 0.20
0.20
1.00
72.71
90.58
0.70 0.15
0.15
1.00
71.85
90.16
0.80 0.10
0.10
1.00
71.89
90.16
0.90 0.05
0.05
1.00
71.40
89.88
16

---

C Implementation of the PLD Loss
import torch
2 import torch. fecch.n..uunctionn functional as F
4 def
(flat_logitat torch. Tensor, flat Flat_labels: torch LongTensor
) -> torch. torch.Longhin LongTensor:
For each example, sort logits ascending, remove the true label, and oppend it the end so occupies the last (top-1) position.
10
sorted torch. sort (flat_looogs, dim=-1, Fescending-False) mask = sorted_idx 1= flat_labees (-squeeze(1)) flat_logits. size(-1)
[N, V] # [M, V]
13
assert torch. all (mask (ddmm==11) = 1),
15
"Must remove exactly true label per " sorted_excl corted_idx view(-1, V 1) return torch. cat [sorted_excl, flat_ labels unsqueeze(-1)1, dim=-1) # [M, v]
17
# [M, V-1]
19
def plackett_luce (studenttt__ torch. Tensor, teacher_logitas torch. Tensor, labels: torch. LongTensor, temperature: float 1.0 torch. Tensor::
26
Computes the PLD loss:
sum_k alpha_k &o logsumezp_ s_k alpha_k softman teacherrtteccerpppttt (ptt_.
28
flat_s, flat_t, flat_lbl prepare_fort for_ classification
30
student_logita. labels, teacher_logits
32
ranking create_ad sheate_ddjusted_eee maa.usttddrrnnin((faat.... flat_1bl)
# [N V]
s_perm torch, gethergtterrre (flat_s, dim=-1, index-ranking) t_perm Norch.gathen (flat_t, dim--1, indexnga
V]
EN, V]
lumsu torch. lorchhlogcumeuuexp..c. (s_perm, dim=-1)
[N, V]
per per_pos_lose = log_ cumsuu s_perm
# [N, VJ
softmax teacher_prod s.softmaxtt_perm (t_perm / temperature, dim=-1)
# [N, V]
weighted = per_pos_lose teacher_prol
# [N, V]
return weighted sum (dim=-1) mean
scalar
D Additional Experimental Setup
For these supplementary experiments, we adopt homogeneous teacher-student configuration drawn from the MobileNetV4 (29] family. Specifically, the student network is MobileNetV4 Convolutional Small, and the teacher network is MobileNetV4 HeboeeettHHybridd Large. To ensure that any observed differencees exactly the same training hyperparameters as
arise solely from the choice of loss function, in our main experiments, including learning-rate schedules, batch size, weight decay, and the data- augmentation pipeline.
To comprehensively evaluate the relative merits of Plackett-Luce Distillation (PLD) compared to DIST and KD, we conduct experiments across multiple optimizers, various divergence weightings, runtime and several logit-standardization schemes. We further quantify distributional alignment by measuring the KI. divergence between student and teacher softmax outputs throughout training. Finally, to gain geometric insight into each loss, we visualize the loss landscapes of PL.D, KD, and DIST.
17

---

E Optimizer Ablation Study
In main experiments, adopt the Lamb 49
optimizer as preseribed by the Timm brary. To evaluate the robustness of the Placket-LLuce Distillation (PLD) loss under different optimization schemes, we perform an ablation study in which we replace Lamb with three alter- natives: AdamW W [2I]. Adan [46], and AdaBelief [56]. Table reports the Top-1 accuracy of the MobileNetV4 Small student on ImageNet- IK for each optimizer, under the KD, DIST, and PLD losses. PLD consistently outperforms both KD and DIST, achieving an average Top- gain of 0.825% over DIST and 2.21 % over KD, with maximum improvements of 0.9 % and 2.70%, respectively.
43 li-
Table 9: Top-1 accuracy (%) of the MobileNerV44Smaall student under different optimizers, comparing DIST, KD, and PLD. ADIST PLD DIST, AKD PLD
KD.
Optimizer
DIST
PLD
ADIST
ÁKD
AdaBelief
69.91
68.92
70.80
0.89
1.88
A
69.94
68.89
70.85
0.91
1.96
Adan
70.07
68.52
70.82
0.75
2,30
Lamb (base)
70.41
68.46
71.16
0.75
2.70
F Logit Standardization Analysis
Logit standardization-ccnterr or normalizing the teacher and student logits before applying the distillation loss-has been proposed [35] to improve optimization stability and distribution alignment. We evaluate four variants the MobileNetV4-Smalll student (distilled from MobileNetV4-Hybric. Large) using Lamb optimizer. Specifically, we consider:
DIST std. logits: standardize both teacher and student logits before computing the DIST loss; KD + std. logits: standardize both teacher and student logits before the KL--based KD loss; PLD std. logits: standardize both teacher and student logits before the PLD loss; PLD + std. teacher logits: standardize only the teacher logits for the PLD softmax weighting.
Table 10 reports Top-1 accuracy ImageNet- IK for each standardization variant, using the un- standardized PLD baseline of 71.16%. While standardizing logits yields modest gains for both DIST and KD, it does not benefit PLD: applying standardization to either both teacher and student logits or to the teacher logits alone leads to a slight degradation PLD's performance.
Table 10: Effect of logit standardization on Top-1 accuracy (%). PLD baseline (no standardization):
71.16%. A 71.16 Aec.
Method
Top-1 Acc. (%)
A
DIST + std. logits
71.12
0.04
KD + std. logits
69.14
2.02
PLD std. logits
70.66
0.50
PLD std. teacher logits
70.81
0.35
G Classification Accuracy Across Divergences
In the main text we adopt the standard knowledge-distillttin loss based on the forward Kullback- Leibler divergence:
LKD LCE(Es. y) + (1 (1-a) DKL (softmax(z:/T)) softmax(z,/T)).
with 0.1 and T In addition this forward KL term, we evaluate two alternatives: the reverse Kullback-Leible divergence DKL. (softmax(z./11) softmax(z:/1)) and the Jensen- Shannon divergence Das Dus(soft msx softmax(zz/ (x./))x Table [] reports Top-1 accuracy (%) using each divergence measure. SeasennShhanae yields modest gain over forward KI,, while reverse KL performs comparably.
18

---

Table 11: Top-1 accuracy (%) of vanilla KD under different divergence measures. The PLD baseline (fixed across rows) is 71.16% %. AKD = PLD KD.
Divergence
Top-1
AKD
Forward KL
68.46
2.70
Reverse KL
67.4
3.72
Censen-Shanme
69.83
1.33
H
Distribution Matching Analysis
We evaluate how well each distillation loss aligns the student's output distribution with the teacher's by measuring the KL divergence between their softmax outputs at the end of training. Taboc[[]reporis these KL values for both homogeneous teacher-student pairs (same model family) and heterogencous pairs (cross-archh crosssarccitecture). Lower KL indicates tighter alignment. Although the PLD loss trains the student respect the teacher's preference ordering, in the homogeneous setups PLD achieves better alignment than DIST. However, since the KD loss explicitly minimizes a KL term, it yields the lowest divergence overall, outperforming both DIST and PLD.
Table 12: KL divergence of student vs. teacher softmax outputs under DIST, KD, and PLD. Lower is better.
Teacher
Student
DIST
KD
PLD
Homogeneous
MobileNer\44-Large
MohileNcrV44Meddum
0.67
0.55
0.60
MobileNeeVV+-Lage
MobileNerV44-mme
0.95
0.85
0.83
MobaeeNetV4+-arge
MobileNatV4--yyrri--edddd
0.64
0.52
0.60
ViT-Large/
ViT-Base/11
0.47
0.42
0.72
VÝT-Large/16
ViT-Small/16
0.55
0.53
0.69
Heterogeneous
MobileNetV4-Large
ResNer50
0.71
0.60
0.65
NobileNarV---lyyrri-Meeem
ResNet50
1.27
0.27
1.50
ViT-Base/16
ResNe150
0.43
0.39
0.66
VIT- ViT-Large/16
ResNet50
0.46
0.44
0.59
Runtime Analysis
We evaluate the computational efficiency PLD compared with other distillation methods discussed in Section 5 PLD involves single sorting operation to obtain the teacher-optimal permutation, similar to ListMEEE [45]; therefore, its complexity is only O log C.) far lower than the OCC!) enumeration of all possible rankings. In practice, this adds negligible overhead: PLD trains nearly the same speed as logit-based methods such as KD, DIST, and DKD. Table 13 reports the total training time (in minutes) for all CIFAR-100 experiments Table [
Despite its list-wise formulation, PLD's 's runtime remains comparable to KD and DIST, and substantially faster than other methods such RKD VID.
J Loss Landscape Visualization
To gain geometric insight into the optimization landscapes induced by our three distillation losses (DIST, KD, PLD), plot 3D surfaces and 2D contours over two-dimensional slice of the student- logit space.
Setup: Let R random teacher-logit vector normalized to unit norm, and let d,, d2 E RVV be two random orthonormal directions. For a grid of (a, (8) E [-5|[4], 5]]41]] we define student logits s(a, 8) = + a dj + 3 de and compute each loss L(s(a, :3), () every grid point. Figure plots the loss landscapes for DIST [3] KD 12]. and PLD (ours), While KD and PLD both induce
19

---

Table 13: Total training time (minutes) for all CIFAR-100 experiments. PLD achieves comparable efficiency to KD, DIST, and DKD despite its list-wise formulation.
Model (Teacher
CE
PLD
KD
DIST
DKD
AT
FitNet
PKT
RKD
SP
VID
Student)
ResNer32x4
32.06
62.84
62.18
62.24
62.46
65.39
63.96
62.53
68.36
62.73
78.40
ResNet8x4
ResNet50
MobileNetV2
23.62
102.59
102.23
102.49
102.54
108.17
113.18
102.28
128.65
102.78
132.06
ResNet56
22.97
33.42
33.57
33.81
33.46
34.55
33.30
33.89
36.21
33.67
36.57
ResNer20
WideResNet-40-2
27.67
33.10
37.56
36.73
34.27
36.77
38.05
36.84
37.62
37.01
37.35
WideResNNet-4-11
FlD Cormur11-2
Surten (T-31
2202
ans bang
(a) 2.0
(b)
SM5
An12
201
200
400
(c) T 0.5
(d) 7 0.1
Figure 3: PLD loss surfaces at different teacher temperatures. (Top row) = 2.0 and = 1.0; (Bottom row) T 0.5 and T 0.1. Lowering below 1.0 flattens convexity.
convex surfaces, DIST dips sharply yet remains effectively planar in the (a, 3) slice. Moreover, the contour for PLD more tightly centered around the origin than that of KD.
J.1 Temperature Sensitivity of the PLD Loss Surface
To investigate the effect of the teacher-softmax temperature T' on the geometry of the PLD loss landscape, fix the same two-dimensional (a, () slice and compute the PLD surface four representative temperatures: T' E (2.0, 1.0, 0.5, 0.1}. Figure shows both the 3D surface and 2D contour plots for each T. We observe that reducing T from 2.0 down to 1.0 produces only minor changes, whereas further lowering T below 1.0 flattens the curvature.
20

---

DIST Surface
DIST Contour
1.496
1.456
1.4
1.416
1.376
1.3
1.336
1.256
-4-
1.216
-4
1.176
-2
(a) DIST loss
Surface
KD Contour
0,714
0.709
0.71
0.70
0.699
0.69
0.684
4-022
2
B
(b) KD loss
Surface
5.938
5.935
5.935
2
5.932
5.930
5.929
5. 5.925
5.926
5. .20
5.923
5.920
5..117
A2024
5.914
B
5.911
(c) PLD loss
Figure 4: Loss landscapes of three distillation methods: (a) DIST exhibits a sharp dip yet remains effectively planar; (b) KD shows moderate convexity; (c) PLD (ours) exhibits better convexity with contours mostly centered at the origin.
21

---

NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Question:
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report mean Top-1 accuracies on CIFAR- along with their correspond- ing standard errors computed over three independent runs with different random seeds.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
22

---

Justification: This work is foundational on model compression and does not present direct application or risk. Indirectly, more efficient models may reduce energy and hardware costs, but we do not see specific societal harms arising from the method itself.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No high-risk or dual-use data/models are released in this work
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do release any new datasets or large models with this submission.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human-subject data or crowdsourcing is involved.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is required.
Answer: [NA]
Justification: No large language models are used in our core methodology.
23


Reviews:
[{"review_id": "g7v14X5fxI", "reviewer": "Reviewer_fk7Y", "summary": "This paper proposes Plackett-Luce Distillation (PLD), a novel list-wise, choice-theoretic knowledge distillation loss that leverages the Plackett-Luce permutation model to enforce a teacher-optimal ranking of classes. By interpreting teacher logits as worth scores and weighting each selection step by the teacher's confidence, PLD provides a convex, translation-invariant surrogate that unifies several losses. Experiments on ImageNet-1K demonstrate consistent accuracy improvements over standard KL-based distillation (KD) and correlation-based distillation (DIST) across various student-teacher architectures.", "questions": "See Weaknesses.", "limitations": "1.The proposed method may require a lot of additional training time. \n2.The experiments are limited to image classification. Can the proposed method works on other tasks?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.The paper introduces a well-founded framework based on choice theory and the Plackett-Luce model.\n2.The method is convex and differentiable, with closed-form gradients, making it efficient and amenable to gradient-based optimization.\n3.The proposed method eliminates the need for manual tuning of weights between the cross-entropy and distillation terms.\n\nWeaknesses\n1.Converting the teacher's outputs into a ranked list and then using the Plackett-Luce model to weight labels based on logits seems to me like a simple remapping of the original soft labels. I don't think this approach extracts significantly more useful information from the soft labels.\n2.The comparison methods in this paper are too limited. Please refer to other knowledge distillation (KD) papers and include more baseline algorithms for comparison.\n3.This paper introduces the Plackett-Luce model and requires computation of closed-form gradients, which incurs significant additional training cost. Please provide a comparison of training time to quantify this overhead.\n4.Potential applications beyond image classification (e.g., NLP, Image Segmentation) are not explored experimentally.\n5.This paper is difficult to understand.", "was_revised": true, "final_justification": "Thank you for your constructive and insightful comments on my manuscript. I have carefully considered your feedback, and I am pleased to inform you that your responses have addressed the majority of my concerns. Your suggestions have significantly clarified several aspects of the paper, and I now feel more confident in its contribution to the field.\n\nAs a result, I have revised my assessment and have decided to increase the score accordingly.", "rebuttal": "* We sincerely thank Reviewers fk7Y, Bcg7, ySdm, and x7Lh, as well as the Area and Senior Area Chairs, for their thorough and constructive feedback on PLD.\n* We are pleased that the reviewers recognize our method’s strong theoretical foundation (fk7Y, Bcg7), its efficiency and practical significance (fk7Y), the clarity of our derivations and writing (Bcg7, ySdm), and the robustness of our empirical evaluation (Bcg7, ySdm).\n* We greatly appreciate the suggestions to broaden our baseline comparisons, quantify runtime overhead, extend experiments to other tasks, and polish the manuscript’s readability. We will incorporate all additional experiments (CIFAR-100 baselines, COCO detection and runtime comparisons) into the revised manuscript.\n* We also thank the reviewers for highlighting minor typos and stylistic improvements, which we have addressed in the updated draft.\n\nThank you Reviewer fk7Y for the detailed and constructive feedback. We’re glad the reviewers (including fk7Y and others) recognize PLD’s choice-theoretic foundation, its convex differentiable surrogate with closed-form gradients, and its elimination of manual tuning between cross-entropy and distillation. Below we (1) show how PLD transfers richer intra-class information beyond soft-label remapping, (2) present expanded CIFAR-100 baselines under identical hyperparameters, (3) provide empirical runtime analyses, (4) demonstrate extensions to object detection, and (5) describe improvements to clarity and consistency in the manuscript.\nWe hope these additions and clarifications address your concerns, and we would be grateful for your reconsideration of the score.\n\n**Q1. Is converting the teacher’s outputs into a ranked list and then using the Plackett–Luce model to weight labels based on logits merely a remapping of the original soft labels, or does it extract substantially more useful information?**\n\nAnswer:  PLD extracts richer intra-class information than standard KD by supervising the full teacher permutation with confidence-weighted list-wise ranking, instead of just remapping soft labels.\n\nIn knowledge distillation we add extra constraints to the loss to extract richer information from the teacher. Cross-entropy alone only pushes the student toward the top class, whereas distillation terms transfer subtler inter-class relations. PLD uses *weighted list-wise ranking*: given a teacher ranking $a > b > c > d > e$, supervising this single permutation not only enforces $a$ as the top choice but also embeds every relative preference (e.g., $a>b>c$, $b>c>d$, etc.). Furthermore, each selection step is weighted by the teacher’s softmax confidence (so $\\alpha_1 = q^T_a \\gg \\alpha_5 = q^T_e$), meaning PLD emphasizes high-confidence distinctions like $a>b$ over low-confidence ones like $d>e$, all within one forward pass.\n\nBy contrast, standard KD (CE + KL) depends on softmax normalization: when the teacher is highly confident, the target is near one-hot and carries little structure, so the KL term must be softened via temperature (Hinton et al., 2015) just to expose any ordering. PLD instead directly minimizes the negative log-likelihood of the *entire* teacher-optimal permutation $\\pi^* = (y, \\text{argsort}(t)\\setminus\\{y\\})$, aligning all ordering relations in a single convex loss (Sec. 4.1, L183–L184), and uses confidence weights $\\alpha_k = q^T_{\\pi^*_k}$ to prioritize the strongest signals (Sec. 4.2). This yields substantially richer intra-class information than mere soft-label remapping.\n\n**Q2: Inclusion of additional KD baselines.** \n\nAnswer: We included DIST (NeurIPS 2022) as a baseline and structured our ImageNet-1K experiments to compare models horizontally across architectures and scales; to further strengthen the empirical comparison we added CIFAR-100 results against other recent KD methods, all trained under the same controlled hyperparameters: AdamW (β₁=0.9, β₂=0.999), learning rate $=0.001$, weight decay $=0.5$, batch size $=128$, for 250 epochs with a cosine-annealing schedule and a 10-epoch linear warm-up. Results are reported as mean $\\pm$ std over three runs. Following prior work (e.g., KD and DIST) we set the temperature $\\tau=4$ where applicable. \n\nTable A: CIFAR‑100 performance under identical hyperparameters\n\n| Model (Teacher→Student) | CE | PLD | KD | DIST | DKD | AT | FitNet | PKT | RKD | SP | VID |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet32x4ResNet8x4 | 71.56 ± 0.16 | **73.97 ± 0.05** | 72.11 ± 0.10 | 73.21 ± 0.10 | 73.08 ± 0.27 | 72.51 ± 0.37 | 72.59 ± 0.30 | 72.86 ± 0.17 | 71.45 ± 0.41 | 72.45 ± 0.08 | 71.70 ± 0.32 |\n| ResNet50 MobileNetV2 | 64.42 ± 0.67 | 68.16 ± 0.38 | 66.61 ± 1.16 | 66.98 ± 0.74 | **69.51 ± 0.42** | 52.28 ± 0.84 | 62.77 ± 0.05 | 66.08 ± 0.11 | 65.16 ± 0.21 | 65.30 ± 0.29 | 64.14 ± 0.82 |\n| resnet56resnet20 | 67.57 ± 0.25 | **70.28 ± 0.21** | 69.45 ± 0.24 | 69.66 ± 0.49 | 67.80 ± 0.03 | 68.42 ± 0.16 | 66.75 ± 0.27 | 68.66 ± 0.20 | 68.53 ± 0.18 | 69.36 ± 0.27 | 68.52 ± 0.12 |\n| WideResNet‑40‑2WideResNet‑40‑1 | 70.26 ± 0.24 | **73.50 ± 0.3** | 72.44 ± 0.36 | 72.74 ± 0.33 | 73.22 ± 0.38 | 71.99 ± 0.34 | 70.86 ± 0.33 | 71.43 ± 0.16 | 11.32 ± 1.84 | 73.49 ± 0.20 | 70.61 ± 0.39 |\n\n\n\n**Q3. Does introducing the Plackett–Luce model with its closed-form gradients incur significant additional training cost?**\n\nAnswer: PLD’s runtime is on par with state-of-the-art KD methods while yielding the high efficiency. PLD enforces only a single teacher‑optimal permutation (O(C log C) via sorting) rather than summing over all C! rankings—just as ListMLE  and P‑ListMLE do—making it computationally feasible and directly comparable to other state‑of‑the‑art distillation methods. Below is the run time of Table A’s methods: \n\n**Table B: Total training time (minutes)**\n\n| Runtime | CE | PLD | KD | DIST | DKD | AT | FitNet | PKT | RKD | SP | VID |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet32x4ResNet8x4 | 32.06 | 62.84 | 62.18 | 62.24 | 62.46 | 65.39 | 63.96 | 62.53 | 68.36 | 62.73 | 78.40 |\n| ResNet50 MobileNetV2 | 23.62 | 102.59 | 102.23 | 102.49 | 102.54 | 108.17 | 113.18 | 102.28 | 128.65 | 102.78 | 132.06 |\n| resnet56resnet20 | 22.97 | 33.42 | 33.57 | 33.81 | 33.46 | 34.55 | 33.30 | 33.89 | 36.21 | 33.67 | 36.57 |\n| WideResNet‑40‑2WideResNet‑40‑1 | 27.67 | 33.1 | 37.56 | 36.73 | 34.27 | 36.77 | 38.05 | 36.84 | 37.62 | 37.01 | 37.35 |\nDespite its list‑wise nature, PLD’s runtime is on par with KD, DIST, and DKD, and substantially faster than RKD or VID. Furthermore, we found PLD delivers the high accuracy gain over time\n\n**Table C: Accuracy gain over time (ΔTop‑1 / runtime)**\n\n| gain | CE | PLD | KD | DIST | DKD | AT | FitNet | PKT | RKD | SP | VID |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet32x4ResNet8x4 | 0 | **0.038** | 0.009 | 0.027 | 0.024 | 0.015 | 0.016 | 0.021 | -0.002 | 0.014 | 0.002 |\n| ResNet50 MobileNetV2 | 0 | 0.036 | 0.021 | 0.025 | **0.05** | -0.112 | -0.015 | 0.016 | 0.006 | 0.009 | -0.002 |\n| resnet56resnet20 | 0 | **0.081** | 0.056 | 0.062 | 0.007 | 0.025 | -0.025 | 0.032 | 0.027 | 0.053 | 0.026 |\n| WideResNet‑40‑2WideResNet‑40‑1 | 0 | **0.0979** | 0.0581 | 0.0676 | 0.0864 | 0.0471 | 0.0158 | 0.0318 | −1.5670 | 0.0873 | 0.0094 |\n\n**Q4. Why are potential applications beyond image classification (e.g., NLP, image segmentation) not explored experimentally?**\n\nAnswer: As discussed in our Conclusion (Sec 6), PLD naturally extends to any task where student and teacher share the same output logits, including detection or segmentation. To validate this, we distilled Faster R‑CNN detectors on COCO for two teacher→student pairs using DKD  and PLD:\n\n**Table D: Object detection performance (COCO)**\n\n| Teacher → Student | Method | AP | AP₅₀ | AP₇₅ | AP_s | AP_m | AP_l |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet‑50 → MobileNetV2‑FPN | DKD | **29.24** | **50.42** | 29.87 | **16.15** | **31.11** | 38.07 |\n| ResNet‑50 → MobileNetV2‑FPN | PLD | 28.91 | 49.63 | **29.88** | 16.05 | 30.41 | **38.59** |\n| ResNet‑101 → ResNet‑18‑FPN | DKD | 32.11 | 53.43 | 33.9 | 18.46 | 34.43 | 41.5 |\n| ResNet‑101 → ResNet‑18‑FPN | PLD | **32.47** | **53.83** | **34.17** | **18.5** | **34.97** | **42.12** |\n| ResNet‑101 → ResNet‑50‑FPN | DKD | 36.54 | **58.57** | 39.46 | **21.74** | **39.8** | **47.31** |\n| ResNet‑101 → ResNet‑50‑FPN | PLD | **36.6** | 58.28 | **39.58** | 21.37 | 39.76 | 47.24 |\n\nGiven PLD’s competitive performance, we hope other researchers will adopt it for additional domains such as NLP, as discussed in Section 6 (L303).\n\n**Q5. This paper is difficult to understand.**\n\nAnswer: While we appreciate this feedback, several other reviewers found the paper clear and accessible. For example, Reviewer Bcg7 writes, “The paper is well written and clear. The method is well-motivated and the derivations are fairly easy to follow.” and Reviewer ySdm notes, “The organization of this paper is very clear, so readers can understand the paper easily.” Nonetheless, we’ve further smoothed the narrative flow by breaking down long, dash‑connected sentences into shorter, standalone sentences and adding brief transitional phrases, making the text easier to follow. We further appreciate your detailed suggestions for improving the writing and will incorporate them in the revised manuscript.\n\n**Baselines & References**\n\n1. **KD**:     arXiv:1503.02531\n2. **DIST**:  arXiv:2205.10536\n3. **DKD**:   arXiv:2203.08679\n4. **AT**:      arXiv:1612.03928\n5. **FitNet**:arXiv:1412.6550\n6. **PKT**:   arXiv:1803.10837\n7. **RKD**:  arXiv:1904.05068\n8. **SP**:     arXiv:1907.09682\n9. **VID**:    arXiv:1904.05835"}, {"review_id": "7dE1JLFzTj", "reviewer": "Reviewer_Bcg7", "summary": "This paper presents a novel method for knowledge distillation (KD).  The main idea is to use a new loss function that encourages the student network's rankings for each class to match the teacher's rankings for each class and also weights each class according to the teacher's softmax class probabilities.  This loss function is based on the Plackett-Luce permutation model.  The paper shows how this new loss subsumes cross-entropy loss as well as some list-based losses.  Experimental results showing the improvement over standard KD and DIST (a recent KD method) on the ImageNet-1k dataset with various teacher-student network combinations.", "questions": "Are repeated experiments to compute error bars really too computationally expensive?\n\nWhy not train/test on CIFAR-100 with standard teacher-student network pairings to allow comparisons across a broad set of prior KD methods?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and clear.  The method is well-motivated and the derivations are fairly easy to follow.\n\nThe paper is theoretically sound.  It shows a clear derivation of the new loss function and explains its connection to the Plackett-Luce permutation model.  The paper also shows how the Plackett-Luce Distillation (PLD) Loss subsumes cross-entropy loss and P-ListMLE loss.\n\nThe experiments on the ImageNet-1K dataset show improvement of PLD over KD and DIST using different teacher-student combinations (including both homogenous and heterogeneous teacher-student pairs).\n\nWeaknesses:\n\nUnfortunately, the experiments do not include error bars from multiple runs as has been common in many previous papers on KD.  While it is true that it would be computationally expensive to repeat all of the experiments multiple times on ImageNet-1K, it is not out of the question to repeat them say 3 times each.\n\nThe method does not compare against many previous KD methods (only KD, DIST, ListMLE and p-ListMLE).  Many past papers on KD used the CIFAR-100 dataset for training and testing with some standard teacher-student network pairings (for example, see Yonglong Tian, Dilip Krishnan, and Phillip Isola, \"Contrastive representation distillation\", in ICLR, 2020).  Including experiments on CIFAR-100 using the same teacher-student pairings would allow comparisons to a large number of previous KD methods.", "was_revised": true, "final_justification": "After reading the other reviews and the authors' rebuttal,  I think the paper should be accepted to NeurIPS.  The authors addressed my concerns about the lack of uncertainty estimates and lack of comparison to prior work (via results on CIFAR-100) and should include these new results in the final version of the paper.", "rebuttal": "* We sincerely thank Reviewers fk7Y, Bcg7, ySdm, and x7Lh, as well as the Area and Senior Area Chairs, for their thorough and constructive feedback on PLD.\n* We are pleased that the reviewers recognize our method’s strong theoretical foundation (fk7Y, Bcg7), its efficiency and practical significance (fk7Y), the clarity of our derivations and writing (Bcg7, ySdm), and the robustness of our empirical evaluation (Bcg7, ySdm).\n* We greatly appreciate the suggestions to broaden our baseline comparisons, quantify runtime overhead, extend experiments to other tasks, and polish the manuscript’s readability. We will incorporate all additional experiments (CIFAR-100 baselines, COCO detection and runtime comparisons) into the revised manuscript.\n* We also thank the reviewers for highlighting minor typos and stylistic improvements, which we have addressed in the updated draft.\n\nThank you Reviewer Bcg7 for the detailed and constructive feedback. We are glad you found the paper well written, clear, theoretically sound, and that the improvements of PLD over KD and DIST across teacher-student pairs are convincing. We also appreciate your pointer to relevant prior work such as Contrastive Representation Distillation (Tian et al., ICLR 2020), which helped guide our expansion of baselines and the design of additional CIFAR-100 experiments. Without these suggestions it would have been very difficult to coordinate and execute so many additional experiments in the limited time available. Below we (1) expand the empirical comparison with additional CIFAR-100 baselines, including uncertainty estimates from three runs to provide error bars (2) broaden the set of KD methods evaluated to strengthen the comparison and (3) demonstrate extensions to object detection. We hope these additions and clarifications address your concerns, and we would be grateful for your reconsideration of the score.\n\n**Inclusion of additional KD baselines and tasks: CIFAR‑100 Baselines and Object Detection**\n\n\nWe included DIST (NeurIPS 2022) as a baseline and structured our ImageNet-1K experiments to compare models horizontally across architectures and scales; to further strengthen the empirical comparison we added CIFAR-100 results against other recent KD methods, all trained under the same controlled hyperparameters: AdamW (β₁=0.9, β₂=0.999), learning rate $=0.001$, weight decay $=0.5$, batch size $=128$, for 250 epochs with a cosine-annealing schedule and a 10-epoch linear warm-up. Results are reported as mean $\\pm$ std over three runs. Following prior work (e.g., KD and DIST) we set the temperature $\\tau=4$ where applicable. \n\nTable A: CIFAR‑100 performance under identical hyperparameters\n\n| Model (Teacher→Student) | CE | PLD | KD | DIST | DKD | AT | FitNet | PKT | RKD | SP | VID |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet32x4ResNet8x4 | 71.56 ± 0.16 | **73.97 ± 0.05** | 72.11 ± 0.10 | 73.21 ± 0.10 | 73.08 ± 0.27 | 72.51 ± 0.37 | 72.59 ± 0.30 | 72.86 ± 0.17 | 71.45 ± 0.41 | 72.45 ± 0.08 | 71.70 ± 0.32 |\n| ResNet50 MobileNetV2 | 64.42 ± 0.67 | 68.16 ± 0.38 | 66.61 ± 1.16 | 66.98 ± 0.74 | **69.51 ± 0.42** | 52.28 ± 0.84 | 62.77 ± 0.05 | 66.08 ± 0.11 | 65.16 ± 0.21 | 65.30 ± 0.29 | 64.14 ± 0.82 |\n| resnet56resnet20 | 67.57 ± 0.25 | **70.28 ± 0.21** | 69.45 ± 0.24 | 69.66 ± 0.49 | 67.80 ± 0.03 | 68.42 ± 0.16 | 66.75 ± 0.27 | 68.66 ± 0.20 | 68.53 ± 0.18 | 69.36 ± 0.27 | 68.52 ± 0.12 |\n| WideResNet‑40‑2WideResNet‑40‑1 | 70.26 ± 0.24 | **73.50 ± 0.3** | 72.44 ± 0.36 | 72.74 ± 0.33 | 73.22 ± 0.38 | 71.99 ± 0.34 | 70.86 ± 0.33 | 71.43 ± 0.16 | 11.32 ± 1.84 | 73.49 ± 0.20 | 70.61 ± 0.39 |\n\nAs discussed in our Conclusion (Sec 6), PLD naturally extends to any task where student and teacher share the same output logits, including detection or segmentation. To validate this, we distilled Faster R‑CNN detectors on COCO for two teacher→student pairs using DKD  and PLD:\n\n**Table B: Object detection performance (COCO)**\n\n| Teacher → Student | Method | AP | AP₅₀ | AP₇₅ | AP_s | AP_m | AP_l |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet‑50 → MobileNetV2‑FPN | DKD | **29.24** | **50.42** | 29.87 | **16.15** | **31.11** | 38.07 |\n| ResNet‑50 → MobileNetV2‑FPN | PLD | 28.91 | 49.63 | **29.88** | 16.05 | 30.41 | **38.59** |\n| ResNet‑101 → ResNet‑18‑FPN | DKD | 32.11 | 53.43 | 33.9 | 18.46 | 34.43 | 41.5 |\n| ResNet‑101 → ResNet‑18‑FPN | PLD | **32.47** | **53.83** | **34.17** | **18.5** | **34.97** | **42.12** |\n| ResNet‑101 → ResNet‑50‑FPN | DKD | 36.54 | **58.57** | 39.46 | **21.74** | **39.8** | **47.31** |\n| ResNet‑101 → ResNet‑50‑FPN | PLD | **36.6** | 58.28 | **39.58** | 21.37 | 39.76 | 47.24 |\n\n**Baselines & References**\n\n1. **KD**: Hinton, G., Vinyals, O., & Dean, J. (2015). *Distilling the Knowledge in a Neural Network*. arXiv:1503.02531\n2. **DIST**: Huang, T., You, S., Wang, F., Qian, C., & Xu, C. (2022). *Knowledge Distillation from A Stronger Teacher (DIST)*. arXiv:2205.10536\n3. **DKD**: Chen, Z., Liu, W., & Tao, D. (2022). *Decoupled Knowledge Distillation*. arXiv:2203.08679\n4. **AT**: Zagoruyko, S., & Komodakis, N. (2016). *Paying More Attention to Attention*. arXiv:1612.03928\n5. **FitNet**: Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). *FitNets: Hints for Thin Deep Nets*. arXiv:1412.6550\n6. **PKT**: Passalis, N., & Tefas, A. (2018). *Learning Deep Representations with Probabilistic Knowledge Transfer*. arXiv:1803.10837\n7. **RKD**: Park, W., Kim, D., Lu, Y., & Cho, M. (2019). *Relational Knowledge Distillation*. arXiv:1904.05068\n8. **SP**: Tung, F., & Mori, G. (2019). *Similarity‑Preserving Knowledge Distillation*. arXiv:1907.09682\n9. **VID**: Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., & Dai, Z. (2019). *Variational Information Distillation for Knowledge Transfer*. arXiv:1904.05835"}, {"review_id": "syYGB8HnKe", "reviewer": "Reviewer_ySdm", "summary": "This paper targets logit distillation, which is knowledge distillation with merely logit outputs. \nPrevious logit distillation methods all rely on cross-entropy loss. \nIn this paper, they introduce a convex, translation-invariant surrogate, Plackett-Luce Distillation (PLD) Loss. \nPLD views logits as “worth” scores in the classical Plackett-Luce permutation model.\nWith the designed mechanism, PLD can not only act as the cross-entropy loss but also capture the full rank structure provided by the teacher model. \nExperiment results on the classification task prove the effectiveness of the proposed method.", "questions": "See weakness 1, 3. I think adding these experimental results will make this paper more insightful.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. The organization of this paper is very clear, so readers can understand the paper easily. \n2. The idea of this paper is reasonable. I am happy to see that a new loss is introduced to logit distillation, since loss really plays a crucial role in logit distillation. \n3. The experiment results cover various network architectures, including ViT. The improvements of the proposed method is clear and convincing. \n4. The authors have discussions on the limitations of the proposed method. I think it is helpful to the research community. \n5. They also provide the pseudo code and detailed theoretical analysis of the proposed method. \n\nCons:\n1. I am curious about the comparison results with more methods, eg. DKD (CVPR2022) [1].\n\n[1] Decoupled Knowledge Distillation \n\n2. The authors can reorganize the experiment section, moving key comparison results forward, and put tables of baseline methods behind them. \n\n3. Now they only provide the experiment results in classification tasks. I think the current results are enough for a NeurIPS paper, but I am still curious about the method performance on other tasks, eg. object detection.", "was_revised": true, "final_justification": "After reading the rebuttal, I keep my score.", "rebuttal": "* We sincerely thank Reviewers fk7Y, Bcg7, ySdm, and x7Lh, as well as the Area and Senior Area Chairs, for their thorough and constructive feedback on PLD.\n* We are pleased that the reviewers recognize our method’s strong theoretical foundation (fk7Y, Bcg7), its efficiency and practical significance (fk7Y), the clarity of our derivations and writing (Bcg7, ySdm), and the robustness of our empirical evaluation (Bcg7, ySdm).\n* We greatly appreciate the suggestions to broaden our baseline comparisons, quantify runtime overhead, extend experiments to other tasks, and polish the manuscript’s readability. We will incorporate all additional experiments (CIFAR-100 baselines, COCO detection and runtime comparisons) into the revised manuscript.\n* We also thank the reviewers for highlighting minor typos and stylistic improvements, which we have addressed in the updated draft.\n\nThank you Reviewer ySdm for the thoughtful and encouraging feedback. We’re pleased you found the paper’s organization clear, the motivation sound, and the empirical gains convincing across architectures including ViT. Your pointer to DKD (CVPR 2022) helped us broaden our baseline set, and your suggestion to reorganize the experimental section has improved clarity and flow. Without these suggestions it would have been very difficult to coordinate and execute so many additional experiments in the limited time available. In the revised manuscript we will (1) include DKD among the CIFAR-100 baselines and present key results up front with supporting tables moved to follow, (2) extend PLD to object detection on COCO to demonstrate its applicability beyond classification, and (3) refine the experimental layout to highlight these comparisons. We hope these additions further strengthen your confidence in our work.\n\n\n**Inclusion of additional KD baselines and tasks: CIFAR‑100 Baselines and Object Detection**\n\nWe included DIST (NeurIPS 2022) as a baseline and structured our ImageNet-1K experiments to compare models horizontally across architectures and scales; to further strengthen the empirical comparison we added CIFAR-100 results against other recent KD methods, all trained under the same controlled hyperparameters: AdamW (β₁=0.9, β₂=0.999), learning rate $=0.001$, weight decay $=0.5$, batch size $=128$, for 250 epochs with a cosine-annealing schedule and a 10-epoch linear warm-up. Results are reported as mean $\\pm$ std over three runs. Following prior work (e.g., KD and DIST) we set the temperature $\\tau=4$ where applicable. \n\nTable A: CIFAR‑100 performance under identical hyperparameters\n\n| Model (Teacher→Student) | CE | PLD | KD | DIST | DKD | AT | FitNet | PKT | RKD | SP | VID |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet32x4ResNet8x4 | 71.56 ± 0.16 | **73.97 ± 0.05** | 72.11 ± 0.10 | 73.21 ± 0.10 | 73.08 ± 0.27 | 72.51 ± 0.37 | 72.59 ± 0.30 | 72.86 ± 0.17 | 71.45 ± 0.41 | 72.45 ± 0.08 | 71.70 ± 0.32 |\n| ResNet50 MobileNetV2 | 64.42 ± 0.67 | 68.16 ± 0.38 | 66.61 ± 1.16 | 66.98 ± 0.74 | **69.51 ± 0.42** | 52.28 ± 0.84 | 62.77 ± 0.05 | 66.08 ± 0.11 | 65.16 ± 0.21 | 65.30 ± 0.29 | 64.14 ± 0.82 |\n| resnet56resnet20 | 67.57 ± 0.25 | **70.28 ± 0.21** | 69.45 ± 0.24 | 69.66 ± 0.49 | 67.80 ± 0.03 | 68.42 ± 0.16 | 66.75 ± 0.27 | 68.66 ± 0.20 | 68.53 ± 0.18 | 69.36 ± 0.27 | 68.52 ± 0.12 |\n| WideResNet‑40‑2WideResNet‑40‑1 | 70.26 ± 0.24 | **73.50 ± 0.3** | 72.44 ± 0.36 | 72.74 ± 0.33 | 73.22 ± 0.38 | 71.99 ± 0.34 | 70.86 ± 0.33 | 71.43 ± 0.16 | 11.32 ± 1.84 | 73.49 ± 0.20 | 70.61 ± 0.39 |\n\nAs discussed in our Conclusion (Sec 6), PLD naturally extends to any task where student and teacher share the same output logits, including detection or segmentation. To validate this, we distilled Faster R‑CNN detectors on COCO for two teacher→student pairs using DKD  and PLD:\n\n**Table B: Object detection performance (COCO)**\n\n| Teacher → Student | Method | AP | AP₅₀ | AP₇₅ | AP_s | AP_m | AP_l |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet‑50 → MobileNetV2‑FPN | DKD | **29.24** | **50.42** | 29.87 | **16.15** | **31.11** | 38.07 |\n| ResNet‑50 → MobileNetV2‑FPN | PLD | 28.91 | 49.63 | **29.88** | 16.05 | 30.41 | **38.59** |\n| ResNet‑101 → ResNet‑18‑FPN | DKD | 32.11 | 53.43 | 33.9 | 18.46 | 34.43 | 41.5 |\n| ResNet‑101 → ResNet‑18‑FPN | PLD | **32.47** | **53.83** | **34.17** | **18.5** | **34.97** | **42.12** |\n| ResNet‑101 → ResNet‑50‑FPN | DKD | 36.54 | **58.57** | 39.46 | **21.74** | **39.8** | **47.31** |\n| ResNet‑101 → ResNet‑50‑FPN | PLD | **36.6** | 58.28 | **39.58** | 21.37 | 39.76 | 47.24 |\n\n**Baselines & References**\n\n1. **KD**: Hinton, G., Vinyals, O., & Dean, J. (2015). *Distilling the Knowledge in a Neural Network*. arXiv:1503.02531\n2. **DIST**: Huang, T., You, S., Wang, F., Qian, C., & Xu, C. (2022). *Knowledge Distillation from A Stronger Teacher (DIST)*. arXiv:2205.10536\n3. **DKD**: Chen, Z., Liu, W., & Tao, D. (2022). *Decoupled Knowledge Distillation*. arXiv:2203.08679\n4. **AT**: Zagoruyko, S., & Komodakis, N. (2016). *Paying More Attention to Attention*. arXiv:1612.03928\n5. **FitNet**: Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2015). *FitNets: Hints for Thin Deep Nets*. arXiv:1412.6550\n6. **PKT**: Passalis, N., & Tefas, A. (2018). *Learning Deep Representations with Probabilistic Knowledge Transfer*. arXiv:1803.10837\n7. **RKD**: Park, W., Kim, D., Lu, Y., & Cho, M. (2019). *Relational Knowledge Distillation*. arXiv:1904.05068\n8. **SP**: Tung, F., & Mori, G. (2019). *Similarity‑Preserving Knowledge Distillation*. arXiv:1907.09682\n9. **VID**: Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., & Dai, Z. (2019). *Variational Information Distillation for Knowledge Transfer*. arXiv:1904.05835"}, {"review_id": "fdXI1FlFza", "reviewer": "Reviewer_x7Lh", "summary": "The authors propose a general framework for logit-based knowledge distillation from a choice-theoretic perspective. They introduce Plackett-Luce Distillation (PLD), a weighted list-wise ranking loss that interprets logits as \"worth\" scores. The proposed method outperforms standard logit-based baselines such as KD and DIST on standard image classification benchmarks.", "questions": "1. The writing can be improved for clarity and polish. In particular, there is an overuse of em dashes, which disrupts the flow of sentences. Additionally, lines 186 and 187 show inconsistent naming of ListMLE (referred to as both “likelihood” and “maximum-likelihood”), which should be made consistent.\n\n2. The comparison is restricted to only two distillation methods (KD and DIST), whereas many other logit-based distillation methods exist. A broader comparison would strengthen the empirical claims.\n\n3. In the Introduction, the primary motivation is that the Plackett-Luce model can unify cross-entropy and distillation-derived weights, unlike other methods which require an explicit cross-entropy term. However, it is not convincingly argued why having a separate cross-entropy term is a significant drawback in practice. Also, the theoretical benefit of adopting a list-wise ranking loss for image classification tasks remains unclear. what advantage does PLD offer over traditional approaches in this specific context?\n\n4. In PLD, the logits from the teacher model are scaled by a temperature parameter \\tau, but the student model's logits are not. This is inconsistent with the formulation used in temperature-scaled KL divergence, where both teacher and student logits are scaled. Could the authors explain the rationale behind this asymmetric design choice?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper presents an interesting and novel direction by framing logit-based distillation through the lens of choice theory, offering a unified perspective on distillation losses.\n\n2. The proposed PLD method achieves superior performance compared to existing logit-based distillation approaches (e.g., KD and DIST) across standard benchmarks.\n\nWeaknesses:\n\n1. Writing needs further improvement.\n\n2. Too few comparison methods.\n\n3. The motivation is not very clear.", "was_revised": true, "final_justification": "The author's rebuttal addressed my most of the concerns. Thus I will raise my score to boardline accept.", "rebuttal": "* We sincerely thank Reviewers fk7Y, Bcg7, ySdm, and x7Lh, as well as the Area and Senior Area Chairs, for their thorough and constructive feedback on PLD.\n* We are pleased that the reviewers recognize our method’s strong theoretical foundation (fk7Y, Bcg7), its efficiency and practical significance (fk7Y), the clarity of our derivations and writing (Bcg7, ySdm), and the robustness of our empirical evaluation (Bcg7, ySdm).\n* We greatly appreciate the suggestions to broaden our baseline comparisons, quantify runtime overhead, extend experiments to other tasks, and polish the manuscript’s readability. We will incorporate all additional experiments (CIFAR-100 baselines, COCO detection and runtime comparisons) into the revised manuscript.\n* We also thank the reviewers for highlighting minor typos and stylistic improvements, which we have addressed in the updated draft.\n\nThank you Reviewer x7Lh for the detailed and constructive feedback. We are glad you found PLD a novel and effective direction compared to standard logit-based baselines. Below we (1) address writing clarity and consistency by reducing overused em dashes and standardizing the terminology around ListMLE (now consistently described as the likelihood loss); (2) expand empirical comparisons with additional KD baselines on CIFAR-100 under controlled hyperparameters and with an object detection task; (3) clarify the motivation by explaining the practical drawbacks of a separate CE term and the benefit of list-wise ranking in image classification; and (4) clarify the asymmetric temperature design. We hope these additions and clarifications address your concerns, and we would be grateful for your reconsideration of the score.\n\n**Q1: Writing clarity & consistency**\nWe have carefully revised the manuscript to improve readability and ensure consistent terminology. All over‑long, dash‑connected sentences have been broken into shorter, standalone sentences, and we’ve replaced excess em dashes with commas or parentheses for smoother flow. We also updated lines 186–187 to standardize every occurrence of ListMLE—short for *Listwise Maximum Likelihood Estimation*—to “ListMLE (likelihood loss),” removing the term “maximum‑likelihood” to match the original paper’s terminology [38].\n\n**Reference**\n[38] F. Xia, T.-Y. Liu, J. Wang, W. Zhang, & H. Li. *Listwise approach to learning to rank: theory and algorithm.* ICML pages 1192–1199, 2008.\n\n**Q2. Inclusion of additional KD baselines and tasks: CIFAR‑100 Baselines and Object Detection**\n\nAnswer: We included DIST (NeurIPS 2022) as a baseline and structured our ImageNet-1K experiments to compare models horizontally across architectures and scales; to further strengthen the empirical comparison we added CIFAR-100 results against other recent KD methods, all trained under the same controlled hyperparameters: AdamW (β₁=0.9, β₂=0.999), learning rate $=0.001$, weight decay $=0.5$, batch size $=128$, for 250 epochs with a cosine-annealing schedule and a 10-epoch linear warm-up. Results are reported as mean $\\pm$ std over three runs. Following prior work (e.g., KD and DIST) we set the temperature $\\tau=4$ where applicable. \n\n**Table A: CIFAR‑100 performance**\n\n| Model (Teacher→Student) | CE | PLD | KD | DIST | DKD | AT | FitNet | PKT | RKD | SP | VID |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet32x4ResNet8x4 | 71.56 ± 0.16 | **73.97 ± 0.05** | 72.11 ± 0.10 | 73.21 ± 0.10 | 73.08 ± 0.27 | 72.51 ± 0.37 | 72.59 ± 0.30 | 72.86 ± 0.17 | 71.45 ± 0.41 | 72.45 ± 0.08 | 71.70 ± 0.32 |\n| ResNet50 MobileNetV2 | 64.42 ± 0.67 | 68.16 ± 0.38 | 66.61 ± 1.16 | 66.98 ± 0.74 | **69.51 ± 0.42** | 52.28 ± 0.84 | 62.77 ± 0.05 | 66.08 ± 0.11 | 65.16 ± 0.21 | 65.30 ± 0.29 | 64.14 ± 0.82 |\n| resnet56resnet20 | 67.57 ± 0.25 | **70.28 ± 0.21** | 69.45 ± 0.24 | 69.66 ± 0.49 | 67.80 ± 0.03 | 68.42 ± 0.16 | 66.75 ± 0.27 | 68.66 ± 0.20 | 68.53 ± 0.18 | 69.36 ± 0.27 | 68.52 ± 0.12 |\n| WideResNet‑40‑2WideResNet‑40‑1 | 70.26 ± 0.24 | **73.50 ± 0.3** | 72.44 ± 0.36 | 72.74 ± 0.33 | 73.22 ± 0.38 | 71.99 ± 0.34 | 70.86 ± 0.33 | 71.43 ± 0.16 | 11.32 ± 1.84 | 73.49 ± 0.20 | 70.61 ± 0.39 |\n\nAs discussed in our Conclusion (Sec 6), PLD naturally extends to any task where student and teacher share the same output logits, including detection or segmentation. To validate this, we distilled Faster R‑CNN detectors on COCO for two teacher→student pairs using DKD  and PLD:\n\n**Table B: Object detection performance (COCO)**\n\n| Teacher → Student | Method | AP | AP₅₀ | AP₇₅ | AP_s | AP_m | AP_l |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| ResNet‑50 → MobileNetV2‑FPN | DKD | **29.24** | **50.42** | 29.87 | **16.15** | **31.11** | 38.07 |\n| ResNet‑50 → MobileNetV2‑FPN | PLD | 28.91 | 49.63 | **29.88** | 16.05 | 30.41 | **38.59** |\n| ResNet‑101 → ResNet‑18‑FPN | DKD | 32.11 | 53.43 | 33.9 | 18.46 | 34.43 | 41.5 |\n| ResNet‑101 → ResNet‑18‑FPN | PLD | **32.47** | **53.83** | **34.17** | **18.5** | **34.97** | **42.12** |\n| ResNet‑101 → ResNet‑50‑FPN | DKD | 36.54 | **58.57** | 39.46 | **21.74** | **39.8** | **47.31** |\n| ResNet‑101 → ResNet‑50‑FPN | PLD | **36.6** | 58.28 | **39.58** | 21.37 | 39.76 | 47.24 |\n\n**Q3.1: why having a separate cross-entropy term is a significant drawback in practice?**\n\nAnswer: The separate CE mixing weight is brittle, expensive to tune, and cannot adapt per sample; PLD subsumes CE into a single data-dependent ranking loss, eliminating that hyperparameter.\n\nStandard KD minimizes$$\\mathcal{L} = \\alpha\\\\mathrm{CE}(y,s) + (1-\\alpha)\\\\mathrm{KL}(q^T\\parallel q^S),$$\nwhich requires tuning both the CE weight $\\alpha$ and the distillation weight $1-\\alpha$. As Figure 1a shows (Sec. 1), small changes in $\\alpha$ can shift Top-1 accuracy by more than 1%, and setting $\\alpha=0$ (dropping CE entirely) causes a notable performance drop. This global hyperparameter is brittle, it doubles the tuning burden compared to CE alone, and it forces costly grid searches (Appendix B), especially because variants like DIST introduce additional weights such as temperature $\\tau$ and balancing terms $\\beta,\\gamma$.\n\nMoreover, individual teacher output distributions vary per example: highly skewed logits require stronger softening to reveal inter-class structure, while flatter logits require less. A fixed $\\alpha$ cannot adapt to this heterogeneity. In contrast, PLD’s first Plackett–Luce factor coincides with CE on the true class, and each subsequent ranking step is weighted by the teacher’s own softmax mass $\\alpha_k = q^T_{\\pi^*_k}$, making the supervision inherently data-dependent (Sec. 4.2). This unification subsumes CE into a single convex ranking loss and reduces the total hyperparameter count by one (removing $\\alpha$).\n\nMotivated by Sutton’s “bitter lesson” that general methods scale best with compute, PLD offers a unified, self-tuning objective that reduces tuning overhead and focuses compute on scale rather than hyperparameter search.\n\n**Q3.2: What’s the advantage of list-wise ranking for image classification?**\n\nAnswer: During inference, classification sorts the output logits—for example, it computes $\\arg\\max_i s_i$ or selects the top-$k$ values. Training typically uses the cross-entropy loss$$\\mathcal{L}_{\\mathrm{CE}} = -\\sum_i y_i \\log s_i.$$\nThis loss enforces $s_y > s_i$ for all $i \\neq y$. As shown in Section 4.1 (L172–L173), this constraint matches the first selection step in the Plackett–Luce model. Softmax guarantees $s_i > 0$ for every class, and cross-entropy training avoids overconfidence, yielding smooth output distributions. As a result, the detailed ordering among non-target classes is not fully exploited. In this sense, cross-entropy behaves like a relaxed ranking loss.Knowledge distillation captures these residual relationships via the teacher’s soft distribution $q^T$. For example, classes $a,b,c,d,e$ with true label $e$, cross-entropy enforces four independent constraints: $s_e > s_a$, $s_e > s_b$, $s_e > s_c$, and $s_e > s_d$. A list-wise ranking loss instead enforces the full ordering$$e > a > b > c > d\n$$in a single objective. This objective also enforces sub-rankings such as $a > b > c$ and $b > c > d$.Empirically, Table 2 shows that PLD’s ListMLE surrogate outperforms cross-entropy. This confirms that supervising the entire teacher-optimal permutation yields stronger Top-1 accuracy gains. Similar benefits of list-wise losses over pairwise losses have been observed in learning-to-rank methods such as ListNet \\[1].\n\n**Reference**\n\n\\[1] *Learning to rank: from pairwise approach to listwise approach.* ICML 2007.\n\n\n**Q4: Why is temperature applied only to the teacher logits in PLD, rather than symmetrically to both teacher and student as in temperature-scaled KL divergence?**\n\n**Answer:**  Applying a temperature $\\tau$ to the student logits $s \\mapsto s' = s/\\tau$ simply rescales the entire PLD loss by $1/\\tau$. Specifically,$$\\mathcal{L}(s') = -\\sum_k \\alpha_k \\log P_{\\mathrm{student}}(\\pi^*; s/\\tau)= \\frac{1}{\\tau}\\,\\mathcal{L}(s).\n$$Under gradient descent with learning rate $\\eta$, the update$$s \\leftarrow s - \\eta\\,\\frac{\\partial \\mathcal{L}(s')}{\\partial s}\n= s - \\frac{\\eta}{\\tau}\\,\\frac{\\partial \\mathcal{L}(s)}{\\partial s}$$is equivalent to using the original loss with an effective learning rate $\\eta' = \\eta/\\tau$. Thus student-side softening does not add modeling expressivity; it only changes the step size. In Supplementary Sec. 3 (Table 2) We find that standardizing both teacher and student logits yields 70.66% Top-1 accuracy, while standardizing only the teacher logits gives 70.81%, compared to 71.16% with no logit scaling. \n\n**Baselines & References**\n1. **KD**:arXiv:1503.02531\n2. **DIST** arXiv:2205.10536\n3. **DKD**: arXiv:2203.08679\n4. **AT**: arXiv:1612.03928\n5. **FitNet**:arXiv:1412.6550\n6. **PKT**:arXiv:1803.10837\n7. **RKD**:  arXiv:1904.05068\n8. **SP**:     arXiv:1907.09682\n9. **VID**:    arXiv:1904.05835"}]

Author rebuttal:

Meta-review: This paper introduces Plackett–Luce Distillation (PLD), a choice-theoretic, list-wise knowledge distillation loss that unifies cross-entropy and distillation terms in a convex, differentiable framework. The method is theoretically well-motivated and demonstrates consistent improvements over standard KD and DIST on ImageNet-1K.

Reviewers initially raised several concerns. Reviewer fk7Y questioned whether PLD extracted more information beyond soft-label remapping, criticized the limited baseline coverage, and flagged potential runtime overhead and unclear writing. Reviewer Bcg7 found the paper clear and theoretically sound but requested uncertainty estimates and comparisons to a broader set of KD methods. Reviewer ySdm found the paper well-organized and convincing but asked for inclusion of DKD baselines and suggested reorganization of the experimental section, while also expressing curiosity about applicability beyond classification. Reviewer x7Lh felt the writing needed polish, the motivation was not clearly argued, and empirical comparisons were too narrow.

The rebuttal addressed these points: the authors provided expanded CIFAR-100 baselines including DKD and other methods, reported runtime analysis showing PLD comparable to KD and DIST, added COCO detection experiments to demonstrate generalization beyond classification, and revised the manuscript for clarity. These additions persuaded most reviewers. Reviewer Bcg7 and ySdm endorsed acceptance, satisfied with the new results. Reviewer x7Lh acknowledged the clarifications and raised the score to borderline accept. Reviewer fk7Y recognized that many concerns were addressed and also raised the score.

Overall, the consensus is that PLD is a technically solid and good contribution to knowledge distillation. With two reviewers recommending acceptance and two moving to borderline accept after rebuttal, the balance of opinion is that the paper merits acceptance.
Decision: Accept (poster)