Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels
Authors: Hyeonsu Jeong
Hye Won Chung
Venue: ICLR 2025 Poster (2025)
Keywords: self-distillation
partial label learning
label noise correction
training with soft labels
multi-class classification
Primary area: learning theory
TL;DR: We show that multi-round self-distillation functions as a form of label averaging, which is effective in correcting label noise. Additionally, we propose an efficient one-step distillation approach, inspired by partial label learning.

Abstract: We investigate the mechanisms of self-distillation in multi-class classification, particularly in the context of linear probing with fixed feature extractors where traditional feature learning explanations do not apply. Our theoretical analysis reveals that multi-round self-distillation effectively performs label averaging among instances with high feature correlations, governed by the eigenvectors of the Gram matrix derived from input features. This process leads to clustered predictions and improved generalization, mitigating the impact of label noise by reducing the model's reliance on potentially corrupted labels. We establish conditions under which multi-round self-distillation achieves 100\% population accuracy despite label noise. Furthermore, we introduce a novel, efficient single-round self-distillation method using refined partial labels from the teacher's top two softmax outputs, referred to as the PLL student model. This approach replicates the benefits of multi-round distillation in a single round, achieving comparable or superior performance--especially in high-noise scenarios--while significantly reducing computational cost.

Published as a conference paper at ICLR 2025
RETHINKING SELF-DISTILLATION" LABEL AVERAG- ING AND ENHANCED SOFT LABEL REFINEMENT WITH PARTIAL LABELS
Hyeonsu Jeong & Hye Won Chung
School of Electrical Engineering
Korea Advanced Institute of Science and Technology (KAIST)
Daejeon, South Korea (hs jhongg122,, hwchung ) akaist.ac.ke
ABSTRACT
We investigate the mechanisms of self-distillation in multi-class classification, particularly in the context of linear probing with fixed feature extractors where traditional feature learning explanations do not apply. Our theoretical analysis reveals that multi-round self-distillation effectively performs label averaging among instances with high feature correlations, governed by the eigenvectors of the Gram matrix derived from input features. This process leads to clustered predictions and improved generalization, mitigating the impact of label noise by reducing the model's 's reliance on potentially corrupted labels. We establish conditions under which multi-round self-distillation achieves 100% population accuracy despite label noise. Furthermore, we introduce novel, efficient single-round seff-dstilllation method using refined partial labels from the teacher's top two softmax outputs, referred to as the PLL student model. This approach replicates the benefits of multi- round distillation a single round, achieving comparable or superior performance- especially in high-noise scena significantly reducing computational cost.
INTRODUCTION
Knowledge distillation, initially introduced by Hinton et al. (2015), has emerged as a powerful technique in deep learning, where smaller student model is trained to replicate the predictions of a larger, more complex teacher model. In classification tasks, the teacher's output probability distribution, often obtained via softmax, serves as "soft targets" for training the student model. This approach has proven effective across various domains including computer vision, speech recognition, and natural language processing (Chebotar & Waters 2016; Cui et al.) 2017 Asif et al.) 2019 Liu et al. 2019). yielding smaller, more efficient models that often generalize better than similarly sized models trained without distillation.
Building upon this foundation, self-distillation eliminates the need for a larger teacher model by using a model of the same architecture as its own teacher. In this process, models of the same size are trained over multiple rounds, with each iteration learning from the outputs of the previous one. Surprisingly, self-distillation has been shown enhance performance, especially when repeated over multiple rounds (Furlanello et al. 2018: Zhang et al. 2019). This improvement is intriguing because it cannot be fully explained by traditional theories that focus on student mimicking a more complex teacher, prompting a deeper exploration into the mechanisms underlying self-distillation.
Recent research has begun provide diverse explanations for the benefits of diff-disillation. For example, Allen-Zhu & Li (2022) describe self-distillation as feature-learning process in neural networks, particularly under a multi-view data distribution where each class possesses multiple orthogonal features (e.g.. "windows" and "wheels" for cars). The teacher's softmax outputs help the student model learn richer set of features, improving generalization. Another line research emphasizes that the benefits of self-distillation extend beyond neural networks to classical regression settings (Mobahi et al.) 2020; Dong et al.) 2019:: Das & Sanghavi] 2023). In these contexts, self- distillation acts regularizer, preventing overfitting and promoting generalization.

---

Published as a conference paper at ICLR 2025
Despite these insights, several critical questions remain unaddressed. First, can self-distillation provide benefits in training a simple linear classifier atop a fixed pre-trained feature extractor-a process known linear probing? In this setting, traditional explanations involving feature learning do not apply, as the feature extractor is fixed. With the emergence of large-scale pre-trained networks, or foundation models (Bommasani al.) (02T), linear probing has become an increasingly popular method to leverage these networks' capabilities for downstream tasks. Yet, it remains unclear whether self-distillation can enhance this process without additional feature learning, and how repeated rounds of distillation impact the model outputs.
Furthermore, can the benefits of multi-round self-distillation be achieved more efficiently-in just a single round-by developing methods that replicate these gains without repeated training cycles? While multi-round self-distillation has shown performance improvements, can be computa- tionally intensive due to the need for multiple training iterations. Developing efficient methods to achieve similar gains without multiple rounds is therefore of significant practical interest.
Our Contributions In this paper, we address these questions by:
Analyzing Self-Distillation in Linear Probing:
Our analysis reveals that self-distillation effec- tively performs label averaging among instances with high feature correlations when generating predictions the training data. This averaging is governed by the eigenvectors of the Gram matrix derived from input features (defined in Sec. 2.2). As distillation progresses, the top-K eigenvectors-whese K is the number of classes-dominate this process, leading increasingly clustered predictions around instances from the same ground-truth class. This finding provides insights into how diff-istillation benefits even in simple linear probing without feature updates.
Quantifying Gains in Label-Noise Scenarios: We quantify the number of distillation rounds needed to achieve 100% population accuracy in presence of label noise that is, correctly predicting the ground-truth label for any training instance as the number of instances goes to infinity. This number depends on the relative strength between signals from the given (potentially noisy) labels and the averaged labels of instances with high feature correlation. As the latter signal strengthens with more distillation rounds, the student model can correctly classify samples with noisy labels after a few iterations, demonstrating the robustness imparted by multi-round self-distillation.
Introducing an Efficient Single-Round Method using Partial Label Refinement: We introduce a novel self-distillation approach that replicates the benefits of multi-round distillation in a single round. This method trains the student with refined partial labels, focusing on the top two softmax outputs from the teacher. We demonstrate that this approach outperforms traditional multi-round self-distillation in high-noise scenarios, while significantly reducing computational cost. Our findings supported by experiments on synthetic and real datasets (Figure 3 and Section 6
Related Works Since its introduction by Hinton et al. (2155) and subsequent extensions (Romero et al. 2014 Tian et al. 2019 Lopez-Paz et al. 2015), several studies have sought to theoretically analyze the effects of knowledge distillation and self-distillation. Many of these works rely specific assumptions about datasets, model architectures, or loss functions to facilitate tractable analysis. Mobahi et al. () Dong et al. (219) explored self-distillation in classical regression settings using mean squared error loss. They demonstrated that multi-round distil reduces the number of basis functions in the model, which helps prevent overfitting initially but can lead to underfitting after excessive rounds. In another line of research focusing on binary classification with cross-entropy loss, |Phuon & Lampert (2019): Ji & Zhu (202); Das & Sanghavi (2023) analyzed the effects of distillation on linear networks or logistic regression under label noise. Das & Sanghavi (2023) specifically identified the range of label corruption where the student model outperforms the teacher.
In contrast to these studies, our paper examines multi-round self- fit--ditillation for multi-class classifica- tion with cross-entropy loss, particularly in the context of linear probing where the feature extractor is fixed and feature learning is absent. We demonstrate that the effect of self-distillation can be interpreted as label averaging among highly correlated instances, offering a new perspective on the mechanism of self-distillation. While Allen-Zhu & Li (2022) explain the benefits of self-distillation through feature learning under a multi-view data distribution, our results show that significant benefits can be achieved through label averaging alone, even without feature evolution. Specifically, our theoretical analyses (Theorems [.I) illustrate how self-distillation leads to clustered outputs based on feature correlations, enhancing generalization by reducing dependence on given (potentially
2

---

Published as a conference paper at ICLR 2025
noisy) labels. Additionally, findings can be extended to include feature learning by considering the evolution of the feature map over multiple distillation rounds (as discussed in Corollary C.I).
Our work also introduces a novel method for efficiently distilling "dark knowledge" from the teacher model. Instead of utilizing the full softmax output as in traditional distillation, we refine the teacher's outputs into set of feasible labels by focusing the top two predictions. This approach is analogous to Partial Label Learning (PLL) (Cour et al. 2011), where each training sample is associated with a of candidate labels rather than a single ground truth label. Various PLL methods aim improve
classification accuracy by weighting candidate labels using logits (Lv et al. 2020; Wang et al. 102T
or leveraging feature space topology (Wang et al., 2019 Xu et al., 2019 Lyu et 2019) Our method differs by directly employing the refined partal labels derived from the teacher's s outputs, achieving the same benefits as multi-round distillation just one round. Importantly, our approach outperforms traditional multi-round distillation in high-noise scenarios, as it consistently includes the ground-truth label among the top two predictions, as demonstrated by Theorem 5.1
PRELIMINARIES AND OVERVIEW OF RESULTS
Notation We denote vectors (matrices) by lowercase (uppercase) bold letters. For vector and matrix A, let [v]: or 0; represent the i-th entry, and [A]i, the (i,j)-th entry. Let e(i) be unit vector with in the i-th entry. We use || I|FF for the Frobenius norm. Let 1axm R2xm be all-one matrix.
2.1 SETTING: SELF-DISTILLATION FOR K-CLASS CLASSIFICATION
We consider the problem of K-class classification, where each sample x E X' is associated with a ground class label y(x) E y == {1, 2, ..... K). Let P denote the underlying distribution over the sample space X' V. Assume that we have a feature map Φ : X' - >X C Rdd, We are given a dataset [(((x;) (. ())) Kr consisting of Kn feature-label pairs, where each (Ø(x;). ): (.) E X x y. For simplicity, we assume that the feature vectors are normalized, i.e., [|)(x;) 12
Vi E [Kn]. The given label Di may be noisy, while the ground-truth label of x; is y/ := (x,). We further assume that the dataset is balanced with respect to both the ground-truth labels and the given labels, meaning that for each class l E [K]. there are n samples with y/ = & and n samples with J. =
We consider classifier O Rd K based multinomial logistic regression (softmax regression). For an input x, the classifier outputs y = (0 o(x)), where a : RKK - 1)x is the softmax function. The predicted class label for inputxx is then arg max (00 o(x))] The population accuracy
defined as [xxx((x)>~> 1 arg max (o ((x)] = v(x)) This setting equivalent to linear probing in the context of neural networks, where the network consists of fixed feature extractor Φ X > X followed by tunable fully connected linear layer parameterized by O and softmax layer. As the training objective, we consider the (z-regularized cross-entropy (CE) loss between the target labels (e.g.. e(ik) for the teacher model) and the model's predictions y; == o(0 (xx:)):
Σ CE(e(()). + ||10
Kn
(T(0) = Kn
2
where À > 0 is a regularization parameter, and CE(e(()),))) = = (e())) loglydle
In self-distillation, the student model, which shares the same architecture as the teacher, minimizes the same objective (T] over the same training set {ø(x;)}; ). but with new targets obtained from the teacher's output. Let 07 == arg ming (r) be the teacher's optimal parameters. The new target for
an input x; becomes the teacher's output prediction y == o (0IT)(x:)) rather than the given label (())). Self-distillation can be repeated iteratively, where the i-th distilled model uses the outputs of
the (1 (1-15 1)-th model as its targets. Denoting the output of the (t (t- 11th model for the i-th instance as
y; (t-1) the training objective for the t-th student model 0 E d for & = given by
1 Kn
11101
(() = Kn
CE
(t-1) . (!
(2)
1=1
for y! = ((x))):)) (0) (0)+)))))) is the same objective the teacher model ①1 when y
e(ji).
3

---

Published as a conference paper at ICLR 2025
The optimal 0 that minimizes
(2 for the t-th distilled model, denoted by @(0), is expressed as
( yi (((-1))
Kn
yi
Kn
y (1 y:
o(x,)
T
o(x;) where a; (e)
Knd
Knd
since it satisfies the condition 4r(0 d0 Kn E ((x; (y, y - + A0 = 0. Note that
((t) a linear combination of the training instances {o(x;)) with coefficients a (t) determined by
the difference between the target y; (B-1) and the output y,
(2) The key question is how the optimal classifier ( and the corresponding output y (0 ((x)) evolve as increases.
To answer this question, we need to analyze the coefficients {a
( Kn in (3 which satisfy Jim1
Kn Σ
Knda, 1 = y(( (T ((x:)) (((OT((()))
y((
(0 (x)),
(()))a
vi [Kn], (4)
( } is challenging due to the non-linearity of the softmax function To address this, consider the following linear approximation of the softmax: Assumption 1. We approximate the softmax (vv) for the logits v E RK by
derived from the definition. Solving 4 for {ar")
exp(U1)
exp(uk) Σ exp(1:)
1
1 UK
o(v)=
(5)
E E (1.)
K K+ - 24
K K+
This linear approximation has been considered in the analysis knowledge distillation, as ini- tially explored by Hinton et al. (2015), where the softmax output layer converts the logits us- ing (v/T) with high temperature 5 > O. When T = 1, the approximation remains valid the logits are of sufficiently small magnitude. From B the logits of the t-th distilled model Kn
for an input (xx.) are given by ((4) ((x)) = Σ/n1 (o(x;), (x This is equivalent to (0())(x))) Thus, the magnitude of the logits becomes small if the average difference between the softmax outputs of the teacher (i.e., the (1 1) model) and the student (i.e., the i-th model) across the training instances is sufficiently small. In Appendix D.I we empirically demonstrate that this approximation holds well for blass-balancee datasets with a sufficient number of samples. Using the fact that the logits zero-mean, i.e., ΣX vi O, for all j. we have (v) = (v(v) == K 1x + v.. Thus, the softmax output can be approximated as uniform vector k K perturbed by a (scaled) logit v. Under this approximation, we derive closed-form expressions for {a (t) in (4) and analyze how the model's outputs change over multiple rounds of self eff-distillation.
Σja, Knd
(e(x)) (o(0tt=))(x)))
which holds since T (xx;) Σドー (Ø(x;), ((x))a,, and Σ
2.2 OVERVIEW OF OUR RESULTS
Given a set of input features (o(x))) we define the Gram matrix & E KnnKnn as [b]e := (o(x,), o(x;)), Vi,j j [Kn], where (:,-) denotes the inner product. The eigen-decomposition of is written as ΣxE Kn A.V.V. with orthonormal eigenvectors VI...., VKn E Kn and the
corresponding eigenvalues d 2 ... AKn 2 0.
Theorem 2.1 (Informal version of Thm. 3.1. (8)
As the number of self-distillation rounds E z ...... knl IR KxKK of the 1-th distilled model for the inputs of
increases, the output (0) [y
the training dataset (ø(x;),0: Kn evolves as
1 1KKKn= (Y(0))
Kn Σ
A
(())
-
V.V.
K
K
K
where (U) = (e( ... e(āxn)) represents the one-hot encoded vectors of the given labels.
This theorem reveals that after rounds of self-distillation, the model's predictions become weighted average of the given labels, with weights determined by the transformed Gram matrix $(() :=
4

---

Published as a conference paper at ICLR 2025
(X +.)) Note that ((t) retains the same eigenvectors the Gram matrix s, but its eigenvalues take the form (A.((K2m) + X.)),, maintaining the same decreasing order as the eigenvalues of Φ for i E [Kn]. The eigenvalues of (+() decay exponentially with t since \:/(Knn + X,) < 1. Consequently, the label averaging process becomes increasingly dominated by the top eigenvectors of <iit, as shown in Fig. 2a See Section for details.
The label averaging effect of multi-round self-distillation is particularly advantageous in label-noise scenarios, as it allows the model's output prediction for each training instance to be influenced not only by the given (potentially noisy) label but also by the labels of other instances with high feature correlations. To quantify the benefits of multi-round self-distillation in the presence of label noise, consider low-rank structure for the Gram matrix +b R KK=KKK, Kn:
i= j:
C,
Yi Uz. U.
(7)
d
Vi + yj. h(y:) = h(Us),
0,
h(y)) & h(Uj),
class correlation matrix
superclass correlation matrix
where c d 2 and h : [K] + [R] maps each class to a superclass This block-wise structure of the Gram matrix motivated by input feature correlations observed in real datasets, such as those obtained from neural networks pre-trained on Im- ageNet, as shown in Fig. I Under the structure 7 the top-K K eigenvectors of + reveal clusters of instances shar- ing the same ground- truth class, inferred from their feature correlations. Consequently, the label averaging by ((),, in (6. causes predictions for the instances from the same ground-truth class become more similar as illustrated in Fig. 2bb, even when labels are noisy. This clustering mitigates the impact of label noise by reducing the model's reliance on potentially corrupted given labels. Our main result quantifies the benefits of multi-round self- distillation by establishing conditions under which the t-th distilled model can achieve 100% population accuracy. These conditions are equivalent to those where the model can correctly classify all training data, including label-noisy instances, in the limit of n -> 20.
set of confusable classes (R < K).
Gram Matrix of CIFAR-100
-44
increase,
The Gram matrix + of 50,000 instances of CIFAR-100, extracted from a ResNet34 network, pre-trained on Im- ageNet. Similar block-wise structures of the Gram matrix are observed for six different real datasets in Fig. 7
Figure 1:
Theorem 2.2 (Informal version of Thm. 4.1). Given label corruption matrix C E K where [C] == Hi Y1 = k. di k/H/n, the 1-th distilled model trained on this label-noisy dataset achieves 100% population accuracy if
[C],, [C],, + 1/
Wk. k' E [K] with ls + K,
(8)
where Aα (K/(Kn Xx) and AR+1 = Ax+1 + with Ax 1 - c nlcc d).
AK+1 = C, as per the assumed Gram matrix in
This theorem shows that as the number of distillation rounds t increases, the condition for achieving 100% population accuracy becomes less stringent due to the eigenvalue ratio, (Ax //x+1)( increasing exponentially with This allows the model to tolerate higher levels of label corruption while still guaranteeing correct classification, highlighting the robustness conferred by multi-round self- distillation in noisy environments. See Section 4 for a detailed analysis.
Lastly, to reduce the computational cost of multiple distillation rounds, we propose novel method that achieves the benefits of multi-round self distiliiilnt in a single round. The key idea is to refine the teacher's 's output into partial labels, focusing the top two predicted classes. Specifically, assign equal weights (1/2) to the top two label positions in the teacher's softmax output and use this refined target train the student model.
Theorem 2.3 (Informal ver. of Thm. 5.I). The student model trained using partial labels, refined from the leacher's outputs, achieves 100% population accuracy, if [C)Rkk & > [C),,t, Vk. k' k) E [K]-
5

---

Published as a conference paper at ICLR 2025
Eigenspectrum of 6!!!
9.3
IA. o 6.30
16.2 1..0
9.1.0.
found
Class
Claps
hound
10 Aound
20 Aound
0.5
0.2
105s
Class
-lles
Casss
Class
0. 6.1
[6.0 1..
Figure 2: (a) The evolution the eigenvalues (()) as the distillation rounds & increases. Notably, only [[= 4) top-eigenvalues dominate the others i progresses, (b) A 2D plot showing the softmax outputs the teacher model and the 4th and 6th student models for the training instances. Each corner represents the one-hot vectors of class. As the distillation round f increases, instances form distinct clusters based on their true labels (represented by colors dots). driven by the label averaging among highly correlated input instances within the same class. See Appendix G for full evolution.
This condition requires that, for each class k, the proportion of samples correctly labeled as k is greater than the proportion mislabeled as any other class k'. Under this mild condition, our method effectively attains the label averaging benefits of multi-round distillation without the need for repeated distillations. See Section for details. Full proofs of our theoretical results are available in Appendix.
LABEL AVERAGING EFFECT OF MULTI-ROUND DELF-DISTILLATION
We present a closed-form solution for the softmax outputs Y [y( (1)
yKa R KxKn the
t-th student model, in terms of the Gram matrix E 1gKnxK Kn defined as [ (o(x.), ((x,)).
Theorem 3.1. As the number of self-distillation rounds t Zt increases, the output predictions for the training instances evolve as
( - -KKKn/ (Y ( -1xk)1 IKn (IKn + 2)(KKn)))
(9)
This result shows that each round of self-distillation effectively averages the given labels (columns of Y())) by repeatedly multiplying (Ina (Ixn. + //(K3nn)) -)). By applying Woodbury's matrix identity and using the eigen-decomposition of + = ΣKE A.V.v, we can express the right-hand
side of as in (6). using (()) IKn (IKn 2/(33AA) "I E
Kn
K V.V.
The relationship between the Gram matrix and the label averaging matrix $ reveals the effect of multi-roum distiliiilaa,,n, summarized as follows:
Clustering Based on Feature Correlation:
The Gram matrix + defined as ( == (6(x;), o(x,) )).
captures the pairwise feature correlations among the training instances. instances from the same ground-truth class exhibit higher feature correlations than those from different classes-as assumed 7 the top K eigenvectors of indicate clusters representing each class (See Appendix for details). Since ( shares the same eigenvectors causes the predictions for each instance to be increasingly influenced by the average labels of other instances within the same ground-truie -ruth class the number of distillation rounds increases.
the label averaging process via (t)
Dominance of Top Eigenvectors: The effect of multi-round self-distillation reflected in the eigenspectrum of (()) (f) where the eigenvalues are X = (A:/(K2 A,)), for E [Kn]. As t increases, these eigenvalues decay different rates depending X; Specifically, the top K eigenvalues (associated with the largest X;) decay more slowly than the others. In Fig. 2a we illustrate how the eigenvalues
(f) evolve with increasing t. We use a training dataset with 100 samples per class, where the Gram matrix follows block-wise
K 4 classes and structure specified in 7 with parameters c = 0.4 and 0.1 for = 1, and is perturbed by random matrix E with entries [E], U 0..5, 0.05). As the top-K K eigenvalues increasingly

---

Published as a conference paper at ICLR 2025
Accuracy Gain of SD
Output of Clean Sample
Output of Nolsy Sample
True Label
True Label
16
Other Labels
Label
Accuracy 196
08
Other Labels
18-
Outpur
Output
66
Teach
Student
3rd Student
PLL Student
Corruption Ratio
Distilation round
Distillaion round round
(a)
(h)
Figure 3: (a) Prediction accuracy of the teacher and student models for different distillation rounds t as the label corruption rate increases. The 100% population accuracy regime expands for student models with more distillation rounds. (b) Softmax outputs for K" = 4 classification as increases, using the same setup Fig. 2 with 50% label corruption rate. The BLUE line shows the average softmax value at the true label, the ORANGE line for noisy samples at the given label, and the GREEN line for other labels. Shaded regions show 11 standard deviation. For clean samples, the true label maintains the highest value, while for noisy samples, the true label starts lower than the given label but surpasses it after a few rounds. The shaded region shrinks with & due to label clustering.
dominate, the model's predictions become more clustered among instances of the same ground- truth class, as shown in Fig. (b] This clustering effect persists as long as the top- K eigenvectors of d align with the ground-truth classes. The robustness of this alignment, even under perturbations from E, is ensured provided that the perturbations are small relative to the eigen-gap (Ax (Ax-Ax41)
established by the Davis-Kahin theorem (Davis & Kahan 1970).
GAIN OF MULTI-ROUND SELF-DISTILLATION IN LABEL-NOISE SCENARIOS
In this section, we quantify the gains of multi-round self-distillation by identifying conditions under which the 1-th student model can achieve 100% accuracy, even when trained on noisy dataset.
Theorem 4.1. Suppose we have access to the population, i.e., n - co. Les E R be the label corruption matrix of the training dataset. defined as [C] == Ki yr k. Us k//nn. We assume that [C],,k' 0 whenever h(k) * h(kk), ). where h(k) denotes the superclass of class Then, the t-th distilled model, trained on this label-noisy dataset with Gram matrix & described in 7 achieves 100% population accuracy if
[C],, > [C]R, + 1/ ((AK/AK+1)'
Yk, Re' [K] with # K
(10)
where do > AK) and SK+1 = AK41/ (K3n/ Ak+1), with Ax = 1 - c c+nnc- n(c-d)
and AK+1 1-c. Conversely, there exist k2, E [K] with + 1e such that
[C]i, [C] +1/ ((xx ((AK//R-I))~-11)
(11)
then the (t- (t- 1)-th distilled model fails to correctly classify label-noisy samples from class k that have been mislabelled as i = k,, and thus fails to achieve 100% population accuracy.
These conditions illustrate the advantages of multi-round self-distillation. Specifically, when,
[C]s.kr + +1/ ((K///K4)) -1) [C]&, < [C] + 1/ ((AK/AK+))
(12)
for some k k E [K] with l & k', the i-th distilled model can correctly classify label-noisy samples from the class that have been mislabelled as Ø; = k', while the (t - 1)-th distilled model cannot. This demonstrates that multi-round self-distillation. through the label averaging effect described in Sec. 3 enhances the model's robustness to label noise, allowing it to tolerate higher rates of label corruption while still achieving 100% population accuracy. This phenomenon is illustrated in Fig, Ba where we compare the classification accuracy of the teacher and student models across increasing rounds of distillation (L), as the label commuttion rate increases, using the same experimental setup as
7

---

Published as a conference paper at ICLR 2025
Fig. 2 As the distillation round t increases, the range of label corruption for which the student model maintains 100% accuracy expands, as the condition (62) becomes easier to satisfy. In Fig. 3b we illustrate how the softmax output components [y(*)], Jc. for
[K], evolves for clean samples (left) and label-noisy samples (right) as the distillation round t increases, with a 50% label corruption rate. For label-noisy samples, while the teacher model (t = 1) tends to assign the highest softmax value to the incorrect noisy label due to memorization, after a few rounds of distillation (t 2 3), the true label begins to attain the highest value in the softmax output, thus improving classification performance.
Proof Sketch of Theorem 4.1 The proof relies on analyzing the evolution of the model's outputs over multiple distillation rounds under the Gram matrix model (7). The key result is Lemma 4.1
Lemma 4.1. Under the Gram matrix model (7. the output of the t-th distilled model for the training instance (o(x,), . ik) with true class yi is given by
y: = p'e(ğ)) + (a' jey + (r:- - -0"g))) (')))) + +1 (1-rr)(x//A
(13)
where p. q. are defined as
n(c d)) Ts = (1 -1--6 + n(c - d) + K,nd))/ (K2 - 1-c n(c d) + K2nd),
p = (1 c)/ nd 1 q:= (1- c1+
nd + -1- c c+nccc (-c-n(())) d)); (14)
with K2 being the number of classes in superclass s h(yi), By: = + (immm)) e(ūi) is the average label vector over samples of the same true class yo, @k(y)) K,n [()hh()))))) (i,, the average label vector over samples in the same superclass h(y;),, and 1 is the all-ones vector in R
This expression shows that the model's output y is weighted combination of the given label e(j.).
the average label vector ey over samples from the same true class y;, the average label vector ()(ss)) over samples from the same superclass h(,.), and the uniform vector. Since 1 > Ts > q > p > O, as the distillation round increases, the weights assigned to these components shift away from the individual noisy label c(ū:) towards the average labels of the same class and superclass, eventually making the output converge to the uniform vector as t -> 20. Theorem 4.1 can be proved by analyzing (13) and finding the condition under which the correct label y/4 attains the highest value in the softmax output components [y Jy("))
Jk. & E [K]] for both clean (Ú: y:) and noisy (js + y:) samples.
A NEW SELF-DISTILLATION METHOD WITH PARTIAL LABELS
In this section, we propose novel self- diffdistillatton method that achieves the benefits of multi-round self-distillation, as outlined in Thm. 4.1 in just a single-round. The key insight stems from the observation that, under mild conditions on the label corruption rates, the teacher's softmax output at the true label (represented by the blue lines in Fig. at = 1) consistently ranks at least second- even if it is lower than the given label. As self-distillation
highest-ever even for label-noisy samples-evee eeeeneee-eeenneee progresses through multiple rounds, the softmax output (or confidence) for the true label decreases in clean samples while increasing in noisy samples due to the label-averaging effect among instances from the same ground-truth class. This phenomenon improves classification accuracy for noisy samples but reduces the model's confidence in clean samples. This trend is illustrated in Fig. 3b where the blue lines for clean (left) and noisy (right) samples converge to similar levels as t increases.
Motivated by this observation, we aim to design a self-distillation method where, in single round, the true label achieves the highest output value, even for noisy samples. To achieve this, we introduce the following label refinement scheme for the teacher's output: We generate two-hot
vector Y: from the teacher's output y
[(0T)(xx)) by selecting the top two labels with the highest values and assigning a weight of 1/2 to each, setting all the other entries to zero. These refined outputs become the new targets for training the student model 6, trained with the objective
fp(0) = शे. Knn CE y (P) + ALO for y (P) - o oxx)). We refer to this student model as the Partial Label Learning (PLL) student model, following the concept of PLL (Cour et al. 2011), which trains a classifier using a candidate set of labels for each input instance. Our label refinement emulates the effect of multi-round self- distiliitlttoon by significantly boosting the confidence (assigning weight 1/2) to the true label for noisy samples while reducing it for clean samples. As a result, the PLL student model achieves 100% population accuracy after single round of distillation.
2
8

---

Published as a conference paper at ICLR 2025
Theorem 5.1. The PLL student model, trained on the label-noisy dataset with the Gram matrix + from 77. achieves 100% population accuracy if
[C]&,& > [C],,ts, vk, E [K] with + k',
(15)
where the label-corruption matrix C is defined by [C|s,a == Ki y/i k,ū; k)1/n, and assume that [C]& = whenever likk) o h(k)).
Proof Sketch. The condition (15) ensures that, for each class k. the proportion of samples correctly labeled as k is greater than the proportion mislabeled as any other class k' * k. Under this condition, the teacher's 's softmax output includes the true label among the top two highest values for both clean and label-noisy samples. By using the refined partial labels yı, single round of self-distillation generates an output for the i-th sample similar to (13):
(1
() = py: + (q - y
Σ
8,
+ (ro - 4)
8z Knn (1
Σ
1K + -r-) re)
(16)
K
This expression ensures that the true label yi has the highest value in y; y! for all k F V/. each partial label y, assigns weight of 1/2 to the true label.
i.e., by (1)
ly
Remark 1 (Comparison with Multi-Round Self-Distillationn. The condition 62) for achieving 100% accuracy with the 1-th self-distilled model is more stringent than the condition (15) for the PLL student model. Therefore, the PLL student model not only replicates the effect of multi-round distillation in a single round but also achieves better performance in scenarios with high label-noise rates, where (15) is satisfied but (62) is not. This advantage is illustrated in Fig. Ba where the PLL student model achieves 100% population accuracy across a wider range of label corruption rates.
EXPERIMENT
In this section, we validate our theoretical findings through experiments on real datasets. We employ two-layer neural network composed of linear layer followed by softmax layer on top of
fixed feature extractor. We conduct experiments on six multi-class image classification benchmarks:
CIFAR- Krizhevsky et al. 2009). Caltech- faltech-101/256 101/256 (Griffin al. 2007 Flowers- (Nilsback)
& Zisserman 2008), Food-101 101 BBossard et al. 2014), and StanfordCars (Krause et al. 2013)). utilizing
the Py Torch torchvision library. For our fixed feature extractor, we use ResNet34 (He al. 2016)
networks pre- traitrained on ImageNet. More experimental details can be found in Appendix E We
explore three different label corruption scenarios: symmetric, asymmetric, and superclass corruption,
as described in Table [ for label corruption rate 1) E [0, 1] and Ω. == (1 E [K] h()) = h(k)).
Table I: Definition of label corruption matrix C for different corruption scenarios
Symmetric corruption
Asymmetric corruption
Superclass corruption
k. W R(R), W + ke R(A).
Th. Rh -
k' = k
4 K k. R-
[C
2g
[C],,
k' of k, k Qe.
1 + Qz.
R
Fig. Jillustttas the accuracy improvement of student models compared to the base (teacher) model in the superclass corruption scenario. Results for other datasets and corruption scenarios are available in Appendix Across all datasets, we observe that test accuracy progressively increases with the number of distillation rounds. Notably, the PLL student model outperforms other multi-round distillation models at higher corruption rates (1 2 0.6), achieving this gain efficiently in just a single round of distillation. These results support our theoretical findings, showing the effectiveness of multi-round self-distillation and especially the PL. student model-in mitigating the effects of label noise.
Verification of the Label Averaging Effect To verify the occurrence of label averaging effect by self-distillation in real datasets, plot the evolution of model outputs across distillation steps. We arbitrarily select four classes from the CIFAR- 100 dataset and conduct experiments only on samples from these classes. Under 50% symmetric label corruption, visualize the output evolution of

---

Published as a conference paper at ICLR 2025
StanfordCars
16.0
10.5
Destation Co OR
F
FIL
Diotattin
Comuption Radio o 2961
Comuption Ratio (1)
Compption Berto 2944
Figure 4: Distillation gain in accuracy (%) by each student model compared to the teacher in the superclass label corruption scenario. PLAIN represents the improvement of the multi-round students over the teacher model, while ORANGE represents the improvement by the PLL student model.
both the teacher and student models 2D plot in Fig. 5a As the number of distillation rounds increases, we observe that the outputs tend to form clusters based on their true labels. Additionally, some instances initially misclassified by the teacher are correctly classified by the student model due to the label averaging effect. This suggests that the distillation process helps the student model recover from some of the teacher's errors, especially in cases where the labels are corrupted.
Ablation Study of Top-k Targets In the PLL student model. using larger set of partial labels in- creases the likelihood of including the true label but introduces a trade-off by reducing the confidence in instances that are already well-classified. Our theoretical analysis in Sec. 5 indicates that the true label's prediction is at least the second highest in the teacher's 's output, supporting the use of a top-2 candidate set. To validate this choice, we conduct experiments under the same conditions as Fig. using the top-k partial labels as targets for various k. In Fig. [b] we plot the accuracy gain of each PLL student model with top-k partial labels, chosen from the teacher's output, on the CIFAR-100 dataset. Results for other datasets are provided in Appendix H.1 As shown, using more candidates generally leads to lower test accuracy across most datasets, supporting our choice top two labels.
4th
with Top-k Labels
Class
Class
Glass
Class
Class
CH Diollattn Gato
Luie
1953
Class 3
Class
Class
Class
Class
Class 4
6.0
15.88.11
Comuuttio Rorlo (60
(b)
Figure 5: (a) Visualization of the softmax outputs of the teacher model, and the first and fourth student models on the CIFAR-100 dataset, reduced four-class classification outputs for clarity. (b) Accuracy improvement (%) for each PL.I student model using top-k partial labels compared to the teacher model the superclass label corruption scenario on the CIFAR-108 dataset.
7 CONCLUSION
We investigated self-distillation for multi-class classification in the context of linear probing with fixed feature extractors. Our theoretical analysis showed that multi-round self-distillation effectively performs label averaging among instances with high feature correlations, which helps correct label noise by reducing the model's reliance on potentially corrupted labels. Additionally, we introduced the PLL student model as an efficient approach for label refinement, offering practical method to distill the teacher's "dark knowledge" into candidate set of labels. This method achieves the benefits of multi-round self- celf-distillation in just single round, providing practical and efficient way to improve model performance. Limitations and future works are available Appendix
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported by the National Research Foundation Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024-00408000 and RS-2025-00516153), and by the Institute for Information & communications Technology Planning & Evaluation (IITP) grants funded by the Korean government (MSIT) (RS-2024-00444866)
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. In The Eleventh International Conference on Learning Representations, 2022.
Umar Asif, Jianbin Tang, and Stefan Harrer. Ensemble knowledge distillation for learning improved and efficient networks. arXiv preprint (0xxvv::909000999 2019.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.0m 07258. 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446 Springer, 2014.
discriminative compo-
Yevgen Chebotar and Austin Waters. Distilling knowledge from ensembles of neural networks for speech recognition. In Interspeech, pp. 3439-3443, 2016.
Timothee Cour, Ben Sapp. and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research, 12:1501-1536, 2011.
Jia Cui, Brian Kingsbury, Bhuvana Ramabhadran, George Saon, Tom Sercu, Kartik Audhkhasi. Abhinav Sethy, Markus Nussbaum-Thhon and Andrew Rosenberg. Knowledge distillation across ensembles of multilingual models for low-resource languages. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4825 4829. IEEE, 2017.
Rudrajit Das and Sujay Sanghavi. Understanding self-distillation in the presence of label noise. In Proceedings of the 40th International Conference on Machine Learning. 2023.
Chandler Davis and William Morton Kahan. The rotation of cigenvectors by perturbation. iii. SIAM
Journal on Numerical Analysis, 7(1):1-46, 1970.
Bin Dong, Jikai Hou, Yiping Lu, and Zhihua Zhang. Distillation early stopping? harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network. arXiv preprint arXiv: 1910.01555 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representationss 2021. URL [tLL:ttts/////eeeeee net/Eozum?id-Yicc
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference Machine Leurning, pp. 1607-1616.
PMLR, 2018.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 1503.0500..220 2015.

---

Published as a conference paper at ICLR 2025
Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. Advances in Neural Information Processing Systems, 33: 00823-208333 2020.
Jonathan Krause, Michael Stark, Jía Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Xiaodong Liu, Pengcheng Hc, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neu- ral networks via knowledge distillation for natural language understanding. arXiv preprint arXiv: 2019.
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. arXiv preprint arXiv:1571.03643, 2015.
Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In international conference on machine learning, pp. 6500-6510, PMLR, 2020.
Gengyu Lyu, Songhe Feng, Tao Wang, Congyan Lang, and Yidong Li. Gm-pll: Graph matching based partial label learning. IEEE Transactions Knowledge and Data Engineering, 33(2): 521-535, 2019.
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. Advances in Neural Information Processing Systems, 33:3351-3361 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pp. 722-729. IEEE, 2008
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation, In International conference on machine learning, pp. 5142-5151 PMLR, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 2014.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna- lional Conference on Learning Representations, 2019.
Deng-Bae Bao Wang, Li Li, and Min-Ling Zhang. Adaptive graph guided disambiguation for partial label learning. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 83-91, 2019.
Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico: Contrastive label disambiguation for partial label learning. International Conference on Learning Representations, 2021.
Ning Xu, Jiaqi Lv, and Xin Geng. Partial label learning via label enhancement. In Proceedings of the AAAI Conference on artificial intelligence, volume 33, pp. 5557-5564, 2019.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3713-3722, 2019.
Zhilu Zhang and Mert Sabuneu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances neural information processing systems, 31, 2018
12

---

Published as a conference paper at ICLR 2025
A BROADER IMPACT
Our paper theoretically analyzes self-distillation, particularly highlighting its advantages in multi- class classification problems with label noise. We also explore the integration of self-distillation with partial label learning, demonstrating its superior performance gains over traditional multi-round self-distillation, especially in high noise rate regimes. This research directly addresses the prevalent challenges in handling real-world data, where acquiring clean and accurately labeled datasets is often difficult and costly.
With the recent breakthroughs in large-scale pre-trained neural networks or foundation models, various studies have been conducted to effectively utilize well-trained large neural networks as feature extractor backbones. We believe that our result can be applied to such scenarios and theoretically demonstrate the performance gains achievable by self-distillation through a simple linear probing within a pre-trained neural network.
By enhancing our understanding of knowledge distillation and self-distillation, we also provide the way for the development of more efficient models, capable of matching the performance of their larger and more complex counterparts, which is especially beneficial scenarios with limited resources.
B LIMITATIONS AND FUTURE WORKS
In this work, we theoretically analyzed the effect of multi-round self-distillation and self-distillation with refined outputs by deriving closed-form solutions for the outputs of the student model as the number of distillation rounds increases (Sec. (3) The main technical challenge in this analysis lies in the non-linearity of softmax functions. We addressed this by using linear approximation of the softmax function, similar to the original work by |Hintnn et al. (2013), as detailed in Assumption [ We justified this assumption within proper parameter regimes through numerical experiments in Sec. D.I Generalizing our results to fully incorporate the non- non-linear nature of the softmax function remains a future work.
After deriving closed-form solutions for the outputs of the distilled models in Theorem 3.1 in terms of the general Gram matrix 4, representing instance-wisg feature correlations, we considered particular low-rank feature correlation map (7) to conduct further analysis on the student models' outputs. The practicality of this feature correlation model validated using six different real datasets, as detailed in Sec. D.2 demonstrating its applicability in real-world model training. We also provided additional analysis for four additional feature correlation maps in Sec. J including a more generalized version of the superclass feature correlation model. Relaxing the assumption on a class-wis feature correlation, for example by generalizing it to
(((;)),;, ((xx) (X,, vU) (a, w),
remains future work.
Lastly, our main analysis and experimental results show the gain self-distillation training a linear classifier on top of a fixed feature extractor. Our results can also be combined with the feature learning perspective of self-distillation by considering the feature map evolving over the distillation rounds, illustrated in Corollary C.I Understanding the role of self-distillation in more general setups, including distilling foundational models with more diverse applications, such as natural language processing, is an important direction for future work.
13

---

Published as a conference paper at ICLR 2025
C SELF-DISTILLATION WITH FEATURE LEARNING
We next provide an extended version of our analysis for the full-to-full self-distillation scenarios, where not only the last linear layer but also the feature extractor can be updated through multi-round self-distillations. To illustrate this point, consider the quantified gains of self-distillation in label-noise scenarios presented in Theorem [..] This theorem provides the sufficient number of distillation rounds required for the student's s softmax output to assign the highest value to the ground-truth label for both clean and noisy samples. The condition depends not only on the class-wiss wise label corruption rates but also on the relative gap in feature correlations between samples of the same ground-truth class, parameterized by c, and those of different ground-trure classes, parameterized by in the Gram matrix model + (7).
We can generalize our theory by allowing these correlations to evolve over distillation rounds, denoted by c(9 and d(s), due feature updates during celf-distillation. Recent work by Allen -hu & Li (2022) demonstrates that and student models, enabling the student to learn more diverse features when using the teacher's softmax outputs as targets. This suggests that the intra-class feature correlation c( may increase with each distillation round i, enhancing the separation between classes
's effectiveness arises from an implicit ensemble of teacher
Assuming the class-wise feature map defined in (7) and allowing c and change over distillation steps, our parameters p and in 14), which govern the label averaging effect, also become functions of é:
(1)
p = K2nd + c6)
(17)
( + n(((() (())
( K2nd + 1 c0) + n(c(() - d())
can be generalized Under the extended class-wise feature correlation assumption, our Theorem 4.1 to:
Corollary C.1 (Extended version). Under the evolving feature correlation model, the t-th distilled model achieves 100% population accuracy if
[C],,& [C]R, + (p000)
If the student model learns more diverse features than the teacher model, leading to increased intra-class feature correlation cl as i increases, then /p)) also increases. This means that the regime for a student model to achieve 100% population accuracy expands, ultimately leading to performance gains from self-distillation. 'huu, our result can be integrated with the aspect of Allen-Zhu Li (2022), where the gain from self-distillation is explained by diverse features learned by the student model. Specifically, Corollary [.I quantifies the gains both from the feature learning and label averaging effects of self-distillation under label noise scenarios. The proof of Corollary C.I
can be found in Appendix K.3
D JUSTIFICATION OF ASSUMPTIONS
In this section, provide empirical justifications for the assumptions we have imposed in the theoretical analysis.
D.I JUSTIFICATION OF SOFTMAX APPROXIMATION (ASSUMPTION I
In Assumption 1 approximate the softmax function (vv) the linear function 1/K v/K, assuming a small enough magnitude of v E R To justify our approximation, let us recall the equation for y
Kn
Kn (0(x;), o(x;)) (x - -x,0) (18)
(1) y =
Σ (x(x)) o(x)))aa))
Knd
14

---

Published as a conference paper at ICLR 2025
We numerically validate Assumption by analyzing the softmax approximation error. The softmax approximation error is defined as the maximum difference - between the softmax outputs derived in Lemma 4.1 based the linear approximation of the softmax, and the outputs obtained by
numerically solving (8) for {y: (() Ji=1- (see Appendix E.4 for details.)
We set K = 4, À 10-1 and the Gram matrix + follows a block-wise structure as in with parameters c = 0.4 and = 0.1. Under the symmetric label corruption with rate = 0.5, we analyze the softmax error the number of samples per class n increases. As shown in Fig. 6 the softmax error becomes negligible, especially as the number of samples exceeds 10*.
0.035
softmax error
0.030
0.025
0.020
0.015
0.005
0.000
10a
10*
10a
# of samples
Figure 6: We plot the softmax approximation error as the number samples increases. We use
K 4. x 10 0.4, d 0.1, and impose the symmetric label corruption with rate 0.5.
D.2 JUSTIFICATION OF CLASS-WISE FEATURE CORRELATION MAP
In we argue that the feature correlation matrix exhibits block diagonal structure, i.c.,
1
(d(x)),(xx))
C,
yi = y,. =
(19)
d, ys + リı h(y:) = h(y)):
0. h(y:) + h(U,).
We empirically demonstrate the validity of this claim diverse real-world datasets (with dataset details in Appendix [ET)). We first conduct hierarchical clustering to rearrange the order of classes so that the highly-correlated classes become adjacent. The results of hierarchical clustering
presented in Appendix L.2 Under the rearranged class orders, the Gram matrices (input feature correlation matrices) have block-diagonal matrix structure for the real datasets, shown in Fig. 7
Additionally, we assume that feature correlation between any two instances is determined by their corresponding true labels. To verify this assumption, we investigate the mean and standard deviation values of the feature correlation for all pairs of instances, for three cases as follows: 1) = Mj 2) Y: # yj, h(y; = h(y;), ), and 3) h(y) # h(y)). We calculate the mean and standard deviation for all sample pairs corresponding to each case. As shown in Table we find that there exists a gap in the mean feature correlations between the three cases. This empirical evidence supports the validity of our class-wise feature map, modeled as 77
Table 2: Statistics for three different cases of instance-wise feature correlation.
Dataset
# of superclass
y/ y/
y: yj h(x:) kws))
h(w))) # h(u))
CIFAR-100
0.25++0.16
0.02+10.12
-0.03++0.10
Caltech- 101
4
0.42+0.15
0.03++0.88
-0.02++0.07
Caltech-256
0.35-10.16
0.0210008
-0.03+00.07
Plowers-102
2
0.36:10.15
0.031:0.1
-0.0440.13
Food-101
0.17+00.15
0.01+0.12
-0.02+t.112
StanfordCars
2
0.21+0.16
0.06+0.14
-0.07+0.13
15

---

Published as a conference paper at ICLR 2025
CIFAR-100
Caltech-101
Caltech-256
0.4
0.3
Flowers-102
Food-101
StanfordCars
Figure 7: Class-wise feature correlation map six different real-world datasets. The features are extracted by pre-trained ResNet34. Block diagonal patterns are exhibited in all the six datasets, as assumed our feature correlation map (7).
D.3 EFFECT OF TEMPERATURE SCALING
Let us define the softmax 0² RK (0, 1)K with temperature scaling parameter o=(v) exp(ex /T) for the logits [e....., ux] RKK We set and denote of (+)) o-) in the main document. In this section, show that the temperature scaling of the softmax, combined with the proper scaling of the cross-entropy (CE) loss function in the
exp(v:/r) /T)
E. exp(v./r)" .... EE, exp(e./7)
regularized CE loss [ results in the same optimal solution for 0 as in 1. Let y
(t) represent the
output of the t-th student model for the i-th sample. The objective function for the f-th student model (O), scaled by temperature is given by
72
CE 11-1) ( ((x:))) + 1|101
(0)
(20)
Kn
fort E N. Then, the optimal 0, denoted 0* satisfies
Kn
Kn
T-
y (t-1)
y! ((xi) = Σ ((x:)
(21)
Knd
Substituting the optimal 0" yields
y o- (0 ((xx))
Kn
(o(x;)) ((x)))))))))
Kn
(o(x)), x(x;)) (y -x("") 25/))
0
Knd
16

---

Published as a conference paper at ICLR 2025
This implies that even with temperature scaling with 7, the equation of the optimal prediction y; remains unchanged for f (0). In other words, temperature scaling does not affect the optimal prediction.
E EXPERIMENTAL DETAILS
In this section, we provide the details the experiments presented in Sec. 6 Our code is publicly available at
E. IMAGE CLASSIFICATION DATASETS
To validate our theoretical findings, we conduct experiments on several real-world datasets using the PyTorch torchvision library. Below are the details of the real-world datasets used.
C-FAR-100 (Krizhevsky et al. 2009): CIFAR-100 consists of 60,000 32x32 images in 100 classes. This dataset covers a wide range object categories such as animals, vehicles, and household items, with each class having 600 images.
Caltech-101 (Griffin al. 2007): Caltech- 101 contains images from 101 object categories with background category named "BACKGROUND "ooogle" For each category, there are about 40 to 800 samples, with average of 50 samples per category. We removed the background category and divided the dataset into training and validation sets using an 8:2 ratio.
Caltech-256 (Griffin al., 2007): Caltech-255 256 is large-scale dataset consisting of 30,607 images across 256 object categories. Similar to the Caltech- 101 dataset, we removed the "clutter" category and divided the dataset into training and validation sets using 8:2 ratio.
Flowers-102 NNilsback Zisserman, 2008): The Flowers- -0ce dataset contains 102 different categories of flowers, with total of 8, 8,189 images. Each class consists of varying numbers of images, ranging from 40 to 258 images per class. Since the test set of the Flowers- 102 dataset is larger than the training set, swapped the training and test sets for use.
Food- 101 (Bossard et al. 2014): Food-10 101 is large-scale dataset specifically designed for food recognition tasks. It contains 101,000 images of food items categorized into 101 classes, with each class consisting of 1,000 images.
StanfordCars (Krause et al. 2013): The StanfordCars dataset is focused on car recognition and consists of 16, 18,185 images 196 classes of cars. The dataset includes images of cars taken from various viewpoints and under different lighting conditions.
E.2 CHOOSING AN APPROPRIATE HYPERPARAMETER
In real experiments, we extract the features of input instances the diverse datasets using the pre-trained ResNet34 model. Recall the definition of p and q from
1-c
p K2n
(22)
1 1-c+ d)
1 = K2nd +1 - n(c-d)
(23)
In this setting, we visualize the variation of the q/p ratio as À changes across the six real datasets to find an appropriate considering, the intra-class feature correlation and the inter-class feature correlation d from T When the g ratio is too small too large, becomes hard to demonstrate the effect distillation from Thm. 4.1
In Fig. 8 observe that achieve q/p ratio close to 2, À should be selected around 10 for Caltech- 101, Flowers- 102, and StanfordCars, while around 10 44 for CIFAR-100, Caltech-256, and Food-101. Therefore, in the main text, set À to 5 10-4 for Caltech-100 101, Flowers- 102, and StanfordCars, and x 10 for CIFAR- 100, Caltech-256. and Food-101 to illustrate the impact of distillation. Additional experimental results for various values of are presented in Appendix L.]
17

---

Published as a conference paper at ICLR 2025
G. 2.5
2.5
2
1.5
Flowers-102
Caltech-256
Stanford-Cars
Food-101
1e-4
2e-4
5e-4
le-3
1e-5
2e-5
5e-5
1e-4
A
Figure 8: Variation of q/p with changing weight decay A.
E.3 TRAINING
We train two-layer network consisting of a linear layer and softmax layer. Our loss function is defined as the cross-entropy between the softmax output of the teacher model and the soft- max output of the student model. mitigate uncertainty, we conduct three repeated runs for all experiments with seeds 0, and 2. We perform grid search over learning rates, in the set {0.1, 0.05, 0.01, 0.005, 0.001). Each model is trained for 200 epochs, employing the SGD optimizer with momentum value of 0.9. In experiments, student model often leads to instability during training. The PLL student model trains with a set of candidate labels for each sample with equal weights-in case, the top two labels with weights of 1/2 each. Using CE loss with equally weighted candidate labels can cause instability since the model may converge incorrectly when the candidate
observe that using CE loss with the PLL
includes incorrect labels. Hence, for the stable convergence of the PLL student model, we use Generalized Cross Entropy(GCE) (Zhang & Sabuncul 2018) loss with the hyperparameter 0.7. Our neural networks are trained using multiple NVIDIA RTX A6000 GPUs.
E,4 NUMERICAL METHOD FOR CALCULATING THE OUTPUTS OF STUDENT MODELS
For synthetic experiments in Fig. 2 using numerical methods, instead of applying the approximation in Assumption Recall from (3) the equation for g(t):
we compute the outputs of multi-round self self-distillation models
Kn yi
(1-1)
(1) y
( T
o(xx))
Knd
The optimal output of the f-th disitlled model {y; (1)
Ji=i satisfies the following equation:
Kn (((x,),(xx)) (y---)) y!1))
y
(24)
Knd
Therefore, given the feature correlations and the output of the (t 1)-th distilled student model ( Kn. be found by
(xx(--)) J= the optimal output for the t-th distilled student model (v(") (t)
J==1
minimizing the difference between the left-hand side and right- hand side of (24). The corresponding
pseudo-code for this process is outlined below:
18

---

Published as a conference paper at ICLR 2025
Algorithm 1: Optimal Output Calculation by Numerical Method
Input: K: number of classes, n: number samples per class, A: weight-decay, 7: total distillation steps, fy: learning rate, convergence tolerance
Input: Given label (target) Y [yl (0)
...... .YK. E ER Kn
(0)
Output: Output of t-th distilled model Y = [y] 3..1.0
y (t)
Kn E RRKKKn for t = :....? T
for f 1,2,..., do
Step Initialize Y for each instance = ......., Kn do
ΣKn uk (0, 1);
while iter < max_iter do Step 2: Compute Y for each instance i = 1,2, ..... Kn do
8/
Σ Kn (0(x,), (xx)))
(y) (t-1)
-y,00 r)
Knd
Step 3: Compute Loss L; ((Y | (
10
Step 4: Update Y Y ( VV y ((Y(()
11
Step 5: Check for Convergence;
if (() < then
15
Break;
19

---

Published as a conference paper at ICLR 2025
F FULL EXPERIMENTAL RESULTS ON THREE DIFFERENT LABEL CORRUPTION SCENARIOS
For Theorems 4. and 5.1 the main text, we formulate the gain of self-distillatio using the label corruption matrix C without making any specific assumptions about the label corruption. To validate our theoretical results, we conduct experiments with various types of label corruption. Fig. shows the accuracy gain of the student models for six different datasets under the three corruption scenarios, described Table [ The precise accuracies of each distillation models can be found in Appendix M Overall, we observe that there is gain as the distillation step & increases in multi-round distillation, and we confirm that the PLL student model outperforms other multi-round distillation models. This results validate our theoretical finding in Sec. 2.c..aand and 5 where show that self-distillation plays role as label averaging, facilitating the correction of label noise.
Caltech-101
Caltech-256
CIFAR-100
20
Die Eation Carn
2010
114
Coruption Rati [1)]
Corruption a
Complian Saria g1al
Flowers-102
Food-101
StanfordCars
bosn
3016
12.5
bis
Comsuption Ratio
Comption forts g.1
(a) Superclass label corruption (Full version of Fig.
Caltech-101
Caltech-256
CIFAR-100
Deti Belitatton Cain (50
Diejtaton
20
Comppton Rate 1941
Comuation Rate (1))
Compation 294
Flowers-102
Food-101
StanfordCars
3016
Dittatoon Car
mettarmn oan
Compption Ratto
Comuation Ratio (y
Compptio
(b) Symmetric label corruption
20

---

Published as a conference paper at ICLR 2025
20
Diettation Oan
F6L
PAL
50
-30
50
90
2
76
Flowers-102
Food-101
StanfordCars
300
Die Dietatioo can
6-4
Compationn Sarto (.66
Corrupt on Radio n (94)
Corruption Ratto 2944
(c) Asymmetric label corruption
Figure 9: Distillation gain in accuracy (%) by each student model compared to the teacher model in three different label corruption scenarios. PLAIN represents the gap between the multi-roum SD students and the teacher model, while ORANGE represents the gap between the PLL students and the teacher model. We use weight decay value À x 10 10-4 for Caltech- 101, Flowers- 102, and StanfordCars datasets, and À 10 for CIFAR-100, Caltech-256, and Food-101 datasets.
G FULL OUTPUT EVOLUTION OF FIG. 2B AND FIG. 5A
In Sec. 3] our analysis reveals that multi-round self-distillation effectively performs label averaging among instances with high feature correlations. This process leads to clustered predictions and improved generalization. To verify our claim, conduct experiments on both synthetic and real data and illustrate the clustering effect of softmax outputs of the student models as the number of distillation rounds increases. Fig. I0 presents the full version of Fig. 2bb depicting the output evolution for K class classification task for synthetic dataset. Each dotted circle represents cluster containing all instances true label. As the number of distillation round progresses, the student model's 's outputs converge unique clustering points, depending on its true label.
GCAR
5132 Bose!
12112
C355
Ciss
Cass
05552
Coss
Clas2
Class
Cuss
Class
Class
Class
Class
Class
aacko mefaer
Student
Student
7th Student
Student
9th Student
Olsss
Class
Cass
Clsss
Clsss
Cass
Clsss 1
Cleas 2
class
CIsss 2
Class
Class
Cass
Class 4
Clsss
Clss
Class
Class
Class
Class
natcu mal.
Figure 10: (Full version of Fig. 2b) The output evolution of the teacher model and the multi-round self-distilled student model. Each corner represents the one-hot vectors of the respective class. We 100 and À 3.125 x 10-4, with 50% symmetric label corruption. The Gram matrix is assumed to follow a block-wir wise structure as described in (7). with parameters c = 0.4 and = 0.1, added random perturbation matrix E, where each entry [ Eli . (7-0055. 0.05).
21

---

Published as a conference paper at ICLR 2025
E11 Buee
Claas
C
Geay
15.00
Figure 11: (Full version of Fig- Sap The output evolution of the teacher model and the multi-roun self-distilled student models for the 4-class CIFAR- 1-00 dataset. Each corner represents the one-hot vectors of the respective class. We À 2.5 10-1, with 50% symmetric label corruption.
Fig. I is the full version of Fig. Sa For the CIFAR-100 dataset, we arbitrarily select 4 classes and use only the instances belonging to these 4 classes, with 500 samples per class. As the distillation step t progresses, the outputs form clusters due to the label averaging effect.
H ABLATION STUDY
H.1 TARGETING TOP-K PARTIAL LABELS IN PLL STUDENT MODEL
In Sec. [] we theoretically demonstrate the effectiveness of our novel student model, the PLL student model. By selecting the top two label positions from teacher model's output, we ensure inclusion of the ground-truth label in the partial label list. While increasing the number of candidate labels may improve the likelihood of including the true label, it can also reduce confidence in the correct label for both noisy and clean-label samples, potentially leading to performance degradation. We conduct additional ablation studies on the sizes of candidate sets to validate the appropriateness of using only the top-two labels, shown in Fig. 12 We observe that the model targeting the top-two vectors achieves the best trade-offss between accurately including the correct label in the partial label list while maintaining high enough confidence for the correct label within the list, resulting in the best test accuracies across six benchmark datasets.
Caitech-101
Caltech-256
CIFAR-100
topil
Top
Destaatin Dain (96)
F4a-5
Destudon Can
tee-5
Eop
Corruption Radio y (50)
Comusttin Ratio (90)
Comruption Ratto o/hi
Dien wabnn Ownn 180
top-
50
TO
Figure 12: Full version of 5b Distillation gain in accuracy (%) by each top-k PLL student model compared to the teacher model in the superclass label corruption scenario. ORANGE represents the improvement by our proposed top-2 PLL student model compared to the teacher model, while PLAIN represents the gap between the top-k k PLL students and the teacher model for >
22

---

Published as a conference paper at ICLR 2025
H.2 EMPLOYING LARGER FEATURE EXTRACTOR BACKBONE
We also conduct additional experiments using a larger backbone-a pretrained Vision Transformer (Vit-B) (Dosovitskiy al. 2021) model-as fixed feature extractor. The distillation gains in test accuracy (%) over six benchmark datasets across different label corruption rates are presented in
Fig. 13
Caltech-256
alto Can
Cua att cal
26
6
90
2
54
Flowers-102
Food-101
StanfordCars
301
12.5
Didk Lobnn Gann
Corsuption Betlo (1)
Corruptin Rat o (H))
Compaton Karis g16)
Figure 13: Distillation gain in accuracy (%) by each student model compared to the teacher model in three different label corrpption scenarios, using ViT-B fixed feature extractor. PLAIN represents the gap between the multi-round SD students and the teacher model, while ORANGE represents the gap between the PLL students and the teacher model. We use weight decay value Caltech- 101, Flowers- and StanfordCars datasets, and À = 5x 10-5 for CIFAR-100, Caltech-256, and Food-101 datasets.
x 10-4 for
Using larger backbone like ViT-B enhances feature extraction capabilities, resulting in greater disparity between intra-class and inter-class feature correlations. For example, when calculating feature correlations the CIFAR- 100 dataset using the pretrained VTT-B model, observe that the average intra-class feature correlation increases to 0.35, compared to 0.25 with ResNet34, This higher intra-class correlation amplifies the clustering effect of self-distillation model predictions, allowing significant performance improvements to be achieved with fewer distillation steps, as implied by our Theorem 4.
In experiments with the larger backbone (ViT-B), we find that most of the distillation gains occur within the first few rounds. As shown in Fig. 13 in the first and second distillation steps when using ViT-B, although additional distillation steps still bring slight gains in high noise rate regimes. The PLL student model also effectively achieves the gains of multi-round self-distillation in a single round in high label-noise regimes for the ViT-B backbone. These additional experiments confirm that our approach effective with larger models, further validating the versatility and robustness of our method.
nearly all performance improvements observed
23

---

Published as a conference paper at ICLR 2025
1 PROOF OF THEOREM 1.1
Recall ( the equation for the coefficients {a; (8) Kn.
Kn (xxx)) ((x))) ,(xx))))
(t-1) Knda (9) = y,(
(25)
By applying the softmax approximation (Assumption D.. we have
Kn
12nda (0) Ky!
((xx))
1x.
(26)
Define the output matrix Y() E TR. Kx and the coefficient matrix A (0) E RKKKn as:
Y( = [y]
A (1) = [a knl
(27)
Knd
We also define the Gram matrix + E KanKn, where each entry is given by
[ = (o(x:), ((x;)), Vi,j E [Kn].
(28)
Denoting the i-th column of a matrix A by A A t]. 26
can be written as
Kn
K [ KY 1)
[3 Σ
D A (4) [. 1x
j=1
= -
(A (0) + [b. 1 1xxKnl..
Since 4 is symmetric, we have
+ ② (Y 1KKKn.
Also, from the definition of A (1), we obtain the relation between Y and (r- as follows:
Knd (((-1) - - K (Yil-))
(r(t-1)-(())
K 1kxkn) (K²ndI + $) $)-;;
1 (Ixn+ +
(Y(L-1) (r(t-1)-x((0) -x(0) (x((-1)- 1 K (xxXss))
$
K2nd
(y(t-1)
1 K (k*kk) - (x(0)= -(x00 - 1kxkaa
(Yie--)- K ZlaxKn]
-1
IKn +
K2nX
1 (Y R (1XXK) (r(r-1)-
1 [IXXKn)
IKn
IKn + K2nd
-17
K
From this recursive relation, express (1) in terms of (0) and + as:
K IKXKn = (x10) 1 Y
[ IKn - IKnn +
Kand $
(29)
K
Similarly, for the outputs of the student model with partial labels, the only difference is that (0) is replaced by Y:
(P)
K 1KXKn = ( K
1 lkxx)))
[Ian (Ina + K2nd
+
-1
(30)
where (P) is the output of the student model with partial labels using the input Y
",,,...
SKa).
24

---

Published as a conference paper at ICLR 2025
J PROOF OF LEMMA 4.1 AND THE CLOSED-FORM OUTPUT SOLUTIONS FOR FIVE DIFFERENT FEATURE CORRELATIONS
A
Recall the closed-form solution for Y from 6
YO)) 1xKK= = (Y()) 1KKKn K
Kn Σ
K² + X ) v.v.
(31)
K
where {(A,, v represents the eigervelue-cigenvvetoo pairs of the Gram matrix 4, satisfying + = Ei= Ava. In this section, we compute {(),, vi) for diverse class-wise feature correlation scenarios and simplify the closed-fome solution for Y
Assume the feature correlation between two instances (é(x:): ((x)) is determined by their true labels:
J i + j. (o(x)))((xx)) (w(y). vj),
f1.
(32)
Define the class-wise feature correlation matrix W E BKXk with entries
[Wk, w(i,)), Vi.j E [K]-
(33)
Without loss of generality, we assume that the dataset is sorted by true labels y,. Then, the Gram matrix o can be written as
P WP + +(L [W] (W).E.P))
(34)
block matrix with n. blocks
diagonal matrix
where (zz) is diagonal matrix with elements from the vector z, and P
R KKxKn is defined as:
0..
0 0...0 0
0...0 0 1...
P (e(y)) ... e(yk.))
(35)
0 0
In the following sub-sections, we derive closed-fore solutions for the model outputs c for five different class-wise feature correlations W.
J.1 CASE I: POSITIVE CORRELATIONS ONLY AMONG SAME SAME-CLASS INSTANCES
Assume that the wiss feature correlation w(ys Mi) is defined as follows:
(yt Mj)
C,
yi = yj y + Øj
(36)
0.
for 0. From 34 the Gram matrix & be computed
Φ = PT WP + D(L [W] [ P)
=PT(ck P + D - || , P)
nc
P + (1 L)
Let us represent P in 35 by P = (p....., Pxl
for p; E II Kn, can be shown that & has two
Kn. types of eigenvalue-eigennector pairs ();, 1=
25

---

Published as a conference paper at ICLR 2025
K pairs with eigenvalues
= = nc --,, C, for = I,..,,,K. K,
and the corresponding eigenvectors vn Pt
(Kn K) pairs with the eigenvalues c) c) and the corresponding eigenvectors, denoted by for i = K + 1,...,K. Kn.
Note that Link K+1 V.v. the sum of outer-product of (Kn - K) eigenvectors satisfies
v.v. IKn
Σ i=1
v.v.
Ikn 'PPP.
(37)
i=++
Hence, the label averaging matrix (t) given as
A
(1t)
V.V.
K2nd
X
- c ne
P
K2n K2nx+1-c) )(1
1-c
1 P
K2n1 +
For simplicity, we define constants p and q:
1-c
1 1-c+ q= -
p Kan\+1-e
(38)
The label averaging matrix T (1) can be expressed as
(d'--')PTP +p'IKn.
(39)
class-wise label avernging matrix
Then, the closed-form of ( is
Y
(0)
1xKK
Φ11) + K
1KXKn
K
(H - - + +'IIkn)
(0)
1xKKn
+ IxxKn
K
K
1 (q' p¹) p00pp P 1 x1 1l)kkk
(
where the last equality holds due 1x P P kxk=
(!)
Finally, we arrive at the closed form of y:
p'e(ū) + -(g' - p')
Σ
1 e(ú,) + 7 g')kk.
(40)
J.2 CASE II: CLASS-DEPENDEN INTRA-CLASS CORRELATIONS
Let the feature correlation between two samples with the same true label change depending on the class. We model the feature correlation between instances as function of their true labels:
(Wa)), Yi = .
wii, 93)
(41)
0. VE + yj-
Here, we assume w(y:) for all y: [K]. The Gram matrix & can be computed
T P WP D(1 W
p (I.
+ +D[((1 (())){
K
(1
T
= > ni)
Vn Pi
(n
+ D([[1 D[[[1-Wl) (()) P).
It can be checked that + has types of eigenvalue-eigennector pairs:
26

---

Published as a conference paper at ICLR 2025
K pairs with eigenvalues
X; = mi) + - w(i),
i= 1...., K,
and the corresponding eigenvectors V; Vn Pii
(Kn - K) pairs with eigenvalues
[ Akn]
w (1),,...... w(1)
w ((K), w())..., ww w()
(42)
(n-1) eigenvalues
(n- 1) eigenvalues
and the corresponding eigenvectors v; for i - K + 1...., Kn.
Hence, the label averaging matrix (()) is given as
Kn
K2nd Ao ) viV.
X - w(i) + mw(i)
$
(+)
i=1
1
K
1 - wii
Σ VJV]
P. P
K2nd + (i) + (i)
K2nd w(1)
(1d)=1-w(i)
Note that
Σ VoV =
O, ,,..,,,,,,,. 1,....,00)
p p P
(43)
(1))=
1-m)
(K-i)n
For simplicity, let define two vectors p 1 wii)
[11 Pl...., PK]
q = [g [q ......., qK] E
(i) + nw (i)
P: K2n
9i
K2nd + w(i) + mi)
(44)
Then, (() can be simplified as
K
K
(0)
(af
plpp pa + Σ p D D(........ -...,,,,,,.,,
n
(1-1)m
(K-ijn
P P')P +D(p'P).
class-wise lobel averaging matrix
Therefore, arrive the closed form for y!!
yi (t)
(0)
1
1K
$10 [1,i]
+
1
K
K
Ply e(ūi) + n
1
( - pby)
Σ
e(ū,) + K (1 -gy) 1x.
(45)
(:
J.3 CASE III: CORRELATION GAP BETWEEN INTRA- AND INTER-CLASS INSTANCES
Let be the feature correlation between samples with the same true label, and let d be the correlation between two samples with different true labels. The class-wise feature correlation wyy 8j) is defined follows:
(it 81)
C,
y yj
(46)
d,
3: + Vj.
for 2 0. The Gram matrix can be computed as
+ P WP D(11-((1[[],) w
1 P)
= P T [d1 + (c d) P + (1 c)
Note that W = dl + dI- is a type of diagonal-shift to low-rank matrix, which has two types of eigenvalue-eigennecoor pairs {a,, u; K
27

---

Published as a conference paper at ICLR 2025
pair with eigenvalue
01 Kd + (c- d),
and the corresponding eigenvector i
VK Ixi
(K - 1) pairs with eigenvalues
o, = (c- d), i 2,...,K.
and the corresponding eigenvectors u, Then, the Gram matrix 4 can be expressed as
1 _PTT, Vn
( u. (1 -c]
Φ = P P (1- k) E no;
u,
can be checked that has three types of eigenvalue pairs:
pair with eigenvalue
\ = noi + + (1-c) c) = Knd + nncc - d) + (1 cc., c),
and the corresponding eigenvector
V1
P Pu
u1
p Plk:
vn
Kn
(K - 1) pairs with eigenvalues
X; = no, + (1 = n(c - d) + (1 (1-c), = K,
and the corresponding eigenvectors
V;
P puu; Vn
u1
i= 2,..., K;
(Kn K) pairs with eigenvalues
A (1-c),, i= K + 1,.... Kn,
and the corresponding eigenvectors v;
Define p. and r according to the following equations:
1
p K2nd+1-
1 c+ n(c- d)
q K2nd C. + n((c d)'
K2nd 1 n d) + Knd
n(c - d) + Knd
Then, the label averaging matrix (1)) is given
KR
A
$
10
Σ
V.V.
K2nA + Xo
1=1
P
PTuu
9¹
P
u P
P
Σ V.V.
n
n
K+1
P
1 1 I P
LpT
In
- 1
P p-pp p²
IKn
1
- p
k
u
P
Kn
K
iml
r'-9'
q¹ d'-p'p
1Ka
+p'Ina-
Kn
class- wine label averaging matrix
28

---

Published as a conference paper at ICLR 2025
Finally, we obtain the closed form for (0)
yi
y/ Y K
1xxKn
$ [. +
1
1K
K
=p'c(0:) (a' -p²)
Σ c(ts)
( ( 9²)
Kn -d(s)
c(ū))
+ (11)))
1K
():
= =p'e(ū)) + (a' -p')
Σ
e(ū;))
+1 9¹) 1 TO) L
(47)
The last equality holds due to the balanced given labels, (úi). across the K classes. Note that the closed form for y; consists of the following three components, across all the three considered so far:
(t)
e(ji): the given label,
Eiiws e(ū;): the average label vector among samples from the same true class Yi-
Ix: the uniform vector.
K
J.4 CASE IV: SUPERCLASS FEATURE CORRELATION (PROOF OF LEMMA 4.1
Let us assume that there exists set of superclasses, where the classes within the same superclass have higher feature correlation, while the classes from different superclasses have zero feature correlation. Let h [K] [[]] be function that map the class index to superclass index. The model for the superclass feature correlation is defined
C,
y1 = y;;
w(y 81)
d, hy))
(48)
0. h(,) # h(y,)
for d O. Without loss of generality, assume that the labels are sorted according to the superclass index, which implies that h is a non-decreasing function. Then, the class-wise feature correlation matrix W R can be expressed as the of rank- R block diagonal matrix and diagonal matrix:
W
dQ QQQQa + (c dd)
(49)
block dingonal matrix
diagonal matrix
where Q E IR Rx K are defined as
1...
0...0
0 ....
0...0 0
Q
=
(50)
K2
Kn
where Σ K, K. with K, := Kk E [K] l(kk) = i for E [R],
Let us represent Q in 50 as Q = [q......
qR] T for qi E R Then, W can be written as
R
W
K2d
qi
(T,.9 qi
(c d)IK.
VK;
It be shown that W has types of eigenvalue-eigennecoo pairs (0;, u, K
29

---

Published as a conference paper at ICLR 2025
R pairs with eigenvalues
o, o K,d + (c d),
i= i=.,...,
and the corresponding eigenvectors u, vk. 9:-
(K R) pairs with the eigenvalues
o, (c d),
i= R i=R+h..., K,
and corresponding eigenvectors u, Then, o can be expressed as follows:
K
T 1 (12 +11 I)
Φ = P WP + (1 C) = E no;
Thus, it can be shown that & has three types of eigenvalue-eigenvector pairs:
pairs with eigenvalues
A no; + (1 c) = K,nd + n(c- d) + (1 - c)2 i= 1=1..... R.
and the corresponding eigenvectors
V,
P vn
u1
Knn qi
(K- - R) pairs with eigenvalues
X no; (1 - = n(c - d) + (1 c)2 i= R K,
and the corresponding eigenvectors
P vn
Vi
(Kn - K) pairs of eigenvalues
A (1-c), i= K + 1,..., Kn,
with eigenvectors Vi.
Define p. q and r [rI,...,TRI
E R as follows:
1-c
K²nA+1-c
1- c n(c -d) d)
q K2n + c nncc d)
1 c + nlc - d) + K,nd
== K2n + - e + n(c d) K i i [R].
Then, the label averaging matrix ( is given
Kn.
A
(1)
V.V.
K2nd
X
Kn
TTTu u
P
d'
Σ
p
P
Σ v.v.
n
i=R+1
i=K+1
R
d'd T
R
1 K
uuu
uuu
P
Ig
Σ nuu
P P+p*
Inn
Σ
p u
P
T=1
t=1
i=1
q¹
r² p'
p
P
p PTP 1'p+p'In I'Inn
K2n
n
Tp: rr--9"
o' d-
P+
Kin
class-wise label averaging matrix
superclass wise label averaging mutris
30

---

Published as a conference paper at ICLR 2025
Finally, we arrive at the closed form of !!??
p(فا)
y p'e((i)+ p²)
e(ūr) +(1--g)
E elUs)
+(1-r5) 1k
Kan (jth(yz)=s)
K
(ク:
(51)
where 8 h(vs) is the superclass of ys- The closed form for y! consists of the following four components:
e(ii): the given label,
a E e(ū;): the average label vector among samples from the same true class yi+
Knn E(j:(on)=s)] e(ú,):: the average label vector among samples from the same superclass class
h(y)),
K I:: the uniform vector.
J.5 CASE V: EXTENDED SUPERCLASS FEATURE CORRELATION
As the last case, we allow non-zero inter-superclass correlation, fixed as a constant e, The model for the class- feature correlation is defined
C,
y/4 = yj
w(yi, Mi) d,
+ yソ. h(,,)(()+)))))))))))))) h(v));
(52)
h(y.) # h(y,)
for d O. Under this model, the class-wise feature correlation matrix W is expressed as:
W == (d e)Q Q + (c (c-d)[K
(53)
and the Gram matrix 4 is given by:
K P) P WP + (1 e) t- = P WP D([1 [W
Then, the label averaging matrix of the p(t) can be formulated as:
1
(!) = IKn
+ [IKn
$
KZnd
KKnx+11
IKn
Inn
P WP
Kand
K²nA
Since PP" nIKn the following matrix inversion holds for any U RR Kxk and R:
UI + P UP)
IKn + PT
(oI ( P.
Thus, we have
K2 K2nd+1-
x2n
K2nd
Inn + K2nd
PTWP)
= K2nd +
I Inn
PT
V
P
K2nd
K2n 12nA+1-2
where V V is given by:
K2nt+1-
W
V
IK + K²X
K2nd
Substituting back into (t),, we get:
((1 K2 + Inn PT
Kand
K2nd Knn+1-c I- I- V-1 P
T (+)
CLAIM: For any 2 1, U U kxk and v E ER
1p UP
+ uI kk
L
(U - I -'I
(54)
n
31

---

Published as a conference paper at ICLR 2025
Proof. Since PPT nlkns have:
1-
P UP + vInn
= I'Ian" +
1
n¹
PPPUPP
1
i
n
1=1
=''
+
1 UC-ipp
1=1
= -'' +
P
v Ut-i P
-=
I'IKn +
SPT((U + vIn) - o'Ix|P.
n
From 54
we can express (+)) as
(1)
K2nA =1-c Inn + n
K²nd
_ (I (1-V
(11--11111 (1-
K²nd
Ix
K2nd +
(55)
The main problem is to calculate the inverse V, which can written as
K2nd c n(c d) IK + K2Y
1xxk + d-c K2λ QTQ
(56)
K2nd
By simple matrix calculations, have
RxxIRxt = K1axRi
1 Q Q Q ((K)))
Q QQQ Q Q D ((K))))
Hence, the following matrix inversion holds true for the corresponding a > 0, B and y of (56)
(aIx + 8lkk QQ (==Q)-==((
- -
B
Kxk Q DTD ((
ala + Kg)
where 6; is given as
7 Kg a
6, a + K,3 - Kiy
a(a + KB)
Define p. q, 8 and (ri....,rni rn) E R according to the following equations:
1 c p K2nA 1
1 - c (c d)
9 KPnd + c n(c d)
- c (cc- d) + K,nd
i [7]]
R2nd + 1 e c n(c d) + K
1- c- ((c - d) + Kne
8 K2nd+1- + n(c d) Kne
Then, the coefficients for Ix, 1XXK, 1 of V-1 in 56 can be written, respectively, as follows:
1
-
3
1
y
a + KB)
K a +
K3
K
32

---

Published as a conference paper at ICLR 2025
and
1
2 K18y
6, a + K3 + Kr7
a a(a + K3)
1
1a + KB K1
a+ + K3 + Kr7
a(a + KB)
K- Ki
K;
Ka+ K3B Kry
a
a: + K3
de
K2nd
K3X K2nd + 1 c+ n/c d) + Kind
(K k2)
Ki(K2nA)
K2nd + - c n(c d) + K2nd + - n(c- -dc-d + Kne
K k:)(d- e)
Kild- e)
(ri 11-99 q) K(K,d Ke)
(r: ss).
KKdd
Substituting the expanded form of V-1, we get:
<t -p'IKn 1 PT VI-V- -pIk P
n
=p'Ikn 1p_ (1Ix
q 9-8
1xxk + (( -p'Ix) P.
K
Also, for j E zt, the following holds:
9-- K
1xxx Q
D x, R
Q))
S)
j
9
8-b
(Q (Q"
1K*K +
1
K
k
k=1
(q 8)$
1KKK +
q
Q
T Q
(Q
K
k
K
#=1
(q S)
Ki-- {
i
(9-s) 1
1KKK
D
+
Q
K
K
K
Hence, we get
qIL + 9-8 S 1xxK + QTD((11(2))))
= q + Σ (1)--) (0--)19-1110 1 + 1((6,1k..x118 R
K
q'Ixx + x(t)lxxxx
+ Q D(u /
where v(t) and gh are defined as
v(t) = q +
9 -
- q¹:
K
(0 q + Ki
1 19-8
+ 0.)}}
q Ki
( (q-b 8
}
i [R].
K
K
33

---

Published as a conference paper at ICLR 2025
Finally, get the closed-form of the label averaging matrix o as
((ok + 9 K
$ = -I'Ikn + PTT
1xK + Q (((6))))))) - -p'IK
=p'In= + - P P D(u4 / + u(l)
R
P
11 Ki
KixKi
P+
g' --P'PD
+p' + ((t)
KuxKn-
Knn
label averaging mairiix
superclass wise label averaging matrix
Then, the closed-form of y! for the extended feature map is given as follows:
: = (a'
Σ
e(ús)
Σ
e(s))
+ -11-
-9--))
1K
Kgn
K
{
(=st(u))=s)]
(57)
where hyy) = 8. Note that when e = 0, vt) and uus are given as
v11) = 0, (1) = -
which recovers the result in (51)
K PROOF OF THEOREM 4.1 THEOREM 5.1 AND COROLLARY C.
In this section, present the complete proofs of Theorem 4.1 and 5.1 and Corollary C.1 As in Sec. we define the corruption matrix g x by
{i
k. U. k7
[C],,
Then, the mean of the given labels for samples that share the same true label, denoted as E ( e(ū,). corresponds to the y row of the matrix C, expressed as:
Σ
c(ú,) Cly.,
n
(J
Here, we aim to find the conditions for the prediction vector y!" to have the maximum value at the true label position Yi, indicating correct prediction. To achieve this, we use the dlosed-form solution
13 of y presented in Lemma 4.1
K.1 PROOF OF THEOREM
According to Lemma 4.1
the closed-form outputs of the 1-th distilled model can be formulated as
y (1)
p'e(1)) (g' (=
e(0,)) + (r: - x¹) ( Σ (ンツッ==ル)
Σ
e(ūr)) + (1 r!) 1x (j-h(,))=*)
where = h((:)) is the superclass of yis K2 is the size of the superclass s, and and r. are defined in (14) Also, assume that the label error happens only within each superclass, which implies that [C|,,'' = if (kk) * h(kk). Remind that we assume that the dataset is balanced with respect to both the ground-trus coound-truth labels and given labels, i.c., li 3/- = k}] Hi Ø = k]] for Yk [K]. Then, we can simplify the third term, the mean of superclass labels, using the indicator function, denoted as 1, as follows:
(r - - (
Σ
e(i,)) = r -
[L(b(L) s)..... ...(()))) = =s)],
(58)
K.n (=sh(o,))s)
K.
34

---

Published as a conference paper at ICLR 2025
which means that third term is uniform inside the superclass, while zero outside the superclass. Now, we will analyze the prediction of clean samples and label-noise samples.
For the clean samples, i.c., Ds ybs the prediction of the t-th distilled model for the i-th sample can be written as
/ + --')'(C[mmn + + gg)/Ka + +1++11111111 (1-r9)/K, p¹
== y:i
ly = (d' - p')[Clun + (rl - /')/K, + r,
+ ye, h(k)) = h(vs):
(1-r))/K.
h(k) + h(y:)-
Then, the condition for the correct classification of the clean sample is given by
1
Cluss [C] (a/p)- Vk y:.
(59)
For the noisy samples such that D: + yi, the prediction of the t-th distilled model can be written
(af - p')[C)n mn + (r.: - q¹)/K, + (1-r5)KK,
yı: = Úi; k + yis Dis h(k)) = h(y)):
p² + (d - p')(Clm + (ri (1-)/
[ =
(g' - (f'-p')(Cr p')[Clm II. + (rt - q///K, + +1 (1-r)/
r)/K,
((k) # h(y:)-
Thus, the condition for the correct classification of the noisy sample given as
[Clyven > >[Cle
and
(60)
(a/p)'
[C]usin [C] Js
Vk + ys.
(61)
Note that 59 is the easiest condition, and (60) is the most stringent condition. Thus, if
[C K + 1// ((g/p)* - 1), Vk,
[K] with k k
(62)
then the t-th student model can correctly classify both the noisy samples and clean samples. On the other hand, if
[C],,, < [C],, +1/ ((q/p)-11-1)
(63)
then the (t- 1) distilled model fails to correctly classify label-noisy samples from class k that have been mislabelled as 0; = k' and thus fails to achieve 100% population accuracy.
To find the regimes where the student model outperforms the teacher model, we need find / z+ that satisfies
CC + (g/p)--1-1 > [C] > [C]a,tr + (g/p)-1 -1
(64)
In such i, the student model correctly classifies the samples whose true labels are k and the given labels are k', but the teacher model makes mistakes.
K.2
PROOF OF THEOREM 5.1
The closed form solution for the output of the PLL student model for the i-th training instance is given by
(
y! py: + (g p)
Σ E (::0==0))
1x Σ Y, +(1 (1-rr)) r.) K (jshtyy)=s}
yj
(r. 4) K₂n
(65)
n
for s = h()). This can be derived from Lemma 4.1 by plugging in the refined teacher's output y, at the position of e(ū)) and considering the more step of distillation t 1. Here, y, is the two-hot vector with weights 1/2, at the positions of top two labels with the highest values the teacher's predictions.
35

---

Published as a conference paper at ICLR 2025
Suppose that yy... [Cly, holds for all & + ys. From the conditions (59) and 61 the true
label always included in the top-two candidate set. Let us define Y/ as
yi arg max C),,,- k
(66)
Note that Ot is always in the superclass of (v:).. Then, the equation for y, can be expressed as:
1
y: 2
-((x) + " ze(ū))
(67)
for the clean sample, and
1
y: = 2
(3:) + e(()))
(68)
2
for the noisy sample.
Then, 1 E [.w) y, and R,n E (=:((,))=)) Ny are given as
Σ
Y, 2 _((1)) + 1210 Cl.... e(ū)) + 1
1
1
n (
jtt
and
yj プソー K. Kgn
[1(h(1) .).... ... 1(((K =)];
(jah(1s)=R)
since the true labels and the given labels are balanced. Hence, for the clean sample, the prediction of the student model for the i-th sample be written
b
+ 2-
2 + R,
Pe-g
I-r. K Will (C)..))) Ta 1-r. K, K
= Ysi
ly
1=
+
(
k.
+ 1-te K
k Uss
$ C wick
+ Ys Dis ((,k(()) h(y:);
h(k) + h(:)).
Here, [y; I), has the maximum value at k = yi. because (C] yi.ue + [C] M. is less than or equal to 1.
For the noisy sample, when DI Ui. we have
2 + + p--4
K. + K
1-r-
= ys:
ly (P)
Jx
2 + + [ U... [C]
2--9 K.
+ 1-ne
k Ús:
[C wick + r= + K
+ yss ūi, (ikk)(k) = h(v));
K.
1-r. K
h(k)) * h(vs).
On the other hand, when D: * Dü, we have
g + p- r-- 1-re
K.
K
yEi
F + 2 [
Cl + r--g
K. + 1-n.
= 9:i
K
4 ([C)ri si:: + [C),, + K.
1-n
I D::
K
Clyu.k K, K
ra-q 1-r,
+ yes Ds. Ut. ((k)) h(ys): h(k) * h(y:))
K
Therefore, the prediction of the PLL student model has the highest probability at the true label for all noisy samples as well.
Throughout all the cases, it can be concluded that the PLL model consistently achieves 100% population accuracy when the PLL candidate sets of size two always contain the label. Note that the condition for this situation is given by (15).
To summarize, the t-th distilled model attains 100% population accuracy when
[C],, [C],,e/ + 1/ ((q/p)² 1), Vke k' [K] with k/
holds. For the PL.L student model, the condition for achieving 100% population accuracy is given by [C] [C], Vk, k E [K] with k + k/
which implies that the PLL student model outperforms any t-th distilled model in the population accuracy.
36

---

Published as a conference paper at ICLR 2025
K.3 PROOF OF COROLLARY C.1
Suppose that the class-wise feature correlation at the I-distillation round is given by
j: Vi = U₁ i i J Yi # Uj h(v:)) h(1)): h(y) A h(y)).
clt)
Φ ('00, == (6(x,), o(x,))
(69)
d(
0,
Then, following the similar analysis in Sec.
J.4 the label averaging matrix p(0)
IKn (IKn + (((1)((KnA))
] the 1-th distillation round can be expressed as follows:
((t)_ 1 TT
(q(0)
DI + D
(()))
1K X.
R
P II'''I
(70)
K,
b==1
where p((), 100 (t) are defined as
1 c1)
(1))
K2nd + 1 cltj²
1 1 -cit) + n(((t) - d(t1)
(())
K2nd c11) n((() d(t))'
1- (1) (c ((t)) + (an
(1))
K2nd + - elt) + n((()) - d(t)) + K Kyndo)
For simplicity notation, define D RRR.R RKKK as
ft
U1g Kg
D(u) D
K, X,,
(71)
1=11
CLAIM. For any t 1,
i T
0 Ix
+
D
(
P +
II
Knn
-=1
==1
1=1
Proof. When t = 1, the above claim holds trivially. Next, suppose our claim holds true for
Then, have
R
k
J
II 1=1
TT
II p Ix
D
II
(1)
(i)
P
In
i=1
1=1
and
(b+1) T (t ( I + D (l.(tt)) - (1+1)) [==1 )] P (
Since PPT = nIn and D(u)D(v) = D(u v). we have
k+1
k+1
II
a (1) = 1P [aIx + D(D P
II
Knn
1=1
for some and b. Then, a and b, can be computed as following:
4
(g) - ((++1)) + (k+1)
1
p ( (g (++1) (1()-1)
1=1
1
37

---

Published as a conference paper at ICLR 2025
and
k
b, = =(r- (1( - -9 1))
(0)) - (gg
p(ett))
1=1
k
+ (r!1t11) - (1)
Il p()) + +p(kt1) 2=1
(1(k)1) (k+
k+1
k+1
Thus, by the mathematical induction, our claim holds true for all 1.
From our claim, we can derive closed-fore solution of y in the evolving feature map (69) follows:
as
(1
II c(D)) +
I 0
p())
Σ
e(Us)
1=1
(72)
+
(1)
Σ
e(ts) +
1
1k
Kon (ih(()))=a)
K
The only difference from the original equation is that p², g¹, and r have changed to Ie (() and [[: re respectively. Hence, Corollary C.1 holds.
II p(),
38

---

Published as a conference paper at ICLR 2025
L ADDITIONAL EXPERIMENTAL RESULTS
L.1 ABLATION STUDY: VARYING WEIGHT-DECAY VALUE X
We perform an ablation study for À values that make q/p near 2 as explained in Appendix B.2 For these À values, we observe that there is a consistent accuracy gain as the distillation round t increases. Also, we observe that our PLL student model outperforms the multi-round SD models.
Caltach-103
Caltecb-256
CIT68-169
Flowere-102
Food-163
StanfordCare
(a) Superclass label corruption
Caltech 103
Caltech226
CIFAR-100
Comptionaat
Fla 102
Food-105
StrnfontCare
(b) Symmetric label corruption
Caltech- 101
Culbech- 256
CIPAR 100
Flouers-102
Food-103
StanfordCar
Tatto JTN
(c) Asymmetric label corruption
Figure 14: Accuracy gap for each student model compared to the teacher model in three different label corruption scenarios. PLAIN represents the gap between multi-round SD students and the teacher model, while ORANGE represents the gap between PLL students and the teacher model. We use weight decay value À= = 2 10-4 for Caltech- 101, Flowers-102, and StanfordCars datasets, and = 2 x 10-5 for CIFAR-100, 100, Caltech-256, and Food-101 datasets.
39

---

Published as a conference paper at ICLR 2025
Caftectls 101
Coltech-250
CIPAR 100
Flower-162
Food 103
StanfordCars
Sarvatio
Superclass label corruption
Caltech-101
Callsch-225 256
CPAAR-30
Goeg Flowers 102
Food-18
StanfordCars
(b) Symmetric label corruption
Celbech-102
Cattech-
102
Foodi11
StanfordCaars
Somaptal Meti
(c) Asymmetric label corruption
Figure 15: Accuracy gap for each student model compared to the teacher model in a three label corruption scenarios. PLAIN represents the gap between multi-round SD students and the teacher model, while ORANGE represents the gap between PLL students and the teacher model. We use weight decay value x = 1 x 10-3 for Caltech-101. Flowers- 102, and StanfordCars datasets, and = 1 x 10 for CIFAR-100, 100, Caltech-256, and Food-101 datasets.
40

---

Published as a conference paper at ICLR 2025
L.2 HIERARCHICAL CLUSTERING RESULTS
We conduct hierarchical clustering to estimate the super-class on six different real datasets. The figures below show the hierarchical clustering results of each dataset. Clustering is performed to place similar classes adjacent to each other.
kanganse
skaci
lamu
peal
dlppaant
rrocedie
Fumingo
Tummou head
couger body
eral
Caterpalar
honkonil
cogrin
Figure 16: Hierarchical clustering results for CIFAR- 100, Caltech- 101 and Caltech- 256 datasets
bans
furn 1th
cocod
statyuu
teaver
Soupion
Aater:
kous
Uggar
koster
and
sctopus
Shark
sraportly
Eutberly
pain
plaza
CIFAR-100
Humoue or
Caltech-101
tomp
Caltech-256
bany
woman
borse:
chandulier
tammi
rese
ewer
apccer bull
eunfiover
cxp.5 s
tainatts
mounnaln
read
purtdd
cil
magil
helmaree
eno
leny
biem
chat
bortle
can
shatithur
grand puus
Votorbi lec
Paces eary
raveet poppeer
Teadphons
gramephion
Lukee
motortyde
Micycle
<lotric
metrrome
acord
Wine State
Swoohare
segede
minore
41

---

Published as a conference paper at ICLR 2025
makoow
Jusker bosu
pird caradiee
cam chonder
crom, Jestez
hurnon
mrond
Marne Bower
hor and sour SoAD
macarant theare
bearded Iie
planc ansiouram by
mettrrd
biohoo of booder
peocoaten
pood
Figure 17: Hierarchical clustering results for Flowers- 102, Food- 101 and Stanford datasets
glnger
scagheert carbonars
ssaghottt bolograne
carration
burtecuup
beod Salad
beaf Conpaccio
prod, maskower
jroe Lon soter
ceviche
edenaam
huncirch
Ioterspemmam
puscamett
mallbe
preak muted
artichio
borl Frwer
cosweed salid
handlaaved podiet ordhid
syptura
them spols
mussds
magnall
cudh
Capeitonet
sh marggaid
Cheass plats
storness pertion
mul
yoouu
troreta
Swera
StanfordCars
Flowers-104
swestwilln
Food Food-101 101
rancakee
the
Ice mamn
wotorcuss
nal pensttcke
Braniu
purp conetwwrr
cametio
erowberyy chomtakke
pemm
etanc
bhethysh 5h
dobe-foaa
buly. buar
hona mama
Canteruayy tels
azaina
povt sapo
Mobous
pork chos
By
Deremus
crnsue madime
ponarta
Pah and chioe
tiper (iy
columbine na
Med calmmat
ptrra
poss flswer
Clemadt
apote gre
pr mic es feathers
French boast
booster Laaddaee
burbutenn dakay
pertts rone
breakrsat bunto
crnekate
bolero does Mlue
samale
pours
krrin ruls
ircen drdee
reidng Back
Buttoll
takoped
pink pr mise
king procua
alsin oos felly
criled cheess sarowich
sall
partsc bread
penrian
Prnnn miss
aoursi0
42

---

Published as a conference paper at ICLR 2025
M DETAILED EMPIRICAL RESULTS
In this section, we report the full results of the experiments presented in the Sec. Appendix and
Table 3: (Details Fig. 9a Test accuracy of Caltech- 101 dataset applying superclass label corruption, where weight decay value À= 5 x 10"
Distillation Stap
D.D
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
95.03==0.00%
93,4720.22%
90,40:00.35%
84.31 86%
78.7620.89%%
67.80
1.775
16.38:20.98%
95.10.10.05%
93.7410.20%
90.9410.22%%
19.8810.65%
68.5911.288%
95.05.10.099
93.74.10.248
90.8410.37%%
85.60.1.011%
80.1510..66%
68.87.11.289
47.85.1.200%
18.66.11.599
94.9 94.97:10.11 17%
03.74-10.20%%
90.92:10.355
16.54
80.05:00.52%
69.01 11:1111:311
48.0 48.04-12.18%
18.63:11.78%
94 855-
+2.68-0.115)
900 18 +6.8
85 55.60--1. 1.00--1166 16%
50.15+00.52
69.01 1228
47.96+2.1555
18. 1.889
PLL
14.0
92.67.0.41%
89. 59.69.0053%%
25.54
81.1110.77%
70.79
48.5812.40%
19.781144%%
Table 4: (Details of Fig. 9a) Test accuracy of Caltech-256 dataset applying superclass label corruption, where weight decay value
10-b
Distillation Step
(Teacher)
$1.20=0.225
77.96+0.225%
72.61+0.34%
69.57=00.05%
63.68+0.588
52.63+00.74%
28.
83.67:#0.07% %
81.98::0.15%
78.16+00.7%%
72.66=00.3%%
69. 69.79:00.1% 77%
63.93:0.699
52.97=00:33%
28.677: 1.82%
83 53.60100.08%
11.9%
78. 12:10.111%
72.5610.4116
09.83110.23%
63.89910.68%
52.914
28 1.83%
83. 53.53.10.08%
81 $1.88.00 11.88.....49
78.02.00.09% 0..0%
72.5310.399%
69 00.80..0.24%%
63. 10.66%
52.9210.95%
28. 1.88%
83.53.10.05%
51.83.1..21%
77.93.10110 10%
72.42.10.39%
0...0.2
60.65%
28.
PLL
82.64.10.24%
80.23.10.29%
76.08.11.13%
71.0010.46%
69 69.63.11.67%)
65.37.11.45%
54.33
28.68.1..74%
Table 5: (Details of Fig. 9ah Test accuracy of CIFAR-100 dataset applying superclass label corruption, where weight decay value
10
Distillation
0.1
0.3
0.7
(Teacher)
70.6010.05%
65.8010.111%
09.58..0.22%
52.38.10.19%
16.7
19.5911..25%
28.33
13.21:10.22%
71.731..222%
69.55110.11%
66.03.10.200
62.13:10.21%
58.66100600%
53.2411.113%
43.10.10.37%
22.62.11.22%%
72,06 72
69. 20...6
66. 66.34:10.20%
62.35.10.12%
58.63:10.57%
53.4110.551%
43.57.10.32%
23. 1..3%.11.13%%
72.03-+0.225 25%
69 83- 279
66.4+-0.215
62.40+-0.10 129
56.70-00.559
53.52:+0.552 5.52--0.529
43.72-+0.3255
23.53+11179
72.0410.28%%
69.86.00288%
86.1910.17%
62.4110.14% 14%
58.7410.54%
53.63110.52%
43. 10.33%
23.65
69.06.10.16%
67.85.10005%
66.9110.23%
63.88.10.28%
61.3811..3%%
57.1310..44%
48.60.10.87%
26.4610.73%%
Table 6: (Details of Fig. 9a) Test accuracy of Flowers- 102 dataset applying superclass label corruption, where weight decay value À 5 x 10
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacee)
08.10:0.17%
67.75:10.44%
51:93:11006
22.55:10.48%
9.64:10.289
$7.94.10.29%
$3.04-10.28%
28.6
73.2011.099%
41.80112.691
29.02.11.64%
87.78.10.399%
83.66.10.00
75.3910.21%%
63.14.1..88% 58%%
53.7312.525%
43.53112.06%
30.23111.01%
13.0111.001%
$7.29100.999
83.82.10. 14.1%
75.82.10.415
63.73.11.52%
54.12.12.1%%
44.25:11 60%%
50.72:11.066
13:27.11.75
87.033-60.245
03.92:00.35%
75.92:00.605
64.22
54.44:11.550
44.61:11 39%
31,18+1 1.06%
13.43111988
85.62:60.20%
79.71:00.39%
74 04-83.00.32
63.86:11 0.70%
56.90-3.111%
46.37:11.11%
34.44:1.25%%
14.22:0.42%
Table (Details of Fig. Dab Test accuracy Food- dataset applying superclass label corruption,
where weight decay value
5 x 10
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
63.52.00.02%
61.7710..6%%
67.9610..5%%
52/4410.234 23%
48.35.00.10%
42.36100.15% 12%
32, 000.9%%
15.6910.36%%
63.4510.00%%
62.41:10.09%
10.066
53.35.10.00%
49.57.10.28%
39.2111.73%%
19.00113.881
63. 63.29-+0.08%
62.22:+0.079.
50.68-+0.244
56.0++0.10
53.30+0.189
49.61-+0.20
39.
19.11+43.88%
63. 17+0.08%
62.13++0.099,
59.63-+0.225
55.97+0.11%
53, 13.24-0017%%
49.61 00/61++0.00 28%
39. 3..33%99..7%
19.13433.93%
63:00::0.12%
62.04=00.13%
59. 59.555+-0.2396
55.96=00.119
33.24:50.17%
49 29.633+0.2665
30.35::3.8044
19.97%9
PL.
63.31#0.099%
61.93=00.0%%
59.99=0.37%
$7.74=0.07%
55.7330..5%%
53.24400.9%
43.02+5.77%
21.0647.16%
Table 8: (Details of Fig.
9a
Test accuracy of StanfordCars dataset applying superclass label corruption.
where weight decay value À 5 x 10-1
Distillation Step
0.0
0.1
0.3
D.5
1.6
0.7
0.8
0.9
(Tescher)
43.0210.03%
35.9210.27%
26.60110.1 13%
17.5210.11%
13.43.0..2%%
9.88.10221%
5.92=0.145
2.93-00291
43.18.0..65
20.77.10.095
16.$110.169
12.22.10.13%
7.60-0.255
3.20-0035%
06.97:00006
38.8010.1 15%
31.5 01.5810.06%
21.7610..11%
17.65:00007%
12.63:10.00
7.85 0818
3.27=0422
42.38++0 14%
38.96
0.15%
32. 30++00000000 225
21 11.00+00922
17.78+00.22%
12.73++0.1669
7.93 0.25% 2555
1.32--0.525.
41.76++0.019
38.79+0.360 36%
32.65=0.211
21.814 1.8%%
17.96++0.24%
12.68:+0.22%
7.82-0.366 9636
3.33=11663
41.2810.111%
36.32.0.18%
31.3610..29%
23.55110.22%
19.67.10.4%%
14.1710.40%
8.88.10.00%
3.0110055%%
43

---

Published as a conference paper at ICLR 2025
Table 9: (Details of Fig. 9b Test accuracy of Caltech-10 101 dataset applying symmetric label corruption. where weight decay value À= 5 x 10
Distiliaion Step
0.D
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Tencher)
95.03+0.07%%
93.51=00.18
91.11 48%
57.92++0.11 17%
84, 44.55
80.76-10.5499
69. 10.59-=0.36%
45.97-t0.666%
95.10+0..55% %
93.70+0.22%
91. 91.74+0.525%
48.40-00447%
85.29+0.78%
81.14+00-55%
70.97-00.44%
47.56+1.399
95.05=00.09%
95.63-0.23%%
91.84+047%%
58.63=0.644%
71.22+0.71%
47.85+1.43%%
94.9720.17%%
93/47200.14%
91.785.0.35%
60.61
85.41
81.0910..88%
71.29:00.64%
47.75:11.51%
91.8510.15%%
93.4110.07%
00.80100.66%
$8.59100577
85. 05.50....81%
11.0110.4665
71.4110.87%
47.6211.43%
PL.1.
94.033+0.195
92.78+0.525%
91.53++05505
88.63+0.725%
N5.56+2..25%
82.51+0..19%
73.69-1444%
51.77+2988%
Table 10: (Details of Fig. 9b
Test accuracy of Caltech-256 dataset applying symmetric label corruption, where weight decay value X = 5 x 10-5
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
83.63==0.033%
81.76=00.15%
78. 78.59:40.395
74.50==0.294 29%
11.47=0.122%
66.45+00.766 76%%
57.47=00.42%
34.41=00.1%%
83.67:00.0755 %
78.78:...99% 82.0110.35%%
74.68:0..26%
71.6410.11%%
66 66.86200.87
57 311.%
34.311.0.12%
65.60++3.0089
81.90+0.300 90%
78.71++0.55 57%
74.61+0.31
71. 14.5%
16.44++.. 85F
57. 31%%
34.25+0.15%%
83.53-40.08%
31.%%
78.67+00.375
74.57=0.228%
71.51=0.15%%
66.77400.985
14.050..12%
83 53.53:60.05%
$1.78:00.2%%
78.65:10.35%
74.57=00.26%
11.42:0.18%
66.69:10.7356
57.17:0.311%
34.07:10.15%
PLL
82.64:00.25%
80.43:00711%
76.47:0.41%
73.33=00.68%
70.60=00.00%
68.16:11111%
0..38:
32.5%
Table 11: (Details of Fig. 9b) Test accuracy of CIFAR-100 dataset applying symmetric label corrup- tion, where weight decay value = 5 x 10-5
Distillation Step
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
70.00+0.05%
66.07+0.06%%
61.66++0.21%
55.98+0.10%
51 51.60+0.22%%
45.78+00.08%
35.83-+0. 19%
18.81++0. 14%
2
71.73+00.225
30.02=0.144%
67.41++0.36%
63.4440.433%
62.43=0..0%%
58.83+t0.815
51.42+00.664
34. 44.38+-0.70%%
02.05::0.22%
00.25=0.188%
67.7210.35%
64.59
18.49
59.03 7.%
51.7720.55%
75%%%
72.03-10.25%
70.2810.199%
67.77..0.99%
64.6810.39%
62.51.10.18%
59.1610.77%
1.0.5%
35.62..0.87%%
72.04.10.28%
70.27.00.14 1496
67.76.10.43%
64.72.10.37%
62 62.6%%10.17%
59.23310.809
52.05
35.76110.95%
PLL
69.16.10.16%
68.00.0..21%
67.48.10.34%
65.52_.0.14%
63.6610..7%%
61.09100.35%
56.2110.118%
10.55.0.255%
Table 12: (Details of Fig. 9b Test accuracy of Flowers-102 dataset applying symmetric label corruption, where weight decay value = 10-4
Distillation Step
(TTacber)
88.10:10.17% %
80.3910..06%
87.16110.88%
43.82:1.2255 11.4121.79%
34.481.0553%
22. 22.55:10.08%
1..75
29.2%.....%%
$2.91
74.1110.42%
62.2211..7%%
55.3911.24%
45.52.11.44%
32.3911.8854
12.81.11.04%%
$7.78.10.399
$3.82.10.6618 6-
15.26110.47%
7.
46.63119.338
33. 13.5012.1199
13.89%
75.
33.39-00.68%
2.54%
57.78-00.324
68%
2.511.
14.25:1.77
67.03.10.24%
%
57.7811..2%%
47.09.0.799%
PLL
85.62.10.00%
80.07.00.49%
66.83.2.73% 74.48.10.85%
60.3912.68%%
50.2611.388%
35.88.1.400%
1..3%
Table 13: (Details of Fig. 9b) Test accuracy of Food- 101 dataset applying symmetric label corruption, where weight decay value
10 10-5
Distillatoon Step
(Teatber)
63.52.1..02%
61.92.10.100 10%
58.24.10.24%
13.39.0.23%
49.73110.07%
44.31.10.23%
35.49.0.249%
18.44.10.3%%
45.45---.47%
62.56-10.10%
60.21+00.0%%
56.99-0.15%
53.22++.3388
51.21+40.88%
44 11.%9%0.11%
24:92:+-43305
63.29.10.0 63.2910.08%%
62.4710.09%
60.15110.07%
16.951
53.24.12.37%
51.22.00.41%
44.7910.11%
24.94.14.366
63.07.0.088
62.39 129.
60.10.10.07%
56.92100.199
55.2012.40%%
51.19100.36%
1..0099
63.000:3.110 12%
62.25::0.11%
60.00:1:.005
53.19::2.41%
25. 25.044114.45%
PLAL
63.31:10.09%
62.03::0.124
60.2410.05%
07.97:0..29%
%
54.26:12.81%
54.01.1..09%
49.2310.189%
29.3618803%
Table 14: (Details of Fig. 9b) Test accuracy of StanfordCars dataset applying symmetric label corruption, where weight decay value À 5 40-4
Distillation
0.1
0.3
(Teacher)
26.51+0.3248
13.26+
5.45-0.17%
2.49-0.125
2
4..18:0.06%
37.8940.17 17%
29.77+0.30%%
21.2740.23%
16.76 10%
12.20:10119%
6.89-0.344 0.34%
2.77=0
3
42.97100006
38.86100006%
30.9110.433%
1..951..17%
16.89.0.400 46%
12.40:00.14%
7.20
2.84=00254
42.38-H0
38.95-0.11%
31.36+00.55%
22. 22.15+00.66%
16-91+0.566%
12.32-00.7%%
7.26-0.55
2.90-0.339
41.76:00.01%
38.86440.10
31.61:00.338
1..00.00.4%
16.72-00.00%
12:19...66
0.22-0.622%
2.92=00345
PLL
41.28:501445 14%
36.5840.199%
31.97:00.66%
24.76:0.37%
20.56=0.27%
14.6110.99%%
8.26:.0.6%
2.93:00341%
44

---

Published as a conference paper at ICLR 2025
Table 15: (Details of Fig. 9c Test accuracy of Caltech- 101 dataset applying asymmetric label corruption, where weight decay value À = 5 10-1
Distiliaion Step
D.D
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Tencher)
95.03+0.07%%
93663+0.22 22%
91 1..00-+01119 19%
87 47.69:00.41%
85. 1466
80.22:+
108%%
70.87=00.115
45.
45.12:+0:925
95.10+0..55%
93.66-+0229%
48.27
35.93-0.35%%
80.6+11.0 100%
71.74+00.60%
95.05=00.09%
93.55=0.335%
58.31=0.400%
71.8%=0.811%
94.972:0.17%
93.53-20.2316
91.6110.53%
88.40100337 87%%
85 08.899
80.63:2 1.45%
01.79
46.83:11.372
91.8510.15%
93.53.1.2216
91.5710.655%
$8.46.0 38.4610..35%
85 65.98100.22%
80.57110.55%
71.72110.82%
16.87.11.255%
PL.1
94.033+0.199
92.74-+0.09%
90184++0.689
88.71-11.44
87.14+0.53%
81.43+00.88%
72.50+2..54%
48.73+2.36%
Table 16: (Details of Fig. 9c
Test accuracy of Caltech-255 dataset applying asymmetric label corruption, where weight decay value X = 5 x 10-5
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
83.63=+0.033%
1886
78.52:00.20%
74.35=00.19% 19%
71.22=00.8%
66 (6.50++0.3375
57
57.42=00711%
35.1 1115:1111105
83.67::0.07 %
81.8610.10%
%
74.82:.0499%
01.2%
66.761:0.00 28%
07.72
18.1911188%
35.601+31008%
81.72+0.14%
50%
14.80-00555%
71.32+00.3%
66.73+0.2395
37.644+0.99%
15.16+1.19%%
83.53-40.088%
78.53-40.2885
54.74400.54%
71.27=00.25%
57.57+1.9254
38.11=11188%
83.53-60.05%
81 11.65550.18%
78.51:0.322%
34.7220.545%
71.21:60.44%
66.63-20.3156
57.54:11.01%
15.11:81.24%
PLL
82.64::0.25%
80.38:0.211%
75.87:10.49%
74.06
20.82s:0885%
68.04::0.44%
$7.91=2.38%
34.65:22.46%
Table 17: (Details of Fig. 9c Test accuracy of CIFAR-100 dataset applying asymmetric label corruption, where weight decay value X = 5 x 10-5.
Distillation Step
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
70.0+00.05%
66 16.113+0.188
61.97+0.11%
55.76+0.19%
51.61+0.119%
45.46+0..3 32%
35.83-00.11%
18.88++0.322 33%%
2
71.73+0..25%
69 87%
67.29+00199
64.21=0.11%
01.51
58.55-t0.2155
52.36=#0.32%%
$4.72+-0.40%%
02.05::0.22%
20.01
67.3710..19% 19%
64.43220.1%
62.252.0.48%
58.7810..185 18%
62.62200.32%
36. 111.
72.03-10.25%
61.5010..8%%
62.29100.22%
72.04.10.28%
70.06.10.f4 46%
67.47.10.14%
68.57%.0.18%
62. 42.32..0.411
58. 1.0.25%
52,
52.84.10.266%
36.46.10.16%%
PLL
69.16.10.16%
68.4110.15%
67.59.10.20%
2...26
63.55 2...28%
60.731.0.22%
56.51.00.10%
40.53.10.55%
Table 18: (Details of Fig. 9c Test accuracy of Flowers- 102 dataset applying asymmetric label corruption, where weight decay value = 10-4
Distillation Step
(TTacber)
88.10:10.17% %
80.33:10.12%
17.7%
52.29:1.45%
1..90
34.28:1..0%%
22.22:0.32%
11.08:11.1%%
$7.91.10.299
73.8910.46%% 20.21
16%%
57.1912.98%
46.50.1.42%%
51.2412.05%%
15.95.11.93%
$7.78.10.399
53.60100.119 61
75.9210.722%
65.2012.211%
39.44.1..33%
48.3311.008 1.08%
32.39122228%
15.9511.629
72%
65.
92
60.20-11.111
88%
52.61++2,149
15.82-1..445 1.74%
67.03.10.24%
83.82.0085%
7..57..1.15%
60.5210..11%
32.32.2..44%
15.72
PLL
85.62..0.0%%
19.84.10.18%
75.2311.744%
66.73.2.0055
62.2221.67%
51.27
36.4411.600%
17.4812.36%
Table 19: (Details of Fig. 9c Test accuracy of Food- 101 dataset applying asymmetric label corruption, where weight decay value
10 10-5
Distillation Step
(Teatber)
63.52.1..02%
61.8 61.83:10.11 14%
58.27.10.100 10%
53.17.10.12
49.74:10.179 17%
44.34:10.22 22%
35.16610.209
18. 18.62.10.34%%
45.45---.47%%
62.43+0.11%%
60.17-0.12%
56. 66.44-0008%
53.95--0.649%
51.077+0.40P
43.86 :...1%-..1%
25.16-4-3311.
63.29.100 63.2910.08%
62.3110.12% 10%
12%
50.04%
53.97.0.663%
51.0610.43%
43.9210.17%
25.24.14.35%
63.17.0.08%
10..99...39
60.04.1..15%
16..119.119
53.96100.639
31.03.19.44%
43.94 17%
25.31.14.39%
63.00-20.110 12%
62.10::01144
18%
76.76:::0
44%%
170.7%
25.35:14.39%
PL.L
63.31:10.09%
60.11:10.13%
57.69 % 56.71 0..73%
53.861.0.29%
48.52:0.33%
29.71.17.46%
Table 20: (Details of Fig. 9c Test accuracy of StanfordCars dataset applying asymmetric label corruption, where weight decay value À 5 4
Distillation
0.1
0.3
(Teacher)
26.38+0.31
17.2240.31%
13.0400.24%
9.48-0.15%
5.35-0.199 19%
2
43.18:0.06%
37.84:0.0554
29.76:1..266
21.00+#.311%
16.7650.35%
11.79.0.39%%
6.78-0.465
3.03 20%
3
42.97100006%
38.7010.004%
31.1310.149
21.7110.53%
16.9710.62%
12.05::..326
0.08
3.12::0.200
42.38-0.
38.73-0.0%%
11.8990024%%
21.76+00.905
16 16.84-0.77%%
12.00-0.39%
7.17-0.579
3.13-0 20%
41.76:0..01%
38.5240.04%
21.77:11.5%%
11.87:0.366%
7.19=0.588%
3.18:0.166%
PLL
41.28:501445 14%
36.6340.2246
31.57:0..0%
24.78:00.56%
20.69:0.344
13.17:0.71%
8.20%:1.14%
3.17=0335%
45

---

Published as a conference paper at ICLR 2025
Table 21: (Details of Fig. 14a) Test accuracy of Caltech- 101 dataset applying superclass label corruption, where weight decay value X = 2
10-1
Distilition Step
D.D
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Tencher)
95.01+9 10%
93.45-00.24 24
901 00.58+00.338
84 94.29:00.915
76, 76.74+00.223
67.76 16%
47.37+1..75%
16.36--1.025
95.10+0.05%% %
93.70+0.195%
90.M-+002775
84,41-0.855
79 09.0000.44%
68.45+1331%
49.42++1.99
184771141%
95.05+00.55%
95.61=0.14%
84. $4.39=00.00 80%
19.99=0.49%
18.51:11.58%
94.95150.05%
93.61
90.73::0.25%
84.25
79.92:0.50%%
68.70211.325
49.40
18.61:11.67%
94.91
93.5710.15%
90.63100175
84.16 7.9.99
79. 19.9010.5004
1.35%
49. 19.25522.0406
18,9912.425
PI..
93.82+00966
92.67-+0.41%
89.75++0.505%
84.31++0.726
81.13+0.67%
51.67+2.74% 71187+1113%
19.72+144%%
Table 22: (Details of Fig. 14a) Test accuracy of Caltech-256 dataset applying superclass label corruption, where weight decay value X = 2 x 10-5
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.9
(Taacher)
83.63=#0.01%
S1 S1.69900.23%
77.96+0.22485
72.60=00.36% 36%
69. 10.57=00.03%
63 633666+::.5776
52.61=#0.754
28.32:t1 1.56%
83.6810.0855 %
81.971:0.16%
78.1410.17%
72.62:0..2%%
09.77..0.18%
63.871:0
52, 0.99%%
28.62:11.82%
83.64++3.008
81.91+001995 19%
78.09+0.09%
72.52+0.37%
00.76+0.21%
65.84++0 69%
52. 01.1919-
28.54+11.115
53.55-40.008%
81 $1.83+0.223%
78.03+0.1115 11%
72.46+0.334
00.72=0.215%
63.784006638
52.76=00:94%
28
83. 33.50%
21.78:00.216
77.92:0.105
72.45.0.33%
89 00.68
63.72400.005 20%
52.73:60.98%
28.40:11.895
PLL
82.49:0..12%
30.25=0.311%
16.09:1.115
71.00=0.46%6
6..63
65.42::1.446
1.237
28.70:12.70%
Table 23: (Details of Fig. 14a Test accuracy of CIFAR-100 dataset applying superclass label corruption, where weight decay value X = 2 x 10-5
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Taacher)
70.35+00.02%
64.82-00.07%
58, 08.54+00.255
51.24+0.08%
45. 45.54+0.115%
38. 38.39++..20 20%
27. 37-.34.-...
12.82++0.175
2
71.45=0.18%
68.02=1.0099
61. 61.70=0.199
59.01=00116%
52.27=#0.276
42.63-=0.400%
22.044+-0.997%
71.59200.248
68. 1.06%
66 36.262:0.3%%
62.07 37%
59.11::0.200
13.07.1.15%
43.05100.006
22.751:0.95%
71.6310.23%%
68.1411.00%%
66.31110.39%
10.11110.400
19.1510..11%
43.1710.22%
22.83310.97%.
71 11.62.10.18%%
68. 68.14:11005%
66.34..0.88%
62.13.1...00 40%
59. 19.0.9002
53.18.11 53.18.11.699
43.23.10.27%
22.95.1..000
PLL
69. 29.30..0.20%%
60.21%
65.85.10.46%
62.96.00.34%
60.65 1...1%%
56.6110.333%
47.50.10.811 % 25.56.10.80%
Table 24: (Details of Fig. 14a Test accuracy of Flowers- dataset applying superclass label corruption, where weight decay value = 10-4
Distillation Step
(TTacber)
88.01:10.17% %
80.36110.09%
56.41110.33%
50.361:0.9%%
1..66
31.01::1.36%
21.14:20.99%
9.22:10.28%
09.9%
32.35
12.12.1..88%
58.04
1..95
39.34.11.93%
27.4811..66%
10.36.10.48%
87.751.0.24% 24%%
$2.88.10.17%
73.6910.811
59.6111..609
3..36
40.62.11.32%
2..35
10.69.10.76%
87.
74.12-0.85%
$1.24-H0.324
41.
74%
1.59%
10.69-20.909
10.09%
83.0410.49%
74.05.10.88%
60.33.1..98%
51.57.10.22%
41.27.10.50%
29.25 1..7%
10.65.1.009%
PLL
85.23310.18%
79.08.10.26%
30.33.11.88%
57.91
50.29
40.16:11.295
28.92
11.671...9%
Table 25: (Details of Fig. 14a Test accuracy of Food-10 dataset applying superclass label corruption, where weight decay value
10-5
Distillation Step
(Teatber)
63.4610.033%
61 01.66110.173
57.79.10.244
52.15:10.244
48.09100066
42.04.10.14 14%
32.5611.00%%
15.58.10.46%
63.52-0.05%%
62.25-00.99%
59. 50.72-+0.26%
56. 14-10.2.7%
51.75+2.59%
47.45+2.69%
37. 37.58++3.69%
19.13+44.1%%
63. 63.28.12 63.28.00.08%
10.3%%
59 59.66.10.28%
56.121...25
51.6812.56%
47.4212.68%
37.54.13.70%
19.15114.05%
63.1310.044%
61.99110.29
59. 39.56.10.24%
56. 03.08.5 10008....00 1...
51 31.63322.59
47.3912.688 686
37.52.13.69%.
19.16.14.06%
62.98:101 08%%
61.90):0.29%
50.47:10.23%
56
51.59::2.52%
47.57
37.52:13719%
19.15144.9%
PL.L
63.36:10.144 14%
61.85::0.00%
59.9310.366
57.20:0.12%%
53.57.13.23%
50.75133.99%
40.59:15.33%
21.3017.40%
Table 26: (Details of Fig. 14a
Test accuracy of StanfordCars dataset applying superclass label corruption, where weight decay value À
4
Distillation Step
0.1
0.3
0.7
(Teacher)
24.40+0.135
15.74+0.09%
11.94=0.24%
8.95=0.09%
5.51-0.14%
2
42.97:00.1 15%g
35.96:20.254
27.42+0.2288
18.23300.77%
14.48:00.45%
9.97=00.7%
6.04=0.199
3.15
3
43.03-0.188%
36.90110211%
28.61110.45%
18.6010.633%
14.99.0.44%%
10:28:#1:2%%
6.29-0.311
3.21=:03384
02:924000055
37.36++
28.77+0..99%
18.75=0.72%%
15.13-+0.445
10.32+1..669
6.30-00399%
3.25--0.37%
42.82:0.07%%
37.74:0.36%
28.68:00.66%
18.7340.68%
10.41:1.48%%
6.28-00.3%%
PLL
41.30=0.000%
34.5740.46%
27.97:40.00%
18.51
14.61
9.44%
5.99=0.369%
2.98=00366 -636
46

---

Published as a conference paper at ICLR 2025
Table 27: (Details of Fig. 14b) Test accuracy of Caltech- 101 dataset applying symmetric label corruption, where weight decay value X = 2 x 10 10-
Distillaton Step
D.D
0.1
0.3
0..5
0.6
0.7
0.8
0.9
(Taacher)
95.01
93.51=
91.09::0.51%
87 12%
$4.52::0.43%
80.72::0.5756 57%
69. 69.57=00.31%6
45.87:::0733%
05.10.10.05% %
93.70.0..27%
01.72..0544%
88.38 1.0.50%
81.0310.55%% 8..3%%
20.89100.441
47.57.11.3%%
91.90.00.42%
88,
55.22.10.85%
54%
70.95.0.78%
47.4311.49%
94.95:10.005%
93.49 29%
18::0000000 3818
88.
85. 15.12:00.73%
80
431
70. 00::000000000
47.48:1:2259
94.91 +4.91-+0.47%
91.43-00266%
91. 91.72++0.39 39%
88. 49.%%
85.06+00.788
N0.78++0.475
+..772+0.84%
47.31++1.22%
PI.1.
93.82++0.96%
92.78+00448%
91.51-+0.5258
88.81-00725%
25.56++2.39%
82.39-00.1%%
73.71+1442%
$1.69+2992%
Table 28: (Details of Fig. 14b) Test accuracy of Caltech- 256 dataset applying symmetric label corruption, where weight decay value X = 2 x 10-5
Distillation Step
0.1
0.6
0.7
0.9
(Teacher)
53.63.10.01%
81. 16.11..10 1-0.
78.5910.39%%
74.5010.29%
71.4910.148
66.44.10.33 75%
57.44.00.4%%
34.4110.17%
03.6K:10.088 56
0..34%
74.79:00.40%
74.65:0..5%
71.63:0.108%
66 04.844+-..8 87%
57. 0.3392
34.62:0..00%
83.64+00000
30%
78.68+0..57 57%
74.60++0.266
71.51+0.109%
66.81 82%
57.25-+0.37
34 +4.6)+00355%
83.55-40.08%%
81
78.61=+0.41%
71.43+0.09%
76%
$7.09=00.322
34.48+0.40%
83. 33.50%
31.76:50.34%
78.55:10.38%%
24.4620.225%
71.39:0.11%%
66.724007655
57 37.02%00.3%%
34.60110,44%
PLL.
82.49100129%
80.45:00.71%
76.51
73.33:10.55%
70.60:0.19%
58.41.00.31% 1..1%
34.30E1.03%
Table 29: (Details of Fig. 14b) Test accuracy of CIFAR-100 dataset applying symmetric label corruption, where weight decay value
10-5
Distillation Step
(Taacher)
70.35+0..22%
65 (8.18++0.0 14%
60.68+0.177 17%
55.04+0001%%
50 50.50+0.233%
44.+000011
34 34.64++0.11
18.09+00 10%
11.45=0.188%
69 (0.27==0.63%
07.30++0.299:
64 (3%%
62.07=0000%%
58. 38.56=+0..3396
51.33-00.87%
33.35 33.35=+t0.70%%
71 31.59:30.2445
69 60:20.429
67. 333%
64.43:0.10 10%
62.21:10.07% 078
58. 74-44550 805
51, $1.78::0.659
34. 34.36:10.739%
71.63+0.23%
69.64+0.39%
67.41+00.33%
64.55+0.109
00.26+00.08%
58 03.83+0.7777 77%%
$1.87-00.61%
$4.41-+0.744%
18.6240.18%
69.6240.38%%
67.39:40.32%%
64.4740.08%
62.30+0.111%
58, 85.85500.816
51.93=00.60%
54.42400.6%%
PL..
69. 20.30+0..0%%
67. 34:30.00433
67, 14+60.1%
65.20=00111
63.35++0.39%
60.54+0.30%%
55.18=00.815 % 3..94+1.43%%
Table 30: (Details of Fig. 14b) Test accuracy of Flowers-102 dataset applying symmetric label corruption, where weight decay value x
10-4
Distillation Step
(Teacher)
88.01 #0.17%
79.74:10.549
66 18+0088%
49.4::1.00 1.79%
41.31:1145%
32.12:+0.80
21 11.01:0.1199
10.07::1.55%%
87.75+01.29%
82.45-00.35%
72.55+1..3%%
58.01 +0.19+
52.88-0.338
42.97+00 56%
29 00.08+1.200
11.37+1.889%
67.75=00.244
53.40=00.415
73.53+1156%%
59.12=00.48%
55.
$3.59::1.0659
45.42+t0.88 8.442
30 1.4%%
11.99:22.72%
87.55200.16%
83.73:0.58%
1.54%
59.08:20.48%
16.37%=1.13%
45.78::1.045
30.7521.71%
11.99
$7.0910.30%%
53.56.10.52%
13.95
59.02.0.733%
16.70.1.188%
45.88.10.97%
1..65
11.96.1..74%
PLL
85.233.0.18%
18.63
71.2412.18%%
18.50
1..3%.1..33%
44.31.0.50%%
3..56.1.144%
11.86.13.0%%
Table 31: (Details of Fig. 14b Test accuracy of Food-101 dataset applying symmetric label corruption, where weight decay value
10
Distillation
(Tencher)
(3.46620003%
61:81=#0.124
35.06-:0223%%
53. 35.10=00.244
49.
18:24400.45%
63.92:10.05% %
62.27710.32%
59 50.86690.00%
56.9610.114%
53.22=2.40%
51.15:00.8%%
43.4721.81%
21.67:14.58%
63. 63.28.10.00 63.28.10.08%
62.13.10.66%
59.7110.48%
56.9110.1446
53.2112.415%
51.1210.34%%
4..48.1.82%%
21.63:1-1611%
63.0340
62.01110.20
00.59..0.11%
56. $6.8510.133 13%
53.1612243%
51.09110.30
13.49111.825
21.61.1-4.63%
62.96:50.08%
33.3%
59.53:10.50%
56. 56.81 13%
58.1262.45%
$1.03:00336
43.47=1.81%
21 21.60%:4.62%
63.36:0.144%
61.65=00522%
50.90=
58.00:00.19%
54.21=2.77%
53.98:00.0%%
47.61=2.34%%
23.58:8.33%
Table 32: (Details of Fig. 14b) Test accuracy of StanfordCars dataset applying symmetric label
corruption, where weight decay value X = 2 10-4
Distillation Step
0.0
0.1
0.5
0.6
0.7
0.8
(Tencher)
42.62:40.096
54, 14.19
24.50-0.43%
15.53-40.28
11.61
00.
8.43=0.11%
2.38+0.1155
2
42.97:10.154
35.63:00311%
17.45
18.63:00.44%
13.95
44%
9.26==0.84%
5.43:40.41%
2.57:00.1 17%
3
13.03:10.18%
0.37%
0.7%
19:21 7..7%%
14,26
55%
9.35.0..8%
5.62100.5%
2.73:...114
42.92:+0.09%
37.40=0.575 57%
28.70-0.9555 9.5%%
19.31-0.84%
14.29
59%
9.20-40
5.70++0.49%
2.74+0..15 12%
42.82:4.007%
37.62:00.43%
18.79
19.3110.94%
.44%
9.27=
9.74::0.42%
2.76=00.1%%
PLL
41. 41.30::0.06/
0..69:00.02%
27. 27.86.00.699
19.14:10.00%
14.58:2.188%
9.29::1.35%
5.43:10.4093
2 20%
47

---

Published as a conference paper at ICLR 2025
Table 33: (Details of Fig. 14ch Test accuracy of Caltech- 101 dataset applying asymmetric label corruption, where weight decay value À = 2 10
Distillaton Step
D.D
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Taacher)
95.01
93. 33.59900.20%
91.05=00.1%%
$7.6920.4416
85.0k:00.71% 71%
80.17::1.07%
70.74::0.85%
45. 15.16:2:.944%
05.10..0.05% %
93.66.10.34%
91.34.1..5%%
38.23110.661
2..2%%
80.57.110085
71.61
46.39111466%
93.5110.34%%
91.47.10.50%
88.33.10.444 44.44%
15..3%
46.43.11.495
94.95:10.055%
93.51 30%
01.42:00.009
88:25::04423
85.77::0.299
80.26:1 1.3
07..088%%
46. 16.35:11
+4.91-+0.47%
93.53-+0.24%
91. 01.36++0.544 549%
88.25+0.455%
85, 85.70-00.22%
80.18++1 10%%
71 73.00.6666%
46.37 46.37++1.515
PI.1.
93.82++0966%
92.72+00.0%%
901 001-80+007398
89.82-10.615 5%
07.44%
81.36+-0.74%
72.45+2.411%
4.469++2.66%
Table 34: (Details of Fig. 14ch
Test accuracy of Caltech- 256 dataset applying asymmetric label
corruption, where weight decay value X = 2 x 10-5
Distiliation Step
0.1
0.6
0.7
0.9
(Teacher)
53.63.10.01%
81 1..5510.119 19%
78.51.10.20%
74.3410.188%
11.22.00.2%%
66. 16.50.10.335 10.1%%%%%
57.42.10.71%
15.12.11.0%%
03.6K:00.08 5
81.86-10.100
70.59:00.25%
74.80:00.49%
71. 0..26
66 74. 28%
57. 77.699:00886
35.16+11.16%
83.64++3.000
81.72+0.14%% 14%
78.53+00.279
74.77+00.55 52%
71.28-00444%
66 30%
57. 07-58-0-95%
1..10+
83.55-40.08%%
74.69=0.544%
21.17=00.4%%
33.50%
11.65
78.45:60.285
24.65:00.33%
71.14:20.41%
66.633:9.3155
57 57.43:
34.99
PL..
82.49100129%
80.37:0.199
75.88:00550%
74.0611.28%
70.82:00.87%
08.02.19.85
57.96 2.339
34.7012,493
Table 35: (Details of Fig. 14ch Test accuracy of CIFAR-100 100 dataset applying asymmetric label corruption, where weight decay value X
10-5.
Distillation Step
(Taacee))
70.35+0..12%
(5.40+0.115%
61.05++0.175 17%
54.78+00.2%%
50.68+0.299
44 18.38++0.00 36%
34. 14.81+00.039%
18:21++0.250
11.45=00188%
60.13=+0.60%
61.27++0.31%%
64.15=0..88
02.20=00.288
58.23+t0.19 19%
51.96=001134
54.15=t0.885
71 31.59:33.2445 9
69. 30:3004999.
67.36:59.27% 27%
64.48:0.13%
3m
62.26:10.29 278
58.49-50. 18%
52.37-50.18% 18%
35.27:51.0288
71.63+0.23%%
69.37+0.50%
67 07.56+00.25%
64.53+00.10 10%
62.27-+0.26%
58. 18.56-+1.21%
2.2.7.
35.53+11.024
18.6240.18%
50..0.
67.38+0.21%%
64.5440.10%
62.28+0.28%%
58.57-40.235
52.51=0.22%%
PL..
69. 20.30+0..0%%
67. 67.25=00.49%
67. 67.1240.22%
64.88=0.116%
63.10:+0.02%
60.40=00.22%
55.71=0..16%
38.92+1.46%
Table 36: (Details of Fig. 14ch Test accuracy of Flowers- 102 dataset applying asymmetric label corruption, where weight decay value \
10-4
Distillation Step
(Teacher)
88.01 +0.17%
79:97:0.11 17%
66:33:1.099
49.77::1.49%
12.84=1.188%
32.19::1785%
20.62:00112
10.33:110000
87.75+01.99%
81 61.96-00.66%
72.58-+0.74 74%
59.31+3.68%
51.43+3.19%
44.1+2.78%
28. 28.53+33129%
13.59+1.115%
87.75+00.244 24%
82.45=00.85%
74.02+ 1.56%
3.3%
54.58+2.39%
45. 15.29942.7496
30.00-+2.588%
14.87++1.611
87.55::0.16%
1.03%
74.61:2.04%
3.2
24.97:22.05%
15.46%:1.86% %
30. 2.1499
15.00:1.8865
17.0910.30%
92.48.1..21%
74.58.12.155
62.55.3.200%
55.0311.89%
45.2312.622%
30.49.2.47%
PLL
85.233.0.18%
78.37..0.72%
70.23.11.09%
59.7713.86%
54.71.2.20%
44.8011.85%
28.76.11.96%
15.5911.45%
Table 37: (Details of Fig. 14c
Test accuracy of Food-101 dataset applying asymmetric label corrup-
tion, where weight decay value
x 10-5
Distillation
(Tencher)
(3.4666:003%
61.74:00.16%
38.06:20.000 10%
10%
49 59,49=0.1124
43.97:40.21%
18.46+40.38%
63.92:10.05% %
62.3110.06%
60.00:0.22%%
33.8
53. 53.8%
50.90110.41%
41.82233.47%
22.22:.3.52%
63.28-10.088
62.1910.08%
59.9510.19%
45.79 41%
53.8510.56%
$0.8610..46%
41.80 11.80%
22.21113.49%
63.1310.044%
$2.0710.001
19.8910.15%
53 53.50100539
90.82199415%
41.7813.499
(2.20.13.499
62.96::0.08%
61.93:90.01%
59 19.844.0.16%
55.71211424%
53.79:0.554%
50.83:60.45%
41.77=3.51%
22.20:t3.49%
63.36:0.114%
61.74:0.211%
50.97:0.25%%
56.55=1.52%
55.59=00.88%
53.78:0.23%%
45.67::4.6%%
24.38:5..8%%
Table 38: (Details of Fig. 14ch Test accuracy of StanfordCars dataset applying asymmetric label corruption, where weight decay value À 2 10-4
Distillation Step
0.0
0.1
0.5
0.6
0.7
0.8
(Tencher)
42.62:40.096
24.35-0.31%
15.23-40.00
01.51
8.59=0.09%
5.07+40.2199
0.96++0.085
2
42.97:10.154
35.63=00222 24%
0.504
18.6540.664
12.42-0.67%
9.16=50.88%
5.19:0.35%
2.85-0.166%
3
03.03:10.18%
0.378
0..9
19.12
12.50_0..46%
9.29
5.29110.228
2.55:10.244
42.92:+0.09%
37.33-0.274
28. 2222----8225
19.19-1.0%%
12.56-0.79%%
9.45=11388%
5.29+0.433%
2.5%+0.211
42.52:0.07%%
37.62:0.38%%
1.797
9.52&1.37%
5.30::0.42%
2.5
PLL
41. 41.30::0.06//
34.85:00400%
28. 0.:300008..
19.00:00.0%%
12.33-0.93
8.49:11355%
5.22:10.62%
2.47:0.25%
48

---

Published as a conference paper at ICLR 2025
Table 39: (Details of Fig. 15a) Test accuracy of Caltech- 101 dataset applying superclass label corruption, where weight decay value X = 10-3
Distiliaion Step
D.D
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Tencher)
35.08+0..038
33,49--0.255 25%
90.46 505
84.57-00.835
63%
78, 08.70
67.78++1.17 117%
47.45+t1.70
16.44-5009995
95.14-0.15% %
93.91+0.32%
91.9%%0.22%%
35.56-15 85.56+0.779%
80.01-0.71%
19.80+11.177 1.7%
19.65-11.700 1.76%
18.4++1
95.
93.89=0.2255
85.81+1.00%%
1.21%
1.8%%
95.05-1:0.05%
93.91
90.78::0.225
$5.9321.022%
80.45
69.45100.665
10.10
18.84:21.965
94.93100008%
93. 93.80100.21f
90.75110.19%
86.00_1.0445
30.59.10.85%
69. 19.64.10.77%6
50.12 1.85%
18.89.11.911
PI.1.
94. 34.35++0.16%
92.68+0.50%%
89.73+00.5258
85.52+2.40%
01.20--0.65%
70.91-1.00%
51.50+22.92%
20.01+ 1.66%
Table 40: (Details of Fig. 15a) Test accuracy of Caltech-256 dataset applying superclass label corruption, where weight decay value X = x 10-4
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
83.65==0..33%
01.73=0..256
71.97:00.24%
72.64=00.3%%
69. 69.58=00.07%
52. 52.62=00.75%
28.39:
%
82.0110.11%
18.17100.18%
72.9220.11%
69 14%
64.0410.6645
53.04-10.92%6
28.733: 1.81%
15.59++3.099
$1.95+00.23%
78.13+0.119%
72.96+0..14%
69.91-0.18%
64.09+00.66 64%
53.05+1 .000%
28.75++ 800%
83.59:=0.066%
72.96+0.13%
69.91=0.22%%
64.09:0.6115
28.78+11.9%%
83 83.55:5:.044%
$1.81:0.2226
78.0240.12%
72.93.0.17%
69.9410.26%
64.03::0662%
53.07
28.65:&1.81%
PLL
82.64::0.25%
80.20=0.2996
16.08:11.6%%
71.65:0997%
69.61=1.70%%
65.36:11.9%
1..34
28.67:42.73%
Table 41: (Details of Fig. 15a Test accuracy of CIFAR-100 dataset applying superclass label
corruption, where weight decay value X = x 10-4
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
71:20+0..22%
66.72+0.199 19%
61.01++0.165 16%
53.95+0.22%%
48 48.46+0.006%
41.28-+0. 10%
29. 81--0.366%
13.89+00.27%
2
72.25=+0.066%
69.67=00115%
66 16.36::-.299.
62.48=0.15%% 15%%
59. -.0%0
53.96=t:03315
43. 43.38%00.28%
22.69:+00.818
72.441:0.03%
69 18%
86.68:10.145
62.68
59. 09.65
$4.12220.365
23.33::0.67% 64.66
72.47.10.11%
29..66.0.26%
99.6010066%%
$1.27110.228
23.60110.721%
72.45.10.14%
70.0410.22%%
66 66.85.11..11%
62.85.10.179 17%
59 09/68.15 00.68..0.59%
54.40.10.322
44. 14.55.10.69%
23.74.10.77%%
PLL
69.90.1..12%
68.57.10.21%
67.50.10.27%
6..65..0.30%
62.26.10.38%
36..36%
49.89.10.71
28.20.11411%
Table 42: (Details of Fig. 15a Test accuracy of Flowers- dataset applying superclass label corruption, where weight decay value
1 10-3
Distillation Step
(Taaceer)
88.241.0.66%
41.83::0.41%
70.20110:22%
55.36111.00%
46.60:1.77%%
35.72&11.55%
24.97
10.07:10.18%
87.97.10.09%
83
$3.82.10.35%
75.7510..3%%
61.3119.63%
55.78112.44%
45.72.12.78%
12.25
12.39.11.0%%
$7.61.19.23%%
10.511 51%
17.8110..33%
1.039
57.4212.138
48.2411.200%
33. 13.999.
13.3311144%
14%
84.61-10.35%
76.24-10.565 56.6%
39%
809
49.05:1.55
13.86-1:1
86.6010.112%
$4.6110.42%
78.5910.733%
68.4311.20 1.27%
1.85%
48.9511.485
33.86%
1.69%
PLL
85.56110.46%%
81.11
79.02..0.94%
72.48.22.64%
64.8712.766
54.3111.04%
41.47.1.25%%
15.2312.53%
Table 43: (Details of Fig. 15a Test accuracy of Food-10 101 dataset applying superclass label corruption, where weight decay value
10-4
Distillation Step
(Teateer)
63 63.6010..3%%
61.95:10.129 12%
58 08.22.10.25%
52.88.10.249
48.83:10.119 11%
43.01.10.11 13%
15.88.10.27% 33.23:10.55%
63.66 63.66-10..15 %
62.41+0.119%
59 90.66-0.3195
56. 56.24+00189%
51.45-0..11%
49.62+0.28%%
59.57+3.93%%
21.95:+3.54%%
63. 63.38.10.066
62.2710.15%
59. 99.9910.29%
56.2510.211%
53.47110.23%
89.70 69.7010.311% 31%
39. 59.74.13.96%
22.18.3..488
63.2 33.27.0.044
10..7....77 17%
59. 79.70.0.28%
56. 36.26.2 100.11
53.49 1...2..0
69.76.10.20
39. 1..045
22.33.13.74%
63.12:80.04%
30.65:102265
53.51::0.22%
39. 39.95-1-40000
22.45:13.81%
PLAL
63.41:10.03%%
61 01.82:50.1559
60.10:0.41%
07.15:.0.11% %
56.39.0.33%%
53.261.0004%
63.5%:6..1%%
27.0617.03%%
Table 44: (Details of Fig. 15a Test accuracy of StanfordCars dataset applying superclass label
corruption, where weight decay value À 1
Distillation
0.1
0.3
0.7
(Teacher)
20.05+0.04%
19.7740.22%
15.44-0.29%
11.40.9.99%
6.77=0.139
3:16-00275%
2
42.87 10%
39.95-0.09%
32. 25.
20:00:0.254
18.7 3..36
13.75=10.09%
8.31=0.18%
3.47=004344
3
41.74..0.0 12%
38.5410.36%
32.83-00.54%
24.2810.26%
19.2010..00%
14.58.10.05%
8.64=
3.57-10.53%
40.37+00
37.44 34%
$2.49+0.55%
24.34=0.279:
19.21+0.87%
14.70+0.16%%
8.69-0 0.0%
3.54-0 545
39.03sa0.1
36.20:40227%
31.7620.58
23.91:4037%
19.05
14.61:40200%
8.59
3.46=0.555
PLL
40.60:403364
35.9644020%%
30.3240.45%
23.65:0.45%
19.43=0.75%%
15.18:00.55%
8.80=:0.12%
3.26=0.624
49

---

Published as a conference paper at ICLR 2025
Table 45: (Details of Fig. 15b) Test accuracy of Caltech-10 101 dataset applying symmetric label corruption, where weight decay value X = 10-3
Distiliaion Step
D.D
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Tencher)
05.08
93.49-00165 16%
91. 01.13+00,425
57.96 19%
54 60:00.5666
80-80-5005555 55%
69. 40.74-:0..11%
46 36.04---0.600 600%
95.14-0.15%%
93.74-0.23%
88. 47%%
85 15.60-0.6%%
81,45+00. 36%
71.31+0..60
47.77+1.18%
95.
93.63=0.332
91.92+0.31%%
58.09=00665%
85 65.81=0.900%
71.58=0.85%
95.05-1:0.05%
93.53-20.26%
91.97:10.29%
88.755:0.67%
86.0610.88%
81.57.:0.00 38%
71 76766
18.43:11.48%
94.9310.008%
93.4910.12%
91.97.10.14%
00.692 %
86.06.1 1.04%
$1.55:10.279
71 1...7%
48.4811.445
PL.1
94. 14.35++0.16%
92.72+00.55%
91.51++0.57%
88.61+0071%%
85 25.6242.285%
82.53+-0.28%
73.64-1..58
51.69+3.19%
Table 46: (Details of Fig. 15b)
Test accuracy of Caltech-256 dataset applying symmetric label corruption, where weight decay value X = x 10-4
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
83.65==0..33%
81.77=0.13%6
78 18.5%
74.50:=0.299 29%
11.48:00.13%
66.48::0.744% 74%%
57,47=00.44%
16.%
%
02.031..334%
0..75
71 11%
66.81100755% 75%
57. 033%
34.3910..1%% 13%
08.59++3.099
51.91+00.31%
78.77+00 40%%
74.75+0..27%
71.70+00.66%
66.78++0 70%%
47.45-00.28%
34. 14.36+00.14%
83.59:=0.006%
78.76+00.775
74.70=00318
71.63=0.195
14.3240.16%
83 83.55:5:.:44%
81 31.822003326
78.71:00.39%
74.64-20.27%
01.57:0.200%
66.87.10.6995
57.43:60.33%
34.25100188%
PLL
82.64400.25%
80.41=0.71%6
76.47::0.44%
73:29=0.6556
70.5920.18%
6..80::1006%
0..13
32.47:
Table 47: (Details of Fig. 15b) Test accuracy of CIFAR-100 dataset applying symmetric label
corruption, where weight decay value X = x 10-4
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
71:20+0.022%
67.18+0000%%
62.92+0.375
57.28+0.145%
53.12+0..7%% 177%
47.33-+0.033%
37. 37.33-+0.28%
19.89++0.13%
2
72.25=00.06%
10.21=0.12%%
67.52:+0.09%%
14.52=0.11% 11%
62.35=00.25%%
58. 58.50++0.300 30%
51.96=00.688
34 43.922t-..47%
72.44100003%
10.45:00100% 10%
67.721:0.06%
12.
62.53
58.73::0.22
52.29
35.99
72.47.10.11%
70.56.10.17%
67.7310.08%
68.82.0.18%
62.601.0.24%
38.88.10.23%
1.0.8%
12.45.00.14%
70.6310..1%
67.72.10.13%
64.87
62.67.10.24%%
58.95.10.27%
52. 1...5%
36. 36.58.10..44%
PLL
69.90.0.112%
69.13.10.18%
67.93.10.10%
65.76.00.9%%
64.08.0..11%
61.4110.11%%
57.02..0.19%
41.18.1..2%%
Table 48: (Details of Fig. 15b)
Test accuracy of Flowers-102 dataset applying symmetric label
corruption, where weight decay value
10-3
Distillation Step
(Taacber)
88.24110.16%
51.57
89.77::10045 1.4%%
55.42#2.00%
48.06:21.45%
37.97.1..23%
25.2910.21%
11.60:1.53%%
87.97.10.09%
1..9%
76.80 2.%
66.3712.47%
58.33.12.16%
48.66.11.000
34. 14.28.0.68%%
14.97.2.228%
87.61110.23% 23%
7..77..0.769.
78. 10.2000040
67.3512.009%
20.27.2.009
51 1..3..1335
15.52.11.85% 3..6221..239
14%
78 73%
67.56+2.004
62.81-2.767%
52. 900%
15.7811.7455 14.14791.19.
86.6010.112%
$4.67.10.46%
78.66.10.70%
63.24.12.94%
36.31.11.13%
PLL
85.56:10.465%
81.4110.44%%
79.7410.32%
73.56.1.511%
66.7312.47%
5..6711.68%%
42.71.2.37%
17.9714.24%
Table 49: (Details of Fig. 15bb Test accuracy of Food-100 dataset applying symmetric label corruption, where weight decay value
10"4
Distillation Step
(Teatber)
63 63.6010..33%
62.08.10.099
58. 06.54.00.22
53.82:10.219
50.2110009%%
44.87.10.10 19%
16....10.55%
18.91.10.23%%
63.66 63.66---..65% %
62.61-0..9%
60.20-00.03%
56.98-11 16.9--0-19%%
54.72-0025%%
51.24-H0.31%
44.59
28.08-+00.0 19%%
63. 63.38.10.06%
62.4810.07%
60.0910.044
57.01 10.08%
10.8%%
51.31.10.35%
44.82 14.82.00.0%%
28.30.00.00%
63.2 9
62.39 1...09
59.9610..0%%
56.98.10. 10.08%
54.75.0.21%
51.32.10.35%
44.90 1..000.0.06%
28.48100.19%
63.12:80.04%%
62.31
56.
45.
28.56:1.0..
PL.L
63.41:10.03%
61.98:20.189.
60 60.16:10.07%
57.98:20.25 %
56.22:10.0%%
54.04:00.1%%
49.17:20.10%
34.731.0.88%
Table 50: (Details of Fig. 15b) Test accuracy of StanfordCars dataset applying symmetric label corruption, where weight decay value À 10-3
Distillation
0.1
0.3
0.7
(Teacher)
28.95+0.27%
19.9240.11%
15.40:0.24%%
6.30=00299
2.79-0.175
2
42.87:00. 10%
39.15-0.11%
31.7940.21
21.35.0.21%
18.36.0..8%%
13.52-10.33%
7.74=0.40%
3,20
3
41.74.10.1 12%
38.8710.02%
23.24
24.2010..14%
19.25.0.25%%
14.16:0.5000
8.09_
3.34=:0322:
40.37+00
38.0740.0256
31.96+40.44
34.11+00.07%
19.47+0.345
14.31+0.52%%
8.24-0.67%
3.35-0 36%
39.03sa0.1
36.91
31.12+0.21%%
23.7240.124
19.24:80.14%
14.15:10.88%
8.20=
3.39:
PLL
40.60:40.366
36.2040.09%%
30.18:50.11%
24.00+0.18%%
20.28::0.31 %
15.37:00.11%
1.93%
3.73::0.11%
50

---

Published as a conference paper at ICLR 2025
Table 51: (Details of Fig. 15ch Test accuracy of Caltech- 101 dataset applying asymmetric label corruption, where weight decay value X = 10-3
Distiliaion Step
D.D
0.1
0.3
0.5
0.66
0..
0.8
0.9
(Tencher)
35.08+0..338
93 33.64-50 23%
91. 11.13+00.245
57 17.69-0.411%
55.0880.7118
80.52:+1
70.97=00.555
45.18+-0.949
95.14-0.15%% %
93.74+0.275
%
11.65+00.61%
88. 42%%
86 86.04-+0.366
72.04-+0.60P
1.32%
95.
93.57=0.33
56.56.90
86. 26.6.6%
88%
72.24-00.00
95.05-1:0.05%
93.55:20.244
2-..919 91.67:10.633%
06.39
80.86:2:0.8 87%
02.85
47.41:11.21%
94.9310.008%
93.57.10.2%
91.1110.69%
88.61 100409
06.24.1...88%
80.84.10.953
02.2010
47.4811.266%
PI..
94. 14.35++0.16%
92.70--0.12%
90.78+00.71%
89.67-0..66%
87.23+1..6%%
81.32+00.2%%
72.47+22.65%
48.68++2.60%
Table 52: (Details of Fig. 15ch
Test accuracy of Caltech-256 dataset applying asymmetric label corruption, where weight decay value X = x 10-4
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Teacher)
83.65=90..3%%
81.65=0017%%
78,
03.500+-.2205
74.38=00.8%% 18%
71.22=0.28%
66 16.54400.334
57.43=#0.33%
35. 15:15:t1 1.0%%
%
01.8810.10%
%
14.47
71.3320.411%
66 66.833:0.22 27%6
57.79 00.799.0.89KK
16.281:116%%
08.59++3.099
81.74-+0.14% 14%
78.63+0..275 27.%
71.41-0.444%
66.80++0.25
57.70+ 1.05%
35.27+1..50
83.59:=0.006%
74.8==0.51
71 41.45%
57.7241.09%
3..27411315%
83 $3.55:50.045
81.71:0.11%6
78.54:60.2955
34.88:60.51%
71.50:10.40%
66.79:10.255
17.71
18.27:
PLL
82.64:00.25%
80.41=0.21%6
75 25.86::0.49%
73.98&1.2296
20.80=:0.945%
68.01:0090%
$7.83=2..6%
34.6042,4156
Table 53: (Details of Fig. 15ch Test accuracy of CIFAR-100 dataset applying asymmetric label
corruption, where weight decay value X = x 10-4
Distillation Step
0.0
0.1
0.3
0.5
0.6
0.7
0.8
0.9
(Taacher)
71:20+0.022%
17.%%
63. 12440.166
57. 57.15+0.11%
53.07+0.16%
46.98+90.22 24%
37. 11%
19.93+0.31%%
2
72.25=00.06%
70 10.38
67.68:-0.4009
18%
02.22=00.08%
58.61+00.549
52.52=00.42%
34. $4.98=+0.37%%
72.441:0.03%
70.74:0.199
67.762:0.40%
133%
62.3620..05%
58. 38.86690.51%
53.07=00.28%
36 16.28::0.17%
72.47.10.11%
16..5....16%
61.7510.17%
53.28110.2%%
36.58.10.11%
12.45.0..14%
70.89 1.15%
67.75.10.36%
64.81100.33%
62.52.10.12%
58.98.10.51%%
53.37.10.18%
36.83.10.10%%
PLL
69.90.1..12%
69.20110.05%
68.00.10.33%
65.71.0..9%
63.85 1.0.33%
61.62.90.45%
57.10.1.110%
41.51.1053%%
Table 54: (Details of Fig. 15c
Test accuracy of Flowers- dataset applying asymmetric label corruption, where weight decay value = 10-3
Distillation Step
(Taacber)
88.24110.16%
81.14:10.12%
70.62::1332
56.08.:2.00%
19.0%
38.26:11.44%
24.93:1.644%
12.09
87.97.10.09%
10.%%
76.80100.57%
66 36.20%33.26%
60.33.1..80%
89.97.33477%
$6.08
17.03.11.42%
87.61119.23%
84.1510.26%
78.04.10449
68.0712.8888 2.88%
62.06
51.14.13.14%
36.6711.749)
17.7511.229
14%
84.2%-0032%%
16.56-90.65%
2.66%
51.37+43.15%
36.
86.6010.122%
84.22.10.52%
51.4713.47%
18.1711.19%
PLL
85. 85.56:10:46%%
81.27.10.11%
79.67.00.0%%
74.93.2.52%
69.48.1.30%%
59.9311.54%
45.2911.47%
20.5912.17%
Table 55: (Details of Fig. 15c Test accuracy of Food- 101 dataset applying asymmetric label corrup- tion, where weight decay value
x 10-4
Distillation
(Teatber)
63 63.6010..33%
62.02.10.169
58.53.10111 11%
53 53.6010.11
50.25:10.15%
44.9510..00
35.84-10.00%
19.06.10.399
63.66--0.15%% %
62.50-t0.01%;
1..7:-..99
57.03+001144 14%%
54 14.44-0.249%
51.08-+-0.44%
44.21=0.200%
28. 16.
63.38.10.066
62.4710.05%
$6.98.0.111
$1.6810.22%%
51.1310.47%
14.2%%
28.46.10.80%
63.2 33.27.00044
62.34
59.92100330% 30%
56.96 0639610..00 1.0.1% 12%
51.6710.17% 17%
51.1510.42%
44.49100288 28%
28. 28.591.0.81%
63.12:80.049%
62.27:10.069 0600%
18%
1...2%
26.73:10.87%
PL.L
63.41:10.03%%
61.95::0.128
50.93:30.11%
57.94:0.12%
56.341...3%%
53.97.1..29%
49.18:00.23%
36.361.1.76%
Table 56: (Details of Fig. 15ch Test accuracy of StanfordCars dataset applying asymmetric label corruption, where weight decay value À
1
Distillation
0.1
0.3
(Teacher)
28.72+0.3655
15.18++0.222
11.01:0.13%%
6.18-0.20%
2.71
2
42.87:00. 10%6
39.03-0.06%
31.85:102225
23.23:60.00
18.52:00.47%
13.75:10.28%
0.71=0.25%
3.27 16%
3
41.74.10.1 12%
38.66.10088%
32.29-6..88%
24.1240.324
19.12.0.622%
14.66.0.11
7.95
3.30 233%
40.37+00
$7.75+0.10 12%
14%
24.08+00.21%
19.06
1490+40.09
7.91-0.299
3.32-0 26%
39.03sa0.1
36.6240.100
31.2440.20%
23.8240.25%
18.78:00.72%
14.69:20.88%
7.84=0.40%
PLL
40.60:403364
36.0240.19%
30.40:00.2488
24.64:00.99%
19.52:0.66%
15.30:0.44%%
0.68:-0.3%%
3.36::0.32%
51


Reviews:
[{"review_id": "TO9MF2nwKD", "reviewer": "Reviewer_UkvB", "summary": "This work aims to analyze self-distillation in linear probing with neural network feature extractors. The contributions of this work are threefold: (1) the analysis reveals that self-distillation effectively performs label averaging among instances with highly correlated features when generating predictions on the training data; (2) the analysis quantifies the number of distillation rounds needed to achieve 100% population accuracy in the presence of label noise; and (3) based on the theoretical analysis, the authors introduce a novel self-distillation approach that achievers similar benefits of multi-round distillation in a single round.", "questions": "1. The second contribution of this work is to determine the number of distillation rounds required to achieve 100% population accuracy in the presence of label noise. On first reading, the term \"100% population accuracy\" may not be immediately clear. Introducing a brief explanation of this concept in the first place would be beneficial for readers who may be less familiar with the subject.\n\n2. Could you explain the intuition behind the condition (Eq. 10) in Theorem 4.1? At first glance, it seems almost implausible to reduce training losses to zero in the presence of label noise, even with multiple rounds of self-distillation. \n\n3. It seems that the terms “true label” and “given label” are not properly defined. As I understand it, the “given label” refers to the target label, which may include noise, while the “true label” represents the oracle.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Solid Presentation\nOverall, this manuscript is well-written and polished. The introduction provides a satisfying overview of the rich theoretical analysis, which is a strong aspect of the work. Some parts of the analysis are somewhat difficult to follow, but experts in this specific field likely won’t struggle to understand the details.\n\n2. Interesting theoretical analysis\nOne of the main theoretical contributions of this paper is demonstrating that the effect of self-distillation can be interpreted as label averaging among highly correlated instances. This interpretation offers a new perspective on the mechanism of self-distillation, even in the absence of feature evolution. Initially, I thought this was merely a special case of Allen-Zhu and Li’s study. However, after reviewing previous work, I discovered a notable difference in how the effects of self-distillation are analyzed, which I believe is worth sharing with the community.\n\n3. A New Approach to Converting Multi-Round Self-Distillation into a Single Round\nThe proposed method takes a different approach by directly using the refined partial labels derived from the teacher's outputs, achieving the same benefits as multi-round distillation in just one round. This is especially appealing, as one of the main drawbacks of self-distillation is the need for multiple rounds, which makes training very inefficient. A single-shot method is highly desirable, and we hope this approach will be widely adopted for training classifiers in noisy label settings.", "weaknesses": "1. Most sections are well-written, but some parts would benefit from clearer descriptions or revision. Please refer to the section below.\n\n2. The experiments are somewhat weak, but this is understandable since this work is more theoretical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OgJrfHX1s5", "reviewer": "Reviewer_XuSH", "summary": "This paper presents a theoretical analysis of the mechanisms behind self-distillation in a linear probing setting. The analysis reveals that after $t$ rounds of self-distillation, the model's predictions converge to a weighted average of the provided labels, with the weights determined by the Gram matrix. Building on this finding, the authors investigate the effects of label noise and the efficiency of the self-distillation method. Experiments demonstrate the effectiveness of proposed single-round self-distillation method.", "questions": "See my questions in weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper is well-written, particularly in Section 2, which formulates self-distillation and provides a clear overview of the results.\n* This paper presents an interesting result in Theorem 2.1, establishing a connection between the predictions of the $t$-th distilled model and the given (possibly noisy) labels.", "weaknesses": "The main weakness about this paper is the significance of research problem.\n\n* This paper focuses on self-distillation with linear probing and provides a theoretical analysis in this context. I believe that both self-distillation and linear probing are valuable techniques, but I am unclear about the purpose of combining self-distillation with linear probing. As far as I know, linear probing is widely used in self-supervised learning as a method to evaluate learned features. Why should we combine linear probing with self-distillation, especially in scenarios involving label noise?\n\n* The proposed theory assumes a fixed feature extractor and therefore cannot be applied to trainable feature extractors, which are more commonly used in task adaptation and transfer learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4N94TdySLJ", "reviewer": "Reviewer_g4HH", "summary": "This paper examines the multi-round self-distillation for multi-class classification in the context of linear probing. By approximating the softmax function as a linear function and considering the feature correlation matrix as a low-rank structure, this paper derives a quantified closed-form solution for the output of the $t$-th student model, showing the effect of self-distillation can be interpreted as label averaging among highly correlated instances. The authors then derive the conditions of the label corruption matrix achieving 100% population accuracy for multi-round self-distillation, in the context of balance and superclass corruption. The authors further prove that distilling teacher's top-2 outputs enjoys better theoretical properties. Extensive experiments are conducted and show consistency with the proposed theories.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed theory is solid and valuable, especially providing insight into the effectiveness of distilling with partial labels.\n- The authors provide numerical and visual analysis of the approximation of the softmax function and the feature correlation matrix, which is reasonable and convincing.\n- This paper is well-organized and clearly written.", "weaknesses": "In the experiments, Generalized Cross Entropy loss is applied for PLL which differs from the setting of CE loss in the main theory. The authors need to explain whether applying GCE loss is fair for comparison or provide experimental results of CE loss for PLL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eIrm9THUln", "reviewer": "Reviewer_CZ26", "summary": "In this paper, the authors proposed introduce a single-round self-distillation method using refined partial labels from the teacher’s top two softmax outputs, referred to as the PLL student model. This approach replicates the benefits of multiround distillation in a single round. Experiments on several datasets demonstrate the proposed method is effective.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The analysis of self-distillation in linear probing reveals that self-distillation effectively performs label averaging among instances with high feature correlations when generating predictions on the training data.", "weaknesses": "Here are a few questions I would like to ask the authors:\n\n1. What are the clear insights or significances of Sections 2.2, 3, and 4 in relation to the proposed method in Section 5? Could the authors summarize the roles and conclusions of Sections 2.2, 3, and 4, and their implications for Section 5? These sections are not sufficiently clear.\n\n2. In Section 5, the authors state, \"selecting the top two labels with the highest values and assigning a weight of 1/2 to each, setting all the other entries to zero.\" What is the rationale behind this approach?\n\n3. The experiments use ResNet34 as the backbone for linear probe experiments; however, training ResNet34 with all parameters is not particularly time-consuming or resource-intensive. Should a backbone with a larger parameter count be chosen for the experiments?\n\n4. When comparing existing PLL methods, the authors claim, \"Our method differs by directly employing the refined partial labels derived from the teacher’s outputs, achieving the same benefits as multi-round distillation in just one round.\" Does this imply that other methods incorporating the teacher's output could achieve similar or even better results? Are there corresponding experiments to support this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)