Title: Learning From Biased Soft Labels
Venue: NeurIPS 2023
Accepted: yes
Keywords: Soft label; knowledge distillation; weakly-supervised learning; Machine learning.
Abstract: Since the advent of knowledge distillation, many researchers have been intrigued by the $\textit{dark knowledge}$ hidden in the soft labels generated by the teacher model. This prompts us to scrutinize the circumstances under which these soft labels are effective. Predominant existing theories implicitly require that the soft labels are close to the ground-truth labels. In this paper, however, we investigate whether biased soft labels are still effective. Here, bias refers to the discrepancy between the soft labels and the ground-truth labels. We present two indicators to measure the effectiveness of the soft labels. Based on the two indicators, we propose moderate conditions to ensure that, the biased soft label learning problem is both $\textit{classifier-consistent}$ and $\textit{Empirical Risk Minimization}$ (ERM) $\textit{learnable}$, which can be applicable even for large-biased soft labels. We further design a heuristic method to train Skillful but Bad Teachers (SBTs), and these teachers with accuracy less than 30\% can teach students to achieve accuracy over 90\% on CIFAR-10, which is comparable to models trained on the original data. The proposed indicators adequately measure the effectiveness of the soft labels generated in this process. Moreover, our theoretical framework can be adapted to elucidate the effectiveness of soft labels in three weakly-supervised learning paradigms, namely incomplete supervision, partial label learning and learning with additive noise. Experimental results demonstrate that our indicators can measure the effectiveness of biased soft labels generated by teachers or in these weakly-supervised learning paradigms.

Learning From Biased Soft Labels

Abstract

Since the advent of knowledge distillation, many researchers have been intrigued by the dark knowledge hidden in the soft labels generated by the teacher model. This prompts us to scrutinize the circumstances under which these soft labels are effective. Predominant existing theories implicitly require that the soft labels are close to the ground-truth labels. In this paper, however, we investigate whether biased soft labels are still effective. Here, bias refers to the discrepancy between the soft labels and the ground-truth labels. We present two indicators to measure the effectiveness of the soft labels. Based on the two indicators, we propose moderate conditions to ensure that, the biased soft label learning problem is both classifier-consistent and Empirical Risk Minimization (ERM) learnable , which can be applicable even for large-biased soft labels. We further design a heuristic method to train Skillful but Bad Teachers (SBTs), and these teachers with accuracy less than 30% can teach students to achieve 90% accuracy on CIFAR-10, which is comparable to models trained on the original data. The proposed indicators adequately measure the effectiveness of the soft labels generated in this process. Moreover, our theoretical framework can be adapted to elucidate the effectiveness of soft labels in three weakly-supervised learning paradigms, namely incomplete supervision, partial label learning and learning with additive noise. Experimental results demonstrate that our indicators can measure the effectiveness of biased soft labels generated by teachers or in these weakly-supervised learning paradigms.

1 Introduction

Knowledge distillation [2, 17, 20] has achieved remarkable achievements in a wide range of applications. It has emerged as a popular paradigm for model compression [22, 21] and transfer learning [41, 34] by distilling knowledge from the big model (teacher) to the small model (student). Students inherit the knowledge of the teacher by imitating the soft labels generated by the teacher model. Many experiments show that learning from soft labels can be very effective, even surpassing learning from ground-truth labels [2, 17, 37]. However, there are still many mysteries why these soft labels are effective. Most existing theories do not apply to soft labels that significantly deviate from the ground-truth labels and lack explicit indicators to evaluate the soft labels [39, 58, 7, 35]. In this paper, we mainly focus on the effectiveness of the biased soft labels, especially large-biased soft labels.

It is significant to study the effectiveness of biased soft labels. Firstly, expert-labeled annotation typically requires a substantial investment of manpower and time. But the soft labels in weakly supervised learning could be cheap and biased [59, 46]. Evaluating the biased soft labels is instructive for weakly supervised learning. Secondly, these biased soft labels can apply for privacy protection

(a)

0

1

2

3

4

5

6

7

8

9

Class indexes

0.0

0.2

0.4

0.6

0.8

1.0

Values

0.9112

Bird

Others

(b)

0

1

2

3

4

5

6

7

8

9

Class indexes

0.0

0.2

0.4

0.6

0.8

1.0

Values

0.1867

Bird

Others

(c)

[15, 11]. When they are employed for subsequent training, they can conceal the ground-truth labels. It is consistent with the requirements of privacy and utility in privacy protection [27].

After realizing the significance of biased soft labels, we study the biased soft labels and reveal that large-biased soft labels can also teach good students. This phenomenon inspire us to seek indicators to measure the effectiveness of soft labels. As a result, we propose two intuitive indicators, namely unreliability degree and ambiguity degree . The intuition behind them is to convert the soft label into a topk set ( k is a constant), which contains the topk labels of the soft label. Unreliability degree is the probability that the ground-truth label is not in the topk set while ambiguity degree is the upper bound of the probability that the incorrect label and the correct label co-occur in the topk set.

Furthermore, based on the proposed indicators, we present moderate conditions to ensure that, the biased soft label learning problem is classifier-consistent [28] and Empirical Risk Minimization (ERM) learnable [6]. The former guarantees that the learners can converge to the optimal solution as learning from original labeled data. The latter implies that learners' performance can generalize to the entire data distribution. Our theory not only guarantees the effectiveness of biased soft labels, but also provides explicit indicators to evaluate the soft labels.

Biased soft labels are prevalent in weakly supervised learning [24, 44, 59], and our theory can offer theoretical insights for these paradigms. In detail, we apply the theory to three classic weaklysupervised learning paradigms: incomplete supervision [3], partial label learning [5] and learning with additive noise [10]. In these weakly-supervised learning paradigms, soft labels are biased and we provide a theoretical guarantee for the learners in these fields. In other words, we prove that the biased soft labels in the three paradigms are effective to train a good model.

It is important to note that, Yuan et al. [53] argues that poorly-trained teachers can teach good students. In fact, their soft labels are a mixture of the teacher's outputs and the ground-truth labels with a ratio of 0 . 1 : 0 . 9 , which makes the soft labels close to the ground-truth labels as illustrated in Figure 1(b). Differently, our experiments reveal that large-biased soft labels can also teach good students. Furthermore, the effectiveness of these soft labels can be measured by unreliability degree and ambiguity degree. In addition, experiments on three weakly-supervised learning paradigms also demonstrate that biased soft label learning problem is learnable and the proposed indicators are effective. Our contributions can be summarized as follows:

We find that learning from large-biased soft labels may also achieve comparable performance and intend to explore the underlying mechanisms behind the effectiveness of the biased soft labels.

Two indicators have been proposed to measure the effectiveness of soft labels. Based on the indicators, we present moderate conditions to guarantee the effectiveness of the soft labels. It is proved that the biased soft label learning problem is classifier-consistent and ERM learnable.

A heuristic method is designed to train skillful but bad teachers, i.e., teachers with low accuracy but who can teach good students. We can explain this phenomenon with unreliability degree and ambiguity degree.

The theory provides a theoretical view for the learners in three weakly-supervised learning paradigms. Specifically, we provide theoretical guarantees for the learnability of these paradigms from the perspective of soft labels. Experimental results are consistent with our theory.

2 Related Work

Knowledge Distillation and Label Smoothing Knowledge Distillation (KD) was initially proposed in model compression [22, 21] and then applied to transfer learning [41, 34]. There is growing interest in why distilling can transfer information and what the dark knowledge hidden in the soft labels is. Furthermore, the dark knowledge also exists in the variants of knowledge distillation, which are introduced in the appendix A.1. Label Smoothing (LS) [38] is a regularization method to improve performance by mixing the uniform noise into the ground-truth label. It is convinced that soft labels can restrain overconfidence of the student model. Essentially, both KD and LS can be unified as learning from soft labels Yuan et al. [53].

Label Enhancement Label Distribution Learning (LDL) [13] was proposed to exploit the label distribution to mirror the relationship between the label and the instance, where the formalization of the label distribution is identical to the soft labels mentioned above. In the remainder of the paper, we use nomenclature soft labels. Due to the high cost of labeling the soft labels, Label Enhancement (LE) [48] was proposed to recover the soft label from the logical label by exploiting the implicit correlation among different labels. Numerous novel algorithms have been designed in recent years that aim to improve the predictive model with the soft labels [47, 57]. Wang and Geng [42] applied the margin theory to the soft labels and designed the adaptive margin loss.

In fact, most existing explanations of the soft labels are empirically and experimentally validated, while the rigorous theoretical analyses usually have strong assumptions regarding the model or data distribution. Phuong and Lampert [35] explored the mechanism of distillation where the teacher model and the student model are linear. Allen-Zhu and Li [1] supposed that the instance could be decomposed into multiple independent features and had a linear relationship with the sample, and then proved the effectiveness of the soft labels. Wang and Yoon [43] solved the objective functional problem of self-distillation with the Green's function, which assumes that the network can reach the optimal solution. Menon et al. [31] and Zhou and Song [58] regarded the generated soft labels as the posterior probability and assumed the existence of the Bayes probability. There is also some work analyzing soft labels from the perspective of transfer risk [19, 18]. Most existing theories suggest that students can achieve good performance only when the soft labels are close to the ground-truth labels, and lack explicit indicators to measure the effectiveness of the soft labels.

3 Methodology

3.1 Preliminary

Let X be the instance space, Y = { 1 , 2 , . . . , c } be the label space with c classes and ∆ be the soft-label space over Y , i.e., ∆ = { d ∈ R c | ∑ i ∈Y d i = 1 , d i ≥ 0 for all i ∈ Y} . Define D as the data distribution over X × Y and H as the hypothesis space from X to Y . Each h ∈ H is called the learner or hypothesis. When h ∈ H is a neural network, it often employs a softmax function in the final layer and takes the highest value as the prediction. Slightly abusing notation, we use h ( x ) ∈ ∆ to denote the soft label generated by model h , and ˜ h ( x ) ∈ Y to represent the model's prediction.

̸

Next, we introduce some important concepts in the Probably Approximately Correct (PAC) learning [40]. The Natarajan dimension [32] was proposed to characterize the capacity of multiclass hypothesis spaces and we denote d H as the Natarajan dimension of H . The expected classification error of a hypothesis h ∈ H is defined as Err D ( h ) = E ( x ,y ) ∼D [ I ( h ( x ) = y )] , where I ( · ) is the indicator function. Given a finite dataset z , the empirical classification error is defined as Err z ( h ) = 1 n ∑ n i =1 I ( h ( x ) = y ) . The Empirical Risk Minimization (ERM) learner h ERM is defined as h ERM = arg min h ∈H Err ∆ z ( h | f ) , while the optimal learner h ∗ is defined as h ∗ = arg min h ∈H Err D ( h ) .

̸

Considering that this study involves two models: generating soft labels and learning from the soft labels. To avoid ambiguity, we use f to represent the model generating soft labels (teacher) and h to denote the model trained with soft labels (student). It should be noted that f ∈ H can be a neural network or a map based on certain rules, such as label smoothing.

In order to gain a deeper insight into the concept of biased soft labels, we have provided precise definitions for relevant terms below. These definitions may not have a direct correlation with the theoretical analysis presented in section 3.2, but they serve to elucidate the boundaries of our research.

Definition 1 (Bias of soft labels) . Give a dataset D consisting of n samples, the feature vector for the i -th sample is denoted as x i and the corresponding label is denoted as y i . Let f represent a model or a mapping rule. The bias of the soft labels generated by f on dataset D is

where f y i ( x i ) refers to the component of the soft label f ( x i ) that corresponds to the true label y i .

Definition 2 (Large-biased soft labels) . Soft labels generated by f on dataset D is called biased soft labels when Bias( f, D ) > 0 and called large-biased soft labels when Bias( f, D ) ≥ 1 .

The bias, as defined here, essentially represents the disparity between the soft label and the true label. Here, we employ the Manhattan distance, also referred to as L1-norm, although other metrics can also be suitable. A threshold of Bias ≥ 1 is employed to categorize soft labels as "large-biased." In practical scenarios, the choice of the threshold can be adapted based on specific contexts. After establishing the concept of large-biased soft labels, we proceed to define "bad teachers."

Definition 3 (Bad teachers) . We define f as a bad teacher if the soft labels it generates on dataset D are large-biased. Typically, D is the training set for f .

The definition of the bad teachers is based on the definition of large-biased soft labels. Similarly, here we adapt 1 as the boundary value for bad teachers. It's important to note that when we use "good/bad" to describe teachers, we are referring to the performance of teachers on the dataset D . On the other hand, when we say teachers are "skillful," we are emphasizing that they are adept at instructing students, leading to excellent student performance. Due to the fact that the performance of the student model is contingent on both the model structure and the complexity of the dataset, we are unable to establish precise boundaries for "skillful teachers." In this context, 'skillful' merely signifies that the teacher produces students with acceptable performance.

3.2 Theoretical Analysis of Soft Labels

In this subsection, we define two indicators, unreliability degree and ambiguity degree, to measure the effectiveness of soft labels. Based on these indicators, we present moderate conditions to ensure that the biased soft label learning problem is classifier-consistent and Empirical Risk Minimization (ERM) learnable. The underlying intuition behind the indicators is to convert the soft label into a set. Let Ω k ( d ) = { i ∈ Y | i ranks topk in d } be the set of top k labels in the soft label d . Here, k is a constant ranging in { 1 , 2 , . . . , c -1 } . When k = 1 , Ω k ( d ) has only one element, i.e., the prediction. For soft labels generated by teacher f , we define the unreliability degree as,

However, it is not enough to measure the soft labels by merely unreliability degree. For example, for images whose ground-truth labels are 1 , if label 2 always appears in Ω k ( d ) , then the student model is unable to distinguish label 1 from label 2 . Therefore, we introduce the ambiguity degree [4] and extend it to more general soft labels (induced by teacher f )

̸

̸

Ambiguity degree bound the probability of co-occurrence . In other words, if a model f is with ambiguity degree γ k ( f ) , then Pr( i ∈ Ω k ( f ( x )) | i = y, x, y ) ≤ γ k ( f ) . The smaller η or γ is, the more effective the soft labels are. However, when k increases, η will decrease and γ is inverse, which means k should be selected cautiously.

Theorem 1. Training with soft labels generated by the teacher model f , if γ k ( f ) < 1 -η k ( f ) 1 -η k ( f ) , then the optimal student model h ∗ ∈ H satisfies h ∗ = arg min h ∈H Err D ( h | f ) .

The proof can be found in A.2. Theorem 1 ensures that the student model h learning from the teacher can converge to the optimal learner over the entire data distribution. This property is known as classifier-consistency [12]. However, it does not provide the sample complexity bounds of the learning problem. In other words, it does not establish a connection between the generalization bound and the number of training samples.

Next, we provide our main result, a sufficient condition for the ERM learnability of the biased soft label learning problem. In the previous subsection, we presented the definition of expected classification error and empirical classification error, which are based on the label space Y . Here, given the soft label generating model f , we define the soft label-based expected error

and the soft label-based empirical error on dataset z = { ( x i , y i ) } n i =1

In the above equations, the set Ω k ( f ( x i )) is determined by the soft label f ( x i ) . We denote H η as the set of teacher models whose generated soft labels are with unreliability degree η , i.e., H η = { f : η k ( f ) = η } . With such soft labels, we analyze the performance of the ERM learners (students). The definition of the ERM learner is provided in the preliminary. Specifically, based on the unreliability degree in (1) and the ambiguity degree in (2), we provide a sufficient condition to ensure that, the biased soft label learning problem is ERM learnable.

Theorem 2 (Main theory) . For k ∈ { 1 , 2 , . . . , c -1 } , assume the unreliability degree η k ( f ) and the ambiguity degree γ k ( f ) of the soft labels generated by teacher model f , denoted concisely as η k and γ k , and satisfy 0 < η k , γ k < 1 and η k + γ k < 1 . Let θ k = log 2(1 -η k ) 1 -η k + γ k and suppose the Natarajan dimension of the hypothesis space H is d H . Define

Then when n > n 0 , the ERM learner satisfies Err D ( h ERM | f ) < ε with probability 1 -δ .

The proof can be divided into two lemmas. Let define H ε be the set of hypotheses with error at least ε , i.e, H ε = { h ∈ H : Err D ( h | f ) ≥ ε } . Our target is to bound the probability Pr( h ∈ H ε ) , which measures the generalization of the learner h . Since the entire soft label space is inaccessible, H ε is evaluated by the mediator R n,ε as follows:

Then, our goal is to prove that Pr( z ∈ R n,ε | f ∈ H η ) ≤ δ . In other words, given dataset z , the student model h has the generalization bound greater than ε with probability at most δ . Since the teacher model f is intractable and the soft label space ∆ is unknown, it is very difficult to directly calculate the conditional probability Pr( z ∈ R n,ε | f ∈ H η ) . We bound it by introducing a testing set z ′ . Lemma 1 is adapted from [[9], Lemma 11.1.5] and [[16], Corollary 2.6].

Lemma 1. For a testing set z ′ ∈ ( X × Y ) n , we can define the set S n,ε as

Then, we have Pr(( z , z ′ ) ∈ S n,ε | f ∈ H η ) ≥ 1 2 Pr( z ∈ R n,ε | f ∈ H η ) , for n > 8 log 2 ε .

The proof of Lemma 1 can be found in appendix A.3. With lemma 1, we just need to estimate Pr(( z , z ′ ) ∈ S n,ε | f ∈ H η ) . It seems more complicated to introduce the testing set z ′ but we can swap training/testing instance pairs, which is a classic method in the proof of learnability, to refine the data distribution on X × Y into each single instance.

Lemma 2. On the same condition of theorem 2, we have

The proof of Lemma 2 can be found in appendix A.4. With Lemma 1 and Lemma 2, we can prove Theorem 2, which is detailed in the appendix A.5.

In this section, we establish two essential properties of the soft labels. Classifier-consistency guarantees the effectiveness of the soft labels in a macroscopic perspective, and ERM learnability provides a microcosmic generalization bound for the student model h . The corresponding threshold conditions are presented to ensure the student model can learn from the soft labels. The theory is applicable to all biased soft labels. This inspires us to consider that accuracy alone may not be a sufficient measure of a teacher's teaching ability. In the section 5.1, we design a simple heuristic algorithm to generate such skillful but bad teachers, i.e., teachers with low accuracy but can teach good students. These results are illustrative for the comprehension and development of the soft label based algorithms.

4 Biased Soft Labels in Weakly-Supervised Learning

The labels in weakly-supervised learning (WSL) could be incomplete, inexact, inaccurate [59] because accurate annotation is often expensive and difficult to obtain. Soft labels are widely used in WSL. Many weakly-supervised learning paradigms can be transformed into learning from soft labels. Due to the lack of the supervisory information, the soft labels could be large-biased but the model can still learn from them. In this section, our theory is adapted to three classic weakly-supervised learning paradigms and can provide theoretical guarantee for the learnability of these problems. These findings reflect that the theory is promising and extensible.

4.1 Incomplete Supervision

In incomplete supervision, there are labeled data and unlabeled data. A common approach is to use the model to label the unlabeled data and then learn with all data iteratively. The soft labels of the unlabeled data are probably biased but make a significant contribution to the training process. We propose an ideal accuracy function and, based on it, we provide a theoretical analysis on incomplete supervision from the perspective of soft labels. Suppose there are N labeled data and M unlabeled data sampled from X × Y . The predictive model h is an ERM learner on both labeled data and unlabeled data. The label of unlabeled data will be updated iteratively.

Assumption 1. For N labeled data and M unlabeled data whose soft labels have unreliability degree η and ambiguity degree γ , the ERM learner h has a deterministic accuracy funtion ρ ( η, γ ) , the probability that h predict correctly. The model architecture, data distribution and optimization are implicitly included in ρ ( η, γ ) .

Assumption 2. Given that smaller values of η or γ result in more supervised information in the soft labels, we assume ρ ( η, γ ) decreases with η and γ .

̸

We suppose that the incorrect labels share the equal probability to appear in the topk set. The distribution of the incorrect labels can be characterized in a more refined manner. For instance, assume that there is a upper bound of p ( i | x ) p ( j | x ) , i, j ∈ Y , i, j = y, i = j . We employ simplified assumptions because they can still capture and reflect this process. Based on the ideal ρ ( η, γ ) , we delineate the progressive performance of h . Let ρ t denote the accuracy of h at epoch t and we have

̸

In practice, as learner h learns from labeled data and unlabeled data, the performance of h will improve and the soft labels of the unlabeled data will be more effective. Consequently, h and the soft labels may achieve a dynamic equilibrium.

Theorem 3. Based on the ideal accuracy function ρ ( η, γ ) , with a moderate initial state η 0 , γ 0 satisfying Theorem 2, if final accuracy of ρ final exists, it can be calculated by the following fixed point equation:

where k accords with the topk set in η and γ , c is the number of class labels. If ρ ( η, γ ) is k L -Lipschitz continuous ( k L < 1 -1 c ), then ρ final exists and is unique.

The proof is detailed in A.6. In fact, the deterministic ρ ( η, γ ) is unattainable due to the indeterminacy of the optimization and the potential uncertainty in the data distribution. An intuitive extension is to assume ρ ( η, γ ) is a probability distribution related to the training specifics, which can be

further investigated. Theorem 3 is coarse yet in agreement with the general intuition. The model h improves as the soft labels envolve and finally reach the bottleneck restricted by the model, data and optimization.

In this section, we demonstrate the potential benefits of our theory in several classic weakly-supervised learning paradigms. There remain numerous domains associated with soft labels. It is essential to possess an appropriate theory to analyze the soft labels for the corresponding algorithms. Our theory can be instrumental for comprehending and constructing the soft label based algorithms.

4.2 Partial Label Learning

In Partial Label Learning (PLL), each instace is typically assigned a set of possible labels, i.e., the candidate label set s [28, 46, 49]. The corresponding candidate label space is denoted by S , i.e., the non-empty power set of Y . Traditional PLL assumes the ground-truth label must be in the candidate label set. But recently, Lv et al. [29] considers that ground-truth label could be not in the candidate label set, which is named as Unreliable Partial Label Learning (UPLL).

In UPLL, there are two basic concepts, partial rate ν and unreliable rate µ . Partial rate ν is the ratio of incorrect labels in the candidate label set to total labels. A lower partial rate usually indicates a better performance of the model. Unreliable rate µ is the probability of ground-truth label y not in the candidate label set, which can be formally stated as

The discrete candidate label set s can be transformed into the soft label by

where | s | is the cardinality of set s . So PLL also can be viewed as learning from biased soft labels. Then we have the following corollary.

Corollary 1. For UPLL with partial rate ν and unreliable rate µ , regard the generated soft labels as teacher. Then, we have η = µ and γ = ν . With the same conditions in Theorem 2, UPLL is ERM learnable and the sample complexity remains unchanged.

This corollary show that UPLL is ERM learnable under a moderate condition. We provide new insights from the perspective of soft labels.

4.3 Learning with Additive Noise

Additive noise mechanism [30, 14] is an important methodology for differential privacy. Specifically, Laplace noise or Gaussian noise is added to data for protecting privacy. The privacy budget can be controlled by adjusting the scale of the noise. After normalization, the noisy labels are also biased soft labels in nature. In fact, given the probability density function of noise, we can calculate the corresponding unreliability degree and ambiguity degree in order to measure the effectiveness of the noisy labels. With the biased soft labels, the task is to train a utility model with strong privacy guarantees. Our theory can guarantee the utility of such soft labels.

To calculate unreliability degree and ambiguity degree, we refer to order statistic [8]. Order statistic analyze the i th-smallest value of random samples from a continuous distribution. We denote the order distribution Order ( d, n, i ) as the i th-smallest value of n samples from distribution d . The software Mathematics [45] provide an efficient API for estimating the order distribution.

Corollary 2. Let d denote the noise distribution (e.g. Laplace noise and Gaussian noise) and regard the noisy soft labels as the teacher. With the k in Eq.1 and the total classes c , for k ≤ c -1 , we can compute the η and γ as

With the same conditions in Theorem 2, the problem learning with additive noise is ERM learnable and the sample complexity remains unchanged.



























k

(

f

)

k

(

f

)















(a)



































k

(

f

)

k

(

f

)















(b)

























k

(

f

)

k

(

f

)





















(c)

As the scale of the noise increases, η and γ will increase, i.e., the effectiveness of the soft labels will decrease. This result agrees with the practical situation.

5 Experiments

Our experimentally investigate whether biased soft labels are still effective and whether the proposed indicators can measure the effectiveness of these soft labels , which consist of three parts. Firstly, we design a simple heuristic algorithm to generate Skillful but Bad Teachers (SBTs), which have low accuracy (less than 35%) but can teach good students. Secondly, experimental results demonstrate that students learning from SBTs can achieve comparable performance as models trained on the original data. We can explain these phenomenons with unreliability degree and ambiguity degree. Thirdly, we conduct experiments in weakly-supervised learning paradigms, and the results also confirm that unreliability degree and ambiguity degree can reflect the effectiveness of the biased soft labels. Due to space limitations, we present a part of experimental results in the appendix. Additionally, we provide the details of the experiment setup in appendix A.7.

5.1 Skillful but Bad Teachers

In this subsection, we introduced the design of the SBTs. The intuition behind SBTs, inspired by Theorem 2, is to inhibit correct predictions and reduces the unreliability degree and ambiguity degree simultaneously. So we design some heuristic loss functions and have some hyperparameters that qualitatively control the unreliability degree and ambiguity degree. More specifically, the designed loss functions are intended to keep the ground-truth label within the topk set of the soft labels, without necessarily requiring it to be at the top. Here, k is an empirical constant, which we have set to 3 or 4 in training SBTs.

Firstly, SBTs will punish those correctly predicted instances as





































k

(

f

)

k

(

f

)















(a)































k

(

f

)

k

(

f

)















(b)

where ℓ ( · , · ) is the cross entropy loss function. In practice, the value of the ground-truth label decreases significantly, which causes the ground-truth label to fall out of the topk set, i.e., large η k . So the ground-truth label y is compensated when y / ∈ Ω k ( f ( x )) :

The compensation term is designed to improve the topk accuracy of SBTs, which keeps the statistical effectiveness of the soft labels generated by SBTs. In practice, we found there was a strong correlation among the topk labels, which leaded to the confusion between the ground-truth label and similar labels, i.e., large γ k . To decrease this correlation, we propose an effective method to make the labels in Ω k ( f ( x )) as independent as possible. We randomly select k -1 labels except the ground-truth label y . Then the selected k -1 labels are employed as the learning objectives:

where s rnd is the set of the k -1 random labels excluding y . Consequently, the total objective of SBTs is as follows:

where L ce ( x , y ) is the vanilla cross-entropy loss between the output of the model and the ground-truth label, and α 1 , α 2 , α 3 are the trade-off parameters.

5.2 Effectiveness of the Proposed Indicators

In this subsection, our aim is to demonstrate that, despite the soft labels generated by SBTs are large-biased, students still achieve high accuracy. As outlined in Table 2 of the appendix, we adapt four different classic metrics: Chebyshev distance, KL divergence, Manhattan distance, and Euclidean distance. These metrics are employed to measure the discrepancies between the large-biased soft labels generated by SBTs and the ground-truth labels. The soft labels generated by SBTs exhibit substantial differences when compared to those from normally trained teachers. Nevertheless, they remain effective in instructing good students. It should be noted that the accuracy of the students mirrors the effectiveness of the soft labels. However, accuracy is an indirect measure contingent on the student model. The proposed unreliability degree and ambiguity degree can directly measure the effectiveness of these soft labels. The smaller the unreliability degree and ambiguity degree, the more effective the soft labels, and subsequently, the students' accuracy tends to be higher. This phenomenon is illustrated in Figure 2, which agrees with our expectations.

In the experiments, we set k = 4 for unreliability degree η k ( f ) and ambiguity degree γ k ( f ) ( k in the indicators can be different from k in training SBTs). Since η k ( f ) and γ k ( f ) are probabilities over the entire data distribution X × Y and difficult to compute, we estimate them empirically by the the generated soft labels on the test data. Note that training with the ground-truth labels can achieve accuracy 95.29% on CIFAR-10, 78.13% on CIFAR-100 and 72.53% on Tiny ImageNet. We can find many interesting results in Figure 2:

When these soft labels are generated by SBTs with accuracy less than 30%, which means they are quite different from the ground-truth labels, the students can still achieve accuracy much higher than the teachers.

As η k ( f ) and γ k ( f ) decrease, Acc (i.e., accuracy of the student) increases. The proposed indicators exhibit an inverse correlation with the students' accuracy, which reflects the effectiveness of the proposed indicators.

For the more complicated dataset like CIFAR-100 and Tiny ImageNet, accuracy (Acc) is more sensitive to η k ( f ) and γ k ( f ) . This implies that, for more complex tasks, the quality of the annotation is more crucial.

The hyperparameters in Eq.(8) are pivotal in controlling the indicators of these soft labels, which is elucidated in A.8. Furthermore, we conducted experiments to investigate whether the effectiveness of soft labels is influenced by different model backbones, as detailed in A.9. We tested a range of architectures, including wideresnet 28x2, 28x4, 40x2, and 40x4. The four distinct backbones exhibited consistent patterns, indicating that the proposed indicators are effective across different backbones. In addition, we present the overall distribution of these soft labels in A.10. The experimental results demonstrate that biased soft labels can also teach good students and the proposed indicators can measure the effectiveness of these soft labels , which confirm the validity of our theory.

5.3 Biased Soft Labels in Weakly-Supervised Learning

The aforementioned weakly-supervised learning paradigms have engendered a lot of specialized algorithms. The intention of this paper is not to devise more efficacious algorithms, but to evaluate the effectiveness of the soft labels in the weakly-supervised learning paradigms by unreliability degree and ambiguity degree. The results of partial label learning are shown in Table 1, while the results of learning with additive noise are shown in Figure 3. Due to space limitations, the results of incomplete supervision are shown in A.11. From the three experiments, we can observe that the accuracy of the students decrease when unreliability degree and ambiguity degree increase. All the results reflect that these indicators can measure the effectiveness of the soft labels well, which is consistent with our theory. In addition, since the soft labels in partial label learning and learning with additive noise are generated by a mapping rule rather

than a neural network, the results of these experiments are more stable and smooth.

6 Conclusion

In this paper, we find that even large-biased soft labels can teach a good student and focus on the effectiveness of the biased soft labels. It motivates us to rethink when the biased soft labels (or teachers) are effective. We propose two indicators, unreliability degree and ambiguity degree, to measure the effectiveness of soft labels. Based on the proposed indicators, we provide moderate conditions that guarantee the classifier-consistency and ERM learnability of the biased soft label learning problem. Our theoretical framework can be adapted to elucidate the effectiveness of soft labels in three weakly-supervised learning paradigms, incomplete supervision, partial label learning and learning with additive noise. We design a heuristic method to train Skillful but Bad Teachers (SBTs), which validate that large-biased soft labels can teach good students. Besides, the effectiveness of both soft labels generated by SBTs and soft labels in the weakly-supervised learning paradigms can be measured by the proposed indicators well, which is consistent with our theory.

References

Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and selfdistillation in deep learning. CoRR , abs/2012.09816, 2020.

Cristian Buciluˇ a, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 535-541, 2006.

Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks , 20(3):542-542, 2009.

Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research , 12:1501-1536, 2011.

Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research , 12:1501-1536, 2011.

Amit Daniely, Sivan Sabato, Shai Ben-David, and Shai Shalev-Shwartz. Multiclass learnability and the erm principle. In Proceedings of the 24th Annual Conference on Learning Theory , pages 207-232. JMLR Workshop and Conference Proceedings, 2011.

Tri Dao, Govinda M Kamath, Vasilis Syrgkanis, and Lester Mackey. Knowledge distillation as semiparametric inference. In International Conference on Learning Representations , 2020.

HA David and HN Nagaraja. Order statistics. Encyclopedia of Statistical Sciences , 2004.

Richard M Dudley, H Kunita, F Ledrappier, and RM Dudley. A course on empirical processes. In Ecole d'été de probabilités de Saint-Flour XII-1982 , pages 1-142. Springer, 1984.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 , pages 265-284. Springer, 2006.

Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. Advances in Neural Information Processing Systems , 33:10948-10960, 2020.

Xin Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering , 28(7): 1734-1748, 2016.

Badih Ghazi, Noah Golowich, Ravi Kumar, Pasin Manurangsi, and Chiyuan Zhang. Deep learning with label differential privacy. Advances in Neural Information Processing Systems , 34:27131-27145, 2021.

Badih Ghazi, Noah Golowich, Ravi Kumar, Pasin Manurangsi, and Chiyuan Zhang. Deep learning with label differential privacy. Advances in neural information processing systems , 34:27131-27145, 2021.

Evarist Giné and Joel Zinn. Some limit theorems for empirical processes. The Annals of Probability , pages 929-989, 1984.

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. CoRR , abs/1503.02531, 2015.

Daniel Hsu, Ziwei Ji, Matus Telgarsky, and Lan Wang. Generalization bounds via distillation. arXiv preprint arXiv:2104.05641 , 2021.

Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. Advances in Neural Information Processing Systems , 33:20823-20833, 2020.

Zhong Ji, Jingwei Ni, Xiyao Liu, and Yanwei Pang. Teachers cooperation: team-knowledge distillation for multiple cross-domain few-shot learning. Frontiers of Computer Science , 17(2):172312, 2023.

Qing Jin, Jian Ren, Oliver J Woodford, Jiazhuo Wang, Geng Yuan, Yanzhi Wang, and Sergey Tulyakov. Teachers do more than teach: Compressing image-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13600-13611, 2021.

Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems , 31, 2018.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896, 2013.

Hankook Lee, Sung Ju Hwang, and Jinwoo Shin. Rethinking data augmentation: Self-supervision and self-distillation. CoRR , abs/1910.05872, 2019.

Liping Liu and Thomas Dietterich. Learnability of the superset label learning problem. In International Conference on Machine Learning , pages 1629-1637. PMLR, 2014.

Milan Lopuhaä-Zwakenberg, Boris Škori´ c, and Ninghui Li. Fisher information as a utility metric for frequency estimation under local differential privacy. In Proceedings of the 21st Workshop on Privacy in the Electronic Society , pages 41-53, 2022.

Jiaqi Lv, Miao Xu, Lei Feng, Gang Niu, Xin Geng, and Masashi Sugiyama. Progressive identification of true labels for partial-label learning. In International Conference on Machine Learning , pages 6500-6510. PMLR, 2020.

Jiaqi Lv, Biao Liu, Lei Feng, Ning Xu, Miao Xu, Bo An, Gang Niu, Xin Geng, and Masashi Sugiyama. On the robustness of average losses for partial-label learning. CoRR , abs/2106.06152, 2021.

Mani Malek Esmaeili, Ilya Mironov, Karthik Prasad, Igor Shilov, and Florian Tramer. Antipodes of label differential privacy: Pate and alibi. Advances in Neural Information Processing Systems , 34:6934-6945, 2021.

Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In International Conference on Machine Learning , pages 7632-7642. PMLR, 2021.

Balas K Natarajan. On learning sets and functions. Machine Learning , 4:67-97, 1989.

Balas K Natarajan. On learning sets and functions. Machine Learning , 4(1):67-97, 1989.

Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. Boosting self-supervised learning via knowledge transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9359-9367, 2018.

Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International Conference on Machine Learning , pages 5142-5151. PMLR, 2019.

Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , pages 400-407, 1951.

Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550 , 2014.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2818-2826, 2016.

Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. CoRR , abs/2002.03532, 2020.

Leslie G Valiant. A theory of the learnable. Communications of the ACM , 27(11):1134-1142, 1984.

Vladimir Vapnik, Rauf Izmailov, et al. Learning using privileged information: similarity control and knowledge transfer. J. Mach. Learn. Res. , 16(1):2023-2049, 2015.

Jing Wang and Xin Geng. Label distribution learning machine. In International Conference on Machine Learning , pages 10749-10759. PMLR, 2021.

Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Yude Wang, Jie Zhang, Meina Kan, and Shiguang Shan. Learning pseudo labels for semi-and-weakly supervised semantic segmentation. Pattern Recognition , 132:108925, 2022.

Alfred North Whitehead. An introduction to mathematics . Courier Dover Publications, 2017.

Dong-Dong Wu, Deng-Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. In International Conference on Machine Learning , pages 24212-24225. PMLR, 2022.

N. Xu, J. Shu, R. Zheng, X. Geng, D. Meng, and M. Zhang. Variational label enhancement. IEEE Transactions on Pattern Analysis &amp; Machine Intelligence , (01):1-15, sep 2022. ISSN 1939-3539. doi: 10.1109/TPAMI.2022.3203678.

Ning Xu, Yun-Peng Liu, and Xin Geng. Label enhancement for label distribution learning. IEEE Transactions on Knowledge and Data Engineering , 33(4):1632-1643, 2019.

Ning Xu, Congyu Qiao, Xin Geng, and Min-Ling Zhang. Instance-dependent partial label learning. Advances in Neural Information Processing Systems , 34:27119-27130, 2021.

Junxiao Xue, Shibo Huang, Huawei Song, and Lei Shi. Fine-grained sequence-to-sequence lip reading based on self-attention and self-distillation. Frontiers of Computer Science , 17(6):176344, 2023.

Anbang Yao and Dawei Sun. Knowledge transfer via dense cross-layer mutual-distillation. In European Conference on Computer Vision , pages 294-311. Springer, 2020.

Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 1285-1294, 2017.

Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3903-3911, 2020.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR , abs/1605.07146, 2016.

Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3713-3722, 2019.

Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4320-4328, 2018.

Xingyu Zhao, Yuexuan An, Ning Xu, and Xin Geng. Fusion label enhancement for multi-label learning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022 , pages 3773-3779. ijcai.org, 2022.

Helong Zhou and Liangchen Song. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. In Proceedings of International Conference on Learning Representations (ICLR) , 2021.

Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National science review , 5(1):44-53, 2018.

Xiatian Zhu, Shaogang Gong, et al. Knowledge distillation by on-the-fly native ensemble. Advances in neural information processing systems , 31, 2018.

A Appendix

A.1 Variants of Knowledge Distillation

Self-distillation [55, 25, 50] treats the mixture of the outputs and the ground-truth labels as target soft labels, and the proportion is iteratively adjusted during the training. Ensemble KD [52, 60] employs the ensemble of the soft labels (teachers) to improve generalization. Besides, in mutual learning [56, 51], there is no explicit teacher network and multiple students learn from each other by synthesizing soft labels of other students. Soft labels play an important role in all these variants.

A.2 Proof of Theorem 1

For simplification, we denote the probability of event A as p ( A ) .

The coefficient of ℓ ( h ( x ) , y ) is:

̸

For i = y , there is

Therefore, when 1 -η > γ (1 -η ) + η , i.e., γ < 1 -η 1 -η , we have h ∗ = arg min h ∈H Err D ( h | f ) .

A.3 Proof of Lemma 1

Let z , z ′ be the training set and the testing set, and each of them is of size n . Define H ( z ) = { h ∈ H : Err ∆ z ( h | f ) = 0 } as the set of zero-empirical-risk hypotheses. We can bound Pr( z ∈ R n,ε | f ∈ H η ) with Pr(( z , z ′ ) ∈ S n,ε | f ∈ H η ) as follows

When n > 8 log 2 ε , we have Pr(( z , z ′ ) ∈ S n,ε | f ∈ H η ) ≥ 1 2 Pr( z ∈ R n,ε | f ∈ H η ) , which completes the proof.

A.4 Proof of Lemma 2

We follow the methodology of proving the ERM learnability in [26]. Here we need to bound Pr( S n,ε | f ∈ H η ) . The key behind the proof is to refine Err ∆ z ( h | f ) and Err ∆ z ′ ( h | f ) . We use a classic method, i.e., swap, to refine the probability of a single instance. A swap σ ( z , z ′ ) = ( z σ , z ′ σ ) means exchanging some instances between the training set z and testing set z ′ while keeping size n unchanged. Each ( x, y ) ∈ z is a training instance while each ( x, y ) ∈ z ′ is a testing instance. There are 2 n different swaps in total and we define G as the set of all swaps. Firstly, we use swap to describe Pr( S n,ε | f ∈ H η ) .

Here, the expectations are with respect to ( x, y, x ′ , y ′ ) . To further refine S n,ε , we define S h n,ε for a certain classifier h as

Let H | ( x, x ′ ) be the hypothesis space that have different prediction for instances ( x, x ′ ) . Next, we have the bound

By [33], the complexity of the hypothesis space H | ( x, x ′ ) can be bounded as

Let d x i = f ( x i ) be the soft label of x i induced by model f and Ω k ( d x i ) be the topk set of the soft label. Then,

For the pair of ( x, y, x ′ , y ′ ) , we consider the number of instances of all cases. Specifically, let u 1 , u 2 and u 3 represent the number of both incorrectly predicted instances, one incorrectly predicted instances and both correctly predicted instances. Besides, we define u σ as the number of instances where ( x σ , y σ ) is incorrectly predicted while ( x ′ σ , y ′ σ ) is correctly predicted. Afterwards, the number of incorrectly predicted instances in the testing set is u 1 + u 2 -u σ .

̸

For Pr ( ˜ h ( x σ i ) ∈ Ω k ( d x i ) | x σ , y σ , f ∈ H η ) , we count instances which have been swapped. On the one side, there are u 2 + u 3 -u σ instances are correctly predicted ( ˜ h ( x σ i ) = y σ i ) where we have Pr ( ˜ h ( x σ i ) ∈ Ω k ( d x i ) | f ∈ H η ) = 1 -η by (1). On the other side, there are u 1 + u σ instances are incorrectly predicted ( ˜ h ( x σ i ) = y σ i ), where we have Pr ( ˜ h ( x σ i ) ∈ Ω k ( d x i ) | f ∈ H η ) ≤ γ by (2). So, we have

And then, the conditional probability can be bounded as follow:

There are 2 n different swaps and we sum all.

where n = u 1 + u 2 + u 3 . The 2 u 1 + u 3 in the first step means ways of swapping both correct instances and both incorrect instances. The index of j equals u 2 different swaps of one correctly predicted instances. According to the assumption η + γ ≤ 1 , we have 0 < γ 1 -η < 1 -η + γ 2(1 -η ) < 1 . For u 1 + u 2 ≥ ε 2 n , when u 1 = 0 and u 2 = ε 2 n , the right side reaches its maximum as follows:

We have proved the Lemma 2.

A.5 Proof of Theorem 2

With Lemma 1 and Lemma 2, we have

We set θ as

Since η + γ < 1 , we get θ > 0 . We need to bound Pr( R n,ε | f ∈ H η ) with η , which means

Note that the function f ( x ) = log 1 a + ax --1 ≥ 0 . Let a = θε 2 +log( 1 2 -2 η ) d H and x = n . It can be inferred that

With the bound of log n , we get the linear inequality of n . Since the aforementioned results hold for all k ∈ { 1 , 2 , . . . , c -1 } , Let

When n > n 0 , we get Pr( R n,ε | f ∈ H η ) < η and the proof is finished.

0

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

6

7

8

9

0.08

0.10

0.12

0.14

0.16

0.18

(a)

0

10

20

30

40

50

60

70

80

90

0

10

20

30

40

50

60

70

80

90

0.05

0.10

0.15

0.20

(b)

A.6 Proof of Theorem 3

Generally, for epoch t , we denote η t and γ t as the the unreliability degree and ambiguity degree of the the soft labels of the unlabeled data. The model will achieve the accuracy of epoch t , ρ ( η t , γ t ) . We refer to it as ρ t for simplicity. Then, for next epoch t +1 , the unreliability degree and ambiguity degree of next epoch can be estimated η t +1 ≤ 1 -ρ t . The estimation of ambiguity degree is a little bit more complicated. Incorrect label share equal probability γ t +1 because we assume the noise is uniformly distributed. Then we have

For ρ ( η, γ ) monotonically decreases, with the upper bounds on η t +1 and γ t +1 , we can get a lower bound on ρ t +1 as

If ρ final = lim t →∞ ρ t exists, it must satisfy the fix point equation,

Next, we prove that if ρ ( η, γ ) is k L -Lipschitz continuous ( k L < 1 -1 c ), then ρ final exists and is unique. We define

where ( η, γ ) ∈ [0 , 1] 2 . l 1 -norm is employed as the norm on [0 , 1] 2 and denote d ( · , · ) as the distance function. We want to show that ψ is a contraction mapping . For ( η 1 , γ 1 ) , ( η 2 , γ 2 ) ∈ [0 , 1] 2 ,

where (1 + 1 c -1 ) · k L ∈ [0 , 1) . So ψ is a contraction mapping and there is a unique fixed point ( η, γ ) that ψ ( η, γ ) = ( η, γ ) . That means ρ final exists and is unique.

Further,

A.7 Experiment Setup

We consider three benchmark image datasets CIFAR-10, CIFAR-100 [23] and Tiny ImageNet, and generate the soft labels with different hyperparameters of the teacher model. The student model is trained with the generated soft labels and aims to distinguish the ground-truth label. The accuracy of the student model is employed to measure the effectiveness of the soft labels. Datasets are divided into training, validation, testing set in the ratio of 4:1:1. For the fairness of the experiments, all student models are WideResNet28 × 2 architecture [54] on each dataset. In all experiments, we use mini-batch SGD [36] with a batch size of 128 and a momentum of 0.9. Each model is trained with maximum epochs T = 200 and employs early stopping strategy with patience 20. We report final performance using the test accuracy corresponding to the best accuracy on validation set. For the all experiments, we employ a basic re-weighting strategy to train the student model. Specifically, in each epoch, we train the student model and update the soft labels with the softmax outputs of it.

Weakly-supervised learning For PLL, we employ a common strategy to zeroize the soft labels that are not in the candidate set. For learning with unlabeled data, we warm-up the model for 5 epochs. Then, we train with all data every epoch and label the unlabeled data with the model every 5 epochs. Each dataset is divided into labeled set, unlabeled set, testing set in the ratio of 1:19:4.

A.8 Hyperparameters in (8)

We set the number of the random labels (in 7) as 3 while k in η k and γ k as 4 in all experiments. The punishment factor α 1 ranges from 0 to 0.4 and the compensation factor α 2 ranges from 0.9 to 1.3. The weight of the random labels α 3 is from 1.6 to 2.3. In detail, we observe the following phenomenons:

As mentioned in 1, we cannot take the accuracy of the teacher as the sole criterion to evaluate the pedagogical efficacy. γ defined in (2) can be seen as the coarse measure of the imbalance of the label. For the student model, the smaller γ could mean the better performance if the number of random labels is fixed.

Both the punishment and the random labels are applied to decrease the top1 accuracy within a reasonable range. For the simple dataset, large punishment are needed to protect privacy and for the complicated dataset like CIFAR-100, we enhance the ratio of random labels to reduce the effect of similar labels.

These soft labels are very different from the ground-truth labels, which means low accuracy. In these soft labels, the average degree of the ground-truth labels is around 0.2.







































k

(

f

)

k

(

f

)



















































(a)







































(b)

A.9 Experiments of different backbones

Figure 6 demonstrates whether the effectiveness of soft labels is influenced by different backbones. We experimented with wideresnet 28x2, 28x4, 40x2, and 40x4. In the Figure 2, we omitted the Unreliability degree and Ambiguity degree, which are the same the that of Figure 2 in the paper, for clarity. The four distinct backbones displayed consistent trends, further suggesting that the proposed indicators are effective across different backbones.

A.10 Overall distribution of the Soft Labels Generated by SBTs

As illustrated in Figure 4, the horizontal axis represents the ground-truth label while the vertical axis represents the mean of the soft labels. The diagonal can be seen as the degree of correctly predicted labels. We can see that the ground-truth label is dominant in the soft labels. On the other side, the figure can be seen as a simple measure of the similarity between labels.

A.11 Experiments of Incomplete Supervision

The predictive model label the unlabeled data and then learn with all data alternately. As shown in Figure 5, Acc (the accuracy of the model) improves as the soft labels evolve. Accordingly, unreliability degree η k ( f ) decreases and ambiguity degree γ k ( f ) remain unchanged, which means the soft labels of the model are more effective. The figure shows the dynamics in the training with unlabeled data, which is consistent with the theory in 4.1. Furthermore, we find that the soft labels of unlabeled data exhibit a high ambiguity degree γ k ( f ) , indicating that some categories may be very similar and difficult for the model to distinguish.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyse the learnability properties of (biased) soft labels, e.g., originating from a teacher model in a student-teacher setup, with respect to classifier consistency and ERM learnability. To this end, two indicator measures of the quality of the proxy labels are suggested, which are used in a theoretical analysis to derive bounds about the aforementioned quality dimensions. Moreover, a heuristic loss for training skillful but bad teachers is developed.\n\nSTRENGTHS:\n- Investigates a highly relevant matter\n- Technically sound and valid theoretical results\n- Covers multiple special cases of weakly-supervised learning as a practically relevant problem setup\n\nWEAKNESSES:\nMajor:\n- I find the distinction between incomplete supervision and partial label learning confusing. Typically, incomplete supervision and partial label learning are more or less the same thing. What the authors here refer to in the case of incomplete supervision is semi-supervised learning, which is a special case of partial label learning, namely that you can either observe a precise (i.e., unambiguous) label, or no label at all resp. the complete target space. In order to preserve consistency with related literature on this (e.g., as in your reference [58]), I would recommend sticking to this ontology.\n- The empirical analysis of SBTs does not contain any baseline. Overall, it is hard to judge the appropriateness of the developed indicators and the SBT loss proposals, perhaps the experimental setup can be overhauled.\n- It is hard to assess the looseness resp. tightness of the bound in Theorem 2. I would love to see the authors elaborating on this matter, e.g., by putting individual term components in a context, comparing it to “classical” bounds in multi-class learning, potentially with respect to label noise.\n\nMinor:\n- The page limit for submissions was 9 pages, but there is content on page 10.\n- I think there is a missing $\\max$ (or a different aggregator) in Eq. (1), right?\n- Also, “ambiguity degree” is a term that is already being used in the PLL literature, as also indicated by referring to [5]. This is not precisely an equivalent ambiguity degree formulation, so it should be distinguished.\n\nQUESTIONS:\n- How did you determine the hyperparameters being used in the experiments? Did you repeat the experiments multiple times with different seeds? How about the standard deviations?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the effectiveness of biased soft labels in knowledge distillation and weakly-supervised learning. The paper introduces two indicators to measure the effectiveness of soft labels, and proposes moderate conditions to ensure that biased soft label learning is classifier-consistent and ERM learnable. The paper also presents a heuristic method to train skillful but bad teachers, and shows that they can teach students to achieve high accuracy on CIFAR-10/100. The paper applies the theoretical framework to three weakly-supervised learning paradigms, and validates the indicators with experiments.\n\nSTRENGTHS:\n1. The paper is well-written and a pleasure to read.\n2. The paper provides thorough theoretical analysis for three different weak supervision settings.\n\nWEAKNESSES:\n1. The paper only conducts experiments on CIFAR-10/100 datasets, and does not provide experiments on larger and mainstream datasets, such as ImageNet. The paper also does not compare the proposed method with other state-of-the-art weakly-supervised methods, especially those based on knowledge distillation.\n2. The paper is suggested to provide more visualizations of the the prediction results (class probabilities) of large-biased soft labels (generated by SBTs) and good student.\n3. The paper does not provide experiments on different backbones, and cannot demonstrate the effectiveness of the proposed indicators and method on different architectures. The paper also does not discuss how the choice of the backbone affects the performance and robustness of the method.\n4. The paper exceeds the page limit. The paper is suggested to be shortened to meet the page limit requirement.\n\nQUESTIONS:\n1. In the introduction, the paper claims that “Empirical Risk Minimization (ERM) learners’ performance can generalize to the entire data distribution.”\n    However, previous studies have pointed out the drawbacks of ERM, such as (1) neural networks trained with ERM change their predictions drastically when evaluated on examples just outside the training distribution[1], also known as adversarial examples; (2) ERM allows large neural networks to memorize (instead of generalize from) the training data even in the presence of strong regularization, or in classification problems where the labels are assigned at random [1]. \n    In general, using ERM for optimization may affect the generalization ability of the model. Does this contradict the conclusion of this paper? Does the conclusion of this paper have any prior studies or experiments to support it?\n[1]Zhang, Hongyi, et al. \"mixup: Beyond empirical risk minimization.\" *arXiv preprint arXiv:1710.09412* (2017).\n\n2. The paper mentioned “skillful but bad teachers”. How to define the skillfulness of the bad teachers? \n3. As you mentioned in Section 5.3, is your main contribution the two methods of evaluating soft labels (unreliability degree and ambiguity degree), rather than the heuristic method of “bad teacher can teach good student”? \n4. The conclusion that “the accuracy of the students decrease when unreliability degree and ambiguity degree increase” does not seem to be obvious in Figure 2(b). Also, in Figure 3 (a) and (b), the ambiguity degree γ does not change much as the student accuracy increases. The same situation also occurs in Figure 5 in the appendix, which seems to not support the effectiveness of the ambiguity degree γ indicator in these weakly-supervised settings.\n\nLIMITATIONS:\nSome of the paper’s experimental results show that the two indicators proposed by the paper cannot well characterize the effectiveness of soft labels in all weakly-supervised settings. For example, under some of the weakly-supervision settings, the paper shows that the indicators are not consistent with the performance of the student model. The paper does not provide sufficient explanation or analysis for this phenomenon, and does not discuss how to improve or modify the indicators to better capture the effectiveness of soft labels.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a clever theoretical framework for studying learnability under supervision with imperfect soft labels.\n\nSTRENGTHS:\n- Learning from imperfect soft labels is an important and interesting area that is prevalent not only in knowledge distillation but also in cognitive science, human-AI interaction, and pretty much any modeling that aim to take into account noise and uncertainty. \n- The authors conduct a promising theoretical analysis of this setting by recasting it as a noisy top-k oracle problem (where teachers provide a set of k labels which with some probability contains the true label).\n\nWEAKNESSES:\n- Clarity could be improved by providing more intuitions and explanations for terminology throughout. For example, \"biased soft label\" is never formally defined, and the implied definition (based on the notions of ambiguity and unreliability) does not fully agree with what I would normally think of when thinking of statistical notions of bias. Some additional editorial revision (fixing typos, etc.) would also be helpful, but I am not taking that into account in my score. \n- I think the assumptions are not quite as moderate as the authors imply and would like some more clarification about these. For example, one of the assumptions seem to be that labels occur in the top-k with equal probability, but this is clearly not the case in practice (e.g. you may believe a picture of a dog could be a wolf, but certainly not an airplane) which the authors mention when designing their experiments as they find that the top-k labels are correlated. The experiments are then designed to satisfy this assumption which means there is very little evidence that the proposed theory describes a realistic setting.  Another assumption (Assumption 2 in 4.1) seems to pre-assume that the metrics considered by the authors are inversely correlated with accuracy which feels circular since I believe this is one of the things the authors want to claim. \n- As mentioned in the section above, the empirical results are currently unconvincing. In addition to the assumption mentioned above, it seems like in both experiments, the true label has the highest probability in >15% of the cases. For CIFAR10 this is reasonable since random chance is 10%, but for CIFAR100, this is quite high. Is there a disconnect between the theory setting and the experimental setting, in that the theory is focused on learning from the top-k set where all top-k labels are treated equally while in the experimental setting, learning still takes into account the actual probabilities? Perhaps a more compelling experiment would be to isolate various top-k cases and show that the curves are robust to changes in k?\n- There are a number of existing papers that study the informativeness of soft labels. It would be great to see some discussion of how this study fits into that literature.\n\nQUESTIONS:\nSee weaknesses section.\n\nLIMITATIONS:\n- The authors have listed both limitations and assumptions, though I think the assumptions may be more of a limitation than suggested.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to study the effectiveness of biased soft labels for knowledge distillation. They propose two indicators to measure the effectiveness of the biased soft labels: unreliability degree and ambiguity degree.  They provide a theoretical guarantee that the biased soft labels are effective in training a good student in three weakly-supervised learning paradigms: incomplete supervision, partial label learning, and learning with additive noise.  Their experiments reveal that largely biased soft labels can also teach good students and the proposed indicators are effective in measuring the effectiveness of soft labels.\n\nSTRENGTHS:\n* The motivation of the paper is original and novel. It is intriguing to see that learning from largely biased soft labels can achieve comparable performance, and the proposed indicators can be a valuable contribution to measuring their effectiveness in learning a good student for downstream tasks.\n* The paper provides an in-depth theoretical guarantee of the biased soft labels in three weakly supervised learning settings.\n* The experiment results validate that the proposed indicators (unreliability and ambiguity degree) are indeed effective in measuring the effectiveness of the soft labels (i.e. high accuracy of students) across the three weakly supervised learning paradigms. \n* Overall, the paper sufficiently informs the readers of technical and implementation details.\n\nWEAKNESSES:\n* The paper shows limited experiment results on CIFAR-10 and CIFAR-100 (Figure 2). The authors should perform experiment on a wider range of benchmark datasets to validate the generality of the method. Are the proposed indicators effective in more complex datasets as the authors suggested?\n* The introduction could be better organized. The paper should clearly present the definition of a large-biased soft label and better motivate the readers on why utilizing them is valuable. \n* The paper exceeds the nine page limit.\n\nQUESTIONS:\n* Does the proposed indicators work well across other benchmark datasets?\n\nLIMITATIONS:\nThe authors did not address the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the concept of learning from weak, or \"soft,\" labels that may be biased or diverge from the ground truth labels in a dataset. In contrast to existing theories that focus on the importance of close alignment between soft and ground truth labels, the authors probe the efficacy of learning from significantly biased soft labels. They introduce two indicators to gauge the effectiveness of these soft labels and propose conditions under which learning from such labels can be successful, including large-biased labels.\n\nThe authors further devise a heuristic method for training what they call Skillful but Bad Teachers (SBTs), referring to models with relatively low accuracy that can nevertheless effectively train high-performing student models. They show that these teachers can achieve up to 90% accuracy on the CIFAR-10 dataset, demonstrating the validity of their approach.\n\nAdditionally, the authors adapt their theoretical framework to examine the utility of soft labels in three specific weakly-supervised learning paradigms: incomplete supervision, partial label learning, and learning with additive noise. Experimental results are presented to support the proposed indicators and the viability of biased soft labels in these scenarios.\n\nKey contributions include:\n\n    Discovery that learning from largely biased soft labels can achieve comparable performance, and an exploration of the mechanisms behind this phenomenon.\n    The proposal of two indicators to evaluate the effectiveness of soft labels and conditions to ensure their usefulness.\n    A heuristic method to train SBTs, using new concepts of unreliability degree and ambiguity degree.\n    A theoretical framework that illuminates the role of soft labels in three weakly-supervised learning paradigms, accompanied by theoretical guarantees for their learnability and supporting experimental results.\n\nSTRENGTHS:\nTheoretical Analysis: The paper provides a comprehensive theoretical framework for analyzing the effectiveness of soft labels, which are often employed in the realm of machine learning for teaching student models.\n\nDefinitions and Indicators: The paper introduces and defines new concepts like unreliability degree and ambiguity degree, and relates them to the effectiveness of soft labels. This could provide valuable insight for the development of future machine learning models.\n    \nExtension to Weakly-Supervised Learning (WSL): The research effectively applies the theoretical findings to weakly-supervised learning paradigms, thus demonstrating the applicability and extensibility of their findings.\n\nWEAKNESSES:\nSelective Conditions: While the paper provides conditions for classifier-consistency and ERM learnability, it doesn't clearly outline how to meet these conditions in a real-world context. Furthermore, the condition of balancing unreliability degree and ambiguity degree might be challenging to achieve in practice.\n\nQUESTIONS:\nIt's mentioned that the soft labels generated by SBTs are large-biased, which could potentially impact the model's ability to generalize to new, unseen data. While the authors note that students still have good accuracy despite the bias, how does the authors propose the current approach tackles this situation?\n\nLIMITATIONS:\nThe process of inhibiting correct predictions, reducing unreliability and ambiguity degrees, and randomly selecting k-1 labels for each training instance could increase the computational complexity and time required to train the models.\n\nThe evaluation is mainly based on the accuracy of the student models, which might not be the most comprehensive measure. There could be other performance metrics that are important in the context of this problem.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}