SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hua Yuan $^{1,2}$ , Yu Shi $^{1,2}$ , Ning Xu $^{1,2}$ , Xu Yang $^{1,2}$ , Xin Geng $^{1,2*}$ , Yong Rui $^{3*}$
$^{1}$  School of Computer Science and Engineering, Southeast University, Nanjing 210096, China
$^{2}$  Key Laboratory of New Generation Artificial Intelligence Technology and Its
Interdisciplinary Applications (Southeast University), Ministry of Education, China
3 Lenovo Research, Beijing 100085, China
{yuanhua,seushiyu,xning,xuyang_palm,xgeng}@seu.edu.cn，yongrui@lenovo.com
Since the advent of knowledge distillation, many researchers have been intrigued by the dark knowledge hidden in the soft labels generated by the teacher model. This prompts us to scrutinize the circumstances under which these soft labels are effective. Predominant existing theories implicitly require that the soft labels are close to the ground-truth labels. In this paper, however, we investigate whether biased soft labels are still effective. Here, bias refers to the discrepancy between the soft labels and the ground-truth labels. We present two indicators to measure the effectiveness of the soft labels. Based on the two indicators, we propose moderate conditions to ensure that, the biased soft label learning problem is both classifier-consistent and Empirical Risk Minimization (ERM) learnable, which can be applicable even for large-biased soft labels. We further design a heuristic method to train Skillful but Bad Teachers (SBTs), and these teachers with accuracy less than  $30\%$  can teach students to achieve  $90\%$  accuracy on CIFAR-10, which is comparable to models trained on the original data. The proposed indicators adequately measure the effectiveness of the soft labels generated in this process. Moreover, our theoretical framework can be adapted to elucidate the effectiveness of soft labels in three weakly-supervised learning paradigms, namely incomplete supervision, partial label learning and learning with additive noise. Experimental results demonstrate that our indicators can measure the effectiveness of biased soft labels generated by teachers or in these weakly-supervised learning paradigms.
Knowledge distillation [2, 17, 20] has achieved remarkable achievements in a wide range of applications. It has emerged as a popular paradigm for model compression [22, 21] and transfer learning [41, 34] by distilling knowledge from the big model (teacher) to the small model (student). Students inherit the knowledge of the teacher by imitating the soft labels generated by the teacher model. Many experiments show that learning from soft labels can be very effective, even surpassing learning from ground-truth labels [2, 17, 37]. However, there are still many mysteries why these soft labels are effective. Most existing theories do not apply to soft labels that significantly deviate from the ground-truth labels and lack explicit indicators to evaluate the soft labels [39, 58, 7, 35]. In this paper, we mainly focus on the effectiveness of the biased soft labels, especially large-biased soft labels.
It is significant to study the effectiveness of biased soft labels. Firstly, expert-labeled annotation typically requires a substantial investment of manpower and time. But the soft labels in weakly supervised learning could be cheap and biased [59, 46]. Evaluating the biased soft labels is instructive for weakly supervised learning. Secondly, these biased soft labels can apply for privacy protection
(b)
Figure 1: (a) Images of the birds in CIFAR-10. (b) An example of the soft label for "Bird" in Yuan et al. [53]. (c) An example of the large-biased soft label for "Bird". Compared with Yuan et al. [53], the value of the ground-truth label (Bird) in the large-biased soft label is probably much smaller, and is likely not the top of the large-biased soft label.
[15, 11]. When they are employed for subsequent training, they can conceal the ground-truth labels. It is consistent with the requirements of privacy and utility in privacy protection [27].
After realizing the significance of biased soft labels, we study the biased soft labels and reveal that large-biased soft labels can also teach good students. This phenomenon inspire us to seek indicators to measure the effectiveness of soft labels. As a result, we propose two intuitive indicators, namely unreliability degree and ambiguity degree. The intuition behind them is to convert the soft label into a top- $k$  set ( $k$  is a constant), which contains the top- $k$  labels of the soft label. Unreliability degree is the probability that the ground-truth label is not in the top- $k$  set while ambiguity degree is the upper bound of the probability that the incorrect label and the correct label co-occur in the top- $k$  set.
Furthermore, based on the proposed indicators, we present moderate conditions to ensure that, the biased soft label learning problem is classifier-consistent [28] and Empirical Risk Minimization (ERM) learnable [6]. The former guarantees that the learners can converge to the optimal solution as learning from original labeled data. The latter implies that learners' performance can generalize to the entire data distribution. Our theory not only guarantees the effectiveness of biased soft labels, but also provides explicit indicators to evaluate the soft labels.
Biased soft labels are prevalent in weakly supervised learning [24, 44, 59], and our theory can offer theoretical insights for these paradigms. In detail, we apply the theory to three classic weakly-supervised learning paradigms: incomplete supervision [3], partial label learning [5] and learning with additive noise [10]. In these weakly-supervised learning paradigms, soft labels are biased and we provide a theoretical guarantee for the learners in these fields. In other words, we prove that the biased soft labels in the three paradigms are effective to train a good model.
It is important to note that, Yuan et al. [53] argues that poorly-trained teachers can teach good students. In fact, their soft labels are a mixture of the teacher's outputs and the ground-truth labels with a ratio of  $0.1:0.9$ , which makes the soft labels close to the ground-truth labels as illustrated in Figure 1(b). Differently, our experiments reveal that large-biased soft labels can also teach good students. Furthermore, the effectiveness of these soft labels can be measured by unreliability degree and ambiguity degree. In addition, experiments on three weakly-supervised learning paradigms also demonstrate that biased soft label learning problem is learnable and the proposed indicators are effective. Our contributions can be summarized as follows:
- We find that learning from large-biased soft labels may also achieve comparable performance and intend to explore the underlying mechanisms behind the effectiveness of the biased soft labels.
- Two indicators have been proposed to measure the effectiveness of soft labels. Based on the indicators, we present moderate conditions to guarantee the effectiveness of the soft labels. It is proved that the biased soft label learning problem is classifier-consistent and ERM learnable.
- A heuristic method is designed to train skillful but bad teachers, i.e., teachers with low accuracy but who can teach good students. We can explain this phenomenon with unreliability degree and ambiguity degree.
- The theory provides a theoretical view for the learners in three weakly-supervised learning paradigms. Specifically, we provide theoretical guarantees for the learnability of these paradigms from the perspective of soft labels. Experimental results are consistent with our theory.
Knowledge Distillation and Label Smoothing Knowledge Distillation (KD) was initially proposed in model compression [22, 21] and then applied to transfer learning [41, 34]. There is growing interest in why distilling can transfer information and what the dark knowledge hidden in the soft labels is. Furthermore, the dark knowledge also exists in the variants of knowledge distillation, which are introduced in the appendix A.1. Label Smoothing (LS) [38] is a regularization method to improve performance by mixing the uniform noise into the ground-truth label. It is convinced that soft labels can restrain overconfidence of the student model. Essentially, both KD and LS can be unified as learning from soft labels Yuan et al. [53].
Label Enhancement Label Distribution Learning (LDL) [13] was proposed to exploit the label distribution to mirror the relationship between the label and the instance, where the formalization of the label distribution is identical to the soft labels mentioned above. In the remainder of the paper, we use nomenclature soft labels. Due to the high cost of labeling the soft labels, Label Enhancement (LE) [48] was proposed to recover the soft label from the logical label by exploiting the implicit correlation among different labels. Numerous novel algorithms have been designed in recent years that aim to improve the predictive model with the soft labels [47, 57]. Wang and Geng [42] applied the margin theory to the soft labels and designed the adaptive margin loss.
In fact, most existing explanations of the soft labels are empirically and experimentally validated, while the rigorous theoretical analyses usually have strong assumptions regarding the model or data distribution. Phuong and Lampert [35] explored the mechanism of distillation where the teacher model and the student model are linear. Allen-Zhu and Li [1] supposed that the instance could be decomposed into multiple independent features and had a linear relationship with the sample, and then proved the effectiveness of the soft labels. Wang and Yoon [43] solved the objective functional problem of self-distillation with the Green's function, which assumes that the network can reach the optimal solution. Menon et al. [31] and Zhou and Song [58] regarded the generated soft labels as the posterior probability and assumed the existence of the Bayes probability. There is also some work analyzing soft labels from the perspective of transfer risk [19, 18]. Most existing theories suggest that students can achieve good performance only when the soft labels are close to the ground-truth labels, and lack explicit indicators to measure the effectiveness of the soft labels.
Let  $\mathcal{X}$  be the instance space,  $\mathcal{Y} = \{1,2,\dots ,c\}$  be the label space with  $c$  classes and  $\Delta$  be the soft-label space over  $\mathcal{V}$ , i.e.,  $\Delta = \{\pmb {d}\in \mathbb{R}^c |\sum_{i\in \mathcal{V}}\pmb {d}_i = 1,\pmb {d}_i\geq 0$  for all  $i\in \mathcal{V}\}$ . Define  $\mathcal{D}$  as the data distribution over  $\mathcal{X}\times \mathcal{Y}$  and  $\mathcal{H}$  as the hypothesis space from  $\mathcal{X}$  to  $\mathcal{V}$ . Each  $h\in \mathcal{H}$  is called the learner or hypothesis. When  $h\in \mathcal{H}$  is a neural network, it often employs a softmax function in the final layer and takes the highest value as the prediction. Slightly abusing notation, we use  $h(\pmb {x})\in \Delta$  to denote the soft label generated by model  $h$ , and  $\tilde{h} (\pmb {x})\in \mathcal{V}$  to represent the model's prediction.
Next, we introduce some important concepts in the Probably Approximately Correct (PAC) learning [40]. The Natarajan dimension [32] was proposed to characterize the capacity of multiclass hypothesis spaces and we denote  $d_{\mathcal{H}}$  as the Natarajan dimension of  $\mathcal{H}$ . The expected classification error of a hypothesis  $h \in \mathcal{H}$  is defined as  $\mathrm{Err}_{\mathcal{D}}(h) = \mathbb{E}_{(x,y) \sim \mathcal{D}}[\mathbb{I}(h(x) \neq y)]$ , where  $\mathbb{I}(\cdot)$  is the indicator function. Given a finite dataset  $\mathbf{z}$ , the empirical classification error is defined as  $\mathrm{Err}_{\mathbf{z}}(h) = \frac{1}{n} \sum_{i=1}^{n} \mathbb{I}(h(\mathbf{x}) \neq y)$ . The Empirical Risk Minimization (ERM) learner  $h^{\mathrm{ERM}}$  is defined as  $h^{\mathrm{ERM}} = \arg \min_{h \in \mathcal{H}} \mathrm{Err}_{\mathbf{z}}^{\Delta}(h|f)$ , while the optimal learner  $h^*$  is defined as  $h^* = \arg \min_{h \in \mathcal{H}} \mathrm{Err}_{\mathcal{D}}(h)$ .
Considering that this study involves two models: generating soft labels and learning from the soft labels. To avoid ambiguity, we use  $f$  to represent the model generating soft labels (teacher) and  $h$  to denote the model trained with soft labels (student). It should be noted that  $f \in \mathcal{H}$  can be a neural network or a map based on certain rules, such as label smoothing.
In order to gain a deeper insight into the concept of biased soft labels, we have provided precise definitions for relevant terms below. These definitions may not have a direct correlation with the theoretical analysis presented in section 3.2, but they serve to elucidate the boundaries of our research.
Definition 1 (Bias of soft labels). Give a dataset  $\mathcal{D}$  consisting of  $n$  samples, the feature vector for the  $i$ -th sample is denoted as  $\pmb{x}_i$  and the corresponding label is denoted as  $y_i$ . Let  $f$  represent a model or a mapping rule. The bias of the soft labels generated by  $f$  on dataset  $\mathcal{D}$  is
$$
\operatorname {B i a s} (f, \mathcal {D}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left[ 1 - f _ {y _ {i}} \left(\boldsymbol {x} _ {i}\right) \right],
$$
where  $f_{y_i}(\pmb{x}_i)$  refers to the component of the soft label  $f(\pmb{x}_i)$  that corresponds to the true label  $y_i$ .
Definition 2 (Large-biased soft labels). Soft labels generated by  $f$  on dataset  $\mathcal{D}$  is called biased soft labels when  $\mathrm{Bias}(f, \mathcal{D}) > 0$  and called large-biased soft labels when  $\mathrm{Bias}(f, \mathcal{D}) \geq 1$ .
The bias, as defined here, essentially represents the disparity between the soft label and the true label. Here, we employ the Manhattan distance, also referred to as L1-norm, although other metrics can also be suitable. A threshold of  $\mathrm{Bias} \geq 1$  is employed to categorize soft labels as "large-biased." In practical scenarios, the choice of the threshold can be adapted based on specific contexts. After establishing the concept of large-biased soft labels, we proceed to define "bad teachers."
Definition 3 (Bad teachers). We define  $f$  as a bad teacher if the soft labels it generates on dataset  $\mathcal{D}$  are large-biased. Typically,  $\mathcal{D}$  is the training set for  $f$ .
The definition of the bad teachers is based on the definition of large-biased soft labels. Similarly, here we adapt 1 as the boundary value for bad teachers. It's important to note that when we use "good/bad" to describe teachers, we are referring to the performance of teachers on the dataset  $\mathcal{D}$ . On the other hand, when we say teachers are "skillful," we are emphasizing that they are adept at instructing students, leading to excellent student performance. Due to the fact that the performance of the student model is contingent on both the model structure and the complexity of the dataset, we are unable to establish precise boundaries for "skillful teachers." In this context, 'skillful' merely signifies that the teacher produces students with acceptable performance.
3.2 Theoretical Analysis of Soft Labels
In this subsection, we define two indicators, unreliability degree and ambiguity degree, to measure the effectiveness of soft labels. Based on these indicators, we present moderate conditions to ensure that the biased soft label learning problem is classifier-consistent and Empirical Risk Minimization (ERM) learnable. The underlying intuition behind the indicators is to convert the soft label into a set. Let  $\Omega_k(\pmb{d}) = \{i \in \mathcal{Y} \mid i \text{ ranks top-} k \text{ in } \pmb{d}\}$  be the set of top  $k$  labels in the soft label  $\pmb{d}$ . Here,  $k$  is a constant ranging in  $\{1, 2, \dots, c - 1\}$ . When  $k = 1$ ,  $\Omega_k(\pmb{d})$  has only one element, i.e., the prediction. For soft labels generated by teacher  $f$ , we define the unreliability degree as,
$$
\eta_ {k} (f) = \Pr_ {(\boldsymbol {x}, y) \sim \mathcal {X} \times \mathcal {Y}} (y \notin \Omega_ {k} (f (\boldsymbol {x}))). \tag {1}
$$
However, it is not enough to measure the soft labels by merely unreliability degree. For example, for images whose ground-truth labels are 1, if label 2 always appears in  $\Omega_k(\pmb{d})$ , then the student model is unable to distinguish label 1 from label 2. Therefore, we introduce the ambiguity degree [4] and extend it to more general soft labels (induced by teacher  $f$ )
$$
\gamma_ {k} (f) = \max  _ {i \in \mathcal {Y}} \Pr_ {\left(\boldsymbol {x}, y\right) \sim \mathcal {X} \times \mathcal {Y}, y \neq i} \left(i \in \Omega_ {k} (f (\boldsymbol {x}))\right). \tag {2}
$$
Ambiguity degree bound the probability of co-occurrence. In other words, if a model  $f$  is with ambiguity degree  $\gamma_k(f)$ , then  $\operatorname*{Pr}(i \in \Omega_k(f(x)) \mid i \neq y, x, y) \leq \gamma_k(f)$ . The smaller  $\eta$  or  $\gamma$  is, the more effective the soft labels are. However, when  $k$  increases,  $\eta$  will decrease and  $\gamma$  is inverse, which means  $k$  should be selected cautiously.
Theorem 1. Training with soft labels generated by the teacher model  $f$ , if  $\gamma_k(f) < 1 - \frac{\eta_k(f)}{1 - \eta_k(f)}$ , then the optimal student model  $h^* \in \mathcal{H}$  satisfies  $h^* = \operatorname*{arg\min}_{h \in \mathcal{H}} \mathrm{Err}_{\mathcal{D}}(h|f)$ .
The proof can be found in A.2. Theorem 1 ensures that the student model  $h$  learning from the teacher can converge to the optimal learner over the entire data distribution. This property is known as classifier-consistency [12]. However, it does not provide the sample complexity bounds of the learning problem. In other words, it does not establish a connection between the generalization bound and the number of training samples.
Next, we provide our main result, a sufficient condition for the ERM learnability of the biased soft label learning problem. In the previous subsection, we presented the definition of expected classification error and empirical classification error, which are based on the label space  $\mathcal{V}$ . Here, given the soft label generating model  $f$ , we define the soft label-based expected error
$$
\mathrm {E r r} ^ {\Delta} (h | f) = \mathbb {E} _ {(\boldsymbol {x}, y) \sim \mathcal {X} \times \mathcal {Y}} [ \mathbb {I} (\tilde {h} (\boldsymbol {x}) \notin \Omega_ {k} (f (\boldsymbol {x}))) ],
$$
and the soft label-based empirical error on dataset  $\mathbf{z} = \{(x_i,y_i)\}_{i = 1}^n$
$$
\operatorname {E r r} _ {\mathbf {z}} ^ {\Delta} (h | f) = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {I} \left(\tilde {h} \left(\boldsymbol {x} _ {i}\right) \notin \Omega_ {k} (f (\boldsymbol {x} _ {i}))\right).
$$
In the above equations, the set  $\Omega_k(f(\pmb{x}_i))$  is determined by the soft label  $f(\pmb{x}_i)$ . We denote  $H_{\eta}$  as the set of teacher models whose generated soft labels are with unreliability degree  $\eta$ , i.e.,  $H_{\eta} = \{f:\eta_k(f) = \eta \}$ . With such soft labels, we analyze the performance of the ERM learners (students). The definition of the ERM learner is provided in the preliminary. Specifically, based on the unreliability degree in (1) and the ambiguity degree in (2), we provide a sufficient condition to ensure that, the biased soft label learning problem is ERM learnable.
Theorem 2 (Main theory). For  $k \in \{1, 2, \dots, c - 1\}$ , assume the unreliability degree  $\eta_k(f)$  and the ambiguity degree  $\gamma_k(f)$  of the soft labels generated by teacher model  $f$ , denoted concisely as  $\eta_k$  and  $\gamma_k$ , and satisfy  $0 < \eta_k, \gamma_k < 1$  and  $\eta_k + \gamma_k < 1$ . Let  $\theta_k = \log \frac{2(1 - \eta_k)}{1 - \eta_k + \gamma_k}$  and suppose the Natarajan dimension of the hypothesis space  $\mathcal{H}$  is  $d_{\mathcal{H}}$ . Define
$$
\begin{array}{l} n _ {0} (\mathcal {H}, \varepsilon , \delta) = \min _ {k \in \{1, 2, \dots , c - 1 \}} \left\{\frac {2}{\frac {\theta_ {k} \varepsilon}{2} + \log \frac {1}{2 - 2 \eta_ {k}}} (d _ {\mathcal {H}} (\log (2 d _ {\mathcal {H}}) \right. \\ \left. \left. + \log \frac {1}{\frac {\theta_ {k} \varepsilon}{2} + \log \frac {1}{2 - 2 \eta_ {k}}} + 2 \log L\right) + \log \frac {1}{\delta} + 1\right) \Bigg \}. \\ \end{array}
$$
Then when  $n > n_0$ , the ERM learner satisfies  $\mathrm{Err}_{\mathcal{D}}(h^{ERM}|f) < \varepsilon$  with probability  $1 - \delta$ .
The proof can be divided into two lemmas. Let define  $H_{\varepsilon}$  be the set of hypotheses with error at least  $\varepsilon$ , i.e.,  $H_{\varepsilon} = \{h \in \mathcal{H} : \operatorname{Err}_{\mathcal{D}}(h|f) \geq \varepsilon \}$ . Our target is to bound the probability  $\operatorname*{Pr}(h \in H_{\varepsilon})$ , which measures the generalization of the learner  $h$ . Since the entire soft label space is inaccessible,  $H_{\varepsilon}$  is evaluated by the mediator  $R_{n,\varepsilon}$  as follows:
$$
R _ {n, \varepsilon} = \left\{\mathbf {z} \in (\mathcal {X} \times \mathcal {Y}) ^ {n}: \exists h \in H _ {\varepsilon}, \mathrm {E r r} _ {\mathbf {z}} ^ {\Delta} (h | f) = 0 \right\}.
$$
Then, our goal is to prove that  $\operatorname{Pr}(\mathbf{z} \in R_{n,\varepsilon} \mid f \in H_{\eta}) \leq \delta$ . In other words, given dataset  $\mathbf{z}$ , the student model  $h$  has the generalization bound greater than  $\varepsilon$  with probability at most  $\delta$ . Since the teacher model  $f$  is intractable and the soft label space  $\Delta$  is unknown, it is very difficult to directly calculate the conditional probability  $\operatorname{Pr}(\mathbf{z} \in R_{n,\varepsilon} \mid f \in H_{\eta})$ . We bound it by introducing a testing set  $\mathbf{z}'$ . Lemma 1 is adapted from [[9], Lemma 11.1.5] and [[16], Corollary 2.6].
Lemma 1. For a testing set  $\mathbf{z}' \in (\mathcal{X} \times \mathcal{Y})^n$ , we can define the set  $S_{n,\varepsilon}$  as
$$
S _ {n, \varepsilon} = \{(\mathbf {z}, \mathbf {z} ^ {\prime}) \in (\mathcal {X} \times \mathcal {Y}) ^ {2 n}: \exists h \in H _ {\varepsilon}, \operatorname {E r r} _ {\mathbf {z}} ^ {\Delta} (h | f) = 0, \operatorname {E r r} _ {\mathbf {z} ^ {\prime}} ^ {\Delta} (h | f) \geq \frac {\varepsilon}{2} \}.
$$
Then, we have  $\operatorname*{Pr}((\mathbf{z},\mathbf{z}^{\prime})\in S_{n,\varepsilon}\mid f\in H_{\eta})\geq \frac{1}{2}\operatorname*{Pr}(\mathbf{z}\in R_{n,\varepsilon}\mid f\in H_{\eta}),$  for  $n > \frac{8\log 2}{\varepsilon}$
The proof of Lemma 1 can be found in appendix A.3. With lemma 1, we just need to estimate  $\operatorname{Pr}((\mathbf{z}, \mathbf{z}') \in S_{n,\varepsilon} \mid f \in H_{\eta})$ . It seems more complicated to introduce the testing set  $\mathbf{z}'$  but we can swap training/testing instance pairs, which is a classic method in the proof of learnability, to refine the data distribution on  $\mathcal{X} \times \mathcal{Y}$  into each single instance.
Lemma 2. On the same condition of theorem 2, we have
$$
\Pr \left((\mathbf {z}, \mathbf {z} ^ {\prime}) \in S _ {n, \varepsilon} \mid f \in H _ {\eta}\right) \leq (2 n) ^ {d _ {\mathcal {H}}} L ^ {2 d _ {\mathcal {H}}} \exp \left(- \frac {n \theta \varepsilon}{2}\right).
$$
The proof of Lemma 2 can be found in appendix A.4. With Lemma 1 and Lemma 2, we can prove Theorem 2, which is detailed in the appendix A.5.
In this section, we establish two essential properties of the soft labels. Classifier-consistency guarantees the effectiveness of the soft labels in a macroscopic perspective, and ERM learnability provides a microcosmic generalization bound for the student model  $h$ . The corresponding threshold conditions are presented to ensure the student model can learn from the soft labels. The theory is applicable to all biased soft labels. This inspires us to consider that accuracy alone may not be a sufficient measure of a teacher's teaching ability. In the section 5.1, we design a simple heuristic algorithm to generate such skillful but bad teachers, i.e., teachers with low accuracy but can teach good students. These results are illustrative for the comprehension and development of the soft label based algorithms.
4 Biased Soft Labels in Weakly-Supervised Learning