SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $\mathcal{H} \mid (x, x')$  be the hypothesis space that have different prediction for instances  $(x, x')$ . Next, we have the bound
$$
\sum_ {\sigma \in G} \Pr (\sigma (\mathbf {z}, \mathbf {z} ^ {\prime}) \in S _ {n, \varepsilon} \mid x, y, x ^ {\prime}, y ^ {\prime}, f \in H _ {\eta}) \leq \sum_ {h \in \mathcal {H} | (x, x ^ {\prime})} \sum_ {\sigma \in G} \Pr (\sigma (\mathbf {z}, \mathbf {z} ^ {\prime}) \in S _ {n, \varepsilon} ^ {h} \mid x, y, x ^ {\prime}, y ^ {\prime}, f \in H _ {\eta}).
$$
By [33], the complexity of the hypothesis space  $H\mid (x,x^{\prime})$  can be bounded as
$$
\left| \mathcal {H} \mid (x, x ^ {\prime}) \right| \leq (2 n) ^ {d _ {H}} L ^ {2 d _ {H}}.
$$
Let  $d_{x_i} = f(x_i)$  be the soft label of  $x_i$  induced by model  $f$  and  $\Omega_k(d_{x_i})$  be the top-  $k$  set of the soft label. Then,
$$
\begin{array}{l} \Pr \left(\sigma (\mathbf {z}, \mathbf {z} ^ {\prime}) \in S _ {n, \varepsilon} ^ {h} \mid x, y, x ^ {\prime}, y ^ {\prime}, f \in H _ {\eta}\right) \\ = \mathbb {I} \left(\operatorname {E r r} _ {\mathbf {z} ^ {\prime \sigma}} (h | f) \geq \frac {\varepsilon}{2} \mid f \in H _ {\eta}\right) \cdot \Pr \left(\tilde {h} \left(x _ {i} ^ {\sigma}\right) \in \Omega_ {k} \left(d _ {x _ {i}}\right), 1 \leq i \leq n \mid x ^ {\sigma}, y ^ {\sigma}, f \in H _ {\eta}\right) \\ = \mathbb {I} \left(\operatorname {E r r} _ {\mathbf {z} ^ {\prime \sigma}} (h | f) \geq \frac {\varepsilon}{2} \mid f \in H _ {\eta}\right) \cdot \prod_ {i = 1} ^ {n} \Pr \left(\tilde {h} \left(x _ {i} ^ {\sigma}\right) \in \Omega_ {k} \left(d _ {x _ {i}}\right) \mid x ^ {\sigma}, y ^ {\sigma}, f \in H _ {\eta}\right). \\ \end{array}
$$
For the pair of  $(x,y,x^{\prime},y^{\prime})$ , we consider the number of instances of all cases. Specifically, let  $u_{1}$ ,  $u_{2}$  and  $u_{3}$  represent the number of both incorrectly predicted instances, one incorrectly predicted instances and both correctly predicted instances. Besides, we define  $u_{\sigma}$  as the number of instances where  $(x^{\sigma},y^{\sigma})$  is incorrectly predicted while  $(x^{\prime \sigma},y^{\prime \sigma})$  is correctly predicted. Afterwards, the number of incorrectly predicted instances in the testing set is  $u_{1} + u_{2} - u_{\sigma}$ .
$$
\begin{array}{l} \mathbb {I} \left(\operatorname {E r r} _ {\mathbf {z} ^ {\prime \sigma}} (h | f) \geq \frac {\varepsilon}{2} \mid f \in H _ {\eta}\right) = \mathbb {I} \left(u _ {1} + u _ {2} - u _ {\sigma} \geq \frac {\varepsilon}{2} n\right) \\ \leq \mathbb {I} (u _ {1} + u _ {2} \geq \frac {\varepsilon}{2} n). \\ \end{array}
$$
For  $\operatorname{Pr}\left(\tilde{h}(x_i^\sigma) \in \Omega_k(d_{x_i}) \mid x^\sigma, y^\sigma, f \in H_\eta\right)$ , we count instances which have been swapped. On the one side, there are  $u_2 + u_3 - u_\sigma$  instances are correctly predicted  $(\tilde{h}(x_i^\sigma) = y_i^\sigma)$  where we have  $\operatorname{Pr}\left(\tilde{h}(x_i^\sigma) \in \Omega_k(d_{x_i}) \mid f \in H_\eta\right) = 1 - \eta$  by (1). On the other side, there are  $u_1 + u_\sigma$  instances are incorrectly predicted  $(\tilde{h}(x_i^\sigma) \neq y_i^\sigma)$ , where we have  $\operatorname{Pr}\left(\tilde{h}(x_i^\sigma) \in \Omega_k(d_{x_i}) \mid f \in H_\eta\right) \leq \gamma$  by (2). So, we have
$$
\prod_ {i = 1} ^ {n} \Pr \left(\tilde {h} \left(x _ {i} ^ {\sigma}\right) \in \Omega_ {k} \left(d _ {x _ {i}}\right) \mid x ^ {\sigma}, y ^ {\sigma}, f \in H _ {\eta}\right) \leq (1 - \eta) ^ {u _ {2} + u _ {3} - u _ {\sigma}} \cdot \gamma^ {u _ {1} + u _ {\sigma}}.
$$
And then, the conditional probability can be bounded as follow:
$$
\Pr \left(\sigma (\mathbf {z}, \mathbf {z} ^ {\prime}) \in S _ {n, \varepsilon} ^ {h} \mid x, y, x ^ {\prime}, y ^ {\prime}\right) \leq \mathbb {I} \left(u _ {1} + u _ {2} \geq \frac {\varepsilon}{2} n\right) \cdot (1 - \eta) ^ {u _ {2} + u _ {3} - u _ {\sigma}} \cdot \gamma^ {u _ {1} + u _ {\sigma}}.
$$
There are  $2^{n}$  different swaps and we sum all.
$$
\begin{array}{l} \sum_ {\sigma \in G} \mathbb {I} \left(u _ {1} + u _ {2} \geq \frac {\varepsilon}{2} n\right) \cdot (1 - \eta) ^ {u _ {2} + u _ {3} - u _ {\sigma}} \cdot \gamma^ {u _ {1} + u _ {\sigma}} \\ \leq 2 ^ {u _ {1} + u _ {3}} \cdot \mathbb {I} \left(u _ {1} + u _ {2} \geq \frac {\varepsilon}{2} n\right) \cdot \sum_ {j = 0} ^ {u _ {2}} \binom {u _ {2}} {j} (1 - \eta) ^ {u _ {2} + u _ {3} - j} \cdot \gamma^ {u _ {1} + j} \\ = 2 ^ {n - u _ {2}} \cdot (1 - \eta) ^ {u _ {2} + u _ {3}} \cdot \gamma^ {u _ {1}} \cdot \mathbb {I} \left(u _ {1} + u _ {2} \geq \frac {\varepsilon}{2} n\right) \cdot \sum_ {j = 0} ^ {u _ {2}} \binom {u _ {2}} {j} (\frac {\gamma}{1 - \eta}) ^ {j} \\ = 2 ^ {n - u _ {2}} \cdot (1 - \eta) ^ {n - u _ {1}} \cdot \gamma^ {u _ {1}} \cdot \mathbb {I} \left(u _ {1} + u _ {2} \geq \frac {\varepsilon}{2} n\right) \cdot (1 + \frac {\gamma}{1 - \eta}) ^ {u _ {2}} \\ = \mathbb {I} \left(u _ {1} + u _ {2} \geq \frac {\varepsilon}{2} n\right) \cdot 2 ^ {n - u _ {2}} \cdot (1 - \eta) ^ {n - u _ {1}} \cdot \gamma^ {u _ {1}} \cdot (\frac {1 - \eta + \gamma}{1 - \eta}) ^ {u _ {2}} \\ = \mathbb {I} \left(u _ {1} + u _ {2} \geq \frac {\varepsilon}{2} n\right) \cdot (2 - 2 \eta) ^ {n} \cdot \left(\frac {\gamma}{1 - \eta}\right) ^ {u _ {1}} \cdot \left(\frac {1 - \eta + \gamma}{2 (1 - \eta)}\right) ^ {u _ {2}}. \\ \end{array}
$$
where  $n = u_1 + u_2 + u_3$ . The  $2^{u_1 + u_3}$  in the first step means ways of swapping both correct instances and both incorrect instances. The index of  $j$  equals  $u_2$  different swaps of one correctly predicted instances. According to the assumption  $\eta + \gamma \leq 1$ , we have  $0 < \frac{\gamma}{1 - \eta} < \frac{1 - \eta + \gamma}{2(1 - \eta)} < 1$ . For  $u_1 + u_2 \geq \frac{\varepsilon}{2} n$ , when  $u_1 = 0$  and  $u_2 = \frac{\varepsilon}{2} n$ , the right side reaches its maximum as follows:
$$
\Pr \left(S _ {n, \varepsilon} \mid f \in H _ {\eta}\right) \leq (2 n) ^ {d _ {H}} \cdot L ^ {2 d _ {H}} \cdot (2 - 2 \eta) ^ {n} \cdot \left(\frac {1 - \eta + \gamma}{2 (1 - \eta)}\right) ^ {\frac {n \varepsilon}{2}}.
$$
We have proved the Lemma 2.
A.5 Proof of Theorem 2
With Lemma 1 and Lemma 2, we have
$$
\Pr \left(R _ {n, \varepsilon} \mid f \in H _ {\eta}\right) \leq 2 ^ {d _ {H} + 1} \cdot n ^ {d _ {H}} \cdot L ^ {2 d _ {H}} \cdot (2 - 2 \eta) ^ {n} \cdot \left(\frac {1 - \eta + \gamma}{2 (1 - \eta)}\right) ^ {\frac {n \varepsilon}{2}}.
$$
We set  $\theta$  as
$$
\theta = \log \frac {2 (1 - \eta)}{1 - \eta + \gamma}.
$$
Since  $\eta +\gamma < 1$  , we get  $\theta >0$  . We need to bound  $\operatorname *{Pr}\left(R_{n,\varepsilon}\mid f\in H_{\eta}\right)$  with  $\eta$  , which means
$$
\left(d _ {H} + 1\right) \cdot \log 2 + d _ {H} \log n + 2 d _ {H} \log L + n \log (2 - 2 \eta) - \frac {\theta \varepsilon n}{2} \leq \log \eta .
$$
Note that the function  $f(x) = \log \frac{1}{a} + ax - -1 \geq 0$ . Let  $a = \frac{\frac{\theta\varepsilon}{2} + \log\left(\frac{1}{2 - 2n}\right)}{d_H}$  and  $x = n$ . It can be inferred that
$$
\log n \leq \frac {\frac {\theta \varepsilon}{2} + \log (\frac {1}{2 - 2 \eta})}{d _ {H}} n - \log \frac {\frac {\theta \varepsilon}{2} + \log (\frac {1}{2 - 2 \eta})}{d _ {H}} - 1.
$$
With the bound of  $\log n$ , we get the linear inequality of  $n$ . Since the aforementioned results hold for all  $k \in \{1, 2, \ldots, c - 1\}$ , Let
When  $n > n_0$ , we get  $\operatorname*{Pr}\left(R_{n,\varepsilon} \mid f \in H_\eta\right) < \eta$  and the proof is finished.
Figure 4: Average of the soft labels in CIFAR-10 (a) and CIFAR-100 (b).
A.6 Proof of Theorem 3
Generally, for epoch  $t$ , we denote  $\eta_t$  and  $\gamma_t$  as the unreliability degree and ambiguity degree of the soft labels of the unlabeled data. The model will achieve the accuracy of epoch  $t$ ,  $\rho(\eta_t, \gamma_t)$ . We refer to it as  $\rho_t$  for simplicity. Then, for next epoch  $t + 1$ , the unreliability degree and ambiguity degree of next epoch can be estimated  $\eta_{t + 1} \leq 1 - \rho_t$ . The estimation of ambiguity degree is a little bit more complicated. Incorrect label share equal probability  $\gamma_{t + 1}$  because we assume the noise is uniformly distributed. Then we have
$$
\eta_ {t + 1} + (c - 1) \gamma_ {t + 1} = k.
$$
$$
\gamma_ {t + 1} \leq \frac {c - k - \rho_ {t}}{c - 1}.
$$
For  $\rho (\eta ,\gamma)$  monotonically decreases, with the upper bounds on  $\eta_{t + 1}$  and  $\gamma_{t + 1}$ , we can get a lower bound on  $\rho_{t + 1}$  as
$$
\begin{array}{l} \rho_ {t + 1} = \rho (\eta_ {t + 1}, \gamma_ {t + 1}) \\ \geq \rho (1 - \rho_ {t}, \frac {c - k - \rho_ {t}}{c - 1}). \\ \end{array}
$$
If  $\rho_{\mathrm{final}} = \lim_{t\to \infty}\rho_t$  exists, it must satisfy the fix point equation,
$$
x = \rho (1 - x, \frac {c - k - x}{c - 1}).
$$
Next, we prove that if  $\rho (\eta ,\gamma)$  is  $k_{L}$ -Lipschitz continuous  $(k_{L} < 1 - \frac{1}{c})$ , then  $\rho_{\mathrm{final}}$  exists and is unique. We define
$$
\psi : (\eta , \gamma) \to (1 - \rho (\eta , \gamma), \frac {c - k}{c - 1} - \frac {\rho (\eta , \gamma)}{c - 1}),
$$
where  $(\eta, \gamma) \in [0,1]^2$ .  $l_{1}$ -norm is employed as the norm on  $[0,1]^2$  and denote  $d(\cdot, \cdot)$  as the distance function. We want to show that  $\psi$  is a contraction mapping. For  $(\eta_1, \gamma_1), (\eta_2, \gamma_2) \in [0,1]^2$ ,
$$
\begin{array}{l} d (\psi (\eta_ {1}, \gamma_ {1}), \psi (\eta_ {2}, \gamma_ {2})) \\ = (1 + \frac {1}{c - 1}) | \rho (\eta_ {1}, \gamma_ {1}) - \rho (\eta_ {2}, \gamma_ {2}) | \\ \leq \left(1 + \frac {1}{c - 1}\right) \cdot k _ {L} d \left(\left(\eta_ {1}, \gamma_ {1}\right), \left(\eta_ {2}, \gamma_ {2}\right)\right) \\ \end{array}
$$
where  $(1 + \frac{1}{c - 1})\cdot k_L\in [0,1)$ . So  $\psi$  is a contraction mapping and there is a unique fixed point  $(\eta ,\gamma)$  that  $\psi (\eta ,\gamma) = (\eta ,\gamma)$ . That means  $\rho_{\mathrm{final}}$  exists and is unique.
Table 2: Different metrics of soft labels.
Dataset	Setting	Acc(Teacher)	Cheby	KL	Manhattan	Euclidean	Acc(Student)
CIFAR-10	Supervised	95.29	0.0630	0.3335	0.1259	0.1728	95.68
SBTs	19.07	0.8323	1.9295	1.6646	0.9093	88.53
20.98	0.8312	1.9619	1.6625	0.9129	88.85
22.00	0.8348	1.9197	1.6696	0.9078	89.32
22.95	0.8213	1.8781	1.6427	0.9008	89.57
24.73	0.8357	1.8968	1.6715	0.9058	90.22
26.54	0.8396	1.9110	1.6768	0.9076	90.43
28.07	0.8390	1.9099	1.6489	0.9150	90.58
CIFAR-100	Supervised	78.13	0.1998	1.3763	0.3996	0.2630	79.01
SBTs	21.85	0.9145	3.4739	1.8290	0.9467	68.42
22.74	0.9078	3.3814	1.8156	0.9391	69.28
24.55	0.9093	3.3457	1.8187	0.9392	69.55
25.35	0.9064	3.3299	1.8129	0.9370	69.88
26.53	0.9008	3.2921	1.8016	0.9320	71.22
27.22	0.9046	3.2858	1.8092	0.9329	71.34
28.05	0.9009	3.2560	1.8018	0.9302	72.08
A.7 Experiment Setup
We consider three benchmark image datasets CIFAR-10, CIFAR-100 [23] and Tiny ImageNet, and generate the soft labels with different hyperparameters of the teacher model. The student model is trained with the generated soft labels and aims to distinguish the ground-truth label. The accuracy of the student model is employed to measure the effectiveness of the soft labels. Datasets are divided into training, validation, testing set in the ratio of 4:1:1. For the fairness of the experiments, all student models are WideResNet  $28 \times 2$  architecture [54] on each dataset. In all experiments, we use mini-batch SGD [36] with a batch size of 128 and a momentum of 0.9. Each model is trained with maximum epochs  $T = 200$  and employs early stopping strategy with patience 20. We report final performance using the test accuracy corresponding to the best accuracy on validation set. For the all experiments, we employ a basic re-weighting strategy to train the student model. Specifically, in each epoch, we train the student model and update the soft labels with the softmax outputs of it.
Weakly-supervised learning For PLL, we employ a common strategy to zeroize the soft labels that are not in the candidate set. For learning with unlabeled data, we warm-up the model for 5 epochs. Then, we train with all data every epoch and label the unlabeled data with the model every 5 epochs. Each dataset is divided into labeled set, unlabeled set, testing set in the ratio of 1:19:4.
A.8 Hyperparameters in (8)
We set the number of the random labels (in 7) as 3 while  $k$  in  $\eta_{k}$  and  $\gamma_{k}$  as 4 in all experiments. The punishment factor  $\alpha_{1}$  ranges from 0 to 0.4 and the compensation factor  $\alpha_{2}$  ranges from 0.9 to 1.3. The weight of the random labels  $\alpha_{3}$  is from 1.6 to 2.3. In detail, we observe the following phenomenons:
- As mentioned in 1, we cannot take the accuracy of the teacher as the sole criterion to evaluate the pedagogical efficacy.  $\gamma$  defined in (2) can be seen as the coarse measure of the imbalance of the label. For the student model, the smaller  $\gamma$  could mean the better performance if the number of random labels is fixed.
- Both the punishment and the random labels are applied to decrease the top-1 accuracy within a reasonable range. For the simple dataset, large punishment are needed to protect privacy and for the complicated dataset like CIFAR-100, we enhance the ratio of random labels to reduce the effect of similar labels.
- These soft labels are very different from the ground-truth labels, which means low accuracy. In these soft labels, the average degree of the ground-truth labels is around 0.2.
Figure 5: The curve of the indicators in incomplete supervision.
(b)
Figure 6: The experiments of 4 different backbones: WRN  $28 \times 2$ , WRN  $28 \times 4$ , WRN  $40 \times 2$ , WRN  $40 \times 4$ . On the left are the experiments conducted on CIFAR-10, while on the right are those performed on CIFAR-100.
A.9 Experiments of different backbones
Figure 6 demonstrates whether the effectiveness of soft labels is influenced by different backbones. We experimented with wideresnet  $28\mathrm{x}2$ ,  $28\mathrm{x}4$ ,  $40\mathrm{x}2$ , and  $40\mathrm{x}4$ . In the Figure 2, we omitted the Unreliability degree and Ambiguity degree, which are the same the that of Figure 2 in the paper, for clarity. The four distinct backbones displayed consistent trends, further suggesting that the proposed indicators are effective across different backbones.
A.10 Overall distribution of the Soft Labels Generated by SBTs
As illustrated in Figure 4, the horizontal axis represents the ground-truth label while the vertical axis represents the mean of the soft labels. The diagonal can be seen as the degree of correctly predicted labels. We can see that the ground-truth label is dominant in the soft labels. On the other side, the figure can be seen as a simple measure of the similarity between labels.
A.11 Experiments of Incomplete Supervision
The predictive model label the unlabeled data and then learn with all data alternately. As shown in Figure 5, Acc (the accuracy of the model) improves as the soft labels evolve. Accordingly, unreliability degree  $\eta_{k}(f)$  decreases and ambiguity degree  $\gamma_{k}(f)$  remain unchanged, which means the soft labels of the model are more effective. The figure shows the dynamics in the training with unlabeled data, which is consistent with the theory in 4.1. Furthermore, we find that the soft labels of unlabeled data exhibit a high ambiguity degree  $\gamma_{k}(f)$ , indicating that some categories may be very similar and difficult for the model to distinguish.