SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Without loss of generality, assume that  $\epsilon$  is the largest pseudo-labeling error among  $q$  classes, i.e.,  $\epsilon = \max_{k\in [q]}\sum_{j = 1}^{m}|\mathbb{I}(f_k(x_j)\geq \tau (\alpha_k)) - \mathbb{I}(y_{jk} = 1)| / m$ . Obviously,  $\epsilon$  consists of exactly two types of pseudo-labeling error:
$$
\epsilon_ {1} = \frac {\sum_ {j = 1} ^ {m} \mathbb {I} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right) <   \tau \left(\alpha_ {k}\right) , y _ {j k} = 1\right)}{m} \tag {14}
$$
$$
\epsilon_ {0} = \frac {\sum_ {j = 1} ^ {m} \mathbb {I} (f _ {k} (\boldsymbol {x} _ {j}) \geq \tau (\alpha_ {k}) , y _ {j k} = 0)}{m}
$$
where  $\epsilon_{1}$  calculates the proportion of positive labels being treated as negative ones, and  $\epsilon_{0}$  calculates the proportion of negative labels being treated as positive ones. Then, we prove the following two sides, which provide the bounds for  $\widehat{R}_u'(f)$ . Firstly, we prove its upper bound:
$$
\begin{array}{l} \widehat {R} _ {u} ^ {\prime} (f) = \frac {1}{m} \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {q} \mathbb {I} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right) \geq \tau \left(\alpha_ {k}\right)\right) \ell_ {1} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) + \mathbb {I} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right) <   \tau \left(\alpha_ {k}\right)\right) \ell_ {0} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) \\ \leq \frac {1}{m} \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {q} \mathbb {I} \left(y _ {j k} = 1\right) \ell_ {1} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) + \mathbb {I} \left(y _ {j k} = 0\right) \ell_ {0} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) \\ + \mathbb {I} (y _ {j k} = 0, f _ {k} (\boldsymbol {x} _ {j}) \geq \tau (\alpha_ {k})) \ell_ {1} (f _ {k} (\boldsymbol {x} _ {j})) + \mathbb {I} (y _ {j k} = 1, f _ {k} (\boldsymbol {x} _ {j}) <   \tau (\alpha_ {k})) \ell_ {0} (f _ {k} (\boldsymbol {x} _ {j})) \\ \leq \frac {1}{m} \sum_ {j = 1} ^ {m} \mathcal {L} (f (\boldsymbol {x} _ {j}), \boldsymbol {y} _ {j}) + \epsilon_ {0} \sum_ {k = 1} ^ {q} \ell_ {1} (f _ {k} (\boldsymbol {x} _ {j})) + \epsilon_ {1} \sum_ {k = 1} ^ {q} \ell_ {0} (f _ {k} (\boldsymbol {x} _ {j})) \\ \leq \widehat {R} _ {u} (f) + q B \epsilon \\ \end{array}
$$
where the second line holds based on Eq.(14). Then, we prove its low bound:
$$
\begin{array}{l} \widehat {R} _ {u} ^ {\prime} (f) = \frac {1}{m} \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {q} \mathbb {I} \left(f _ {k} (\boldsymbol {x} _ {j}) \geq \tau \left(\alpha_ {k}\right)\right) \ell_ {1} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) + \mathbb {I} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right) <   \tau \left(\alpha_ {k}\right)\right) \ell_ {0} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) \\ \geq \frac {1}{m} \sum_ {j = 1} ^ {m} \sum_ {k = 1} ^ {q} \mathbb {I} \left(y _ {j k} = 1\right) \ell_ {1} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) + \mathbb {I} \left(y _ {j k} = 0\right) \ell_ {0} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) \\ - \mathbb {I} \left(y _ {j k} = 1, f _ {k} \left(\boldsymbol {x} _ {j}\right) <   \tau \left(\alpha_ {k}\right)\right) \ell_ {1} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) - \mathbb {I} \left(y _ {j k} = 0, f _ {k} \left(\boldsymbol {x} _ {j}\right) \geq \tau \left(\alpha_ {k}\right)\right) \ell_ {0} \left(f _ {k} \left(\boldsymbol {x} _ {j}\right)\right) \\ \geq \frac {1}{m} \sum_ {j = 1} ^ {m} \mathcal {L} (f (\boldsymbol {x} _ {j}), \boldsymbol {y} _ {j}) - \epsilon_ {1} \sum_ {k = 1} ^ {q} \ell_ {1} (f _ {k} (\boldsymbol {x} _ {j})) - \epsilon_ {0} \sum_ {k = 1} ^ {q} \ell_ {0} (f _ {k} (\boldsymbol {x} _ {j})) \\ \geq \widehat {R} _ {u} (f) - q B \epsilon \\ \end{array}
$$
By combining these two sides, we can obtain the following result:
$$
\left| \widehat {R} _ {u} ^ {\prime} (f) - \widehat {R} _ {u} (f) \right| \leq q B \epsilon \tag {15}
$$
For any  $\delta > 0$ , with probability at least  $1 - \delta$ , we have:
$$
\begin{array}{l} R (\hat {f}) \\ \leq \widehat {R} _ {l} (\widehat {f}) + \widehat {R} _ {u} (\widehat {f}) + 2 q L _ {\ell} \mathcal {R} _ {n + m} (\mathcal {F}) + q B \sqrt {\frac {\log \frac {2}{\delta}}{2 N}} \\ \leq \widehat {R} _ {\ell} (\widehat {f}) + \widehat {R} _ {u} ^ {\prime} (\widehat {f}) + q B \epsilon + 2 q L _ {\ell} \mathcal {R} _ {N} (\mathcal {F}) + q B \sqrt {\frac {\log \frac {2}{\delta}}{2 N}} \\ \leq \widehat {R} _ {l} (f) + \widehat {R} _ {u} ^ {\prime} (f) + q B \epsilon + 2 q L _ {\ell} \mathcal {R} _ {N} (\mathcal {F}) + q B \sqrt {\frac {\log \frac {2}{\delta}}{2 N}} \tag {16} \\ \leq \widehat {R} _ {\ell} (f) + \widehat {R} _ {u} (f) + 2 q B \epsilon + 2 q L _ {\ell} \mathcal {R} _ {N} (\mathcal {F}) + q B \sqrt {\frac {\log \frac {2}{\delta}}{2 N}} \\ \leq R (f) + 2 q B \epsilon + 4 q L _ {\ell} \mathcal {R} _ {N} (\mathcal {F}) + 2 q B \sqrt {\frac {\log \frac {2}{\delta}}{2 N}} \\ \end{array}
$$
where the first and fifth lines are based on Eq.(1), and second and fourth lines are due to Lemma 2.
The third line is by the definition of  $\hat{f}$ .
C Details of Experimental Settings
Table 3 reports the detailed characteristics of four benchmark datasets. VOC is a popular multi-label dataset that has been divided into a trainval set containing 5,011 examples and a test set containing 4,952 examples from 20 object categories. MS-COCO is another widely used multi-label dataset, which consists of 82,081 training examples and 40,504 validation examples belonging to 80 different categories. In our experiments, the validation set is used for testing. NUS is incomplete online due to many invalid URLs. Our collection consists of 126,034 training images and 84,226 testing images from 81 classes.
Comparing methods To validate the effectiveness of the proposed method, we compare it with four groups of methods: 1) Three instance-aware pseudo labeling methods: Top-1, which is similar to
Table 3: The detailed characteristics of benchmark datasets.
Dataset	# Training	# Testing	# Classes
VOC	5,011	4,952	20
COCO	82,081	16,416	80
NUS	126,034	84,226	81
FixMatch [38] that selects the most probable label as the ground-truth one; Top- $k$ , which selects the top  $k$  rank labels according to the predicted probabilities as the true labels; IAT, which utilizes the instance-aware thresholding strategy to separates positive and negative labels. 2) Two state-of-the-art MLML methods: LL [19] (includes three variants LL-R, LL-Ct, and LL-Cp), which rejects or corrects the large-loss examples to prevent model from memorizing noisy labels; PLC [46], which performs the pseudo-labeling consistency regularization for recovering the labeling information of potential labels. 3) Two state-of-the-art SSL methods: Adsh [14], which solves the class-imbalance issue in SSL by performing adaptive thresholding; FreeMatch [44], which improves the performance of FixMatch by introducing a self-adaptive class fairness regularization. 4) One state-of-the-art SSMLL method: DRML [43], which performs pseudo labeling by exploring the feature distribution and the label correlation simultaneously. 5) Two baseline methods, BCE, which is the most commonly used loss function for multi-label classification; ASL [34], which improves BCE by utilizing the dynamic down-weighting and hard-thresholding techniques. DRML is the only deep SSMLL method whose source code could be found on the Internet. Furthermore, most MLML methods cannot be applied to the SSMLL scenario, since they assume that a subset of labels have been annotated for each training instance.