SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} R \left(\widetilde {g} _ {\mathrm {C D}}\right) - R \left(g ^ {*}\right) = \left(R \left(\widetilde {g} _ {\mathrm {C D}}\right) - \widetilde {R} _ {\mathrm {C D}} \left(\widetilde {g} _ {\mathrm {C D}}\right)\right) + \left(\widetilde {R} _ {\mathrm {C D}} \left(\widetilde {g} _ {\mathrm {C D}}\right) - \widetilde {R} _ {\mathrm {C D}} \left(\widehat {g} _ {\mathrm {C D}}\right)\right) \\ + \left(\widetilde {R} _ {\mathrm {C D}} \left(\widehat {g} _ {\mathrm {C D}}\right) - R \left(\widehat {g} _ {\mathrm {C D}}\right)\right) + \left(R \left(\widehat {g} _ {\mathrm {C D}}\right) - R \left(g ^ {*}\right)\right) \\ \leq \left| R \left(\widetilde {g} _ {\mathrm {C D}}\right) - \widetilde {R} _ {\mathrm {C D}} \left(\widetilde {g} _ {\mathrm {C D}}\right) \right| + \left| \widetilde {R} _ {\mathrm {C D}} \left(\widehat {g} _ {\mathrm {C D}}\right) - R \left(\widehat {g} _ {\mathrm {C D}}\right) \right| + \left(R \left(\widehat {g} _ {\mathrm {C D}}\right) - R \left(g ^ {*}\right)\right) \\ \leq 4 C _ {\ell} \left(L _ {f} + 1\right) \sqrt {\frac {\ln 2 / \delta}{2 n}} + 4 \left(L _ {f} + 1\right) C _ {\ell} \Delta + 8 L _ {\ell} \Re_ {n} (\mathcal {G}). \\ \end{array}
$$
The first inequality is derived because  $\widetilde{g}_{\mathrm{CD}}$  is the minimizer of  $\widetilde{R}_{\mathrm{CD}}(g)$ . The second inequality is derived from Theorem 5 and Theorem 3. The proof is completed.
G Limitations and Potential Negative Social Impacts
G.1 Limitations
This work focuses on binary classification problems. To generalize it to multi-class problems, we need to convert multi-class classification to a set of binary classification problems via the one-versus-rest or the one-versus-one strategies. In the future, developing methods directly handling multi-class classification problems is promising.
G.2 Potential Negative Social Impacts
This work is within the scope of weakly supervised learning, which aims to achieve comparable performance while reducing labeling costs. Therefore, when this technique is very effective and prevalent in society, the demand for data annotations may be reduced, leading to the increasing unemployment rate of data annotation workers.
H Additional Information about Experiments
In this section, the details of experimental data sets and hyperparameters are provided.
H.1 Details of Experimental Data Sets
The detailed statistics and corresponding model architectures are summarized in Table 4. The basic information of data sets, sources and data split details are elaborated as follows.
For the four benchmark data sets,
- MNIST [61]: It is a grayscale handwritten digits recognition data set. It is composed of 60,000 training examples and 10,000 test examples. The original feature dimension is  $28^{*}28$ , and the label space is 0-9. The even digits are regarded as the positive class while the odd digits are regarded as the negative class. We sampled 15,000 unlabeled data pairs as training data. The data set can be downloaded from http://yann.lecun.com/exdb/mnist/.
Table 4: Characteristics of experimental data sets.
Data Set	# Train	# Test	# Features	# Class Labels	Model
MNIST	60,000	10,000	784	10	MLP
Kuzushiji	60,000	10,000	784	10	MLP
Fashion	60,000	10,000	784	10	MLP
CIFAR-10	50,000	10,000	3,072	10	ResNet-34
Optdigits	4,495	1,125	62	10	MLP
USPS	7,437	1,861	256	10	MLP
Pendigits	8,793	2,199	16	10	MLP
Letter	16,000	4,000	16	26	MLP
- Kuzushiji-MNIST [62]: It is a grayscale Japanese character recognition data set. It is composed of 60,000 training examples and 10,000 test examples. The original feature dimension is  $28^{*}28$ , and the label space is  $\{o', su', na', ma', re', ki', tsu', ha', yo'\}$ . The positive class is composed of 'o', 'su', 'na', 'ma', and 're' while the negative class is composed of 'ki', 'tsu', 'ha', 'ya', and 'wo'. We sampled 15,000 unlabeled data pairs as training data. The data set can be downloaded from https://github.com/rois-codh/kmnist.
- Fashion-MNIST [63]: It is a grayscale fashion item recognition data set. It is composed of 60,000 training examples and 10,000 test examples. The original feature dimension is  $28*28$ , and the label space is  $\{\text{T-shirt}, \text{trouser}, \text{pullover}, \text{dress}, \text{sandal}, \text{coat}, \text{sneaker}, \text{bag}, \text{ankle boot}\}$ . The positive class is composed of 'T-shirt', 'pullover', 'coat', 'shirt', and 'bag' while the negative class is composed of 'trouser', 'dress', 'sandal', 'sneaker', and 'ankle boot'. We sampled 15,000 unlabeled data pairs as training data. The data set can be downloaded from https://github.com/zalandoresearch/fashion-mnist.
- CIFAR-10 [64]: It is a colorful object recognition data set. It is composed of 50,000 training examples and 10,000 test examples. The original feature dimension is  $32*32*3$ , and the label space is {'airplane', 'bird', 'automobile', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'}). The positive class is composed of 'bird', 'deer', 'dog', 'frog', 'cat', and 'horse' while the negative class is composed of 'airplane', 'automobile', 'ship', and 'truck'. We sampled 10,000 unlabeled data pairs as training data. The data set can be downloaded from https://www.cs.toronto.edu/~kriz/cifar.html.
For the four UCI data sets, they can be downloaded from Dua and Graff [65].
- Optdigits, USPS, Pendidigits [65]: They are handwritten digit recognition data set. The train-test split can be found in Table 4. The feature dimensions are 62, 256, and 16 respectively and the label space is 0-9. The even digits are regarded as the positive class while the odd digits are regarded as the negative class. We sampled 1,200, 2,000, and 2,500 unlabeled data pairs for training respectively.
- Letter [65]: It is a letter recognition data set. It is composed of 16,000 training examples and 4,000 test examples. The feature dimension is 16 and the label space is the 26 capital letters in the English alphabet. The positive class is composed of the top 13 letters while the negative class is composed of the latter 13 letters. We sampled 4,000 unlabeled data pairs for training.
H.2 Details of Experiments on the KuaiRec Data Set
We used the small matrix of the KuaiRec [72] data set since it has dense confidence scores. It has 1,411 users and 3,327 items. We clipped the watching ratio above 2 and regarded the examples with watching ratio greater than 2 as positive examples. Following the experimental protocol of He et al. [73], we regarded the latest positive example foe each user as the positive testing data, and sampled 49 negative testing data to form the testing set for each user. The HR and NDCG were calculated at top 10. The learning rate was set to 1e-3 and the dropout rate was set to 0.5. The number of epochs was set to 50 and the batch size was set to 256. The number of MLP layers was 2 and the embedding dimension was 128. The hyperparameters was the same for all the approaches for a fair comparison.
H.3 Details of Hyperparameters
For MNIST, Kuzushiji-MNIST and Fashion-MNIST, the learning rate was set to 1e-3 and the weight decay was set to 1e-5. The batch size was set to 256 data pairs. For training the probabilistic classifier to generate confidence, the batch size was set to 256 and the epoch number was set to 10.
For CIFAR10, the learning rate was set to 5e-4 and the weight decay was set to 1e-5. The batch size was set to 128 data pairs. For training the probabilistic classifier to generate confidence, the batch size was set to 128 and the epoch number was set to 10.
For all the UCI data sets, the learning rate was set to 1e-3 and the weight decay was set to 1e-5. The batch size was set to 128 data pairs. For training the probabilistic classifier to generate confidence, the batch size was set to 128 and the epoch number was set to 10.
The learning rate and weight decay for training the probabilistic classifier were the same as the setting for each data set correspondingly.
I More Experimental Results with Fewer Training Data
Figure 4 shows extra experimental results with fewer training data on other data sets with different class priors.
(a) Kuzushiji
(g) MNIST
(i) Kuzushiji
(j) Fashion
(k) USPS
(1) Letter
(m) Optdigits
Figure 4: Classification performance of ConfDiff-ReLU and ConfDiff-ABS given a fraction of training data as well as Pcomp-Teacher given  $100\%$  of training data with different prior settings  $(\pi_{+} = 0.2$  for the first and the second row and  $\pi_{+} = 0.8$  for the third and the fourth row).
(n)Pendigits
(o) MNIST
(p) CIFAR-10