SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 3.3. Let  $c_{1}, c_{2}$  and  $\sigma > 0$ . Let  $\mathbf{x}_i \in X$  be a set of  $n$  i.i.d. points from  $\mathcal{X}$  with corresponding true labels  $y_i$ . Given  $(h, t) \in \mathcal{H}^{T,g}$ , let  $\mathbb{E}\left[\left(\ell_{0-1}^{\perp}(h, t, \mathbf{x}_i, y_i) - \mathcal{E}(h, t|\mathcal{X})\right)^2\right] = \sigma_i^2 > \sigma^2$  for every  $\mathbf{x}_i$  for  $\sigma_i > 0$  and let  $\sum_{i}^{n} \sigma_i^2 \geq c_1$  then for every  $\epsilon \in [0, (\sum_{i=1}^{n} \sigma_i^2) / \sqrt{c_1}]$  with  $n_v < 12\sigma^2 \log(4c_2) / \epsilon^2$ , the following holds w.p. at least  $1/4$ ,  $\mathcal{E}_a(h, t|\mathcal{X}) > \widehat{\mathcal{E}}_a(h, t|X) + \epsilon$ .
Therefore, if a sufficiently large validation set is not used in each round, there is a constant probability of erroneously deciding on a threshold for auto-labeling. Such a requirement on validation data also applies to active learning if we seek to validate the output model. Bypassing this requirement demands the use of approaches that are different from threshold-based auto-labeling and traditional validation techniques. We note the possibility of using recently proposed active testing techniques [33], a nascent approach to reducing validation data usage.
3.1 Linear Classifier Setting
Next, we consider a simple setting where active learning is known to be optimal to see if TBAL can offer similar performance guarantees. To do so, we instantiate results from 3.2 to homogeneous linear separators under the uniform distribution in the realizable setting. Let  $P_{\mathbf{x}}$  be supported on the unit ball in  $\mathbb{R}^d$ ,  $\mathcal{X} = \{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}|| \leq 1\}$ . Let  $\mathcal{W} = \{\mathbf{w} \in \mathbb{R}^d : ||\mathbf{w}||_2 = 1\} = \mathbb{S}_d$ ,  $\mathcal{H} = \{\mathbf{x} \mapsto \mathrm{sign}(\langle \mathbf{w}, \mathbf{x} \rangle) \forall \mathbf{w} \in \mathcal{W}\}$ , the score function given by  $g(h, \mathbf{x}) = g(\mathbf{w}, \mathbf{x}) = |\langle \mathbf{w}, \mathbf{x} \rangle|$ , and set  $T = [0, 1]$ . For simplicity, we will use  $\mathcal{W}$  in place of  $\mathcal{H}$ .
Corollary 3.4. (Overall Auto-Labeling Error and Coverage) Let  $\hat{\mathbf{w}}_i, \hat{t}_i$  be the ERM solution and the auto-labeling threshold respectively at epoch  $i$ . Let  $n_v^{(i)}, n_a^{(i)}$  denote the number of validation and auto-labeled points at epoch  $i$ . Let the TBAL algorithm run for  $k$ -epochs. Then, for any  $\delta \in (0,1)$ , w.p. at least  $1 - \delta$ ,
$$
\begin{array}{l} \widehat {\mathcal {E}} \big (X _ {p o o l} (A _ {k}) \big) \leq \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \Bigg (\underbrace {\widehat {\mathcal {E}} _ {a} \big (\hat {\mathbf {w}} _ {i} , \hat {t} _ {i} | X _ {v} ^ {(i)} \big)} _ {(a)} + \underbrace {\frac {4}{p _ {0}} \sqrt {\frac {2}{n _ {v} ^ {(i)}} \Big (2 d \log \Big (\frac {e n _ {v} ^ {(i)}}{d} \Big) + \log \Big (\frac {8 k}{\delta} \Big) \Big)}} _ {(b)} \Bigg) \\ + \underbrace {\frac {4}{p _ {0}} \left(\sqrt {\frac {2 k}{N _ {a} ^ {(k)}} \left(2 d \log \left(\frac {e N _ {a} ^ {(k)}}{d}\right) + \log \left(\frac {8 k}{\delta}\right)\right)}\right)} _ {, a n d} \\ \end{array}
$$
$$
\widehat {\mathcal {P}} \big (X _ {p o o l} (A _ {k}) \big) \geq 1 - \min _ {i} \widehat {t} _ {i} \sqrt {4 d / \pi} - 2 k \sqrt {\frac {2}{N} \Big (2 d \log \Big (\frac {e N}{d} \Big) + \log \Big (\frac {8 k}{\delta} \Big) \Big)}.
$$
These results imply that by ensuring the sum of the empirical validation error term (a) and the upper confidence interval to be less than  $\epsilon_{a}$  in each round of the algorithm we can ensure that the overall auto-labeling error remains below  $\epsilon_{a}$ . Furthermore, by applying standard VC theory to the first round, we obtain that  $\hat{t}_{1} \leq 1/2$ . Therefore, right after the first round, we are guaranteed to label at least half of the unlabeled pool. We empirically observe that TBAL has coverage at par with active learning while respecting the auto-labeling error constraint (See Figure 4(a)).
Tightness of the Bounds. We study this in the setting of the Unit-Ball experiment. The upper bound on excess risk in this setting is given in Corollary 3.4 which is an instantiation of our general results to this specific setting. We consider a simplified form of the upper bound by ignoring the constants to get a sense of the rate in terms of the validation data size. We compute this simplified upper bound for different amounts of validation data. We compare these with the maximum auto-labeling error observed over 25 runs of auto-labeling in the Unit-Ball setting with different random seeds
Figure 3: Left: Simplified upper bound from Corollary 3.4 (ignoring constants) on excess auto-labeling error for the Unit-Ball setting i.e. homogeneous linear classifier with  $d = 30$ . Right: The worst observed auto-labeling error over 25 trials in the Unit-Ball experiment.
for each validation data size. The results are in Figure 3. As expected, we see that the worst-case error rate follows a similar rate as our upper bound but the upper bound is conservative. Next, we explain why this is the case.
Our upper bound is slightly conservative, as it is based on a uniform bound over all hypotheses in a given hypothesis class. Since the individual hypotheses whose excess auto-labeling error we need to bound are not known a priori we need to derive a bound on the number of validation samples using which we can guarantee that the excess auto-labeling error of any hypothesis (model) is small with high probability. Note that this is a conservative (worst-case) analysis to get an upper bound on the validation sample complexity. The upper bound has two parts: a) the Rademacher complexity of the hypothesis class and b) a term with the number of validation samples. We note that on the validation samples, it matches lower bounds order-wise (see Lemma 3.3). This is the first analysis to provide these bounds based on uniform convergence without making any assumptions about the data distributions or hypothesis class. We provide further discussion on the role of Rademacher complexity in the Appendix C.1.
We study the effectiveness of TBAL on synthetic and real datasets. We validate our theoretical results and aim to understand the amount of labeled validation and training data required to achieve a certain auto-labeling error and coverage. We also seek to understand whether our findings apply to real data—where labels may be noisy—along with how TBAL performs compared to common baselines.
Baselines. We compare TBAL to the following methods:
a) Passive Learning (PL) queries a subset of the points randomly to train a model from a given model class and then uses it to predict the labels for the remaining unlabeled pool.
b) Active Learning (AL) (using margin-random query strategy, described below) trains a model from a model class and uses it to predict the labels for the remaining unlabeled pool.
c) Passive Labeling + Selective Classification  $(PL + SC)$  first performs passive learning to train a model from a given model class. Then it performs auto-labeling on the unlabeled data using threshold-based selective classification with the model output by passive learning. Only those unlabeled points that are deemed as fit to be labeled by the selection function are auto-labeled.
d) Active Learning + Selective Classification (AL+SC) first performs active learning (using margin-random query strategy) to train a model from a given model class. It then performs auto-labeling using threshold-based selective classification with the model output by AL.
For selective classification in the above methods, we use Algorithm 2 to estimate the threshold and use it to perform auto-labeling. In experiments, we modify Algorithm 2 slightly—instead of estimating a single threshold for all classes, we estimate thresholds for each class separately.
Active Querying Strategy. We use the margin-random query strategy for querying the next batch of training data. In this strategy, the algorithm first sorts the points based on the margin (uncertainty) score and then selects the top  $C n_b$  ( $C > 1$ ) points from which  $n_b$  points are picked at random. This is a simple and computationally efficient method that balances the exploration and exploitation trade-off. We note that other active-querying strategies exist; we use margin-random as our standard querying strategy to keep the focus on comparing auto-labeling—not active learning approaches.
Nv	Error (%)	Coverage (%)
TBAL	AL+SC	TBAL	AL+SC
100	3.10 ±1.80	0.68 ±0.81	71.43 ±8.86	96.95 ±1.01
400	1.97 ±0.76	0.59 ±0.18	93.99 ±2.39	97.89 ±0.50
800	1.64 ±0.50	0.66 ±0.19	96.26 ±1.33	98.06 ±0.53
1200	1.39 ±0.39	0.67 ±0.19	96.67 ±0.84	98.10 ±0.45
1600	1.33 ±0.30	0.70 ±0.19	97.13 ±0.45	98.16 ±0.44
2000	1.28 ±0.34	0.71 ±0.21	97.15 ±0.54	98.20 ±0.34
\(N_v\)	Error (%)	Coverage (%)
TBAL	AL+SC	TBAL	AL+SC
100	3.10 ±1.80	0.68 ±0.81	71.43 ±8.86	96.95 ±1.01
400	1.65 ±0.65	0.32 ±0.15	93.27 ±2.50	96.91 ±0.99
800	1.08 ±0.47	0.24 ±0.16	96.01 ±1.16	96.31 ±1.36
1200	0.78 ±0.27	0.17 ±0.11	96.82 ±0.84	95.96 ±1.40
1600	0.65 ±0.20	0.13 ±0.08	96.93 ±0.57	95.70 ±1.38
2000	0.54 ±0.16	0.21 ±0.11	97.23 ±0.42	96.36 ±1.13
Table 1: Unit-Ball. Effect of variation of validation data size  $(N_v)$  with and without using a UCB on error estimates. We keep training data size  $N_q$  fixed at 500 and use error threshold  $\epsilon_a = 1\%$ . We report the mean and std. deviation over 10 runs with different random seeds. Left: with  $C_1 = 0$ . Right: with  $C_1 = 0.25$ .
\( {\mathrm{N}}_{\mathrm{v}} \)	Error (%)	Coverage (%)
TBAL	AL+SC	TBAL	AL+SC
200	4.77 ±0.18	3.35 ±0.80	83.14 ±3.65	78.53 ±7.05
400	4.57 ±0.26	3.53 ±0.73	90.70 ±3.11	86.39 ±5.11
600	4.32 ±0.17	3.70 ±0.63	92.96 ±0.46	88.90 ±4.83
800	4.66 ±0.20	3.84 ±0.70	92.42 ±0.89	88.67 ±3.88
1000	4.67 ±0.16	3.90 ±0.68	92.89 ±0.91	89.79 ±3.09
Nv	Error (%)	Coverage (%)
TBAL	AL+SC	TBAL	AL+SC
200	2.28 ±0.21	3.11 ±0.86	68.24 ±6.20	57.77 ±13.09
400	1.29 ±0.10	1.98 ±0.40	63.81 ±4.86	63.06 ±10.70
600	1.41 ±0.20	1.81 ±0.22	69.64 ±3.98	62.92 ±9.20
800	1.62 ±0.30	2.04 ±0.35	67.45 ±3.72	63.22 ±7.89
1000	1.64 ±0.23	1.97 ±0.26	70.28 ±2.82	66.11 ±8.00
Table 2: IMDB. Effect of variation of validation data size  $(N_v)$  with and without using a UCB on error estimates. We keep training data size  $N_q$  fixed at 500 and use error threshold  $\epsilon_{a} = 5\%$ . We report the mean and std. deviation over 10 runs with different random seeds. Left: with  $C_1 = 0$ . Right: with  $C_1 = 0.25$ .
Datasets. We use the following synthetic and real datasets. We also provide empirical results on MNIST and another synthetic dataset in the Appendix. For each dataset, we split the data into two sufficiently large pools. One is used as  $X_{pool}$  on which auto-labeling algorithms are run and the other is used as  $X_{val}$  from which the algorithms subsample validation data.
a) Unit-Ball is a synthetic dataset of uniformly sampled points from the  $d$ -dimensional unit ball. The true labels are generated using a homogeneous linear separator with  $\mathbf{w} = [1 / \sqrt{d}, \dots, 1 / \sqrt{d}]$ . We use  $d = 30$  and generate  $N = 20K$  samples, out of which  $16K$  are in  $X_{pool}$  and  $4K$  are in  $X_{val}$ . The dataset has just two classes but there is no margin between them.
b) Tiny-ImageNet [1] is a subset of the larger ImageNet [14] dataset, designed for image classification tasks. It consists of 200 classes, each with 500 training images and 50 validation and test images. With a total of  $100K$  images, Tiny ImageNet provides a diverse and challenging dataset. We use pre-computed embeddings of the images using CLIP [50].
c) IMDB Reviews [41] is a comprehensive collection of movie reviews, consisting of  $50K$  individual reviews. It is a balanced dataset of positive and negative labels. We use the standard train set of size  $25K$  and split it into  $X_{pool}$  and  $X_{val}$  of sizes  $20K$  and  $5K$  respectively. We compute embeddings of reviews using a pre-trained model bge-large-en [70] from the Massive Text Embedding Benchmark (MTEB) [44, 19].
d) CIFAR-10 [36] is an image dataset with 10 classes. We randomly split the standard training set into  $X_{pool}$  of size  $40K$  and the validation pool of size  $10K$ . We use the raw features for training.
Models and Training. For the linear models, we use SVM with the usual hinge loss and train it to loss tolerance  $10^{-5}$ . To train a multi-layer perceptron (MLP) on the pre-computed embeddings of IMDB and Tiny-ImageNet we use SGD with a learning rate of 0.05, 0.1 respectively, and batch size of 64. To train the medium CNN we use SGD with a learning rate of  $10^{-2}$ , batch size 256, and momentum of 0.9. More details on model training are in the Appendix.
The score function  $g$ . For SVMs we use the standard implementations of [69, 46] in sklearn to get the prediction probabilities and use them as the score function. Neural networks use softmax output.
4.1 Role of Validation Data
The TBAL algorithm uses validation data to estimate the auto-labeling errors at various thresholds to determine the threshold for automatically labeling points accurately. Thus, it is crucial to have
\( {\mathrm{N}}_{\mathrm{v}} \)	Error (%)	Coverage (%)
TBAL	AL+SC	TBAL	AL+SC
2000	\( {0.0} \pm {0.0} \)	\( {0.0} \pm {0.0} \)	\( {0.0} \pm {0.0} \)	\( {0.0} \pm {0.0} \)
4000	\( {13.88} \pm {5.42} \)	\( {13.31} \pm {10.79} \)	\( {0.72} \pm {0.55} \)	\( {0.48} \pm {0.04} \)
6000	\( {14.18} \pm {0.76} \)	\( {11.52} \pm {0.82} \)	\( {17.29} \pm {0.72} \)	\( {8.18} \pm {1.12} \)
8000	\( {13.97} \pm {0.14} \)	\( {11.31} \pm {0.51} \)	\( {36.36} \pm {1.78} \)	\( {23.40} \pm {1.15} \)
10000	\( {13.42} \pm {0.29} \)	\( {11.14} \pm {0.54} \)	\( {43.79} \pm {0.93} \)	\( {33.38} \pm {0.72} \)
Nv	Error (%)	Coverage (%)
TBAL	AL+SC	TBAL	AL+SC
2000	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0	0.0 ±0.0
4000	10.50 ±6.01	7.37 ±4.57	0.47 ±0.05	0.48 ±0.06
6000	10.61 ±0.62	7.71 ±1.03	10.16 ±1.10	4.31 ±1.10
8000	9.90 ±0.63	6.80 ±0.77	25.84 ±1.57	14.43 ±2.01
10000	8.97 ±0.36	6.87 ±0.48	32.19 ±1.34	21.96 ±1.35
Table 3: Tiny-ImageNet. Effect of variation of validation data size  $(N_v)$  with and without using a UCB on error estimates. We keep training data size  $N_q$  fixed at  $10k$  and use error threshold  $\epsilon_{a} = 10\%$ . We report the mean and std. deviation over 5 runs with different random seeds. Left: with  $C_1 = 0$ . Right: with  $C_1 = 0.25$ .
Figure 4: Results for varying  $N_{q}$ , the maximum number of samples algorithm can use for training while providing sufficient validation samples.
accurate estimates of the auto-labeling errors. Our analysis shows that to get such good estimates, large amounts of validation data are needed. In this section, we study the effect of varying the amount of validation data on auto-labeling performance.
Setup. We fix the maximum training data size  $N_{q}$  and run the algorithm with different amounts of validation data. We also consider the two cases where the algorithm uses an upper confidence bound on the error estimate and where it does not. We use the Unit-Ball, IMDB, Tiny-ImageNet, and datasets for this study with  $N_{q} = 500, 500$ , and  $10K$  respectively, and the auto-labeling error thresholds  $\epsilon_{a} = 1\%$ ,  $5\%$ ,  $10\%$ , respectively. Initial seed data of size  $n_{s}$  is  $20\%$  of  $N_{q}$  and query batch size  $n_{b}$  is  $5\%$  of  $N_{q}$ ;  $C = 2$  for both AL and TBAL for both datasets. We give the same initial seed samples of size  $n_{s}$  to all the methods to ensure they have the same starting point.
Results. Tables 1,2 and 3 demonstrate the impact of validation data on the performance of TBAL and other algorithms. The auto-labeling error and coverage of TBAL and other methods are affected by the amount of validation data provided. When the validation data is insufficient, the auto-labeling error of TBAL increases. However, as more validation data is used, the auto-labeling error and coverage of TBAL improves. Providing too little  $X_{val}$  can lead to incorrect estimates of the auto-labeling error, which in turn results in poor auto-labeling performance. This is further highlighted in our theoretical analysis as seen in Theorem 3.2. We also take a more nuanced look at the performance when the algorithm uses an upper confidence bound (with  $C_1 = 0.25$ ) on the estimates and when it does not. We see the effects of not using any upper confidence bound (i.e.  $C_1 = 0$ ). The left side tables in Tables 1,2 and 3 show the results when  $C_1 = 0$  and the right side tables show the results when  $C_1 = 0.25$ . These show that not using UCB leads to high auto-labeling error (i.e. not meeting the guarantees) even when there is a sufficient amount of validation data. This can happen with high coverage as well—yielding a dataset with large errors. On the other hand using UCB, i.e.  $C_1 = 0.25$ , the algorithm can keep the auto-labeling error below the given threshold but suffers in coverage.
In the above Tables, we omitted AL, PL, and  $\mathrm{PL + SC}$  for clarity. We provide full results with all baselines in Appendix D.3.
4.2 Role of Training Data Size
The labels queried for model training also play an important role in the process while incurring costs to obtain. The next experiment focuses on the impact of training data on auto-labeling.
\( {\mathrm{N}}_{\mathrm{q}} \)	Error (%)	Coverage (%)
TBAL	AL+SC	TBAL	AL+SC
200	\( {1.67} \pm {0.29} \)	\( {2.15} \pm {0.45} \)	\( {73.30} \pm {3.49} \)	\( {57.17} \pm {11.09} \)
400	\( {1.63} \pm {0.19} \)	\( {1.61} \pm {0.29} \)	\( {72.59} \pm {3.16} \)	\( {64.53} \pm {16.61} \)
600	\( {1.67} \pm {0.21} \)	\( {1.83} \pm {0.30} \)	\( {71.38} \pm {2.15} \)	\( {70.50} \pm {5.68} \)
800	\( {1.67} \pm {0.27} \)	\( {1.90} \pm {0.31} \)	\( {69.10} \pm {4.51} \)	\( {65.74} \pm {10.14} \)
1000	\( {1.62} \pm {0.22} \)	\( {1.97} \pm {0.35} \)	\( {73.42} \pm {2.84} \)	\( {68.05} \pm {5.56} \)
\( {\mathrm{N}}_{\mathrm{q}} \)	Error (%)	Coverage (%)
TBAL	AL+SC	TBAL	AL+SC
2000	\( {9.22} \pm {1.04} \)	\( {7.42} \pm {0.71} \)	\( {17.51} \pm {1.16} \)	\( {9.33} \pm {0.66} \)
4000	\( {9.30} \pm {0.38} \)	\( {6.97} \pm {0.39} \)	\( {25.01} \pm {1.20} \)	\( {14.25} \pm {1.71} \)
6000	\( {9.12} \pm {0.22} \)	\( {6.85} \pm {0.26} \)	\( {28.06} \pm {0.75} \)	\( {17.51} \pm {0.36} \)
8000	\( {9.21} \pm {0.14} \)	\( {7.38} \pm {0.53} \)	\( {30.88} \pm {0.64} \)	\( {21.18} \pm {0.90} \)
10000	\( {8.95} \pm {0.23} \)	\( {7.10} \pm {0.26} \)	\( {32.31} \pm {1.21} \)	\( {22.34} \pm {0.61} \)
Table 4: Results for varying  $N_{q}$ , the maximum number of samples algorithm can use for training. Left: IMDB with  $N_{v} = 1000, C_{1} = 0.25, \epsilon_{a} = 5\%$ . Right: Tiny-ImageNet with  $N_{v} = 10K, C_{1} = 0.25, \epsilon_{a} = 10\%$ . Mean and std. deviations are reported.
Setup. We limit the amount of training data the algorithm can use and record the resulting auto-labeling error and coverage. We ensure all algorithms have sufficiently large but equal amounts of validation data. We run on Unit-Ball, IMDB, Tiny-Imagenet, and CIFAR-10 datasets with the same values of  $n_s$ ,  $n_b$ , and  $C$  as in previous experiments.
Results. Figures 4(a), 4(b), and 2(b) indicate that TBAL and methods utilizing selective classification (AL+SC, PL+SC) maintain a high level of accuracy, even in scenarios where minimal training samples are used. This is expected as the threshold estimation method (when used with sufficient validation data) will find auto-labeling thresholds such that the auto-labeling error does not exceed  $\epsilon_{a}$ . The impact of training data size can be seen clearly in the coverage achieved by the algorithms. As expected, with fewer training samples the model has low accuracy leading to low coverage. However, as more samples are acquired, a more accurate model within the function class is learned, resulting in increased coverage. The Appendix D has additional discussion and results.
We briefly review related work, deferring a more detailed discussion to the Appendix A.
There is a rich body of work on active learning (AL) [57, 12, 30, 28, 8] focused on learning the best model in a function class with less labeled data than passive learning. Various AL algorithms have been developed and analyzed, e.g., uncertainty sampling [62, 45], disagreement region based [9, 27], margin based [4] and abstention based methods that minimize the Chow's excess risk [72].
Selective classification (SC) equips a given classifier with the option to abstain from prediction in order to guarantee prediction quality. The foundations for SC are laid down in [17, 67, 18, 68] where results on the error rate in the prediction region and the coverage of the given classifier are provided. However, these works lack practical algorithms to find the prediction region. A recent work [26] gives a disagreement-based active learning strategy to learn a selective classifier.
A recent paper [49] studies a TBAL-like algorithm for auto-labeling. It focuses on the cost of training incurred when these systems use large-scale model classes for auto-labeling. It proposes an algorithm to predict the training set size that minimizes the overall cost and provides an empirical evaluation.
Weak supervision is another line of work aimed at auto-labeling that does not rely on obtaining human labels but instead uses potentially noisy but cheaply available sources to infer labels [51, 22]. In contrast, we are focused specifically on analyzing the performance of TBAL algorithms [58].
In this work, we analyzed threshold-based auto-labeling systems and derived sample complexity bounds on the amount of human-labeled validation data required to guarantee the quality of machine-labeled data. Our study shows that these methods can accurately label a reasonable size of data using seemingly bad models when good confidence functions are available. Our analysis points to the hidden downside of these systems in terms of a large amount of validation data usage and calls for more sample-efficient methods including active testing.
This work was partly supported by funding from the American Family Data Science Institute. We thank Stephen Mussmann, Changho Shin, Albert Ge, Yi Chen, Kendall Park, and Nick Roberts for their valuable inputs. We thank the anonymous reviewers for their valuable comments and constructive feedback on our work.