SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Harit Vishwakarma
h Vishwakarma@cs.wisc.edu
fredsala@cs.wisc.edu
Heguang Lin
hglin@seas.upenn.edu
Ramya Korlakai Vinayak
ramya@ece.wisc.edu
Creating large-scale high-quality labeled datasets is a major bottleneck in supervised machine learning workflows. Threshold-based auto-labeling (TBAL), where validation data obtained from humans is used to find a confidence threshold above which the data is machine-labeled, reduces reliance on manual annotation. TBAL is emerging as a widely-used solution in practice. Given the long shelf-life and diverse usage of the resulting datasets, understanding when the data obtained by such auto-labeling systems can be relied on is crucial. This is the first work to analyze TBAL systems and derive sample complexity bounds on the amount of human-labeled validation data required for guaranteeing the quality of machine-labeled data. Our results provide two crucial insights. First, reasonable chunks of unlabeled data can be automatically and accurately labeled by seemingly bad models. Second, a hidden downside of TBAL systems is potentially prohibitive validation data usage. Together, these insights describe the promise and pitfalls of using such systems. We validate our theoretical guarantees with extensive experiments on synthetic and real datasets<sup>1</sup>.
Machine learning (ML) models with millions or even billions of parameters are used to obtain state-of-the-art performance in many applications, e.g., object identification [53], machine translation [63], and fraud detection [71]. Such large-scale models require training on large-scale labeled datasets. As an outcome, the typical supervised ML workflow begins with the construction of a large-scale high quality dataset. Datasets with up to millions of labeled data points have played a pivotal role in the advancement of computer vision. However, collecting labeled data is an expensive and time consuming process. A common approach is to rely on the services of crowdsourcing platforms such as Amazon Mechanical Turk (AMT) to get groundtruth labels.
Even with crowdsourcing, obtaining labels for the entire dataset is expensive. To reduce costs, data labeling systems that partially rely on using a model's predictions as labels have been developed. Such systems date back to teacher-less training [21]. Modern examples include Amazon Sagemaker Ground Truth [58] and others [61, 55, 2, 64]. These approaches can be broadly termed auto-labeling.
Auto-labeling systems aim to label unlabeled data using predictions from ML models that are often trained on small amounts of human labeled data which can produce incorrect labels. The shelf life of datasets is longer than those of models, e.g., ImageNet continues to be a benchmark for many computer vision tasks [14] fifteen years after its initial development. As a result, to reliably train new models on auto-labeled datasets and deploy them, we need a thorough understanding of how reliable the datasets output by these auto-labeling systems are. Unfortunately, many widely
1https://github.com/harit7/TBAL
Threshold Based Auto Labeling System
Figure 1: High-level workflow threshold-based auto-labeling (TBAL). Box (B) shows the key component estimating the auto-labeling region using validation data and auto-labeling points in it.
used commercial auto-labeling systems [58, 55] are largely opaque with limited public information on their functionality. It is therefore unclear whether the quality of the datasets obtained can be trusted. To address this, we study the high level workflow of a popular threshold-based auto-labeling (TBAL) system (see Figure 1). We emphasize that our goal is to understand such systems and their performance—not to promote them as a superior alternative to other approaches. Our goal is:
Goal. Develop a fundamental understanding of TBAL systems. This is crucial since there is a lack of theoretical understanding of the reliability of these systems despite their wide adoption.
The TBAL systems we study (Figure 1) work iteratively. At a high level, in each iteration, the system trains a model on currently available human labeled data and decides to label certain parts of unlabeled data using the trained model by finding high-accuracy regions using validation data. It then collects human labels on a small portion of unlabeled data that is deemed helpful for training the current model in the next iteration. The validation data is created by sampling i.i.d. points from the unlabeled pool and querying human labels for them. In addition to training data, the validation data is a major driver of the cost and accuracy of auto-labeling and will be a key component in our study.
Our Contributions. We study TBAL systems (Figure 1) and make the following contributions:
- Provide the first theoretical characterization of TBAL systems, developing tradeoffs between the quantity of manually labeled data and the quantity and quality of auto-labeled data (Section 3).
- Empirical results validating our theoretical understanding on real and synthetic data (Section 4).
Our results reveal two important insights. Promisingly, even poor quality models are capable of reliably labeling at least some data when we have access to sufficient validation data and a good confidence function that can accurately quantify the confidence of a given model on any data point. On the downside, in certain scenarios, the quantity of the validation data required to reach a certain quantity and quality of auto-labeled data can be high.
2 Threshold-Based Auto-Labeling Algorithm
We begin with the problem setup and describe the TBAL algorithm that is inspired by the commercial systems [58]. Then we provide experiments and theoretical analysis shedding light on the pros and cons of TBAL. We emphasize that TBAL is not our proposal and our goal is to understand the effectiveness of such an auto-labeling system.
Notation. Let the instance and label spaces be  $\mathcal{X}$  and  $\mathcal{Y} = \{1, \dots, k\}$ . We assume that there is some deterministic but unknown function  $f^* : \mathcal{X} \mapsto \mathcal{Y}$  that assigns true label  $y = f^*(\mathbf{x})$  to any  $\mathbf{x} \in \mathcal{X}$ . We also assume that there is a noiseless oracle  $\mathcal{O}$  that can provide the true label  $y \in \mathcal{Y}$  for any given  $\mathbf{x} \in \mathcal{X}$ . Let  $X_{pool} \subseteq \mathcal{X}$  denote a sufficiently large pool of unlabeled data to be labeled.
The goal of an auto-labeling algorithm is to produce accurate labels  $\tilde{y}_i\in \mathcal{V}$  for points  $\mathbf{x}_i\in X_{pool}$  while minimizing the number of queries to the oracle. Let  $[m]\coloneqq \{1,2,\ldots ,m\}$ ,  $A\subseteq [N]$  be the set of indices of auto-labeled points, and  $X_{pool}(A)$  be these points. The auto-labeling error  $\widehat{\mathcal{E}} (X_{pool}(A))$
and the coverage  $\widehat{\mathcal{P}}(X_{pool}(A))$  are defined as
$$
\widehat {\mathcal {E}} \left(X _ {\text {p o o l}} (A)\right) := \frac {1}{N _ {a}} \sum_ {i \in A} \mathbb {1} \left(\tilde {y} _ {i} \neq f ^ {*} (\mathbf {x} _ {i})\right) \text {a n d} \widehat {\mathcal {P}} \left(X _ {\text {p o o l}} (A)\right) := \frac {| A |}{N} = \frac {N _ {a}}{N}, \tag {1}
$$
where  $N_{a}$  denotes the size of auto-labeled set  $A$ . TBAL algorithm aims to auto-label the dataset so that  $\widehat{\mathcal{E}}(X_{pool}(A)) \leq \epsilon_{a}$  while maximizing coverage  $\widehat{\mathcal{P}}(X_{pool}(A))$  for any given  $\epsilon_{a} \in (0,1)$ .
Hypothesis Class and Confidence Function. A TBAL algorithm is given a fixed hypothesis space  $\mathcal{H}$  and a confidence function  $g:\mathcal{H}\times \mathcal{X}\mapsto T\subseteq$ $\mathbb{R}^+$  that quantifies the confidence of  $h\in$ $\mathcal{H}$  on any data point  $\mathbf{x}\in \mathcal{X}$ . Confidence functions include prediction probabilities and margin scores. For example, when  $\mathcal{H}$  is a set of unit-norm homogeneous linear classifiers, i.e.  $h_{\mathbf{w}}(\mathbf{x}) = \mathrm{sign}(\mathbf{w}^T\mathbf{x})$  with  $\mathbf{w}\in \{\mathbf{w}\in \mathbb{R}^d:||\mathbf{w}||_2 = 1\}$ , a reasonable confidence function is  $g(h_{\mathbf{w}},\mathbf{x}) = |\mathbf{w}^T\mathbf{x}|$ .
Note that the target  $f^{*}$  might not be in the hypothesis space  $\mathcal{H}$ . Our analysis (Section 3) shows that the TBAL algorithm can work well, i.e., accurately label a reasonable fraction of unlabeled data with simpler hypothesis classes that do not contain the target hypothesis  $f^{*}$ . We illustrate this with a simple example in Section 2.3 and Figure 2. Note as well that the features  $x$  could be raw features or representations from self-supervised techniques, pre-trained models, etc. We analyze TBAL in settings (i) with no assumptions on the features and (ii) when the features are linearly separable.
2.2 Description of the algorithm
The TBAL algorithm is given in Algorithm 1. It starts with an unlabeled pool  $X_{pool}$  and an auto-labeling error threshold  $\epsilon$ . For ease of exposition, the algorithm is given the labeled validation set  $D_{val}$  of size  $N_v$  separately. In practice, it is created by selecting points
at random from  $X_{pool}$ . The algorithm starts with an initial batch of  $n_s$  random data points and obtains oracle labels for these. The algorithm works in an iterative manner using the following steps.
Algorithm 1 Threshold-based Auto-Labeling (TBAL)
Input: Unlabeled pool  $X_{pool}$ , auto labeling error threshold  $\epsilon_{a}$ , seed data size  $n_{s}$ , batch size for active query  $n_{b}$ , labeled validation data pool  $D_{val}$ .
Output:  $D_{out} = \{(\mathbf{x}_i,\tilde{y}_i):\forall \mathbf{x}_i\in X_{pool}\}$
1:  $X_{u}^{(1)} = X_{pool}$ ;  $D_{val}^{(1)} = D_{val}$
2:  $D_{query}^{(1)} = \text{randomly_query_batch}(X_u^{(1)}, n_s)$
3: Remove queried points from  $X_{u}^{(1)}$
4:  $D_{train}^{(0)} = \phi$ ;  $i = 1$ ;  $D_{out} = D_{out}^{(1)} = D_{query}^{(1)}$
5: while  $X_{u}^{(i)} \neq \phi$  do
6:  $D_{train}^{(i)} = D_{train}^{(i - 1)}\cup D_{query}^{(i)}$
7:  $\hat{h}_i =$  empirical_risk_min(H,  $D_{train}^{(i)}$
8:  $\hat{t}_i = \text{Estimate Threshold}(X_u^{(i)}, D_{val}^{(i)}, \epsilon_a, \hat{h}_i, n_0)$
9:  $D_{auto}^{(i)} = \{(\mathbf{x},\hat{h}_i(\mathbf{x}):\mathbf{x}\in X_u^{(i)},g(\hat{h}_i,\mathbf{x})\geq \hat{t}_i\}$
10:  $X_{u}^{(i + 1)} = \{\mathbf{x}:\mathbf{x}\in X_{u}^{(i)},g(\hat{h}_{i},\mathbf{x}) <   \hat{t}_{i}\}$
11:  $D_{val}^{(i + 1)} = \{(\mathbf{x},y)\colon (\mathbf{x},y)\in D_{val}^{(i)},g(\hat{h}_i,\mathbf{x}) <   \hat{t}_i\}$
12:  $D_{query}^{(i + 1)} = \mathrm{active\_query\_batch}(\hat{h}_i,X_u^{(i + 1)},n_b)$
13: Remove queried points from  $X_{u}^{(i + 1)}$
14:  $D_{out} = D_{out}\cup D_{auto}^{(i)}\cup D_{query}^{(i + 1)}$
15:  $i = i + 1$
16: end while
Algorithm 2 Estimate Threshold
Input:  $X_{u}^{(i)},D_{val}^{(i)},\epsilon_{a},\hat{h}_{i},n_{0}$
Output: Threshold  $\hat{t}_i$
1:  $T_{i} = \{g(\hat{h}_{i},\mathbf{x}):\left(\mathbf{x},y\right)\in D_{val}^{(i)}\}$
2: for  $t \in T_i$  do
3:  $N_{t} = |\{\mathbf{x}\in X_{v}^{(i)}:g(\hat{h}_{i},\mathbf{x})\geq t\} |$
4: end for
5:  $\tilde{T}_i = \{t:t\in T_i,N_t\geq n_0\} \cup \{\infty \}$
6:  $\hat{t}_i = \min \{t\in \tilde{T}_i:\widehat{\mathcal{E}}_a(\hat{h}_i,t|X_v^{(i)}) + C_1\hat{\sigma}_i\leq \epsilon_a\}$
1. Oracle labeled data obtained in each iteration  $i$  is added to the training pool  $D_{train}^{(i)}$ . It is used to train a model  $\hat{h}_i$  by performing empirical risk minimization (ERM).
2. Find the region where  $\hat{h}_i$  can auto-label accurately. The algorithm estimates a threshold  $\hat{t}_i$  on the confidence score above which it can auto-label with the desired auto-labeling accuracy on the validation data (see Algorithm 2). Thresholds that have too little validation data are discarded, since their estimates are unreliable. The minimum threshold is found such that the sum of the estimated error  $\widehat{\mathcal{E}}_a(\hat{h}_i,t|X_v^{(i)})$  (see eq. (2)) and an upper confidence bound using the standard deviation of the estimated error, is at most the given auto-labeling error threshold.
3. Auto-label the points in the pool,  $X_{u}^{(i)}$ , which have confidence  $g(\hat{h}_i, \mathbf{x}) > \hat{t}_i$ . These are added to the set  $D_{out}$  and removed from the unlabeled pool. The validation points that fall in the auto-labeled region are also removed from the validation set so that in the next round the validation
(a) TBAL and AL+SC on Circles dataset
(b) Auto labeling performance of various methods
Figure 2: Comparison of TBAL, active learning (AL) followed by selective classification  $(\mathrm{AL} + \mathrm{SC})$  and passive learning (PL) followed by selective classification  $(\mathrm{PL} + \mathrm{SC})$  on the Circles dataset (Sec. 2.3) using linear classifiers and confidence functions. (a) Samples auto-labeled, queried, and left unlabeled. (b) The auto-labeling error and coverage achieved by the algorithms. (50 trials.)
set and the unlabeled pool are from the same region and the same distribution. Removing the auto-labeled points from  $X_{\mathrm{pool}}$  is a crucial step in the TBAL algorithm that enables it to focus only on the remaining unlabeled regions in the next iteration.
4. If there are points left in  $X_{\mathrm{pool}}$ , the algorithm selects points using some active querying strategy [57], obtains human labels for them, and adds them to the training pool. Note that the auto-labeled data is not added into the training set.
This process continues until there are no data points left to be labeled. The algorithm then outputs the labeled dataset, which is a mixture of human- and machine-labeled points.
2.3 Comparison between Auto-Labeling, Active Learning and Selective Classification
What is the difference between TBAL and methods such as active learning and selective classification?
Active learning. The goal of active learning [57] (AL) is to find the best model in hypothesis class  $\mathcal{H}$  by training with less labeled data compared to passive learning. This is usually achieved by obtaining labels for the most informative points. Note that the end goal is to output a model from the function class whose predictions on new data as good as the best model in the function class could.
Selective Classification. The goal of selective classification (SC) [17] is to find the best combination of the hypothesis and selection functions to minimize error and maximize coverage of selection regions.
Auto-Labeling. The output of an auto-labeling procedure is a labeled dataset (not a model). When the hypothesis class is of lower complexity, it is often not possible to find a good classifier. The goal of an auto-labeling system is to label as much of the unlabeled data as accurately as possible with a given function class and with limited labeled data from humans.
Is active learning alone enough to auto-label data? AL has been found to be effective in reducing the number of labels needed to learn versus passive learning, particularly in low-noise cases [28]. Doing auto-labeling using AL followed by SC may be effective in such settings. However, in real-world scenarios, noise levels may be higher and the hypothesis class could be misspecified. In these instances, using the model learned through active learning to automatically label all data may result in a high number of errors.
We illustrate this difference between AL, SC, and auto-labeling through an example. Suppose the data consists of two concentric circles, one for each class, with the same number of points per class (Figure 2(a)). This setting is not linearly separable. We run TBAL, AL, and AL followed by SC with an error tolerance of  $\epsilon_{a} = 1\%$  and linear classifiers and confidence functions. The results are shown in Figure 2. Note that the multiple optimal linear classifiers will all incur an error of  $50\%$ . AL algorithms can only output models that make at least  $50\%$  error. If we naively use the output model for auto-labeling, we can obtain near full coverage but incur around  $50\%$  auto-labeling error. If we use the model output by AL with threshold-based SC, labeling error is reduced. However, it can only label  $\approx 25\%$  of the unlabeled data. In contrast, TBAL can label almost all of the data accurately (close to  $100\%$  coverage) with less than  $1\%$  auto-labeling error.
The performance of the TBAL (Algorithm 1) depends on many factors including the hypothesis class, the accuracy of the confidence function, the data sampling strategy, and the size of the training and validation data. In particular, the amount of validation data plays a critical role in determining the accuracy of the confidence function, which in turn affects the accuracy and coverage.
We derive bounds on the auto-labeling error and the coverage for Algorithm 1 in terms of the size of the validation data, the number of auto-labeled points  $N_{a}^{(k)}$ , and the Rademacher complexity of the extended hypothesis class  $\mathcal{H}^{T,g}$  induced by the confidence function  $g$ . Our first result, Theorem (3.2), applies to general settings and makes no assumptions on the particular form of the hypothesis class, the data distribution, and the confidence function. We then instantiate and specialize the results for a specific setting in Section 3.1. We introduce some notation to aid in stating our results,
Definition 3.1. (Hypothesis Class with Abstain) The function  $g$ , set  $T$  and  $\mathcal{H}$  induce an extended hypothesis class  $\mathcal{H}^{T,g} \coloneqq \mathcal{H} \times T$ . Any function  $(h,t) \in \mathcal{H}^{T,g}$  is defined as  $(h,t)(\mathbf{x}) = h(\mathbf{x})$  if  $g(h,\mathbf{x}) \geq t$  and  $\bot$  otherwise. Here  $(h,t)(\mathbf{x}) = \bot$  means  $(h,t)$  abstains in classifying the point  $\mathbf{x}$ .
Error Definitions. Let  $S \subseteq \mathcal{X}$  denote a non-empty sub-region of  $\mathcal{X}$  and  $S \subseteq S$  be a finite set of i.i.d. samples from  $S$ . The subset  $S(h, t) \subseteq S$  denotes the regions where  $(h, t)$  does not abstain i.e.  $S(h, t) := \{\mathbf{x} \in S : (h, t)(\mathbf{x}) \neq \bot\}$ , and the conditional probability mass associated with it is  $\mathbb{P}(h, t | S) := \mathbb{P}(S(h, t) | S)$ , and its empirical counterpart  $\widehat{\mathbb{P}}(h, t | S) := |S(h, t)| / |S|$ . We use  $\mathbb{P}(S)$  to denote the probability mass of set  $S$  and  $\mathbb{P}(S' | S)$  for the conditional probability of subset  $S' \subseteq S$  given  $S$ . The population level and empirical auto-labeling errors are defined as follows:
$$
\mathcal {E} _ {a} (h, t | \mathcal {S}) := \mathbb {E} _ {\mathbf {x} | \mathcal {S}} [ \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x}, y) ] / \mathbb {P} (h, t | \mathcal {S}),
$$
$$
\widehat {\mathcal {E}} _ {a} (h, t \mid S) := \left(\sum_ {\mathbf {x} _ {i} \in S (h, t)} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i})\right) / | S (h, t) | \tag {2}
$$
Here  $\ell_{0 - 1}^{\perp}(h,t,\mathbf{x},y)\coloneqq \ell_{0 - 1}(h,\mathbf{x},y)\cdot \ell_{\perp}(h,t,\mathbf{x})$  with  $\ell_{0 - 1}(h,\mathbf{x},y)\coloneqq \mathbb{1}(h(\mathbf{x})\neq y)$ , and  $\ell_{\perp}(h,t,\mathbf{x})\coloneqq \mathbb{1}(g(h,\mathbf{x})\geq t)$ .
Rademacher Complexity. The Rademacher complexities for the function classes induced by the  $\mathcal{H}, T, g$  and the loss functions are defined as  $\Re_n(\mathcal{H}^{T,g}) := \Re_n(\mathcal{H}, \ell_{0-1}) + \Re_n(\mathcal{H}^{T,g}, \ell_{\perp})$ . Let  $\hat{h}_i$  and  $\hat{t}_i$  be the ERM solution and the auto-labeling threshold at epoch  $i$ . Let  $p_0 \in (0,1)$  be a constant such that  $\mathbb{P}(\hat{h}_i, \hat{t}_i | \mathcal{X}^{(i)}) \geq p_0$  for all  $i$ . Let  $X_v^{(i)}$  denote the validation set, and  $n_v^{(i)}$  and  $n_a^{(i)}$  the number of validation and auto-labeled points at epoch  $i$ . Let  $\widehat{\mathcal{E}}_a(\hat{h}_i, \hat{t}_i | X_v^{(i)})$  be the empirical conditional risk of  $\hat{h}_i$  in the region where  $g(\hat{h}_i, \mathbf{x}) \geq \hat{t}_i$  evaluated on the validation data  $X_v^{(i)}$ .
We provide the following guarantees on the auto-labeling error and the coverage achieved by TBAL.
Theorem 3.2. (Overall Auto-Labeling Error and Coverage) Let  $k$  denote the number of rounds of the TBAL Algorithm 1. Let  $n_v^{(i)}, n_a^{(i)}$  denote the number of validation and auto-labeled points at epoch  $i$  and  $n^{(i)} = |X^{(i)}|$ . Let  $X_{pool}(A_k)$  be the set of auto-labeled points at the end of round  $k$ .  $N_a^{(k)} = \sum_{i=1}^k n_a^{(i)}$  be the total number of auto-labeled points. Then, for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \widehat {\mathcal {E}} \big (X _ {p o o l} (A _ {k}) \big) \leq \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \Big (\underbrace {\widehat {\mathcal {E}} _ {a} \big (\hat {h} _ {i} , \hat {t} _ {i} | X _ {v} ^ {(i)} \big)} _ {(a)} + \underbrace {\frac {4}{p _ {0}} \big (\Re_ {n _ {v} ^ {(i)}} \big (\mathcal {H} ^ {T , g} \big) + \frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {v} ^ {(i)}} \log (\frac {8 k}{\delta})} \big)} _ {(b)} \Big) \\ + \underbrace {\frac {4}{p _ {0}} \left(\sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \Re_ {n _ {a} ^ {(i)}} \left(\mathcal {H} ^ {T , g}\right) + \sqrt {\frac {k}{N _ {a} ^ {(k)}} \log \left(\frac {8 k}{\delta}\right)}\right)} _ {(c)}, \quad a n d \\ \end{array}
$$
$$
\widehat {\mathcal {P}} \big (X _ {p o o l} (A _ {k}) \big) \geq \sum_ {i = 1} ^ {k} \mathbb {P} \big (\mathcal {X} ^ {(i)} (\hat {h} _ {i}, \hat {t} _ {i}) \big) - 2 \Re_ {n ^ {(i)}} \big (\mathcal {H} ^ {T, g} \big) - \sqrt {\frac {2 k ^ {2}}{N} \log \big (\frac {8 k}{\delta} \big)}.
$$
Discussion. We interpret this result, starting with the auto-labeling error term  $\widehat{\mathcal{E}}(X_{pool}(A_k))$ . The term (a)  $\widehat{\mathcal{E}}_a(\hat{h}_i, \hat{t}_i | X_v^{(i)})$  is the empirical conditional error in the auto-labeled region computed on the validation data in  $i$ -th round, which is at most  $\epsilon_a$ . Thus, summing term (a) over all the rounds is at most  $\epsilon_a$ . The term (b) provides an upper bound on the excess error over the empirical estimate term
(a) as a function of the Rademacher complexity of  $\mathcal{H}^{T,g}$  and the validation data used in each round. The last term (c) captures the variance in the overall estimate as a function of the total number of auto-labeled points and the Rademacher complexity of  $\mathcal{H}^{T,g}$ . If we let  $n_v^{(i)}\geq n_v$  i.e. the minimum validation points ensured in each round, then we can see the second term is  $\mathcal{O}(\Re_{n_v}(\mathcal{H}^{T,g}))$  and the third term is  $\mathcal{O}(\sqrt{1 / n_v})$ .
Therefore, validation data of size  $\mathcal{O}\left(1 / \epsilon_a^2\right)$  in each round is sufficient to get a  $\mathcal{O}(\epsilon_{a})$  bound on the excess auto-labeling error. The terms with Rademacher complexities suggest that it is better to use a hypothesis class and confidence function such that the induced hypothesis class has low Rademacher complexity. While such a hypothesis class might not be rich enough to include the target function, it would still be helpful for efficient and accurate auto-labeling of the dataset which can then be used for training richer models in the downstream task. The coverage term provides a lower bound on the empirical coverage  $\widehat{\mathcal{P}} (X_{pool}(A_k))$  in terms of the true coverage of the sequence of estimated hypotheses  $\hat{h}_i$  and threshold  $\hat{t}_i$ .
We note that the size of the validation data needed to guarantee the auto-labeling error in each round by Algorithm 1 is optimal up to log factors. This follows by applying a result on the tail probability of the sum of independent random variables due to Feller [20]: