SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \widehat {\mathcal {E}} (X _ {p o o l} (A _ {k})) \leq \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \left(\underbrace {\widehat {\mathcal {E}} _ {a} (\hat {h} _ {i} , \hat {t} _ {i} | X _ {v} ^ {(i)})} _ {(a)} + \frac {4}{p _ {0}} \left(\underbrace {\Re_ {n _ {v} ^ {(i)}} \left(\mathcal {H} ^ {T , g}\right) + \Re_ {n _ {a} ^ {(i)}} \left(\mathcal {H} ^ {T , g}\right)} _ {(b)}\right) \right. \\ \left. + \frac {4}{p _ {0}} \underbrace {\sqrt {\frac {1}{n _ {v} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)}} _ {(c)}\right) + \underbrace {\frac {4}{p _ {0}} \sqrt {\frac {k}{N _ {a} ^ {(k)}} \log \left(\frac {8 k}{\delta}\right)}} _ {(d)} \\ \end{array}
$$
We first simplify the terms dependent on  $n_v^{(i)}$  as follows. Here we use the inequality  $\sqrt{a} + \sqrt{b} \leq \sqrt{2(a + b)}$  for any  $a, b \in \mathbb{R}^+$ .
$$
\begin{array}{l} \Re_ {n _ {v} ^ {(i)}} \left(\mathcal {H} ^ {T, g}\right) + \sqrt {\frac {1}{n _ {v} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)} \leq \sqrt {\frac {2 d}{n _ {v} ^ {(i)}} \log \left(\frac {e n _ {v} ^ {(i)}}{d}\right)} + \sqrt {\frac {1}{n _ {v} ^ {(i)}} \log \left(\frac {4 k}{\delta}\right)}, \\ \leq \sqrt {\frac {2}{n _ {v} ^ {(i)}} \left(2 d \log \left(\frac {e n _ {v} ^ {(i)}}{d}\right) + \log \left(\frac {8 k}{\delta}\right)\right)}. \\ \end{array}
$$
Next, we simplify the terms dependent on  $n_a^{(i)}$  as follows. First, we substitute the Rademacher complexity using the bound in Lemma C.6 and then apply the same steps as in the proof of Theorem 3.2 to bound  $\sum_{i=1}^{k} \sqrt{n_a^{(i)}} / N_a^{(k)}$  by  $\sqrt{k / N_a^{(k)}}$  followed by the application of  $\sqrt{a} + \sqrt{b} \leq \sqrt{2(a + b)}$  to get the final term.
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \Re_ {n _ {a} ^ {(i)}} \left(\mathcal {H} ^ {T, g}\right) + \sqrt {\frac {k}{N _ {a} ^ {(k)}} \log \left(\frac {8 k}{\delta}\right)} \leq \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \sqrt {\frac {2 d}{n _ {a} ^ {(i)}} \log \left(\frac {e n _ {a} ^ {(i)}}{d}\right)} + \sqrt {\frac {k}{N _ {a} ^ {(k)}} \log \left(\frac {8 k}{\delta}\right)} \\ = \sum_ {i = 1} ^ {k} \frac {\sqrt {n _ {a} ^ {(i)}}}{N _ {a} ^ {(k)}} \sqrt {2 d \log \left(\frac {e n _ {a} ^ {(i)}}{d}\right)} + \sqrt {\frac {k}{N _ {a} ^ {(k)}} \log \left(\frac {8 k}{\delta}\right)} \\ \leq \sum_ {i = 1} ^ {k} \frac {\sqrt {n _ {a} ^ {(i)}}}{N _ {a} ^ {(k)}} \sqrt {2 d \log \left(\frac {e N _ {a} ^ {(k)}}{d}\right)} + \sqrt {\frac {k}{N _ {a} ^ {(k)}} \log \left(\frac {8 k}{\delta}\right)} \\ \leq \sqrt {\frac {2 d k}{N _ {a} ^ {(k)}} \log \left(\frac {e N _ {a} ^ {(k)}}{d}\right)} + \sqrt {\frac {k}{N _ {a} ^ {(k)}} \log \left(\frac {8 k}{\delta}\right)} \\ \leq \sqrt {\frac {2 k}{N _ {a} ^ {(k)}} \left(2 d \log \left(\frac {e N _ {a} ^ {(k)}}{d}\right) + \log \left(\frac {8 k}{\delta}\right)\right)}. \\ \end{array}
$$
C.3 Homogeneous Linear Classifiers with Uniform Distribution
Here we instantiate Theorem 3.2 for the case of homogeneous linear separators under the uniform distribution in the realizable setting. Formally, let  $P_{\mathbf{x}}$  be a uniform distribution supported on the unit ball in  $\mathbb{R}^d$ ,  $\mathcal{X} = \{\mathbf{x} \in \mathbb{R}^d : ||\mathbf{x}|| \leq 1\}$ . Let  $\mathcal{W} = \{\mathbf{w} \in \mathbb{R}^d : ||\mathbf{w}||_2 = 1\} = \mathbb{S}_d$  and  $\mathcal{H} = \{\mathbf{x} \mapsto \mathrm{sign}(\langle \mathbf{w}, \mathbf{x} \rangle) \forall \mathbf{w} \in \mathcal{W}\}$ , the score function is given by  $g(h, \mathbf{x}) = g(\mathbf{w}, \mathbf{x}) = |\langle \mathbf{w}, \mathbf{x} \rangle|$  and set  $T = [0, 1]$ . For simplicity, we will use  $\mathcal{W}$  in place of  $\mathcal{H}$ .
Corollary 3.4. (Overall Auto-Labeling Error and Coverage) Let  $\hat{\mathbf{w}}_i, \hat{t}_i$  be the ERM solution and the auto-labeling margin threshold respectively at epoch  $i$ . Let  $n_v^{(i)}, n_a^{(i)}$  denote the number of validation and auto-labeled points at epoch  $i$ . Let the auto-labeling algorithm run for  $k$ -epochs. Then, for any
$\delta \in (0,1)$ , w.p. at least  $1 - \delta /2$
$$
\begin{array}{l} \widehat {\mathcal {E}} (X _ {p o o l} (A _ {k})) \leq \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \left(\underbrace {\widehat {\mathcal {E}} _ {a} (\hat {\mathbf {w}} _ {i} , \hat {t} _ {i} | X _ {v} ^ {(i)})} _ {(a)} + \underbrace {\frac {4}{p _ {0}} \sqrt {\frac {2}{n _ {v} ^ {(i)}} \left(2 d \log \left(\frac {e n _ {v} ^ {(i)}}{d}\right) + \log \left(\frac {8 k}{\delta}\right)\right)}} _ {(b)}\right) \\ + \underbrace {\frac {4}{p _ {0}} \left(\sqrt {\frac {2 k}{N _ {a} ^ {(k)}} \left(2 d \log \left(\frac {e N _ {a} ^ {(k)}}{d}\right) + \log \left(\frac {8 k}{\delta}\right)\right)}\right)} _ {c} \\ \end{array}
$$
and w.p. at least  $1 - \delta / 2$
$$
\widehat {\mathcal {P}} (X _ {p o o l} (A _ {k})) \geq 1 - \min _ {i} \widehat {t} _ {i} \sqrt {4 d / \pi} - 2 k \sqrt {\frac {2}{N} \left(2 d \log \left(\frac {e N}{d}\right) + \log \left(\frac {8 k}{\delta}\right)\right)}.
$$
Proof. The bound on auto-labeling error follows directly from Theorem C.7 as the VC dimension for this setting is  $d$ . For the coverage bound, we utilize the fact that the distribution  $P_{\mathbf{x}}$  is the uniform distribution over the unit ball. This enables us to obtain explicit lower bounds on the coverage. The details are given in Lemma C.8 and Lemma C.9.
Lemma C.8. Let the auto-labeling algorithm run for  $k$ -epochs and let  $\hat{\mathbf{w}}_i, \hat{t}_i$  be the ERM solution and the auto-labeling margin threshold respectively at epoch  $i$ . Let  $\mathcal{X}^{(i)}$  be the unlabeled region at the beginning of epoch  $i$ , then we have,
$$
\sum_ {i = 1} ^ {k} \mathbb {P} \left(\mathcal {X} ^ {(i)} \left(\hat {\mathbf {w}} _ {i}, \hat {t} _ {i}\right)\right) \geq 1 - \min  _ {i} \hat {t} _ {i} \sqrt {4 d / \pi}. \tag {15}
$$
Proof. Let  $\mathcal{X}(\hat{\mathbf{w}}_i, t_i) = \{\mathbf{x} \in \mathcal{X} : |\langle \hat{\mathbf{w}}_i, \mathbf{x} \rangle| \geq \hat{t}_i\}$  denote the region that can be auto-labeled by  $\hat{\mathbf{w}}_i, \hat{t}_i$ . However, since in each round the remaining region is  $\mathcal{X}^{(i)}$  the actual auto-labeled region of epoch  $i$  is  $\mathcal{X}_a^{(i)} = \{\mathbf{x} \in \mathcal{X}^{(i)} : |\langle \hat{\mathbf{w}}_i, \mathbf{x} \rangle| \geq \hat{t}_i\}$ . Let  $\bar{\mathcal{X}}(\hat{\mathbf{w}}_i, t_i)$  denote the complement of set  $\mathcal{X}(\hat{\mathbf{w}}_i, t_i)$ .
Now observe that  $\mathcal{X}_a = \cup_{i=1}^k \mathcal{X}_a^{(i)}$  and  $\mathcal{X}(\hat{\mathbf{w}}_k, \hat{t}_k) \subseteq \mathcal{X}_a$  because any  $\mathbf{x} \in \mathcal{X}(\hat{\mathbf{w}}_k, \hat{t}_k)$  is either auto-labeled in previous rounds  $i < k$  or if not then it will be auto-labeled in the  $k^{th}$  round. More specifically, any  $\mathbf{x} \in \mathcal{X}(\hat{\mathbf{w}}_k, \hat{t}_k)$  is either in  $\cup_{i=1}^{k-1} \mathcal{X}_a^{(i)}$  and if not then it must be in  $\mathcal{X}_a^{(k)}$ . Thus the sum of probabilities,
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \mathbb {P} \left(\mathcal {X} ^ {(i)} (\hat {\mathbf {w}} _ {i}, \hat {t} _ {i})\right) = \sum_ {i = 1} ^ {k} \mathbb {P} \left(\mathcal {X} _ {a} ^ {(i)}\right) \\ = \mathbb {P} \left(\mathcal {X} _ {a}\right) \\ \geq \min  _ {i} \mathbb {P} \left(\mathcal {X} \left(\hat {\mathbf {w}} _ {i}, \hat {t} _ {i}\right)\right) \\ = 1 - \max  _ {i} \mathbb {P} \big (\bar {\mathcal {X}} (\hat {\mathbf {w}} _ {i}, \hat {t} _ {i}) \big) \\ \geq 1 - \min  _ {i} \hat {t} _ {i} \sqrt {4 d / \pi} \\ \end{array}
$$
The last step used Lemma 4 from [3]) with  $\gamma_{1} = \hat{t}_{i}$  and  $\gamma_{2} = 0$  to upper bound  $\mathbb{P}(\tilde{\mathcal{X}} (\hat{\mathbf{w}}_i,\hat{t}_i))$  by  $\hat{t}_i\sqrt{4d / \pi}$ . The lemma is stated as follows in Lemma C.9,
Lemma C.9. ([3] (Lemma 4)) Let  $d \geq 2$  and let  $\mathbf{x} = [x_1, \ldots, x_d]$  be uniformly distributed in the  $d$ -dimensional unit ball. Given  $\gamma_1 \in [0, 1]$ ,  $\gamma_2 \in [0, 1]$ , we have:
$$
\mathbb {P} \big ((x _ {1}, x _ {2}) \in [ 0, \gamma_ {1} ] \times [ \gamma_ {2}, 1 ] \big) \leq \frac {\gamma_ {1} \sqrt {d}}{2 \sqrt {\pi}} \exp \left(- \frac {(d - 2) \gamma_ {2} ^ {2}}{2}\right)
$$
C.4 Lower Bound
Lemma 3.3. Let  $c_{1}, c_{2}$  and  $\sigma > 0$ . Let  $\mathbf{x}_i \in X$  be a set of  $n$  i.i.d. points from  $\mathcal{X}$  with corresponding true labels  $y_i$ . Given  $(h, t) \in \mathcal{H}^{T,g}$ , let  $\mathbb{E}\left[\left(\ell_{0-1}^{\perp}(h, t, \mathbf{x}_i, y_i) - \mathcal{E}(h, t|\mathcal{X})\right)^2\right] = \sigma_i^2 > \sigma^2$  for every  $\mathbf{x}_i$  for  $\sigma_i > 0$  and let  $\sum_{i}^{n} \sigma_i^2 \geq c_1$  then for every  $\epsilon \in [0, \frac{\sum_{i=1}^{n} \sigma_i^2}{\sqrt{c_1}}]$  with  $n_v < \frac{12\sigma^2}{\epsilon^2} \log(4c_2)$  the following holds w.p. at least  $1/4$ ,  $\mathcal{E}_a(h, t|\mathcal{X}) > \widehat{\mathcal{E}}_a(h, t|X) + \epsilon$ .
Proof. It follows by application of Feller's result stated in lemma C.10.  $\square$
Lemma C.10. (Feller, Lower Bound on Tail Probability of Sum of Independent Random Variables) There exists positive universal constants  $c_1$  and  $c_2$  such that for any set of independent random variables  $X_1, \ldots, X_m$  satisfying  $E[X_i] = 0$  and  $|X_i| \leq M$ , for every  $i \in \{1, \ldots, m\}$ , if  $\sum_{i=1}^{m} \mathbb{E}[(X)_i^2] \geq c_1$ , then for every  $\epsilon \in [0, \frac{\sum_{i=1}^{m} \mathbb{E}[(X_i)^2]}{M\sqrt{c_1}}]$
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {m} X _ {i} > \epsilon\right) \geq c _ {2} \exp \left(\frac {- \epsilon^ {2}}{1 2 \sum_ {i = 1} ^ {m} \mathbb {E} [ (X _ {i}) ^ {2} ]}\right). \tag {16}
$$
In this section, we discuss additional experiments on the role of hypothesis class in auto-labeling datasets and experiments for studying the role of confidence function in auto-labeling. Finally, we visualize PaCMAP embeddings of the CIFAR-10 and MNIST data points to get a sense of auto-labeling regions in various rounds of the algorithm.
D.1 Additional Experiments on Role of the Hypothesis Class
First, we provide details of the datasets,
XOR is a synthetic dataset. Recall that it is created by uniformly drawing points from 4 circles, each centered at the corners of a square of with side length 4 centered at the origin. Points in the diagonally opposite balls belong to the same class. We generate a total of  $N = 10,000$  samples, out of which we keep 8,000 in  $X_{pool}$  and 2,000 in the validation pool  $X_{val}$ .
MNIST [15] is a standard image dataset of hand-written digits. We randomly split the standard training set into  $X_{pool}$  and the validation pool  $X_{val}$  of sizes 48,000 and 12,000 respectively. While training a linear classifier on this dataset we flatten the  $28 \times 28$  images to vectors of size 784.
(a) Output of TBAL and AL+SC on XOR dataset
(b) Auto labeling performance of various methods
Figure 5: Comparison of Threshold-Based Auto-Labeling (TBAL) and Active-Learning followed by Selective Classification (AL+SC) on XOR-dataset. Left figure (a) shows samples that were auto-labeled, queried, and left unlabeled by these methods. Right figure (b) shows the auto-labeling error and coverage achieved. The lines show the mean and the shaded region shows 1-standard deviation estimated over 10 trials with different random seeds.
XOR Experiment. We run the TBAL algorithm 1 with an error tolerance of  $\epsilon_{a} = 1\%$ . We use  $20\%$  of  $N_{q}$  as seed training data and keep query size  $n_b$  as  $5\%$  of  $N_{q}$ . We compare it with active learning and active learning followed by selective classification. The given function class and selective classifier are both linear for all the algorithms. The results are shown in Figure 5. Clearly, there is no linear classifier that can correctly classify this data. We note that there are multiple optimal classifiers in the function class of linear classifiers and they will all incur an error of  $25\%$ . So, active learning algorithms can only output models that make at least  $25\%$  error. If we naively use the output model for auto-labeling, we can obtain near full coverage but incur  $25\%$  auto-labeling error. If we use the model output by active learning with threshold-based selective classification, then it can attain lower error in labeling. However, it can only label  $\approx 25\%$  of the unlabeled data. In contrast, the TBAL algorithm can label almost all of the data accurately, i.e., attain close to  $100\%$  coverage, with an error close to  $1\%$  auto-labeling error.
MNIST Experiment. For training LeNet [37] we use SGD with a learning rate of 0.1, batch size of 32, and train for 20 epochs. We use auto-labeling error threshold  $\epsilon_{a} = 5\%$ . We use  $20\%$  of  $N_{q}$  as seed training data and keep query size  $n_{b}$  as  $5\%$  of  $N_{q}$ . The results are presented in Figure 6 we observe that TBAL using less powerful models can still yield highly accurate datasets with a significant fraction of points labeled by the models. This confirms the notion that bad models can still provide good datasets.
D.2 Role of Confidence Function
The confidence function  $g$  is used to obtain uncertainty scores is an important factor in auto-labeling. In particular, for threshold-based auto-labeling we expect the scores of correctly classified and incorrectly classified points to be reasonably well separated and if this is not the case then the
(a) Auto-labeling MNIST data using a linear classifier. The validation size used here is  $12\mathrm{k}$
(b) Auto-labeling MNIST data using LeNet classifier. The validation size used here is  $12k$
Figure 6: Auto-labeling performance on MNIST data using different models (hypothesis classes) as a function of samples available for training. The left figure (a) shows the results with the linear classifier and the right figure (b) shows the results with the LeNet classifier. The auto-labeling error threshold  $\epsilon_{a} = 5\%$  in both experiments and the algorithms are given the same amount of validation data. The lines show the mean and the shaded region shows 1-standard deviation estimated over 5 trials with different random seeds.
(a) Auto-labeling CIFAR-10 data using a small network and softmax scores. Validation size  $= 10\mathrm{k}$
(b) Auto-labeling CIFAR-10 data using a small network and energy scores. Validation size  $= 10k$
Figure 7: Auto-labeling performance on CIFAR-10 data using a small network and different scoring functions. The left figure (a) shows the results with softmax scores and the right figure (b) shows the results with the energy score. The auto-labeling error threshold  $\epsilon_{a} = 10\%$  in both experiments and methods are given the same amount of validation data. The lines show the mean and the shaded region shows 1-standard deviation estimated over 5 trials with different random seeds.
algorithm will struggle to find a good threshold even if the given classifier has good accuracy in certain regions.
Setup. We perform auto-labeling on the CIFAR-10 dataset using a small CNN network with 2 convolution layers followed by 3 fully connected layers [48]. We use two different scores for auto-labeling, a) Usual softmax output b) Energy score with temperature  $= 1$  [38]. We vary the maximum number of training samples  $N_{q}$  and keep  $20\%$  of  $N_{q}$  as seed samples and query points in the batches of  $10\%$  of  $N_{q}$ . The model is trained for 50 epochs, using SGD with a learning rate of 0.05, batch size  $= 256$ , weight decay  $= 5e^{-4}$ , and momentum  $= 0.9$ , and use  $\epsilon_{a} = 10\%$ .
Results. The results with softmax scores and energy scores used as confidence functions can be seen in Figures 8(a) and 8(b) respectively. We see that for both of these cases, TBAL does not obtain a coverage of more than  $\approx 6\%$ . We observe that using the energy score as the confidence function performed marginally better than using the softmax scores. We note that this is the case even though the test accuracies of the trained models were around  $50\%$  for most of the rounds. Note that CIFAR-10 has 10 classes, so an accuracy of  $50\%$  is much better than random guessing and one would expect to be able to auto-label a significant chunk of the data with such a model. However, the softmax scores and energy scores are not well calibrated, and therefore, when used as confidence functions, they result in a poor separation between correct and incorrect predictions by the model. This can be seen in Figure 8 where neither of the softmax and energy scores provides a good separation between the correct and incorrect predictions. We can also see that the energy score is marginally better in terms of the separation, which allows it to achieve slightly better auto-labeling coverage in comparison to using softmax scores. This suggests that more investigation is needed to understand the properties of good confidence functions for auto-labeling which is left to future work.
(a) Histogram of scores in round 2.
Figure 8: Histograms of scores computed on the validation data in a few rounds of TBAL run on CIFAR-10 with a small net. We picked two rounds where it auto-labeled the most i.e. around 800 points.
(b) Histogram of scores in round 6.
For a more detailed visualization of the rounds of TBAL for this experiment, see Figures 10 and 11 in the Appendix.
D.3 Detailed Results
In the main paper, we omitted AL, PL, and  $\mathrm{PL + SC}$  for clarity and due to lack of space. We provide results including these baselines in the following tables for IMDB, Tiny-ImageNet, and Unit-Ball datasets.
\( N_v \)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
200	4.77 ±0.18	3.35 ±0.80	3.68 ±0.38	6.60 ±1.06	6.16 ±0.10	83.14 ±3.65	78.53 ±7.05	81.20 ±1.78	98.75 ±0.00	97.50 ±0.00
400	4.57 ±0.26	3.53 ±0.73	3.96 ±0.33	6.60 ±1.06	6.16 ±0.10	90.70 ±3.11	86.39 ±5.11	87.43 ±2.01	98.75 ±0.00	97.50 ±0.00
600	4.32 ±0.17	3.70 ±0.63	4.17 ±0.27	6.60 ±1.06	6.16 ±0.10	92.96 ±0.46	88.90 ±4.83	90.12 ±1.62	98.75 ±0.00	97.50 ±0.00
800	4.66 ±0.20	3.84 ±0.70	4.15 ±0.35	6.60 ±1.06	6.16 ±0.10	92.42 ±0.89	88.67 ±3.88	89.37 ±1.45	98.75 ±0.00	97.50 ±0.00
1000	4.67 ±0.16	3.90 ±0.68	4.21 ±0.35	6.60 ±1.06	6.16 ±0.10	92.89 ±0.91	89.79 ±3.09	90.18 ±1.39	98.75 ±0.00	97.50 ±0.00
Table 6: IMDB. Effect of variation of validation data size  $(N_v)$  without using a UCB (i.e.,  $C_1 = 0$ ) on error estimates. We keep training data size  $N_q$  fixed at 500 and use error threshold  $\epsilon_a = 5\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
\(N_v\)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
200	2.28 ±0.21	3.11 ±0.86	2.86 ±0.35	6.60 ±1.06	6.16 ±0.10	68.24 ±6.20	57.77 ±13.09	60.45 ±1.63	98.75 ±0.00	97.50 ±0.00
400	1.29 ±0.10	1.98 ±0.40	1.54 ±0.11	6.60 ±1.06	6.16 ±0.10	63.81 ±4.86	63.06 ±10.70	68.32 ±6.60	98.75 ±0.00	97.50 ±0.00
600	1.41 ±0.20	1.81 ±0.22	1.87 ±0.07	6.60 ±1.06	6.16 ±0.10	69.64 ±3.98	62.92 ±9.20	69.84 ±3.07	98.75 ±0.00	97.50 ±0.00
800	1.62 ±0.30	2.04 ±0.35	2.33 ±0.35	6.60 ±1.06	6.16 ±0.10	67.45 ±3.72	63.22 ±7.89	72.50 ±2.28	98.75 ±0.00	97.50 ±0.00
1000	1.64 ±0.23	1.97 ±0.26	1.93 ±0.13	6.60 ±1.06	6.16 ±0.10	70.28 ±2.82	66.11 ±8.00	73.04 ±2.06	98.75 ±0.00	97.50 ±0.00
Table 7: IMDB. Effect of variation of validation data size  $(N_v)$ , using a UCB (i.e.,  $C_1 = 0.25$ ) on error estimates. We keep training data size  $N_q$  fixed at 500 and use error threshold  $\epsilon_a = 5\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
\(N_q\)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
200	4.57 ±0.21	4.30 ±0.20	3.97 ±0.06	6.44 ±0.20	6.20 ±0.09	93.46 ±1.01	90.60 ±2.91	91.97 ±0.28	99.50 ±0.00	99.00 ±0.00
400	4.60 ±0.09	3.75 ±0.83	3.92 ±0.86	11.86 ±13.54	6.81 ±1.26	92.55 ±0.66	84.27 ±19.15	89.95 ±2.92	99.00 ±0.00	98.00 ±0.00
600	4.93 ±0.10	3.99 ±0.91	4.69 ±0.09	6.31 ±1.28	6.33 ±0.10	92.45 ±0.84	91.69 ±3.99	91.20 ±0.42	98.50 ±0.00	97.00 ±0.00
800	4.76 ±0.12	3.55 ±0.69	4.37 ±0.14	6.91 ±1.49	6.12 ±0.10	92.15 ±1.05	89.98 ±3.38	89.97 ±0.69	98.00 ±0.00	96.00 ±0.00
1000	4.49 ±0.06	4.19 ±0.31	4.25 ±0.29	5.65 ±0.25	6.14 ±0.11	92.25 ±0.96	92.28 ±2.13	89.47 ±0.70	97.50 ±0.00	95.00 ±0.00
Table 8: IMDB. Effect of variation of  $N_{q}$ , the maximum number of samples the algorithm can use for training, without using a UCB (i.e.,  $C_1 = 0$ ) on error estimates. We keep validation data size  $N_{v}$  fixed at 1000 and use error threshold  $\epsilon_{a} = 5\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
\( {\mathrm{N}}_{\mathrm{q}} \)	Error (%)	Coverage (%)
TBAL	AL+SC	PL+SC	AL	PL	TBAL	AL+SC	PL+SC	AL	PL
200	\( {1.67} \pm {0.29} \)	\( {2.15} \pm {0.45} \)	\( {1.59} \pm {0.10} \)	\( {6.44} \pm {0.20} \)	\( {6.20} \pm {0.09} \)	\( {73.30} \pm {3.49} \)	\( {57.17} \pm {11.09} \)	\( {57.39} \pm {4.15} \)	\( {99.50} \pm {0.00} \)	\( {99.00} \pm {0.00} \)
400	\( {1.63} \pm {0.19} \)	\( {1.61} \pm {0.29} \)	\( {1.76} \pm {0.13} \)	\( {11.86} \pm {13.54} \)	\( {6.81} \pm {1.26} \)	\( {72.59} \pm {3.16} \)	\( {64.53} \pm {16.61} \)	\( {58.48} \pm {1.79} \)	\( {99.00} \pm {0.00} \)	\( {98.00} \pm {0.00} \)
600	\( {1.67} \pm {0.21} \)	\( {1.83} \pm {0.30} \)	\( {1.67} \pm {0.08} \)	\( {6.31} \pm {1.28} \)	\( {6.33} \pm {0.10} \)	\( {71.38} \pm {2.15} \)	\( {70.50} \pm {5.68} \)	\( {65.71} \pm {2.14} \)	\( {98.50} \pm {0.00} \)	\( {97.00} \pm {0.00} \)
800	\( {1.67} \pm {0.27} \)	\( {1.90} \pm {0.31} \)	\( {1.79} \pm {0.09} \)	\( {6.91} \pm {1.49} \)	\( {6.12} \pm {0.10} \)	\( {69.10} \pm {4.51} \)	\( {65.74} \pm {10.14} \)	\( {73.21} \pm {2.57} \)	\( {98.00} \pm {0.00} \)	\( {96.00} \pm {0.00} \)
1000	\( {1.62} \pm {0.22} \)	\( {1.97} \pm {0.35} \)	\( {1.70} \pm {0.12} \)	\( {5.65} \pm {0.25} \)	\( {6.14} \pm {0.11} \)	\( {73.42} \pm {2.84} \)	\( {68.05} \pm {5.56} \)	\( {64.18} \pm {2.11} \)	\( {97.50} \pm {0.00} \)	\( {95.00} \pm {0.00} \)
Table 9: IMDB. Effect of variation of  $N_{q}$ , the maximum number of samples the algorithm can use for training, using a UCB (i.e.,  $C_1 = 0.25$ ) on error estimates. We keep validation data size  $N_{v}$  fixed at 1000 and use error threshold  $\epsilon_{a} = 5\%$ . We report the mean and std. deviation over 10 runs with different random seeds.
D.4 Auto Labeling Visualization
In this section, we visualize the process of TBAL. We use the dimensionality reduction method, PaCMAP [66], to visualize the features of the samples. For neural network models, we visualize the PaCMAP embeddings of the penultimate layer's output and for linear models, we use PaCMAP on the raw features. In these figures, each row corresponds to one TBAL round. Each figure shows a few selected rounds of auto-labeling. Each figure has four columns (left to right), which show: a) The samples that are labeled by TBAL in the round are shown in that row. b) The embeddings for training samples in that round. c) The embeddings for validation data points in that round. d) The score distribution for the validation dataset in that round.
In Figure 9 we see visualizations for auto-labeling on the MNIST data using linear models. In this setting the data exhibits clustering structure in the PaCMAP embeddings learned on the raw features and the confidence (probability) scores produced are also reasonably well calibrated which leads to good auto-labeling performance.