SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Tiny imagenet dataset. http://cs231n.stanford.edu/tiny-imagenet-200.zip. Accessed: May 16, 2023.
[2] Airbus. Airbus active labeling blog. https://acubed.airbus.com/blog/wayfinder/ automatic-data-labeling-strategies-for-vision-based-machine-learning-and-ai/, 2022. Accessed: 2022-11-18.
[3] Maria-Florina Balcan, Andrei Z. Broder, and Tong Zhang. Margin based active learning. In COLT, 2007.
[4] Maria-Florina Balcan and Phil Long. Active and passive learning of linear separators under log-concave distributions. In Conference on Learning Theory, pages 288–316. PMLR, 2013.
[5] Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th annual international conference on machine learning, pages 49-56, 2009.
[6] Kamalika Chaudhuri, Sham M. Kakade, Praneeth Netrapalli, and Sujay Sanghavi. Convergence rates of active learning for maximum likelihood estimation. In Proceedings of the 28th International Conference on Neural Information Processing Systems, 2015.
[7] C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1):41-46, 1970.
[8] Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. Batch active learning at scale. In Advances in Neural Information Processing Systems, volume 34, pages 11933-11944, 2021.
[9] David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine Learning, 15(2):201-221, 1994.
[10] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory, pages 67-82. Springer, 2016.
[11] Sanjoy Dasgupta. Coarse sample complexity bounds for active learning. In Y. Weiss, B. Schölkopf, and J. Platt, editors, Advances in Neural Information Processing Systems, volume 18. MIT Press, 2006.
[12] Sanjoy Dasgupta. Two faces of active learning. Theoretical Computer Science, 412(19):1767-1781, 2011. Algorithmic Learning Theory (ALT 2009).
[13] Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Monteleoni. Analysis of perceptron-based active learning. In International conference on computational learning theory, pages 249-263. Springer, 2005.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
[15] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
[16] Giulia DeSalvo, Mehryar Mohri, and Umar Syed. Learning with deep cascades. In Proceedings of the Twenty-Sixth International Conference on Algorithmic Learning Theory (ALT 2015), 2015.
[17] Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. JMLR, 11:1605-1641, aug 2010.
[18] Ran El-Yaniv and Yair Wiener. Active learning via perfect selective classification. Journal of Machine Learning Research, 13(2), 2012.
[19] Hugging Face. Hugging face mteb leaderboard. https://huggingface.co/spaces/mteb/leaderboard, 2023. Accessed: 2023-08-07.
[61] Superb-AI. Superb ai automated data labeling service. https://www(superb-ai.com/product/automate, 2022. Accessed: 2022-11-18.
[62] Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classification. Journal of machine learning research, 2(Nov):45-66, 2001.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS 2017), 2017.
[64] Rudy Venguswamy, Mike Levy, Anirudh Koul, Satyarth Praveen, Tarun Narayanan, Ajay Krishnan, Jenessa Peterson, Siddha Ganju, and Meher Kasam. Curator: A No-Code Self-Supervised Learning and Active Labeling Tool to Create Labeled Image Datasets from Petabyte-Scale Imagery. In EGU General Assembly Conference Abstracts, pages EGU21-6853, April 2021.
[65] Deng-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. In Advances in Neural Information Processing Systems, volume 34, pages 11809-11820, 2021.
[66] Yingfan Wang, Haiyang Huang, Cynthia Rudin, and Yaron Shaposhnik. Understanding how dimension reduction tools work: An empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization. Journal of Machine Learning Research, 22(201):1-73, 2021.
[67] Yair Wiener and Ran El-Yaniv. Agnostic selective classification. Advances in neural information processing systems, 24, 2011.
[68] Yair Wiener and Ran El-Yaniv. Agnostic pointwise-competitive selective classification. Journal of Artificial Intelligence Research, 52:171–201, 2015.
[69] Ting-Fan Wu, Chih-Jen Lin, and Ruby Weng. Probability estimates for multi-class classification by pairwise coupling. Advances in Neural Information Processing Systems, 16, 2003.
[70] Shitao Xiao, Zheng Liu, Peitan Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
[71] Yufan Zeng and Jiashan Tang. Rlc-gnn: An improved deep architecture for spatial-based graph neural network with application to fraud detection. Applied Sciences, 11, 06 2021.
[72] Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. arXiv preprint arXiv:2204.00043, 2022.
The supplementary material is organized as follows. We provide a detailed discussion on related works in Section A. We give details of the definitions and notation in Section B.1. The notations are also summarized in the Table 5 in section B. Then we give the proof of the main theorem (Theorem 3.2) followed by proofs of supporting lemmas in section C. We provide details of its instantiation for finite VC-dimension hypothesis classes and the homogeneous linear separators case in Section C.3. Then, we provide the technical details of the lower bound (Lemma 3.3). Then we provide details of additional experiments in Section D. In Section D.4 we provide insights into auto-labeling using PaCMAP [66] visualizations of auto-labeled regions in each round.
There is a rich body of work on active learning on empirical and theoretical fronts [57, 12, 30, 28, 8, 54]. In active learning, the goal is to learn the best model in the given function class with fewer labeled data than in classical passive learning. To this end, various active learning algorithms have been developed and analyzed, e.g., uncertainty sampling [62, 45], disagreement region based [9, 27], margin based [3, 4], importance sampling based [5] and others [6]. Active learning has been shown to achieve exponentially smaller label complexity than passive learning in noiseless and low-noise settings [13, 3, 27, 28, 4, 11, 30, 6, 34, 32]. This suggests, in these settings auto-labeling using active learning followed by selective classification is expected to work well. However, in practice we do not have favorable noise conditions and the hypothesis class could be misspecified i.e. it may not contain the Bayes optimal classifier. In such cases, [31] proved lower bounds on the label complexity of active learning that are order wise same as passive learning. These findings have motivated more refined goals for active learning – abstain on hard to classify points and do well on the rest of the points. This idea is captured by the Chow's excess risk [7] and some of the recent works [60, 59, 47, 72] have proved exponential savings in label complexity for active learning when the goal is to minimize Chow's excess risk. The classifier learned by these methods is equipped with the abstain option and hence it can be readily applied for auto-labeling. However, the problem of misspecification of the hypothesis class still remains. Nevertheless, it would be interesting future work to explore the connections between auto-labeling and active learning with abstention. We also note that similar works on learning with abstention are done in the context of passive learning [10].
Another closely related line of work is the selective classification where the goal is to equip a given classifier with the option to abstain from the prediction in order to guarantee prediction quality. The foundations for selective classification are laid down in [17, 67, 18, 68] where they give results on the error rate in the prediction region and the coverage of a given classifier. However, they lack practical algorithms to find the prediction region. A recent work [26] proposes a new disagreement-based active learning strategy to learn a selective classifier.
Recent work studies a practical algorithm for threshold-based selective classification on deep neural networks [25]. The algorithm estimates the prediction threshold using training samples and they bound the error rate of the selective classifier using [23]. We note that their result is applicable to a specific setting of a given classifier. In contrast, in the TBAL algorithm analyzed in this paper, selective classification is done in each round and the classifiers are not given a priori but instead learned via ERM on training data which is adaptively sampled in each round.
Another related work [49] studies an algorithm similar to TBAL for auto-labeling. Their emphasis is on the cost of training incurred when these systems use large-scale model classes for auto-labeling. They propose an algorithm to predict the training set size that minimizes the overall cost and provides an empirical evaluation.
Well-calibrated uncertainty scores are essential to the success of threshold-based auto-labeling. However, in practice, such scores are often hard to get. Moreover, neural networks can produce overconfident ( unreliable) scores [29]. Fortunately, there are plenty of methods in the literature to deal with this problem [46, 69]. More recently, various approaches have been proposed for uncertainty calibration for neural networks [24, 42, 65, 35, 40, 56]. A detailed study of calibration methods and their impact on auto-labeling is beyond the scope of this work and left as future work.
There is another line of work emerging towards auto-labeling that does not rely on getting human labels but instead uses potentially noisy but cheaply available sources to infer labels [51, 52, 22]. The focus of this paper, however, is on analyzing the performance of TBAL algorithms [58, 2] that have emerged recently as auto-labeling solutions in systems.
B Definitions and Notation
B.1 Basic Definitions
Let  $\mathcal{X}$  be the instance space and  $p(\mathbf{x})$  be a density function supported on  $\mathcal{X}$ . For any  $\mathbf{x}_i \in \mathcal{X}$  let  $y_i$  be its true label. Let  $X = \{\mathbf{x}_1, \ldots, \mathbf{x}_N\}$  be a set of  $N$  i.i.d samples drawn from  $\mathcal{X}$ . Let set  $S \subseteq \mathcal{X}$  denote a non-empty sub-region of  $\mathcal{X}$  and  $S \subseteq X \cap S$  be a set of  $n > 0$  i.i.d. samples.
Definition B.1. (Hypothesis Class with Abstain) We can think of the function  $g$  along with set  $T$  as inducing an extended hypothesis class  $\mathcal{H}^{(T,g)}$ . Let  $\mathcal{H}^{T,g} = \mathcal{H} \times T$ . For any function  $(h,t) \in \mathcal{H}^{(T,g)}$  is defined as
$$
(h, t) (\mathbf {x}) := \left\{ \begin{array}{l l} h (\mathbf {x}) & \text {i f} g (h, \mathbf {x}) \geq t \\ \perp & \text {o . w .} \end{array} \right. \tag {3}
$$
Here  $(h,t)(\mathbf{x}) = \bot$  means the hypothesis  $(h,t)$  abstains in classifying the point  $\mathbf{x}$ . Otherwise, it is equal to  $h(\mathbf{x})$ .
The subset  $S(h,t) \subseteq S$  where  $(h,t)$  does not abstain and its complement  $\bar{S}(h,t)$  where  $(h,t)$  abstains, are defined as follows,
$$
\mathcal {S} (h, t) := \left\{\mathbf {x} \in \mathcal {S}: (h, t) (\mathbf {x}) \neq \bot \right\}, \quad \bar {\mathcal {S}} (h, t) := \left\{\mathbf {x} \in \mathcal {S}: (h, t) (\mathbf {x}) = \bot \right\}
$$
Probability Definitions. The probability  $\mathbb{P}(\mathcal{S})$  of subset  $S\subseteq \mathcal{X}$  and the conditional probability of any subset  $S^{\prime}\subseteq S$  are given as follows,
$$
\mathbb {P} (\mathcal {S}) := \mathbb {P} (\mathcal {S} | \mathcal {X}) := \int_ {\mathbf {x} \in \mathcal {S}} p (\mathbf {x}) d \mathbf {x}, \qquad \mathbb {P} (\mathcal {S} ^ {\prime} | \mathcal {S}) := \frac {\mathbb {P} (\mathcal {S} ^ {\prime} | \mathcal {X})}{\mathbb {P} (\mathcal {S} | \mathcal {X})}, \qquad \mathbb {P} (h, t | \mathcal {S}) := \mathbb {P} (\mathcal {S} (h, t) | \mathcal {S})
$$
The empirical probabilities of  $S$  and  $S' \subseteq S$  are defined as follows,
$$
\widehat {\mathbb {P}} (S) := \frac {| S |}{| X |}, \qquad \widehat {\mathbb {P}} (S ^ {\prime} | S) := \frac {| S ^ {\prime} |}{| S |}, \qquad \widehat {\mathbb {P}} (h, t | S) := \frac {| S (h , t) |}{| S |}
$$
Loss Functions. The loss functions are defined as follows,
$$
\ell_ {0 - 1} (h, \mathbf {x}, y) := \mathbb {1} (h (\mathbf {x}) \neq y),
$$
$$
\ell_ {\perp} (h, t, \mathbf {x}) := \mathbb {1} (g (h, \mathbf {x}) \geq t),
$$
$$
\ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x}, y) := \ell_ {0 - 1} (h, \mathbf {x}, y) \cdot \ell_ {\perp} (h, t, \mathbf {x}).
$$
Error Definitions. Define the conditional error in set  $S \subseteq \mathcal{X}$  as follows,
$$
\mathcal {E} (h, t | \mathcal {S}) := \mathbb {E} _ {\mathbf {x} | \mathcal {S}} [ \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x}, y) ] = \int_ {\mathbf {x} \in \mathcal {S}} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x}, y) \cdot \frac {p (\mathbf {x})}{\mathbb {P} (\mathcal {S})} d \mathbf {x}
$$
Then, the conditional error in set  $S(h,t)$  i.e. the subset of  $S$  on which  $(h,t)$  does not abstain,
$$
\mathcal {E} _ {a} (h, t | \mathcal {S}) := \mathcal {E} (h, t | \mathcal {S} (h, t)) := \mathbb {E} _ {\mathbf {x} | \mathcal {S} (h, t)} [ \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x}, y) ] = \frac {\mathcal {E} (h , t | \mathcal {S})}{\mathbb {P} (h , t | \mathcal {S})}
$$
Similarly, define their empirical counterparts as follows,
$$
\widehat {\mathcal {E}} (h, t | S) := \frac {1}{| S |} \sum_ {\mathbf {x} _ {i} \in S} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i}),
$$
$$
\widehat {\mathcal {E}} _ {a} (h, t | S) := \widehat {\mathcal {E}} (h, t | S (h, t)) := \frac {1}{| S (h , t) |} \sum_ {\mathbf {x} _ {i} \in S (h, t)} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i}),
$$
$$
\sum_ {\mathbf {x} _ {i} \in S} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i}) = \sum_ {\mathbf {x} _ {i} \in S (h, t)} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i}) = \sum_ {\mathbf {x} _ {i} \in S (h, t)} \ell_ {0 - 1} (h, \mathbf {x} _ {i}, y _ {i})
$$
Symbol	Definition
X	feature space.
Y	label space.
H	hypothesis space.
h	a hypothesis in H.
x,y	x is an element in X and y is its true label.
S,S	S⊆X is a sub-region in X, S = {x1, . . ., xn} i.i.d. samples in S.
Xpool	unlabeled pool of data points.
X(v(i),nu(i))	set of validation points at the beginning of ith round and nu(i) = |X(v(i)|.
X(a(i),na(i))	set of auto-labeled points in ith round and na(i) = |X(a(i)|.
hati,hati	ERM solution and auto-labeling thresholds respectively in ith round.
X(i)	unlabeled region left at the beginning of ith round.
X(i)	unlabeled pool left at the beginning of ith round.
ma(i)	number of auto-labeling mistakes in ith round.
k	number of rounds of the TBAL algorithm.
Xpool(Ak)	set of all auto-labeled points till the end of round k.
g	confidence function g: H × X → T. Where T ⊆ R+, usually T = [0,1]
HT,g	Cartesian product of H and T the range of g.
Na(k)	∑i=1k na(i).
l0-1(h,x,y)	1(h(x) ≠ y).
l⊥(h,t,x)	1(g(h,x) ≥ t).
l0-1(h,t,x,y)	l0-1(h,x,y) · l⊥((h,t),x).
Rn(H, l0-1)	Eσ,S[ suph∈H 1/n ∑i=1n σiℓ0-1(h, xi, yi)].
Rn(HT,g, l⊥)	Eσ,S[ sup(h,t)∈HT,g 1/n ∑i=1n σiℓ⊥(h,t, xi)].
Rn(HT,g, l⊥0-1)	Eσ,S[ sup(h,t)∈HT,g 1/n ∑i=1n σiℓ0-1(h,t, xi, yi)].
Rn(HT,g)	Rn(H, l0-1) + Rn(HT,g, l⊥).
E(h,t|S)	Ex|S[ l0-1(h,t, x, y)].
ˆE(h,t|S)	1/S| ∑i=1n l0-1(h,t, xi, yi).
P(h,t|S)	Ex|S[ l⊥(h,t, x)].
ˆP(h,t|S)	1/S| ∑i=1n l⊥(h,t, xi, yi).
Ea(h,t|S)	E(h,t|S)/P(h,t|S).
ˆEa(h,t|S)	ˆE(h,t|S)/P(h,t|S).
Table 5: Glossary of variables and symbols used in this paper.
Rademacher Complexity. The Rademacher complexities for the function classes induced by the  $\mathcal{H}, T, g$  and the loss functions are defined as follows,
$$
\mathfrak {R} _ {n} \big (\mathcal {H}, \ell_ {0 - 1} \big) := \mathbb {E} _ {\sigma , S} \Big [ \sup  _ {h \in \mathcal {H}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \ell_ {0 - 1} (h, \mathbf {x} _ {i}, y _ {i}) \Big ].
$$
$$
\mathfrak {R} _ {n} \big (\mathcal {H} ^ {T, g}, \ell_ {\perp} \big) := \mathbb {E} _ {\sigma , S} \Big [ \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \ell_ {\perp} (h, t, \mathbf {x} _ {i}) \Big ].
$$
$$
\mathfrak{R}_{n}\bigl(\mathcal{H}^{T,g},\ell_{0 - 1}^{\perp}\bigr):= \mathbb{E}_{\sigma ,S}\Big[\sup_{(h,t)\in \mathcal{H}^{T,g}}\frac{1}{n}\sum_{i = 1}^{n}\sigma_{i}\ell_{0 - 1}^{\perp}(h,t,\mathbf{x}_{i},y_{i})\Big].
$$
$$
\mathfrak {R} _ {n} \left(\mathcal {H} ^ {T, g}\right) := \mathfrak {R} _ {n} \left(\mathcal {H}, \ell_ {0 - 1}\right) + \mathfrak {R} _ {n} \left(\mathcal {H} ^ {T, g}, \ell_ {\perp}\right)
$$
B.2 Glossary
The notation is summarized in Table 5 below. More detailed notation is in section B.1.
C.1 Proofs for the General Setup
We begin by restating the theorem here and then give the proof.
Theorem 3.2. (Overall Auto-Labeling Error and Coverage) Let  $k$  denote the number of rounds of the TBAL Algorithm 1. Let  $n_v^{(i)}, n_a^{(i)}$  denote the number of validation and auto-labeled points at epoch  $i$  and  $n^{(i)} = |X^{(i)}|$ . Let  $X_{pool}(A_k)$  be the set of auto-labeled points at the end of round  $k$ .  $N_a^{(k)} = \sum_{i=1}^k n_a^{(i)}$  denote the total number of auto-labeled points. Then, for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \widehat{\mathcal{E}}\Big(X_{pool}(A_{k})\Big)\leq \sum_{i = 1}^{k}\frac{n_{a}^{(i)}}{N_{a}^{(k)}}\Bigg(\underbrace{\widehat{\mathcal{E}}_{a}\big(\hat{h}_{i},\hat{t}_{i}|X_{v}^{(i)}\big)}_{(a)} + \frac{4}{p_{0}}\big(\underbrace{\Re_{n_{v}^{(i)}}\big(\mathcal{H}^{T,g}\big) + \frac{2}{p_{0}}\sqrt{\frac{1}{n_{v}^{(i)}}\log(\frac{8k}{\delta})}}_{(b)}\big) \\ + \frac {4}{p _ {0}} \left(\underbrace {\sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \Re_ {n _ {a} ^ {(i)}} \left(\mathcal {H} ^ {T , g}\right) + \sqrt {\frac {k}{N _ {a} ^ {(k)}} \log \left(\frac {8 k}{\delta}\right)}} _ {(c)}\right), \quad a n d \\ \end{array}
$$
$$
\widehat {\mathcal {P}} (X _ {p o o l} (A _ {k})) \geq \sum_ {i = 1} ^ {k} \mathbb {P} \big (\mathcal {X} ^ {(i)} (\hat {h} _ {i}, \hat {t} _ {i}) \big) - 2 \Re_ {n ^ {(i)}} \big (\mathcal {H} ^ {T, g} \big) - \sqrt {\frac {2 k ^ {2}}{N} \log \left(\frac {8 k}{\delta}\right)}.
$$
Proof. Recall the definition of auto-labeling error,
$$
\widehat {\mathcal {E}} \Big (X _ {p o o l} (A _ {k}) \Big) = \sum_ {i = 1} ^ {k} \frac {m _ {a} ^ {(i)}}{N _ {a} ^ {(k)}}, \qquad m _ {a} ^ {(i)} = n _ {a} ^ {(i)} \cdot \widehat {\mathcal {E}} _ {a} \big (\hat {h} _ {i}, \hat {t} _ {i} | X ^ {(i)} \big).
$$
Here,  $m_{a}^{(i)}$  is the number of auto-labeling mistakes made by the Algorithm in the  $i$ th round and  $\hat{\mathcal{E}}_a(\hat{h}_i,\hat{t}_i|X^{(i)})$  is the auto-labeling error in that round. Note that we cannot observe these quantities since the true labels for the auto-labeled points are not available. To estimate the auto-labeling error of each round we make use of validation data. Using the validation data we first get an upper bound on the true error rate of the auto-labeling region i.e.  $\mathcal{E}_a(\hat{h}_i,\hat{t}_i|\mathcal{X}^{(i)})$  in terms of the auto-labeling error on the validation data  $\hat{\mathcal{E}}_a(\hat{h}_i,\hat{t}_i|X_v^{(i)})$  and then get an upper bound on empirical auto-labeling error rate  $\hat{\mathcal{E}}_a(\hat{h}_i,\hat{t}_i|X^{(i)})$  using the true error rate of the auto-labeling region.
We get these bounds by application of Lemma C.1 with  $\delta_3 = \delta /4k$  for each round and then apply union bound over all  $k$  epochs. Note that we have to apply the lemma twice, first to get the concentration bound w.r.t the validation data and second to get the concentration w.r.t to the auto-labeled points.
$$
\mathcal {E} _ {a} \big (\hat {h} _ {i}, \hat {t} _ {i} | \mathcal {X} ^ {(i)} \big) \leq \widehat {\mathcal {E}} _ {a} \big (\hat {h} _ {i}, \hat {t} _ {i} | X _ {v} ^ {(i)} \big) + \frac {4}{p _ {0}} \Re_ {n _ {v} ^ {(i)}} \big (\mathcal {H} ^ {T, g} \big) + \frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {v} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)}.
$$
$$
\widehat {\mathcal {E}} _ {a} \big (\hat {h} _ {i}, \hat {t} _ {i} | X ^ {(i)} \big) \leq \mathcal {E} _ {a} \big (\hat {h} _ {i}, \hat {t} _ {i} | \mathcal {X} ^ {(i)} \big) + \frac {4}{p _ {0}} \Re_ {n _ {a} ^ {(i)}} \big (\mathcal {H} ^ {T, g} \big) + \frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {a} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)}.
$$
Substituting  $\mathcal{E}_a\big(\hat{h}_i,\hat{t}_i|\mathcal{X}^{(i)}\big)$  by its upper confidence bound on the validation data.
$$
\begin{array}{l} \widehat {\mathcal {E}} _ {a} \left(\hat {h} _ {i}, \hat {t} _ {i} \mid X ^ {(i)}\right) \leq \widehat {\mathcal {E}} _ {a} \left(\hat {h} _ {i}, \hat {t} _ {i} \mid X _ {v} ^ {(i)}\right) + \frac {2}{p _ {0}} \Re_ {n _ {v} ^ {(i)}} \left(\mathcal {H} ^ {T, g}\right) + \frac {2}{p _ {0}} \Re_ {n _ {a} ^ {(i)}} \left(\mathcal {H} ^ {T, g}\right) \\ + \frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {v} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)} + \frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {a} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)}. \\ \end{array}
$$
Having an upper bound on the empirical auto-labeling error for  $i^{th}$  round gives us an upper bound on the number of auto-labeling mistakes  $m_{a}^{(i)}$  made in that round. It allows us to upper bound the total auto-labeling mistakes in all  $k$  rounds and thus the overall auto-labeling error as detailed below,
Since we have an upper bound on the empirical auto-labeling error in each round, we have an upper bound for each  $m_{a}^{(i)}$ , which are used as follows to get the bound on the auto-labeling error,
$$
\begin{array}{l} \widehat {\mathcal {E}} (X _ {p o o l} (A _ {k})) = \sum_ {i = 1} ^ {k} \frac {m _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \\ = \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \cdot \frac {m _ {a} ^ {(i)}}{n _ {a} ^ {(i)}} \\ = \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \cdot \widehat {\mathcal {E}} _ {a} \left(\hat {h} _ {i}, \hat {t} _ {i} \mid X ^ {(i)}\right) \\ \leq \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \cdot \left(\mathcal {E} _ {a} \left(\hat {h} _ {i}, \hat {t} _ {i} \mid \mathcal {X} ^ {(i)}\right) + \frac {4}{p _ {0}} \Re_ {n _ {a} ^ {(i)}} \left(\mathcal {H} ^ {T, g}\right) + \frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {a} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)}\right) \\ \leq \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \cdot \left(\widehat {\mathcal {E}} _ {a} (\hat {h} _ {i}, \hat {t} _ {i} | X _ {v} ^ {(i)}) + \frac {4}{p _ {0}} \Re_ {n _ {v} ^ {(i)}} (\mathcal {H} ^ {T, g}) + \frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {v} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)} \right. \\ \left. + \frac {4}{p _ {0}} \Re_ {n _ {a} ^ {(i)}} \left(\mathcal {H} ^ {T, g}\right) + \frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {a} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)}\right) \\ \leq \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \cdot \left(\hat {\mathcal {E}} _ {a} (\hat {h} _ {i}, \hat {t} _ {i} | X _ {v} ^ {(i)}) + \frac {4}{p _ {0}} \Re_ {n _ {v} ^ {(i)}} (\mathcal {H} ^ {T, g}) + \frac {4}{p _ {0}} \Re_ {n _ {a} ^ {(i)}} (\mathcal {H} ^ {T, g}) + \right. \\ \frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {v} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)} + \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \cdot \left(\frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {a} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)}\right) \\ \end{array}
$$
The last term is simplified as follows,