SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \cdot \left(\frac {2}{p _ {0}} \sqrt {\frac {1}{n _ {a} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)}\right) = \frac {2}{p _ {0}} \cdot \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \sqrt {\frac {1}{n _ {a} ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)} \\ = \frac {2}{p _ {0}} \cdot \frac {1}{N _ {a} ^ {(k)}} \sum_ {i = 1} ^ {k} \sqrt {n _ {a} ^ {(i)} \log \left(\frac {8 k}{\delta}\right)} \\ = \frac {2}{p _ {0}} \cdot \sqrt {\log \left(\frac {8 k}{\delta}\right)} \cdot \sum_ {i = 1} ^ {k} \frac {\sqrt {n _ {a} ^ {(i)}}}{N _ {a} ^ {(k)}} \\ \leq \frac {2}{p _ {0}} \cdot \sqrt {\log \left(\frac {8 k}{\delta}\right)} \cdot \sqrt {\frac {k}{N _ {a} ^ {(k)}}} \\ = \frac {2}{p _ {0}} \cdot \sqrt {\frac {k}{N _ {a} ^ {(k)}} \log \left(\frac {8 k}{\delta}\right)} \\ \end{array}
$$
The last inequality follows from the application of the inequality  $||\mathbf{u}||_1 \leq \sqrt{k} ||\mathbf{u}||_2$  for any vector  $\mathbf{u} \in \mathbb{R}^k$ . Here we let  $\mathbf{u} = [\sqrt{n_a^{(1)}}, \ldots, \sqrt{n_a^{(k)}}]$ , and since  $\forall i \sqrt{n_a^{(i)}} > 0$  so,  $\sum_{i=1}^{k} \sqrt{n_a^{(i)}} = ||\mathbf{u}||_1$
and  $N_{a}^{(k)} = ||\mathbf{u}||_{2}^{2}$
$$
\frac {\sum_ {i = 1} ^ {k} \sqrt {n _ {a} ^ {(i)}}}{N _ {a} ^ {(k)}} = \frac {| | \mathbf {u} | | _ {1}}{| | \mathbf {u} | | _ {2} ^ {2}} \leq \frac {\sqrt {k} | | \mathbf {u} | | _ {2}}{| | \mathbf {u} | | _ {2} ^ {2}} = \frac {\sqrt {k}}{| | \mathbf {u} | | _ {2}} = \sqrt {\frac {k}{N _ {a} ^ {(k)}}}
$$
To get the bound on coverage we follow the same steps except that we can use all the unlabeled pool of size  $n(i)$  to estimate the coverage in each round which gives us the bound in terms of  $n^{(i)}$  and  $N$  as follows,
$$
\begin{array}{l} \widehat {\mathcal {P}} (X _ {p o o l} (A _ {k})) = \frac {1}{N} \sum_ {i = 1} ^ {k} n _ {a} ^ {(i)} \\ = \frac {1}{N} \sum_ {i = 1} ^ {k} n ^ {(i)} \cdot \frac {n _ {a} ^ {(i)}}{n ^ {(i)}} \\ = \frac {1}{N} \sum_ {i = 1} ^ {k} n ^ {(i)} \cdot \widehat {\mathbb {P}} \left(X _ {a} ^ {(i)} \mid X ^ {(i)}\right) \\ = \frac {1}{N} \sum_ {i = 1} ^ {k} n ^ {(i)} \cdot \widehat {\mathbb {P}} (\hat {h} _ {i}, \hat {t} _ {i} | X ^ {(i)}) \\ \geq \sum_ {i = 1} ^ {k} \frac {n ^ {(i)}}{N} \left(\mathbb {P} \big (\hat {h} _ {i}, \hat {t} _ {i} \big | \boldsymbol {\mathcal {X}} ^ {(i)} \big) - 2 \mathfrak {R} _ {n ^ {(i)}} \left(\mathcal {H} ^ {T, g}\right) - \sqrt {\frac {1}{n ^ {(i)}} \log \left(\frac {8 k}{\delta}\right)}\right) \\ \geq \sum_ {i = 1} ^ {k} \frac {n ^ {(i)}}{N} \left(\mathbb {P} \left(\hat {h} _ {i}, \hat {t} _ {i} \mid \mathcal {X} ^ {(i)}\right) - 2 \Re_ {n ^ {(i)}} \left(\mathcal {H} ^ {T, g}\right)\right) - \sqrt {\frac {k}{N} \log \left(\frac {8 k}{\delta}\right)} \\ \end{array}
$$
We bound the first term as follows,
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} \frac {n ^ {(i)}}{N} \mathbb {P} \Big (\hat {h} _ {i}, \hat {t} _ {i} \big | \mathcal {X} ^ {(i)} \Big) = \sum_ {i = 1} ^ {k} \frac {n ^ {(i)}}{N} \cdot \frac {\mathbb {P} \big (\mathcal {X} ^ {(i)} (\hat {h} _ {i} , \hat {t} _ {i}) \big)}{\mathbb {P} \big (\mathcal {X} ^ {(i)} \big)} \\ \geq \sum_ {i = 1} ^ {k} \left(\mathbb {P} \left(\mathcal {X} ^ {(i)}\right) - \sqrt {\frac {1}{N} \log \left(\frac {8 k}{\delta}\right)}\right) \cdot \left(\frac {\mathbb {P} \left(\mathcal {X} \left(\hat {h} _ {i} , \hat {t} _ {i}\right)\right)}{\mathbb {P} \left(\mathcal {X} ^ {(i)}\right)}\right) \\ \geq \sum_ {i = 1} ^ {k} \mathbb {P} \left(\mathcal {X} ^ {(i)} \left(\hat {h} _ {i}, \hat {t} _ {i}\right)\right) - \sqrt {\frac {1}{N} \log \left(\frac {8 k}{\delta}\right)} \\ \end{array}
$$
Substituting it back we get,
$$
\begin{array}{l} \widehat {\mathcal {P}} \big (X _ {p o o l} (A _ {k}) \big) \geq \sum_ {i = 1} ^ {k} \mathbb {P} \big (\mathcal {X} ^ {(i)} (\hat {h} _ {i}, \hat {t} _ {i}) \big) - 2 \Re_ {n ^ {(i)}} \big (\mathcal {H} ^ {T, g} \big) - k \sqrt {\frac {1}{N} \log \left(\frac {8 k}{\delta}\right)} - \sqrt {\frac {k}{N} \log \left(\frac {8 k}{\delta}\right)} \\ \geq \sum_ {i = 1} ^ {k} \mathbb {P} \left(\mathcal {X} ^ {(i)} (\hat {h} _ {i}, \hat {t} _ {i})\right) - 2 \Re_ {n ^ {(i)}} \left(\mathcal {H} ^ {T, g}\right) - \sqrt {\frac {4 k ^ {2}}{N} \log \left(\frac {8 k}{\delta}\right)} \\ \end{array}
$$
For the last step we use the inequality  $\sqrt{a} +\sqrt{b}\leq \sqrt{2(a + b)}$  for any  $a,b\in \mathbb{R}^+$
Rademacher complexity and validation error trade-off. The bound contains validation errors at different thresholds. We revisit the definition of validation error appearing in the bound. Let  $X_{v} = \{x_{1},\ldots x_{n_{v}}\}$  be the validation samples and  $y_{i}$  be the label corresponding to  $x_{i}$ . Given any  $h\in \mathcal{H}$  and the confidence function  $g$ , we have different subsets  $X_{v}(h,t)$  of the validation points for
which the model's confidence is higher than  $t$ . More precisely,  $X_{v}(h,t) = \{x_{i}\in X_{v}:g(h,x_{i})\geq t\}$  and the validation error  $\hat{\mathcal{E}}_a(h,t|X_v)$  is computed on each of these subsets as follows,  $\tilde{\mathcal{E}}_a(h,t|X_v) = \frac{1}{|X_v(h,t)|}\sum_{x_i\in X_v(h,t)}\mathbf{1}(h(x_i)\neq y_i)$  this error is different from the overall validation error which is computed over the entire set of validation points  $X_{v}$ :
$$
\hat {\mathcal {E}} (h | X _ {v}) = \frac {1}{| X _ {v} |} \sum_ {x _ {i} \in X _ {v}} \mathbf {1} (h (x _ {i}) \neq y _ {i}).
$$
The TBAL method computes  $\hat{\mathcal{E}}_a(h,t|X_v)$  at different thresholds  $(t)$  and selects the threshold at which it is at most  $\epsilon$ . Thus even if the overall validation error  $\hat{\mathcal{E}}(h|X_v)$  is bad, there could still be regions in the space where the conditional validation error  $\hat{\mathcal{E}}_a(h,t|X_v)$  is small. This can be easily seen in Figure 2 in the paper. Here we are doing auto-labeling using a linear function class that has low Rademacher complexity. All the models in this class have high overall validation error  $\hat{\mathcal{E}}(h|X_v)$  but there are subsets where the conditional validation error  $\hat{\mathcal{E}}_a(h,t|X_v)$  is small and TBAL is able to find those subsets. Thus we see that working with low Rademacher complexity classes might lead to high overall validation error. However, it does not affect TBAL as long as there are regions of low conditional validation error  $\hat{\mathcal{E}}_a(h,t|X_v)$ . Furthermore, our upper bound on excess auto-labeling error depends on  $\hat{\mathcal{E}}_a(h,t|X_v)$  which due to the TBAL procedure is at most  $\epsilon$  and hence it is not in conflict with the Rademacher complexity term.
Next, we state the result for uniform convergence between  $\mathcal{E}_a(h,t|S),\widehat{\mathcal{E}}_a(h,t|S)$  and give its proof.
Lemma C.1. For any  $\delta_3, p_0 \in (0,1)$ , let  $S$  and  $S$  be defined as above. Let  $\mathbb{P}(h,t|S) \geq p_0$  and  $\hat{\mathbb{P}}(h,t|S) \geq p_0 \forall (h,t) \in \mathcal{H}^{T,g}$ , the following holds w.p. at least  $1 - \delta_3/2$
$$
\left| \mathcal {E} _ {a} (h, t | S) - \widehat {\mathcal {E}} _ {a} (h, t | S) \right| \leq \frac {4}{p _ {0}} \Re_ {n} (\mathcal {H} ^ {T, g}) + \frac {2}{p _ {0}} \sqrt {\frac {1}{n} \log \left(\frac {2}{\delta_ {3}}\right)} \quad \forall (h, t) \in \mathcal {H} ^ {T, g}. \tag {4}
$$
Proof. We begin with proving one side of the inequality and the other side is shown by following the same steps. The proof is based on applying the uniform convergence results for  $\widehat{\mathcal{E}}(h, t|S)$  and  $\widehat{\mathbb{P}}(h, t|S)$  from Lemma C.2. The main difficulty here is that  $\mathbb{E}_S[\widehat{\mathcal{E}_a}(h, t|S)] \neq \mathcal{E}_a(h, t|S)$ , so we cannot directly get the above result from standard uniform convergence bounds.
We prove it, by using the results from the Lemma C.2 and restricting the region  $S$  such that it has probability mass at least  $p_0$ .
By definitions of  $\mathcal{E}_a(h,t|S)$  and  $\widehat{\mathcal{E}}_a(h,t|S)$  we have,
$$
\mathcal {E} (h, t | \mathcal {S}) = \mathbb {P} (h, t | \mathcal {S}) \cdot \mathcal {E} _ {a} (h, t | \mathcal {S}) \qquad \text {a n d} \qquad \widehat {\mathcal {E}} (h, t | S) = \widehat {\mathbb {P}} (h, t | S) \cdot \widehat {\mathcal {E}} _ {a} (h, t | S).
$$
Let  $\xi_{1} = \sqrt{(1 / n)\log(2 / \delta_{1})},\xi_{2} = \sqrt{(1 / n)\log(2 / \delta_{2})}$ . From lemma C.2 we have,
$$
\mathcal {E} (h, t | \mathcal {S}) \leq \widehat {\mathcal {E}} (h, t | S) + 2 \Re_ {n} \left(\mathcal {H} ^ {T, g}\right) + \xi_ {1} \quad \forall (h, t) \in \mathcal {H} ^ {T, g} \quad \text {w . p .} 1 - \delta_ {1} / 2. \tag {5}
$$
$$
\hat {\mathbb {P}} (h, t | S) \leq \mathbb {P} (h, t | S) + 2 \Re_ {n} \left(\mathcal {H} ^ {T, g}\right) + \xi_ {2} \quad \forall (h, t) \in \mathcal {H} ^ {T, g} \quad \text {w . p .} 1 - \delta_ {2} / 2. \tag {6}
$$
Plugging in the above definitions of errors in equation (5) we get,
$$
\mathbb {P} (h, t | \mathcal {S}) \cdot \mathcal {E} _ {a} (h, t | \mathcal {S}) \leq \widehat {\mathbb {P}} (h, t | S) \cdot \widehat {\mathcal {E}} _ {a} (h, t | S) + 2 \Re_ {n} (\mathcal {H} ^ {T, g}) + \xi_ {1}. \tag {7}
$$
$$
\mathcal {E} _ {a} (h, t | \mathcal {S}) \leq \frac {\widehat {\mathbb {P}} (h , t | S)}{\mathbb {P} (h , t | \mathcal {S})} \widehat {\mathcal {E}} _ {a} (h, t | S) + 2 \frac {\Re_ {n} (\mathcal {H} ^ {T , g})}{\mathbb {P} (h , t | \mathcal {S})} + \frac {\xi_ {1}}{\mathbb {P} (h , t | \mathcal {S})}. \tag {8}
$$
Substituting  $\widehat{\mathbb{P}}(h, t|S)$  from equation 6 in the above equation, we get the following w.p.  $(1 - \delta_1/2)(1 - \delta_2/2), \forall (h, t) \in \mathcal{H}^{T,g}$ ,
$$
\mathcal {E} _ {a} (h, t | \mathcal {S}) \leq \left(\frac {\mathbb {P} (h , t | \mathcal {S}) + 2 \Re_ {n} (\mathcal {H} ^ {T , g}) + \xi_ {2}}{\mathbb {P} (h , t | \mathcal {S})}\right) \widehat {\mathcal {E}} _ {a} (h, t | S) + \frac {2 \Re_ {n} (\mathcal {H} ^ {T , g})}{\mathbb {P} (h , t | \mathcal {S})} + \frac {\xi_ {1}}{\mathbb {P} (h , t | \mathcal {S})}.
$$
$$
\begin{array}{l} = \left(1 + \frac {2 \Re_ {n} \left(\mathcal {H} ^ {T , g}\right)}{\mathbb {P} (h , t | \mathcal {S})} + \frac {\xi_ {2}}{\mathbb {P} (h , t | \mathcal {S})}\right) \widehat {\mathcal {E}} _ {a} (h, t | S) + \frac {2 \Re_ {n} \left(\mathcal {H} ^ {T , g}\right)}{\mathbb {P} (h , t | \mathcal {S})} + \frac {\xi_ {1}}{\mathbb {P} (h , t | \mathcal {S})}. \\ = \widehat {\mathcal {E}} _ {a} (h, t | S) + \frac {2 \Re_ {n} (\mathcal {H} ^ {T , g})}{\mathbb {P} (h , t | S)} \cdot \widehat {\mathcal {E}} _ {a} (h, t | S) + \frac {\xi_ {2}}{\mathbb {P} (h , t | S)} \widehat {\mathcal {E}} _ {a} (h, t | S) + \frac {2 \Re_ {n} (\mathcal {H} ^ {T , g})}{\mathbb {P} (h , t | S)} \\ + \frac {\xi_ {1}}{\mathbb {P} (h , t | \mathcal {S})}. \\ \end{array}
$$
Using upper bound  $\widehat{\mathcal{E}}_a(h,t|S)\leq 1$  in the second and third terms,
$$
\mathcal {E} _ {a} (h, t | \mathcal {S}) \leq \widehat {\mathcal {E}} _ {a} (h, t | S) + 4 \frac {\Re_ {n} (\mathcal {H} ^ {T , g})}{\mathbb {P} (h , t | \mathcal {S})} + \frac {\xi_ {1} + \xi_ {2}}{\mathbb {P} (h , t | \mathcal {S})} \quad \forall (h, t) \in \mathcal {H} ^ {T, g} w. p. \geq 1 - (\delta_ {1} + \delta_ {2}) / 2.
$$
Using  $\mathbb{P}(h,t|\mathcal{S})\geq p_0$
$$
\mathcal {E} _ {a} (h, t | \mathcal {S}) \leq \widehat {\mathcal {E}} _ {a} (h, t | S) + \frac {4}{p _ {0}} \Re_ {n} (\mathcal {H} ^ {T, g}) + \frac {\xi_ {1} + \xi_ {2}}{p _ {0}} \quad \forall (h, t) \in \mathcal {H} ^ {T, g} w. p. \geq 1 - (\delta_ {1} + \delta_ {2}) / 2.
$$
Letting  $\delta_1 = \delta_2 = \delta_3$  and  $\xi_{1} = \xi_{2} = \frac{\xi\cdot n_{0}}{2}$  gives  $\xi = \sqrt{\frac{4}{p_0^2n}\log\left(\frac{2}{\delta_3}\right)}$  and
$$
\mathcal {E} _ {a} (h, t | \mathcal {S}) \leq \widehat {\mathcal {E}} _ {a} (h, t | S) + \frac {4}{p _ {0}} \Re_ {n} (\mathcal {H} ^ {T, g}) + \xi \quad \forall (h, t) \in \mathcal {H} ^ {T, g} w. p. \geq 1 - \delta_ {3}.
$$
This proves one side of the result, and the other side of the result follows similarly.
Lemma C.2. Let  $S \subseteq \mathcal{X}$  be a sub-region of  $\mathcal{X}$  and  $S = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$  be a set of  $n$  i.i.d samples in  $S$  drawn from distribution  $P_{\mathbf{x}}$ . Let  $\{y_1, \ldots, y_n\}$  be the corresponding true labels, let  $\Re_n(\mathcal{H}^{T,g})$  be the rademacher complexity of class  $\mathcal{H}^{T,g}$  then for any  $\delta_1, \delta_2 \in (0,1)$  we have,
$$
\left| \mathcal {E} (h, t | \mathcal {S}) - \widehat {\mathcal {E}} (h, t | S) \right| \leq 2 \Re_ {n} \left(\mathcal {H} ^ {T, g}\right) + \sqrt {\frac {1}{n} \log \left(\frac {2}{\delta_ {1}}\right)} \quad \forall (h, t) \in \mathcal {H} ^ {T, g} \quad w. p. 1 - \delta_ {1} / 2. \tag {9}
$$
$$
\left| \mathbb {P} (h, t | \mathcal {S}) - \hat {\mathbb {P}} (h, t | S) \right| \leq 2 \Re_ {n} \left(\mathcal {H} ^ {T, g}\right) + \sqrt {\frac {1}{n} \log \left(\frac {2}{\delta_ {2}}\right)} \quad \forall (h, t) \in \mathcal {H} ^ {T, g} \quad w. p. 1 - \delta_ {2} / 2. \tag {10}
$$
Proof. The proof is similar to the standard proofs for Rademacher complexity based generalization error bound. Since we work with the modified loss function and hypothesis class to include the abstain option, for completeness we give the proof here. The proofs for error and probability bounds are very much the same except for the change in the loss function. We give the proof for the error bound here.
The result follows by applying McDiarmid's inequality on the function  $\phi(S)$  defined as below,
$$
\phi (S) := \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \mathcal {E} (h, t | \mathcal {S}) - \widehat {\mathcal {E}} _ {S} (h, t | S).
$$
To apply McDiarmid's inequality we first show that  $\phi(S)$  satisfies the bounded difference property (Lemma C.4). This gives us,
$$
\mathcal {E} (h, t | \mathcal {S}) - \widehat {\mathcal {E}} _ {S} (h, t | S) \leq \phi (S) \leq \mathbb {E} _ {S} [ \phi (S) ] + \sqrt {\frac {1}{n} \log \left(\frac {2}{\delta_ {1}}\right)} \quad \forall (h, t) \in \mathcal {H} ^ {T, g} \quad \text {w . p .} 1 - \frac {\delta_ {1}}{2}.
$$
Using the bound on  $\mathbb{E}_S[\phi (S)]$  from Lemma C.3 we get,
$$
\mathcal {E} (h, t | \mathcal {S}) \leq \widehat {\mathcal {E}} (h, t | S) + 2 \Re_ {n} \left(\mathcal {H} ^ {T, g}\right) + \sqrt {\frac {1}{n} \log \left(\frac {2}{\delta_ {1}}\right)} \quad \forall (h, t) \in \mathcal {H} ^ {T, g} \quad \text {w . p .} 1 - \frac {\delta_ {1}}{2}.
$$
Similarly, the bound for the other side is obtained which holds w.p.  $1 - \delta_1 / 2$ , and combining both we get eq. (9).
The bound of probabilities is obtained by following the same steps as above but with a different loss function,  $\ell_{\perp}$ , since  $\mathbb{P}(h,t|\mathcal{S})$  is the probability mass of the region where  $(h,t)$  does not abstain.
Lemma C.3. Let  $S \subseteq \mathcal{X}$  be a sub-region of  $\mathcal{X}$  and  $S = \{\mathbf{x}_1, \ldots, \mathbf{x}_n\}$  be a set of  $n$  i.i.d samples in  $S$  drawn from distribution  $P_{\mathbf{x}}$ . Let  $\{y_1, \ldots, y_n\}$  be the corresponding true labels and let  $\Re_n(\mathcal{H}^{T,g})$  be the Rademacher complexity of the function class  $\mathcal{H}^{T,g}$  defined over  $n$  i.i.d. samples. Then we have,
$$
\mathbb {E} _ {S} \left[ \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \mathcal {E} (h, t | S) - \widehat {\mathcal {E}} (h, t | S) \right] \leq 2 \Re_ {n} \left(\mathcal {H} ^ {T, g}\right). \tag {11}
$$
Proof. Let  $\tilde{S} = \{\tilde{\mathbf{x}}_1, \tilde{\mathbf{x}}_2, \ldots, \tilde{\mathbf{x}}_n\}$  be another set of independent draws from the same distribution as of  $S$  and let the corresponding labels be  $\{\tilde{y}_1, \ldots, \tilde{y}_n\}$ . These samples are usually termed as ghost samples and do not need to be counted in the sample complexity.
$$
\begin{array}{l} \mathbb {E} _ {S} \Big [ \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \mathcal {E} (h, t | S) - \widehat {\mathcal {E}} (h, t | S) \Big ] = \mathbb {E} _ {S} \Big [ \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \mathbb {E} _ {\tilde {S}} \big [ \widehat {\mathcal {E}} (h, t | \tilde {S}) \big ] - \widehat {\mathcal {E}} (h, t | S) \Big ]. \\ = \mathbb {E} _ {S} \Big [ \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \mathbb {E} _ {\tilde {S}} \big [ \widehat {\mathcal {E}} (h, t | \tilde {S}) - \widehat {\mathcal {E}} (h, t | S) \big ] \Big ]. \\ \leq \mathbb {E} _ {S} \left[ \mathbb {E} _ {\tilde {S}} \left[ \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \left[ \hat {\mathcal {E}} (h, t | \tilde {S}) - \hat {\mathcal {E}} (h, t | S) \right] \right] \right]. \\ = \mathbb {E} _ {S, \tilde {S}} \Big [ \sup  _ {h \in \mathcal {H} ^ {T, g}} \left[ \widehat {\mathcal {E}} (h, t | \tilde {S}) - \widehat {\mathcal {E}} (h, t | S) \right] \Big ]. \\ = \mathbb {E} _ {S, \tilde {S}} \Big [ \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \left[ \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {0 - 1} ^ {\perp} (h, t, \tilde {\mathbf {x}} _ {i}, \tilde {y} _ {i}) - \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i}) \right] \Big ]. \\ = \mathbb {E} _ {\sigma , S, \tilde {S}} \Big [ \sup  _ {h \in \mathcal {H} ^ {T, g}} \Big [ \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \ell_ {0 - 1} ^ {\perp} (h, t, \tilde {\mathbf {x}} _ {i}, \tilde {y} _ {i}) - \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i}) \Big ] \Big ]. \\ \leq \mathbb {E} _ {\sigma , \tilde {S}} \left[ \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \ell_ {0 - 1} ^ {\perp} (h, t, \tilde {\mathbf {x}} _ {i}, \tilde {y} _ {i}) \right] + \\ \mathbb {E} _ {\sigma , S} \Big [ \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i}) \Big ]. \\ = 2 \Re_ {n} \left(\mathcal {H} ^ {T, g}, \ell_ {0 - 1} ^ {\perp}\right). \\ \leq 2 \Re_ {n} \left(\mathcal {H} ^ {T, g}\right). \\ \end{array}
$$
In the last step, we used the upper bound on the Rademacher complexity from Lemma C.5.
Lemma C.4. (Bounded Difference) Let  $S$  be a set of i.i.d samples from  $P_{\mathbf{x}}$  then for  $\phi(S) \coloneqq \sup_{(h,t) \in \mathcal{H}^{T,g}} \mathcal{E}(h,t|S) - \widehat{\mathcal{E}}(h,t|S)$ , with probability at least  $1 - \delta$ ,
$$
\phi (S) \leq \mathbb {E} _ {S} [ \phi (S) ] + \sqrt {\frac {1}{| S |} \log \left(\frac {1}{\delta}\right)} \tag {12}
$$
Proof. It is proved by showing that  $\phi(S)$  satisfies the conditions (in particular the bounded difference assumption) needed for the application of McDiarmid Inequality. To see this, Let  $S = \{\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_i, \ldots, \mathbf{x}_n\}$  and let  $S' = \{\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_i', \ldots, \mathbf{x}_n\}$ , i.e.  $S$  and  $S'$  may differ only on the  $i^{th}$  sample.
$$
\begin{array}{l} | \phi (S) - \phi (S ^ {\prime}) | = \Big | \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \mathcal {E} (h, t | \mathcal {S}) - \widehat {\mathcal {E}} (h, t | S) - \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \mathcal {E} (h, t | \mathcal {S}) - \widehat {\mathcal {E}} (h, t | S ^ {\prime}) \Big |. \\ \leq \Big| \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \left(\mathcal {E} (h, t | \mathcal {S}) - \widehat {\mathcal {E}} (h, t | \mathcal {S}) - \mathcal {E} (h, t | \mathcal {S}) + \widehat {\mathcal {E}} (h, t | S ^ {\prime})\right) \Big|. \\ = \Big| \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \left(\widehat {\mathcal {E}} (h, t | S) - \widehat {\mathcal {E}} (h, t | S ^ {\prime})\right) \Big|. \\ = \Big| \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \left(\frac {1}{| S |} \sum_ {\mathbf {z} _ {j} \in S} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {j}, y _ {j}) - \frac {1}{| S ^ {\prime} |} \sum_ {\mathbf {z} _ {j} \in S ^ {\prime}} \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {j}, y _ {j})\right) \Big|. \\ \end{array}
$$
$$
\begin{array}{l} = \Big | \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \left(\frac {1}{n} \sum_ {j \neq i} \left(\ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {j}, y _ {j}) - \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {j}, y _ {j})\right) \right. + \\ \left. \frac {1}{n} \left(\ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i}) - \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i} ^ {\prime}, y _ {i} ^ {\prime})\right)\right) \Bigg). \\ = \Big| \sup  _ {(h, t) \in \mathcal {H} ^ {T, g}} \left(\frac {1}{n} \big (\ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i}, y _ {i}) - \ell_ {0 - 1} ^ {\perp} (h, t, \mathbf {x} _ {i} ^ {\prime}, y _ {i} ^ {\prime}) \big)\right) \Big |. \\ \leq \frac {1}{n} \\ \end{array}
$$
The last step follows since  $\ell_{0 - 1}^{\perp}$  is a 0-1 loss function so letting  $\ell_{0 - 1}^{\perp}(h,t,\mathbf{x}_i,y_i) = 1$  and  $\ell_{0 - 1}^{\perp}(h,t,\mathbf{x}_i',y_i') = 0$  gives an upper bound on the difference. Thus we can apply McDiarmid Inequality here and get the bound.
The relationship between the Rademacher complexities is obtained using the following Lemma C.5 due to [16].
Lemma C.5. ([16]) Let  $\ell_{0-1}, \ell_{\perp}, \ell_{0-1}^{\perp}$  be the loss functions defined as above and the Rademacher complexities on  $n$  i.i.d. samples  $S$  be  $\Re_n(\mathcal{H}, \ell_{0-1}), \Re_n(\mathcal{H}^{T,g}, \ell_{\perp}), \Re_n(\mathcal{H}^{T,g}, \ell_{0-1}^{\perp})$  respectively. Then,
$$
\Re_ {n} \left(\mathcal {H} ^ {T, g}, \ell_ {0 - 1} ^ {\perp}\right) \leq \Re_ {n} \left(\mathcal {H}, \ell_ {0 - 1}\right) + \Re_ {n} \left(\mathcal {H} ^ {T, g}, \ell_ {\perp}\right) =: \Re_ {n} \left(\mathcal {H} ^ {T, g}\right). \tag {13}
$$
Detailed proof of this lemma can be found in [16]. The result follows by expressing  $\ell_{0 - 1}\cdot \ell_{\perp}$  as  $(\ell_{0 - 1} + \ell_{\perp} - 1)_+$  and then applying Talagrand's contraction lemma [39].
C.2 Bounds for Finite VC-Dimension Classes
Here we specialize the auto-labeling error and coverage bounds to the setting of finite VC-dimension classes and then instantiate for a specific setting of homogeneous linear classifiers and uniform distribution.
Lemma C.6. [43] (Corollary 3.8 and 3.18). Let the VC-dimension of function class induced by  $\mathcal{F}$  be any class of functions from  $\mathcal{X} \mapsto \mathcal{Y} \cup \{\bot\}$ , and  $\ell : \mathcal{Y} \cup \{\bot\} \mapsto \{0,1\}$  be a 0-1 function. Then,
$$
\Re_ {n} (\mathcal {F}, \ell) \leq \sqrt {\frac {2 \mathcal {V} (\mathcal {F} , \ell)}{n} \log \left(\frac {e n}{\mathcal {V} (\mathcal {F} , \ell)}\right)}. \tag {14}
$$
Corollary C.7. (Auto-Labeling Error and Coverage for Finite VC-dimension Classes) Let  $k$  denote the number of rounds of TBAL algorithm 1. Let  $\mathcal{V}(\mathcal{H}^{T,g}) = d$  Let  $X_{pool}(A_k)$  be the set of auto-labeled points at the end of round  $k$ .  $N_a^{(k)} = \sum_{i=1}^k n_a^{(i)}$  denote the total number of auto-labeled points. With probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \widehat {\mathcal {E}} (X _ {p o o l} (A _ {k})) \leq \sum_ {i = 1} ^ {k} \frac {n _ {a} ^ {(i)}}{N _ {a} ^ {(k)}} \left(\underbrace {\widehat {\mathcal {E}} _ {a} (\hat {h} _ {i} , \hat {t} _ {i} | X _ {v} ^ {(i)})} _ {(a)} + \underbrace {\frac {4}{p _ {0}} \sqrt {\frac {2}{n _ {v} ^ {(i)}} \left(2 d \log \left(\frac {e n _ {v} ^ {(i)}}{d}\right) + \log \left(\frac {8 k}{\delta}\right)\right)}} _ {(b)}\right) \\ + \underbrace {\frac {4}{p _ {0}} \left(\sqrt {\frac {2 k}{N _ {a} ^ {(k)}} \left(2 d \log \left(\frac {e N _ {a} ^ {(k)}}{d}\right) + \log \left(\frac {8 k}{\delta}\right)\right)}\right)} _ {c} \\ \end{array}
$$
$$
\widehat {\mathcal {P}} \left(X _ {\text {p o o l}} \left(A _ {k}\right)\right) \geq \sum_ {i = 1} ^ {k} \mathbb {P} \left(\mathcal {X} ^ {(i)} \left(\hat {h} _ {i}, \hat {t} _ {i}\right)\right) - 2 k \sqrt {\frac {2}{N} \left(2 d \log \left(\frac {e N}{d}\right) + \log \left(\frac {8 k}{\delta}\right)\right)}.
$$
Proof. The proof follows by substituting the Rademacher complexity bounds for finite VC dimension function classes from Lemma C.6 in the general result from Theorem 3.2.