SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $L_{S}(h) \coloneqq 1 / |\mathcal{S}| \sum_{(x,y) \in \mathcal{S}} \mathbb{1}[h(x) \neq y]$  is a standard empirical loss estimate<sup>1</sup>, and  $h_{\mathcal{H}_i}$  is an arbitrarily chosen hypothesis from  $\mathcal{H}_i$  that is used in the loss estimate of every  $h \in \mathcal{H}_i$ . This leads to an unbiased estimator given that every  $h \in \mathcal{H}_i$  labels the sample from this part of space in exactly the same way.
The utility of this estimator is that by choosing an arbitrary representative  $h_{\mathcal{H}_i}$ , we can estimate the loss of all hypotheses still in contention to precision  $O(\epsilon)$  on  $R_i^c$  with  $\tilde{O}(1/\epsilon^2)$  samples, removing the usual dependence of the VC-dimension. On the other hand, as the disagreement region shrinks,  $\mathbb{P}_{D_G}(x \in R_i)$  shrinks as well, so while we will still need to invoke uniform convergence to get reliable loss estimates in  $R_i$ , the precision to which we need to estimate losses in this part of space decreases with every iteration, and eventually the overall dependence on the VC-dimension is diminished. This later observation is the standard source of gains in DBAL [18, 9, 24].
After forming these loss estimates on each group, we construct unbiased loss estimates for the worst case over groups via
$$
L_{\mathcal{S};R_{i}}^{\max}(h):= \max_{g\in \mathcal{G}}L_{\mathcal{S};R_{i}}(h\mid g).
$$
These loss estimates inherit concentration properties from the two-part estimator above. We draw enough samples at each iteration  $i$  such that we essentially learn the multi-group problem to precision  $2^{\lceil \log (1 / \epsilon) \rceil - i} \epsilon$ .
We note that Algorithm 1 assumes access to the underlying group marginals measures  $\mathbb{P}_{D_G}$ . This is common in the active learning literature [18, 20]. Probabilities of events in instance space can be estimated to arbitrary accuracy using only unlabeled data, so this assumption is not dangerous to our goal of lowering label complexities.
${}^{1}$  taken to be an arbitrary constant if  $S = \varnothing$  ; see the Appendix for details.
5.2 Guarantees
Vitally, the scheme given in Algorithm 1 is consistent. It is a lemma of ours that the number of samples drawn at each iteration is sufficiently large that the true error of any  $h \in \mathcal{H}_{i+1}$  is no more than  $2^{\lceil \log(1/\epsilon) \rceil - i\epsilon}$ . Thus, after  $\lceil \log(1/\epsilon) \rceil$  iterations, the ERM hypothesis on  $L_{S;R_i}^{\max}(\cdot)$  is then  $\epsilon$ -optimal with high probability.
We can bound the label complexity of the algorithm using standard techniques from DBAL. A ubiquitous quantity in the analysis of disagreement-based schemes is that of the "disagreement coefficient" [9, 25]. The general idea is that the disagreement coefficient bounds the rate of decrease in  $r$  of the measure of the disagreement region of a ball of radius  $r$  around  $h^*$  in the pseudo-metric  $\rho_g(h,h')\coloneqq \mathbb{P}_{D_g}\left(h(x)\neq h'(x)\right)$ . Precisely, we use the following definition of the disagreement coefficient in our analysis [10, 24]: given a group  $D_{g}$ , the disagreement coefficient on  $g$  is
$$
\theta_{g}:= \sup_{h\in \mathcal{H}}\sup_{r^{\prime}\geq 2\nu +\epsilon}\frac{\mathbb{P}_{D_{g}}\left(x\in\Delta(B_{g}(h,r^{\prime}))\right)}{r^{\prime}},
$$
where  $B_{g}(h,r^{\prime})\coloneqq \{h^{\prime}\in \mathcal{H}:\rho_{g}(h,h^{\prime})\leq r^{\prime}\}$  is a ball of radius  $r^\prime$  about  $h$  in pseudo-metric  $\rho_{g}$ . We further note the maximum disagreement coefficient over the groups  $\mathcal{G}$  as  $\theta_{\mathcal{G}}\coloneqq \max_g\theta_g$ .
The disagreement coefficient is trivially bounded above by  $1 / \epsilon$ , but can be bounded independently of  $\epsilon$  in many cases [10, 25]. For example, when  $\mathcal{H}$  is the class of linear separators in  $d$  dimensions and the underlying marginal distribution is uniform over the Euclidean unit sphere, the disagreement coefficient is  $\Theta (\sqrt{d})$  [9].
Theorem 1. For all  $\epsilon >0$ ,  $\delta \in (0,1)$ , collections of groups  $\mathcal{G}$ , and hypothesis classes  $\mathcal{H}$  with  $d < \infty$  with probability  $\geq 1 - \delta$ , the output  $\hat{h}$  of Algorithm 1 satisfies
$$
L _ {\mathcal {G}} ^ {\max} (\hat {h}) \leq L _ {\mathcal {G}} ^ {\max} (h ^ {*}) + \epsilon ,
$$
and its label complexity is bounded by
$$
\tilde {O} \left(G \theta_ {\mathcal {G}} ^ {2} \left(\frac {\nu^ {2}}{\epsilon^ {2}} + 1\right) \left(d \log (1 / \epsilon) + \log (1 / \delta)\right) \log (1 / \epsilon) + \frac {G \log (1 / \delta) \log (1 / \epsilon)}{\epsilon^ {2}}\right).
$$
Here, the  $\tilde{O}$  notation hides factors of  $\log (\log (1 / \epsilon))$  and  $\log (G)$ ; we leave all proofs for the Appendix.
Theorem 1 tells us that Algorithm 1 enjoys the following upside over passive and collaborative learning approaches: the dependence on the standard interaction of the VC-dimension  $d$  and  $1 / \epsilon^2$  is removed, and replaced with  $Gd\theta_{\mathcal{G}}^{2}\log^{2}(1 / \epsilon)\nu^{2} / \epsilon^{2}$ , which in settings with small disagreement coefficients and low noise rates, will be significant for small  $\epsilon$ .
5.3 Comparison to Lower Bounds in Collaborative Learning
We compare our label complexity guarantees to results in collaborative learning, where the learner has the power to ask for samples from specific group distributions, but not selectively label these samples. This is a strictly more demanding comparison than to pure passive settings, but a fair one, given that active learners have the option of executing any collaborative learning strategy. In collaborative learning, for finite hypothesis classes  $\mathcal{H}$ , it is known that
$$
\Omega \left(\frac {\log (| \mathcal {H} |)}{\epsilon^ {2}} + \frac {G \log (\min (| \mathcal {H} | , G) / \delta)}{\epsilon^ {2}}\right)
$$
total labels over all groups are necessary [3]. We consider comparing this lower bound to a simplified version of the label complexity guarantee in Theorem 1:
$$
\tilde {O} \left(d G \theta_ {\mathcal {G}} ^ {2} \log^ {2} (1 / \epsilon) + \frac {G \log (1 / \epsilon)}{\epsilon^ {2}}\right),
$$
thus implicitly assuming  $\nu$  is neglectably small, and making all comparisons up to factors logarithmic in  $G$ ,  $1 / \delta$  and  $\log (1 / \epsilon)$ . This former assumption is a standard assumption under which we may hope an agnostic active learner to succeed [15].
Algorithm 2 Group Realizable Algorithm
procedure GROUP_REALIZABLE  $(\mathcal{H},\epsilon ,\delta$  , active learner  $\mathcal{A}$ $\{U_g(\cdot)\}_{g = 1}^G,\{O_g(\cdot)\}_{g = 1}^G))$  for  $g\in [G]$  do  $\hat{h}_g\gets \mathcal{A}(\mathcal{H},\epsilon /6,\delta /2G,U_g(\mathcal{X}),O_g)$ $S_g^\prime \gets 144 / \epsilon^2 (2d\ln (24 / \epsilon) + \ln (8G / \delta))$  samples from oracle  $U_{g}(\mathcal{X})$ $\hat{S}_g\gets \{(x,\hat{h}_g(x)):x\in S_g'\}$  end for return  $\hat{h} = \arg \min_{h\in \mathcal{H}}\max_{g\in [G]}\frac{1}{|\hat{S}_g|}\sum_{(x,\hat{y})\in \hat{S}_g}\mathbb{1}[h(x)\neq \hat{y} ]$    
end procedure
Even the simplified upper bound does not admit the cleanest comparison this to lower bound, due to our excess factor of  $\log (1 / \epsilon)$  in the second term. However, it does showcase that while we pay slightly more per group than necessary, under conditions amenable to active learning, we pay significantly less per dimension  $d$ . Particularly for small  $\epsilon$ , one can see that's approximately sufficient that  $G < o(\theta_G^2 (\log (1 / \epsilon)\epsilon)^2)^{-1})$  for the simplified upper bound to beat the lower bound.
For a more fine-grained comparison that in some sense underestimates the power of Algorithm 1, assume that the following condition governs the relationship of  $G$ ,  $d$ , and  $\epsilon$ :
$$
G \log (1 / \epsilon) \leq d <   \left(\theta_ {\mathcal {G}} ^ {2} \epsilon^ {2} \log^ {2} (1 / \epsilon)\right) ^ {- 1}.
$$
Then the simplified bound is smaller in order than the lower bound above.
6 Group Realizable Learning
A special case of the learning problem, where extreme active learning gains can be readily seen, comes when the hypothesis class  $\mathcal{H}$  achieves zero noise rate on each group  $D_g$ . This setting has been considered in the passive "multi-group learning" literature [6]. Formally speaking, in the group realizable setting, the following condition holds:
$$
\forall g \in [ G ], \exists h _ {g} ^ {*} \in \mathcal {H} \text {s . t .} L _ {\mathcal {G}} \left(h _ {g} ^ {*} \mid g\right) = 0,
$$
i.e. for all groups in the collection  $\mathcal{G}$ , there is some hypothesis achieving 0 error on that group. Note that this differs from the fully realizable setting where there is some  $h^* \in \mathcal{H}$  with  $L_{\mathcal{G}}^{\max}(h^*) = 0$ . While fully realizable implies group realizable, the converse is not true. Thus, group realizability represents an intermediate regime between the realizable setting and the full agnostic settings.
6.1 Algorithm
In the group realizable case, it is possible to show a reduction of the problem of active learning over hypothesis classes with respect to a single distribution.
This can be accomplished as follows. For each  $D_g$ , we call as a subroutine an active learner that is guaranteed to find an order  $\epsilon$ -optimal hypothesis  $\hat{h}_g^*$  with high probability over its queries. It then gathers new unlabeled samples from each  $D_g$ , and instead of requesting labels from  $O_g(\cdot)$ , labels each unlabeled point with  $\hat{h}_g^*$ . The final step is to do an empirical risk minimization on these artificially labeled samples with respect to the multi-group objective. See Algorithm 2 for a formal specification of the strategy.
6.2 Guarantees
The strategy given in Algorithm 2 leads to a consistent active learning scheme, provided the active learners called as subroutines have standard guarantees that can be inherited.
Theorem 2 gives a guarantee to this end. The proof follows from an argument similar to one used in [10] - because the subroutine calls return hypotheses with near 0 error on each group, the artificially labeled training set used in the ERM step looks nearly identical to a counterfactual training set for the ERM step constructed by querying labels  $O_{g}(x)$  for each unlabeled  $x$ . This is similar to the idea in [24]. We present Theorem 2 assuming access to a classical, realizable active learner due to [26].
Theorem 2. Suppose Algorithm 2 is run with the active learner  $\mathcal{A}_{CAL}$  of [26]. Then for all  $\epsilon >0$ ,  $\delta \in (0,1)$ , hypothesis classes  $\mathcal{H}$  with  $d < \infty$ , and collections of groups  $\mathcal{G}$  with the group realizability property under  $\mathcal{H}$ , with probability  $\geq 1 - \delta$ , the output  $\hat{h}$  satisfies
$$
L _ {\mathcal {G}} ^ {\max } (\hat {h}) \leq L _ {\mathcal {G}} ^ {\max } (h ^ {*}) + \epsilon ,
$$
and the number of labels requested is
$$
\tilde {O} \left(d G \theta_ {\mathcal {G}} \log (1 / \epsilon)\right).
$$
Thus, when disagreement coefficients across the collection of groups are bounded independently of  $\epsilon$ , the usual, passive dependence on  $1 / \epsilon^2$  is replaced by  $\log (1 / \epsilon)$ .
In the passive multi-group setting of [7], it has been shown that  $\tilde{O} (\log (|\mathcal{H}|) / \gamma \epsilon)$  samples are sufficient for group realizable learning, where we recall  $\gamma$  is a lower bound on the probability of getting a sample in each group [6].
7 Full Agnostic Approximation
7.1 Inconsistency of the Reduction in the Full Agnostic Regime
Algorithm 2 admits clean analysis, and nicely harnesses the power of realizable active learners for a single distribution. One might wonder if a similar strategy might provide a consistent strategy in full agnostic regime. Unfortunately, the direct application of Algorithm 2 using agnostic learners does not yield a consistent active learning algorithm. In fact, consistency fails even when for each  $g \in [G]$ ,  $h_g^*$  is the Bayes optimal classifier on  $D_g$ , and  $\nu_g := \inf_{h \in \mathcal{H}} L_{\mathcal{G}}(h \mid g)$  is small. This lack of consistency comes down to the fact that labeling with the Bayes optimal underestimates noise rates on each group, which in turn may bias the output of the ERM step.
7.2 A  $3\nu$ -Approximation Algorithm
Although the strategy of creating an artificially labeled training set with near-optimal hypotheses on each group fails outside of the group realizable case, it possesses a nice approximation property.
We give a guarantee to this end in Theorem 3. It states that if we call an active learner with agnostic guarantees on each group  $D_g$ , and then use the outputs  $\hat{h}_g^*$  to artificially label a new batch of unlabeled data from each group, using ERM on this artificially labeled data gives at worst a  $2\nu + \epsilon$ -optimal hypothesis with high probability.
Theorem 3. Suppose Algorithm 2 is run with the agnostic active learner  $\mathcal{A}_{DHM}$  of [15]. Then for all  $\epsilon >0$ ,  $\delta \in (0,1)$ , hypothesis classes  $\mathcal{H}$  with  $d < \infty$ , and collections of groups  $\mathcal{G}$ , with probability  $\geq 1 - \delta$ , the output  $\hat{h}$  satisfies
$$
L _ {\mathcal {G}} ^ {\max} (\hat {h}) \leq L _ {\mathcal {G}} ^ {\max} (h ^ {*}) + 2 \cdot \max _ {g \in [ G ]} \nu_ {g} + \epsilon \leq 3 \cdot L _ {\mathcal {G}} ^ {\max} (h ^ {*}) + \epsilon ,
$$
$$
\tilde {O} \left(d G \theta_ {\mathcal {G}} \left(\log^ {2} (1 / \epsilon) + \frac {\nu^ {2}}{\epsilon^ {2}}\right)\right).
$$
The proof is very similar to that of Theorem 2, but notes in addition that  $\hat{h}_g^*$  mislabels on a roughly  $\nu_g$ -fraction of the unlabeled samples from each group  $G$ . This allows us to upper bound the distortion of the ERM step.
In this work, we have taken a first look at active multi-group learning. Though the design of general agnostic strategies in this setting is quite challenging, an interesting future direction may be the search for strategies that work in more specific cases, for example extending our work in the group realizable setting. In particular, the search for algorithms with small label complexities under specific low-noise conditions, such as Tsybakov noise on each  $D_g$ , may prove fruitful [27].
[25] Steve Hanneke. Theory of active learning. 2014.
[26] David A. Cohn, Les E. Atlas, and Richard E. Ladner. Improving generalization with active learning. Machine Learning, 15:201-221, 1994.
[27] Jean-Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers under the margin condition. arXiv preprint math/0507180, 2005.
[28] Vladimir Vapnik. Statistical Learning Theory. Wiley, 1998.
[29] Wassily Hoeffding. Probability Inequalities for sums of Bounded Random Variables. Springer New York, 1994.
9 Appendix
9.1 Guarantees for General Agnostic Algorithm
In this section, we give proofs for the guarantees of Algorithm 1. We begin with some definitions, starting with how empirical loss estimates are made.
Definition 1. Given a hypothesis  $h \in \mathcal{H}$ , and a set of pairs  $S = \{(x_i, y_i) : x_i \in \mathcal{X}, y_i \in \mathcal{Y}\}_{i=1}^N$ , let
$$
L _ {\mathcal {S}} (h) := \frac {1}{N} \left(\sum_ {i = 1} ^ {N} \mathbb {1} [ h (x _ {i}) \neq y _ {i} ]\right)
$$
the standard empirical loss of  $h$  on  $\mathcal{S}$ . Let  $L_{\emptyset}(h) \coloneqq 1$ .
The convention to let  $L_{\emptyset}(h) = 1$  allows us to "collapse" the two-part loss estimates in the case the probability of drawing an unlabeled sample in a specific region is 0; under the specification of the algorithm,  $S = \emptyset$  if and only if the probability of a sample falling in the disagreement region or its complement is 0 under  $D_g$ , in which case we can safely ignore estimation in one of these regions.
Definition 2. Given a set of classifiers  $\mathcal{H}' \subseteq \mathcal{H}$ , we say "  $\mathcal{H}'$  agrees on a subset  $S \subseteq \mathcal{X}$ " if for each  $x \in S$  and for each pair  $(h, h') \in \mathcal{H}' \times \mathcal{H}'$ , it holds that  $h(x) = h'(x)$ .
We now recall the two-part estimator for the loss of a hypothesis introduced above.
Definition 3. Fix a group distribution  $D_g$ , some  $\mathcal{H}' \subseteq \mathcal{H}$ , a hypothesis  $h \in \mathcal{H}'$ , and some  $R \subseteq \mathcal{X}$  which is measurable with respect to each marginal of  $D_g$  and for which  $\mathcal{H}'$  agrees on  $R^c$ . Given sets of pairs  $S_{R,g}$  and  $S_{R^{c},g}$ , and some arbitrarily chosen classifier  $h_{\mathcal{H}'} \in \mathcal{H}'$ , let
$$
L _ {\mathcal {S}; R} (h \mid g) := \mathbb {P} _ {D _ {g}} (x \in R) \cdot L _ {\mathcal {S} _ {R, g}} (h) + \mathbb {P} _ {D _ {g}} (x \in R ^ {c}) \cdot L _ {\mathcal {S} _ {R} ^ {c, g}} \left(h _ {\mathcal {H} ^ {\prime}}\right).
$$
As mentioned in the main body,  $h_{\mathcal{H}'}$  must be used in the estimate of the loss under  $D_g$  in the "agreement region" for all  $h \in \mathcal{H}$ . The extent to which this estimator is useful can be captured by standard uniform convergence arguments. To this end, we first introduce a function that will prove to control its deviations nicely.
Definition 4. Given a confidence parameter  $\delta \in (0,1)$ , a group distribution  $D_g \in \mathcal{G}$ , some  $R \subseteq \mathcal{X}$  that is measurable with respect to each marginal  $D_g \in \mathcal{G}$ , and sample sizes  $m, m' > 0$ , define the function
$$
\Gamma_ {g} (\delta , R, m, m ^ {\prime}) := \left\{ \begin{array}{l l} & \mathbb {P} _ {D _ {g}} (x \in R) \bigg (\frac {1}{m} + \sqrt {\frac {\ln (8 / \delta) + d \ln (2 e m / d)}{m}} \bigg) + \sqrt {\frac {\ln (4 / \delta)}{2 m ^ {\prime}}} \\ & \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad i f   \mathbb {P} _ {D _ {g}} (x \in R) > 0,   \mathbb {P} _ {D _ {g}} (x \in R ^ {c}) > 0 \\ & \frac {1}{m} + \sqrt {\frac {\ln (8 / \delta) + d \ln (2 e m / d)}{m}} \\ & \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad i f   \mathbb {P} _ {D _ {g}} (x   \in   R) > 0,   \mathbb {P} _ {D _ {g}} (x   \in   R ^ {c}) = 0 \\ & \sqrt {\frac {\ln (4 / \delta)}{2 m ^ {\prime}}} \\ & \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad i f   \mathbb {P} _ {D _ {g}} (x   \in   R) = 0,   \mathbb {P} _ {D _ {g}} (x   \in   R ^ {c}) > 0. \end{array} \right.
$$
Lemma 1. Fix  $\delta \in (0,1)$ , a set of group distributions  $\mathcal{G}$ , and a group distribution  $D_g \in \mathcal{G}$  arbitrarily. Further, fix a subset  $R \subseteq \mathcal{X}$  measurable with respect to each marginal of  $D_g \in \mathcal{G}$ , and a set of classifiers  $\mathcal{H}' \subseteq \mathcal{H}$  with the property that  $\mathcal{H}'$  agree on  $R^c$ . Suppose we query  $m > 0$  unlabeled samples from  $U_g(R)$ , and  $m' > 0$  samples from  $U_g(R^c)$ . Suppose further that we label the output via calls to  $O_g(\cdot)$ , forming the labeled samples  $S_{R,g}$  and  $S_{R^{c},g}$ , respectively; if either  $\mathbb{P}_{D_g}(x \in R) = 0$  or  $\mathbb{P}_{D_g}(x \in R^c)$ , then we set the corresponding sample to be  $\emptyset$ . Then with probability  $\geq 1 - \delta$ , it holds for all  $h \in \mathcal{H}'$  that
$$
\left| L _ {\mathcal {G}} (h \mid g) - L _ {\mathcal {S}; R} (h \mid g) \right| \leq \Gamma_ {g} (\delta , R, m, m ^ {\prime}).
$$
Further, for all  $\gamma > 0$ , if  $m \geq \frac{16(\mathbb{P}_{D_g}(x \in R))^2}{\gamma^2}$  ( $2d\ln(8/\gamma) + \ln(8/\delta)$ ) and  $m' \geq \frac{2\ln(4/\delta)}{\gamma^2}$ , then  $\Gamma_g(\delta, R, m, m') < \gamma$ .
Proof. We begin with the case where both  $\mathbb{P}_{D_g}$ $(x\in R)\neq 0$  and  $\mathbb{P}_{D_g}$ $(x\in R^c)\neq 0$ . In this case, we are able to draw unlabeled samples from both regions, and neither  $S_{R,g}$  nor  $S_{R,g}$  is  $\emptyset$ .
By a lemma of Vapnik [28], we have that with probability  $\geq 1 - \delta /2$  over the draw of  $m$  samples from  $U_{g}(R)$  and their labeling via  $O_{g}(\cdot)$ , that simultaneously for each  $h\in \mathcal{H}'$ :
$$
\left| L _ {\mathcal {S} _ {R, g}} (h) - \mathbb {P} _ {D _ {g}} \left(h (x) \neq y | x \in R\right) \right| \leq \frac {1}{m} + \sqrt {\frac {\ln (8 / \delta) + d \ln (2 e m / d)}{m}}.
$$
In  $R^c$ , all  $h \in \mathcal{H}'$  agree, and so estimating the conditional loss for each  $h \in \mathcal{H}'$  in this region is as statistically hard as estimating a single Bernoulli parameter, which we do by arbitrarily choosing a classifier to use for the loss estimate in this part of space. Thus, by definition of the two-part estimator and Hoeffding's inequality [29], we have with probability  $\geq 1 - \delta / 2$  for all  $h \in \mathcal{H}'$  simultaneously
$$
\left| L _ {\mathcal {S} _ {R ^ {c}, g}} \left(h _ {\mathcal {H} ^ {\prime}}\right) - \mathbb {P} _ {D _ {g}} \left(h (x) \neq y \mid x \in R ^ {c}\right) \right| \leq \sqrt {\frac {\ln (4 / \delta)}{2 m ^ {\prime}}}.
$$
By a union bound, with probability  $\geq 1 - \delta$ , both of these events take place, and so for all  $h \in \mathcal{H}'$  simultaneously,
$$
\begin{array}{l} L _ {\mathcal {G}} (h \mid g) = \mathbb {P} _ {D _ {g}} (h (x) \neq y \mid x \in R) \cdot \mathbb {P} _ {D _ {g}} (x \in R) \\ + \mathbb {P} _ {D _ {g}} (h (x) \neq y \mid x \in R ^ {c}) \cdot \mathbb {P} _ {D _ {g}} (x \in R ^ {c}) \\ \leq \left(L _ {\mathcal {S} _ {R, g}} (h) + 1 / m + \sqrt {(\ln (8 / \delta) + d \ln (2 e m / d)) / m}\right) \cdot \mathbb {P} _ {D _ {g}} (x \in R) \\ + \left(L _ {\mathcal {S} _ {R ^ {c}, g}} \left(h _ {\mathcal {H} ^ {\prime}}\right) + \sqrt {\ln (4 / \delta) / 2 m ^ {\prime}}\right) \cdot \mathbb {P} _ {D _ {g}} \left(x \in R ^ {c}\right) \\ \leq L _ {\mathcal {S}; R} (h \mid g) + \Gamma_ {g} (\delta , R, m, m ^ {\prime}). \\ \end{array}
$$
The lower bound leading to the absolute value is analogous. Vapnik [28] also tells us that for any  $\gamma' > 0$ , a sample of size  $m \geq \frac{4}{\gamma'^2}$  ( $2d\ln(4/\gamma') + \ln(8/\delta)$ ) is sufficient to yield
$$
1 / m + \sqrt {(\ln (8 / \delta) + d \ln (2 e m / d)) / m} <   \gamma^ {\prime}.
$$
Let  $\gamma' = \gamma / 2\mathbb{P}_{D_g}(x \in R)$ . Thus, substituting for  $\gamma'$  and bounding the probability inside the natural log above by 1,
$$
m \geq \mathbb {P} _ {D _ {g}} (x \in R) ^ {2} \frac {1 6}{\gamma^ {2}} (2 d \ln (8 / \gamma) + \ln (8 / \delta))
$$
implies that
$$
\frac {1}{m} + \sqrt {\frac {\ln (8 / \delta) + d \ln (2 e m / d)}{m}} <   \frac {\gamma}{2 \mathbb {P} _ {D _ {g}} (x \in R)}.
$$
As a corollary to Hoeffding, if  $m' \geq 2\ln(4/\delta)/\gamma^2$ , then  $\sqrt{\log(4/\delta)/2m'} < \gamma/2$ . Thus, we may write
$$
\Gamma_ {g} (\delta , R, m, m ^ {\prime}) = \mathbb {P} _ {D _ {g}} (x \in R) \left(\frac {1}{m} + \sqrt {\frac {\ln (8 / \delta) + d \ln (2 e m / d)}{m}}\right) + \sqrt {\frac {\ln (4 / \delta)}{2 m ^ {\prime}}} <   \gamma / 2 + \gamma / 2 = \gamma .
$$
Now suppose that  $\mathbb{P}_{D_g}(x\in R^c) = 0$ . In this case, we have  $S_{R^c,g} = \emptyset$ . Again, we have that with probability  $\geq 1 - \delta /2$
$$
\left| L _ {\mathcal {S} _ {R, g}} (h) - \mathbb {P} _ {D _ {g}} \left(h (x) \neq y \mid x \in R\right) \right| \leq \frac {1}{m} + \sqrt {\frac {\ln (8 / \delta) + d \ln (2 e m / d)}{m}}.
$$
When  $\mathbb{P}_{D_g}(x\in R^c) = 0$  , it holds that  $\mathbb{P}_{D_g}(x\in R) = 1$  , and so
$$
\begin{array}{l} L _ {\mathcal {G}} (h \mid g) = \mathbb {P} _ {D _ {g}} (h (x) \neq y \mid x \in R) \cdot \mathbb {P} _ {D _ {g}} (x \in R) \\ + \mathbb {P} _ {D _ {g}} (h (x) \neq y \mid x \in R ^ {c}) \cdot \mathbb {P} _ {D _ {g}} (x \in R ^ {c}) \\ = \mathbb {P} _ {D _ {g}} (h (x) \neq y \mid x \in R) \\ \leq L _ {\mathcal {S} _ {R, g}} (h) + 1 / m + \sqrt {(\ln (8 / \delta) + d \ln (2 e m / d)) / m} \\ = L _ {\mathcal {S}; R} (h \mid g) + \Gamma_ {g} (\delta , R, m, m ^ {\prime}), \\ \end{array}
$$