SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the final equality comes from fact that  $\mathbb{P}_{D_g}(x\in R^c) = 0$  and  $\mathbb{P}_{D_g}(x\in R) = 1$ , as well as the definitions of  $L_{\mathcal{S};R}(h\mid g)$  and  $\Gamma_g(\delta ,R,m,m')$ . Similarly to the above, if we let  $\gamma^{\prime} = \gamma /2\mathbb{P}_{D_g}(x\in R) = \gamma /2$ , then
$$
m \geq \frac {1 6}{\gamma^ {2}} (2 d \ln (8 / \gamma) + \ln (8 / \delta))
$$
$$
\frac {1}{m} + \sqrt {\frac {\ln (8 / \delta) + d \ln (2 e m / d)}{m}} <   \frac {\gamma}{2},
$$
which by the definition of  $\Gamma_g(\delta, R, m, m')$  when  $\mathbb{P}_{D_g}(x \in R^c) = 0$  gives us  $\Gamma_g(\delta, R, m, m') < \gamma / 2 < \gamma$ . The case where  $\mathbb{P}_{D_g}(x \in R) = 0$  follows the previous argument for when  $\mathbb{P}_{D_g}(x \in R^c) = 0$ .
Definition 5. Given a collection of group distributions  $\mathcal{G}$ , some  $\mathcal{H}' \subseteq \mathcal{H}$ , a hypothesis  $h \in \mathcal{H}'$ , some subset  $R \subseteq \mathcal{X}$  measurable with respect to each marginal of  $D_g \in \mathcal{G}$ , and labeled samples  $\mathcal{S}_{R,k}$  and  $\mathcal{S}_{R^c,k}$ , we define the empirical estimate of the multi-group loss of  $h$  parameterized by  $R$  via
$$
L _ {\mathcal {S}; R} ^ {\max } (h) := \max  _ {g \in [ G ]} L _ {\mathcal {S}; R} (h \mid g).
$$
Having recalled the way in which we form empirical estimates for the group worst-case loss of a given hypothesis, we can show a simple concentration lemma for this group worst-case loss estimator using the concentration property for individual groups proved in Lemma 1,
Lemma 2. Fix  $\delta \in (0,1)$ , a set of group distributions  $\mathcal{G}$ , a subset  $R \subseteq \mathcal{X}$  measurable with respect to each marginal of  $D_g \in \mathcal{G}$ , and a set of classifiers  $\mathcal{H}' \subseteq \mathcal{H}$  that agree on  $R^c$ . Suppose for each  $g \in [G]$ , we query  $m_g > 0$  unlabeled samples from  $U_g(R)$ , and  $m_g' > 0$  samples from  $U_g(R^c)$ . Suppose further that we label the outputs via calls to  $O_g(\cdot)$ , forming the labeled samples  $\mathcal{S}_{R,g}$  and  $\mathcal{S}_{R^c,g}$ , respectively, for each  $g \in [G]$ ; if  $\mathbb{P}_{D_g}(x \in R) = 0$  or  $\mathbb{P}_{D_g}(x \in R^c) = 0$ , then we set the corresponding sample to be  $\emptyset$ . Then with probability  $\geq 1 - \delta$ , it holds for all  $h \in \mathcal{H}'$  that
$$
\left| L _ {\mathcal {G}} ^ {\max } (h) - L _ {\mathcal {S}; R} ^ {\max } (h) \right| \leq \max  _ {g ^ {\prime} \in [ G ]} \Gamma_ {g ^ {\prime}} (\delta / G, m _ {g ^ {\prime}}, m _ {g ^ {\prime}} ^ {\prime}).
$$
Proof. By Lemma 1 and a union bound, it holds with probability  $\geq 1 - \delta$  that on all  $D_g$ , for all  $h \in \mathcal{H}'$  simultaneously, that
$$
\left| L _ {\mathcal {G}} (h \mid g) - L _ {\mathcal {S}; R} (h \mid g) \right| \leq \Gamma_ {g} (\delta / G, m _ {g}, m _ {g} ^ {\prime}).
$$
Thus we may write
$$
\begin{array}{l} \left| L _ {\mathcal {G}} ^ {\max } (h) - L _ {\mathcal {S}; R} ^ {\max } (h) \right| = \left| \max  _ {g ^ {\prime} \in [ G ]} L _ {\mathcal {G}} (h \mid g ^ {\prime}) - \max  _ {g ^ {\prime} \in [ G ]} L _ {\mathcal {S}; R} (h \mid g) \right| \\ \leq \max  _ {g ^ {\prime} \in [ G ]} \left| L _ {\mathcal {G}} \left(h \mid g ^ {\prime}\right) - L _ {\mathcal {S}; R} \left(h \mid g ^ {\prime}\right) \right| \\ \leq \max  _ {g ^ {\prime} \in [ G ]} \Gamma_ {g ^ {\prime}} (\delta / G, m _ {g ^ {\prime}}, m _ {g ^ {\prime}} ^ {\prime}). \\ \end{array}
$$
We now use Lemma 2 to show that Algorithm 1 is conservative enough that the optimal hypothesis  $h^*$  is never eliminated from contention throughout the run of the algorithm with high probability.
Lemma 3. Fix  $\delta \in (0,1)$ , a collection of group distributions  $\mathcal{G}$ , and a hypothesis class  $\mathcal{H}$  with  $d < \infty$  arbitrarily. With probability  $\geq 1 - \delta$ , it holds after each iteration  $i$  of Algorithm 1 that  $h^* \in \mathcal{H}_{i+1}$ .
Proof. By Lemmas 1 and 2, and a union bound over iterations, the number of samples labeled at each iteration is sufficient for us to conclude that with probability  $\geq 1 - \delta$ , for every iteration  $i$
and for each  $h\in \mathcal{H}_i$  , it holds that
$$
\left| L _ {\mathcal {S}; R _ {i}} ^ {\max } (h) - L _ {\mathcal {G}} ^ {\max } (h) \right| \leq 2 ^ {I - i} \epsilon / 8.
$$
We give an inductive argument conditioned on this high probability event. When  $i = 1$ , we have  $h^{*} \in \mathcal{H}_{1}$  because  $\mathcal{H}_{1} = \mathcal{H}$ , and  $h^{*} \in \mathcal{H}$  by definition. If  $h^{*} \in \mathcal{H}_{i}$  for  $i \geq 1$ , then  $h^{*} \in \mathcal{H}_{i + 1}$  if and only if
$$
L _ {\mathcal {S}; R _ {i}} ^ {\max } \left(h ^ {*}\right) \leq L _ {\mathcal {S}; R _ {i}} ^ {\max } \left(\hat {h} _ {i}\right) + 2 ^ {I - i} \epsilon / 4.
$$
When for each  $h\in \mathcal{H}_i$  , it holds that  $|L_{\mathcal{S};R_i}^{\max}(h) - L_{\mathcal{G}}^{\max}(h)|\leq 2^{I - i}\epsilon /8$  , we may write
$$
\begin{array}{l} L _ {\mathcal {S}; R _ {i}} ^ {\max} (h ^ {*}) - L _ {\mathcal {S}; R _ {i}} ^ {\max} (\hat {h} _ {i}) \leq L _ {\mathcal {S}; R _ {i}} ^ {\max} (h ^ {*}) - L _ {\mathcal {G}} ^ {\max} (h ^ {*}) + L _ {\mathcal {G}} ^ {\max} (\hat {h} _ {i}) - L _ {\mathcal {S}; R _ {i}} ^ {\max} (\hat {h} _ {i}) \\ \leq \left| L _ {\mathcal {S}; R _ {i}} ^ {\max } \left(h ^ {*}\right) - L _ {\mathcal {G}} ^ {\max } \left(h ^ {*}\right) \right| + \left| L _ {\mathcal {G}} ^ {\max } \left(\hat {h} _ {i}\right) - L _ {\mathcal {S}; R _ {i}} ^ {\max } \left(\hat {h} _ {i}\right) \right| \\ \leq 2 ^ {I - i} \epsilon / 8 + 2 ^ {I - i} \epsilon / 8 \\ = 2 ^ {I - i} \epsilon / 4, \\ \end{array}
$$
where the first inequality comes from the optimality of  $h^*$ . Thus, we must have  $h \in \mathcal{H}_{i+1}$ .
Now, using the fact that the optimal hypothesis stays in contention throughout the run of the algorithm, we can give a guarantee on the true error of each hypothesis  $h \in \mathcal{H}_{i+1}$ . The idea is that using concentration and the small empirical error of each  $h \in \mathcal{H}_{i+1}$ , we can say that the true errors of each  $h \in \mathcal{H}_{i+1}$  are similar to the true errors of the ERM hypothesis  $\hat{h}_i$ , and then use the true error of  $\hat{h}_i$  as a reference point to which we can compare the true error of  $h \in \mathcal{H}_{i+1}$  and  $h^*$ .
Lemma 4. Fix  $\delta \in (0,1)$ , a collection of group distributions  $\mathcal{G}$ , and a hypothesis class  $\mathcal{H}$  with  $d < \infty$  arbitrarily. Then with probability  $\geq 1 - \delta$ , after every iteration  $i$  of Algorithm 1, it holds for all  $h \in \mathcal{H}_{i+1}$  that
$$
\left| L _ {\mathcal {G}} ^ {\max } (h) - L _ {\mathcal {G}} ^ {\max } \left(h ^ {*}\right) \right| \leq 2 ^ {I - i} \epsilon .
$$
Proof. If  $h \in \mathcal{H}_{i+1}$ , then by the specification of the algorithm, it holds that
$$
L _ {\mathcal {S}; R _ {i}} ^ {\max } (h) - L _ {\mathcal {S}; R _ {i}} ^ {\max } (\hat {h} _ {i}) \leq 2 ^ {I - i} \epsilon / 4.
$$
Because  $\hat{h}_i$  is the ERM hypothesis at iteration  $i$ , it holds that  $L_{\mathcal{S};R_i}^{\max}(\hat{h}_i) - L_{\mathcal{S};R_i}^{\max}(h) \leq 0 < 2^{I - i}\epsilon /4$ , and thus we may conclude
$$
\left| L _ {\mathcal {S}; R _ {i}} ^ {\max } (h) - L _ {\mathcal {S}; R _ {i}} ^ {\max } (\hat {h} _ {i}) \right| \leq 2 ^ {I - i} \epsilon / 4.
$$
By Lemma 2 and the number of samples labeled at each iteration, with probability  $\geq 1 - \delta$ , it holds for all iterations and for all  $h \in \mathcal{H}_i$  that
Conditioned on this event, if  $h\in \mathcal{H}_{i + 1}$  , we have
$$
\begin{array}{l} \left| L _ {\mathcal {G}} ^ {\max } (h) - L _ {\mathcal {G}} ^ {\max } (\hat {h} _ {i}) \right| = \left| L _ {\mathcal {G}} ^ {\max } (h) - L _ {\mathcal {S}; R _ {i}} ^ {\max } (h) + L _ {\mathcal {S}; R _ {i}} ^ {\max } (h) - L _ {\mathcal {S}; R _ {i}} ^ {\max } (\hat {h} _ {i}) + L _ {\mathcal {S}; R _ {i}} ^ {\max } (\hat {h} _ {i}) - L _ {\mathcal {G}} ^ {\max } (\hat {h} _ {i}) \right| \\ \leq \left| L _ {\mathcal {G}} ^ {\max } (h) - L _ {\mathcal {S}; R _ {i}} ^ {\max } (h) \right| + \left| L _ {\mathcal {S}; R _ {i}} ^ {\max } (h) - L _ {\mathcal {S}; R _ {i}} ^ {\max } (\hat {h} _ {i}) \right| + \left| L _ {\mathcal {S}; R _ {i}} ^ {\max } (\hat {h} _ {i}) - L _ {\mathcal {G}} ^ {\max } (\hat {h} _ {i}) \right| \\ \leq 2 ^ {I - i} \epsilon / 8 + 2 ^ {I - i} \epsilon / 4 + 2 ^ {I - i} \epsilon / 8 \\ = 2 ^ {I - i} \epsilon / 2. \\ \end{array}
$$
By Lemma 3, it holds that  $h^* \in \mathcal{H}_{i+1}$  whenever  $\left| L_{\mathcal{S};R_i}^{\max}(h) - L_{\mathcal{G}}^{\max}(h) \right| \leq 2^{I - i}\epsilon / 8$  for all  $h \in \mathcal{H}_i$  at all iterations. Thus, this bound on the true error difference with the ERM  $\hat{h}_i$  applies to  $h^*$ , and we may write for arbitrary  $h \in \mathcal{H}_{i+1}$  that
$$
\left| L _ {\mathcal {G}} ^ {\max} (h) - L _ {\mathcal {G}} ^ {\max} (h ^ {*}) \right| \leq \left| L _ {\mathcal {G}} ^ {\max} (h) - L _ {\mathcal {G}} ^ {\max} (\hat {h} _ {i}) \right| + \left| L _ {\mathcal {G}} ^ {\max} (\hat {h} _ {i}) - L _ {\mathcal {G}} ^ {\max} (h ^ {*}) \right| \leq 2 ^ {I - i} \epsilon ,
$$
which is the desired result.
2We do not directly apply Lemma 1 with  $\gamma = \epsilon 2^{I - i} / 8$  here. We use this quantity in the outer dependence on  $\gamma$  of Lemma 1, but for the natural log dependence on  $\gamma$ , we sub in  $\epsilon /8$  to simplify the analysis. Thus we take slightly more samples than Lemma 1 directly suggests. Because we take the largest probability of the disagreement region over groups as  $m_{i}$ , it holds that  $m_{g}$  is at the smallest the sample size suggested by Lemma 1 for each  $g$ .
Definition 6. Given a group distribution  $D_g \in \mathcal{G}$ , a hypothesis  $h \in \mathcal{H}$ , and a radius  $r \geq 0$ , let the "  $D_g$  - disagreement ball in  $\mathcal{H}$  of radius  $r$  about  $h$ " be
$$
B _ {g} (h, r) := \left\{h ^ {\prime} \in \mathcal {H}: \rho_ {g} \left(h, h ^ {\prime}\right) \leq r \right\},
$$
where  $\rho_g(h,h')\coloneqq \mathbb{P}_{D_g}\left(h(x)\neq h'(x)\right)$
Definition 7. Given a group distribution  $D_g \in \mathcal{G}$  and a hypothesis class  $\mathcal{H}$ , let the "disagreement coefficient" of  $D_g$  be defined as
$$
\theta_ {g} := \sup  _ {h \in \mathcal {H}} \sup  _ {r ^ {\prime} \geq 2 \nu + \epsilon} \frac {\mathbb {P} _ {D _ {g}} (x \in \Delta (B _ {g} (h , r ^ {\prime})))}{r ^ {\prime}}.
$$
We further define the disagreement coefficient over a collection of group distributions  $\mathcal{G}$  as
$$
\theta_ {\mathcal {G}} := \max  _ {g ^ {\prime} \in [ G ]} \theta_ {g ^ {\prime}}.
$$
Given these definitions, we are now ready to state the main theorem. The consistency comes from what we showed in Lemma 4: as the true error for each  $h \in \mathcal{H}_{i+1}$  decreases with each iteration, after enough iterations we will have each  $h \in \mathcal{H}_{i+1}$  having  $\epsilon$ -optimality.
The label complexity bound follows standard ideas in the DBAL literature; see for example [9, 24]. Essentially, what we do is show that at each iteration  $i$ , because the true error of any  $h \in \mathcal{H}_i$  on the multi-group objective can't be too large, the disagreement of  $h$  and  $h^*$  on any single group cannot be too large. This leads to a bound on the size of the disagreement region for each  $g$ .
Theorem 1. For all  $\epsilon >0$ ,  $\delta \in (0,1)$ , collections of group distributions  $\mathcal{G}$ , and hypothesis classes  $\mathcal{H}$  with  $d < \infty$ , with probability  $\geq 1 - \delta$ , the output  $\hat{h}$  of Algorithm 1 satisfies
$$
\tilde {O} \left(G \theta_ {\mathcal {G}} ^ {2} \left(\frac {\nu^ {2}}{\epsilon^ {2}} + 1\right) \left(d \log (1 / \epsilon) + \log (1 / \delta)\right) \log (1 / \epsilon) + \frac {G \log (1 / \epsilon) \log (1 / \delta)}{\epsilon^ {2}}\right).
$$
Proof. Lemma 4 says that the number of samples drawn at each iteration is sufficiently large that with probability  $\geq 1 - \delta$ , for all  $i \in [I]$ , it holds that for all  $h \in \mathcal{H}_{i+1}$ , that we have  $\left|L_{\mathcal{G}}^{\max}(h) - L_{\mathcal{G}}^{\max}(h^{*})\right| \leq 2^{I - i} \epsilon$ . Thus, after  $I = \lceil \log(1 / \epsilon) \rceil$  iterations, the output  $\hat{h}$  satisfies the consistency condition.
To see the label complexity, which is the sum of the number of labels we query at each iteration, we note at iteration  $i$ , we label no more than
$$
1 0 2 4 \left(\frac {m _ {i}}{\epsilon 2 ^ {I - i}}\right) ^ {2} \left(2 d \log \left(\frac {6 4}{\epsilon}\right) + \ln \left(\frac {8 G \lceil \log (1 / \epsilon) \rceil}{\delta}\right)\right) + \frac {1 2 8 \ln (4 G \lceil \log (1 / \epsilon) \rceil / \delta)}{\epsilon^ {2}}
$$
samples for each group distribution  $D_g$ , where  $m_i = \max_{g'} \mathbb{P}_{D_{g'}}(x \in \Delta(\mathcal{H}_i))$ . The only term here that depends on  $i$  is  $\frac{m_i}{\epsilon 2^{I - i}}$ . By Lemma 4, with probability  $\geq 1 - \delta$ , it holds for each  $i > 1$  that  $\left| L_{\mathcal{G}}^{\max}(h) - L_{\mathcal{G}}^{\max}(h^*) \right| \leq 2^{I - i + 1} \epsilon$ ; this holds automatically at  $i = 1$  by the setting of  $I = \lceil \log(1 / \epsilon) \rceil$ . Thus, at arbitrary  $i$  and for arbitrary  $g \in [G]$ , we may write
$$
\begin{array}{l} \rho_ {g} (h, h ^ {*}) = \mathbb {P} _ {D _ {g}} (h (x) \neq h ^ {*} (x)) \\ = \mathbb {P} _ {D _ {g}} (h (x) \neq y, h ^ {*} (x) = y) + \mathbb {P} _ {D _ {g}} (h (x) = y, h ^ {*} (x) \neq y) \\ \leq \mathbb {P} _ {D _ {g}} (h (x) \neq y) + \mathbb {P} _ {D _ {g}} \left(h ^ {*} (x) \neq y\right) \\ = L _ {\mathcal {G}} (h \mid g) + L _ {\mathcal {G}} \left(h ^ {*} \mid g\right) \\ \leq L _ {\mathcal {G}} ^ {\max } (h) + L _ {\mathcal {G}} ^ {\max } \left(h ^ {*}\right) \\ = L _ {\mathcal {G}} ^ {\max } (h) - L _ {\mathcal {G}} ^ {\max } \left(h ^ {*}\right) + L _ {\mathcal {G}} ^ {\max } \left(h ^ {*}\right) + L _ {\mathcal {G}} ^ {\max } \left(h ^ {*}\right) \\ \leq 2 ^ {I - i + 1} \epsilon + 2 \nu , \\ \end{array}
$$
where we recall  $\nu$  is the noise rate on the multi-group objective. Thus, with probability  $\geq 1 - \delta$ , for each  $i \in I$  and  $g \in [G]$ , it holds that
$$
\mathcal {H} _ {i} \subseteq B _ {g} \left(h ^ {*}, 2 ^ {I - i + 1} \epsilon + 2 \nu\right).
$$
Given this observation, we may then write, for all  $g$ , that
$$
\mathbb {P} _ {D _ {g}} (x \in \Delta (\mathcal {H} _ {i})) \leq \mathbb {P} _ {D _ {g}} (x \in \Delta (B _ {g} (h ^ {*}, 2 \nu + 2 ^ {I - i + 1} \epsilon))),
$$
as if there are  $h, h' \in \mathcal{H}_i$  that disagree on some  $x$ , we have  $h, h' \in B_g(h^*, 2\nu + 2^{I - i + 1}\epsilon)$ , and so  $h, h'$  also realize disagreement on  $x$  for the larger set of classifiers. Recalling the definition of  $m_i$ , this allows us to bound the sum of terms depending on  $i$  for each distribution  $D_g$  as
$$
\begin{array}{l} \sum_ {i = 1} ^ {I} \left(\frac {m _ {i}}{\epsilon 2 ^ {I - i}}\right) ^ {2} \leq \sum_ {i = 1} ^ {I} \left(\frac {\max _ {g ^ {\prime}} \mathbb {P} _ {D _ {g}} \left(x \in \Delta \left(B _ {g ^ {\prime}} \left(h ^ {*} , 2 \nu + 2 ^ {I - i + 1} \epsilon\right)\right)\right)}{2 ^ {I - i} \epsilon}\right) ^ {2} \\ \leq \sum_ {i = 1} ^ {I} \left(\max _ {g ^ {\prime}} \frac {\mathbb {P} _ {D _ {g}} \left(x \in \Delta \left(B _ {g ^ {\prime}} \left(h ^ {*} , 2 \nu + 2 ^ {I - i + 1} \epsilon\right)\right)\right)}{2 \nu + 2 ^ {I - i + 1} \epsilon} \cdot \frac {2 \nu + 2 ^ {I - i + 1} \epsilon}{2 ^ {I - i} \epsilon}\right) ^ {2} \\ \leq 4 \left(\frac {\nu + \epsilon}{\epsilon}\right) ^ {2} \sum_ {i = 1} ^ {I} \left(\max _ {g ^ {\prime}} \frac {\mathbb {P} _ {D _ {g}} \left(x \in \Delta (B _ {g ^ {\prime}} (h ^ {*} , 2 \nu + 2 ^ {I - i + 1} \epsilon))\right)}{2 \nu + 2 ^ {I - i + 1} \epsilon}\right) ^ {2} \\ \leq 4 \left(\frac {\nu + \epsilon}{\epsilon}\right) ^ {2} \sum_ {i = 1} ^ {I} \left(\max  _ {g ^ {\prime}} \sup  _ {h \in \mathcal {H}} \sup  _ {r \geq 2 \nu + \epsilon} \frac {\mathbb {P} _ {D _ {g}} (x \in \Delta (B _ {g ^ {\prime}} (h , r)))}{r}\right) ^ {2} \\ = 4 \lceil \log (1 / \epsilon) \rceil \left(\frac {\nu + \epsilon}{\epsilon}\right) ^ {2} \left(\max _ {g ^ {\prime}} \theta_ {g ^ {\prime}}\right) ^ {2} \\ = 4 \lceil \log (1 / \epsilon) \rceil \left(\frac {\nu + \epsilon}{\epsilon}\right) ^ {2} \theta_ {\mathcal {G}} ^ {2}. \\ \end{array}
$$
The label complexity bound then follows by noting the algorithm labels the same amount of samples for all  $G$  groups each iteration, and ignoring the factors of  $\log (G)$  and  $\log (\log (1 / \epsilon))$ .
9.2 Group-Realizable Guarantees
Theorem 2. Suppose Algorithm 2 is run with the active learner  $\mathcal{A}_{CAL}$  of [26]. Then for all  $\epsilon >0$ ,  $\delta \in (0,1)$ , hypothesis classes  $\mathcal{H}$  with  $d < \infty$ , and collections of group distributions  $\mathcal{G}$  that are group realizable with respect to  $\mathcal{H}$ , with probability  $\geq 1 - \delta$ , the output  $\hat{h}$  satisfies
Proof. The label complexity follows directly from the guarantees given in [15]. By a union bound, we with probability  $\geq 1 - \delta$ , have that for all  $g \in [G]$ , that  $\mathcal{A}_{CAL}$  returns  $\hat{h}_g$  with the property that
$$
L _ {\mathcal {G}} \left(\hat {h} _ {g} \mid g\right) \leq \epsilon / 6.
$$
Fix some  $g \in [G]$  arbitrarily. Consider a counterfactual training set  $S_g$ , unseen by the learner, constructed by labeling each example  $x \in S_g'$  via the oracle call  $O_g(x)$ . Then Vapnik [28] tells us that  $m_g := |S_g'|$  is sufficiently large that with probability  $\geq 1 - \delta/2$ , for each  $h \in \mathcal{H}$  simultaneously, we have
$$
\left| L _ {\mathcal {G}} (h \mid g) - L _ {S _ {g}} (h) \right| <   \epsilon / 6.
$$
Again by the union bound, this uniform convergence on  $S_g$  and the guarantee on the runs of  $\mathcal{A}_{CAL}$  both hold for each  $g \in [G]$ . Conditioned on this high probability event, we can first note that for some arbitrary  $h \in \mathcal{H}$ ,
$$
\begin{array}{l} \left| L _ {S _ {g}} (h) - L _ {\hat {S} _ {g}} (h) \right| = \left| \frac {1}{m _ {g}} \sum_ {i = 1} ^ {m _ {g}} \mathbb {1} [ h (x _ {i}) \neq y _ {i} ] - \mathbb {1} [ h (x _ {i}) \neq \hat {h} _ {g} (x _ {i}) ] \right| \\ \leq \frac {1}{m _ {g}} \sum_ {i = 1} ^ {m _ {g}} \left| \mathbb {1} [ h (x _ {i}) \neq y _ {i} ] - \mathbb {1} [ h (x _ {i}) \neq \hat {h} _ {g} (x _ {i}) ] \right| \\ \leq \frac {1}{m _ {g}} \sum_ {i = 1} ^ {m _ {g}} \mathbb {1} [ y _ {i} \neq \hat {h} _ {g} (x _ {i}) ] \\ = L _ {S _ {g}} (\hat {h} _ {g}) \\ \leq L _ {\mathcal {G}} \left(\hat {h} _ {g}\right) + \epsilon / 6 \\ \leq \epsilon / 6 + \epsilon / 6 \\ = \epsilon / 3, \\ \end{array}
$$
where the second to last inequality comes from the uniform convergence over  $S_g$ , and the final inequality comes from the success of the runs of  $\mathcal{A}_{CAL}$ . Then for arbitrary  $h$ , combining Vapnik's guarantee and the inequality we just showed, we may write:
$$
\begin{array}{l} \left| L _ {\mathcal {G}} (h \mid g) - L _ {\hat {S} _ {g}} (h) \right| = \left| L _ {\mathcal {G}} (h \mid g) - L _ {S _ {g}} (h) + L _ {S _ {g}} (h) - L _ {\hat {S} _ {g}} (h) \right| \\ \leq \left| L _ {\mathcal {G}} (h \mid g) - L _ {S _ {g}} (h) \right| + \left| L _ {S _ {g}} (h) - L _ {\hat {S} _ {g}} (h) \right| \\ <   \epsilon / 6 + \epsilon / 3 \\ = \epsilon / 2. \\ \end{array}
$$
Given this guarantee on the representativeness of the artificially labeled samples on each group  $g$ , we have a guarantee for the representativeness over the worst case. For arbitrarily  $h \in \mathcal{H}$ , we may write
$$
\begin{array}{l} \left| L _ {\mathcal {G}} ^ {\max } (h) - \max  _ {g \in [ G ]} L _ {\hat {S} _ {g}} (h) \right| = \left| \max  _ {g \in [ G ]} L _ {\mathcal {G}} (h \mid g) - \max  _ {g \in [ G ]} L _ {\hat {S} _ {g}} (h) \right| \\ \leq \max  _ {g \in [ G ]} \left| L _ {\mathcal {G}} (h \mid g) - L _ {\hat {S} _ {g}} (h) \right| \\ \leq \epsilon / 2. \\ \end{array}
$$
Thus, by the fact that  $\hat{h}$  is the ERM on the artificially labeled dataset, we have
$$
L _ {\mathcal {G}} ^ {\max} (\hat {h}) \leq \max _ {g \in [ G ]} L _ {\hat {S} _ {g}} (\hat {h}) + \epsilon / 2 \leq \max _ {g \in [ G ]} L _ {\hat {S} _ {g}} (h ^ {*}) + \epsilon / 2 \leq L _ {\mathcal {G}} ^ {\max} (h ^ {*}) + \epsilon .
$$
9.3 Approximation Guarantees
Theorem 3. Suppose Algorithm 3 is run with the active learner  $\mathcal{A}_{DHM}$  of [15]. Then for all  $\epsilon >0$ ,  $\delta \in (0,1)$ , hypothesis classes  $\mathcal{H}$  with  $d < \infty$ , and collections of groups  $\mathcal{D}$ , with probability  $\geq 1 - \delta$ , the output  $\hat{h}$  satisfies
Proof. The proof is almost identical to that of Theorem 2. The label complexity bound follows directly from [10]. Similar to before, we have that for all  $g \in [G]$ ,  $\mathcal{A}_{DHM}$  returns  $\hat{h}_g$  with the property that
$$
L _ {\mathcal {G}} (\hat {h} _ {g} \mid g) \leq L _ {\mathcal {G}} (h _ {g} ^ {*} \mid g) + \epsilon / 6.
$$
Fix some  $g \in [G]$  arbitrarily. On a counterfactual training set  $S_g$ , unseen by the learner, constructed by labeling each example  $x \in S_g'$  via the oracle call  $O_g(x)$ , it holds that  $m_g := |S_g'|$  is sufficiently large that with probability  $\geq 1 - \delta / 2$ , for each  $h \in \mathcal{H}$  simultaneously, we have
By the union bound, this uniform convergence and the guarantee on the runs of  $\mathcal{A}_{DHM}$  both hold. Thus, we can first note that for some arbitrary  $h\in \mathcal{H}$
$$
\begin{array}{l} \left| L _ {S _ {g}} (h) - L _ {\hat {S} _ {g}} (h) \right| = \left| \frac {1}{m _ {g}} \sum_ {i = 1} ^ {m _ {g}} \mathbb {1} [ h (x _ {i}) \neq y _ {i} ] - \mathbb {1} [ h (x _ {i}) \neq \hat {h} _ {g} (x _ {i}) ] \right| \\ \leq \frac {1}{m _ {g}} \sum_ {i = 1} ^ {m _ {g}} \left| \mathbb {1} [ h (x _ {i}) \neq y _ {i} ] - \mathbb {1} [ h (x _ {i}) \neq \hat {h} _ {g} (x _ {i}) ] \right| \\ \leq \frac {1}{m _ {g}} \sum_ {i = 1} ^ {m _ {g}} \mathbb {1} [ y _ {i} \neq \hat {h} _ {g} (x _ {i}) ] \\ \leq L _ {\mathcal {G}} \left(\dot {h} _ {g} \mid g\right) + \epsilon / 6 \\ \leq L _ {\mathcal {G}} \left(h _ {g} ^ {*} \mid g\right) + \epsilon / 3 \\ = \nu_ {g} + \epsilon / 3. \\ \end{array}
$$
where the second to last inequality comes from uniform convergence over  $S_G$ , and the final equality comes from the correctness guarantee of  $\mathcal{A}_{DHM}$ . Then for arbitrary  $h$ , combining Vapnik's guarantee and the inequality we just showed, we may write:
$$
\begin{array}{l} \left| L _ {\mathcal {G}} (h \mid g) - L _ {\hat {S} _ {g}} (h) \right| = \left| L _ {\mathcal {G}} (h \mid g) - L _ {S _ {g}} (h) + L _ {S _ {g}} (h) - L _ {\hat {S} _ {g}} (h) \right| \\ \leq \left| L _ {\mathcal {G}} (h \mid g) - L _ {S _ {g}} (h) \right| + \left| L _ {S _ {g}} (h) - L _ {\hat {S} _ {g}} (h) \right| \\ <   \epsilon / 6 + \nu_ {g} + \epsilon / 3 \\ = \nu_ {g} + \epsilon / 2. \\ \end{array}
$$
Then, as above, we have, for arbitrarily  $h\in \mathcal{H}$
$$
\left| L _ {\mathcal {G}} ^ {\max } (h) - \max  _ {g \in [ G ]} L _ {\hat {S} _ {g}} (h) \right| \leq \max  _ {g \in [ G ]} \left| L _ {\mathcal {G}} (h \mid g) - L _ {\hat {S} _ {g}} (h) \right| \leq \max  _ {g \in [ G ]} \nu_ {g} + \epsilon / 2 \leq \nu + \epsilon / 2,
$$
where the final inequality comes from the fact that if any hypothesis has less than  $\nu_{g}$  error on all groups, it would be optimal on group  $g$ . Thus, by the fact that  $\hat{h}$  is the ERM on the artificially labeled dataset, we have
$$
L _ {\mathcal {G}} ^ {\max } (\hat {h}) \leq \max  _ {g \in [ G ]} L _ {\hat {S} _ {g}} (\hat {h}) + \nu_ {g} + \epsilon / 2 \leq \max  _ {g \in [ G ]} L _ {\hat {S} _ {g}} (h ^ {*}) + \nu_ {g} + \epsilon / 2 \leq L _ {\mathcal {G}} ^ {\max } (h ^ {*}) + 2 \nu + \epsilon \leq 3 \cdot L _ {\mathcal {G}} ^ {\max } (h ^ {*}) + \epsilon
$$