SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {V a r} \left[ \Delta_ {i} - \alpha \left(\mathcal {E} _ {i} - \mathcal {E} _ {i - 1}\right) \mid \mathcal {F} _ {i} \right] \leq 2 \alpha \left(\mathbb {E} \left[ | \Delta_ {i} | \mid \mathcal {F} _ {i} \right] + \alpha \eta \max  _ {x} \frac {q (x)}{\mathcal {D} _ {X} (x)}\right) \\ \leq 2 \alpha \left(\mathbb {E} \left[ | \Delta_ {i} | | \mathcal {F} _ {i} \right] + \frac {1}{2} \mathbb {E} \left[ \Delta_ {i} | \mathcal {F} _ {i} \right]\right) \\ \leq 3 \alpha \mathbb {E} \left[ | \Delta_ {i} | | \mathcal {F} _ {i} \right] \\ \lesssim \alpha \mathbb {E} \left[ \Delta_ {i} | \mathcal {F} _ {i} \right]. \\ \end{array}
$$
By picking  $\alpha$  small enough,  $\sum_{i < k} \operatorname{Var}\left[\Delta_i - \alpha(\mathcal{E}_i - \mathcal{E}_{i-1})|\mathcal{F}_i\right] \leq \frac{1}{8}\mu_k$ . Moreover,  $|\Delta_i - \alpha(\mathcal{E}_i - \mathcal{E}_{i-1})| \leq 2\alpha$  always. Therefore by Freedman's inequality, with  $1 - \delta$  probability we have for any  $k$  that
$$
\begin{array}{l} \psi_ {k} - \alpha \mathcal {E} _ {k} \geq \mu_ {k} - \sqrt {\frac {4 \alpha^ {2}}{9} \log^ {2} \frac {1}{\delta} + 2 \sum_ {i <   k} \operatorname {V a r} _ {i - 1} [ \Delta_ {i} - \alpha (\mathcal {E} _ {i} - \mathcal {E} _ {i - 1}) ] \log \frac {1}{\delta}} - \frac {2 \alpha}{3} \log \frac {1}{\delta} \\ \geq \mu_ {k} - \sqrt {\frac {4 \alpha^ {2}}{9} \log^ {2} \frac {1}{\delta} + \frac {1}{4} \mu_ {k} \log \frac {1}{\delta}} - \frac {2 \alpha}{3} \log \frac {1}{\delta} \\ \geq \mu_ {k} - \max \left\{\frac {2 \sqrt {2} \alpha}{3} \log \frac {1}{\delta}, \frac {\sqrt {2}}{2} \sqrt {\mu_ {k} \log \frac {1}{\delta}} \right\} - \frac {2 \alpha}{3} \log \frac {1}{\delta} \\ \geq \mu_ {k} - \max  \left\{\frac {\sqrt {2}}{2} \log \frac {1}{\delta}, \frac {\sqrt {2}}{2} \mu_ {k} \right\} - \frac {2 \alpha}{3} \log \frac {1}{\delta} \\ \geq \left(1 - \frac {\sqrt {2}}{2}\right) \mu_ {k} - \sqrt {2} \log \frac {1}{\delta} \\ \geq - \sqrt {2} \log \frac {1}{\delta} \\ \end{array}
$$
Rearrange and we proved the lemma.
Combining Lemma A.7 and Lemma A.8, we can show  $C$  is small with high probability as the lemma follows.
Lemma A.9. For  $k = O\left(m^{*}\log \frac{|H|}{\delta}\right)$ , with probability at least  $1 - 2\delta$ ,  $h^* \in S_k$  and  $|C| \leq O\left(\log \frac{|H|}{\delta}\right)$  at iteration  $k$ .
Proof. By union bound, with probability at least  $1 - 2\delta$ , Lemma A.6 and A.8 will hold at the same time. This means  $h^*$  is added to  $S_k$ . By definition,  $0 \geq \phi_k \geq \phi_0 + \psi_k$ , so  $\psi_k \leq 2\log |H|$ . Therefore, by Lemma A.7 and A.8, the number of balls added  $|C|$  is  $O\left(\log |H| + \log \frac{1}{\delta}\right) = O\left(\log \frac{|H|}{\delta}\right)$ .
A.5 Putting Everything Together
We proved that after  $O\left(m^{*}\log \frac{|H|}{\delta}\right)$  iterations,  $h^* \in S_i$  and  $C$  is small with high probability. Hence, running the stage two algorithm to return a desired hypothesis will not take much more queries. We are ready to put everything together and finally prove Theorem 1.1.
Proof. Let's pick  $c_{1}, c_{4}, c_{5}$  as in Theorem 2.3 and pick the confidence parameter to be  $\frac{\delta}{3}$ . Then by Lemma A.9, with probability  $1 - \frac{2\delta}{3}$ , the first  $O\left(\log \frac{|H|}{\delta}\right)$  ball added to  $S_{i}$  will contain  $h^{*}$ . Since each ball added to  $C$  has radius  $3c_{4}\eta + 3c_{5}\varepsilon$ , the best hypothesis in  $C$  has error  $(3 + 3c_{4})\eta + 3c_{5}\varepsilon$ . By Theorem 2.2, with probability  $1 - \frac{\delta}{3}$ , the algorithm will return a hypothesis with error  $(9 + 9c_{4})\eta + 9c_{5}\varepsilon \leq \eta + \varepsilon$ . Therefore, by union bound, the algorithm will return a desired hypothesis with probability  $1 - \delta$ . This proves the correctness of the algorithm.
The stage one algorithm makes
$$
O \left(m ^ {*} \left(H, \mathcal {D} _ {X}, c _ {4} \eta , c _ {5} \varepsilon - 2 \eta , \frac {9 9}{1 0 0}\right) \log \frac {| H |}{\delta}\right) \leq O \left(m ^ {*} \left(H, \mathcal {D} _ {X}, c _ {4} \eta , \frac {c _ {5}}{2} \varepsilon , \frac {9 9}{1 0 0}\right) \log \frac {| H |}{\delta}\right)
$$
queries. The stage two algorithm makes  $O\left(|C|\log \frac{|C|}{\delta}\right)$  queries by Theorem 2.2. Note that  $C$  is a  $c_{4}\eta +c_{5}\varepsilon$ -packing because the center of added balls are at least  $c_{4}\eta +c_{5}\varepsilon$  away, so  $m^{*}\left(H,\mathcal{D}_{X},\frac{c_{4}}{2}\eta ,\frac{c_{5}}{2}\varepsilon ,\frac{99}{100}\right)\geq \log |C|$ . Since  $|C|\leq \log \frac{|H|}{\delta}$  by Lemma A.9, stage two algorithm takes  $O\left(\left(m^{*}\left(H,\mathcal{D}_{X},\frac{c_{4}}{2}\eta ,\frac{c_{5}}{2}\varepsilon ,\frac{99}{100}\right) + \log \frac{1}{\delta}\right)\log \frac{|H|}{\delta}\right)$  queries. Picking  $c_{2} = c_{4},c_{3} = \frac{c_{5}}{2}$ , we get the desired sample complexity bound.
To compute the packing at the beginning of the algorithm, we need to compute the distance of every pair of hypotheses, which takes  $O(|H|^2|\mathcal{X}|)$  time. Computing  $r$  in each round takes  $O(|H||\mathcal{X}|)$  time and solving the optimization problem takes  $O(|\mathcal{X}|)$  time. Therefore, the remaining steps in stage one take  $O\left(m^{*}|H||\mathcal{X}|\log \frac{|H|}{\delta}\right)$  time. Stage two takes  $O\left(\log \frac{|H|}{\delta}\log \frac{\log\frac{|H|}{\delta}}{\delta}\right)$  time. Therefore, the overall running time is polynomial of the size of the problem.
Similarly, we can prove Theorem 2.3, which is a stronger and more specific version of Theorem 1.1.
Proof. By Lemma A.9 (with  $m^*$  replaced by  $\frac{1}{\beta}$  and setting confidence parameter to  $\frac{\delta}{3}$ ) after  $O\left(\frac{1}{\beta}\log \frac{N}{\delta}\right)$  queries, with probability at least  $1 - \frac{2\delta}{3}$ , a hypothesis in  $C$  will be within  $c_{4}\eta +c_{5}\varepsilon$  to  $h^{*}$  and  $|C| = O\left(\log \frac{N}{\delta}\right)$ . From Theorem 2.2, with probability at least  $1 - \frac{\delta}{3}$ , stage two algorithm then outputs a hypothesis  $\hat{h}$  that is  $9c_{4}\eta +9c_{5}\varepsilon$  from  $h^\prime$  so err  $(\hat{h})\leq 9c_4\eta +9c_5\varepsilon \leq \eta +\varepsilon$  by the choice of the constants. The stage two algorithm makes  $O\left(\log \frac{N}{\delta}\log \frac{\log\frac{N}{\delta}}{\delta}\right)$  queries. Overall, the algorithm makes  $O\left(\frac{1}{\beta}\log \frac{N}{\delta} +\log \frac{N}{\delta}\log \frac{\log\frac{N}{\delta}}{\delta}\right)$  queries and succeeds with probability at least  $1 - \delta$ .
B Query Complexity Lower Bound
In this section we derive a lower bound for the agnostic binary classification problem, which we denote by AGNOSTICLEARNING. The lower bound is obtained from a reduction from minimum set cover, which we denote by SETCOVER. The problem SETCOVER consists a pair  $(U, S)$ , where  $U$  is a ground set and  $S$  is a collection of subsets of  $U$ . The goal is to find a set cover  $C \subseteq S$  such that  $\bigcup_{S \in C} S = U$  of minimal size  $|C|$ . We use  $K$  to denote the cardinality of the minimum set cover.
Lemma B.1 (Dinur and Steurer [2014], Corollary 1.5). There exists hard instances SETCOVERHARD with the property  $K \geq \log |U|$  such that for every  $\gamma > 0$ , it is NP-hard to approximate SETCOVERHARD to within  $(1 - \gamma)\ln |U|$ .
Proof. This lemma directly follows from Dinur and Steurer [2014, Corollary 1.5]. In their proof, they constructed a hard instance of SETCOVER from LABELCOVER. The size of the minimum cover  $K \geq |V_1| = Dn_1$  and  $\log |U| = (D + 1)\ln n_1 \leq K$ . So the instance in their proof satisfies the desired property.
Then we prove the following lemma by giving a ratio-preserving reduction from SETCOVER to AGNOSTICLEARNING.
Lemma B.2. If there exists a deterministic  $\alpha$ -approximation algorithm for AGNOSTICLEARNING  $\left(H, \mathcal{D}_x, \frac{1}{3|\mathcal{X}|}, \frac{1}{3|\mathcal{X}|}, \frac{1}{4|H|}\right)$ , there exists a deterministic  $2\alpha$ -approximation algorithm for SETCOVERHARD.
Proof. Given an instance of SETCOVERHARD, for each  $s \in S$ , number the elements  $u \in s$  in an arbitrary order; let  $f(s, u)$  denote the index of  $u$  in  $s$ 's list (and padding 0 to the left with the extra bit). We construct an instance of AGNOSTICLEARNING as the following:
1. Let the domain  $\mathcal{X}$  have three pieces:  $U, V \coloneqq \{(s,j) \mid s \in S, j \in [1 + \log |s|]\}$ , and  $D = \{1, \dots, \log |U|\}$ , an extra set of  $\log |U|$  more coordinates.
2. On this domain, we define the following set of hypotheses:
(a) For  $u \in U$ , define  $h_u$  which only evaluates 1 on  $u \in U$  and on  $(s, j) \in V$  if  $u \in s$  and the  $j$ 'th bit of  $(2f(s, u) + 1)$  is 1.
(b) For  $d \in D$ , define  $h_d$  which only evaluates 1 on  $d$ .
(c) Define  $h_0$  which evaluates everything to 0.
3. Let  $\mathcal{D}_X$  be uniform distribution over  $\mathcal{X}$  and set  $\eta = \frac{1}{3|\mathcal{X}|}$  and  $\varepsilon = \frac{1}{3|\mathcal{X}|}$ . Set  $\delta = \frac{1}{4|H|}$ .
Any two hypotheses satisfy  $\| h_1 - h_2 \| \geq \frac{1}{|\mathcal{X}|} > \varepsilon = \eta$ , so  $\mathrm{err}(h^*) = 0$ . First we show that  $m^* \left(H, \mathcal{D}_x, \frac{1}{3|\mathcal{X}|}, \frac{1}{3|\mathcal{X}|}, \frac{1}{4|H|} \right) \leq K + \log |U|$ . Indeed there exists a deterministic algorithm using  $K + \log |U|$  queries to identify any hypothesis with probability 1. Given a smallest set cover  $C$ , the algorithm first queries all  $(s, 0) \in V$  for  $s \in C$ . If  $h^* = h_u$  for some  $u$ , then for the  $s \in S$  that covers  $u$ ,  $(s, 0)$  will evaluate to true. The identity of  $u$  can then be read out by querying  $(s, j)$  for all  $j$ . The other possibilities  $-h_d$  for some  $d$  or 0—can be identified by evaluating on all of  $D$  with  $\log U$  queries. The total number of queries is then at most  $K + \log |U|$  in all cases, so  $m^* \leq K + \log |U| \leq 2K$ .
We now show how to reconstruct a good approximation to set cover from a good approximate query algorithm. We feed the query algorithm  $y = 0$  on every query it makes, and let  $C$  be the set of all  $s$  for which it queries  $(s, j)$  for some  $j$ . Also, every time the algorithm queries some  $u \in U$ , we add an arbitrary set containing  $u$  to  $C$ . Then the size of  $C$  is at most the number of queries. We claim that  $C$  is a set cover: if  $C$  does not cover some element  $u$ , then  $h_u$  is zero on all queries made by the algorithm, so  $h_u$  is indistinguishable from  $h_0$  and the algorithm would fail on either input  $h_0$  or  $h_u$ . Thus if  $A$  is a deterministic  $\alpha$ -approximation algorithm for AGNOSTICLEARNING, we will recover a set cover of size at most  $\alpha m^* \leq \alpha (K + \log |U|) \leq 2\alpha K$ , so this gives a deterministic  $2\alpha$ -approximation algorithm for SETCOVERHARD.
Similar results also hold for randomized algorithms, we just need to be slightly careful about probabilities.
Lemma B.3. If there exists a randomized algorithm for AGNOSTICLEARNING  $\left(H, \mathcal{D}_x, \frac{1}{3|\mathcal{X}|}, \frac{1}{3|\mathcal{X}|}, \frac{1}{4|H|}\right)$ , there exists a randomized  $2\alpha$ -approximation algorithm for SETCOVERHARD with success probability at least  $\frac{2}{3}$ .
Proof. We use the same reduction as in Lemma B.2. Let  $A$  be an algorithm solves AGNOSTICLEARNING  $\left(H, \mathcal{D}_x, \frac{1}{3|\mathcal{X}|}, \frac{1}{3|\mathcal{X}|}, \frac{1}{4|H|}\right)$ . To obtain a set cover using  $A$ , we keeping giving  $A$  label 0 and construct the set  $C$  as before. Let  $q_C$  be a distribution over the reconstructed set  $C$ . Assume that by contradiction with probability at least  $\frac{1}{3}$ ,  $C$  is not a set cover. Then, with probability at least  $1/3$ , there is some element  $v$  such that both  $h_v$  and  $h_0$  are consistent on all queries the algorithm made; call such a query set "ambiguous".
Then what is the probability that the agnostic learning algorithm fails on the input distribution that chooses  $h^*$  uniformly from  $H$ ? Any given ambiguous query set is equally likely to come from any of the consistent hypotheses, so the algorithm's success probability on ambiguous query sets is at most  $1/2$ . The chance the query set is ambiguous is at least  $\frac{2}{3|H|}$ : a  $\frac{1}{3H}$  chance that the true  $h^*$  is  $h_0$  and the query set is ambiguous, and at least as much from the other hypotheses making it ambiguous. Thus the algorithm's fails to learn the true hypothesis with at least  $\frac{1}{3|H|}$  probability, contradicting the assumed  $\frac{1}{4|H|}$  failure probability.
Therefore, a set cover of size at most  $2\alpha K$  can be recovered with probability at least  $\frac{1}{3}$  using the agnostic learning approximation algorithm.
The following theorem will then follow.
Proof. Let's consider the instance of set cover constructed in Lemma B.2. Let  $c = 0.1$  and note that  $0.1\log |H|\leq 0.49\log \frac{|H|}{2}$ . If there exists a polynomial time  $0.49\log \frac{|H|}{2}$  approximation algorithm for the instance, then there exists a polynomial time  $0.98\log \frac{|H|}{2}\leq 0.98\log |U|$  approximation algorithm for SETCOVERHARD, which is a contradiction to Lemma B.1.