SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| \hat {H} - P \| _ {\mathrm {T V}} \leq W (\hat {H}) + 2 \| H ^ {*} - P \| _ {\mathrm {T V}} = W (\hat {H}) + 2 \mathrm {O P T} \\ \leq \alpha \Gamma_ {h} + \epsilon + 2 \epsilon^ {\prime} + 2 O P T \leq (\alpha + 2) O P T + \epsilon + 2 \epsilon^ {\prime}. \\ \end{array}
$$
Corollary C.2. For every parameters  $\epsilon''$  and  $\delta''$  in  $(0,1)$ , there exists an  $(\alpha = 5, \epsilon = \epsilon'', \delta = \delta'')$ -proper learner (without promise) that uses  $O\left(\log(n / \delta'') / (\epsilon'')^2\right)$  samples and runs  $\tilde{O}\left(n / \left(\delta'' (\epsilon'')^2\right)\right)$  time.
Proof. First, we start off with the parameters: We set  $\epsilon := \epsilon'' / 6$ ,  $\epsilon' := \epsilon'' / 6$ ,  $\delta' = \delta'' / 2$ , and  $\delta = \delta'' / (2t)$  where  $t := \lceil \log_2(1 / \epsilon) \rceil + 1$  is the number of iterations in Algorithm 5. Using the Chernoff bound and reusing all samples, we can estimate all the Scheffé sets with additive error at most  $\epsilon'$  with probability at least  $1 - \delta'$ .
Note that based on Corollary A.1, the random ladder tournament (with an arbitrarily small confidence parameter) is a  $(\alpha = 3,\epsilon ,\delta)$ -proper learner (with promise) that uses  $O\left(\log (n / \delta) / \epsilon^2\right)$  samples and runs in  $O\left(n\log (n / \delta^{\prime \prime}) / \left(\delta^{\prime \prime}\left(\epsilon^{\prime \prime}\right)^{2}\right)\right)$  time. We use this learner as our  $\mathcal{A}$  and run the reduction algorithm  $\mathcal{B}$  with the parameters we specified earlier. Now, using Theorem 7, we obtain a proper learner without promise that satisfies the following guarantee with probability at least  $1 - \delta '' =$ $1 - (\delta^{\prime} + t\cdot \delta)$ :
$$
\| \hat {H} - P \| _ {\mathrm {T V}} \leq (\alpha + 2) \cdot \mathrm {O P T} + (\alpha + 1) \epsilon + 2 \epsilon^ {\prime} = 5 \cdot \mathrm {O P T} + \epsilon^ {\prime \prime}.
$$
Regarding the time complexity, the while loop in Algorithm 5 is repeated  $t$  times. In each iteration, we compute  $W(\tilde{H})$  which takes  $O\left(n\log (n / \delta^{\prime \prime}) / (\epsilon^{\prime \prime})^{2}\right)$  time, and we call the random ladder tournament (i.e.,  $\mathcal{A}$ ). Hence, the overall time complexity is:
$$
O \left(t \cdot \frac {n \log (n / \delta^ {\prime \prime})}{\delta^ {\prime \prime} (\epsilon^ {\prime \prime}) ^ {2}}\right) = O \left(\frac {n \cdot \log (n / \delta^ {\prime \prime}) \cdot \log (1 / \epsilon^ {\prime \prime})}{\delta^ {\prime \prime} (\epsilon^ {\prime \prime}) ^ {2}}\right) = \tilde {O} \left(\frac {n}{\delta^ {\prime \prime} (\epsilon^ {\prime \prime}) ^ {2}}\right).
$$
Memory-data consumption of reduction algorithm: Besides the sample and memory usage of  $\mathcal{A}$ ,  $\mathcal{B}$  needs to keep track of  $O(1)$  numbers and also computes  $\tilde{W}(\tilde{H})$ . Now, given  $b$  bits of memory, we can partition  $\mathcal{H}$  into consecutive blocks of size  $r = b / (\log (\log (n) / \epsilon))$ . At every round, we process one block of size  $r$ , we keep track of the Scheffé counts of  $\tilde{H}$  and the hypotheses in the block. For each round we use  $O(\log (n) / \epsilon^2)$  samples, are we have  $O(n / r)$  rounds. Thus, the total number of samples is  $s := O(n / r \cdot (\log (n) / \epsilon^2))$ . Thus, for this part we will have:
$$
s \cdot b = O \left(\frac {n \log n}{\epsilon^ {2}} \cdot \log \left(\frac {\log n}{\epsilon}\right)\right).
$$
Remark 8. Using the reduction algorithm with the memory-data tradeoff described above, one can translate our main theorem, Theorem 1, to a  $(\alpha = 11,\epsilon ,0.1)$ -proper learner (without promise) incurring additional factors of poly  $(\log \log (n),\log (1 / \epsilon))$
D Auxiliary lemmas and facts
Fact D.1. [adapted from Theorem 4 in [MS08]] For every pair  $i, j \in [n]$ , let  $q_{ij}$  denote an estimate of the probabilities of the Scheffé set of  $H_i$  and  $H_j$  according to  $P$ . Let  $\hat{H}$  be the hypothesis that minimizes the following:
$$
\hat{H} := \operatorname *{arg  max}_{H_{i}\in \mathcal{H}}\min_{j\in [n]\setminus \{i\}}\left|H_{i}\left(\mathcal{S}(H_{i},H_{j})\right) - q_{ij}\right|.
$$
For every  $\epsilon$  and  $\delta$ , if we estimate  $q_{ij}$  using  $O(\log (n / \delta) / \epsilon^2)$  samples from  $P$ , then  $\hat{H}$  satisfies the following guarantee:
$$
\left\| \hat {H} - X \right\| _ {T V} \leq 3 O P T + \epsilon .
$$
- If  $H_{1}$  is  $\Gamma$ -close to  $P$ , then the algorithm returns  $H_{1}$ .
- If  $H_{1}$  is  $(\max(\Gamma, \|H_{2} - P\|_{TV}) + 2\|H_{2} - P\|_{TV} + \epsilon)$ -far from  $P$ , then the algorithm returns  $H_{2}$ .
Proof. Let  $q, p_1$ , and  $p_2$  denote the true probability of the Scheffé set of  $H_1$  and  $H_2$  according to  $P$ ,  $H_1$ , and  $H_2$ :
$$
q := P \left(\mathcal {S} \left(H _ {1}, H _ {2}\right)\right), \quad p _ {1} := H _ {1} \left(\mathcal {S} \left(H _ {1}, H _ {2}\right)\right), \quad p _ {2} := H _ {2} \left(\mathcal {S} \left(H _ {1}, H _ {2}\right)\right).
$$
And, let  $\hat{q}$  be the estimate of  $q$  from the samples. The Using the Hoeffding bound, one can show that  $|q - \hat{q}|$  is at most  $\epsilon' \coloneqq \epsilon / 2$  with probability  $1 - \delta$ , since we use  $O(\log(1/\delta)/\epsilon^2)$  samples. Now, if  $\|H_1 - P\|_{\mathrm{TV}}$  is at most  $\Gamma$ , then  $|p_1 - q|$  must be at most  $\Gamma$  implying:
$$
\left| p _ {1} - \hat {q} \right| \leq \left| p _ {1} - q \right| + \left| q - \hat {q} \right| \leq \Gamma + \epsilon^ {\prime}.
$$
Therefore, the algorithm outputs  $H_{1}$  as the winner.
Now, we show the second guarantee. If the total variation distance between  $H_{1}$  and  $P$ ,  $\| H_{1} - P\|_{\mathrm{TV}}$  is greater than  $(\max (\Gamma ,\| H_2 - X\|_{\mathrm{TV}}) + 2\| H_2 - X\|_{\mathrm{TV}} + \epsilon)$ , then by the triangle inequality and the definition of total variation distance, we will have:
$$
\begin{array}{l} \left| p _ {1} - \hat {q} \right| \geq \left| p _ {1} - p _ {2} \right| - \left| p _ {2} - \hat {q} \right| \geq \left| p _ {1} - p _ {2} \right| - \left| p _ {2} - q \right| - \left| q - \hat {q} \right| \\ \geq \| H _ {1} - H _ {2} \| _ {\mathrm {T V}} - \| H _ {2} - P \| _ {\mathrm {T V}} - \epsilon^ {\prime} \quad (\text {B y}) \\ \geq \left\| H _ {1} - P \right\| _ {\mathrm {T V}} - 2 \left\| H _ {2} - P \right\| _ {\mathrm {T V}} - \epsilon^ {\prime} \\ > \max  (\Gamma , \| H _ {2} - X \| _ {\mathrm {T V}}) + \epsilon - \epsilon^ {\prime} \\ \geq \max  \left(\Gamma + \epsilon^ {\prime}, | p _ {2} - q | + \epsilon^ {\prime}\right) \\ \geq \max  \left(\Gamma + \epsilon^ {\prime}, | p _ {2} - \hat {q} | + | q - \hat {q} |\right) \\ \geq \max  \left(\Gamma + \epsilon^ {\prime}, | p _ {2} - \hat {q} |\right) \\ \end{array}
$$
Thus, the algorithm outputs  $H_{2}$  as the winner.
Proof. Using Proposition 3.24 in [Vad12] the mapping is in fact a class of pairwise independent hash functions. That is, the following holds:
Fix two indices  $i\in \mathbb{Z}_p$  and  $x\in [n^{\prime}]$  . The probability of mapping  $x$  to  $i$  is:
$$
\Pr_ {a, b} \left[ f _ {a, b} (x) = i \right] = \frac {1}{q}.
$$
Fix four indices  $i,j\in \mathbb{Z}_p$  and  $x,y\in [n^{\prime}]$  where  $x\neq y$  . The probability of mapping  $x$  to  $i$  and  $y$  to  $j$  is:
$$
\Pr_ {a, b} \left[ f _ {a, b} (x) = i \text {a n d} f _ {a, b} (y) = j \right] = \frac {1}{q ^ {2}}
$$
Suppose  $\mathcal{H}_e$  and  $\mathcal{H}_d$  indicate the set of excellent and decent hypotheses respectively. Fix an index  $i$  for which  $H_i$  is an excellent hypothesis. Suppose one of the indices in  $[n']$ , say  $x$ , is mapped to  $i$ . Now, the probability of another hypothesis in  $\mathcal{H}_e \cup \mathcal{H}_d$  being chosen for a fixed index  $y \neq x$  in  $[n']$  is:
$$
\begin{array}{l} \Pr_{a,b}\big[H_{f_{a,b}(y)(\mathrm{mod} q)}\in \mathcal{H}_{e}\cup \mathcal{H}_{d}\mid f_{a,b}(x) = i\big] = \sum_{j\text{s.t.} H_{j}\in ((\mathcal{H}_{e}\cup \mathcal{H}_{d})\backslash \{H_{i}\})}\Pr_{a,b}[f_{a,b}(y) = j\mid f_{a,b}(x) = i] \\ = \sum_ {j \text {s . t .} H _ {j} \in ((\mathcal {H} _ {e} \cup \mathcal {H} _ {d}) \backslash \{H _ {i} \})} \frac {\operatorname {P r} _ {a , b} [ f _ {a , b} (y) = j \text {a n d} f _ {a , b} (x) = i ]}{\operatorname {P r} _ {a , b} [ f _ {a , b} (x) = i ]} \\ = \frac {t _ {e} + t _ {d} - 1}{q}. \\ \end{array}
$$
Hence, the expected number of hypotheses (other than  $H_{i}$ ) that are selected is:
$$
\begin{array}{l} \mathbf {E} \left[ \# y \in [ n ^ {\prime} ] \backslash \{x \} \text {s . t .} H _ {f _ {a, b} (y) (\mathrm {m o d} q)} \in \mathcal {H} _ {e} \cup \mathcal {H} _ {d} \mid f _ {a, b} (x) = i \right] = \frac {(n ^ {\prime} - 1) \cdot \left(t _ {e} + t _ {d} - 1\right)}{q} \\ \leq \frac {n ^ {\prime} \cdot \left(t _ {e} + t _ {d}\right)}{q} \leq \frac {1}{2}. \\ \end{array}
$$
The last inequality is due to the assumption we have:  $(t_e + t_d) / q \leq 1 / (2n')$ . Next, by Moarkov's inequality, the probability that the number of  $y$ 's are at least one (that is two times the expected value) is at most  $1/2$ . Hence, if  $x$  is mapped to  $H_i$ , with probability at least  $1/2$ , there is no other excellent or decent hypothesis is selected.
Now, we compute the probability that an excellent hypothesis is selected and no other excellent or decent hypothesis is selected as follows:
$\mathbf{Pr}_{a,b}[$  One excellent hypothesis and no decent one]
$$
\begin{array}{l} = \sum_ {x \in [ n ^ {\prime} ]} \sum_ {H _ {i} \in \mathcal {H} _ {e}} \Pr \left[ f _ {a, b} (x) = i (\bmod q) \text {a n d n o o t h e r h y p o t h e s i s i n} \mathcal {H} _ {e} \cup \mathcal {H} _ {d} \text {i s s e l e c t e d} \right] \\ = \sum_ {x \in [ n ^ {\prime} ]} \sum_ {H _ {i} \in \mathcal {H} _ {e}} \Pr \left[ f _ {a, b} (x) = i (\bmod q) \right] \\ \end{array}
$$
$$
\begin{array}{l} \cdot \Pr \left[ \operatorname {n o} y \in \left[ n ^ {\prime} \right] \backslash \{x \} \text {s . t .} H _ {f _ {a, b} (y) (\mathrm {m o d} q)} \in \mathcal {H} _ {e} \cup \mathcal {H} _ {d} \mid f _ {a, b} (x) = i (\mathrm {m o d} q) \right] \\ \geq \sum_ {x \in [ n ^ {\prime} ]} \sum_ {H _ {i} \in \mathcal {H} _ {e}} \Pr \left[ f _ {a, b} (x) = i (\bmod q) \right] \cdot \frac {1}{2} = \frac {n ^ {\prime} \cdot t _ {e}}{2 q} \\ \end{array}
$$
Fact D.2. Let  $n, a, b$  be three positive integers. Then, we have:
$$
\left\lceil \frac {\lceil n / a \rceil}{b} \right\rceil = \left\lceil \frac {n}{a b} \right\rceil .
$$
Proof. Let  $r$  denote the remainder of  $n$  divided by  $a$ . If  $r = 0$ , then the claim is trivial. Therefore, assume  $r$  is in  $\{1, 2, \ldots, a - 1\}$ . Let  $s$  be the remainder of  $n$  divided by  $ab$ . Clearly,  $s$  has to be of the form:  $t \cdot a + r$  for  $t \in \{0, 1, \ldots, b - 1\}$ . Hence, we have
$$
\begin{array}{l} \left\lceil \frac {\lceil n / a \rceil}{b} \right\rceil = \left\lceil \frac {n}{a b} + \frac {a - r}{a b} \right\rceil = \left\lfloor \frac {n}{a b} \right\rfloor + \left\lceil \frac {s + a - r}{a b} \right\rceil \\ = \left\lfloor \frac {n}{a b} \right\rfloor + \left\lceil \frac {(t + 1) \cdot a}{a b} \right\rceil = \left\lfloor \frac {n}{a b} \right\rfloor + 1 = \left\lceil \frac {n}{a b} \right\rceil . \\ \end{array}
$$
Fact D.3. Given the definition of the  $g_{\ell}$ 's in Equation (2), for any positive integer  $\ell$ , we have:
$$
\prod_ {i = 1} ^ {\ell} g _ {i} \geq r ^ {1. 5 ^ {\ell} - 1}.
$$
Proof. By the properties of the geometric series, we obtain:
$$
\prod_ {i = 1} ^ {\ell} g _ {i} \geq \prod_ {i = 1} ^ {\ell} (\sqrt {r}) ^ {1. 5 ^ {i - 1}} = (\sqrt {r}) ^ {\sum_ {i = 1} ^ {\ell} 1. 5 ^ {i - 1}} = (\sqrt {r}) ^ {(1. 5 ^ {\ell} - 1) / (1. 5 - 1)} = r ^ {1. 5 ^ {\ell} - 1}.
$$