SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Note that there are two main sets of values that we need to store. First, the values of the Scheffé counts at every round that allow us to perform the all-go-against-all tournaments. Second, the
indices of the winner hypotheses that go to the next round. Below, we discuss bounding the memory usage for each part.
Memory usage of Scheffé counts: Here, we show to store the Scheffé counts, we need  $O(n \cdot r \cdot \log (\log (1 / \delta) / \epsilon))$  bits at every round. For a single group of size  $g_{\ell}$ , there are  $O(g_{\ell}^{2})$  pairs of hypotheses, therefore Scheffé sets, so we require  $O(g_{\ell}^{2} \log m_{\ell})$  bits to count the number of samples in all the sets. Thus, in round  $\ell$ , we need the following number of bits:
$$
\begin{array}{l} \text {s p a c e} = \# \text {g r o u p s} \cdot O \left(g _ {\ell} ^ {2} \cdot \log m _ {\ell}\right) = O \left(\frac {s _ {\ell}}{g _ {\ell}} \cdot g _ {\ell} ^ {2} \cdot \log m _ {\ell}\right) \\ = O \left(s _ {\ell} \cdot g _ {\ell} \cdot \left(\log g _ {\ell} + \log \left(\frac {\log (1 / \delta)}{\epsilon^ {2}}\right)\right)\right) \\ = n \cdot \log \left(\frac {\log (1 / \delta)}{\epsilon}\right) \cdot O \left(\frac {s _ {\ell}}{n} \cdot g _ {\ell} \cdot (\log g _ {\ell})\right) \\ \end{array}
$$
Next, we show the last term in the last line is  $O(r)$ . For round 1, the memory bound holds since  $\sqrt{r} \cdot \log (\sqrt{r}) \leq r$ . Now for  $\ell \geq 2$  using Equation (3), we have:
$$
O \left(\frac {s _ {\ell}}{n} \cdot g _ {\ell} \cdot (\log g _ {\ell})\right) = O \left(\frac {g _ {\ell} \cdot \log g _ {\ell}}{r ^ {(1 . 5) ^ {\ell - 1} - 1}}\right) = O \left(\frac {r \cdot g _ {\ell} \cdot \log g _ {\ell}}{g _ {\ell} ^ {2}}\right) = O (r).
$$
Hence, the bit complexity at every step is:
$$
O \left(n \cdot r \cdot \log \left(\frac {\log (1 / \delta)}{\epsilon}\right)\right).
$$
Storing the indices: Note that to run the comparisons, one needs the true indices of the hypotheses to query the PDF-comparator or draw a sample. Thus, in the trivial implementation of our algorithm, we potentially need  $O(n \log n)$  bits to keep track of the indices of the winners at every round. In this section, we explain how one can implement our algorithm in a memory-efficient manner, so keeping track of the winners' indices does not take more than  $O(n)$  at every round.
We start by the following assumption that there is natural ordering among the hypotheses:  $H_{1}, H_{2}, \ldots, H_{n}$ . In this way, for the first round we do not need to remember the set of indices, we only need to remember  $n$ . At every level, we group "adjacent" hypotheses together and preserve this natural ordering. More precisely, at round one with group size  $g_{1}$ , the groups are:
$$
\begin{array}{l} \mathcal {G} _ {1} := \left\{H _ {1}, H _ {2}, \dots , H _ {g _ {1}} \right\}, \\ \mathcal {G} _ {2} := \left\{H _ {g _ {1} + 1}, H _ {g _ {1} + 2}, \dots , H _ {2 g _ {1}} \right\}, \\ \end{array}
$$
$$
\mathcal {G} _ {\left\lceil \frac {n}{g _ {1}} \right\rceil} := \{H _ {\left\lfloor \frac {n - 1}{g _ {1}} \right\rfloor \cdot g _ {1} + 1}, H _ {\left\lfloor \frac {n - 1}{g _ {1}} \right\rfloor \cdot g _ {1} + 2}, \ldots , H _ {n} \}.
$$
For the next round, we keep a list of  $\lceil n / g_1\rceil$  winners. However, instead of fully writing down the index of the winners, we use  $O(\log (g_{1}))$  bits per each, and save the index of the winner within the group. For the next round, we partition the hypotheses into groups of size  $g_{2}$  keeping the "adjacent" hypotheses in the same group. We repeat the same process to store the winners. However now, each winner are "representing"  $O(g_{1}\cdot g_{2})$  hypotheses, so we need  $O(\log (g_{1}\cdot g_{2}))$  bits to store its index. We continue this approach. At round  $\ell >1$ , we have a list of  $s_\ell = O(n / \prod_i^{\ell -1}g_i)$  hypotheses, and we require  $O(\log \prod_{i}^{\ell -1}g_{i})$  many bits to do so. Thus, at every step we need  $O(n)$  bits of memory.
Note that at every point in the process, it is easy to compute the actual index from the stored index. If right before starting round  $i$ , the  $j$ -th index in the list is  $k$ , then the true index is:
$$
\operatorname {N E W I N D E X} (i, j, k) = (j - 1) \cdot \prod_ {i} ^ {\ell - 1} g _ {i} + k.
$$
B.3 Filtering acceptable hypotheses
In this section, we describe a filtering algorithm that allows us to pick an acceptable hypothesis with a modest chance. Our algorithm is simple: we draw roughly  $n' \ll n$  hypothesis at random. If one of these hypothesis is OPT-close, and there is no decent hypothesis among them, Algorithm 3 returns an excellent hypothesis. On the other hand, if there are a lot of decent hypotheses, then there is a modest chance that a randomly selected hypothesis is decent, hence an acceptable one.
In the description of Algorithm 3, we assume we have a class of hypotheses of size  $n$ , and roughly  $O(n)$  bits of memory. However, we aim to use this algorithm in a setting that we can process  $n'$  random hypotheses in roughly  $O(n')$  bits of memory. The primary challenge here is that we need roughly  $O(n' \log n)$  many bits to memorize which hypotheses are participating, and we cannot use the "natural ordering" of the hypotheses that we use in the proof of Theorem 5. Thus, we are looking for a mapping that maps indices in  $[n']$  to a random set of indices in  $[n]$  in a memory-efficient manner. To do so, we relax our requirement regarding that we need to draw  $n'$  random hypotheses uniformly from  $\mathcal{H}$ . Instead, we are looking for a set of random pairwise independent indices while finding the index of the selected elements requires small amount of memory. This relaxation gives us a process that filters acceptable hypotheses:
Theorem 6. Suppose we have a set of  $n$  hypotheses  $\mathcal{H} = \{H_1, H_2, \dots, H_n\}$  and an unknown hypothesis  $P$  which we wish to properly learn in  $\mathcal{H}$ . For every positive integer  $n' < n$ , there exists an algorithm that uses  $O(n' \cdot \log(1/\epsilon))$  bits of memory and  $O((\log n)/\epsilon^2)$  samples from  $P$ , and it outputs  $\hat{H}$  that is excellent or decent with the probability:
$$
\mathbf {P r} \left[ \hat {H} i s e x c e l l e n t o r d e c e n t. \right] \geq \min  \left(\frac {n ^ {\prime}}{1 6 n}, \frac {1}{4 n ^ {\prime}}\right).
$$
Proof. First, we start off by describing how we randomly pick  $n'$  hypotheses from the set of  $n$  hypotheses. We use a standard technique for generating pairwise independent random indices described in [Vad12, Chapter 3.5]. The following is the description of a randomized mapping that for any given index  $x \in [n']$  it gives an index  $i$  in  $[n]$  that can be computed in a memory efficient manner.
Let  $q$  be a prime number between  $n$  and  $2n$  ( $n < q < 2n$ ). Such  $q$  always exists via the Bertrand-Chebyshev theorem for any  $n \geq 2$ . Without loss of generality, we assume we have  $q$  hypotheses by adding fake hypotheses to  $\mathcal{H}$  that lose to any other hypothesis. Now, consider the finite field of  $\mathbb{Z}_q$ . Proposition 3.24 in [Vad12] implies that if we use two random numbers  $a$  and  $b$  in  $\mathbb{Z}_q$ , then the following mapping generates a set of pairwise independent indices:
$$
f _ {a, b} (x) := a \cdot x + b (\mathrm {m o d} q).
$$
More precisely, by iterating  $x$  from 1 to  $n'$ , this randomized mapping selects  $n'$  hypotheses with the following indices:  $f_{a,b}(1), f_{a,b}(2), \ldots, f_{a,b}(n')$ . Note that to compute this mapping, we only need to memorize  $a$  and  $b$ , which requires  $O(\log n)$  bits, significantly smaller compared to  $O(n' \log n)$  which we would need to memorize  $n'$  random indices. Using the pairwise independence of the indices generated by this mapping, it is not too difficult to show that with some modest chance we pick exactly one excellent hypothesis and no decent hypothesis with this mapping.
Lemma B.2. For a prime  $q$ , assume we have a class of  $q$  hypotheses that contains  $t_e \geq 1$  excellent hypotheses and  $t_d$  decent hypotheses. Let  $a \neq 0$  and  $b$  be two random numbers in  $\mathbb{Z}_q$  selected uniformly at random. Let  $\mathcal{H}'$  be the set of following hypotheses:  $\{H_{f_{a,b}(x)} : x \in [n']\}$  for which  $f_{a,b}(x) \coloneqq a \cdot x + b \pmod{q}$ . If  $(t_e + t_d) / q \leq 1/(2n')$ , then the probability that  $\mathcal{H}'$  contains exactly one excellent hypothesis and no decent hypothesis is at least  $n' \cdot t_e / (2q)$ .
See Section D for the proof of this lemma.
Without loss of generality, assume  $H_0$  is  $H_{q}$
Algorithm 4 Filtering Algorithm
1: procedure HYPOTHESES-FILTER  $(\mathcal{H},n,\epsilon ,$  sample access to  $P$    
2:  $C\gets$  toss up a coin.   
3: if  $C$  is head then   
4: Draw a random  $i$  in [n]   
5:  $\hat{H}\leftarrow H_{i}$    
6: else   
7:  $q\gets$  the first prime number that is at least  $n$    
8:  $a\gets$  a random number in  $[q - 1]$    
9:  $b\gets$  a random number in [q]   
10:  $\mathcal{H}^{\prime}\gets \{H_{a\cdot x + b (\mathrm{mod} q)}\mid x\in [n^{\prime}]\}$    
11:  $\hat{H}\gets$  HYPOTHESIS-SELECTION-NO-DECENT  $(\mathcal{H}^{\prime},n^{\prime},\epsilon ,0.5)$    
12: Output  $\hat{H}$
Figure 3: The sketch of the filtering algorithm
The algorithm: Assume  $\mathcal{H}$  contains  $t_e \geq 1$  prefect hypotheses and  $t_d$  decent hypotheses. The algorithm is simple: with probability a half we set  $\hat{H}$  to be a random hypothesis from  $\mathcal{H}$ . And, with probability a half we use the randomize mapping to pick  $n'$  hypotheses and run Algorithm 3 with parameters  $\delta = 0.5$  and  $r = 2$  on them. And, we output the output of that algorithm. Figure 3 shows a visual representation of the filtering algorithm.
The probability of getting an acceptable hypothesis by picking a random one is  $(t_e + t_d) / n$ . Note that if this quantity is at least  $1/2n'$ , the statement of the lemma is true. Thus, assume  $(t_e + t_d) / n$  is less than  $1/2n'$  which implies that  $(t_e + t_d) / q$  is less than  $1/2n'$ . Given this condition, by the above lemma, our randomized mapping selects a set of hypotheses with only one excellent hypothesis and no other acceptable hypotheses with probability at least  $n' \cdot t_e / (2q)$ . And, since we set  $\delta$  to a half, with probability at least a half, Algorithm 3 will choose the excellent hypothesis as its output. Using that  $q < 2n$  and  $\mathbf{Pr}[C = \text{tail}] = 1/2$ , with probability at least  $n' \cdot t_e / (8q) \leq n' / (16n)$ ,  $\hat{H}$  is the excellent hypothesis as desired in the statement. Hence, the proof is complete.
B.4 Putting everything together
Proof. Here, we combine the filtering algorithm and the random ladder tournament to prove the upper bound. We first use the filtering algorithm, Algorithm 4, to increase the probability of getting an acceptable hypothesis  $(3\Gamma + \epsilon' - \text{close } P)$  with parameter  $\epsilon' = \epsilon / 4$ . Then, we use the random ladder tournament, Algorithm 2, to select an acceptable hypothesis among the hypotheses that have passed the filtering. Note that in the random ladder tournament, we need to use a new  $\Gamma'$  parameter, that is,  $3\Gamma + \epsilon'$ .
Here, we use the memory bounds provided in Theorem 6 and Lemma 3.1 for these two algorithms. Fix two small constants  $c_0, c_1 \leq 1$  that we determine later. Let  $t$  be the largest integer such that:
$$
t ^ {2} \log \left(\frac {\log n}{\epsilon}\right) t + t \log n \leq c _ {0} b.
$$
Note that given our lower bound for  $b$ , one can find a  $t$  that is at least two. Set  $n'$  as follows:
$$
n ^ {\prime} := \min \left(\frac {c _ {1} \cdot b}{\log (1 / \epsilon)}, 2 \sqrt {n}\right).
$$
We consider two cases in the following:
Case 1:  $n' < 2\sqrt{n}$ . In this case,  $n'$  is equal to  $c_1 \cdot b / \log(1/\epsilon)$ . One can run Algorithm 4 and find an acceptable hypothesis with probability  $p_0 := \min(n'/(16n), 1/(4n')) = n'/(16n)$ . Now in this case, we invoke Algorithm 4  $O(1/p_0)$  times, and in each round we use  $O(\log n/\epsilon^2)$  samples. Furthermore, the random ladder tournament uses  $O((\log p_0^{-1})/(tp_0\epsilon^2))$  samples. Therefore overall, our sample complexity is:
$$
\begin{array}{l} O \left(\frac {1}{p _ {0}} \cdot \left(1 + \frac {1}{t}\right) \cdot \frac {\log n}{\epsilon^ {2}}\right) = O \left(\frac {1}{p _ {0}} \cdot \frac {\log n}{\epsilon^ {2}}\right) = O \left(\frac {n}{n ^ {\prime}} \cdot \frac {\log n}{\epsilon^ {2}}\right) \\ = O \left(\frac {1}{p _ {0}} \cdot \frac {\log n}{\epsilon^ {2}}\right) = O \left(\frac {n \log n}{b \epsilon^ {2}} \cdot \log \frac {1}{\epsilon}\right). \\ \end{array}
$$
Case 2:  $n' = 2\sqrt{n}$ . Similar to the previous case, we would like to run Algorithm 4 and find an acceptable hypothesis with probability  $p_0 \coloneqq \min(n'/(16n), 1/(4n')) = 1/(4n')$ . Now, since  $n' \leq c_1 \cdot b / \log(1/\epsilon)$ , we may be able to run multiple instances of Algorithm 4 in parallel while using the same number of samples. Let  $r = c_1 \cdot b / (n' \cdot \log(1/\epsilon))$ . Now, the number of samples we use for this stage is:
$$
O \left(\frac {1}{r p _ {0}} \cdot \frac {\log n}{\epsilon^ {2}}\right) = O \left(\frac {n ^ {\prime}}{r} \cdot \frac {\log n}{\epsilon^ {2}}\right) = O \left(\frac {\log (1 / \epsilon) \cdot \log n}{b \epsilon^ {2}}\right).
$$
Now, we focus on the number of samples we need for the random ladder tournament. Note that  $t$  should be:
$$
t = \Theta \left(\min \left(\frac {b}{\log n}, \sqrt {\frac {b}{\log ((\log n) / \epsilon)}}\right)\right).
$$
Thus, we use the following amount of samples:
$$
\begin{array}{l} O \left(\frac {1}{t p _ {0}} \cdot \frac {\log n}{\epsilon^ {2}}\right) = O \left(\frac {\sqrt {n}}{b \cdot \min \left((\log n) ^ {- 1} , \left(\sqrt {b} \cdot \log (\log n) / \epsilon\right)\right) ^ {- 1}}\right) \cdot \frac {\log n}{\epsilon^ {2}}\left. \right) \\ = O \left(\frac {\max \left(\log n , \sqrt {b} \log \left((\log n) / \epsilon\right)\right)}{\sqrt {n}} \cdot \frac {n \log n}{b \epsilon^ {2}}\right) = O \left(\frac {n \log n}{b \epsilon^ {2}} \log \left(\frac {\log n}{\epsilon}\right)\right) \\ \end{array}
$$
Putting these two cases together, we obtain the desire sample-memory tradeoffs:
$$
\# \text {s a m p l e s} \cdot \# \text {b i t s} = O \left(\frac {n \log n}{\epsilon^ {2}} \log \left(\frac {\log n}{\epsilon^ {2}}\right)\right).
$$
Regarding the accuracy guarantee, it is not hard to see that in the filtration algorithm, with a modest probability, we pick a distribution that is  $(3\Gamma + \epsilon')$ -close to  $P$ . Now, we run the random ladder tournament with a new  $\Gamma' = 3\Gamma + \epsilon'$ . Then, we get a hypothesis with accuracy:
$$
3 \Gamma^ {\prime} + \epsilon^ {\prime} = 3 (3 \Gamma + \epsilon^ {\prime}) + \epsilon^ {\prime} = 9 \Gamma + 4 \epsilon^ {\prime} = 9 \Gamma + \epsilon .
$$
Hence, the proof is complete.
C Proper learner without promise
Throughout this paper, we mainly focus on proper learners with promise, for which the algorithm is given a parameter  $\Gamma$ , and we are promised that OPT is at most  $\Gamma$ . In this section, we focus on the case where no such information is available to the algorithm. First, we formally define a proper learner without promise.
Definition C.1. Suppose  $\mathcal{A}$  has sample access to an unknown distribution  $P$ . And, it can query the probabilities of the Scheffé sets according to each  $H_{i} \in \mathcal{H}$  and a PDF-comparator for every pair of hypotheses in  $\mathcal{H}$ . We say algorithm  $\mathcal{A}$  is an  $(\alpha, \epsilon, \delta)$ -proper learner for  $\mathcal{H}$  if the following holds: for every distribution  $P$ , and for every class of  $n$  distributions  $\mathcal{H}$ , with probability at least  $1 - \delta$  over its input sample (drawn i.i.d. from  $P$ ) and internal coin tosses,  $\mathcal{A}$  outputs a distribution  $\hat{H} \in \mathcal{H}$  such that:
$$
\| P - \hat {H} \| _ {T V} \leq \alpha \cdot \mathrm {O P T} + \epsilon .
$$
Here, we provide a reduction from a proper learner without promise to a proper learner with promise. At a high level, we perform a binary search over possible values of  $\Gamma$  in  $(0,1)$ . At every step, we try a value of  $\Gamma_{m}$  and run  $\mathcal{A}$  with  $\Gamma = \Gamma_{m}$ . The main challenge is that regardless of  $\Gamma_{m}$  being at least OPT or not,  $\mathcal{A}$  may return a hypothesis that seems close to  $P$ , and it is hard to refute that the suggested value of  $\Gamma$  is at least OPT. For this reason, we look into the output of  $\mathcal{A}$ , say  $\tilde{H}$ , and see if we find evidence that  $\tilde{H}$  is far from  $P$ . We check the distance between  $\tilde{H}$  and  $P$  on every Scheffé set of  $\tilde{H}$ . More precisely, we check if  $W(\tilde{H})$  is larger than  $(\alpha \cdot \mathrm{OPT} + \epsilon)$  where  $W(H)$  is defined as follows for every hypothesis  $H_{i}$ :
$$
W (H _ {i}) := \max  _ {j \in [ n ] \backslash \{i \}} \left| H _ {i} \left(\mathcal {S} (H _ {i}, H _ {j})\right) - P \left(\mathcal {S} (H _ {i}, H _ {j})\right) \right|.
$$
Now, if  $W(\tilde{H})$  is larger than  $(\alpha \cdot \mathrm{OPT} + \epsilon)$ , then one can imply that  $\tilde{H}$  is not  $(\alpha \cdot \mathrm{OPT} + \epsilon)$ -close to  $P$ , therefore,  $\Gamma_{m}$  must have been less than OPT. On the other hand, if  $W(\tilde{H})$  is at most  $(\alpha \cdot \mathrm{OPT} + \epsilon)$ , one can show  $\tilde{H}$  is not too far away from  $P$  (even when  $\Gamma_{m}$  is less than OPT). Putting these observations together and accounting for errors in estimation of  $W(\tilde{H})$ , we obtain a reduction that is described in Algorithm 5, and we prove its accuracy in Theorem 7.
Algorithm 5 Reduction from a proper learner without promise to a proper learner with promise
1: procedure  $\mathcal{B}(\mathcal{H},\alpha ,\epsilon ,\epsilon^{\prime},\delta ,\delta^{\prime}$  sample access to  $P$  , and oracle access to  $\mathcal{A}$    
2:  $\Gamma_{\ell}\gets 0$    
3:  $\Gamma_h\gets 1$    
4:  $\hat{H}\gets \mathcal{A}(\mathcal{H},\Gamma_h,\epsilon ,\delta)$    
5: while  $\Gamma_h - \Gamma_\ell >\epsilon$  do   
6:  $\Gamma_m\leftarrow \frac{\Gamma_h + \Gamma_\ell}{2}$    
7:  $\tilde{H}\gets \mathcal{A}(\mathcal{H},\Gamma_m,\epsilon ,\delta)$    
8:  $\tilde{W} (\tilde{H})\gets$  Estimate  $W(\tilde{H})$  with error at most  $\epsilon^\prime$  and with probability at least  $1 - \delta^{\prime}$    
9: if  $\tilde{W} (\tilde{H})\leq \alpha \Gamma_m + \epsilon +\epsilon '$  then   
10:  $\Gamma_h\gets \Gamma_m$    
11:  $\hat{H}\gets \tilde{H}$    
12: else   
13:  $\Gamma_{\ell}\gets \Gamma_{m}$    
14: Output  $\hat{H}$
Theorem 7. Fix five parameters  $\epsilon, \epsilon', \delta, \delta' \in (0,1)$ , and  $\alpha \geq 1$ . Assume that  $\mathcal{A}$  is an  $(\alpha, \epsilon, \delta)$ -proper learner (with promise). Suppose that we can estimate the probabilities of all the Scheffé sets according to  $P$  with additive error at most  $\epsilon'$  and with probability at least  $1 - \delta'$ . Then,  $\mathcal{B}$ , described in Algorithm 5, is an  $(\alpha + 2, (\alpha + 1)\epsilon + 2\epsilon', t \cdot \delta + \delta')$  proper learner (without promise) where  $t$  is defined as  $\lceil \log_2(1/\epsilon) \rceil + 1$ .
Proof. Our goal here is to show that for the output hypothesis  $\hat{H}$  with probability at least  $1 - t\cdot \delta +\delta^{\prime}$ , we have:
$$
\left\| \hat {H} - P \right\| _ {\mathrm {T V}} \leq (\alpha + 2) \mathrm {O P T} + (\alpha + 1) \epsilon + 2 \epsilon^ {\prime}. \tag {4}
$$
Observe that at every iteration of the while loop in the algorithm,  $\Gamma_h - \Gamma_\ell$  is divided by two. Thus, the while loop takes  $\lceil \log_2(1 / \epsilon) \rceil$  iterations. Thus,  $t$  is the number of times that we invoke  $\mathcal{A}$  in Algorithm 5. Each time  $\mathcal{A}$  works as expected with probability  $1 - \delta$ . Hence, using the union bound, with probability  $1 - t \cdot \delta$ , we can assume all invocations of  $\mathcal{A}$  result in correct answers. In addition, with probability  $1 - \delta'$ , all the estimates we use for  $P\left(S(\hat{H}, H_i)\right)$  in Line 8 are accurate up to an additive error  $\epsilon'$ . Hence,  $\left|\tilde{W}(\tilde{H}) - W(\tilde{H})\right|$  is at most  $\epsilon'$  with probability at least  $1 - \delta'$ . Hence, by the union bound, the following event happens with probability at least  $1 - (t \cdot \delta + \delta')$ : In every invocation of  $\mathcal{A}$ , if OPT is below the given parameter  $\Gamma$ , then  $\mathcal{A}$  returns a hypothesis,  $\tilde{H}$ , that is  $(\alpha \Gamma + \epsilon)$ -close to  $P$ . And,  $\tilde{W}(\tilde{H})$  is at most  $\alpha \Gamma + \epsilon + \epsilon'$ . Hence, for the rest of this proof, we assume this event holds, and we prove Equation 4.
Consider  $\Gamma_h$  and  $\Gamma_{\ell}$  in the final iteration of while loop in Algorithm 5. Observe that the output  $\hat{H}$  is what  $\mathcal{A}$  has found when it was invoked with input  $\Gamma = \Gamma_h$ . Now, we consider two cases as follows:
Case 1: OPT  $\leq \Gamma_h$ : In this case,  $\hat{H}$  is  $(\alpha \cdot \Gamma_h + \epsilon)$ -close to  $P$  by our assumption that  $\mathcal{A}$  works as expected. Thus,  $\| \hat{H} - P \|_{\mathrm{TV}}$  is at most  $\alpha \cdot \Gamma_h + \epsilon$ . This upper bound is written based on  $\Gamma_h$ , while we aim to find an upper bound on the distance that only contains OPT. For this purpose, we need to use the fact that why the binary search algorithm stoped at this particular  $\Gamma_h$ . For  $\Gamma_\ell$ , we have two possibilities: either  $\Gamma_\ell$  is zero; Or, we have run  $\mathcal{A}$  with  $\Gamma = \Gamma_\ell$ , and we have not found a desired  $\tilde{H}$ .
In both cases,  $\Gamma_{\ell}$  is a lower bound on OPT. Thus, we have:
$$
O P T \geq \Gamma_ {\ell} \geq \Gamma_ {h} - \epsilon .
$$
Now, using this relationship between  $\Gamma_h$  and OPT, we obtain:
$$
\| \hat {H} - P \| _ {\mathrm {T V}} \leq \alpha \cdot \Gamma_ {h} + \epsilon \leq \alpha \cdot \mathsf {O P T} + (\alpha + 1) \epsilon ,
$$
implying Equation 4.
Case 2:  $\mathsf{OPT} > \Gamma_h$ : In this case, clearly,  $\Gamma_h$  cannot be one (since OPT is at most one). Thus,  $\Gamma_h$  was set in Line 10 implying that  $\tilde{W}(\hat{H})$  is at most  $\alpha \Gamma_h + \epsilon + \epsilon'$ . Since we assumed the estimation of  $W(\hat{H})$  was accurate up to error  $\epsilon'$ , we have:
$$
W (\hat {H}) \leq \tilde {W} (\hat {H}) + \epsilon^ {\prime} \leq \alpha \Gamma_ {h} + \epsilon + 2 \epsilon^ {\prime}.
$$
Given this bound on  $W(\hat{H})$ , we show that  $\hat{H}$  cannot be far from  $P$ . Let  $H^{*}$  be a hypothesis in  $\mathcal{H}$  that is OPT-close to  $P$ . Then, we use the definition of the Scheffé sets and the triangle inequality to bound the distance between  $\hat{H}$  and  $P$ :
$$
\begin{array}{l} \| \hat {H} - P \| _ {\mathrm {T V}} \leq \| \hat {H} - H ^ {*} \| _ {\mathrm {T V}} + \| H ^ {*} - P \| _ {\mathrm {T V}} \quad (\text {b y}) \\ = \left| \hat {H} \left(\mathcal {S} (\hat {H}, H ^ {*})\right) - H ^ {*} \left(\mathcal {S} (\hat {H}, H ^ {*})\right) \right| + \| H ^ {*} - P \| _ {\mathrm {T V}} \text {(b y d f n . o f S c h e f f e s e t)} \\ \leq \left| \hat {H} \left(\mathcal {S} (\hat {H}, H ^ {*})\right) - P \left(\mathcal {S} (\hat {H}, H ^ {*})\right) \right| \\ + \left| P \left(\mathcal {S} (\hat {H}, H ^ {*})\right) - H ^ {*} (\mathcal {S} (\hat {H}, H ^ {*}) \right| + \| H ^ {*} - P \| _ {\mathrm {T V}} \quad (\text {b y t r i a n g l e i n e q u a l i t y}) \\ \leq \left| \hat {H} \left(\mathcal {S} (\hat {H}, H ^ {*})\right) - P \left(\mathcal {S} (\hat {H}, H ^ {*})\right) \right| + 2 \| H ^ {*} - P \| _ {\mathrm {T V}}. \\ \end{array}
$$
Now, we use the definition of  $W(\hat{H})$  and conclude: