SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\left[\mathrm{PVLS}^{+}21\right]$  Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alche Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research(a report from the neurips 2019 reproducibility program). Journal of Machine Learning Research, 22(164):1-20, 2021. URL: http://jmlr.org/papers/v22/20-303.html.
[RY20] Prasad Raghavendra and Morris Yau. List decodable learning via sum of squares. In Proceedings of the Thirty-First Annual ACM-SIAM Symposium on Discrete Algorithms, SODA '20, page 161-180, USA, 2020. Society for Industrial and Applied Mathematics.
[SZ99] Michael E. Saks and Shiyu Zhou.  $\mathrm{BP_hSPACE(S)}\subseteq \mathrm{DSPACE(S^{3 / 2})}$ . J. Comput. Syst. Sci., 58(2):376-403, 1999.
$\left[\mathrm{VWDP}^{+}22\right]$  Jason Vander Woude, Peter Dixon, Aduri Pavan, Jamie Radcliffe, and N. V. Vinodchandran. Geometry of rounding. CoRR, abs/2211.02694, 2022. arXiv:2211.02694, doi:10.48550/arXiv.2211.02694.
A.1 Missing Proofs from Section 4
To establish Lemmas 4.1 and 4.3, we build on the work of  $(\mathrm{VWDP}^{+}22)$  and (DLPES02). We first introduce the necessary notion and definitions.
We recall the following notation from the main body of the paper. We use  $\mathrm{diam}_{\infty}$  to indicate the diameter of a set relative to the  $\ell_{\infty}$  norm and  $\overline{B}_{\varepsilon}^{\infty}(\vec{p})$  to represent the closed ball of radius  $\varepsilon$  centered at  $\vec{p}$  relative to the  $\ell_{\infty}$  norm. That is, in  $\mathbb{R}^d$  we have  $\overline{B}_{\varepsilon}^{\infty}(\vec{p}) = \prod_{i=1}^{d}[p_i - \varepsilon, p_i + \varepsilon]$ .
Lemma 4.1 is based on the construction of certain geometric partitions of  $\mathbb{R}^d$  called secluded partitions. Such partitions naturally induce deterministic rounding schemes which we use in the proof.
Let  $\mathcal{P}$  be a partition of  $\mathbb{R}^d$ . For a point  $\vec{p} \in \mathbb{R}^d$ , let  $N_{\varepsilon}(\vec{p})$  denote the set of members of the partitions that have a non-empty intersection with the  $\varepsilon$ -ball around  $\vec{p}$ . That is,
$$
N _ {\varepsilon} (\vec {p}) = \{X \in \mathbb {P} \mid \overline {{B}} _ {\varepsilon} ^ {\infty} (\vec {p})) \cap X \neq \emptyset \}
$$
Definition A.1 (Secluded Partition). Let  $\mathcal{P}$  be a partition of  $\mathbb{R}^d$ . We say that  $\mathcal{P}$  is  $(k,\varepsilon)$ -secluded, if for every point  $\vec{p} \in \mathbb{R}^d$ ,  $|N_{\varepsilon(\vec{p})}| \leq k$ .
The following theorem from  $(\mathrm{VWDP}^{+}22)$  gives an explicit construction of a secluded partition with desired parameters where each member of the partition is a hypercube. For such partitions, we use the following notation. For every  $\vec{p}\in \mathbb{R}^d$ , if  $\vec{p}\in \mathbb{P}$ , then the representative of  $\vec{p}$ ,  $rep(\vec{p})$ , is the center of the hypercube  $X$ .
Theorem A.2. For each  $d \in \mathbb{N}$ , there exists a  $(d + 1, \frac{1}{2d})$ -secluded partition, where each member of the partition is a unit hypercube. Moreover, the partition is efficiently computable: Given an arbitrary point  $\vec{x} \in \mathbb{R}^d$ , its representative can be computed in time polynomial in  $d$ .
A.1.1 Proof of Lemma 4.1
Lemma A.3 (Lemma 4.1). Let  $d \in \mathbb{N}$  and  $\varepsilon \in (0, \infty)$ . Let  $\varepsilon_0 = \frac{\varepsilon}{2d}$ . There is an efficiently computable function  $f_{\varepsilon} : \mathbb{R}^d \to \mathbb{R}^d$  with the following two properties:
1. For any  $x \in \mathbb{R}^d$  and any  $\hat{x} \in \overline{B}_{\varepsilon_0}^\infty(x)$  it holds that  $f_{\varepsilon}(\hat{x}) \in \overline{B}_{\varepsilon}^\infty(x)$ .
2. For any  $x \in \mathbb{R}^d$  the set  $\left\{f_{\varepsilon}(\hat{x}) \colon \hat{x} \in \overline{B}_{\varepsilon_0}^\infty(x)\right\}$  has cardinality at most  $d + 1$ .
As explained in the main body, intuitively, item (1) states that if  $\hat{x}$  is an  $\varepsilon_0$ -approximation of  $x$ , then  $f_{\varepsilon}(\hat{x})$  is an  $\varepsilon$ -approximation of  $x$ , and item (2) states that  $f_{\varepsilon}$  maps every  $\varepsilon_0$ -approximation of  $x$  to one of at most  $d + 1$  possible values.
Proof. A high-level idea behind the proof is explained in Figure 1. We scale the  $(d + 1, \frac{1}{2d})$ -secluded unit hypercube partition by  $\varepsilon$  so that each partition member is a hypercube with side length  $\varepsilon$ . Now, for a point  $x$ , the ball  $\overline{B}_{\varepsilon_0}^\infty(x)$  intersects at most  $d + 1$  hypercubes. Consider a point  $\hat{x}_1 \in \overline{B}_{\varepsilon_0}^\infty(x)$ , it is rounded to  $c_1$  (center of the hypercube it resides in). Note that  $c_1$  lies in the ball of radius  $\varepsilon$  around  $x$ , this is because distance from  $x$  to  $\hat{x}_1$  is at most  $\varepsilon_0$  and the distance from  $\hat{x}_1$  to  $c_1$  is at most  $\varepsilon/2$ . By triangle inequality  $c_1$  belongs to  $\overline{B}_{\varepsilon}^\infty(x)$ . We now provide formal proof.
Let  $\mathcal{P}$  be the  $(d + 1, \frac{1}{2d})$ -secluded partition given by Theorem A.2. Thus  $\mathcal{P}$  consists of unit cubes  $[0,1)^d$  with the property that for any point  $\vec{p} \in \mathbb{R}^d$  the closed cube of side length  $1 / d$  centered at  $\vec{p}$  (i.e.  $\overline{B}_{\frac{1}{2d}}^\infty(\vec{p})$ ) intersects at most  $d + 1$  members/cubes of  $\mathcal{P}$ .
We first define a rounding function  $f: \mathbb{R}^d \to \mathbb{R}^d$  as follows: for every  $x \in \mathbb{R}^d$ ,  $f(x) = \text{rep}(x)$ .
Observe that the rounding function  $f$  has the following two properties. (1) For every  $x \in \mathbb{R}^d$ ,  $\| f(x) - x \|_{\infty} \leq \frac{1}{2}$ . This is because every point  $x$  is mapped via  $f$  to its representative, which is the center of the unit cube in which it lies. (2) For any point  $\vec{p} \in \mathbb{R}^d$ , the set  $\left\{f(x) : x \in \overline{B}_{\frac{1}{2d}}^{\infty}(\vec{p})\right\}$  has
Figure 1: Illustration of proof of Lemma 4.1 for  $d = 2$ .
cardinality at most  $d + 1$ . This is because  $\overline{B}_{\frac{1}{2d}}^{\infty}(\vec{p})$  intersects at most  $d + 1$  hypercubes of  $\mathcal{P}$  and for every hypercube  $X$ , all the points in  $X$  are mapped to its center by  $f$ .
The function  $f$  only gives an  $\frac{1}{2}$ -approximation guarantee. In order to get any  $\varepsilon$ -approximation guarantee, we scale  $f$  appropriately.  $f_{\varepsilon}$  is this scaled version of  $f$ .
Define the function  $f_{\varepsilon}:\mathbb{R}^{d}\to \mathbb{R}^{d}$  as follows: for every  $\hat{x}\in \mathbb{R}^d$ ,  $f_{\varepsilon}(\hat{x}) = \varepsilon \cdot f(\frac{1}{\varepsilon}\hat{x})$ . The efficient computability of  $f_{\varepsilon}$  comes from the efficient computability of  $f$ .
We first establish that  $f_{\varepsilon}$  has property (1) stated in the Lemma. Let  $x \in \mathbb{R}^d$  and  $\hat{x} \in \overline{B}_{\varepsilon_0}^\infty(x)$ . Then we have the following (justifications will follow):
$$
\begin{array}{l} \left\| \frac {1}{\varepsilon} \cdot f _ {\varepsilon} (\hat {x}) - \frac {1}{\varepsilon} x \right\| _ {\infty} = \left\| f \left(\frac {1}{\varepsilon} \hat {x}\right) - \frac {1}{\varepsilon} x \right\| _ {\infty} \\ \leq \left\| f \left(\frac {1}{\varepsilon} \hat {x}\right) - \frac {1}{\varepsilon} \hat {x} \right\| _ {\infty} + \left\| \frac {1}{\varepsilon} \hat {x} - \frac {1}{\varepsilon} x \right\| _ {\infty} \\ \leq \left\| f \left(\frac {1}{\varepsilon} \hat {x}\right) - \frac {1}{\varepsilon} \hat {x} \right\| _ {\infty} + \frac {1}{\varepsilon} \| \hat {x} - x \| _ {\infty} \\ \leq \frac {1}{2} + \frac {1}{\varepsilon} \varepsilon_ {0} \\ = \frac {1}{2} + \frac {1}{2 d} \leq 1 \\ \end{array}
$$
The first line is by the definition of  $f_{\varepsilon}$ , the second is the triangle inequality, the third is scaling of norms, the fourth uses the property of  $f$  that points are not mapped a distance more than  $\frac{1}{2}$  along with the hypothesis that  $\hat{x} \in \overline{B}_{\varepsilon_0}^\infty(x)$ , the fifth uses the definition of  $\varepsilon_0$ , and the sixth uses the fact that  $d \geq 1$ .
Scaling both sides by  $\varepsilon$  and using the scaling of norms, the above gives us  $\| f_{\varepsilon}(\hat{x}) - x\|_{\infty}\leq \varepsilon$  which proves property (1) of the lemma.
To see that  $f_{\varepsilon}$  has property (2), let  $x \in \mathbb{R}^d$ . We have the following set equalities:
$$
\begin{array}{l} \left\{f _ {\varepsilon} (\hat {x}) \colon \hat {x} \in \overline {{B}} _ {\varepsilon_ {0}} ^ {\infty} (x) \right\} = \left\{\varepsilon \cdot f \left(\frac {1}{\varepsilon} \hat {x}\right) \colon \hat {x} \in \overline {{B}} _ {\varepsilon_ {0}} ^ {\infty} (x) \right\} \\ = \left\{\varepsilon \cdot f (a) \colon a \in \overline {{B}} _ {\frac {1}{\varepsilon} \varepsilon_ {0}} ^ {\infty} (\frac {1}{\varepsilon} x) \right\} \\ = \left\{\varepsilon \cdot f (a) \colon a \in \overline {{B}} _ {\frac {1}{2 d}} ^ {\infty} (\frac {1}{\varepsilon} x) \right\} \\ \end{array}
$$
The first line is from the definition of  $f_{\varepsilon}$ , the second line is from re-scaling, and the third is from the definition of  $\varepsilon_0$ .
Because  $f$  takes on at most  $d + 1$  distinct values on  $\overline{B}_{\frac{1}{2d}}^{\infty}\left(\frac{1}{\varepsilon} x\right)$ , the set above has cardinality at most  $d + 1$  which proves property (2) of the lemma.
A.1.2 Proof of Lemma 4.2
Lemma A.4 (Lemma 4.2). Let  $d \in \mathbb{N}$ ,  $\varepsilon_0 \in (0, \infty)$  and  $0 < \delta < 1$ . There is an efficiently computable deterministic function  $f: \{0, 1\}^{\ell} \times \mathbb{R}^{d} \to \mathbb{R}^{d}$  with the following property. For any  $x \in \mathbb{R}^{d}$ ,
$$
\operatorname * {P r} _ {r \in \{0, 1 \} ^ {\ell}} \left[ \exists x ^ {*} \in \overline {{B}} _ {\varepsilon} ^ {\infty} (x) \forall \hat {x} \in \overline {{B}} _ {\varepsilon_ {0}} ^ {\infty} (x): f (r, \hat {x}) = x ^ {*} \right] \geq 1 - \delta
$$
Proof. Partition each coordinate of  $\mathbb{R}^d$  into  $2\varepsilon_0$ -width intervals. The algorithm computing the function  $f$  does the following simple randomized rounding:
The function  $f:$  Choose a random integer  $r \in \{1 \dots 2^{\ell}\}$ . Note that  $r$  can be represented using  $\ell$  bits. Consider the  $i^{th}$  coordinate of  $\hat{x}$  denoted by  $\hat{x}[i]$ . Round  $\hat{x}[i]$  to the nearest  $k * (2\varepsilon_0)$  such that  $k \mod 2^{\ell} \equiv r$ .
Now we will prove that  $f$  satisfies the required properties.
First, we prove the approximation guarantee. Let  $x'$  denote the point in  $\mathbb{R}^d$  obtained after rounding each coordinate of  $\hat{x}$ . The  $k$ s satisfying  $k \mod 2^\ell \equiv r$  are  $2^\ell \cdot 2\varepsilon_0$  apart. Therefore,  $x'[i]$  is rounded by at most  $2^\ell \varepsilon_0$ . That is,  $|x'[i] - \hat{x}[i]| \leq 2^\ell \varepsilon_0 = \frac{\varepsilon_0 d}{\delta}$  for every  $i$ ,  $1 \leq i \leq d$ . Since  $\hat{x}$  is an  $\varepsilon_0$ -approximation (i.e., each coordinate  $\hat{x}[i]$  is within  $\varepsilon_0$  of the true value  $x[i]$ ), then each coordinate of  $x'$  is within  $(2^\ell + 1) \varepsilon_0$  of  $x[i]$ . Therefore  $x'$  is a  $(2^\ell + 1) \varepsilon_0$ -approximation of  $x[i]$ . Thus  $x' \in \overline{B}_{\varepsilon}^\infty(x)$  for any choice of  $r$ .
Now we establish that for  $\geq 1 - \delta$  fraction of  $r\in \{1\ldots 2^{\ell}\}$ , there exists  $x^{*}$  such every  $\hat{x}\in \overline{B}_{\varepsilon_0}^\infty (x)$  is rounded  $x^{*}$ . We argue this with respect to each coordinate and apply the union bound. Fix an  $x$  and a coordinate  $i$ . For  $x[i]$ , consider the  $\varepsilon_0$  interval around it.
Consider  $r$  from  $\{1 \ldots 2^{\ell}\}$ . When this  $r$  is chosen, then we round  $\hat{x}[i]$  to the closest  $k * (2\varepsilon_0)$  such that  $k \mod 2^{\ell} \equiv r$ . Let  $p_1^r, p_2^r, \ldots, p_j^r \ldots$  be the set of such points: more precisely  $p_j = (j2^l + r) * 2\varepsilon_0$ . Note that  $\hat{x}[i]$  is rounded to an  $p_j$  to some  $j$ . Let  $m_j^r$  denote the midpoint between  $p_j^r$  and  $p_{j+1}^r$ . I.e.,  $m^r = (p_j^r + p_{j+1}^r)/2$ . We call  $r$  'bad' for  $x[i]$  if  $x[i]$  is close to some  $m_j^r$ . That is,  $r$  is 'bad' if  $|x[i] - m_j^r| < \varepsilon_0$ . Note that for a bad  $r$  there exist  $\hat{x}_1$  and  $\hat{x}_2$  in  $\overline{B}_{\varepsilon_0}^\infty(x)$  so that their  $i^{th}$  coordinates are rounded to  $p_j^r$  and  $p_{j+1}^r$  respectively. The crucial point is that if  $r$  is 'not bad' for  $x[i]$ , then for every  $x' \in \overline{B}_{\varepsilon_0}^\infty(x)$ , there exists a canonical  $p^*$  such that  $x'[i]$  is rounded to  $p^*$ . We call  $r$  bad for  $x$ , if  $r$  is bad for  $x$ , if there exists at least one  $i, 1 \leq i \leq d$  such that  $r$  is bad for  $x[i]$ . With this, it follows that if  $r$  is not bad for  $x$ , then there exists a canonical  $x^*$  such that every  $x' \in \overline{B}_{\varepsilon_0}^\infty(x)$  is rounded to  $x^*$ .
With this, the goal is to bound the probability that a randomly chosen  $r$  is bad for  $x$ . For this, we first bound the probability that  $r$  is bad for  $x[i]$ . We will argue that there exists at most one bad  $r$  for  $x[i]$ . Suppose that there exist two numbers  $r_1$  and  $r_2$  that are both bad for  $x[i]$ . This means that  $|x[i] - m_{j_1}^{r_1}| < \varepsilon_0$  and  $|x[i] - m_{j_2}^{r_2}| < \varepsilon_0$  for some  $j_1$  and  $j_2$ . Thus by triangle inequality  $|m_{j_1}^{r_1} - m_{j_2}^{r_2}| < 2\varepsilon_0$ . However, note that  $|p_{j_1}^{r_1} - p_{j_2}^{r_2}|$  is  $|(j_1 - j_2)2^\ell + (r_1 - r_2)|2\varepsilon_0$ . Since  $r_1 \neq r_2$ , this value is at least  $2\varepsilon_0$ . This implies that the absolute value of difference between  $m_{j_1}^{r_1}$  and  $m_{j_2}^{r_2}$  is at least  $2\varepsilon$  leading to a contradiction.
Thus the probability that  $r$  is bad for  $x[i]$  is at most  $\frac{1}{2^{\ell}}$  and by the union bound the probability that  $r$  is bad for  $x$  is at most  $\frac{d}{2^{\ell}} \leq \delta$ . This completes the proof.
A.1.3 Proof of Lemma 4.3
The proof, which is based on Sperner/KKM Lemma, is present in  $(\mathrm{VWDP}^{+}22)$ . Since our setting is slightly different, for completeness we give a proof.
We first introduce the necessary definitions and notation.
Definition A.5 (Sperner/KKM Coloring). Let  $d \in \mathbb{N}$  and  $V = \{0,1\}^d$  denote a set of colors (which is exactly the set of vertices of  $[0,1]^d$  so that colors and vertices are identified). Let  $\chi :[0,1]^d \to V$
be a coloring function such that for any face  $F$  of  $[0,1]^d$  (of any dimension), for any  $x\in F$ , it holds that  $\chi (x)\in V(F)$  where  $V(F)$  is the vertex set of  $F$  (informally, the color of  $x$  is one of the vertices in the face  $F$ ). Such a function  $\chi$  will be called a Sperner/KKM coloring.
Theorem A.6 (Cubical Sperner/KKM lemma (DLPES02)). Let  $d \in \mathbb{N}$  and  $V = \{0,1\}^d$  and  $\chi : [0,1]^d \to V$  be a Sperner/KKM coloring. Then there exists a subset  $J \subset V$  with  $|J| = d + 1$  and a point  $\vec{y} \in [0,1]^d$  such that for all  $j \in J$ ,  $\vec{y} \in \overline{\chi^{-1}(j)}$  (informally, there is a point  $\vec{y}$  that is touching at least  $d + 1$  different colors).
We will need to relate partitions to Sperner/KKM coloring so that we can use the Sperner/KKM Lemma.
For any co-ordinate  $i$ , let  $\pi$  denote the standard projection map:  $\pi_i : [0,1]^d \to [0,1]$  defined by  $\pi_i(x) \stackrel{\mathrm{def}}{=} x_i$  which maps  $d$ -dimensional points to the  $i^{th}$  coordinate value. We extend this to sets:  $\pi_i(X) = \{\pi_i(x) : x \in X\}$ .
Definition A.7 (Non-Spanning partition). Let  $d \in \mathbb{N}$  and  $\mathcal{P}$  be a partition of  $[0,1]^d$ . We say that  $\mathcal{P}$  is a non-spanning partition if it holds for all  $X \in \mathcal{P}$  and for all  $i \in [d]$  that either  $\pi_i(X) \not\equiv 0$  or  $\pi_i(X) \not\geq 1$  (or both).
Next, we state a lemma that asserts that for any non-spanning partition, there is a Sperner/KKM coloring that respects the partition: that is every member gets the same color.
Lemma A.8 (Coloring Admission). Let  $d \in \mathbb{N}$ , and  $V = \{0,1\}^d$ , and  $\mathcal{P}$  a non-spanning partition of  $[0,1]^d$ . Then there exists a Sperner/KKM coloring  $\chi : [0,1]^d \to V$  such that for every  $X \in \mathcal{P}$ , for every  $x, y \in X$ ,  $\chi(x) = \chi(y)$ .
Now we are ready to prove the Lemma 4.3.
Lemma A.9. (Lemma 4.3) Let  $\mathcal{P}$  be a partition of  $[0,1]^d$  such that for each member  $X\in \mathcal{P}$ , it holds that  $\mathrm{diam}_{\infty}(X) < 1$ . Then there exists  $\vec{p}\in [0,1]^d$  such that for all  $\delta >0$  we have that  $\overline{B}_{\delta}^{\infty}(\vec{p})$  intersects at least  $d + 1$  members of  $\mathcal{P}$ .
Proof. Consider an arbitrary  $X \in \mathcal{P}$ . For each coordinate,  $i \in [d]$ , the set  $\{x_i : x \in X\}$  does not contain both 0 and 1 (if it did, this would demonstrate two points in  $X$  that are  $\ell_{\infty}$  distance at least 1 apart and contradict that  $\mathrm{diam}_{\infty}(X) < 1$ ). Thus,  $\mathcal{P}$  is by definition a non-spanning partition of  $[0,1]^d$ . Since  $\mathcal{P}$  is non-spanning, by Lemma A.8, there is a Sperner/KKM coloring where each point of  $[0,1]^d$  can be assigned one of  $2^d$ -many colors and for any member  $X \in \mathcal{P}$ , all points in  $X$  are assigned the same color. By Lemma A.6, there is a point  $\vec{p} \in [0,1]^d$  such that  $\vec{p}$  belongs to the closure of at least  $d + 1$  colors. Since every point of a partition has the same color, each of these  $d + 1$  colors corresponds to at least  $d + 1$  different partitions. From this, it follows that or any  $\delta > 0$ ,  $\overline{B}_{\delta}^{\infty}(\vec{p})$  intersects at least  $d + 1$  different members of  $\mathcal{P}$ .
A.2 Missing Proofs From Section 6
A.2.1 Proofs of Theorem 6.2, Theorem 6.3
Theorem A.10 (Theorem 6.2). Let  $\mathcal{H}$  be a concept class that is learnable with  $d$  non-adaptive statistical queries, then  $\mathcal{H}$  is  $(d + 1)$ -list replicably learnable. Furthermore, the sample complexity  $n = n(\nu, \delta)$  of the  $(d + 1)$ -list replicable algorithm is  $O\left(\frac{d^2}{\nu^2} \cdot \log \frac{d}{\delta}\right)$ , where  $\nu$  is the approximation error parameter of each statistical query oracle.
Proof. The proof is very similar to the proof of Theorem 5.2. Our replicable algorithm  $B$  works as follows. Let  $\varepsilon$  and  $\delta$  be input parameters and  $\mathcal{D}$  be a distribution and  $f\in \mathcal{H}$ . Let  $A$  be the statistical query learning algorithm for  $\mathcal{H}$ . Let  $STAT(D_f,\nu)$  be the statistical query oracle for this algorithm. Let  $\phi_1,\ldots ,\phi_d$  be the statistical queries made by  $A$ .
Let  $\vec{b} = \langle b[1], b[2], \ldots, b[d] \rangle$  where  $b[i] = E_{\langle x, y \rangle \in \mathcal{D}_f}[\phi_i(\langle x, y \rangle), 1 \leq i \leq d]$ . Set  $\varepsilon_0 = \frac{\nu}{2d}$ . The algorithm  $B$  first estimates the values  $b[i]$  up to an approximation error of  $\varepsilon_0$  with success probably  $1 - \delta / d$  for each query. Note that this can be done by a simple empirical estimation algorithm, that
uses a total of  $n = O\left(\frac{d^2}{\nu^2} \cdot \log \frac{d}{\delta}\right)$  samples. Let  $\vec{v}$  be the estimated vector. It follows that  $\vec{v} \in \overline{B}_{\varepsilon_0}^\infty(\vec{b})$  with probability at least  $1 - \delta$ . Note that different runs of the algorithm will output different  $\vec{v}$ .
Next, the algorithm  $B$  evaluates the deterministic function  $f_{\varepsilon}$  from Lemma 4.1 on input  $\vec{v}$ . Let  $\vec{u}$  be the output vector. Finally, the algorithm  $B$  simulates the statistical query algorithm  $A$  with  $\vec{u}[i]$  as the answer to the query  $\phi_i$ . By Lemma 4.1,  $\vec{u} \in \overline{B}_{\nu}^{\infty}(\vec{b})$ . Thus the error of the hypothesis output by the algorithm is at most  $\varepsilon$ . Since  $A$  is a deterministic algorithm the number of possible outputs only depends on the number of outputs of the function  $f_{\varepsilon}$ , more precisely the number of possible outputs is the size of the set  $\{f_{\varepsilon}(\vec{v}): v \in \overline{B}_{\varepsilon_0}^{\infty}(\vec{b})\}$  which is almost  $d + 1$ , by Lemma 4.1. Thus the total number of possible outputs of the algorithm  $B$  is at most  $d + 1$  with probability at least  $1 - \delta$ .
Theorem A.11 (Theorem 6.3). Let  $\mathcal{H}$  be a concept class that is learnable with  $d$  non-adaptive statistical queries, then  $\mathcal{H}$  is  $\lceil \log \frac{d}{\delta} \rceil$ -certificate replicably learnable. Furthermore, the sample complexity  $n = n(\nu, \delta)$  of this algorithm equals  $O\left(\frac{d^2}{\nu^2 \delta^2} \cdot \log \frac{d}{\delta}\right)$ , where  $\nu$  is the approximation error parameter of each statistical query oracle.
Proof. The proof is very similar to the proof of Theorem 5.4. Our replicable algorithm  $B$  works as follows, let  $\varepsilon$  and  $\delta$  be input parameters and  $\mathcal{D}$  be a distribution and  $f \in \mathcal{H}$ . Let  $A$  be the statistical query learning algorithm for  $\mathcal{H}$  that outputs a hypothesis  $h$  with approximation error  $e_{\mathcal{D}_f}(h) = \varepsilon$ . Let  $STAT(D_f, \nu)$  be the statistical query oracle for this algorithm. Let  $\phi_1, \ldots, \phi_d$  be the statistical queries made by  $A$ .
Let  $\vec{b} = \langle b[1], b[2], \dots, b[d] \rangle$ , where  $b[i] = E_{\langle x, y \rangle \in \mathcal{D}_f}[\phi_i(\langle x, y \rangle)]$ . Set  $\varepsilon_0 = \frac{\nu \delta}{2d}$ . The algorithm  $B$  first estimates the values  $b[i], 1 \leq i \leq d$  up to an additive approximation error of  $\varepsilon_0$  with success probably  $1 - \delta / d$  for each query. Note that this can be done by a simple empirical estimation algorithm that uses a total of  $n = O\left(\frac{d^2}{\nu^2 \delta^2} \cdot \log \frac{d}{\delta}\right)$  samples. Let  $\vec{v}$  be the estimated the vector. It follows that  $\vec{v} \in \overline{B}_{\varepsilon_0}^\infty(\vec{b})$  with probability at least  $1 - \delta$ . Next, the algorithm  $B$  evaluates the deterministic function  $f$  described in Lemma 4.2 with inputs  $r \in \{0, 1\}^\ell$  where  $\ell = \lceil \log \frac{d}{\delta} \rceil$  and  $\vec{v}$ . By Lemma 4.2 for at least  $1 - \delta$  fraction of the  $r$ 's, the function  $f$  outputs a canonical  $\vec{v^*} \in \overline{B}_{\nu}^\infty(\vec{b})$ . Finally, the algorithm  $B$  simulates the statistical query algorithm  $A$  with  $\vec{v^*}[i]$  as the answer to the query  $\phi_i$ . Since  $A$  is a deterministic algorithm it follows that our algorithm  $B$  is certificate replicable. Note that the certificate complexity is  $\ell = \lceil \log \frac{d}{\delta} \rceil$ .
The following theorem states how to convert adaptive statistical query learning algorithms into certificate replicable PAC learning algorithms. This result also appears in the work of (GKM21; ILPS22), though they did not state the certificate complexity. We explicitly state the result here.
Theorem A.12. ((GKM21; ILPS22))[Theorem 6.4] Let  $\mathcal{H}$  be a concept class that is learnable with  $d$  adaptive statistical queries, then  $\mathcal{H}$  is  $\lceil d\log \frac{d}{\delta}\rceil$ -certificate replicably learnable. Furthermore, the sample complexity of this algorithm equals  $O\left(\frac{d^3}{\nu^2\delta^2} \cdot \log \frac{d}{\delta}\right)$ , where  $\nu$  is the approximation error parameter of each statistical query oracle.