SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. The proof uses similar arguments as before. The main difference is that we will evaluate each query with an approximation error of  $\frac{\nu\delta}{d}$  with a probability error of  $d / \delta$ . This requires  $O\left(\frac{d^2}{\nu^2\delta^2} \cdot \log \frac{d}{\delta}\right)$  per query. We use a fresh set of certificate randomness for each such evaluation. Note that the length of the certificate for each query is  $\lceil \log d / \delta \rceil$ . Thus the total certificate complexity is  $\lceil d \log \frac{d}{\delta} \rceil$ .
A.2.2 Proof of Theorem 6.7
We first recall the definition of the concept class  $d$ -THRESHOLD.
Fix some  $d \in \mathbb{N}$ . Let  $X = [0,1]^d$ . For each value  $\vec{t} \in [0,1]^d$ , let  $h_{\vec{t}}: X \to \{0,1\}$  be the concept defined as follows:  $h_{\vec{t}}(\vec{x}) = 1$  if for every  $i \in [d]$  it holds that  $x_i \leq t_i$  and 0 otherwise. Let  $\mathcal{H}$  be the hypothesis class consisting of all such threshold concepts:  $\mathcal{H} = \{h_{\vec{t}} | \vec{t} \in [0,1]^d\}$ .
Theorem A.13 (Theorem 6.7). In the PAC model under the uniform distribution, there is a  $d + 1$ -list replicable algorithm for the  $d$ -THRESHOLD. Moreover, for any  $k < d + 1$ , there does not exist a  $k$ -list replicable algorithm for the concept class  $d$ -THRESHOLD under the uniform distribution. Thus its list complexity is exactly  $d + 1$ .
It is easy to see that  $d$ -THRESHOLD is learnable under the uniform distribution by making  $d$  non-adaptive statistical queries. Thus by Theorem 6.2,  $d$ -THRESHOLD under the uniform distribution admits a  $(d + 1)$ -list replicable algorithm. So we will focus on proving the lower bound which is stated as a separate theorem below.
Theorem A.14. For  $k < d + 1$ , there does not exist a  $k$ -list replicable algorithm for the  $d$ -THRESHOLD in the PAC model under uniform distribution.
The proof is similar to the proof of Theorem 5.5. The reason is that sampling  $d$ -many biased coins with bias vector  $\vec{b}$  is similar to obtaining a point  $\vec{x}$  uniformly at random from  $[0,1]^d$  and evaluating the threshold function  $h_{\vec{b}}$  on it—this corresponds to asking whether all of the coins were heads/1's. The two models differ though, because in the sample model for the  $d$ -COIN BIAS ESTIMATION PROBLEM, the algorithm sees for each coin whether it is heads or tails, but this information is not available in the PAC model for the  $d$ -THRESHOLD. Conversely, in the PAC model for the  $d$ -THRESHOLD, a random draw from  $[0,1]^d$  is available to the algorithm, but in the sample model for the  $d$ -COIN BIAS ESTIMATION PROBLEM the algorithm does not get this information.
Furthermore, there is the following additional complexity in the impossibility result for the  $d$ -THRESHOLD. In the  $d$ -COIN BIAS ESTIMATION PROBLEM, we said by definition that a collection of  $d$  coins parameterized by bias vector  $\vec{a}$  was an  $\varepsilon$ -approximation to a collection of  $d$  coins parameterized by bias vector  $\vec{b}$  if and only if  $\| \vec{b} - \vec{a} \|_{\infty} \leq \varepsilon$ , and we used this norm in the proofs. However, the notion of  $\varepsilon$ -approximation in the PAC model is quite different than this. It is possible to have hypotheses  $h_{\vec{a}}$  and  $h_{\vec{b}}$  in the  $d$ -THRESHOLD such that  $\| \vec{b} - \vec{a} \|_{\infty} > \varepsilon$  but with respect to some distribution  $\mathcal{D}_X$  on the domain  $X$  we have  $err_{\mathcal{D}_X}(h_{\vec{a}}, h_{\vec{b}}) \leq \varepsilon$ . For example, if  $\mathcal{D}_X$  is the uniform distribution on  $X = [0,1]^d$  and  $\vec{a} = \vec{0}$  and  $\vec{b}$  is the first standard basis vector  $\vec{b} = \langle 1,0,\dots,0\rangle$ , and  $\varepsilon = \frac{1}{2}$ , then  $\| \vec{b} - \vec{a} \|_{\infty} = 1 > \varepsilon$ , but  $err_{\mathcal{D}_X}(h_{\vec{a}}, h_{\vec{b}}) = 0 \leq \varepsilon$  because  $h_{\vec{a}}(\vec{x}) \neq h_{\vec{b}}(\vec{x})$  if and only if all of the last  $d-1$  coordinates of  $\vec{x}$  are 0 and the first coordinate is  $>0$ , but there is probability 0 of sampling such  $\vec{x}$  from the uniform distribution on  $X = [0,1]^d$ .
For this reason, we can't just partition  $[0,1]^d$  as we did with the proof of Theorem 5.5 and must do something more clever. It turns out that it is possible to find a subset  $[\alpha,1]^d$  on which hypotheses parameterized by vectors on opposite faces of this cube  $[\alpha,1]^d$  have high PAC error between them. A consequence by the triangle inequality of  $err_{\mathcal{D}_X}$  is that two such hypotheses cannot both be approximated by a common third hypothesis. This is the following lemma states.
Lemma A.15. Let  $d \in \mathbb{N}$  and  $\alpha = \frac{d - 1}{d}$ . Let  $\vec{s}, \vec{t} \in [\alpha, 1]^d$  such that there exists a coordinate  $i_0 \in [d]$  where  $s_{i_0} = \alpha$  and  $t_{i_0} = 1$  (i.e.  $\vec{s}$  and  $\vec{t}$  are on opposite faces of this cube). Let  $\varepsilon \leq \frac{1}{8d}$ . Then there is no point  $\vec{r} \in X$  such that both errunif  $(h_{\vec{s}}, h_{\vec{r}}) \leq \varepsilon$  and errunif  $(h_{\vec{t}}, h_{\vec{r}}) \leq \varepsilon$  (i.e. there is no hypothesis which is an  $\varepsilon$ -approximation to both  $h_{\vec{s}}$  and  $h_{\vec{t}}$ ).
Proof. Let  $\vec{q} = \left\langle \begin{cases} s_i & i = i_0 \\ t_i & i \neq i_0 \end{cases} \right\rangle_{i=1}^d$  which will serve as a proxy to  $\vec{s}$ .
We need the following claim.
Claim A.16. For each  $\vec{x} \in X$ , the following are equivalent:
1.  $h_{\vec{q}}(\vec{x}) \neq h_{\vec{t}}(\vec{x})$
2.  $h_{\vec{q}}(\vec{x}) = 0$  and  $h_{\vec{t}}(\vec{x}) = 1$
3.  $x_{i_0}\in (q_{i_0},t_{i_0}] = (\alpha ,1]$  and for all  $i\in [d]\setminus \{i_0\}$ $x_{i}\in [0,t_{i}]$
Furthermore, the above equivalent conditions imply the following:
4.  $h_{\overline{s}}(\vec{x}) \neq h_{\overline{t}}(\vec{x})$
Proof of Claim A.16.
$(2)\Rightarrow (1)$  : This is trivial.
$(1) \Longrightarrow (2)$ : Note that because  $q_{i_0} = s_{i_0} = \alpha < 1 = t_{i_0}$ , we have for all  $i \in [d]$  that  $q_i \leq t_i$ . If  $h_{\vec{t}}(\vec{x}) = 0$  then for some  $i_1 \in [d]$  it must be that  $x_{i_1} > t_{i_1}$ , but since  $t_{i_1} \geq q_{i_1}$  it would also be the case that  $x_{i_1} > q_{i_1}$ , so  $h_{\vec{q}}(\vec{x}) = 0$  which gives the contradiction that  $h_{\vec{q}}(\vec{x}) = h_{\vec{t}}(\vec{x})$ . Thus  $h_{\vec{t}}(\vec{x}) = 1$ , and since  $h_{\vec{q}}(\vec{x}) \neq h_{\vec{t}}(\vec{x})$  we have  $h_{\vec{q}}(\vec{x}) = 0$ .
(1)  $\Longleftrightarrow$  (3): We partition  $[0, 1]^d$  into three sets and examine these three cases.
Case 1:  $x_{i_0} \in (q_{i_0}, t_{i_0}] = (\alpha, 1]$  and for all  $i \in [d] \setminus \{i_0\}$ ,  $x_i \in [0, t_i]$ . In this case,  $q_{i_0} < x_{i_0}$  so  $h_{\vec{q}}(\vec{x}) = 0$  and for all  $i \in [d]$ $x_i \leq t_i$ , so  $h_{\vec{t}}(\vec{x}) = 1$ , so  $h_{\vec{q}}(\vec{x}) \neq h_{\vec{t}}(\vec{x})$ .
Case 2:  $x_{i_0} \notin (q_{i_0}, t_{i_0}] = (\alpha, 1]$  and for all  $i \in [d] \setminus \{i_0\}$ ,  $x_i \in [0, t_i]$ . In this case, because  $x_{i_0} \in [0, 1]$  and  $x_{i_0} \notin (\alpha, 1]$  we have  $x_{i_0} \leq \alpha = q_{i_0} \leq t_{i_0}$  and also for all other  $i \in [d] \setminus \{i_0\}$ ,  $x_i \leq t_i = q_i$  (by definition of  $\vec{q}$ ). Thus  $h_{\vec{q}}(\vec{x}) = 1 = h_{\vec{t}}(\vec{x})$ .
Case 3: For some  $i_1 \in [d] \setminus \{i_0\}$ ,  $x_{i_1} \notin [0, t_{i_1}]$ . In this case, because  $x_{i_1} \in [0, 1]$ , we have  $x_{i_1} > t_{i_1} = q_{i_1}$ . Thus  $h_{\vec{q}}(\vec{x}) = 0 = h_{\vec{t}}(\vec{x})$ .
Thus, it is the case that  $h_{\vec{q}}(\vec{x}) \neq h_{\vec{t}}(\vec{x})$  if and only if  $x_{i_0} \in (q_{i_0}, t_{i_0}] = (\alpha, 1]$  and for all  $i \in [d] \setminus \{i_0\}$ ,  $x_i \in [0, t_i]$ .
$(1,2,3)\Rightarrow (4)$  : By (2), we have  $x_{i_0} > q_{i_0}$  , and since  $q_{i_0} = s_{i_0}$  by definition of  $\vec{q}$  , it follows that  $x_{i_0} > s_{i_0}$  which means  $h_\vec{s} (\vec{x}) = 0$  . By (3),  $h_{\tilde{t}}(\vec{x}) = 1$  which gives  $h_{\vec{s}}(\vec{x})\neq h_{\tilde{t}}(\vec{x})$
We also need the following Lemma.
Lemma A.17. Let  $d \in \mathbb{N}$  and  $\alpha = \frac{d - 1}{d} = 1 - \frac{1}{d}$ . Then  $(1 - \alpha) \cdot \alpha^{d - 1} > \frac{1}{4d}$ .
Proof. If  $d = 1$ , then  $\alpha = 0$  so  $(1 - \alpha) \cdot \alpha^{d-1} = 1 \geq \frac{1}{4} = \frac{1}{4d}$  (see footnote $^2$ ).
If  $d \geq 2$ , then we utilize the fact that  $(1 - \frac{1}{d})^d \geq \frac{1}{4}$  in the following:
$$
\begin{array}{l} (1 - \alpha) \cdot \alpha^ {d - 1} = (\frac {1}{d}) (1 - \frac {1}{d}) ^ {d - 1} \\ = (\frac {1}{d}) \frac {(1 - \frac {1}{d}) ^ {d}}{1 - \frac {1}{d}} \\ = \frac {(1 - \frac {1}{d}) ^ {d}}{d - 1} \\ \geq \frac {1}{4 (d - 1)} \\ > \frac {1}{4 d}. \\ \end{array}
$$
This completes the proof. As an aside,  $\alpha = \frac{d - 1}{d}$  is the value of  $\alpha$  that maximizes the expression  $(1 - \alpha)\cdot \alpha^{d - 1}$  which is why that value was chosen.
With the above Claim and Lemma in hand, we return to the proof of Lemma A.15. Our next step will be two prove the following two inequalities:
$$
2 \varepsilon <   e r r _ {\mathrm {u n i f}} (h _ {\vec {q}}, h _ {\vec {t}}) \leq e r r _ {\mathrm {u n i f}} (h _ {\vec {s}}, h _ {\vec {t}}).
$$
For the second of these inequalities, note that by the  $(1) \Longrightarrow (4)$  part of claim above, since  $h_{\vec{q}}(\vec{x}) \neq h_{\vec{t}}(\vec{x})$  implies  $h_{\vec{s}}(\vec{x}) \neq h_{\vec{t}}(\vec{x})$  we have
$$
\begin{array}{l} e r r _ {\mathrm {u n i f}} (h _ {\vec {q}}, h _ {\vec {t}}) = \operatorname * {P r} _ {\vec {x} \sim \mathrm {u n i f} (X)} [ h _ {\vec {q}} (\vec {x}) \neq h _ {\vec {t}} (\vec {x}) ] \\ \leq \operatorname * {P r} _ {\vec {x} \sim \operatorname {u n i f} (X)} \left[ h _ {\vec {s}} (\vec {x}) \neq h _ {\vec {t}} (\vec {x}) \right] \\ = e r r _ {\text {u n i f}} \left(h _ {\bar {s}}, h _ {\bar {t}}\right). \\ \end{array}
$$
2This uses the interpretation that  $0^0 = 1$  which is the correct interpretation in the context in which we will use the lemma.
Now, for the first of the inequalities above, we will use the  $(1) \Longleftrightarrow (3)$  portion of the claim, we will use our hypothesis that  $\vec{t} \in [\alpha, 1]^d$  (which implies for each  $i \in [d]$  that  $[0, t_i] \subseteq [0, \alpha]$ ), we will use the hypothesis that  $\varepsilon \leq \frac{1}{8d}$ , and we will use Theorem A.17. Utilizing these, we get the following:
$$
\begin{array}{l} e r r _ {\mathrm {u n i f}} (h _ {\vec {q}}, h _ {\vec {t}}) \\ = \Pr_ {\vec {x} \sim \operatorname {u n i f} (X)} [ h _ {\vec {q}} (\vec {x}) \neq h _ {\vec {t}} (\vec {x}) ] \\ = \Pr_ {\vec {x} \sim \operatorname {u n i f} (X)} \left[ x _ {i _ {0}} \in (\alpha , 1 ] \wedge \forall i \in [ d ] \backslash \left\{i _ {0} \right\}, x _ {i} \in [ 0, t _ {i} ] \right] \\ = \operatorname *{Pr}_{x_{i_{0}}\sim \operatorname {unif}([0,1])}[x_{i_{0}}\in (\alpha ,1)]\cdot \prod_{\substack{i = 1\\ i\neq i_{0}}}^{d}\operatorname *{Pr}_{x\sim \operatorname {unif}([0,1])}[x\in [0,t_{i}]] \\ \geq \operatorname *{Pr}_{x_{i_{0}}\sim \operatorname {unif}([0,1])}[x_{i_{0}}\in (\alpha ,1)]\cdot \prod_{\substack{i = 1\\ i\neq i_{0}}}^{d}\operatorname *{Pr}_{x\sim \operatorname {unif}([0,1])}[x\in [0,\alpha ]] \\ = (1 - \alpha) \cdot \alpha^ {d - 1} \\ > \frac {1}{4 d} \\ \geq 2 \varepsilon . \\ \end{array}
$$
Thus, we get the desired two inequalities:
$$
2 \varepsilon <   e r r _ {\text {u n i f}} \left(h _ {\vec {q}}, h _ {\vec {t}}\right) \leq e r r _ {\text {u n i f}} \left(h _ {\vec {s}}, h _ {\vec {t}}\right).
$$
This nearly completes the proof. If there existed some point  $\vec{r} \in X$  such that both  $err_{\mathrm{unif}}(h_{\vec{s}}, h_{\vec{r}}) \leq \varepsilon$  and  $err_{\mathrm{unif}}(h_{\vec{t}}, h_{\vec{r}}) \leq \varepsilon$ , then it would follow from the triangle inequality of  $err_{\mathrm{unif}}$  that
$$
e r r _ {\text {u n i f}} \left(h _ {\vec {s}}, h _ {\vec {t}}\right) \leq e r r _ {\text {u n i f}} \left(h _ {\vec {s}}, h _ {\vec {r}}\right) + e r r _ {\text {u n i f}} \left(h _ {\vec {r}}, h _ {\vec {r}}\right) \leq 2 \varepsilon
$$
but this would contradict the above inequalities, so no such  $\vec{r}$  exists.
Equipped with the Lemma A.15, we are now ready to prove Theorem A.14.
Proof of Theorem A.14. Fix any  $d \in \mathbb{N}$ , and choose  $\varepsilon$  and  $\delta$  as  $\varepsilon \leq \frac{1}{4d}$  and  $\delta \leq \frac{1}{d + 2}$ . We will use the constant  $\alpha = \frac{d - 1}{d}$  and consider the cube  $[\alpha, 1]^d$ .
Suppose for contradiction such an algorithm  $A$  does exists for some  $k < d + 1$ . This means that for each possible threshold  $\vec{t} \in [0,1]^d$ , there exists some set  $L_{\vec{t}} \subseteq \mathcal{H}$  of hypotheses with three properties: (1) each element of  $L_{\vec{t}}$  is an  $\varepsilon$ -approximation to  $h_{\vec{t}}$ , (2)  $|L_{\vec{t}}| \leq k$ , and (3) with probability at least  $1 - \delta$ ,  $A$  returns an element of  $L_{\vec{t}}$ .
By the trivial averaging argument, this means that there exists at least one element in  $L_{\vec{t}}$  which is returned by  $A$  with probability at least  $\frac{1}{k} \cdot (1 - \delta) \geq \frac{1}{k} \cdot (1 - \frac{1}{d + 2}) = \frac{1}{k} \cdot \frac{d + 1}{d + 2} \geq \frac{1}{k} \cdot \frac{k + 1}{k + 2}$ . Let  $f \colon [\alpha, 1]^d \to [0, 1]^d$  be a function which maps each threshold  $\vec{t} \in [\alpha, 1]^d$  to such an element (the maximum probability element with ties broken arbitrarily) of  $L_{\vec{t}}$ . This is slightly different from the proof of Theorem 5.5 because we are defining the function  $f$  on only a very specific subset of the possible thresholds. The reason for this was alluded to in the discussion following the statement of Theorem A.14.
The function  $f$  induces a partition  $\mathcal{P}$  of  $[\alpha, 1]^d$  where the members of  $\mathcal{P}$  are the fibers of  $f$  (i.e.  $\mathcal{P} = \{f^{-1}(\vec{y}) \colon \vec{y} \in \mathrm{range}(f)\}$ ). For any member  $W \in \mathcal{P}$  and any coordinate  $i \in [d]$ , it cannot be that the set  $\{w_i \colon \vec{w} \in W\}$  contains both values  $\alpha$  and 1—if it did, then there would be two points  $\vec{s}, \vec{t} \in W$  such that  $s_i = \alpha$  and  $t_i = 1$ , but because they both belong to  $W$ , there is some  $\vec{y} \in [0, 1]^d$  such that  $f(\vec{s}) = \vec{y} = f(\vec{t})$ , but by definition of the partition,  $h_{\vec{y}}$  would have to be an  $\varepsilon$ -approximation (in the PAC model) of both  $h_{\vec{s}}$  and  $h_{\vec{t}}$ , but by Lemma A.15 this is not possible.
Thus, the partition  $\mathcal{P}$  is a non-spanning partition of  $[\alpha, 1]^d$  as in the proof of Lemma 4.3, so there is some point  $\vec{p} \in [\alpha, 1]^d$  such that for every radius  $r > 0$ , it holds that  $\overline{B}_r^\infty(\vec{p})$  intersects at least  $d + 1$
members of  $\mathcal{P}$ . In fact, there is some radius  $r$  such that  $\| \vec{t} -\vec{s}\|_{\infty}\leq r$ , then  $d_{\mathrm{TV}}(\mathcal{D}_{A,\vec{s},n},\mathcal{D}_{A,\vec{t},n})\leq \eta$ , for  $\eta$  that lies between 0 and  $\frac{1}{k}\cdot \frac{k + 1}{k + 2} -\frac{1}{k + 1}$ .
Now we get the same type of contradiction as in the proof of Theorem 5.5: for the special point  $\vec{p}$  we have that  $\mathcal{D}_{A,\vec{p},n}$  is a distribution that has  $d + 1\geq k + 1$  disjoint events that each have probability greater than  $\frac{1}{k + 1}$ . Thus, no  $k$ -list replicable algorithm exists.
B Expanded Prior and Related Work
We give a more detailed discussion on prior and related work. This section is an elaboration of the Section 2 from the main body of the paper.
Formalizing reproducibility and replicability has gained considerable momentum in recent years. While the terms reproducibility and replicability are very close and often used interchangeably, there has been an effort to distinguish between them and accordingly, our notions fall in the replicability definition  $(\mathrm{PVLS}^{+}21)$ .
In the context of randomized algorithms, various notions of reproducibility/replicability have been investigated. The work of Gat and Goldwasser (GG11) formalized and defined the notion of pseudodeterministic algorithms. A randomized algorithm  $A$  is pseudodeterministic if, for any input  $x$ , there is a canonical value  $v_{x}$  such that  $\operatorname*{Pr}[A(x) = v_x] \geq 2/3$ . Gat and Goldwasser designed polynomial-time pseudodeterministic algorithms for algebraic computational problems, such as finding quadratic non-residues and finding non-roots of multivariate polynomials (GG11). Later works studied the notion of pseudodeterminism in other algorithmic settings, such as parallel computation, streaming and sub-linear algorithms, interactive proofs, and its connections to complexity theory (GG; GGH18; OS17; OS18; AV20; GGMW20; LOS21; DPVWV22).
In the algorithmic setting, mainly two generalizations of pseudodeterminism have been investigated: multi-pseudodeterministic algorithms (Gol19) and influential bit algorithms (GL19). A randomized algorithm  $A$  is  $k$ -pseudodeterministic if, for every input  $x$ , there is a set  $S_x$  of size at most  $k$  such that the output of  $A(x)$  belongs to the set  $S_x$  with high probability. When  $k = 1$ , we get pseudodeterminism. A randomized algorithm  $A$  is  $\ell$ -influential-bit algorithm if, for every input  $x$ , for most of the strings  $r$  of length  $\ell$ , there exists a canonical value  $v_{x,r}$  such that the algorithm  $A$  on inputs  $x$  and  $r$  outputs  $v_{x,r}$  with high probability. The string  $r$  is called the influential bit string. Again, when  $\ell = 0$ , we get back pseudodeterminism. The main focus of these works has been to investigate reproducibility in randomized search algorithms.
Very recently, pseudodeterminism and its generalizations have been explored in the context of learning algorithms to formalize the notion of replicability. The seminal work of (BLM20) defined the notion of global stability. They define a learning algorithm  $A$  to be  $(n,\eta)$ -globally stable with respect to a distribution  $D$  if there is a hypothesis  $h$  such that  $\operatorname{Pr}_{S\sim D^n}(A(S) = h)\geq \eta$ , here  $\eta$  is called the stability parameter. Note that the notion of global stability is equivalent to Gat and Goldwasser's notion of pseudodeterminism when  $\eta = 2 / 3$ . Since Gat and Goldwasser's motivation is to study pseudodeterminism in the context of randomized algorithms, the success probability is taken as  $2 / 3$ . In the context of learning, studying the stability parameter  $\eta$  turned out to be useful. The work of Bun, Livny and Moran (BLM20) showed that any concept class with Littlestone dimension  $d$  has an  $(m,\eta)$ -globally stable learning algorithm with  $m = \tilde{O} (2^{2^d} / \alpha)$  and  $\eta = \tilde{O} (2^{-2^d})$ , where the error of  $h$  (with respect to the unknown hypothesis) is  $\leq \alpha$ . Then they established that a globally stable learner implies a differentially private learner. This, together with an earlier work of Alon, Livny, Malliaris, and Moran (ALMM19), establishes an equivalence between online learnability and differentially private PAC learnability.
The work of Ghazi, Kumar, and Manurangsi (GKM21) extended the notion of global stability to pseudo-global stability and list-global stability. The notion of pseudo-global stability is very similar to the earlier-mentioned notion of influential bit algorithms of Grossman and Liu (GL19) when translated to the context of learning. The work of (GKM21) used these concepts to design user-level differentially private algorithms.
The recent work reported in (ILPS22) introduced the notion of  $\rho$ -replicability. A learning algorithm  $A$  is  $\rho$ -replicable if  $\operatorname{Pr}_{S_1, S_2, r}[A(S_1, r) = A(S_2, r)] \geq 1 - \rho$ , where  $S_1$  and  $S_2$  are samples drawn from a distribution  $\mathcal{D}$  and  $r$  is the internal randomness of the learning algorithm  $A$ . They designed replicable
algorithms for many learning tasks, including statistical queries, approximate heavy hitters, median, and learning half-spaces. It is known that the notions of pseudo-global stability and  $\rho$ -replicability are the same up to polynomial factors in the parameters (ILPS22; GKM21).
In this work, we study the notions of list and certificate complexities as a measure the degree of (non) replicability. Our goal is to design learning algorithms with optimal list and certificate complexities while minimizing the sample complexity. The earlier works (BLM20; GKM21; ILPS22) did not focus on minimizing these quantities. The works of (BLM20; GKM21) used replicable algorithms as an intermediate step to design differentially private algorithms. The work of (ILPS22) did not consider reducing the certificate complexity in their algorithms and also did not study list-replicability. Earlier works (GKM21; ILPS22) studied how to convert statistical query learning algorithms into certificate replicable learning algorithms, however, their focus was not on the certificate complexity. Here, we study the relationship among (nonadaptive and adaptive) statistical query learning algorithms, list replicable algorithms, and certificate replicable algorithms with a focus on list, certificate and sample complexities.
The study of notions of reproducibility/replicability in various computational fields is an emerging topic. The article  $(\mathrm{PVLS}^{+}21)$  discusses the differences between replicability and reproducibility. In  $(\mathrm{EKK}^{+}23)$ , the authors consider replicability in the context of stochastic bandits. Their notion is similar to the notion studied in (ILPS22). In  $(\mathrm{AJJ}^{+}22)$ , the authors investigate reproducibility in the context of optimization with inexact oracles (initialization/gradient oracles). The setup and focus of these works are different from ours.