SlowGuess/NeurIPS_2023_OCR huggingface dataset:
5 Replicability of Learning Coins Biases
In this section, we establish replicability results for estimating biases of  $d$  coins.
Definition 5.1. The  $d$ -COIN BIAS ESTIMATION PROBLEM is the following problem: Design an algorithm  $A$  (possibly randomized) that gets  $\varepsilon \in (0,1)$ ,  $\delta \in (0,1]$  as inputs, observes independent tosses of an ordered collection of  $d$ -many biased coins with a bias vector  $\vec{b} \in [0,1]^d$ , and outputs  $\vec{v}$  so that  $\| \vec{b} - \vec{v} \|_{\infty} \leq \varepsilon$  with probability  $\geq 1 - \delta$ .
The  $d$ -COIN BIAS ESTIMATION PROBLEM fits in the framework of the general learning task introduced in Section 3 so that we can talk about list and certificate replicable algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM. We describe this now.
For  $d$ -COIN BIAS ESTIMATION PROBLEM we have the following.  $\mathcal{X} = \{0,1\}^d$  (where 0 corresponds to Tail and 1 corresponds to Head) which is the set of representations of all possibilities of flipping  $d$ -many coins. The class of distributions  $\mathcal{D}$  is the set of all  $d$ -fold products of Bernoulli distributions. Each distribution  $D\in \mathcal{D}$  is parameterized with a vector  $v_{D} = \langle b_{1},\dots ,b_{d}\rangle \in [0,1]^{d}$ . When sampled according to  $v_{D}$  we obtain a sample point  $\langle x_1\dots x_d\rangle \in \mathcal{X}$ , where  $\operatorname*{Pr}(x_i = 1) = b_i$ . We take the class of hypothesis  $\mathcal{H}$  to be  $[0,1]^d$  which is the set of all  $d$ -tuples representing biases of a collection of  $d$ -many coins. Lastly, the error function is defined as  $err(D,h) = \| v_D - h\|_\infty$ .
5.1 Replicable Algorithms for  $d$ -COIN BIAS ESTIMATION PROBLEM
In this section, we design list and certificate replicable algorithms for  $d$ -COIN BIAS ESTIMATION PROBLEM.
Theorem 5.2. There exists an  $(d + 1)$ -list replicable algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM. For a given  $\varepsilon$  and  $\delta$ , its sample complexity is  $n = O\left(\frac{d^2}{\varepsilon^2} \cdot \log \frac{d}{\delta}\right)$ , per coin.
Algorithm 1  $(d + 1)$ -list replicable algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM
Input:  $\varepsilon > 0, \delta \in (0,1]$ , sample access to  $d$  coins with biases  $\vec{b} \in [0,1]^d$
$$
\varepsilon_ {0} \stackrel {\mathrm {d e f}} {=} \frac {\varepsilon}{2 d}, \delta_ {0} \stackrel {\mathrm {d e f}} {=} \frac {\delta}{d}
$$
$n \stackrel{\mathrm{def}}{=} O\left(\frac{\ln(1 / \delta_0)}{\varepsilon_0^2}\right) = O\left(\frac{d^2\ln(d / \delta)}{\varepsilon^2}\right)$  for some constant
Let  $f_{\varepsilon}:\mathbb{R}^{d}\to \mathbb{R}^{d}$  be as in Lemma 4.1.
Let  $g: \mathbb{R}^d \to [0,1]^d$  be the function which restricts coordinates to the unit interval (i.e.
$$
g (\vec {y}) \stackrel {{\mathrm {d e f}}} {{=}} \left\langle \left\{ \begin{array}{l l} 0 & y _ {i} <   0 \\ y _ {i} & y _ {i} \in [ 0, 1 ] \\ 1 & y _ {i} > 1 \end{array} \right. \right\rangle_ {i = 1} ^ {d}
$$
Take  $n$  samples from each coin and let  $\vec{a}$  be the empirical biases.
return  $g(f(\vec{a}))$
Proof. Note that when  $\varepsilon \geq 1/2$ , a trivial algorithm that outputs a vector with  $1/2$  in each component works. Thus the most interesting case is when  $\varepsilon < 1/2$ . Our list replicable algorithm is described in Algorithm 1. We will prove its correctness by associating for each possible bias  $\vec{b} \in [0,1]^d$ , a set  $L_{\vec{b}}$  with the three necessary properties: (1)  $|L_{\vec{b}}| \leq d + 1$ , (2)  $L_{\vec{b}} \subseteq \overline{B}_{\varepsilon}^{\infty}(\vec{b})$  (and also the problem specific
restriction that  $L_{\vec{b}} \subseteq [0,1]^d$ ), and (3) when given access to coins of biases  $\vec{b}$ , with probability at least  $1 - \delta$  the algorithm returns a value in  $L_{\vec{b}}$ .
Let  $L_{\vec{b}} = \left\{g(f_{\varepsilon}(\vec{x}))\colon \vec{x}\in \overline{B}_{\varepsilon_0}^{\infty}(\vec{b})\right\}$ . By Lemma 4.1,  $f_{\varepsilon}$  takes on at most  $d + 1$  values on  $\overline{B}_{\varepsilon_0}^{\infty}(\vec{b})$  (which means  $g\circ f_{\varepsilon}$  also takes on at most  $d + 1$  values on this ball) which proves that  $|L_{\vec{b}}|\leq d + 1$ . This proves property (1).
Next we state the following observation which says that the coordinate restriction function  $g$  of Algorithm 1 does not reduce approximation quality. The proof is straightforward.
Observation 5.3. Using the notation of Algorithm 1, if  $\vec{y} \in \overline{B}_{\varepsilon}^{\infty}(\vec{b})$  then  $g(\vec{y}) \in \overline{B}_{\varepsilon}^{\infty}(\vec{b})$ .
We now establish Property (2). We know from Lemma 4.1 that for each  $\vec{x} \in \overline{B}_{\varepsilon_0}^\infty(\vec{b})$  we have  $f_{\varepsilon}(\vec{x}) \in \overline{B}_{\varepsilon}^\infty(\vec{b})$ , and by Observation 5.3,  $g$  maintains this quality and we have  $g(f_{\varepsilon}(\vec{x})) \in \overline{B}_{\varepsilon}^\infty(\vec{b})$ . This shows that  $L_{\vec{b}} \subseteq \overline{B}_{\varepsilon}^\infty(\vec{b})$  proving property (2).
By Chernoff's bounds, for a single biased coin, with  $n = O\left(\frac{\ln(1 / \delta_0)}{\varepsilon_0^2}\right)$  independent samples of the coin we can estimate the bias within  $\varepsilon_0$  with probability at least  $1 - \delta_0$ . Thus, by a union bound, if we take  $n$  samples of each of the  $d$  coins, there is a probability of at most  $d \cdot \delta_0 = \delta$  that at least one of the empirical coin biases is not within  $\varepsilon_0$  of the true bias. Thus, by taking  $n$  samples of each coin, we have with probability at least  $1 - \delta$  that the empirical biases  $\vec{a}$  belong to  $\overline{B}_{\varepsilon_0}^\infty (\vec{b})$ . In the case that this occurs, we have by definition of  $L_{\vec{b}}$  that the value  $g(f_{\varepsilon}(\vec{a}))$  returned by the algorithm belongs to the set  $L_{\vec{b}}$ . This proves property (3). The sample complexity follows since  $\varepsilon_0 = \frac{\varepsilon}{2d}$  and  $\delta_0 = \frac{\delta}{d}$ .
The next result is on certificate replicable algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM.
Theorem 5.4. For every  $\varepsilon$  and  $\delta$ , there is a  $(\lceil \log \frac{d}{\delta} \rceil, \varepsilon, \delta)$ -certificate replicable algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM with sample complexity of  $n = O\left(\frac{d^2}{\varepsilon^2 \delta^2}\right)$  per coin.
Proof. Let  $\varepsilon$  and  $\delta$  be the input parameters to the algorithm and  $\vec{b}$  the bias vector. Set  $\varepsilon_0 = \frac{\varepsilon\delta}{2d}$ . The algorithm  $A$  first estimates the bias of each coin with up to  $\varepsilon_0$  with a probability error parameter  $\frac{\delta}{d}$  using a standard estimation algorithm. Note that this can be done using  $O\left(\frac{d^2}{\varepsilon^2\delta^2}\right)$  tosses per coin. Let  $\vec{v}$  be the output vector. It follows that  $\vec{v} \in \overline{B}_{\varepsilon_0}^\infty(\vec{b})$  with probability at least  $1 - \delta$ . Then it runs the deterministic function  $f$  described in Lemma 4.2 with input  $r \in \{0,1\}^\ell$  with  $\ell = \lceil \log \frac{d}{\delta} \rceil$  and  $\vec{v}$  and outputs the value of the function. Lemma 4.2 guarantees that for  $1 - \delta$  fraction of the  $rs$ , all  $\vec{v} \in \overline{B}_{\varepsilon_0}^\infty(\vec{b})$  gets rounded to the same value by  $f$ . Hence algorithm  $A$  satisfies the requirements of the certificate-replicability. The certificate complexity is  $\lceil \log \frac{d}{\delta} \rceil$ .
We remark that by using the refined definition of certificate replicability mentioned in Section 3, we can obtain a  $(\lceil \log \frac{d}{\rho} \rceil, \varepsilon, \rho, \delta)$ -replicable algorithm with sample complexity  $O\left(\frac{d^2}{\varepsilon^2 \rho^2} \log(1 / \delta)\right)$ . Note that an  $\ell$ -certificate replicable algorithm leads to a  $2^\ell$ -list replicable algorithm. Thus Theorem 5.4 gives a  $O\left(\frac{d}{\delta}\right)$ -list replicable algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM with sample complexity  $O\left(\frac{d^2}{\varepsilon^2 \delta^2}\right)$ . However, this is clearly sub-optimal and Theorem 5.2 gives an algorithm with a much smaller list and sample complexities. Also from the work of Goldrecih (Gol19), it follows that  $\ell$ -list replicable algorithm can be converted into a  $\lceil \log\left(\frac{\ell}{\delta}\right) \rceil$ -certificate replicable algorithm. However, this conversion increases the sample complexity. For example, when applied to  $d$ -COIN BIAS ESTIMATION PROBLEM, the sample complexity becomes  $O\left(\frac{d^6}{\varepsilon^2 \delta^2}\right)$ . In comparison, Theorem 5.4 uses a tailored rounding technique to achieve an algorithm with a much smaller sample complexity.
5.2 An Impossibility Result
This section establishes the optimality of the list complexity for  $d$ -COIN BIAS ESTIMATION PROBLEM.
Theorem 5.5. For  $k < d + 1$ , there does not exist a  $k$ -list replicable algorithm for the  $d$ -COIN BIAS ESTIMATION PROBLEM.
Before proving the theorem, we need a lemma that follows from the Data Processing Inequality; for more detail see the supplementary material. In the following, we use  $\mathcal{D}_{A,\vec{b},n}$  to denote the distribution of the output of an algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM.
Lemma 5.6. For biases  $\vec{a},\vec{b}\in [0,1]^d$  we have  $d_{\mathrm{TV}}\left(\mathcal{D}_{A,\vec{a},n},\mathcal{D}_{A,\vec{b},n}\right)\leq n\cdot d\cdot \| \vec{b} -\vec{a}\|_{\infty}$
Proof. We use the basic fact that an algorithm (deterministic or randomized) cannot increase the total variation distance between two input distributions.
The distribution giving one sample flip of each coin in a collection with bias  $\vec{b}$  is the  $d$ -fold product of Bernoulli distributions  $\prod_{i=1}^{d} \operatorname{Bern}(b_i)$  (which for notational brevity we denote as  $\operatorname{Bern}(\vec{b})$ , so the distribution which gives  $n$  independent flips of each coin is the  $n$ -fold product of this and is denoted as  $\operatorname{Bern}(\vec{b})^{\otimes n}$ ). We will show that for two bias vectors  $\vec{a}$  and  $\vec{b}$ ,  $d_{\mathrm{TV}}\left(\operatorname{Bern}(\vec{b})^{\otimes n}, \operatorname{Bern}(\vec{a})^{\otimes n}\right) \leq n \cdot d \cdot \| \vec{b} - \vec{a} \|_{\infty}$ . This suffices to establish the lemma.
Observe that we have for each  $i\in [d]$
$$
d _ {\mathrm {T V}} \left(\operatorname {B e r n} \left(b _ {i}\right), \operatorname {B e r n} \left(a _ {i}\right)\right) = \left| b _ {i} - a _ {i} \right|.
$$
$$
d _ {\mathrm {T V}} \left(\operatorname {B e r n} (\vec {b}), \operatorname {B e r n} (\vec {a})\right) \leq \sum_ {i = 1} ^ {d} \left| b _ {i} - a _ {i} \right| \leq d \cdot \| \vec {b} - \vec {a} \| _ {\infty}
$$
$$
d _ {\mathrm {T V}} \left(\operatorname {B e r n} (\vec {b}) ^ {\otimes n}, \operatorname {B e r n} (\vec {a}) ^ {\otimes n}\right) \leq n \cdot d \cdot \| \vec {b} - \vec {a} \| _ {\infty}.
$$
Proof of Theorem 5.5. Fix any  $d \in \mathbb{N}$ , and choose  $\varepsilon$  and  $\delta$  as  $\varepsilon < \frac{1}{2}$  and  $\delta \leq \frac{1}{d + 2}$ . Suppose for contradiction that such an algorithm  $A$  does exist for some  $k < d + 1$ . This means that for each possible bias vector  $\vec{b} \in [0,1]^d$ , there exists some set  $L_{\vec{b}} \subseteq \mathcal{H}$  of hypotheses with three properties: (1) each element of  $L_{\vec{b}}$  is an  $\varepsilon$ -approximation to  $h_{\vec{b}}$ , (2)  $|L_{\vec{b}}| \leq k$ , and (3) with probability at least  $1 - \delta$ ,  $A$  returns an element of  $L_{\vec{b}}$ .
By an averaging argument, this means that there exists at least one element in  $L_{\vec{b}}$  which is returned by  $A$  with probability at least  $\frac{1}{k} \cdot (1 - \delta) \geq \frac{1}{k} \cdot (1 - \frac{1}{d + 2}) = \frac{1}{k} \cdot \frac{d + 1}{d + 2} \geq \frac{1}{k} \cdot \frac{k + 1}{k + 2}$ . Let  $f: [0,1]^d \to [0,1]^d$  be a function which maps each bias  $\vec{b}$  to such an element of  $L_{\vec{b}}$ . Since  $\frac{1}{k} \cdot \frac{k + 1}{k + 2} > \frac{1}{k + 1}$ , let  $\eta$  be such that  $0 < \eta < \frac{1}{k} \cdot \frac{k + 1}{k + 2} - \frac{1}{k + 1}$ .
The function  $f$  induces a partition  $\mathcal{P}$  of  $[0,1]^d$  where the members of  $\mathcal{P}$  are the fibers of  $f$  (i.e.  $\mathcal{P} = \{f^{-1}(\vec{y})\colon \vec{y}\in \mathrm{range}(f)\}$ ). By definition, for any member  $X\in \mathcal{P}$  there exists some  $\vec{y}\in \mathrm{range}(f)$  such that for all  $\vec{b}\in X$ ,  $f(\vec{b}) = \vec{y}$ . By definition, we have  $f(\vec{b})\in L_{\vec{b}}\subseteq \overline{B}_{\varepsilon}^{\infty}(\vec{b})$  showing that  $\vec{y}\in \overline{B}_{\varepsilon}^{\infty}(\vec{b})$  and by symmetry  $\vec{b}\in \overline{B}_{\varepsilon}^{\infty}(\vec{y})$ . This shows that  $X\subseteq \overline{B}_{\varepsilon}^{\infty}(\vec{y})$ , so  $\mathrm{diam}_{\infty}(X)\leq 2\varepsilon < 1$ .
Let  $r = \frac{\eta\cdot d}{n}$ . Since every member of  $\mathcal{P}$  has  $\ell_{\infty}$  diameter less than 1, by Lemma 4.3 there exists a point  $\vec{p} \in [0,1]^d$  such that  $\overline{B}_r^\infty (\vec{p})$  intersects at least  $d + 1 > k$  members of  $\mathcal{P}$ . Let  $\vec{b}^{(1)},\ldots ,\vec{b}^{(d + 1)}$  be points belonging to distinct members of  $\mathcal{P}$  that all belong to  $\overline{B}_r^\infty (\vec{p})$ . By definition of  $\mathcal{P}$ , this means for distinct  $j,j^{\prime} \in [d + 1]$  that  $f(\vec{b}^{(j)}) \neq f(\vec{b}^{(j')})$ .
Now, for each  $j \in [d + 1]$ , because  $\| \vec{p} - \vec{b}^{(j)} \|_{\infty} \leq r$ , by Lemma 5.6 we have  $d_{\mathrm{TV}}(\mathcal{D}_{A,\vec{p},n}, \mathcal{D}_{A,b(\vec{j}),n}) \leq n \cdot d \cdot r = \eta$ . However, this gives rise to a contradiction because the probability that  $A$  with access to biased coins  $\vec{b}^{(j)}$  returns  $f(\vec{b}^{(j)})$  is at least  $\frac{1}{k} \cdot \frac{k + 1}{k + 2}$ , and by Lemma 5.6, it must be that  $A$  with access to biased coins  $\vec{p}$  returns  $f(\vec{b}^{(j)})$  with probability at least  $\frac{1}{k} \cdot \frac{k + 1}{k + 2} - \eta > \frac{1}{k + 1}$ ; notationally,  $\operatorname{Pr}_{\mathcal{D}_{A,b(\vec{j}),n}}\left(\left\{f(\vec{b}^{(j)})\right\}\right) \geq \frac{1}{k} \cdot \frac{k + 1}{k + 2}$  and  $d_{\mathrm{TV}}(\mathcal{D}_{A,b(\vec{j}),n}, \mathcal{D}_{A,\vec{p},n}) \leq \eta$ , so  $\operatorname{Pr}_{\mathcal{D}_{A,\vec{p},n}}\left(\left\{f(\vec{b}^{(j)})\right\}\right) \geq \frac{1}{k} \cdot \frac{k + 1}{k + 2} - \eta > \frac{1}{k + 1}$ . This is a contradiction because a distribution cannot have  $d + 1 \geq k + 1$  disjoint events that each have probability greater than  $\frac{1}{k + 1}$ .
6 List and Certificate Replicability in PAC Learning
In this section, we establish replicability results for the PAC model. We first note that PAC model fits in the general learning framework introduced in Section 3. Consider the PAC learning model where we have a domain  $\mathcal{X}'$ , a concept class  $\mathcal{C}$  (a family of subsets of  $\mathcal{X}'$ ), and a hypothesis class  $\mathcal{H}$  (a collection of functions  $\mathcal{X}' \to \{0,1\}$ ). For any distribution  $D$  over  $\mathcal{X}'$  and any concept indicator  $f$  from  $\mathcal{C}$ , let  $D_f$  denote the distribution over  $\mathcal{X}' \times \{0,1\}^d$  obtained by sampling  $x \sim D$  and outputting  $\langle x, f(x) \rangle$ . We define the following learning problem in the general learning framework:  $\mathcal{X} = \mathcal{X}' \times \{0,1\}$ ,  $\mathcal{H} = \mathcal{H}$ ,  $\mathcal{D} = \{D_f : f \in \mathcal{C}$  and  $D$  a distribution over  $\mathcal{X}'\}$ , and  $err(D_f, h) = \operatorname*{Pr}_{x \sim \mathcal{D}'}[f(x) \neq h(x)]$ .
We are interested in statistical query learning which is defined by Kearns (Kea98).
Definition 6.1. A statistical query oracle  $STAT(D_f, \nu)$  takes as an input a real-valued function  $\phi: X \times \{0,1\} \to (0,1)$  and returns an estimate  $v$  such that  $|v - E_{\langle x,y \rangle \in D_f}[\phi(x,y)]| \leq \nu$  (Kea98). We say that an algorithm  $A$  learns a concept class  $\mathcal{H}$  via statistical queries if for every distribution  $D$  and every function  $f \in \mathcal{H}$ , for every  $0 < \varepsilon < 1$ , there exists  $\nu$  such that the algorithm  $A$  on input  $\varepsilon$ , and  $STAT(D_f, \nu)$  as an oracle, outputs a hypothesis  $h$  such that  $err(D_f, h) \leq \varepsilon$ . The concept class is non-adaptively learnable if all the queries made by  $A$  to the statistical query oracle are non-adaptive.
6.1 Replicable PAC Learning Algorithms
Next we state main results of this section. Due to space limitation, all the proofs are in the supplementary material. We first show that any concept class that is learnable with  $d$  non-adpative statistical queries has a  $(d + 1)$ -list replicable algorithm.
Theorem 6.2. Let  $\mathcal{H}$  be a concept class that is learnable with  $d$  non-adaptive statistical queries, then  $\mathcal{H}$  is  $(d + 1)$ -list replicably learnable. Furthermore, the sample complexity  $n = n(\nu, \delta)$  of the  $(d + 1)$ -list replicable algorithm is  $O\left(\frac{d^2}{\nu^2} \cdot \log \frac{d}{\delta}\right)$ , where  $\nu$  is the approximation error parameter of each statistical query oracle.
We note that we can simulate a statistical query algorithm that makes  $d$  adaptive queries to get a  $2^{d}$ -list replicable learning algorithm. This can be done by rounding each query to two possible values (the approximation factor increases by 2). The sample complexity of this algorithm will be  $O\left(\frac{d}{\nu^{2}} \cdot \log \frac{d}{\delta}\right)$ . Next, we design a certificate replicable algorithm for hypothesis classes that admit statistical query learning algorithms.
Theorem 6.3. Let  $\mathcal{H}$  be a concept class that is learnable with  $d$  non-adaptive statistical queries, then  $\mathcal{H}$  is  $\lceil \log \frac{d}{\delta} \rceil$ -certificate replicably learnable. Furthermore, the sample complexity  $n = n(\nu, \delta)$  of this algorithm is  $O\left(\frac{d^2}{\nu^2 \delta^2} \cdot \log \frac{d}{\delta}\right)$ , where  $\nu$  is the approximation error parameter of each statistical query oracle.
We can also consider the case when the statistical query algorithm makes  $d$  adaptive queries. In this case we get the following theorem. Note that the certificate complexity is close to linear in  $d$  as oppose to logarithmic in the case of non-adaptive queries.
Theorem 6.4. Let  $\mathcal{H}$  be a concept class that is learnable with  $d$  adaptive statistical queries, then  $\mathcal{H}$  is  $\lceil d\log \frac{d}{\delta}\rceil$ -certificate replicably learnable. Furthermore, the sample complexity of this algorithm is  $O\left(\frac{d^3}{\nu^2\delta^2}\cdot \log \frac{d}{\delta}\right)$ , where  $\nu$  is the approximation error parameter of each statistical query oracle.
6.2 Impossibility Results in the PAC Model
In this section, we establish matching upper and lower bounds on the list complexity for the concept class  $d$ -THRESHOLD in the PAC model with respect to the uniform distribution. In particular, we establish that this learning task admits a  $(d + 1)$ -list replicable algorithm and does not admit a  $d$ -list replicable algorithm.
Definition 6.5 (d-THRESHOLD). Fix some  $d \in \mathbb{N}$ . Let  $X = [0,1]^d$ . For each value  $\vec{t} \in [0,1]^d$ , let  $h_{\vec{t}}: X \to \{0,1\}$  be the concept defined as follows:  $h_{\vec{t}}(\vec{x}) = 1$  if for every  $i \in [d]$  it holds that  $x_i \leq t_i$  and 0 otherwise. Let  $\mathcal{H}$  be the hypothesis class consisting of all such threshold concepts:  $\mathcal{H} = \{h_{\vec{t}} | \vec{t} \in [0,1]^d\}$ .
We first observe the impossibility of list-replicable algorithms in the distribution-free PAC model. This follows from known results.
Observation 6.6. There is no  $k$ -list replicable algorithm (for any  $k$ ) for  $d$ -THRESHOLD in the PAC model even when  $d = 1$ .
The above observation follows from the works of (ALMM19) and (BLM20). It is known that  $d$ -THRESHOLD has an infinite Littlestone dimension. Suppose it admits  $k$ -list replicable algorithms in the PAC model. This implies that  $d$ -THRESHOLD is globally stable learnable with stability parameter  $1 / k$  (Please see (BLM20) for the definition of global stability). The work of (BLM20) showed that any class that is globally stable learnable with a constant stability parameter is differentially private learnable. The work of (ALMM19) showed that if a concept class is differentially private learnable, then it has a finite Littlestone dimension. Putting these results together, we obtain that if  $d$ -THRESHOLD admits  $k$ -list replicable algorithm, then it has a finite Littlestone dimension which is a contradiction.
The above result implies that for every  $k$  and every learning algorithm  $A$ , there is some distribution  $\mathcal{D}_A$  such that  $A$  is not  $k$ -list replicable with respect to  $\mathcal{D}_A$ . However, a natural question is whether a lower bound holds for a fixed distribution, especially simple distributions such as the uniform distribution. We show that this is indeed the case.
Theorem 6.7. In the PAC model under the uniform distribution, there is a  $d + 1$ -list replicable algorithm for the  $d$ -THRESHOLD. Moreover, for any  $k < d + 1$ , there does not exist a  $k$ -list replicable algorithm for the concept class  $d$ -THRESHOLD under the uniform distribution. Thus its list complexity is exactly  $d + 1$ .
In this work, we investigated the pressing issue of replicability in machine learning from an algorithmic point of view. We observed that perfect replicability is not achievable and hence considered two natural extensions that capture the degree of (non) replicability: list and certificate replicability. We designed replicable algorithms with a small list, certificate, and sample complexities for the  $d$ -COIN BIAS ESTIMATION PROBLEM and the class of problems that can be learned via statistical query algorithms that make non-adaptive statistical queries. We also established certain impossibility results in the PAC model of learning and for  $d$ -COIN BIAS ESTIMATION PROBLEM. There are several interesting research directions that emerge from our work. There is a gap in the sample complexities of the list and certificate replicable algorithms with comparable parameters. Is this gap inevitable? Currently, there is an exponential gap in the replicability parameters between hypothesis classes that can be learned via non-adaptive and adaptive statistical queries. Is this gap necessary? A generic question is to explore the trade-offs between the sample complexities, list complexity, certificate complexities, adaptivity, and nonadaptivity.
We thank the NeurIPS 2023 reviewers for their comments that improved the presentation of the paper. We especially acknowledge reviewer U7Lz for very detailed and insightful comments. We thank an anonymous researcher for pointing out Observation 6.6. We thank Jamie Radcliffe for informative discussions on topics related to this work. We thank authors of (CMY23) for informing us about their work and helpful discussions. Part of this work is supported by NSF grants 2130536 and 2130608. Part of the work was done during Pavan and Vinodchandran's visit to Simons Institute for the Theory of Computing. Part of Peter's work was done while at Ben-Gurion University of Negev.