SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peter Dixon<sup>1</sup>, A. Pavan<sup>2</sup>, Jason Vander Woude<sup>3</sup>, and N. V. Vinodchandran<sup>4</sup>
$^{1}$ Independent Researcher
tooplark@gmail.com
$^{2}$ Iowa State University
pavan@cs.iastate.edu
<sup>3</sup>University of Nebraska-Lincoln
jasonvw@huskers.unl.edu
<sup>4</sup>University of Nebraska-Lincoln
vinod@cse.unl.edu
We investigate replicable learning algorithms. Informally a learning algorithm is replicable if the algorithm outputs the same canonical hypothesis over multiple runs with high probability, even when different runs observe a different set of samples from the unknown data distribution. In general, such a strong notion of replicability is not achievable. Thus we consider two feasible notions of replicability called list replicability and certificate replicability. Intuitively, these notions capture the degree of (non) replicability. The goal is to design learning algorithms with optimal list and certificate complexities while minimizing the sample complexity. Our contributions are the following.
- We first study the learning task of estimating the biases of  $d$  coins, up to an additive error of  $\varepsilon$ , by observing samples. For this task, we design a  $(d + 1)$ -list replicable algorithm. To complement this result, we establish that the list complexity is optimal, i.e. there are no learning algorithms with a list size smaller than  $d + 1$  for this task. We also design learning algorithms with certificate complexity  $\tilde{O}(\log d)$ . The sample complexity of both these algorithms is  $\tilde{O}\left(\frac{d^2}{\varepsilon^2}\right)$  where  $\varepsilon$  is the approximation error parameter (for a constant error probability).
- In the PAC model, we show that any hypothesis class that is learnable with  $d$ -nonadaptive statistical queries can be learned via a  $(d + 1)$ -list replicable algorithm and also via a  $\tilde{O} (\log d)$ -certificate replicable algorithm. The sample complexity of both these algorithms is  $\tilde{O}\left(\frac{d^2}{\nu^2}\right)$  where  $\nu$  is the approximation error of the statistical query. We also show that for the concept class  $d$ -THRESHOLD, the list complexity is exactly  $d + 1$  with respect to the uniform distribution.
To establish our upper bound results we use rounding schemes induced by geometric partitions with certain properties. We use Sperner/KKM Lemma to establish the lower bound results.
Replicability and reproducibility in science are critical concerns. The fundamental requirement that scientific results and experiments be replicable/reproducible is central to the development and evolution of science. In recent years, these concerns have grown as several scientific disciplines turn to data-driven research, which enables exponential progress through data democratization and affordable computing resources. The replicability issue has received attention from a wide spectrum of entities, from general media publications (for example, The Economist's "How Science Goes Wrong," 2013 (eco13)) to scientific publication venues (for example, see (JP05; Bak16)) to professional and scientific bodies such as the National Academy of Sciences, Engineering, and Medicine (NASEM). The emerging challenges to replicability and reproducibility have been discussed in depth by a consensus study report published by NASEM (NAS19).
A broad approach taken to ensure the reproducibility/replicability of algorithms is to make the datasets, algorithms, and code publicly available. Of late, conferences have been hosting replicability workshops to promote best practices and to encourage researchers to share code (see  $\mathrm{(PVLS^{+}21)}$  and (MPK19)). An underlying assumption is that consistent results can be obtained using the same input data, computational methods, and code. However, these practices alone are insufficient to ensure replicability as modern-day approaches use computations that inherently involve randomness.
Computing over random variables results in a high degree of non-replicability, especially in machine learning tasks. Machine learning algorithms observe samples from a (sometimes unknown) distribution and output a hypothesis. Such algorithms are inherently non-replicable. Two distinct runs of the algorithm will output different hypotheses as the algorithms see different sets of samples over the two runs. Ideally, to achieve "perfect replicability," we would like to design algorithms that output the same canonical hypothesis over multiple runs, even when different runs observe a different set of samples from the unknown distribution.
We first observe that perfect replicability is not achievable in learning, as a dependency of the output on the data samples is inevitable. We illustrate this with a simple learning task of estimating the bias of a coin: given  $n$  independent tosses of a coin with unknown bias  $b$ , output an estimate of  $b$  that is within an additive error of  $\varepsilon$  with high probability. It is relatively easy to argue that there is no algorithm that outputs a canonical estimate  $v_{b}$  with probability  $\geq 2/3$  so that  $|v_{b} - b| \leq \varepsilon$ . Suppose, for the sake of contradiction, such an algorithm  $A$  exists. Consider a sequence of coins with biases  $b_{1} < b_{2} < \dots < b_{m}$  where each  $b_{i+1} - b_{i} \leq \eta = 1/10n$ , and  $b_{m} - b_{1} \geq 2\varepsilon$ . For two adjacent biases  $b_{i}$  and  $b_{i+1}$ , the statistical distance (denoted by  $d_{\mathrm{TV}}$ ) between  $\mathcal{D}_{i+1}^{n}$  and  $\mathcal{D}_{i}^{n}$  is  $\leq n\eta$ , where  $\mathcal{D}_{i}^{n}$  is the distribution of  $n$  independent tosses of the  $i^{th}$  coin. Let  $v_{i+1}$  and  $v_{i}$  be the canonical estimates output by the algorithm for biases  $b_{i+1}$  and  $b_{i}$  respectively. Since  $A$  on samples from distribution  $\mathcal{D}_{i}^{n}$  outputs  $v_{i}$  with probability at least  $2/3$  and  $d_{\mathrm{TV}}(D_{i}^{n}, D_{i+1}^{n}) \leq n\eta$ ,  $A(\mathcal{D}_{i+1}^{n})$  must output  $v_{i}$  with probability at least  $2/3 - n\eta$ . Since  $A(\mathcal{D}_{i+1}^{n})$  must output a canonical value  $v_{i+1}$  with probability at least  $2/3$ , this implies that  $v_{i} = v_{i+1}$  (if not the probabilities will add up to  $> 1$ ). Thus, on all biases  $b_{1}, \ldots, b_{m}$ , the algorithm  $A$  should output the same value. This leads to a contradiction since  $b_{1}$  and  $b_{m}$  are  $2\varepsilon$  apart. However, it is easy to see that there is an algorithm for bias-estimation that outputs one of two canonical estimates with high probability using  $n = O(1/\varepsilon^{2})$  tosses: estimate the bias within an error of  $\varepsilon/2$  and round the value to the closest multiple of  $\varepsilon$ . The starting point of our work is these two observations. Even though it may not be possible to design learning algorithms that are perfectly replicable, it is possible to design algorithms whose "non-replicability" is minimized.
We study two notions of replicability called list replicability and certificate replicability which quantify the degree of (non)-replicability of learning algorithms. They are rooted in the pseudodeterminism literature (GG11; Gol19; GL19) which studied concepts known as multi-pseudodeterminism (related to list replicability) and influential bit algorithms (related to certificate replicability). Recently a notion of algorithmic stability and its variants such as list global stability, pseudo global stability, and  $\rho$ -replicability have been studied in the context of learning algorithms (BLM20; GKM21; ILPS22). These notions are very similar to each other and Section 2 discusses the similarities and connections.
Informally, an algorithm  $A$  is  $k$ -list replicable if there is a list  $L$  consisting of  $k$  (approximately correct) hypotheses so that the output of the algorithms  $A$  belongs to  $L$  with high probability. This implies that when the algorithm  $A$  is run multiple times, we see at most  $k$  distinct hypotheses (with
high probability). The value  $k$  is called the list complexity of the algorithm. An algorithm whose list complexity is 1 is perfectly replicable. Thus, list complexity can be considered as a degree of (non) replicability. The goal in this setting is to design learning algorithms that minimize the list complexity  $k$  for various learning tasks.
In certificate replicability, the learning algorithm has access to an  $\ell$ -bit random string that is independent of samples and the other randomness that the algorithm may use. It is required that for most  $\ell$ -bit random strings  $r$ , the algorithm must output a canonical (approximately correct) hypothesis  $h_r$  that depends only on  $r$ . Thus once we fix a good  $\ell$ -bit random string  $r$ , multiple runs of the algorithm will output the same hypothesis with high probability. We call  $\ell$  the certificate complexity of the algorithm. An algorithm with zero certificate complexity is perfectly replicable. Thus  $\ell$  is another measure of the degree of (non) replicability of the algorithm. The goal in this setting is to design learning algorithms that minimize the certificate complexity.
A critical resource in machine learning tasks is the sample complexity—the number of samples that the algorithm observes. This work initiates a study of learning algorithms that are efficient in list complexity, certificate complexity as well as sample complexity. The main contribution of this work is the design of algorithms with an optimal list and certificate complexities with efficient sample complexity for a few fundamental learning tasks.
Estimating the biases of  $d$  coins. We consider the basic problem of estimating the biases of  $d$  coins simultaneously by observing  $n$  tosses of each of the coins which we call  $d$ -COIN BIAS ESTIMATION PROBLEM. The task is to output an approximate bias vector  $\vec{v}$  with probability at least  $(1 - \delta)$  so that  $\| \vec{b} - \vec{v} \|_{\infty} \leq \varepsilon$  where  $\vec{b} = \langle b_1, \dots, b_d \rangle$  is the true bias vector, i.e.,  $b_i$  is the true bias of the  $i$ th coin. For this task, we establish the following results.
- There is a  $(d + 1)$ -list replicable learning algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM whose sample complexity (number of observed coin tosses) is  $n = O\left(\frac{d^2}{\varepsilon^2} \cdot \log \frac{d}{\delta}\right)$  per coin.
- There is a  $\lceil \log \frac{d}{\delta} \rceil$ -certificate replicable algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM with sample complexity  $n = O\left(\frac{d^2}{\varepsilon^2 \delta^2}\right)$  per coin.
- We establish the optimality of the above upper bounds in terms of list complexity. We show that there is no  $d$ -list replicable learning algorithm for  $d$ -COIN BIAS ESTIMATION PROBLEM.
While  $d$ -COIN BIAS ESTIMATION PROBLEM is a basic learning task and is of interest by itself, the techniques developed for this problem are applicable in the context of PAC learning.
PAC learning. We investigate list and certificate replicable PAC learning algorithms.
- We establish the following generic result: Any concept class that can be learned using  $d$  non-adaptive statistical queries can be learned by a  $(d + 1)$ -list replicable PAC learning algorithm with sample complexity  $O\left(\frac{d^2}{\nu^2} \cdot \log \frac{d}{\delta}\right)$  where  $\nu$  is the statistical query parameter. We also show that such concept classes admit a  $\lceil \log \frac{d}{\delta} \rceil$ -certificate replicable PAC learning algorithm with sample complexity  $O\left(\frac{d^2}{\nu^2 \delta^2} \cdot \log \frac{d}{\delta}\right)$ .
- We study the list complexity of the concept class  $d$ -THRESHOLD. Each hypothesis  $h_{\vec{t}}$  in this concept class is described via a  $d$ -dimensional vector  $\vec{t} \in [0,1]^d$ . For a hypothesis  $h_t$ ,  $h_t(\vec{x}) = 1$  if and only if  $x_i \leq t_i$ ,  $1 \leq i \leq d^1$ . We establish that, under the uniform distribution, the list complexity of  $d$ -THRESHOLD is exactly  $d + 1$ .
To establish our upper bound results we use rounding schemes induced by geometric partitions with certain properties. We use Sperner/KKM Lemma as a tool to establish the lower bound results. Due to space restrictions, most of the proofs are given in the supplementary material.
2 Prior and Related Work
Formalizing reproducibility and replicability has gained considerable momentum in recent years. While the terms reproducibility and replicability are very close and often used interchangeably, there
This concept class is the same as the class of axis-parallel rectangles in  $[0,1]^d$  with  $\vec{0}$  as one of the corners.
has been an effort to distinguish between them and accordingly, our notions fall in the replicability definition  $(\mathrm{PVLS}^{+}21)$ . As mentioned earlier, reproducibility, replicability, and related notions have been of considerable interest in the context of traditional randomized algorithms and learning algorithms. Here we discuss the prior work that is most relevant to the present work. A more detailed discussion of the prior and related work is provided in the supplementary material.
The seminal work of (BLM20) defined the notion of global stability in the context of learning, which is similar to the notion of pseudodeterminism (GG11) in the context of traditional algorithms. They define a learning algorithm  $A$  to be  $(n,\eta)$ -globally stable with respect to a distribution  $D$  if there is a hypothesis  $h$  such that  $\operatorname{Pr}_{S\sim D^n}(A(S) = h)\geq \eta$ , here  $\eta$  is called the stability parameter. Using this notion as an intermediate tool they established that every concept class with finite Littlestone dimension can be learned by an approximate differentially private algorithm. The work in (GKM21) extended the notion of global stability to list-global stability and pseudo-global stability. The authors of (GKM21) used these concepts to design user-level differentially private algorithms. The notion of pseudo-global stability is similar to the notion of certificate applicability, however, as defined, the notion of list-global stability differs from our notion of list replicability.
The recent work reported in (ILPS22) introduced the notion of  $\rho$ -replicability. A learning algorithm  $A$  is  $\rho$ -replicable if  $\operatorname*{Pr}[A(S_1,r) = A(S_2,r)] \geq 1 - \rho$ , where  $S_{1}$  and  $S_{2}$  are samples drawn from a distribution  $\mathcal{D}$  and  $r$  is the internal randomness of the learning algorithm  $A$ . They designed replicable algorithms for many learning tasks, including statistical queries, approximate heavy hitters, median, and learning half-spaces. It is known that the notions of pseudo-global stability and  $\rho$ -replicability are the same up to polynomial factors in the parameters (ILPS22; GKM21).
The present work introduces the notions of list and certificate complexities as measures of the degree of (non) replicability. Our goal is to design learning algorithms with optimal list and certificate complexities while minimizing the sample complexity. The earlier works did not focus on minimizing these quantities. The works of (BLM20; GKM21) used replicable algorithms as an intermediate step to design differentially private algorithms. The work of (ILPS22) did not consider reducing the certificate complexity in their algorithms and also did not study list replicability. The main distinguishing feature of our work from prior works is our focus on designing learning algorithms that are efficient in list, certificate, and sample complexities as well as establishing optimality results for list and certificate complexity.
A very recent and independent work of (CMY23) investigated relations between list replicability and the stability parameter  $\nu$ , in the context of distribution-free PAC learning. They showed that for every concept class  $\mathcal{H}$ , its list complexity is exactly the inverse of the stability parameter. They also showed that the list complexity of a hypothesis class is at least its VC dimension. For establishing this they exhibited, for any  $d$ , a concept class whose list complexity is exactly  $d$ . There are some similarities between their work and the present work. We establish similar upper and lower bounds on the list complexity but for different learning tasks:  $d$ -THRESHOLD and  $d$ -COIN BIAS ESTIMATION PROBLEM. For  $d$ -THRESHOLD, our results are for PAC learning under uniform distribution and do not follow from their distribution-independent results. Thus our results, though similar in spirit, are incomparable to theirs. Moreover, their work did not focus on efficiency in sample complexity and also did not study certificate complexity which is a focus of our paper. We do not study the stability parameter.
Finally, we would like to point out that there notions of list PAC learning and list-decodable learning in the learning theory literature (see (CP23) and (RY20) for recent progress on these notions). However, these notions are different from the list replicable learning that we consider in this paper. List PAC learning and list-decodable learning are generalized models of PAC learning. For example, any learning task that is PAC learnable is trivially list PAC learnable with a list size of 1. However, list replicable learning is an additional requirement that needs to be satisfied by a learner. Thus the notion of list and list-decodable PAC learning are different from the notions of list/certificate replicability.
3 List and Certificate Replicability Notions
We define list and certificate replicability notions for general learning tasks. A learning problem is a family  $\mathcal{D}$  of distributions over a domain  $\mathcal{X}$ , a set  $\mathcal{H}$  (representing hypotheses) and an error function  $err: \mathcal{D} \times \mathcal{H} \to [0, \infty)$ . The goal is to learn a hypothesis from  $\mathcal{H}$  by observing samples from a distribution  $D$  where  $D \in \mathcal{D}$  with a small error  $err(D, h)$ . A learning algorithm  $A$  has the following
inputs: (i)  $m$  independent samples from a distribution  $D \in \mathcal{D}$  and (ii)  $\varepsilon \in (0, \infty)$  and (iii)  $\delta \in (0, 1]$ . It may also receive additional inputs.
Definition 3.1 (List Replicability). Let  $k \in \mathbb{N}$ ,  $\varepsilon \in (0, \infty)$ , and  $\delta \in [0, 1]$ . A learning algorithm  $A$  is called  $(k, \varepsilon, \delta)$ -list replicable if the following holds: There exists  $n \in \mathbb{N}$  such that for every  $D \in \mathcal{D}$ , there exists a list  $L \subseteq \mathcal{H}$  of size at most  $k$  such that (i) for all  $h \in L$ ,  $err(D, h) \leq \varepsilon$ , and (ii)
$$
\operatorname * {P r} _ {s \sim D ^ {n}} [ A (s, \varepsilon , \delta) \in L ] \geq 1 - \delta .
$$
For  $k \in \mathbb{N}$ , we call  $A$  a list replicable if for all  $\varepsilon \in (0, \infty)$  and  $\delta \in (0, 1]$ ,  $A$  is  $(k, \varepsilon, \delta)$ -list replicable. We say that  $n$  is the sample complexity of  $A$  and  $k$  is the list complexity of  $A$ .
Definition 3.2 (Certificate Replicability). Let  $\ell \in \mathbb{N}$ ,  $\varepsilon \in (0,\infty)$ , and  $\delta \in [0,1]$ . A learning algorithm  $A$  is called  $(\ell ,\varepsilon ,\delta)$ -certificate replicable if the following holds: There exists  $n\in \mathbb{N}$  such that for every  $D\in \mathcal{D}$  there exists  $h:\{0,1\}^{\ell}\to \mathcal{H}$  such that
$$
\Pr_ {r \in \{0, 1 \} ^ {\ell}} \left[ \Pr_ {s \in D ^ {n}} \left[ A (s, \varepsilon , \delta , r) = h (r) \text {a n d} \operatorname {e r r} (D, h (r)) \leq \varepsilon \right] \geq 1 - \delta \right] \geq 1 - \delta .
$$
We can refine the above definition by introducing another probability parameter  $\rho$  and define a notion of  $(\ell, \varepsilon, \rho, \delta)$ -replicability. The definition is the same as above except that we require the outer probability to be  $1 - \rho$ , and the inner probability is  $1 - \delta$ . For simplicity, we take  $\rho$  to be  $\delta$  and work with the notion of  $(\ell, \varepsilon, \delta)$ -replicability.
The above definitions generalize the notion of perfect replicability, where it is required that the learning algorithm outputs a canonical hypothesis  $h$  (with  $err(D,h)\leq \varepsilon$ ) with probability  $\geq 1 - \delta$ . A motivation for these definitions is to characterize how close we can be to perfect replicability in scenarios where if perfect replicability is not achievable. Note that for list (certificate) replicability, when  $k = 1$  (respectively,  $\ell = 0$ ), we achieve perfect applicability. We note that the above definitions are inspired by multi-pseudodeterminism (Gol19) and influential-bit algorithms (GL19).
4 Primary Lemmas
Our upper bounds are based on certain rounding schemes and the lower bound is based on Sperner/KKM lemma. In this section, we state a few key technical lemmas that will be used in the rest of the paper. The proofs are in the supplementary material. We will use the following notation. We use  $\mathrm{diam}_{\infty}$  to indicate the diameter of a set relative to the  $\ell_{\infty}$  norm and  $\overline{B}_{\varepsilon}^{\infty}(\vec{p})$  to represent the closed ball of radius  $\varepsilon$  centered at  $\vec{p}$  relative to the  $\ell_{\infty}$  norm. That is, in  $\mathbb{R}^d$  we have  $\overline{B}_{\varepsilon}^{\infty}(\vec{p}) = \prod_{i=1}^{d}[p_i - \varepsilon, p_i + \varepsilon]$ .
We first state a lemma that gives a universal deterministic rounding algorithm that is used in designing list replicable algorithms. The lemma is based on the work in  $(\mathrm{VWDP}^{+}22)$  and is a byproduct of certain geometric partitions they call secluded partitions.
Lemma 4.1. Let  $d \in \mathbb{N}$  and  $\varepsilon \in (0, \infty)$ . Let  $\varepsilon_0 = \frac{\varepsilon}{2d}$ . There is an efficiently computable function  $f_{\varepsilon} : \mathbb{R}^d \to \mathbb{R}^d$  with the following two properties:
1. For any  $x \in \mathbb{R}^d$  and any  $\hat{x} \in \overline{B}_{\varepsilon_0}^\infty(x)$  it holds that  $f_{\varepsilon}(\hat{x}) \in \overline{B}_{\varepsilon}^\infty(x)$ .
2. For any  $x \in \mathbb{R}^d$  the set  $\left\{f_{\varepsilon}(\hat{x}) \colon \hat{x} \in \overline{B}_{\varepsilon_0}^{\infty}(x)\right\}$  has cardinality at most  $d + 1$ .
Item (1) states that if  $\hat{x}$  is an  $\varepsilon_0$  approximation of  $x$ , then  $f_{\varepsilon}(\hat{x})$  is an  $\varepsilon$  approximation of  $x$ , and Item (2) states that  $f_{\varepsilon}$  maps every  $\varepsilon_0$  approximation of  $x$  to one of at most  $d + 1$  possible values.
The following lemma gives a universal randomized rounding algorithm that is used in designing certificate replicable algorithms. We note that randomized rounding schemes have been used in a few prior works (SZ99; DPV18; Gol19; GL19; ILPS22). Our rounding scheme is more nuanced as it is geared towards minimizing certificate complexity.
Lemma 4.2. Let  $d \in \mathbb{N}$ ,  $\varepsilon_0 \in (0, \infty)$  and  $0 < \delta < 1$ . There is an efficiently computable deterministic function  $f: \{0, 1\}^{\ell} \times \mathbb{R}^{d} \to \mathbb{R}^{d}$  with the following property. For any  $x \in \mathbb{R}^{d}$ ,
$$
\Pr_ {r \in \{0, 1 \} ^ {\ell}} \left[ \exists x ^ {*} \in \overline {{B}} _ {\varepsilon} ^ {\infty} (x) \forall \hat {x} \in \overline {{B}} _ {\varepsilon_ {0}} ^ {\infty} (x): f (r, \hat {x}) = x ^ {*} \right] \geq 1 - \delta
$$
where  $\ell = \lceil \log \frac{d}{\delta} \rceil$  and  $\varepsilon = (2^{\ell} + 1)\varepsilon_{0} \leq \frac{2\varepsilon_{0}d}{\delta}$ .
The following result is a corollary to a cubical variant of Sperner's lemma/KKM lemma initially developed in (DLPES02) and expanded on in  $(\mathrm{VWDP}^{+}22)$ . We use this to establish our lower bound results.
Lemma 4.3. Let  $\mathcal{P}$  be a partition of  $[0,1]^d$  such that for each member  $X\in \mathcal{P}$ , it holds that  $\mathrm{diam}_{\infty}(X) < 1$ . Then there exists  $\vec{p}\in [0,1]^d$  such that for all  $\delta >0$  we have that  $\overline{B}_{\delta}^{\infty}(\vec{p})$  intersects at least  $d + 1$  members of  $\mathcal{P}$ .