Title: List and Certificate Complexities in Replicable Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Replicability

learning algorithms

sample complexity

PAC Learning
Abstract: We investigate replicable learning algorithms. Informally  a learning algorithm is replicable if the algorithm outputs the same canonical hypothesis over multiple runs with high probability, even when different runs observe a different set of samples from the unknown data distribution. In general, such a strong notion of replicability is not achievable. 
Thus we consider two feasible notions of replicability called {\em list replicability} and {\em certificate replicability}. 
Intuitively, these notions capture the degree of (non) replicability. The goal is to design learning algorithms with optimal list and certificate complexities while minimizing the sample complexity.  Our contributions are the following.

1. We first study the learning task of estimating the biases of $d$ coins, up to an additive error of $\varepsilon$, by observing samples. For this task, we design a $(d+1)$-list replicable algorithm. To complement this result, we establish that the list complexity is optimal, i.e there are no learning algorithms with a list size smaller than $d+1$ for this task. We also design learning algorithms with certificate complexity $\tilde{O}(\log d)$.   The sample complexity of both these algorithms is $\tilde{O}(\frac{d^2}{\varepsilon^2})$ where $\varepsilon$ is the approximation error parameter (for a constant error probability).  

2. In the PAC model, we show that any hypothesis class that is learnable with $d$-nonadaptive statistical queries can be learned via a $(d+1)$-list replicable algorithm and also via a $\tilde{O}(\log d)$-certificate replicable algorithm. The sample complexity of both these algorithms is $\tilde{O}(\frac{d^2}{\nu^2})$ where $\nu$ is the approximation error of the statistical query. We also show that for the concept class \dtep, the list complexity is exactly $d+1$ with respect to the uniform distribution.   

To establish our upper bound results we use rounding schemes induced by geometric partitions with certain properties. We use Sperner/KKM Lemma to establish the lower bound results.

List and Certificate Complexities in Replicable Learning

Abstract

We investigate replicable learning algorithms. Informally a learning algorithm is replicable if the algorithm outputs the same canonical hypothesis over multiple runs with high probability, even when different runs observe a different set of samples from the unknown data distribution. In general, such a strong notion of replicability is not achievable. Thus we consider two feasible notions of replicability called list replicability and certificate replicability . Intuitively, these notions capture the degree of (non) replicability. The goal is to design learning algorithms with optimal list and certificate complexities while minimizing the sample complexity. Our contributions are the following.

-We first study the learning task of estimating the biases of d coins, up to an additive error of ε , by observing samples. For this task, we design a ( d + 1) -list replicable algorithm. To complement this result, we establish that the list complexity is optimal, i.e there are no learning algorithms with a list size smaller than d +1 for this task. We also design learning algorithms with certificate complexity ˜ O (log d ) . The sample complexity of both these algorithms is ˜ O ( d 2 ε 2 ) where ε is the approximation error parameter (for a constant error probability).

-In the PAC model, we show that any hypothesis class that is learnable with d -nonadaptive statistical queries can be learned via a ( d +1) -list replicable algorithm and also via a ˜ O (log d ) -certificate replicable algorithm. The sample complexity of both these algorithms is ˜ O ( d 2 ν 2 ) where ν is the approximation error of the statistical query. We also show that for the concept class d -THRESHOLD, the list complexity is exactly d +1 with respect to the uniform distribution.

To establish our upper bound results we use rounding schemes induced by geometric partitions with certain properties. We use Sperner/KKM Lemma to establish the lower bound results.

1 Introduction

Replicability and reproducibility in science are critical concerns. The fundamental requirement that scientific results and experiments be replicable/reproducible is central to the development and evolution of science. In recent years, these concerns have grown as several scientific disciplines turn to data-driven research, which enables exponential progress through data democratization and affordable computing resources. The replicability issue has received attention from a wide spectrum of entities, from general media publications (for example, The Economist's 'How Science Goes Wrong,' 2013 (eco13)) to scientific publication venues (for example, see (JP05; Bak16)) to professional and scientific bodies such as the National Academy of Sciences, Engineering, and Medicine (NASEM). The emerging challenges to replicability and reproducibility have been discussed in depth by a consensus study report published by NASEM (NAS19).

A broad approach taken to ensure the reproducibility/replicability of algorithms is to make the datasets, algorithms, and code publicly available. Of late, conferences have been hosting replicability workshops to promote best practices and to encourage researchers to share code (see (PVLS + 21) and (MPK19)). An underlying assumption is that consistent results can be obtained using the same input data, computational methods, and code. However, these practices alone are insufficient to ensure replicability as modern-day approaches use computations that inherently involve randomness.

Computing over random variables results in a high degree of non-replicability, especially in machine learning tasks. Machine learning algorithms observe samples from a (sometimes unknown) distribution and output a hypothesis. Such algorithms are inherently non-replicable. Two distinct runs of the algorithm will output different hypotheses as the algorithms see different sets of samples over the two runs. Ideally, to achieve 'perfect replicability,' we would like to design algorithms that output the same canonical hypothesis over multiple runs, even when different runs observe a different set of samples from the unknown distribution.

We first observe that perfect replicability is not achievable in learning, as a dependency of the output on the data samples is inevitable. We illustrate this with a simple learning task of estimating the bias of a coin: given n independent tosses of a coin with unknown bias b , output an estimate of b that is within an additive error of ε with high probability. It is relatively easy to argue that there is no algorithm that outputs a canonical estimate v b with probability ≥ 2 / 3 so that | v b -b | ≤ ε . Suppose, for the sake of contradiction, such an algorithm A exists. Consider a sequence of coins with biases b 1 < b 2 < · · · < b m where each b i +1 -b i ≤ η = 1 / 10 n , and b m -b 1 ≥ 2 ε . For two adjacent biases b i and b i +1 , the statistical distance (denoted by d TV ) between D n i +1 and D n i is ≤ nη , where D n i is the distribution of n independent tosses of the i th coin. Let v i +1 and v i be the canonical estimates output by the algorithm for biases b i +1 and b i respectively. Since A on samples from distribution D n i outputs v i with probability at least 2 / 3 and d TV ( D n i , D n i +1 ) ≤ nη , A ( D n i +1 ) must output v i with probability at least 2 / 3 -nη . Since A ( D n i +1 ) must output a canonical value v i +1 with probability at least 2 / 3 , this implies that v i = v i +1 (if not the probabilities will add up to > 1 ). Thus, on all biases b 1 , . . . , b m , the algorithm A should output the same value. This leads to a contradiction since b 1 and b m are 2 ε apart. However, it is easy to see that there is an algorithm for bias-estimation that outputs one of two canonical estimates with high probability using n = O (1 /ε 2 ) tosses: estimate the bias within an error of ε/ 2 and round the value to the closest multiple of ε . The starting point of our work is these two observations. Even though it may not be possible to design learning algorithms that are perfectly replicable, it is possible to design algorithms whose 'non-replicability' is minimized.

Westudy two notions of replicability called list replicability and certificate replicability which quantify the degree of (non)-replicability of learning algorithms. They are rooted in the pseudodeterminismliterature (GG11; Gol19; GL19) which studied concepts known as multi-pseudodeterminism (related to list replicability) and influential bit algorithms (related to certificate replicability). Recently a notion of algorithmic stability and its variants such as list global stability , pseudo global stability , and ρ -replicability have been studied in the context of learning algorithms (BLM20; GKM21; ILPS22). These notions are very similar to each other and Section 2 discusses the similarities and connections.

1.1 Our Results

Informally, an algorithm A is k -list replicable if there is a list L consisting of k (approximately correct) hypotheses so that the output of the algorithms A belongs to L with high probability. This implies that when the algorithm A is run multiple times, we see at most k distinct hypotheses (with

high probability). The value k is called the list complexity of the algorithm. An algorithm whose list complexity is 1 is perfectly replicable. Thus, list complexity can be considered as a degree of (non) replicability. The goal in this setting is to design learning algorithms that minimize the list complexity k for various learning tasks.

In certificate replicability, the learning algorithm has access to an ℓ -bit random string that is independent of samples and the other randomness that the algorithm may use. It is required that for most ℓ -bit random strings r , the algorithm must output a canonical (approximately correct) hypothesis h r that depends only on r . Thus once we fix a good ℓ -bit random string r , multiple runs of the algorithm will output the same hypothesis with high probability. We call ℓ the certificate complexity of the algorithm. An algorithm with zero certificate complexity is perfectly replicable. Thus ℓ is another measure of the degree of (non) replicability of the algorithm. The goal in this setting is to design learning algorithms that minimize the certificate complexity.

A critical resource in machine learning tasks is the sample complexity -the number of samples that the algorithm observes. This work initiates a study of learning algorithms that are efficient in list complexity, certificate complexity as well as sample complexity. The main contribution of this work is the design of algorithms with an optimal list and certificate complexities with efficient sample complexity for a few fundamental learning tasks.

Estimating the biases of d coins. We consider the basic problem of estimating the biases of d coins simultaneously by observing n tosses of each of the coins which we call d -COIN BIAS ESTIMATION PROBLEM. The task is to output an approximate bias vector ⃗ v with probability at least (1 -δ ) so that ∥ ⃗ b -⃗ v ∥ ∞ ≤ ε where ⃗ b = ⟨ b 1 , · · · b d ⟩ is the true bias vector, i.e., b i is the true bias of the i th coin. For this task, we establish the following results.

-There is a ( d +1) -list replicable learning algorithm for d -COIN BIAS ESTIMATION PROBLEM whose sample complexity (number of observed coin tosses) is n = O ( d 2 ε 2 · log d δ ) per coin.

-There is a ⌈ log d δ ⌉ -certificate replicable algorithm for d -COIN BIAS ESTIMATION PROBLEM with sample complexity n = O ( d 2 ε 2 δ 2 ) per coin.

-We establish the optimality of the above upper bounds in terms of list complexity. We show that there is no d -list replicable learning algorithm for d -COIN BIAS ESTIMATION PROBLEM.

While d -COIN BIAS ESTIMATION PROBLEM is a basic learning task and is of interest by itself, the techniques developed for this problem are applicable in the context of PAC learning.

PAC learning. We investigate list and certificate replicable PAC learning algorithms.

-We establish the following generic result: Any concept class that can be learned using d nonadaptive statistical queries can be learned by a ( d +1) -list replicable PAC learning algorithm with sample complexity O ( d 2 ν 2 · log d δ ) where ν is the statistical query parameter. We also show that such concept classes admit a ⌈ log d δ ⌉ -certificate replicable PAC learning algorithm with sample complexity O ( d 2 ν 2 δ 2 · log d δ ) .

-We study the list complexity of the concept class d -THRESHOLD. Each hypothesis h ⃗ t in this concept class is described via a d -dimensional vector ⃗ t ∈ [0 , 1] d . For a hypothesis h t , h t ( ⃗ x ) = 1 if and only if x i ≤ t i , 1 ≤ i ≤ d 1 . We establish that, under the uniform distribution, the list complexity of d -THRESHOLD is exactly d +1 .

To establish our upper bound results we use rounding schemes induced by geometric partitions with certain properties. We use Sperner/KKM Lemma as a tool to establish the lower bound results. Due to space restrictions, most of the proofs are given in the supplementary material.

2 Prior and Related Work

Formalizing reproducibility and replicability has gained considerable momentum in recent years. While the terms reproducibility and replicability are very close and often used interchangeably, there

has been an effort to distinguish between them and accordingly, our notions fall in the replicability definition (PVLS + 21). As mentioned earlier, reproducibility, replicability, and related notions have been of considerable interest in the context of traditional randomized algorithms and learning algorithms. Here we discuss the prior work that is most relevant to the present work. A more detailed discussion of the prior and related work is provided in the supplementary material.

The seminal work of (BLM20) defined the notion of global stability in the context of learning, which is similar to the notion of pseudodeterminism (GG11) in the context of traditional algorithms. They define a learning algorithm A to be ( n, η ) -globally stable with respect to a distribution D if there is a hypothesis h such that Pr S ∼ D n ( A ( S ) = h ) ≥ η , here η is called the stability parameter . Using this notion as an intermediate tool they established that every concept class with finite Littlestone dimension can be learned by an approximate differentially private algorithm. The work in (GKM21) extended the notion of global stability to list-global stability and pseudo-global stability . The authors of (GKM21) used these concepts to design user-level differentially private algorithms. The notion of pseudo-global stability is similar to the notion of certificate applicability, however, as defined, the notion of list-global stability differs from our notion of list replicability.

The recent work reported in (ILPS22) introduced the notion of ρ -replicability . A learning algorithm A is ρ -replicable if Pr[ A ( S 1 , r ) = A ( S 2 , r )] ≥ 1 -ρ , where S 1 and S 2 are samples drawn from a distribution D and r is the internal randomness of the learning algorithm A . They designed replicable algorithms for many learning tasks, including statistical queries, approximate heavy hitters, median, and learning half-spaces. It is known that the notions of pseudo-global stability and ρ -replicability are the same up to polynomial factors in the parameters (ILPS22; GKM21).

The present work introduces the notions of list and certificate complexities as measures of the degree of (non) replicability. Our goal is to design learning algorithms with optimal list and certificate complexities while minimizing the sample complexity. The earlier works did not focus on minimizing these quantities. The works of (BLM20; GKM21) used replicable algorithms as an intermediate step to design differentially private algorithms. The work of (ILPS22) did not consider reducing the certificate complexity in their algorithms and also did not study list replicability. The main distinguishing feature of our work from prior works is our focus on designing learning algorithms that are efficient in list, certificate, and sample complexities as well as establishing optimality results for list and certificate complexity.

A very recent and independent work of (CMY23) investigated relations between list replicability and the stability parameter ν , in the context of distribution-free PAC learning. They showed that for every concept class H , its list complexity is exactly the inverse of the stability parameter. They also showed that the list complexity of a hypothesis class is at least its VC dimension. For establishing this they exhibited, for any d , a concept class whose list complexity is exactly d . There are some similarities between their work and the present work. We establish similar upper and lower bounds on the list complexity but for different learning tasks: d -THRESHOLD and d -COIN BIAS ESTIMATION PROBLEM. For d -THRESHOLD, our results are for PAC learning under uniform distribution and do not follow from their distribution-independent results. Thus our results, though similar in spirit, are incomparable to theirs. Moreover, their work did not focus on efficiency in sample complexity and also did not study certificate complexity which is a focus of our paper. We do not study the stability parameter.

Finally, we would like to point out that there notions of list PAC learning and list-decodable learning in the learning theory literature (see (CP23) and (RY20) for recent progress on these notions). However, these notions are different from the list replicable learning that we consider in this paper. List PAC learning and list-decodable learning are generalized models of PAC learning. For example, any learning task that is PAC learnable is trivially list PAC learnable with a list size of 1. However, list replicable learning is an additional requirement that needs to be satisfied by a learner. Thus the notion of list and list-decodable PAC learning are different from the notions of list/certificate replicability.

3 List and Certificate Replicability Notions

We define list and certificate replicability notions for general learning tasks. A learning problem is a family D of distributions over a domain X , a set H (representing hypotheses) and an error function err : D × H → [0 , ∞ ) . The goal is to learn a hypothesis from H by observing samples from a distribution D where D ∈ D with a small error err ( D,h ) . A learning algorithm A has the following

inputs: (i) m independent samples from a distribution D ∈ D and (ii) ε ∈ (0 , ∞ ) and (iii) δ ∈ (0 , 1] . It may also receive additional inputs.

Definition 3.1 (List Replicability) . Let k ∈ N , ε ∈ (0 , ∞ ) , and δ ∈ [0 , 1] . A learning algorithm A is called ( k, ε, δ ) -list replicable if the following holds: There exists n ∈ N such that for every D ∈ D , there exists a list L ⊆ H of size at most k such that (i) for all h ∈ L , err ( D,h ) ≤ ε , and (ii)

For k ∈ N , we call Ak -list replicable if for all ε ∈ (0 , ∞ ) and δ ∈ (0 , 1] , A is ( k, ε, δ ) -list replicable. We say that n is the sample complexity of A and k is the list complexity of A .

Definition 3.2 (Certificate Replicability) . Let ℓ ∈ N , ε ∈ (0 , ∞ ) , and δ ∈ [0 , 1] . Alearning algorithm A is called ( ℓ, ε, δ ) -certificate replicable if the following holds: There exists n ∈ N such that for every D ∈ D there exists h : { 0 , 1 } ℓ →H such that

We can refine the above definition by introducing another probability parameter ρ and define a notion of ( ℓ, ε, ρ, δ ) -replicability. The definition is the same as above except that we require the outer probability to be 1 -ρ , and the inner probability is 1 -δ . For simplicity, we take ρ to be δ and work with the notion of ( ℓ, ε, δ ) -replicability.

The above definitions generalize the notion of perfect replicability , where it is required that the learning algorithm outputs a canonical hypothesis h (with err ( D,h ) ≤ ε ) with probability ≥ 1 -δ . A motivation for these definitions is to characterize how close we can be to perfect replicability in scenarios where if perfect replicability is not achievable. Note that for list (certificate) replicability, when k = 1 (respectively, ℓ = 0 ), we achieve perfect applicability. We note that the above definitions are inspired by multi-pseudodeterminism (Gol19) and influential-bit algorithms (GL19).

4 Primary Lemmas

Our upper bounds are based on certain rounding schemes and the lower bound is based on Sperner/KKM lemma. In this section, we state a few key technical lemmas that will be used in the rest of the paper. The proofs are in the supplementary material. We will use the following notation. We use diam ∞ to indicate the diameter of a set relative to the ℓ ∞ norm and B ∞ ε ( p ⃗ ) to represent the closed ball of radius ε centered at p ⃗ relative to the ℓ ∞ norm. That is, in R d we have B ∞ ( p ⃗ ) = ∏ d [ p i -ε, p i + ε ] .

We first state a lemma that gives a universal deterministic rounding algorithm that is used in designing list replicable algorithms. The lemma is based on the work in (VWDP + 22) and is a byproduct of certain geometric partitions they call secluded partitions .

Lemma 4.1. Let d ∈ N and ε ∈ (0 , ∞ ) . Let ε 0 = ε 2 d . There is an efficiently computable function f ε : R d → R d with the following two properties:

For any x ∈ R d and any ˆ x ∈ B ∞ ε 0 ( x ) it holds that f ε (ˆ x ) ∈ B ∞ ε ( x ) .

For any x ∈ R d the set { f ε (ˆ x ): ˆ x ∈ B ∞ ε 0 ( x ) } has cardinality at most d +1 .

Item (1) states that if ˆ x is an ε 0 approximation of x , then f ε (ˆ x ) is an ε approximation of x , and Item (2) states that f ε maps every ε 0 approximation of x to one of at most d +1 possible values.

The following lemma gives a universal randomized rounding algorithm that is used in designing certificate replicable algorithms. We note that randomized rounding schemes have been used in a few prior works (SZ99; DPV18; Gol19; GL19; ILPS22). Our rounding scheme is more nuanced as it is geared towards minimizing certificate complexity.

Lemma 4.2. Let d ∈ N , ε 0 ∈ (0 , ∞ ) and 0 < δ < 1 . There is an efficiently computable deterministic function f : { 0 , 1 } ℓ × R d → R d with the following property. For any x ∈ R d ,

where ℓ = ⌈ log d δ ⌉ and ε = (2 ℓ +1) ε 0 ≤ 2 ε 0 d δ .

The following result is a corollary to a cubical variant of Sperner's lemma/KKM lemma initially developed in (DLPES02) and expanded on in (VWDP + 22). We use this to establish our lower bound results.

Lemma 4.3. Let P be a partition of [0 , 1] d such that for each member X ∈ P , it holds that diam ∞ ( X ) < 1 . Then there exists p ⃗ ∈ [0 , 1] d such that for all δ > 0 we have that B ∞ δ ( p ⃗ ) intersects at least d +1 members of P .

5 Replicability of Learning Coins Biases

In this section, we establish replicability results for estimating biases of d coins.

Definition 5.1. The d -COIN BIAS ESTIMATION PROBLEM is the following problem: Design an algorithm A (possibly randomized) that gets ε ∈ (0 , 1) , δ ∈ (0 , 1] as inputs, observes independent tosses of an ordered collection of d -many biased coins with a bias vector ⃗ b ∈ [0 , 1] d , and outputs ⃗ v so that ∥ ⃗ b -⃗ v ∥ ∞ ≤ ε with probability ≥ 1 -δ .

The d -COIN BIAS ESTIMATION PROBLEM fits in the framework of the general learning task introduced in Section 3 so that we can talk about list and certificate replicable algorithm for d -COIN BIAS ESTIMATION PROBLEM. We describe this now.

For d -COIN BIAS ESTIMATION PROBLEM we have the following. X = { 0 , 1 } d (where 0 corresponds to Tail and 1 corresponds to Head) which is the set of representations of all possibilities of flipping d -many coins. The class of distributions D is the set of all d -fold products of Bernoulli distributions. Each distribution D ∈ D is parameterized with a vector v D = ⟨ b 1 , · · · , b d ⟩ ∈ [0 , 1] d . When sampled according to v D we obtain a sample point ⟨ x 1 · · · x d ⟩ ∈ X , where Pr( x i = 1) = b i . We take the class of hypothesis H to be [0 , 1] d which is the set of all d -tuples representing biases of a collection of d -many coins. Lastly, the error function is defined as err ( D,h ) = ∥ v D -h ∥ ∞ .

5.1 Replicable Algorithms for d -COIN BIAS ESTIMATION PROBLEM

In this section, we design list and certificate replicable algorithms for d -COIN BIAS ESTIMATION PROBLEM .

Theorem 5.2. There exists an ( d + 1) -list replicable algorithm for d -COIN BIAS ESTIMATION PROBLEM . For a given ε and δ , its sample complexity is n = O ( d 2 ε 2 · log d δ ) , per coin.

Algorithm 1 ( d +1) -list replicable algorithm for d -COIN BIAS ESTIMATION PROBLEM

Input: ε > 0 , δ ∈ (0 , 1] , sample access to d coins with biases ⃗ b ∈ [0 , 1] d

Let g : R → [0 , 1] be the function which restricts coordinates to the unit interval (i.e.

) Take n samples from each coin and let ⃗ a return g ( f ( ⃗ a ))

Proof. Note that when ε ≥ 1 / 2 , a trivial algorithm that outputs a vector with 1 / 2 in each component works. Thus the most interesting case is when ε < 1 / 2 . Our list replicable algorithm is described in Algorithm 1. We will prove its correctness by associating for each possible bias ⃗ b ∈ [0 , 1] d , a set L ⃗ b with the three necessary properties: (1) | L ⃗ b | ≤ d +1 , (2) L ⃗ b ⊆ B ∞ ε ( ⃗ b ) (and also the problem specific

restriction that L ⃗ b ⊆ [0 , 1] d ), and (3) when given access to coins of biases ⃗ b , with probability at least 1 -δ the algorithm returns a value in L ⃗ b .

Let L ⃗ b = { g ( f ε ( ⃗ x )): ⃗ x ∈ B ∞ ε 0 ( ⃗ b ) } . By Lemma 4.1, f ε takes on at most d + 1 values on B ∞ ε 0 ( ⃗ b ) (which means g ◦ f ε also takes on at most d +1 values on this ball) which proves that | L ⃗ b | ≤ d +1 . This proves property (1).

Next we state the following observation which says that the coordinate restriction function g of Algorithm 1 does not reduce approximation quality. The proof is straightforward.

Observation 5.3. Using the notation of Algorithm 1, if ⃗ y ∈ B ∞ ε ( ⃗ b ) then g ( ⃗ y ) ∈ B ∞ ε ( ⃗ b ) .

We now establish Property (2). We know from Lemma 4.1 that for each ⃗ x ∈ B ∞ ε 0 ( ⃗ b ) we have f ε ( ⃗ x ) ∈ B ∞ ε ( ⃗ b ) , and by Observation 5.3, g maintains this quality and we have g ( f ε ( ⃗ x )) ∈ B ∞ ε ( ⃗ b ) . This shows that L ⃗ b ⊆ B ∞ ε ( ⃗ b ) proving property (2).

By Chernoff's bounds, for a single biased coin, with n = O ( ln(1 /δ 0 ) ε 2 0 ) independent samples of the coin we can estimate the bias within ε 0 with probability at least 1 -δ 0 . Thus, by a union bound, if we take n samples of each of the d coins, there is a probability of at most d · δ 0 = δ that at least one of the empirical coin biases is not within ε 0 of the true bias. Thus, by taking n samples of each coin, we have with probability at least 1 -δ that the empirical biases ⃗ a belong to B ∞ ε 0 ( ⃗ b ) . In the case that this occurs, we have by definition of L ⃗ b that the value g ( f ε ( ⃗ a )) returned by the algorithm belongs to the set L ⃗ b . This proves property (3). The sample complexity follows since ε 0 = ε 2 d and δ 0 = δ d .

The next result is on certificate replicable algorithm for d -COIN BIAS ESTIMATION PROBLEM.

Theorem 5.4. For every ε and δ , there is a ( ⌈ log d δ ⌉ , ε, δ ) -certificate replicable algorithm for d -COIN BIAS ESTIMATION PROBLEM with sample complexity of n = O ( d 2 ε 2 δ 2 ) per coin.

Proof. Let ε and δ be the input parameters to the algorithm and ⃗ b the bias vector. Set ε 0 = εδ 2 d . The algorithm A first estimates the bias of each coin with up to ε 0 with a probability error parameter δ d using a standard estimation algorithm. Note that this can be done using O ( d 2 ε 2 δ 2 ) tosses per coin. Let ⃗ v be the output vector. It follows that ⃗ v ∈ B ∞ ε 0 ( ⃗ b ) with probability at least 1 -δ . Then it runs the deterministic function f described in Lemma 4.2 with input r ∈ { 0 , 1 } ℓ with ℓ = ⌈ log d δ ⌉ and ⃗ v and outputs the value of the function. Lemma 4.2 guarantees that for 1 -δ fraction of the r s, all ⃗ v ∈ B ∞ ε 0 ( ⃗ b ) gets rounded to the same value by f . Hence algorithm A satisfies the requirements of the certificate-replicability. The certificate complexity is ⌈ log d δ ⌉ .

We remark that by using the refined definition of certificate replicability mentioned in Section 3, we can obtain a ( ⌈ log d ρ ⌉ , ε, ρ, δ ) -replicable algorithm with sample complexity O ( d 2 ε 2 ρ 2 log(1 /δ )) . Note that an ℓ -certificate replicable algorithm leads to a 2 ℓ -list replicable algorithm. Thus Theorem 5.4 gives a O ( d δ ) -list replicable algorithm for d -COIN BIAS ESTIMATION PROBLEM with sample complexity O ( d 2 ε 2 δ 2 ) . However, this is clearly sub-optimal and Theorem 5.2 gives an algorithm with a much smaller list and sample complexities. Also from the work of Goldrecih (Gol19), it follows that ℓ -list replicable algorithm can be converted into a ⌈ log( ℓ δ ) ⌉ -certificate replicable algorithm. However, this conversion increases the sample complexity. For example, when applied to d -COIN BIAS ESTIMATION PROBLEM, the sample complexity becomes O ( d 6 ε 2 δ 2 ) . In comparison, Theorem 5.4 uses a tailored rounding technique to achieve an algorithm with a much smaller sample complexity.

5.2 An Impossibility Result

This section establishes the optimality of the list complexity for d -COIN BIAS ESTIMATION PROBLEM.

Theorem 5.5. For k < d +1 , there does not exist a k -list replicable algorithm for the d -COIN BIAS ESTIMATION PROBLEM .

Before proving the theorem, we need a lemma that follows from the Data Processing Inequality; for more detail see the supplementary material. In the following, we use D A, ⃗ b,n to denote the distribution of the output of an algorithm for d -COIN BIAS ESTIMATION PROBLEM.

Proof. We use the basic fact that an algorithm (deterministic or randomized) cannot increase the total variation distance between two input distributions.

The distribution giving one sample flip of each coin in a collection with bias ⃗ b is the d -fold product of Bernoulli distributions ∏ d i =1 Bern( b i ) (which for notational brevity we denote as Bern( ⃗ b ), so the distribution which gives n independent flips of each coin is the n -fold product of this and is denoted as Bern( ⃗ b ) ⊗ n ). We will show that for two bias vectors ⃗ a and ⃗ b , d TV ( Bern( ⃗ b ) ⊗ n , Bern( ⃗ a ) ⊗ n ) ≤ n · d · ∥ ⃗ b -⃗ a ∥ ∞ . This suffices to establish the lemma.

Observe that we have for each i ∈ [ d ] ,

Hence we have and

Proof of Theorem 5.5. Fix any d ∈ N , and choose ε and δ as ε < 1 2 and δ ≤ 1 d +2 . Suppose for contradiction that such an algorithm A does exists for some k < d +1 . This means that for each possible bias vector ⃗ b ∈ [0 , 1] d , there exists some set L ⃗ b ⊆ H of hypotheses with three properties: (1) each element of L ⃗ b is an ε -approximation to h ⃗ b , (2) | L ⃗ b | ≤ k , and (3) with probability at least 1 -δ , A returns an element of L ⃗ b .

By an averaging argument, this means that there exists at least one element in L ⃗ b which is returned by A with probability at least 1 k · (1 -δ ) ≥ 1 k · (1 -1 d +2 ) = 1 k · d +1 d +2 ≥ 1 k · k +1 k +2 . Let f : [0 , 1] d → [0 , 1] d be a function which maps each bias ⃗ b to such an element of L ⃗ b . Since 1 k · k +1 k +2 > 1 k +1 , let η be such that 0 < η < 1 k · k +1 k +2 -1 k +1 .

The function f induces a partition P of [0 , 1] d where the members of P are the fibers of f (i.e. P = { f -1 ( ⃗ y ): ⃗ y ∈ range( f ) } ). By definition, for any member X ∈ P there exists some ⃗ y ∈ range( f ) such that for all ⃗ b ∈ X , f ( ⃗ b ) = ⃗ y . By definition, we have f ( ⃗ b ) ∈ L ⃗ b ⊆ B ∞ ε ( ⃗ b ) showing that ⃗ y ∈ B ∞ ε ( ⃗ b ) and by symmetry ⃗ b ∈ B ∞ ε ( ⃗ y ) . This shows that X ⊆ B ∞ ε ( ⃗ y ) , so diam ∞ ( X ) ≤ 2 ε < 1 .

̸

Let r = η · d n . Since every member of P has ℓ ∞ diameter less than 1 , by Lemma 4.3 there exists a point p ⃗ ∈ [0 , 1] d such that B ∞ r ( p ⃗ ) intersects at least d +1 > k members of P . Let ⃗ b (1) , . . . , ⃗ b ( d +1) be points belonging to distinct members of P that all belong to B ∞ r ( p ⃗ ) . By definition of P , this means for distinct j, j ′ ∈ [ d +1] that f ( ⃗ b ( j ) ) = f ( ⃗ b ( j ′ ) ) .

Now, for each j ∈ [ d + 1] , because ∥ p ⃗ -⃗ b ( j ) ∥ ∞ ≤ r , by Lemma 5.6 we have d TV ( D A,⃗ p,n , D A, b ⃗ ( j ) ,n ) ≤ n · d · r = η . However, this gives rise to a contradiction because the probability that A with access to biased coins ⃗ b ( j ) returns f ( ⃗ b ( j ) ) is at least 1 k · k +1 k +2 , and by Lemma 5.6, it must be that A with access to biased coins p ⃗ returns f ( ⃗ b ( j ) ) with probability at least 1 k · k +1 k +2 -η > 1 k +1 ; notationally, Pr D A, b ⃗ ( j ) ,n ( { f ( ⃗ b ( j ) ) } ) ≥ 1 k · k +1 k +2 and d TV ( D A, b ⃗ ( j ) ,n , D A,⃗ p,n ) ≤ η , so Pr D A,⃗ p,n ( { f ( ⃗ b ( j ) ) } ) ≥ 1 k · k +1 k +2 -η > 1 k +1 . This is a contradiction because a distribution cannot have d +1 ≥ k +1 disjoint events that each have probability greater than 1 k +1 .

6 List and Certificate Replicability in PAC Learning

In this section, we establish replicability results for the PAC model. We first note that PAC model fits in the general learning framework introduced in Section 3. Consider the PAC learning model where we have a domain X ′ , a concept class C (a family of subsets of X ′ ), and a hypothesis class H (a collection of functions X ′ →{ 0 , 1 } ). For any distribution D over X ′ and any concept indicator f from C , let D f denote the distribution over X ′ ×{ 0 , 1 } d obtained by sampling x ∼ D and outputting ⟨ x, f ( x ) ⟩ . We define the following learning problem in the general learning framework: X = X ′ ×{ 0 , 1 } , H = H , D = { D f : f ∈ C and D a distribution over X ′ } , and err ( D f , h ) = Pr x ∼D ′ [ f ( x ) = h ( x )] .

̸

We are interested in statistical query learning which is defined by Kearns (Kea98).

Definition 6.1. A statistical query oracle STAT ( D f , ν ) takes as an input a real-valued function ϕ : X ×{ 0 , 1 } → (0 , 1) and returns an estimate v such that | v -E ⟨ x,y ⟩∈ D f [ ϕ ( x, y )] | ≤ ν (Kea98). We say that an algorithm A learns a concept class H via statistical queries if for every distribution D and every function f ∈ H , for every 0 < ε < 1 , there exists ν such that the algorithm A on input ε , and STAT ( D f , ν ) as an oracle, outputs a hypothesis h such that err ( D f , h ) ≤ ε . The concept class is non-adaptively learnable if all the queries made by A to the statistical query oracle are non-adaptive.

6.1 Replicable PAC Learning Algorithms

Next we state main results of this section. Due to space limitation, all the proofs are in the supplementary material. We first show that any concept class that is learnable with d non-adpative statistical queries has a ( d +1) -list replicable algorithm.

Theorem 6.2. Let H be a concept class that is learnable with d non-adaptive statistical queries, then H is ( d +1) -list replicably learnable. Furthermore, the sample complexity n = n ( ν, δ ) of the ( d +1) -list replicable algorithm is O ( d 2 ν 2 · log d δ ) , where ν is the approximation error parameter of each statistical query oracle.

We note that we can simulate a statistical query algorithm that makes d adaptive queries to get a 2 d -list replicable learning algorithm. This can be done by rounding each query to two possible values (the approximation factor increases by 2). The sample complexity of this algorithm will be O ( d ν 2 · log d δ ) . Next, we design a certificate replicable algorithm for hypothesis classes that admit statistical query learning algorithms.

Theorem 6.3. Let H be a concept class that is learnable with d non-adaptive statistical queries, then H is ⌈ log d δ ⌉ -certificate replicably learnable. Furthermore, the sample complexity n = n ( ν, δ ) of this algorithm is O ( d 2 ν 2 δ 2 · log d δ ) , where ν is the approximation error parameter of each statistical query oracle.

We can also consider the case when the statistical query algorithm makes d adaptive queries. In this case we get the following theorem. Note that the certificate complexity is close to linear in d as oppose to logarithmic in the case of non-adaptive queries.

Theorem 6.4. Let H be a concept class that is learnable with d adaptive statistical queries, then H is ⌈ d log d δ ⌉ -certificate replicably learnable. Furthermore, the sample complexity of this algorithm is O ( d 3 ν 2 δ 2 · log d δ ) , where ν is the approximation error parameter of each statistical query oracle.

6.2 Impossibility Results in the PAC Model

In this section, we establish matching upper and lower bounds on the list complexity for the concept class d -THRESHOLD in the PAC model with respect to the uniform distribution. In particular, we establish that this learning task admits a ( d +1) -list replicable algorithm and does not admit a d -list replicable algorithm.

Definition 6.5 ( d -THRESHOLD) . Fix some d ∈ N . Let X = [0 , 1] d . For each value ⃗ t ∈ [0 , 1] d , let h ⃗ t : X → { 0 , 1 } be the concept defined as follows: h ⃗ t ( ⃗ x ) = 1 if for every i ∈ [ d ] it holds that x i ≤ t i and 0 otherwise. Let H be the hypothesis class consisting of all such threshold concepts: H = { h ⃗ t | ⃗ t ∈ [0 , 1] d } .

We first observe the impossibility of list-replicable algorithms in the distribution-free PAC model. This follows from known results.

Observation 6.6. There is no k -list replicable algorithm (for any k ) for d -THRESHOLD in the PAC model even when d = 1 .

The above observation follows from the works of (ALMM19) and (BLM20). It is known that d -THRESHOLD has an infinite Littlestone dimension. Suppose it admits k -list replicable algorithms in the PAC model. This implies that d -THRESHOLD is globally stable learnable with stability parameter 1 /k (Please see (BLM20) for the definition of global stability). The work of (BLM20) showed that any class that is globally stable learnable with a constant stability parameter is differentially private learnable. The work of (ALMM19) showed that if a concept class is differentially private learnable, then it has a finite Littlestone dimension. Putting these results together, we obtain that if d -THRESHOLD admits k -list replicable algorithm, then it has a finite Littlestone dimension which is a contradiction.

The above result implies that for every k and every learning algorithm A , there is some distribution D A such that A is not k -list replicable with respect to D A . However, a natural question is whether a lower bound holds for a fixed distribution, especially simple distributions such as the uniform distribution. We show that this is indeed the case.

Theorem 6.7. In the PAC model under the uniform distribution, there is a d + 1 -list replicable algorithm for the d -THRESHOLD . Moreover, for any k < d +1 , there does not exist a k -list replicable algorithm for the concept class d -THRESHOLD under the uniform distribution. Thus its list complexity is exactly d +1 .

7 Conclusions

In this work, we investigated the pressing issue of replicability in machine learning from an algorithmic point of view. We observed that perfect replicability is not achievable and hence considered two natural extensions that capture the degree of (non) replicability: list and certificate replicability. We designed replicable algorithms with a small list, certificate, and sample complexities for the d -COIN BIAS ESTIMATION PROBLEM and the class of problems that can be learned via statistical query algorithms that make non-adaptive statistical queries. We also established certain impossibility results in the PAC model of learning and for d -COIN BIAS ESTIMATION PROBLEM. There are several interesting research directions that emerge from our work. There is a gap in the sample complexities of the list and certificate replicable algorithms with comparable parameters. Is this gap inevitable? Currently, there is an exponential gap in the replicability parameters between hypothesis classes that can be learned via non-adaptive and adaptive statistical queries. Is this gap necessary? A generic question is to explore the trade-offs between the sample complexities, list complexity, certificate complexities, adaptivity, and nonadaptivity.

References

[AJJ + 22] Kwangjun Ahn, Prateek Jain, Ziwei Ji, Satyen Kale, Praneeth Netrapalli, and Gil I. Shamir. Reproducibility in optimization: Theoretical framework and limits, 2022. arXiv:2202.04598 .

[ALMM19] Noga Alon, Roi Livni, Maryanthe Malliaris, and Shay Moran. Private PAC learning implies finite littlestone dimension. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, Phoenix, AZ, USA, June 23-26, 2019 , pages 852-860. ACM, 2019.

[AV20] Nima Anari and Vijay V. Vazirani. Matching is as easy as the decision problem, in the NC model. In Thomas Vidick, editor, 11th Innovations in Theoretical Computer Science Conference, ITCS 2020, January 12-14, 2020, Seattle, Washington, USA , volume 151 of LIPIcs , pages 54:1-54:25. Schloss Dagstuhl - Leibniz-Zentrum f¨ ur Informatik, 2020.

[Bak16] Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature , 533:452-454, 2016.

[BLM20] Mark Bun, Roi Livni, and Shay Moran. An equivalence between private classification and online prediction. In Sandy Irani, editor, 61st IEEE Annual Symposium on Foundations of Computer Science, FOCS 2020, Durham, NC, USA, November 16-19, 2020 , pages 389-402. IEEE, 2020.

[CMY23] Zachary Chase, Shay Moran, and Amir Yehudayoff. Replicability and stability in learning, 2023. arXiv:2304.03757 .

[CP23] Moses Charikar and Chirag Pabbaraju. A characterization of list learnability. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing , STOC 2023, page 1713-1726, New York, NY, USA, 2023. Association for Computing Machinery. doi:10.1145/3564246.3585190 .

[DLPES02] Jesus A. De Loera, Elisha Peterson, and Francis Edward Su. A Polytopal Generalization of Sperner's Lemma. Journal of Combinatorial Theory, Series A , 100(1):126, October 2002. URL: https://www.sciencedirect.com/science/ article/pii/S0097316502932747 , doi:10.1006/jcta.2002.3274 .

[DPV18] Peter Dixon, Aduri Pavan, and N. V. Vinodchandran. On pseudodeterministic approximation algorithms. In Igor Potapov, Paul G. Spirakis, and James Worrell, editors, 43rd International Symposium on Mathematical Foundations of Computer Science, MFCS 2018, August 27-31, 2018, Liverpool, UK , volume 117 of LIPIcs , pages 61:1-61:11. Schloss Dagstuhl - Leibniz-Zentrum f¨ ur Informatik, 2018.

[DPVWV22] Peter Dixon, Aduri Pavan, Jason Vander Woude, and N. V. Vinodchandran. Pseudodeterminism: promises and lowerbounds. In Stefano Leonardi and Anupam Gupta, editors, STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, Rome, Italy, June 20 - 24, 2022 , pages 1552-1565. ACM, 2022.

[eco13] How science goes wrong. The Economist , pages 25-30, 2013.

[EKK + 23] Hossein Esfandiari, Alkis Kalavasis, Amin Karbasi, Andreas Krause, Vahab Mirrokni, and Grigoris Velegkas. Replicable bandits, 2023. arXiv:2210.01898 .

[GG] Shafi Goldwasser and Ofer Grossman. Bipartite perfect matching in pseudodeterministic NC. In Ioannis Chatzigiannakis, Piotr Indyk, Fabian Kuhn, and Anca Muscholl, editors, 44th International Colloquium on Automata, Languages, and Programming, ICALP 2017, July 10-14, 2017, Warsaw, Poland , volume 80 of LIPIcs .

[GG11] Eran Gat and Shafi Goldwasser. Probabilistic Search Algorithms with Unique Answers and Their Cryptographic Applications. Technical Report 136, 2011. URL: https: //eccc.weizmann.ac.il/report/2011/136/ .

[GGH18] Shafi Goldwasser, Ofer Grossman, and Dhiraj Holden. Pseudo-deterministic proofs. In Anna R. Karlin, editor, 9th Innovations in Theoretical Computer Science Conference, ITCS 2018, January 11-14, 2018, Cambridge, MA, USA , volume 94 of LIPIcs , pages 17:1-17:18. Schloss Dagstuhl - Leibniz-Zentrum f¨ ur Informatik, 2018.

[GGMW20] Shafi Goldwasser, Ofer Grossman, Sidhanth Mohanty, and David P. Woodruff. Pseudodeterministic streaming. In Thomas Vidick, editor, 11th Innovations in Theoretical Computer Science Conference, ITCS , volume 151 of LIPIcs , pages 79:1-79:25, 2020.

[GKM21] Badih Ghazi, Ravi Kumar, and Pasin Manurangsi. User-level differentially private learning via correlated sampling. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 2017220184, 2021.

[GL19] Ofer Grossman and Yang P. Liu. Reproducibility and pseudo-determinism in logspace. In Timothy M. Chan, editor, Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2019, San Diego, California, USA, January 6-9, 2019 , pages 606-620. SIAM, 2019. doi:10.1137/1.9781611975482. 38 .

[Gol19] Oded Goldreich. Multi-pseudodeterministic algorithms. Electron. Colloquium Comput. Complex. , TR19-012, 2019. URL: https://eccc.weizmann.ac.il/ report/2019/012 , arXiv:TR19-012 .

[ILPS22] Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell. Reproducibility in learning. In Stefano Leonardi and Anupam Gupta, editors, STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, Rome, Italy, June 20 - 24, 2022 , pages 818-831. ACM, 2022. doi:10.1145/3519935.3519973 .

[JP05] Ioannidis JP. Why most published research findings are false. PLOS Medicine , 2(8), 2005.

[Kea98] Michael J. Kearns. Efficient noise-tolerant learning from statistical queries. J. ACM , 45(6):983-1006, 1998.

[LOS21] Zhenjian Lu, Igor Carboni Oliveira, and Rahul Santhanam. Pseudodeterministic algorithms and the structure of probabilistic time. In Samir Khuller and Virginia Vassilevska Williams, editors, STOC '21: 53rd Annual ACM SIGACT Symposium on Theory of Computing, Virtual Event, Italy, June 21-25, 2021 , pages 303-316. ACM, 2021.

[MPK19] Harshal Mittal, Kartikey Pandey, and Yash Kant. Iclr reproducibility challenge report (padam : Closing the generalization gap of adaptive gradient methods in training deep neural networks), 2019. arXiv:1901.09517 .

[NAS19] Reproducibility and Replicability in Science. https://doi.org/10.17226/ 25303 , 2019. National Academies of Sciences, Engineering, and Medicine.

[OS17] I. Oliveira and R. Santhanam. Pseudodeterministic constructions in subexponential time. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2017, Montreal, QC, Canada, June 19-23, 2017 , pages 665-677, 2017.

[OS18] Igor Carboni Oliveira and Rahul Santhanam. Pseudo-derandomizing learning and approximation. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM 2018 , volume 116 of LIPIcs , pages 55:1-55:19, 2018.

[PVLS + 21] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alche Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research(a report from the neurips 2019 reproducibility program). Journal of Machine Learning Research , 22(164):1-20, 2021. URL: http://jmlr.org/papers/v22/20-303.html .

[RY20] Prasad Raghavendra and Morris Yau. List decodable learning via sum of squares. In Proceedings of the Thirty-First Annual ACM-SIAM Symposium on Discrete Algorithms , SODA '20, page 161-180, USA, 2020. Society for Industrial and Applied Mathematics.

[SZ99] Michael E. Saks and Shiyu Zhou. BP h SPACE(S) ⊆ DSPACE(S 3/2 ). J. Comput. Syst. Sci. , 58(2):376-403, 1999.

[VWDP + 22] Jason Vander Woude, Peter Dixon, Aduri Pavan, Jamie Radcliffe, and N. V. Vinodchandran. Geometry of rounding. CoRR , abs/2211.02694, 2022. arXiv:2211.02694 , doi:10.48550/arXiv.2211.02694 .

A Appendix

A.1 Missing Proofs from Section 4

To establish Lemmas 4.1 and 4.3, we build on the work of (VWDP + 22) and (DLPES02). We first introduce the necessary notion and definitions.

We recall the following notation from the main body of the paper. We use diam ∞ to indicate the diameter of a set relative to the ℓ ∞ norm and B ∞ ε ( p ⃗ ) to represent the closed ball of radius ε centered at p ⃗ relative to the ℓ ∞ norm. That is, in R d we have B ∞ ε ( p ⃗ ) = ∏ d i =1 [ p i -ε, p i + ε ] .

Lemma 4.1 is based on the construction of certain geometric partitions of R d called secluded partitions . Such partitions naturally induce deterministic rounding schemes which we use in the proof.

Let P be a partition of R d . For a point p ⃗ ∈ R d , let N ε ( p ⃗ ) denote the set of members of the partitions that have a non-empty intersection with the ε -ball around p ⃗ . That is,

̸

Definition A.1 (Secluded Partition) . Let P be a partition of R d . We say that P is ( k, ε ) -secluded , if for every point p ⃗ ∈ R d , | N ε ( p ⃗ ) | ≤ k .

The following theorem from (VWDP + 22) gives an explicit construction of a secluded partition with desired parameters where each member of the partition is a hypercube. For such partitions, we use the following notation. For every p ⃗ ∈ R d , if p ⃗ ∈ P , then the representative of p ⃗ , rep ( p ⃗ ) , is the center of the hypercube X .

Theorem A.2. For each d ∈ N , there exists a ( d +1 , 1 2 d ) -secluded partition, where each member of the partition is a unit hypercube. Moreover, the partition is efficiently computable: Given an arbitrary point ⃗ x ∈ R d , its representative can be computed in time polynomial in d .

A.1.1 Proof of Lemma 4.1

Lemma A.3 (Lemma 4.1) . Let d ∈ N and ε ∈ (0 , ∞ ) . Let ε 0 = ε 2 d . There is an efficiently computable function f ε : R d → R d with the following two properties:

For any x ∈ R d and any ˆ x ∈ B ∞ ε 0 ( x ) it holds that f ε (ˆ x ) ∈ B ∞ ε ( x ) .

For any x ∈ R d the set { f ε (ˆ x ): ˆ x ∈ B ∞ ε 0 ( x ) } has cardinality at most d +1 .

As explained in the main body, intuitively, item (1) states that if ˆ x is an ε 0 -approximation of x , then f ε (ˆ x ) is an ε -approximation of x , and item (2) states that f ε maps every ε 0 -approximation of x to one of at most d +1 possible values.

Proof. A high-level idea behind the proof is explained in Figure 1. We scale the ( d +1 , 1 2 d ) -secluded unit hypercube partition by ε so that each partition member is a hypercube with side length ε . Now, for a point x , the ball B ∞ ε 0 ( x ) intersects at most d +1 hypercubes. Consider a point ˆ x 1 ∈ B ∞ ε 0 ( x ) , it is rounded to c 1 (center of the hypercube it resides in). Note that c 1 lies in the ball of radius ε around x , this is because distance from x to ˆ x 1 is at most ε 0 and the distance from ˆ x 1 to c 1 is at most ε/ 2 . By triangle inequality c 1 belongs to B ∞ ε ( x ) . We now provide formal proof.

Let P be the ( d +1 , 1 2 d ) -secluded partition given by Theorem A.2. Thus P consists of unit cubes [0 , 1) d with the property that for any point p ⃗ ∈ R d the closed cube of side length 1 /d centered at p ⃗ (i.e. B ∞ 1 2 d ( p ⃗ ) ) intersects at most d +1 members/cubes of P .

We first define a rounding function f : R d → R d as follows: for every x ∈ R d , f ( x ) = rep ( x ) .

Observe that the rounding function f has the following two properties. (1) For every x ∈ R d , ∥ f ( x ) -x ∥ ∞ ≤ 1 2 . This is because every point x is mapped via f to its representative, which is the center of the unit cube in which it lies. (2) For any point p ⃗ ∈ R d , the set { f ( x ): x ∈ B ∞ 1 2 d ( p ⃗ ) } has

x

ˆ

x

1

ˆ

x

2

ˆ

x

3

c3

c2

c1

B

ε

0

(

x

)

ε

ε

cardinality at most d +1 . This is because B ∞ 1 2 d ( p ⃗ ) intersects at most d +1 hypercubes of P and for every hypercube X , all the points in X are mapped to its center by f .

The function f only gives an 1 2 -approximation guarantee. In order to get any ε -approximation guarantee, we scale f appropriately. f ε is this scaled version of f .

Define the function f ε : R d → R d as follows: for every ˆ x ∈ R d , f ε (ˆ x ) = ε · f ( 1 ε ˆ x ) . The efficient computability of f ε comes from the efficient computability of f .

We first establish that f ε has property (1) stated in the Lemma. Let x ∈ R d and ˆ x ∈ B ∞ ε 0 ( x ) . Then we have the following (justifications will follow):

The first line is by the definition of f ε , the second is the triangle inequality, the third is scaling of norms, the fourth uses the property of f that points are not mapped a distance more than 1 2 along with the hypothesis that ˆ x ∈ B ∞ ε 0 ( x ) , the fifth uses the definition of ε 0 , and the sixth uses the fact that d ≥ 1 .

Scaling both sides by ε and using the scaling of norms, the above gives us ∥ f ε (ˆ x ) -x ∥ ∞ ≤ ε which proves property (1) of the lemma.

To see that f ε has property (2), let x ∈ R d . We have the following set equalities:

The first line is from the definition of f ε , the second line is from re-scaling, and the third is from the definition of ε 0 .

Because f takes on at most d +1 distinct values on B ∞ 1 2 d ( 1 ε x ) , the set above has cardinality at most d +1 which proves property (2) of the lemma.

A.1.2 Proof of Lemma 4.2

LemmaA.4 (Lemma 4.2) . Let d ∈ N , ε 0 ∈ (0 , ∞ ) and 0 < δ < 1 . There is an efficiently computable deterministic function f : { 0 , 1 } ℓ × R d → R d with the following property. For any x ∈ R d ,

where ℓ = ⌈ log d δ ⌉ and ε = (2 ℓ +1) ε 0 ≤ 2 ε 0 d δ .

Proof. Partition each coordinate of R d into 2 ε 0 -width intervals. The algorithm computing the function f does the following simple randomized rounding:

The function f : Choose a random integer r ∈ { 1 . . . 2 ℓ } . Note that r can be represented using ℓ bits. Consider the i th coordinate of ˆ x denoted by ˆ x [ i ] . Round ˆ x [ i ] to the nearest k ∗ (2 ε 0 ) such that k mod 2 ℓ ≡ r .

Now we will prove that f satisfies the required properties.

First, we prove the approximation guarantee. Let x ′ denote the point in R d obtained after rounding each coordinate of ˆ x . The k s satisfying k mod 2 ℓ ≡ r are 2 ℓ · 2 ε 0 apart. Therefore, x ′ [ i ] is rounded by at most 2 ℓ ε 0 . That is, | x ′ [ i ] -ˆ x [ i ] | ≤ 2 ℓ ε 0 = ε 0 d δ for every i , 1 ≤ i ≤ d . Since ˆ x is an ε 0 -approximation (i.e. each coordinate ˆ x [ i ] is within ε 0 of the true value x [ i ] ), then each coordinate of x ′ is within (2 ℓ +1) ε 0 of x [ i ] . Therefore x ′ is a (2 ℓ +1) ε 0 -approximation of x [ i ] . Thus x ′ ∈ B ∞ ε ( x ) for any choice of r .

Now we establish that for ≥ 1 -δ fraction of r ∈ { 1 . . . 2 ℓ } , there exists x ∗ such every ˆ x ∈ B ∞ ε 0 ( x ) is rounded x ∗ . We argue this with respect to each coordinate and apply the union bound. Fix an x and a coordinate i . For x [ i ] , consider the ε 0 interval around it.

Consider r from { 1 . . . 2 ℓ } . When this r is chosen, then we round ˆ x [ i ] to the closest k ∗ (2 ε 0 ) such that k mod 2 ℓ ≡ r . Let p r 1 , p r 2 , . . . p r j . . . be the set of such points: more precisely p j = ( j 2 l + r ) ∗ 2 ε 0 . Note that ˆ x [ i ] is rounded to an p j to some j . Let m r j denote the midpoint between p r j and p r j +1 . I.e, m r = ( p r j + p r j +1 ) / 2 We call r 'bad' for x [ i ] if x [ i ] is close to some m r j . That is, r is 'bad' if | x [ i ] -m r j | < ε 0 . Note that for a bad r there exist ˆ x 1 and ˆ x 2 in B ∞ ε 0 ( x ) so that their i th coordinates are rounded to p r j and p r j +1 respectively. The crucial point is that if r is 'not bad' for x [ i ] , then for every x ′ ∈ B ∞ ε 0 ( x ) , there exists a canonical p ∗ such that x ′ [ i ] is rounded to p ∗ . We call r bad for x , if r is bad for x , if there exists at least one i , 1 ≤ i ≤ d such that r is bad for x [ i ] . With this, it follows that if r is not bad for x , then there exists a canonical x ∗ such that every x ′ ∈ B ∞ ε 0 ( x ) is rounded to x ∗ .

̸

With this, the goal is to bound the probability that a randomly chosen r is bad for x . For this, we first bound the probability that r is bad for x [ i ] . We will argue that there exists at most one bad r for x [ i ] . Suppose that there exist two numbers r 1 and r 2 that are both bad for x [ i ] . This means that | x [ i ] -m r 1 j 1 | < ε 0 and | x [ i ] -m r 2 j 2 | < ε 0 for some j 1 and j 2 . Thus by triangle inequality | m r 1 j 1 -m r 2 j 2 | < 2 ε 0 . However, note that | p r 1 j 1 -p r 2 j 2 | is | ( j 1 -j 2 )2 ℓ +( r 1 -r 2 ) | 2 ε 0 . Since r 1 = r 2 , this value is at least 2 ε 0 . This implies that the absolute value of difference between m r 1 j 1 and m r 2 j 2 is at least 2 ε leading to a contradiction.

Thus the probability that r is bad for x [ i ] is at most 1 2 ℓ and by the union bound the probability that r is bad for x is at most d 2 ℓ ≤ δ . This completes the proof.

A.1.3 Proof of Lemma 4.3

The proof, which is based on Sperner/KKM Lemma, is present in (VWDP + 22). Since our setting is slightly different, for completeness we give a proof.

We first introduce the necessary definitions and notation.

Definition A.5 (Sperner/KKM Coloring) . Let d ∈ N and V = { 0 , 1 } d denote a set of colors (which is exactly the set of vertices of [0 , 1] d so that colors and vertices are identified). Let χ : [0 , 1] d → V

be a coloring function such that for any face F of [0 , 1] d (of any dimension), for any x ∈ F , it holds that χ ( x ) ∈ V ( F ) where V ( F ) is the vertex set of F (informally, the color of x is one of the vertices in the face F ). Such a function χ will be called a Sperner/KKM coloring.

Theorem A.6 (Cubical Sperner/KKM lemma (DLPES02)) . Let d ∈ N and V = { 0 , 1 } d and χ : [0 , 1] d → V be a Sperner/KKM coloring. Then there exists a subset J ⊂ V with | J | = d +1 and a point ⃗ y ∈ [0 , 1] d such that for all j ∈ J , ⃗ y ∈ χ -1 ( j ) (informally, there is a point ⃗ y that is touching at least d +1 different colors).

We will need to relate partitions to Sperner/KKM coloring so that we can use the Sperner/KKM Lemma.

For any co-ordinate i , let π denote the standard projection map: π i : [0 , 1] d → [0 , 1] defined by π i ( x ) def = x i which maps d -dimensional points to the i th coordinate value. We extend this to sets: π i ( X ) = { π i ( x ) : x ∈ X } .

Definition A.7 (Non-Spanning partition) . Let d ∈ N and P be a partition of [0 , 1] d . We say that P is a non-spanning partition if it holds for all X ∈ P and for all i ∈ [ d ] that either π i ( X ) ̸∋ 0 or π i ( X ) ̸∋ 1 (or both).

Next, we state a lemma that asserts that for any non-spanning partition, there is a Sperner/KKM coloring that respects the partition: that is every member gets the same color.

Lemma A.8 (Coloring Admission) . Let d ∈ N , and V = { 0 , 1 } d , and P a non-spanning partition of [0 , 1] d . Then there exists a Sperner/KKM coloring χ : [0 , 1] d → V such that for every X ∈ P , for every x, y ∈ X , χ ( x ) = χ ( y ) .

Now we are ready to prove the Lemma 4.3.

Lemma A.9. (Lemma 4.3) Let P be a partition of [0 , 1] d such that for each member X ∈ P , it holds that diam ∞ ( X ) < 1 . Then there exists p ⃗ ∈ [0 , 1] d such that for all δ > 0 we have that B ∞ δ ( p ⃗ ) intersects at least d +1 members of P .

Proof. Consider an arbitrary X ∈ P . For each coordinate, i ∈ [ d ] , the set { x i : x ∈ X } does not contain both 0 and 1 (if it did, this would demonstrate two points in X that are ℓ ∞ distance at least 1 apart and contradict that diam ∞ ( X ) < 1 ). Thus, P is by definition a non-spanning partition of [0 , 1] d . Since P is non-spanning, by Lemma A.8, there is a Sperner/KKM coloring where each point of [0 , 1] d can be assigned one of 2 d -many colors and for any member X ∈ P , all points in X are assigned the same color. By Lemma A.6, there is a point p ⃗ ∈ [0 , 1] d such that p ⃗ belongs to the closure of at least d +1 colors. Since every point of a partition has the same color, each of these d +1 colors corresponds to at least d +1 different partitions. From this, it follows that or any δ > 0 , B ∞ δ ( p ⃗ ) intersects at least d +1 different members of P .

A.2 Missing Proofs From Section 6

A.2.1 Proofs of Theorem 6.2, Theorem 6.3

Theorem A.10 (Theorem 6.2) . Let H be a concept class that is learnable with d non-adaptive statistical queries, then H is ( d +1) -list replicably learnable. Furthermore, the sample complexity n = n ( ν, δ ) of the ( d +1) -list replicable algorithm is O ( d 2 ν 2 · log d δ ) , where ν is the approximation error parameter of each statistical query oracle.

Proof. The proof is very similar to the proof of Theorem 5.2. Our replicable algorithm B works as follows. Let ε and δ be input parameters and D be a distribution and f ∈ H . Let A be the statistical query learning algorithm for H . Let STAT ( D f , ν ) be the statistical query oracle for this algorithm. Let ϕ 1 , . . . , ϕ d be the statistical queries made by A .

Let ⃗ b = ⟨ b [1] , b [2] , . . . , b [ d ] ⟩ where b [ i ] = E ⟨ x,y ⟩∈D f [ ϕ i ( ⟨ x, y ⟩ ] , 1 ≤ i ≤ d . Set ε 0 = ν 2 d . The algorithm B first estimates the values b [ i ] up to an approximation error of ε 0 with success probably 1 -δ/d for each query. Note that this can be done by a simple empirical estimation algorithm, that

uses a total of n = O ( d 2 ν 2 · log d δ ) samples. Let ⃗ v be the estimated vector. It follows that ⃗ v ∈ B ∞ ε 0 ( ⃗ b ) with probability at least 1 -δ . Note that different runs of the algorithm will output different ⃗ v .

Next, the algorithm B evaluates the deterministic function f ε from Lemma 4.1 on input ⃗ v . Let ⃗ u be the output vector. Finally, the algorithm B simulates the statistical query algorithm A with ⃗ u [ i ] as the answer to the query ϕ i . By Lemma 4.1, ⃗ u ∈ B ∞ ν ( ⃗ b ) . Thus the error of the hypothesis output by the algorithm is at most ε . Since A is a deterministic algorithm the number of possible outputs only depends on the number of outputs of the function f ε , more precisely the number of possible outputs is the size of the set { f ε ( ⃗ v ) : v ∈ B ∞ ε 0 ( ⃗ b ) } which is almost d +1 , by Lemma 4.1. Thus the total number of possible outputs of the algorithm B is at most d +1 with probability at least 1 -δ .

Theorem A.11 (Theorem 6.3) . Let H be a concept class that is learnable with d non-adaptive statistical queries, then H is ⌈ log d δ ⌉ -certificate replicably learnable. Furthermore, the sample complexity n = n ( ν, δ ) of this algorithm equals O ( d 2 ν 2 δ 2 · log d δ ) , where ν is the approximation error parameter of each statistical query oracle.

Proof. The proof is very similar to the proof of Theorem 5.4. Our replicable algorithm B works as follows, let ε and δ be input parameters and D be a distribution and f ∈ H . Let A be the statistical query learning algorithm for H that outputs a hypothesis h with approximation error e D f ( h ) = ε . Let STAT ( D f , ν ) be the statistical query oracle for this algorithm. Let ϕ 1 , . . . , ϕ d be the statistical queries made by A .

Let ⃗ b = ⟨ b [1] , b [2] , · · · , b [ d ] ⟩ , where b [ i ] = E ⟨ x,y ⟩∈D f [ ϕ i ( ⟨ x, y ⟩ )] . Set ε 0 = νδ 2 d . The algorithm B first estimates the values b [ i ] , 1 ≤ i ≤ d up to an additive approximation error of ε 0 with success probably 1 -δ/d for each query. Note that this can be done by a simple empirical estimation algorithm that uses a total of n = O ( d 2 ν 2 δ 2 · log d δ ) samples. Let ⃗ v be the estimated the vector. It follows that ⃗ v ∈ B ∞ ε 0 ( ⃗ b ) with probability at least 1 -δ . Next, the algorithm B evaluates the deterministic function f described in Lemma 4.2 with inputs r ∈ { 0 , 1 } ℓ where ℓ = ⌈ log d δ ⌉ and ⃗ v . By Lemma 4.2 for at least 1 -δ fraction of the r 's , the function f outputs a canonical v ⃗ ∗ ∈ B ∞ ν ( ⃗ b ) . Finally, the algorithm B simulates the statistical query algorithm A with v ⃗ ∗ [ i ] as the answer to the query ϕ i . Since A is a deterministic algorithm it follows that our algorithm B is certificate replicable. Note that the certificate complexity is ℓ = ⌈ log d δ ⌉ .

The following theorem states how to convert adaptive statistical query learning algorithms into certificate replicable PAC learning algorithms. This result also appears in the work of (GKM21; ILPS22), though they did not state the certificate complexity. We explicitly state the result here.

Theorem A.12. ((GKM21; ILPS22))[Theorem 6.4] Let H be a concept class that is learnable with d adaptive statistical queries, then H is ⌈ d log d δ ⌉ -certificate replicably learnable. Furthermore, the sample complexity of this algorithm equals O ( d 3 ν 2 δ 2 · log d δ ) , where ν is the approximation error parameter of each statistical query oracle.

Proof. The proof uses similar arguments as before. The main difference is that we will evaluate each query with an approximation error of νδ d with a probability error of d/δ . This requires O ( d 2 ν 2 δ 2 · log d δ ) per query. We use a fresh set of certificate randomness for each such evaluation. Note that the length of the certificate for each query is ⌈ log d/δ ⌉ . Thus the total certificate complexity is ⌈ d log d δ ⌉ .

A.2.2 Proof of Theorem 6.7

We first recall the definition of the concept class d -THRESHOLD.

Fix some d ∈ N . Let X = [0 , 1] d . For each value ⃗ t ∈ [0 , 1] d , let h ⃗ t : X →{ 0 , 1 } be the concept defined as follows: h ⃗ t ( ⃗ x ) = 1 if for every i ∈ [ d ] it holds that x i ≤ t i and 0 otherwise. Let H be the hypothesis class consisting of all such threshold concepts: H = { h ⃗ t | ⃗ t ∈ [0 , 1] d } .

Theorem A.13 (Theorem 6.7) . In the PAC model under the uniform distribution, there is a d +1 -list replicable algorithm for the d -THRESHOLD . Moreover, for any k < d +1 , there does not exist a k -list replicable algorithm for the concept class d -THRESHOLD under the uniform distribution. Thus its list complexity is exactly d +1 .

It is easy to see that d -THRESHOLD is learnable under the uniform distribution by making d nonadaptive statistical queries. Thus by Theorem 6.2, d -THRESHOLD under the uniform distribution admits a ( d +1) -list replicable algorithm. So we will focus on proving the lower bound which is stated as a separate theorem below.

Theorem A.14. For k < d +1 , there does not exist a k -list replicable algorithm for the d -THRESHOLD in the PAC model under uniform distribution.

The proof is similar to the proof of Theorem 5.5. The reason is that sampling d -many biased coins with bias vector ⃗ b is similar to obtaining a point ⃗ x uniformly at random from [0 , 1] d and evaluating the threshold function h ⃗ b on it-this corresponds to asking whether all of the coins were heads/ 1 's. The two models differ though, because in the sample model for the d -COIN BIAS ESTIMATION PROBLEM, the algorithm sees for each coin whether it is heads or tails, but this information is not available in the PAC model for the d -THRESHOLD. Conversely, in the PAC model for the d -THRESHOLD, a random draw from [0 , 1] d is available to the algorithm, but in the sample model for the d -COIN BIAS ESTIMATION PROBLEM the algorithm does not get this information.

̸

Furthermore, there is the following additional complexity in the impossibility result for the d -THRESHOLD. In the d -COIN BIAS ESTIMATION PROBLEM, we said by definition that a collection of d coins parameterized by bias vector ⃗ a was an ε -approximation to a collection of d coins parameterized by bias vector ⃗ b if and only if ∥ ⃗ b -⃗ a ∥ ∞ ≤ ε , and we used this norm in the proofs. However, the notion of ε -approximation in the PAC model is quite different than this. It is possible to have a hypotheses h ⃗ a and h ⃗ b in the d -THRESHOLDsuch that ∥ ⃗ b -⃗ a ∥ ∞ > ε but with respect to some distribution D X on the domain X we have err D X ( h ⃗ a , h ⃗ b ) ≤ ε . For example, if D X is the uniform distribution on X = [0 , 1] d and ⃗ a = ⃗ 0 and ⃗ b is the first standard basis vector ⃗ b = ⟨ 1 , 0 , . . . , 0 ⟩ , and ε = 1 2 , then ∥ ⃗ b -⃗ a ∥ ∞ = 1 > ε , but err D X ( h ⃗ a , h ⃗ b ) = 0 ≤ ε because h ⃗ a ( ⃗ x ) = h ⃗ b ( ⃗ x ) if and only if all of the last d -1 coordinates of ⃗ x are 0 and the first coordinate is > 0 , but there is probability 0 of sampling such ⃗ x from the uniform distribution on X = [0 , 1] d .

For this reason, we can't just partition [0 , 1] d as we did with the proof of Theorem 5.5 and must do something more clever. It turns out that it is possible to find a subset [ α, 1] d on which hypotheses parameterized by vectors on opposite faces of this cube [ α, 1] d have high PAC error between them. A consequence by the triangle inequality of err D X is that two such hypotheses cannot both be approximated by a common third hypothesis. This is the following lemma states.

Lemma A.15. Let d ∈ N and α = d -1 d . Let ⃗ s, ⃗ t ∈ [ α, 1] d such that there exists a coordinate i 0 ∈ [ d ] where s i 0 = α and t i 0 = 1 (i.e. ⃗ s and ⃗ t are on opposite faces of this cube). Let ε ≤ 1 8 d . Then there is no point ⃗ r ∈ X such that both err unif ( h ⃗ s , h ⃗ r ) ≤ ε and err unif ( h ⃗ t , h ⃗ r ) ≤ ε (i.e. there is no hypothesis which is an ε -approximation to both h ⃗ s and h ⃗ t ).

̸

We need the following claim.

Claim A.16. For each ⃗ x ∈ X , the following are equivalent:

̸

h q ⃗ ( ⃗ x ) = h ⃗ t ( ⃗ x ) 2. h q ⃗ ( ⃗ x ) = 0 and h ⃗ t ( ⃗ x ) = 1

Furthermore, the above equivalent conditions imply the following:

̸

Proof of Claim A.16.

̸

(1) = ⇒ (2): Note that because q i 0 = s i 0 = α < 1 = t i 0 , we have for all i ∈ [ d ] that q i ≤ t i . If h ⃗ t ( ⃗ x ) = 0 then for some i 1 ∈ [ d ] it must be that x i 1 > t i 1 , but since t i 1 ≥ q i 1 it would also be the case that x i 1 > q i 1 , so h q ⃗ ( ⃗ x ) = 0 which gives the contradiction that h q ⃗ ( ⃗ x ) = h ⃗ t ( ⃗ x ) . Thus h ⃗ t ( ⃗ x ) = 1 , and since h q ⃗ ( ⃗ x ) = h ⃗ t ( ⃗ x ) we have h q ⃗ ( ⃗ x ) = 0 .

(1) ⇐⇒ (3): We partition [0 , 1] d into three sets and examine these three cases.

̸

Case 1: x i 0 ∈ ( q i 0 , t i 0 ] = ( α, 1] and for all i ∈ [ d ] \ { i 0 } , x i ∈ [0 , t i ] . In this case, q i 0 < x i 0 so h q ⃗ ( ⃗ x ) = 0 and for all i ∈ [ d ] x i ≤ t i , so h ⃗ t ( ⃗ x ) = 1 , so h q ⃗ ( ⃗ x ) = h ⃗ t ( ⃗ x ) .

Case 2: x i 0 ̸∈ ( q i 0 , t i 0 ] = ( α, 1] and for all i ∈ [ d ] \ { i 0 } , x i ∈ [0 , t i ] . In this case, because x i 0 ∈ [0 , 1] and x i 0 ̸∈ ( α, 1] we have x i 0 ≤ α = q i 0 ≤ t i 0 and also for all other i ∈ [ d ] \ { i 0 } , x i ≤ t i = q i (by definition of ⃗ q ). Thus h q ⃗ ( ⃗ x ) = 1 = h ⃗ t ( ⃗ x ) .

Case 3: For some i 1 ∈ [ d ] \ { i 0 } , x i 1 ̸∈ [0 , t i 1 ] . In this case, because x i 1 ∈ [0 , 1] , we have x i 1 > t i 1 = q i 1 . Thus h q ⃗ ( ⃗ x ) = 0 = h ⃗ t ( ⃗ x ) .

̸

Thus, it is the case that h q ⃗ ( ⃗ x ) = h ⃗ t ( ⃗ x ) if and only if x i 0 ∈ ( q i 0 , t i 0 ] = ( α, 1] and for all i ∈ [ d ] \{ i 0 } , x i ∈ [0 , t i ] .

̸

(1, 2, 3) = ⇒ (4): By (2), we have x i 0 > q i 0 , and since q i 0 = s i 0 by definition of ⃗ q , it follows that x i 0 > s i 0 which means h ⃗ s ( ⃗ x ) = 0 . By (3), h ⃗ t ( ⃗ x ) = 1 which gives h ⃗ s ( ⃗ x ) = h ⃗ t ( ⃗ x ) .

We also need the following Lemma.

Lemma A.17. Let d ∈ N and α = d -1 d = 1 -1 d . Then (1 -α ) · α d -1 > 1 4 d .

Proof. If d = 1 , then α = 0 so (1 -α ) · α d -1 = 1 ≥ 1 4 = 1 4 d (see footnote 2 ).

If d ≥ 2 , then we utilize the fact that (1 -1 d ) d ≥ 1 4 in the following:

This completes the proof. As an aside, α = d -1 d is the value of α that maximizes the expression (1 -α ) · α d -1 which is why that value was chosen.

With the above Claim and Lemma in hand, we return to the proof of Lemma A.15. Our next step will be two prove the following two inequalities:

For the second of these inequalities, note that by the (1) = ⇒ (4) part of claim above, since h q ⃗ ( ⃗ x ) = h ⃗ t ( ⃗ x ) implies h ⃗ s ( ⃗ x ) = h ⃗ t ( ⃗ x ) we have

̸

̸

̸

̸

Now, for the first of the inequalities above, we will use the (1) ⇐⇒ (3) portion of the claim, we will use our hypothesis that ⃗ t ∈ [ α, 1] d (which implies for each i ∈ [ d ] that [0 , t i ] ⊆ [0 , α ] ), we will use the hypothesis that ε ≤ 1 8 d , and we will use Theorem A.17. Utilizing these, we get the following:

̸

Thus, we get the desired two inequalities:

This nearly completes the proof. If there existed some point ⃗ r ∈ X such that both err unif ( h ⃗ s , h ⃗ r ) ≤ ε and err unif ( h ⃗ t , h ⃗ r ) ≤ ε , then it would follow from the triangle inequality of err unif that

but this would contradict the above inequalities, so no such ⃗ r exists.

Equipped with the Lemma A.15, we are now ready to prove Theorem A.14.

Proof of Theorem A.14. Fix any d ∈ N , and choose ε and δ as ε ≤ 1 4 d and δ ≤ 1 d +2 . We will use the constant α = d -1 d and consider the cube [ α, 1] d .

Suppose for contradiction such an algorithm A does exists for some k < d +1 . This means that for each possible threshold ⃗ t ∈ [0 , 1] d , there exists some set L ⃗ t ⊆ H of hypotheses with three properties: (1) each element of L ⃗ t is an ε -approximation to h ⃗ t , (2) | L ⃗ t | ≤ k , and (3) with probability at least 1 -δ , A returns an element of L ⃗ t .

By the trivial averaging argument, this means that there exists at least one element in L ⃗ t which is returned by A with probability at least 1 k · (1 -δ ) ≥ 1 k · (1 -1 d +2 ) = 1 k · d +1 d +2 ≥ 1 k · k +1 k +2 . Let f : [ α, 1] d → [0 , 1] d be a function which maps each threshold ⃗ t ∈ [ α, 1] d to such an element (the maximum probability element with ties broken arbitrarily) of L ⃗ t . This is slightly different from the proof of Theorem 5.5 because we are defining the function f on only a very specific subset of the possible thresholds. The reason for this was alluded to in the discussion following the statement of Theorem A.14.

The function f induces a partition P of [ α, 1] d where the members of P are the fibers of f (i.e. P = { f -1 ( ⃗ y ): ⃗ y ∈ range( f ) } ). For any member W ∈ P and any coordinate i ∈ [ d ] , it cannot be that the set { w i : ⃗ w ∈ W } contains both values α and 1 -if it did, then there would be two points ⃗ s, ⃗ t ∈ W such that s i = α and t i = 1 , but because they both belong to W , there is some ⃗ y ∈ [0 , 1] d such that f ( ⃗ s ) = ⃗ y = f ( ⃗ t ) , but by definition of the partition, h ⃗ y would have to be an ε -approximation (in the PAC model) of both h ⃗ s and h ⃗ t , but by Lemma A.15 this is not possible.

Thus, the partition P is a non-spanning partition of [ α, 1] d as in the proof of Lemma 4.3, so there is some point p ⃗ ∈ [ α, 1] d such that for every radius r > 0 , it holds that B ∞ r ( p ⃗ ) intersects at least d +1

̸

̸

members of P . In fact, there is some radius r such that ∥ ⃗ t -⃗ s ∥ ∞ ≤ r , then d TV ( D A,⃗ s,n , D A,⃗ t,n ) ≤ η , for η that lies between 0 and 1 k · k +1 k +2 -1 k +1 .

Now we get the same type of contradiction as in the proof of Theorem 5.5: for the special point p ⃗ we have that D A,⃗ p,n is a distribution that has d +1 ≥ k +1 disjoint events that each have probability greater than 1 k +1 . Thus, no k -list replicable algorithm exists.

B Expanded Prior and Related Work

We give a more detailed discussion on prior and related work. This section is an elaboration of the Section 2 from the main body of the paper.

Formalizing reproducibility and replicability has gained considerable momentum in recent years. While the terms reproducibility and replicability are very close and often used interchangeably, there has been an effort to distinguish between them and accordingly, our notions fall in the replicability definition (PVLS + 21).

In the context of randomized algorithms, various notions of reproducibility/replicability have been investigated. The work of Gat and Goldwasser (GG11) formalized and defined the notion of pseudodeterministic algorithms . A randomized algorithm A is pseudodeterministic if, for any input x , there is a canonical value v x such that Pr[ A ( x ) = v x ] ≥ 2 / 3 . Gat and Goldwasser designed polynomial-time pseudodeterministic algorithms for algebraic computational problems, such as finding quadratic non-residues and finding non-roots of multivariate polynomials (GG11). Later works studied the notion of pseudodeterminism in other algorithmic settings, such as parallel computation, streaming and sub-linear algorithms, interactive proofs, and its connections to complexity theory (GG; GGH18; OS17; OS18; AV20; GGMW20; LOS21; DPVWV22).

In the algorithmic setting, mainly two generalizations of pseudodeterminism have been investigated: multi-pseudodeterministic algorithms (Gol19) and influential bit algorithms (GL19). A randomized algorithm A is k -pseudodeterministic if, for every input x , there is a set S x of size at most k such that the output of A ( x ) belongs to the set S x with high probability. When k = 1 , we get pseudodeterminism. A randomized algorithm A is ℓ -influential-bit algorithm if, for every input x , for most of the strings r of length ℓ , there exists a canonical value v x,r such that the algorithm A on inputs x and r outputs v x,r with high probability. The string r is called the influential bit string. Again, when ℓ = 0 , we get back pseudodeterminism. The main focus of these works has been to investigate reproducibility in randomized search algorithms.

Very recently, pseudodeterminism and its generalizations have been explored in the context of learning algorithms to formalize the notion of replicability. The seminal work of (BLM20) defined the notion of global stability . They define a learning algorithm A to be ( n, η ) -globally stable with respect to a distribution D if there is a hypothesis h such that Pr S ∼ D n ( A ( S ) = h ) ≥ η , here η is called the stability parameter . Note that the notion of global stability is equivalent to Gat and Goldwasser's notion of pseudodeterminism when η = 2 / 3 . Since Gat and Goldwasser's motivation is to study pseudodeterminism in the context of randomized algorithms, the success probability is taken as 2 / 3 . In the context of learning, studying the stability parameter η turned out to be useful. The work of Bun, Livny and Moran (BLM20) showed that any concept class with Littlestone dimension d has an ( m,η ) -globally stable learning algorithm with m = ˜ O (2 2 d /α ) and η = ˜ O (2 -2 d ) , where the error of h (with respect to the unknown hypothesis) is ≤ α . Then they established that a globally stable learner implies a differentially private learner. This, together with an earlier work of Alon, Livny, Malliaris, and Moran (ALMM19), establishes an equivalence between online learnability and differentially private PAC learnability.

The work of Ghazi, Kumar, and Manurangsi (GKM21) extended the notion of global stability to pseudo-global stability and list-global stability. The notion of pseudo-global stability is very similar to the earlier-mentioned notion of influential bit algorithms of Grossman and Liu (GL19) when translated to the context of learning. The work of (GKM21) used these concepts to design user-level differentially private algorithms.

The recent work reported in (ILPS22) introduced the notion of ρ -replicability . Alearning algorithm A is ρ -replicable if Pr S 1 ,S 2 ,r [ A ( S 1 , r ) = A ( S 2 , r )] ≥ 1 -ρ , where S 1 and S 2 are samples drawn from a distribution D and r is the internal randomness of the learning algorithm A . They designed replicable

algorithms for many learning tasks, including statistical queries, approximate heavy hitters, median, and learning half-spaces. It is known that the notions of pseudo-global stability and ρ -replicability are the same up to polynomial factors in the parameters (ILPS22; GKM21).

In this work, we study the notions of list and certificate complexities as a measure the degree of (non) replicability . Our goal is to design learning algorithms with optimal list and certificate complexities while minimizing the sample complexity. The earlier works (BLM20; GKM21; ILPS22) did not focus on minimizing these quantities. The works of (BLM20; GKM21) used replicable algorithms as an intermediate step to design differentially private algorithms. The work of (ILPS22) did not consider reducing the certificate complexity in their algorithms and also did not study list-replicability. Earlier works (GKM21; ILPS22) studied how to convert statistical query learning algorithms into certificate replicable learning algorithms, however, their focus was not on the certificate complexity. Here, we study the relationship among (nonadaptive and adaptive) statistical query learning algorithms, list replicable algorithms, and certificate replicable algorithms with a focus on list, certificate and sample complexities.

A very recent and independent work of (CMY23) investigated relations between list replicability and the stability parameter ν , in the context of distribution-free PAC learning. They showed that for every concept class H , its list complexity is exactly the inverse of the stability parameter. They also showed that the list complexity of a hypothesis class is at least its VC dimension. For establishing this they exhibited, for any d , a concept class whose list complexity is exactly d . There are some similarities between their work and the present work. We establish similar upper and lower bounds on the list complexity but for different learning tasks: d -THRESHOLD and d -COIN BIAS ESTIMATION PROBLEM. For d -THRESHOLD, our results are for PAC learning under uniform distribution and do not follow from their distribution-independent results. Thus our results, though similar in spirit, are incomparable to theirs. Moreover, their work did not focus on efficiency in sample complexity and also did not study certificate complexity which is a focus of our paper. We do not study the stability parameter.

The study of notions of reproducibility/replicability in various computational fields is an emerging topic. The article (PVLS + 21) discusses the differences between replicability and reproducibility. In (EKK + 23), the authors consider replicability in the context of stochastic bandits. Their notion is similar to the notion studied in (ILPS22). In (AJJ + 22), the authors investigate reproducibility in the context of optimization with inexact oracles (initialization/gradient oracles). The setup and focus of these works are different from ours.

Finally, we would like to point out that there notions of list PAC learning and list-decodable learning in the learning theory literature (see (CP23) and (RY20) for recent progress on these notions). However, these notions are different from the list replicable learning that we consider in this paper. List PAC learning and list-decodable learning are generalized models of PAC learning. For example, any learning task that is PAC learnable is trivially list PAC learnable with a list size of 1. However, list replicable learning is an additional requirement that needs to be satisfied by a learner. Thus the notion of list and list-decodable PAC learning are different from the notions of list/certificate replicability.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work the authors study the problem of replicability in machine learning from a mathematical point of view. They first show that even in the simple problem of estimating the bias of a coin, a strong notion of replicability which asks for an algorithm to output the same answer on independent i.i.d. samples from the true coin as input (and independent internal randomness) is not possible, unless the accuracy guarantees are trivial. Motivated by that, they propose two definitions that aim to capture the degree of non-replicability of the underlying statistical task.\n * The first one is called $L$ list-replicability and, roughly speaking, asks that the algorithm outputs at most $L$ different answers under independent executions, with high probability. Moreover, the answers that belong to this list need to be $\\epsilon$-accurate for the specific application.\n  * The second definition they propose is called $\\ell$ certificate replicability. Under this definition, the algorithm has access to internal random strings of length $\\ell$, and it is required that, for most of the random strings there is a ``canonical'' output $\\mathcal{o}_r$ that is $\\varepsilon$-accurate, i.e., with high probability over the draw of the dataset, when the random string of the algorithm is $r$ its output will be $\\mathcal{o}_r$.\n\nThe authors illustrate the usefulness of these definitions by designing algorithms that satisfy them for various statistical tasks. The bulk of their technical work is to prove Lemma 3.1, Lemma 3.2, and Lemma 3.3, which are later used to establish various results. In particular, Lemma 3.1 is a more fine-grained version of a result from [VWDP+22], Lemma 3.2 is a randomized rounding scheme, that unlike the one proposed in [ILPS22], using a (relatively) small number of random bits, and Lemma 3.3 is a straightforward modification of Sperner's/KKM that has appeared in the literature. \n\nFirst, they consider the problem of estimating the bias of $d$ independent coins from samples. They design a deterministic and efficiently computable algorithm that is $d+1$-list replicable and requires $O\\left(\\frac{d^3}{\\epsilon^2}\\cdot \\log(d/\\delta)\\right)$ samples in total, where $\\epsilon$ is the accuracy parameter and $\\delta$ is the confidence parameter. At this point it is worth mentioning (even though not explicitly stated in the paper) that there exists a (trivial) $2^d$-list replicable algorithm whose sample complexity is $O\\left(\\frac{d}{\\epsilon^2}\\cdot \\log(d/\\delta) \\right)$ -- the authors use this algorithm for a different task (more details on this later). For the same task, they design a $\\log(d/\\delta)$-certificate replicable algorithm with sample complexity $O(\\frac{d^3}{\\varepsilon^2\\delta^2}$ in total. Notice that the sample complexity w.r.t. $\\delta$ is $poly(1/\\delta)$, whereas for the list replicable algorithm it is $poly(\\log(1/\\delta)$. They complement their upper bounds by showing that there is no $k$-list replicable algorithm for this task, for any $k < d+1$, and no $k$-certificate replicable algorithm for any $k < \\log(d)$. This implies that the list complexity parameter is optimal and the certificate complexity parameter is optimal up to a factor of $\\log(1/\\delta)$, but it does not imply that the sample complexity of the algorithms is optimal.\n\nThen, the authors consider the canonical learning task of binary classification in the PAC setting. They first show that any class $H$ that is learnable in the well-known SQ model by non-adaptive queries admits a $d+1$-list replicable algorithm whose sample complexity is $O\\left(\\frac{d^2}{\\varepsilon^2} \\log(d/\\delta)\\right)$. They further show that there is a $\\log(d/\\delta)$-certificate replicable algorithm for this task with sample complexity $O\\left(\\frac{d^2}{\\delta^2\\varepsilon^2} \\log(d/\\delta)\\right)$. In the case of adaptive queries, the only list-replicable algorithm they present is the trivial one with $L=2^d$, but they design a $d\\log(d/\\delta)$-certificate replicable algorithm with sample complexity $O\\left(\\frac{d^3}{\\delta^2\\varepsilon^2} \\log(d/\\delta)\\right).$\n\nFinally, they consider the task of learning $d$-dimensional thresholds on $[0,1]^d$. They observe that if the marginal distribution on $X$ is unknown, there is no list-replicable algorithm for this task. Thus, they restrict the problem by fixing this distribution be uniform. They prove that this restriction helps circumvent the lower bound by showing that there exists a $d+1$-list replicable algorithm for this problem. Moreover, the list replicability parameter cannot be decreased. \n\n\n[VWDP+22]: Jason Vander Woude, Peter Dixon, Aduri Pavan, Jamie Radcliffe, and N. V. Vinodchandran. Geometry of rounding.\n\n[ILPS22]: Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell. Reproducibility in learning.\n\nSTRENGTHS:\nThis work is trying to address a very important problem in ML from a theoretical point of view. It is important to have algorithms with provable replicability guarantees and not just rely on empirical replicability of experimental findings. \nRecently, there has been an interest from the TCS/theoretical ML community to design algorithms with mathematical replicability guarantees. The authors do a good job of placing their work in the literature. \n\nI think the most exciting contributions of the paper are the replicability definitions they propose, which are novel and I find them quite natural and interesting. Moreover, even though the main technical challenges to establish Lemma 3.1, and Lemma 3.3 were already solved in prior work, the use of the rounding scheme of Lemma 3.1 based on secluded partitions in the replicability literature is novel and I believe that it can have further applications in this line of work. Similarly, using Sperner's/KKM lemma to establish lower bounds for list-replicable algorithms could be used to derive lower bounds for other replicability definitions that researchers might come up with in future work. The randomized rounding scheme that is proposed in Lemma 3.2 is a clever, though not very hard, tweak of the one in [ILPS22] that reduces the (potentially infinitely many) random bits that the naive application of [ILPS22] requires.\n\nThe paper is in general well-written and the proof sketches in the main body are easy to follow. The same thing is true for the appendix as well. That being said, I think there are some inaccuracies and some nuances that are not addressed appropriately, which I elaborate on in the Weaknesses/Questions section. \n\n[ILPS22]: Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell. Reproducibility in learning.\n\nWEAKNESSES:\nThe biggest weakness of the paper is that the results are somewhat limited to simple problems. In particular, the results in Section 4 and Section 5.1 can be unified and stated as list/certificate-replicability results for SQ learning (see also my first question to the authors). The next result that the authors prove in the binary classification setting is restricted since it holds for a particular hypothesis class under a particular distribution on the feature space. One immediate extension the authors could consider would be to prove this result for any fixed distribution on $X$ and not just the uniform distribution. Please see below why I believe that this is true. Another obvious direction is to establish sample complexity lower bounds for the various algorithms you have developed.\n\n\nSome inaccuracies/issues in the exposition:\n\n-Line 140: As far as I can tell the notion of list-global stability [GKM21] does not coincide with the notion of list replicability. In particular, a list-globally stable learner outputs a list of hypotheses (not just a single one) and it is required that at least one of the elements of the list will be the canonical output, i.e., it will appear with high probability after every execution. On the other hand, a list replicable algorithm outputs a single element, but whp, this single elements belongs to a canonical list. \n\n-Line 155-156: \"Establishing the optimality of these measures\". As far as I can tell, you didn't establish the optimality of the sample complexity.\n\n-Observation 5.8: Not sure what it means for $k$ to be \"constant\". Even if we allow it to depend on parameters of the class or on $d$ the result is true. \n\n-Proof of Observation 5.8: First of all, I think it is important to state the results of [ALMM19], [BLM20] that prove the claim (at least in the appendix) since many readers might not be familiar with these works. Moreover, to the best of my knowledge, the statement \"any class $H$ has finite Littlestone dimension if and only if for some ``constant'' $k$ there is a $k$-list replicable algorithm for $H$\" does not follow immediately from these two works. The direction that I can see as a direct result is that if $H$ has infinite Littlestone dimension then there does not exist a $k$-list replicable algorithm for $H$. \nIn particular [ALMM19] showed that every class that is approx-DP learnable has finite Littlestone dimension. Subsequently, [BLM20] showed that every Littlestone class has a globally stable learner with appropriate global stability parameters. Assume that H has infinite Littlestone dimension but admits a $k$-list replicable learner. Then, this gives rise to a globally stable learner with stability parameter at most something like $1/k$, which means that the class is approx-DP learnable, hence a contradiction. The converse direction can be obtained by Theorem 2 of [CMY23], however I do not see an immediate way to obtain it from [ALMM19],[BLM20]. Notice that a globally stable learner is not 1-list replicable.\nPlease let me know if I'm missing something.\n\n-Line 368-369. This is not accurate. Be careful with the order of the quantifiers. [ALMM19] did not show that there exists a single \"hard'' distribution $\\mathcal{D}^\\star$ for the class of one-dimensional thresholds. They showed that for any approx-DP learning algorithm $A$, there exists a hard distribution $\\mathcal{D}_A$, i.e., the distribution depends on the algorithm. In fact, it is known that for any fixed $\\mathcal{D}$, any VC class is approx-DP learnable.\n\n-It is a bit confusing to refer to the distribution-dependent learning setting as the PAC learning setting, because most readers would assume that the PAC setting is distribution-free.\n\nTypos and other comments that didn't affect the score/evaluation.\n\n-Line 72 \"multi-pseuodeterminism\" -> multi-pseudodeterminism.\n\n-Line 333, 342, 349, 547, 567, 589 \"reproducibly\" -> replicably.\n\n-Line 100: \"with at least\" -> \"with probability at least\".\n\n-Line 201: \"randomize\" -> randomized.\n\n-Line 201: maybe you can restrict $\\varepsilon$ to be at most one, outside this range it is trivial to satisfy the definition.\n\n-Line 187-188: use singular form for the rounding scheme.\n\n-It might be useful to state the formal definitions of replicability in section 1.1 (see also the next comment).\n\n-Instead of having two separate definitions of an \"accurate\" list replicable and certificate replicable learner based on the statistical task (coins/PAC learning), it might be better to have one definition that captures just what list replicability/certificate replicability means and then specify what \"correctness\" means based on the underlying task, separately -- i.e., have definitions that say what it means for the learner to be list replicable/certificate replicable, have an (application specific) definition on what it means to be accurate and then the accurate list replicable/certificate replicable replicable algorithm is simply the composition of these definitions. \n\n-It might be good to point out the difference between global stability [BLM20] and 1-list replicability. Global stability asks for one canonical output that appears with non-zero probability, whereas 1-list replicability asks for the canonical output to appear whp.\n\n\n-Line 256-263: Maybe you can have it as a Remark/Observation.\n\n-Line 272: You could have an one-line comment that this follows from the data-processing inequality. Also, it looks like the two subscripts of $\\mathcal{D}$ are not aligned. \n\n-Line 406: did you mean if $\\vec{p} \\in X$?\n\n-Line 475: \"are round\" -> are rounded .\n\n-Line 480: \"almost\" -> at most.\n\n-Line 494: you mean $(d-1)$-dimensional face?\n\n-Line 499-500: maybe it's good to define the closure formally (even though the reader can understand what it means).\n\n-Line 698: set notation for $W$ missing.\n\n-Line 704: \"us\" -> is.\n\n-Line 705: \"the\" -> that.\n\n[GKM21]: Badih Ghazi, Ravi Kumar, and Pasin Manurangsi. User-level differentially private learning via correlated sampling.\n\n[ALMM19]: Noga Alon, Roi Livni, Maryanthe Malliaris, and Shay Moran. Private PAC learning implies finite littlestone dimension.\n\n[BLM20]: Mark Bun, Roi Livni, and Shay Moran. An equivalence between private classification and online prediction.\n\n[CMY23} Zachary Chase, Shay Moran, and Amir Yehudayoff. Replicability and stability in learning.\n\nQUESTIONS:\n1) I don't really understand what is the benefit of studying the coin estimation problem in Section 4 separately from the SQ learning problem in Section 5. In particular, as far as I can tell, all the proofs in Section 4 related to estimating the bias of coins hold verbatim for SQ estimation tasks. In fact, they can be phrased as general reduction: given a potentially non-replicable algorithm A that gives answers that are $\\epsilon$-close to the true one in the $\\ell_\\infty$ norm, using the rounding schemes your propose one can get list\\certificate replicable algorithms with the stated sample complexities/number of oracle calls to A. For instance, this can be used to replicably estimate potentially unbounded quantities that concentrate, such as averages of subgaussian random variables etc. Am I missing something?\n\n2) As you have also pointed out in the manuscript, there is a trivial 2^d-list replicable algorithm for the coin estimation problem that needs O(d) samples (assume that $\\epsilon,\\delta$ is a constant) and your algorithm is $d$-list replicable but requires $O(d^3)$ samples. Are you aware of any $poly(d)$-list replicable algorithm that uses $o(d^3)$ samples?\n\n3) Relatedly, do you believe that the $O(d^3)$ sample complexity is necessary to get the $d+1$-list replicability?\n\n4) The dependence of the sample complexity of certificate replicability on the confidence $\\delta$ is a bit unsatisfactory since it is $poly(1/\\delta)$ and not $poly(log(1/\\delta))$. I believe this is, to some extent, an artifact of the way you have defined certificate replicability. You can consider a two-parameter version of the definition that gives rise to more fine-grained sample complexity bounds. To be more precise, you can define an algorithm to be $(\\ell, \\rho)$-certificate replicable if $(1-\\rho)$-fraction of the $\\ell$-bit random strings $r$ have a canonical output $\\mathcal{o}_r$. As in the non-replicable case, an algorithm is $(\\epsilon,\\delta)$-accurate if with probability $1-\\delta$ it outputs an answer whose \"error\" is at most $\\epsilon$. To define what it means to be $(\\ell, \\rho)$-certificate replicable and $(\\epsilon, \\delta)$-accurate you simply require that the algorithm satisfies both definitions. In your definition, you essentially treat the parameters $\\delta, \\rho$ in the same way. The advantage of the definition I suggest is that it will lead to sample complexities of the certificate replicable algorithms you propose that are of the form $poly(1/\\epsilon, 1/\\rho, \\log(1/\\delta)$. These are also the \"correct\" sample complexities that appear in this line of work (see, e.g., [ILPS22], [KKMV23]). Am I missing something?\n\n5) In Line 211, shouldn't the randomness $r$ in the inner probability be fixed? In other words, shouldn't the inner probability be only wrt the random samples of the coins? (as you do in Def. 5.3)\n\n6) In Theorem 5.9, why do you only consider the uniform distribution? I believe that the result should hold for any fixed distribution, but the way to construct the rounding scheme would depend on the distribution (i.e., it will not in general be the one you are proposing).\n\n7) In the proof of Lemma 3.1, line 446, second equality, did you mean to write $\\bar{B}_{\\epsilon_0/\\epsilon}^\\infty(x/\\epsilon)$? If I am not missing something, you can see that by picking $x=1$ for example, what you have currently written is not true.\n\n\n\n[ILPS22]: Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell. Reproducibility in learning.\n\n[KKMV23]: Alkis Kalavasis, Amin Karbasi, Shay Moran, and Grigoris Velegkas. Statistical indistinguishability of learning algorithms.\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers two notions of complexity for stable algorithms: list complexity for list replicable algorithms and certificate complexity for pseudo-globally stable/replicable algorithms. They give matching upper and lower bounds for the list and certificate complexities of the d-coin bias estimation problem. They extend these results to obtain upper bounds on the complexities of adaptive and non-adaptive SQ algorithms and upper and lower bounds on the list complexity of learning d-dimensional thresholds under the uniform distribution.\n\nSTRENGTHS:\nThis paper has a novel perspective on the study of replicable algorithms, by considering how much randomness is required to achieve replicability, and it continues the study of list bounds for list replicability. The rounding techniques and application of Sperner’s lemma used to prove their results are interesting, and I think the fact that they are able to obtain several results for fundamental learning problems from variations of these techniques speaks to their usefulness. I enjoyed reading the paper.\n\nWEAKNESSES:\nThough I enjoyed the paper overall, it was structured in a way that frequently made it difficult to understand or contextualize their results. \n- Formal definitions of the complexity measures studied appear in the context of a particular problem, rather than being properties of general algorithms, which was a bit confusing.\n-  The motivation for these complexity measures is not very clear to me.\n- The explanation of how to simulate a statistical query algorithm making d adaptive queries is not specific enough to be enlightening. Either expanding it or omitting it would make the section more readable.\n- As a reader, I didn’t gain much from the exposition on page 2 showing that perfectly replicability isn’t achievable, and I think it took space away from expanding on why list and certificate complexity are useful notions of non-replicability to minimize. \n\nThe techniques used seem interesting, but seem mostly to have been developed in VWDPRV22. The novel contribution of this paper appears to me to be the adaptation of these results to the problem of certificate and list complexity, and so I think the usefulness of these complexity measures needs much more attention in the paper.\n\nQUESTIONS:\nAs already mentioned, I think these complexity definitions need more motivation. They seem interesting, but how does minimizing list and certificate complexity facilitate replicability? How does list replicability compare to pseudo-global stability?\n\nLIMITATIONS:\nYes, the authors address broader societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper investigates replicable learning algorithms by proposing two notions of replicability, i.e. the list replicability and the certificate replicability. For the problem of estimating the biases of $d$ coins, it designs a $(d+1)$-list replicable algorithm and complements the result by showing that the list size is optimal. For PAC learning model, it shows that any hypothesis class that is learnable with $d$ non-adaptive SQs can be learned by a $(d+1)$-list replicable algorithm or a $\tilde{O}(\log d)$-certificate replicable algorithm.

STRENGTHS:
Designing replicable algorithms is of interest to both the theoretical and applicational community. Empirically, it is important for the models to be reproducible under slight changes in the environment. From the theoretical side, the problem itself is interesting whether provable guarantees can be achieved for algorithmic replicability. The paper generalized the idea of replicability by proposing two notions, allowing the algorithms to be list-replicable or certificate replicable, which characterize the replicability of a broader class of problems from bias estimation of multiple coins to PAC learning thresholds.

WEAKNESSES:
Some parts of the presentation can be improved. For example, as major notions, the definition of list-replicability and certificate-replicability can be placed in Section 1. To now, the ideas seem to be implied from the definition of specific problems instead of standalone definitions, which makes the paper less clear.

QUESTIONS:
To what extent is the notion of list-replicability related to the notion of list PAC learning or list-decodable learning? Since the PAC learning model is considered in this paper, it would be interesting if these notions could be compared.

LIMITATIONS:
No concerns.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the theoretical property of replicability of machine learning algorithms. The paper's authors introduce two notions of list and certificate complexity used to quantify the replicability of such algorithms. They also establish replicable algorithms for the coin bias estimation problem and in the general setting of PAC learning, proving theoretical guarantees on the properties of these algorithms.\n\n---\nPost-rebuttal update: after reading the author response, I am increasing by confidence in the score of 6 by 1 point. I have not worked in the field of working theory, but the results are sensible to me and the rebuttal has addressed some of my questions.\n\nSTRENGTHS:\n* The work studies an important problem of having not only sample-efficient, but also replicable algorithms that would arrive to a narrow set of hypotheses after training on different samples from the data distribution.\n* Although I am not an expert in learning theory, the theoretical analysis appears to be sound and the reasoning steps are explained in sufficient detail\n* The paper covers the related work well, connecting the contributions of the authors to the existing results in a clear way\n\nWEAKNESSES:\n* I feel the work would strongly benefit from an improvement of its presentation. For example, the abstract is several paragraphs long. Furthermore, the introduction contains an example illustratory statement along with its full proof, which makes this section of the paper a bit dense.\n* Some parts of the paper could also be improved by providing more insight into the motivation behind the introduced notions of replicability. It is not fully clear if the certificate complexity has any specific advantages over previous measures of replicability discussed in the CMY23 reference that would make it of interest to the community.\n\nQUESTIONS:\n* Could you provide some intuition into the certificate complexity, for example, into the meaning of the certificate $r$? The proof of Lemma 3.2 in the appendix contains a constructive description of what $r$ does and how it is used, but the work might be easier to understand if we had an example for any standard machine learning problem in a more practical setting (e.g., binary classification of real-valued vectors) that would use $r$ as an input.\n* The introductory section provides useful insight into the motivation behind studing reproducibility, but (unless I am mistaken) it does not address two important real-world considerations. First, while machine learning algorithms observe samples from a distribution, in a supervised offline setting (which is the most popular one) they are trained on a fixed finite-sized sample from that distribution (usually called the training dataset). If the reproducing party trains on the same open dataset, that factor of randomness does not influence the resulting model. Second, all other hardware-based factors of randomness are usually also fixed by setting the random seed at the beginning of training. Would the analysis of replicability be still useful in such a case?\n\nLIMITATIONS:\nThere is a discussion of limitations in the conclusion section, which was helpful in understanding the scope of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-31"}