SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathscr {F} \left(x _ {1} ^ {n}\right) := \left\{f \left(x _ {1} ^ {n}\right): f \in \mathscr {F} \right\},
$$
i.e, the restriction of  $\mathcal{F}$  onto the sample. Let  $s\in \{-1,1\} ^n$  be a Rademacher random vector. That is,  $s_i$  takes on values  $-1,1$  with equal probability independently of other components. Recall that the Rademacher complexity of  $\mathcal{F}$  is given by
$$
\mathcal {R} _ {n} (\mathcal {F}) := \mathbb {E} _ {X, s} \left[ \sup  _ {f \in \mathcal {F}} \frac {1}{n} \langle s, f (X _ {1} ^ {n}) \rangle \right].
$$
Theorem C.1 ([Wainwright, 2019]). For any  $b$ -uniformly bounded class of functions  $\mathcal{F}$ , integer  $n \geq 1$ , and error  $\varepsilon \geq 0$ ,
$$
\sup  _ {f \in \mathcal {F}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} f (X _ {i}) - \mathbb {E} [ f (X) ] \right| \leq 2 \mathcal {R} _ {n} (\mathcal {F}) + \varepsilon
$$
with probability at least  $1 - \exp \left(-\frac{n\varepsilon^2}{2b^2}\right)$ .
In particular, as long as  $\mathcal{R}_n(\mathcal{F}) = o(1)$ , we have uniform convergence of the sample mean.
C.3 Metric Entropy
We write  $B(x, r; \mu)$  to denote the ball of radius  $r$  about a point  $x$  with respect to a metric  $\mu$ . Recall that an  $\varepsilon$ -cover of a set  $T$  with respect to a metric  $\mu$  is a subset  $\theta_1, \ldots, \theta_N \subseteq T$  such that
$$
T \subseteq \bigcup_ {i \in [ N ]} B (\theta_ {i}, \varepsilon ; \mu).
$$
The covering number  $N(\varepsilon; T, \mu)$  is the cardinality of the smallest  $\varepsilon$ -cover.
Proposition C.2 ([Wainwright, 2019]). Fix  $\varepsilon \in (0,1)$ . Let  $B(R; \| \cdot \|)$  denote the  $d$ -dimensional ball of radius  $R$  with respect to  $\| \cdot \|$ . Then
$$
N (\varepsilon , B (R; \| \cdot \|), \| \cdot \|) \leq \left(1 + \frac {2 R}{\varepsilon}\right) ^ {d} \leq \left(\frac {3 R}{\varepsilon}\right) ^ {d}.
$$
We say a collection of zero-mean random variables  $\{Y_{\theta}:\theta \in T\}$  is a sub-Gaussian process with respect to a metric  $\mu$  on  $T$  if
$$
\mathbb {E} \exp \left[ \lambda \left(Y _ {\theta} - Y _ {\theta^ {\prime}}\right) \right] \leq \exp \left[ \frac {\lambda^ {2} \mu^ {2} \left(\theta , \theta^ {\prime}\right)}{2} \right]
$$
for all  $\theta, \theta' \in T$  and  $\lambda \in \mathbb{R}$ .
Proposition C.3. The canonical Rademacher process
$$
Y _ {\theta} := \langle s, \theta \rangle
$$
is a zero-mean sub-Gaussian process with respect to the Euclidean norm on  $T \subseteq \mathbb{R}^n$ .
Proof. Recall that a Rademacher variable is sub-Gaussian with parameter 1. Moreover, the sum of sub-Gaussian variables is sub-Gaussian with parameter equal to the Euclidean norm of the parameters. It follows that  $Y_{\theta} - Y_{\theta'} = \langle s, \theta - \theta' \rangle$  is sub-Gaussian with parameter  $\| \theta - \theta' \|_2$ . The result follows.
Theorem C.4 (One-Step Discretization; [Wainwright, 2019]). Let  $Y_{\theta}, \theta \in T$  be a zero-mean sub-Gaussian process with respect to the metric  $\mu$ . Fix any  $\varepsilon \in [0, D]$  where  $D = \mathrm{diam}(T; \mu)$  such that  $N(\varepsilon; T, \mu) \geq 10$ . Then
$$
\begin{array}{l} \mathbb {E} \left[ \sup  _ {\theta \in T} Y _ {\theta} \right] \leq \mathbb {E} \left[ \sup  _ {\theta , \theta^ {\prime} \in T} Y _ {\theta} - Y _ {\theta^ {\prime}} \right] \\ \leq 2 \mathbb {E} \left[ \sup  _ {\theta , \theta^ {\prime} \in T: \mu (\theta , \theta^ {\prime}) \leq \varepsilon} Y _ {\theta} - Y _ {\theta^ {\prime}} \right] + 4 D \sqrt {\log N (\varepsilon ; T , \mu)}. \\ \end{array}
$$
In general, it may be possible to improve the bound in Theorem C.4 using more sophisticated techniques such as Dudley's entropy integral bound [Wainwright, 2019]. However, the simple inequality from Theorem C.4 suffices for our setting.
Corollary C.5. Fix any  $\varepsilon \in [0, D]$  where  $D = \mathrm{diam}(T; \mu)$  such that  $N(\varepsilon; T, \mu) \geq 10$ . The canonical Rademacher process  $Y_{\theta} = \langle s, \theta \rangle$  for  $\theta \in T \subseteq \mathbb{R}^n$  satisfies
$$
\mathbb {E} \left[ \sup  _ {\theta} Y _ {\theta} \right] \leq 2 \varepsilon \sqrt {n} + 4 D \sqrt {\log N (\varepsilon ; T , \| \cdot \| _ {2})}.
$$
Proof. By Proposition C.3,  $Y_{\theta}$  is a zero-mean sub-Gaussian process and we can apply Theorem C.4 to conclude that
$$
\begin{array}{l} \mathbb {E} \left[ \sup  _ {\| \theta - \theta^ {\prime} \| _ {2} \leq \varepsilon} Y _ {\theta} - Y _ {\theta} ^ {\prime} \right] = 2 \mathbb {E} \left[ \sup  _ {\| v \| _ {2} \leq \varepsilon} \langle v, s \rangle \right] \\ \leq 2 \mathbb {E} \left[ \| s \| _ {2} \cdot \| v \| _ {2} \right] \\ \leq 2 \varepsilon \sqrt {n}. \\ \end{array}
$$
Corollary C.5 gives us a way to control the Rademacher complexity of a function class whose co-domain is well-behaved. We make this notion rigorous in the next section.
C.4 Uniform Convergence of  $(k,p)$ -Clustering Cost
For a fixed set of centers  $F$ , let  $\kappa_F^p: \mathbb{R}^d \to \mathbb{R}$  be given by
$$
\kappa_ {F} ^ {p} (x) := \kappa (x, F) ^ {p}.
$$
Define  $\mathcal{F} \coloneqq \{F \subseteq \mathcal{B}_d : |F| = k\}$ . We take our function class to be
$$
\mathcal {F} := \left\{\kappa_ {F} ^ {p}: F \in \mathcal {F} \right\}.
$$
Let  $\mu : \mathcal{F} \times \mathcal{F} \to \mathbb{R}$  be given by
$$
\mu (A, B) := \max  _ {a \in A, b \in B} \kappa (a, b).
$$
Proposition C.6.  $\mu$  is a metric on  $\mathcal{F}$
Proof. It is clear that  $\mu$  is symmetric and positive definite. We need only show that the triangle inequality holds. Fix  $A, B, C \in \mathcal{F}$  and suppose  $a \in A, c \in C$  are such that  $\kappa(a, c) = \mu(A, C)$ . Then
$$
\begin{array}{l} \mu (A, C) = \kappa (a, c) \\ \leq \kappa (a, b) + \kappa (b, c) \\ \leq \mu (A, B) + \mu (B, C) \\ \end{array}
$$
$\forall b \in B$
Proposition C.7.  $\mathcal{F}$  is  $p$ -Lipschitz parameterized with respect to the metric  $\mu$ .
$$
\begin{array}{l} \left| \kappa_ {F} ^ {1} (x) - \kappa_ {F ^ {\prime}} ^ {1} (x) \right| = \left| \kappa (x, F (x)) - \kappa (x, F ^ {\prime} (x)) \right| \\ \leq \left| \kappa \left(x, F ^ {\prime} (x)\right) + \kappa \left(F ^ {\prime} (x), F (x)\right) - \kappa \left(x, F ^ {\prime} (x)\right) \right| \\ = \kappa \left(F ^ {\prime} (x), F (x)\right) \\ \leq \mu (F, F ^ {\prime}). \\ \end{array}
$$
Now, the function  $g(x):[0,1]\to \mathbb{R}$  given by  $x\mapsto x^p$  is  $p$ -Lipschitz by the mean value theorem:
$$
| g (x) - g (y) | \leq \sup  _ {\xi \in [ 0, 1 ]} g ^ {\prime} (\xi) | x - y | \leq p | x - y |.
$$
The result follows by the fact that the composition of Lipschitz functions is Lipschitz with a constant equal to the product of constants from the composed functions.
Theorem C.8. For any  $\varepsilon \in [0, D = \sqrt{n}]$  such that  $N(\varepsilon; \mathcal{F}(x_1^n), \| \cdot \|_2) \geq 10$ ,
$$
\mathcal {R} _ {n} (\mathcal {F}) \leq \frac {2 \varepsilon}{\sqrt {n}} + 4 \sqrt {\frac {k d \log \frac {3 \sqrt {n} p}{\varepsilon}}{n}} = o (1).
$$
Proof. Remark that  $\kappa_F^p (x_1^n)$  is  $\sqrt{n} p$ -Lipschitz with respect to the metric  $\mu$ . Indeed,
$$
\sum_ {i = 1} ^ {n} \left| \kappa_ {F} ^ {p} (x _ {i}) - \kappa_ {F ^ {\prime}} ^ {p} (x _ {i}) \right| ^ {2} \leq n p ^ {2} \mu (F, F ^ {\prime}) ^ {2}.
$$
Now,  $D\coloneqq \mathrm{diam}(\mathcal{F}(x_1^n);\| \cdot \| _2) = \sqrt{n}$ . We can apply Corollary C.5 with  $T = \mathcal{F}(x_1^n)$  to see that
$$
\begin{array}{l} \mathcal {R} _ {n} (\mathcal {F}) = \frac {1}{n} \mathbb {E} \left[ \sup _ {\theta} Y _ {\theta} \right] \\ \leq \frac {2 \varepsilon}{\sqrt {n}} + 4 \sqrt {\frac {\log N (\varepsilon ; \mathcal {F} (x _ {1} ^ {n}) , \| \cdot \| _ {2})}{n}}. \\ \end{array}
$$
Now, since  $\kappa_F^p (x_1^n)$  is  $\sqrt{n} p$ -Lipschitz, a  $\frac{\varepsilon}{\sqrt{np}}$ -cover for  $\mathcal{F}$  yields an  $\varepsilon$ -cover of  $\mathcal{F}(x_1^n)$ . Hence
$$
N (\varepsilon ; \mathcal {F} (x _ {1} ^ {n}), \| \cdot \| _ {2}) \leq N \left(\frac {\varepsilon}{\sqrt {n} p}; \mathcal {F}, \mu\right).
$$
Note that the cross product of  $k$ $\varepsilon$ -covers of  $\mathcal{B}$  is an  $\varepsilon$ -cover of  $\mathcal{F}$ . Hence
$$
\begin{array}{l} N \left(\frac {\varepsilon}{\sqrt {n} p}; \mathcal {F}, \mu\right) \leq N \left(\frac {\varepsilon}{\sqrt {n} p}; \mathcal {B} _ {d}, \kappa\right) ^ {k} \\ \leq \left(\frac {3 \sqrt {n} p}{\varepsilon}\right) ^ {k d}. \\ \end{array}
$$
Proposition C.2
Substituting this bound on the covering number of  $\mathcal{F}(x_1^n)$  concludes the proof.
Note that the Lipschitz property was crucial to ensure that the exponent in the covering number does not contain  $n$ .
Theorem C.9. Fix  $\varepsilon, \delta \in (0,1)$ . Then with
$$
O \left(\frac {k ^ {2} d ^ {2}}{\varepsilon^ {4}} \log \frac {p}{\delta}\right)
$$
i.i.d. samples from  $\mathbb{P}$
$$
\left| \widehat {\operatorname {c o s t}} (F) - \operatorname {c o s t} (F) \right| \leq \varepsilon
$$
with probability at least  $1 - \delta$  for any set of centers  $F\subseteq \mathcal{B}_d$
Proof. Choose  $\varepsilon \coloneqq 3\leq D = \sqrt{n}$  . By Theorem C.8,
$$
\begin{array}{l} \mathcal {R} _ {n} (\mathcal {F}) \\ \leq \frac {2 \cdot 3}{\sqrt {n}} + 4 \sqrt {\frac {k d \log \frac {3 \sqrt {n} p}{3}}{n}} \\ \leq \frac {6}{\sqrt {n}} + 4 \sqrt {\frac {k d \log \sqrt {n} p}{n}} \\ \leq \frac {6}{\sqrt {n}} + 4 \sqrt {\frac {k d \log p}{n}} + 4 \sqrt {\frac {k d \log \sqrt {n}}{n}} \quad \sqrt {a + b} \leq \sqrt {a} + \sqrt {b} \\ \leq \frac {6}{\sqrt {n}} + 4 \sqrt {\frac {k d \log p}{n}} + 4 \sqrt {\frac {k d}{\sqrt {n}}}. \quad \log \sqrt {n} \leq \sqrt {n} \\ \end{array}
$$
Fix some  $\varepsilon_1\in (0,1)$  . We have
$$
\begin{array}{l} \frac {6}{\sqrt {n}} \leq \varepsilon_ {1} \\ \Longleftrightarrow n \geq \frac {3 6}{\varepsilon_ {1} ^ {2}}. \\ \end{array}
$$
$$
\begin{array}{l} 4 \sqrt {\frac {k d \log p}{n}} \leq \varepsilon_ {1} \\ \Longleftrightarrow n \geq \frac {1 6 k d \log p}{\varepsilon_ {1} ^ {2}}. \\ \end{array}
$$
$$
\begin{array}{l} 4 \sqrt {\frac {k d}{\sqrt {n}}} \leq \varepsilon_ {1} \\ \Longleftrightarrow n \geq \frac {2 5 6 k ^ {2} d ^ {2}}{\varepsilon_ {1} ^ {4}}. \\ \end{array}
$$
By taking the maximum of the three lower bounds, we conclude that
$$
n \geq \frac {2 5 6 k ^ {2} d ^ {2}}{\varepsilon_ {1} ^ {4}} \log p \Rightarrow \mathcal {R} _ {n} (\mathcal {F}) \leq 3 \varepsilon_ {1}.
$$
Fix  $\delta \in (0,1)$ . Observe that  $\mathcal{F}$  is 1-uniformly bounded. Thus by Theorem C.1, we require
$$
\max \left(\frac {2}{\varepsilon_ {1} ^ {2}} \log \frac {1}{\delta}, \frac {2 5 6 k ^ {2} d ^ {2}}{\varepsilon_ {1} ^ {4}} \log p\right) \leq \frac {2 5 6 k ^ {2} d ^ {2}}{\varepsilon_ {1} ^ {4}} \log \frac {p}{\delta}
$$
samples in order to guarantee that
$$
\begin{array}{l} \sup  _ {\kappa_ {F} ^ {p} \in \mathcal {F}} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \kappa_ {F} ^ {p} (X _ {i}) - \mathbb {E} \kappa_ {F} ^ {p} (X) \right| \leq 2 \mathcal {R} _ {n} (\mathcal {F}) + \varepsilon_ {1} \\ \leq 7 \varepsilon_ {1} \\ \end{array}
$$
Choosing  $\varepsilon_{1} = \frac{\varepsilon}{7}$  concludes the proof.
D  $(k,p)$ -Clustering
In this section, we provide the full proofs for Theorem 4.2 and Theorem 4.3, which state the guarantees for the replicable  $k$ -medians and  $k$ -means algorithms, respectively. First, we describe a
useful subroutine for replicable heavy hitters estimation in Appendix D.2. This subroutine is crucial to the replicable coreset algorithm (cf. Algorithm D.3), which we analyze in Appendix D.3. Once we have a coreset, it remains to solve the statistical  $(k,p)$ -clustering problem on a finite distribution. We describe a replicable algorithm for this in Appendix D.4. Our coreset algorithm assumes the knowledge of some constant ratio estimate of OPT. In Appendix D.5, we show how to output such an estimate replicably. Finally, we summarize our findings in Appendix D.6.
D.1 Warm-Up: Replicable SQ Oracle and  $\varepsilon$ -Covers
In order to give some intuition to the reader, we first show how we can use a subroutine that was developed in Impagliazzo et al. [2022] in order to derive some results in the setting we are studying. We first need to define the statistical query model that was introduced in Kearns [1998]
Definition D.1 (Statistical Query Oracle; [Kearns, 1998]). Let  $\mathcal{D}$  be a distribution over the domain  $\mathcal{X}$  and  $\phi : \mathcal{X}^n \to \mathbb{R}$  be a statistical query with true value
$$
v ^ {\star} := \lim  _ {n \rightarrow \infty} \phi (X _ {1}, \dots , X _ {n}) \in \mathbb {R}.
$$
Here  $X_{i}\sim_{i.i.d.}\mathcal{D}$  and the convergence is understood in probability or distribution. Let  $\varepsilon, \delta \in (0,1)^2$ . A statistical query  $(SQ)$  oracle outputs a value  $v$  such that  $|v - v^{\star}| \leq \varepsilon$  with probability at least  $1 - \delta$ .
The simplest example of a statistical query is the sample mean
$$
\phi (X _ {1}, \dots , X _ {n}) = \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {i}.
$$
The idea is that by using a sufficiently large number of samples, an SQ oracle returns an estimate of the expected value of a statistical query whose range is bounded. Impagliazzo et al. [2022] provide a replicable implementation of an SQ oracle with a mild blow-up in the sample complexity which we state below.
Theorem D.2 (Replicable SQ Oracle; [Impagliazzo et al., 2022]). Let  $\varepsilon, \rho \in (0,1)$  and  $\delta \in (0,\rho /3)$ . Suppose  $\phi$  is a statistical query with co-domain  $[0,1]$ . There is a  $\rho$ -replicable SQ oracle to estimate its true value with tolerance  $\varepsilon$  and failure rate  $\delta$ . Moreover, the oracle has sample complexity
The interpretation of the previous theorem is that we can replicably estimate statistical queries whose range is bounded.
We now explain how we can use this result for the statistical  $k$ -means problem under the Euclidean metric, but it is not hard to extend the approach to more general  $(k,p)$ -clustering problems. Consider a fixed set of centers  $F$ . Then, we can replicably estimate the cost of this solution using Theorem D.2 within an additive accuracy  $\varepsilon$  and confidence  $\delta$  using  $O\left(\varepsilon^{-2}\rho^{-2}\log (1 / \delta)\right)$  samples. Thus, a natural approach is to consider an  $\varepsilon$ -cover of the unit-diameter ball and then exhaustively search among solutions whose centers coincide with elements of the  $\varepsilon$ -cover. This is outlined in Algorithm D.1. We are now ready to state our results.
Algorithm D.1 Replicable  $k$ -Means with  $\varepsilon$ -Cover
rKMeansCover: distribution  $\mathbb{P}$ , error  $\varepsilon$ , replicability  $\rho$ , confidence  $\delta$
$G\gets \varepsilon /3$  -cover over the  $d$  -dimensional unit-diameter ball
$\mathcal{F}\gets \left\{F\subseteq G:|F| = k\right\}$  We restrict the solutions to the  $\varepsilon$  -cover.
Output  $\mathrm{argmin}_{F\in \mathcal{F}}$  rSQ-Oracle  $(\mathrm{cost}(F),\varepsilon /3,\rho /\lvert\mathcal{F}\rvert,\delta /\lvert\mathcal{F}\rvert)$
Lemma D.3. For any  $\varepsilon, \delta, \rho \in (0,1)$ ,  $\delta < \rho/3$ , Algorithm D.1 is  $\rho$ -replicable and outputs a solution  $F$  whose cost is at most  $\mathrm{OPT} + \varepsilon$  with probability  $1 - \delta$ . Moreover, it has sample complexity
$$
\tilde {O} \left(\frac {(9 / \varepsilon) ^ {2 k d}}{\rho^ {2} \varepsilon^ {2}} \log \frac {1}{\delta}\right).
$$
Proof (Lemma D.3). We first argue about the replicability of Algorithm D.1. Since we make  $|\mathcal{F}|$  calls to the replicable SQ subroutine with parameter  $\rho / |\mathcal{F}|$ , the overall replicability of the algorithm follows by taking a union bound.
Let us now focus on the correctness of the algorithm. Let  $F^{*}$  be the optimal solution. Consider the solution that we get when we move the centers of  $F^{*}$  to the closest point of  $G$  and let us denote it by  $\hat{F}^{*}$ . Notice that the cost of  $\hat{F}^{*}$  is at most  $\mathrm{OPT} + \varepsilon / 3$ . Furthermore, by a union bound, all the calls to the SQ oracle will return an estimate that is within an additive  $\varepsilon / 3$ -error of the true cost. This happens with probability at least  $1 - \delta$  and we condition on this event for the rest of the proof. Thus the estimated cost of the solution  $\hat{F}^{*}$  will be at most  $\mathrm{OPT} + 2\varepsilon / 3$ . Let  $\widetilde{F}$  be the solution that we output. Its estimated cost is at most that of  $\hat{F}^{*}$  and so its true cost will be at most  $\mathrm{OPT} + \varepsilon$ . This concludes the proof of correctness.
Lastly, we argue about the sample complexity of the algorithm. By Proposition C.2,
$$
\left| \mathcal {F} \right| \leq O \left(\left(9 / _ {\varepsilon}\right) ^ {k d}\right).
$$
By plugging this value into the sample complexity from Theorem D.2, we get
$$
O \left(\frac {(9 / \varepsilon) ^ {2 k d}}{\rho^ {2} \varepsilon^ {2}} \log \frac {(9 / \varepsilon) ^ {k d}}{\delta}\right).
$$
Although Algorithm D.1 provides some basic guarantees, there are several caveats with this approach. Namely, we can only get additive approximations and the dependence of the sample and time complexity on both  $k$ ,  $d$  is exponential.
In the following sections, we will explain how we can overcome these issues. As we alluded to before, our approach combines ideas from coresets estimation through hierarchical grids [Frahling and Sohler, 2005], uniform convergence through metric entropy [Wainwright, 2019], and dimensionality reduction techniques [Makarychev et al., 2019].
D.2 Replicable Heavy-Hitters
In this section, we present a replicable heavy hitters algorithm which is inspired by Impagliazzo et al. [2022] and has an improved sample complexity by a factor of  $O(1 / (v - \varepsilon))$ . We believe that this result could be of independent interest since the replicable heavy hitters algorithm has many applications as a subroutine in more complicated algorithms [Impagliazzo et al., 2022]. Intuitively, this algorithm consists of two phases. In the first phase, we estimate all the candidate heavy hitters and reduce the size of the domain. In the second phase, we estimate the mass of these candidates. We present a new analysis of the second phase using Proposition B.1.
Algorithm D.2 Replicable Heavy Hitters
rHeavyHitters:distribution  $\mathbb{P}$  ,target  $v$  ,error  $\varepsilon$  ,replicability  $\rho$  ,confidence  $\delta$
if  $|\mathcal{X}| < \frac{\ln\frac{2}{\delta(v - \varepsilon)}}{v - \varepsilon}$  then
$\hat{\chi}\gets \chi$
$$
\hat {\mathcal {X}} \leftarrow \left\{\frac {\ln \frac {2}{\delta (v - \varepsilon)}}{v - \varepsilon} \text {s a m p l e s} \sim \mathbb {P} \right\}
$$
$$
S \leftarrow \left\{\frac {6 4 8 \ln^ {2} / \delta + 6 4 8 (| \hat {\mathcal {X}} | + 1) \ln 2}{\rho^ {2} \varepsilon^ {2}} \text {s a m p l e s} \sim \mathbb {P} \right\}
$$
Choose  $v' \in [v - 2/3\varepsilon, v - 1/3\varepsilon]$  uniformly randomly
Output all  $x \in \hat{\mathcal{X}}$  such that  $\hat{\mathbb{P}}_S(x) \geq v'$  {empirical distribution}
Theorem D.4. Fix  $v, \rho \in (0,1)$  and  $\varepsilon \in (0,v), \delta \in (0,\rho /3)$ . Then Algorithm D.2 is  $\rho$ -replicable and returns a list  $L$  of elements  $x$  such that with probability at least  $1 - \delta$ :
(a) If  $\mathbb{P}(x)\leq v - \varepsilon$  , then  $x\notin L$
(b) If  $\mathbb{P}(x)\geq v$  , then  $x\in L$
Moreover, it has sample complexity
$$
\tilde {O} \left(\min  \left(| \mathcal {X} |, \frac {1}{(v - \varepsilon)}\right) \frac {1}{\rho^ {2} \varepsilon^ {2}} \log \frac {1}{\delta}\right).
$$
Proof. First, we wish to capture all  $v - \varepsilon$  heavy hitters in  $\hat{\mathcal{X}}$ . If  $\mathcal{X}$  is sufficiently small, this is easy. Otherwise, we fail to observe each  $(v - \varepsilon)$  heavy hitter with probability at most  $(1 - v + \varepsilon)^{|\hat{\mathcal{X}}|}$ . By a union bound over all  $(v - \varepsilon)^{-1}$  possible heavy hitters, we fail to capture all  $v - \varepsilon$  heavy hitters with probability at most
$$
\begin{array}{l} \frac {(1 - v + \varepsilon) ^ {| \hat {\mathcal {X}} |}}{v - \varepsilon} \leq \frac {1}{v - \varepsilon} \exp \left[ - (v - \varepsilon) \cdot \frac {\ln \frac {2}{\delta (v - \varepsilon)}}{v - \varepsilon} \right] \\ \leq \frac {\delta}{2}. \\ \end{array}
$$
Moving forward, we condition on this step succeeding.
Next, consider  $\hat{p}_1,\hat{p}_2$ , the mass estimates over the course of two runs supported on the candidate sets  $\hat{\mathcal{X}}_1,\hat{\mathcal{X}}_2$ . both follow (possibly different) multinomial distributions of dimension at most  $|\hat{\mathcal{X}} | + 1$  with unknown mean parameters  $p_1(x)$  for  $x\in \hat{\mathcal{X}}_1\cup \{y_1\}$ ,  $p_2(x)$  for  $x\in \hat{\mathcal{X}}_2\cup \{y_2\}$ , respectively, and  $|S|$ . Here  $y_{1},y_{2}$  are dummy elements for observations beyond  $\hat{\mathcal{X}}_1,\hat{\mathcal{X}}_2$ . Suppose we draw  $\frac{\ln 2 / \delta + (|\hat{\mathcal{X}}| + 1)\ln 2}{2\varepsilon^{\prime 2}}$  samples over each of two runs to yield estimates  $\hat{p}_1,\hat{p}_2$ . By Proposition B.1,
$$
\sum_ {x \in \hat {\mathcal {X}} _ {1} \cup \{y _ {1} \}} | \hat {p} _ {1} (x) - p _ {1} (x) | <   2 \varepsilon^ {\prime}
$$
$$
\sum_ {x \in \hat {\mathcal {X}} _ {2} \cup \{y _ {2} \}} | \hat {p} _ {2} (x) - p _ {2} (x) | <   2 \varepsilon^ {\prime}
$$
each with probability at least  $1 - \delta /2$ . Moving forward, we condition on this step succeeding.
Now, consider choosing  $v' \in [v - 2\varepsilon / 3, v - \varepsilon / 3]$  uniformly at random.
Correctness: By choosing  $2\varepsilon' < \varepsilon/3$ , any element  $x$  with true mass at least  $v$  will have empirical mass strictly more than  $v - \varepsilon/3 \geq v'$ . Similarly, any element  $x$  with true mass at most  $v - \varepsilon$  will have empirical mass strictly less than  $v - 2\varepsilon/3 \leq v'$ . Thus we satisfy the correctness guarantees. Note that we satisfy this guarantee with probability at least  $1 - \delta$ .
Replicability:  $v'$  lands in between some  $\hat{p}_1(x), \hat{p}_2(x)$  for  $x \in \hat{\mathcal{X}}_1 \cap \hat{\mathcal{X}}_2$  with total probability at most
$$
\frac {\sum_ {x \in \hat {\mathcal {X}} _ {1} \cap \hat {\mathcal {X}} _ {2}} | \hat {p} _ {1} (x) - \hat {p} _ {2} (x) |}{\varepsilon / 3} \leq \frac {\sum_ {x \in \hat {\mathcal {X}} _ {1} \cap \hat {\mathcal {X}} _ {2}} | \hat {p} _ {1} (x) - p _ {1} (x) |}{\varepsilon / 3} + \frac {\sum_ {x \in \hat {\mathcal {X}} _ {1} \cap \hat {\mathcal {X}} _ {2}} | \hat {p} _ {2} (x) - p _ {2} (x) |}{\varepsilon / 3}
$$
In addition, we wish for  $v'$  to avoid landing "below" any  $\hat{p}_1(x), \hat{p}_2(x)$  where  $x \notin \hat{\mathcal{X}}_1 \cap \hat{\mathcal{X}}_2$ . This happens with probability at most
$$
\frac {\sum_ {x \in \hat {\mathcal {X}} _ {1} \cup \{y _ {1} \} \setminus \hat {\mathcal {X}} _ {2}} | \hat {p} _ {1} (x) - p _ {1} (x) |}{\varepsilon / 3} + \frac {\sum_ {x \in \hat {\mathcal {X}} _ {2} \cup \{y _ {2} \} \setminus \hat {\mathcal {X}} _ {1}} | \hat {p} _ {2} (x) - p _ {2} (x) |}{\varepsilon / 3}.
$$
The sum of the two expressions above is at most  $\frac{4\varepsilon'}{\varepsilon/3}$ . Thus we set
$$
\frac {4 \varepsilon^ {\prime}}{\varepsilon / 3} \leq \frac {\rho}{3}
$$
$$
\varepsilon^ {\prime} \leq \frac {\rho \varepsilon}{3 6}.
$$
The probability of failing to be replicable is then at most  $1 - \rho /3 - 2\delta \geq 1 - \rho$
Sample Complexity: Recall that we set
$$
\varepsilon^ {\prime} = \min  \left(\frac {\rho \varepsilon}{3 6}, \frac {\varepsilon}{6}\right) = \frac {\rho \varepsilon}{3 6}.
$$
If  $|\mathcal{X}|$  is sufficiently small, the sample complexity becomes
$$
\frac {\ln \frac {2}{\delta} + (| \mathcal {X} | + 1) \ln 2}{2 \varepsilon^ {\prime 2}} = \frac {6 4 8 \ln \frac {2}{\delta} + 6 4 8 (| \mathcal {X} | + 1) \ln 2}{\rho^ {2} \varepsilon^ {2}}
$$
Otherwise, the sample complexity is
$$
\frac {\ln \frac {2}{\delta (v - \varepsilon)}}{v - \varepsilon} + \frac {6 4 8 \ln \frac {2}{\delta}}{\rho^ {2} \varepsilon^ {2}} + \frac {6 4 8 \left(\ln \frac {2}{\delta (v - \varepsilon)} + 1\right) \ln 2}{\rho^ {2} \varepsilon^ {2} (v - \varepsilon)}.
$$
D.3 Replicable Coreset