SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now prove that Algorithm 4.1 replicably yields an  $\varepsilon$ -coreset. In Appendix D.3.1, we state Algorithm D.3, an equivalent description of Algorithm 4.1. Then, we show that it is indeed replicable in Appendix D.3.2. Following this, we prove that the cost of any set of centers  $F \subseteq \mathcal{B}_d$  is preserved up to an  $\varepsilon$ -multiplicative error in Appendix D.3.3. In Appendix D.3.4, we show that the outputted coreset is of modest size. Last but not least, we summarize our findings in Appendix D.3.5.
D.3.1 Pseudocode
Before presenting the analysis, we state an equivalent algorithm to Algorithm 4.1 which uses precise notation that is more suitable for analysis.
Starting with the grid that consists of a single cell  $[-1/2, 1/2]^d$  at layer 0, we recursively subdivide each cell into  $2^d$  smaller cells of length  $2^{-i}$  at layer  $i$ . Each subdivided cell is a child of the larger cell, the containing cell is the parent of a child cell, and cells sharing the same parent are siblings. Let  $\mathcal{G}_i$  denote the union of cells at the  $i$ -th level. We write  $\mathbb{P}_i$  to denote the discretized distribution to  $\mathcal{G}_i$ . In other words,  $\mathbb{P}_i = \mathbb{P}\big|_{\sigma(\mathcal{G}_i)}$  is the restriction of  $\mathbb{P}$  to the smallest  $\sigma$ -algebra containing  $\mathcal{G}_i$ . We say a cell on the  $i$ -th level is heavy if the replicable heavy hitters algorithm (cf. Algorithm D.2) returns it with input distribution  $\mathbb{P}_i$ , heavy hitter threshold  $v = \frac{\gamma \cdot \Lambda}{2 - p^i}$ , and error  $v/2$  for some parameter  $\gamma$  to be determined later. A cell that is not heavy is light. The recursion terminates either when there are no more heavy cells on the previous layer, or the current grid length is a sufficiently small fraction of OPT. In the second case, we say the last layer consists of special cells.
If no child cell of a heavy parent is heavy, we mark the parent. Otherwise, we recurse on its heavy children so that one of its descendants will eventually be marked. The recursion must stop, either when there are no more heavy children or when the side length is no more than a fraction of  $\Lambda$ . Note that the light cells with heavy parents and special cells partition  $[-1/2, 1/2]^d$ .
Then, we build  $R$  in reverse as follows. For a marked heavy cell  $Z$ , we set  $R(x)$  to be an arbitrary (but replicable) point in  $Z$ , e.g. its center, for all  $x \in Z$ . This covers all light cells with no heavy siblings as well as special cells. Otherwise, a cell  $Z$  is light with heavy siblings, and its parent is heavy but unmarked. Thus, the parent has some marked heavy descendant  $Z'$ , and we choose  $R(x) = R(x')$  for all  $x \in Z$  and  $x' \in Z'$ .
D.3.2 Replicability
Proposition D.5. Algorithm D.3 terminates after at most
$$
t := \left\lceil \frac {1}{p} \log \frac {1 0 \beta \Delta^ {p}}{\varepsilon \cdot \mathrm {O P T}} + 1 \right\rceil
$$
layers.
Algorithm D.3 Replicable Coreset; Algorithm 4.1 Equivalent
1: rCoreset(distribution  $\mathbb{P}$ , accuracy  $\varepsilon$ , exponent  $p$ , replicability  $\rho$ , confidence  $\delta$ ):  
2: Init  $\mathcal{H}[i] \gets \emptyset$  for all  $i$   
3: Init  $\mathcal{L}[i] \gets \emptyset$  for all  $i$   
4: Init  $\mathcal{S}[i] \gets \emptyset$  for all  $i$   
5:  
6:  $\mathcal{H}(0) \gets \left\{[-1/2, 1/2]^d\right\}$   
7: for  $i \gets 1$ ;  $H[i - 1] \neq \emptyset$ ;  $i \gets i + 1$  do  
8: if  $(2^{-i + 1} \Delta)^p \leq \varepsilon \Lambda / 5$  then  
9:  $\mathcal{S}[i] \gets \text{children}(\mathcal{H}[i - 1])$   
10: else  
11:  $\mathcal{H}[i] \gets \text{children}(\mathcal{H}[i - 1]) \cap \text{HeavyHitters}(\mathbb{P}_i, v = \frac{\gamma \cdot \Lambda}{2 - p_i}, v/2, \rho/t, \delta/t)$  {t is an upper bound on the number of layers.}  
12:  $\mathcal{L}[i] \gets \text{children}(\mathcal{H}[i - 1]) \setminus \mathcal{H}[i]$   
13: end if  
14: end for  
15:  
16: for  $j \gets i - 1$ ;  $j \geq 0$ ;  $j \gets j - 1$  do  
17: for  $Z \in \mathcal{H}[j]$  do  
18: if children $(Z) \cap \mathcal{H}[j + 1] = \emptyset$  then  
19:  $R'(Z) \gets \text{rChoice}(Z)$   
20: else  
21:  $R'(Z) \gets R'(\text{rChoice(children}[Z) \cap \mathcal{H}[j + 1]))$   
22: end if  
23: end for  
24: for  $Z \in \mathcal{L}[j + 1]$  do  
25:  $R(Z) \gets R'(\text{parent}(Z))$   
26: end for  
27: for  $Z \in S[j + 1]$  do  
28:  $R(Z) \gets R'(\text{parent}(Z))$   
29: end for  
30: end for  
31:  
32: Output  $R$
$$
\begin{array}{l} (2 ^ {- i + 1} \Delta) ^ {p} \leq \frac {\varepsilon \Lambda}{5} \\ \Longleftrightarrow 2 ^ {p (- i + 1)} \leq \frac {\varepsilon \Lambda}{5 \Delta^ {p}} \\ \Longleftrightarrow p (- i + 1) \leq \log \frac {\varepsilon \Lambda}{5 \Delta^ {p}} \\ \Longleftrightarrow i \geq \frac {1}{p} \log \frac {5 \Delta^ {p}}{\varepsilon \cdot \Lambda} + 1. \\ \end{array}
$$
But  $1 / \Lambda \leq \beta (1 + \varepsilon) / \mathrm{OPT}$ , concluding the proof.
Corollary D.6. Algorithm D.3 is  $\rho$ -replicable and succeeds with probability at least  $1 - \delta$ . Moreover, it has sample complexity
$$
\tilde {O} \left(\frac {t ^ {2}}{\rho^ {2} \gamma^ {3} \cdot \mathrm {O P T} ^ {3}} \log \frac {1}{\delta}\right).
$$
Proof. The only non-trivial random components of Algorithm D.3 is the heavy hitter estimation. By Theorem D.4, the heavy hitters subroutine call on the  $i$ -th layer is  $\rho / t$ -replicable and succeeds with
probability at least  $1 - \delta / t$ . Also, it has sample complexity
$$
\tilde {O} \left(\frac {t ^ {2} \cdot 2 ^ {3 (- p i + 1)}}{\rho^ {2} \gamma^ {3} \cdot \Lambda^ {3}} \log \frac {1}{\delta}\right).
$$
$$
\begin{array}{l} \sum_ {i = 0} ^ {t} 2 ^ {- 3 p i} \leq \sum_ {i = 0} ^ {\infty} 2 ^ {- 3 p i} \\ = O (1). \\ \end{array}
$$
Thus all in all, Algorithm D.3 is  $\rho$ -replicable and succeeds with probability at least  $1 - \delta$ . It also has sample complexity
$$
\tilde {O} \left(\frac {t ^ {2}}{\rho^ {2} \gamma^ {3} \cdot \Lambda^ {3}} \log \frac {1}{\delta}\right).
$$
Since  $1 / \Lambda \leq \beta (1 + \varepsilon) / \mathrm{OPT}$ , we conclude the proof.
D.3.3 Expected Shift
Proposition D.7. For any  $x \in Z \in \mathcal{L}[i] \cup \mathcal{S}[i]$  outputted by Algorithm D.3,
$$
\kappa (x, R (x)) \leq (2 ^ {- i + 1} \Delta).
$$
Proof.  $x$  is represented by some point within its parent.
Proposition D.8. The output  $S[t]$  of Algorithm D.3 satisfies
$$
\int_ {\mathcal {S} [ t ]} \kappa (x, R (x)) ^ {p} d \mathbb {P} (x) \leq \frac {\varepsilon \Lambda}{5}.
$$
Proof. By computation,
$$
\begin{array}{l} \int_ {\mathcal {S} [ t ]} \kappa (x, R (x)) ^ {p} d \mathbb {P} (x) \leq \int_ {\mathcal {S} [ t ]} \left(2 ^ {- i + 1} \Delta\right) ^ {p} d \mathbb {P} (x) \quad \text {P r o p o s i t i o n D . 7} \\ \leq \int_ {\mathcal {S} [ t ]} \frac {\varepsilon \Lambda}{5} d \mathbb {P} (x) \quad \text {d e f i n i t i o n} t \\ \leq \int_ {\mathcal {X}} \frac {\varepsilon \Lambda}{5} d \mathbb {P} (x) \\ = \frac {\varepsilon \Lambda}{5}. \\ \end{array}
$$
Let  $F$  be an arbitrary set of  $k$  centers. Partition  $\mathcal{L}[i]$  into
$$
\mathcal {L} _ {\text {n e a r}} [ i ] := \left\{Z \in \mathcal {L} [ i ]: \min  _ {x \in Z} \kappa (x, F) ^ {p} \leq \frac {5}{\varepsilon} (2 ^ {- i + 1} \Delta) ^ {p} \right\}
$$
$$
\mathcal {L} _ {\mathrm {f a r}} [ i ] := \left\{Z \in \mathcal {L} [ i ]: \min  _ {x \in Z} \kappa (x, F) ^ {p} > \frac {5}{\varepsilon} (2 ^ {- i + 1} \Delta) ^ {p} \right\}.
$$
Proposition D.9. The output  $\mathcal{L}_{\mathrm{far}}$  of Algorithm D.3 satisfies
$$
\sum_ {i = 0} ^ {t} \int_ {\mathcal {L} _ {\text {f a r}} [ i ]} \kappa (x, R (x)) ^ {p} d \mathbb {P} (x) \leq \frac {\varepsilon}{5} \operatorname {c o s t} (F).
$$
Proof. Any  $x \in Z \in \mathcal{L}_{\mathrm{far}}[i]$  contributes a cost of at least  $\frac{5}{\varepsilon} (2^{-i + 1}\Delta)^p$ . Hence
$$
\begin{array}{l} \sum_ {i = 0} ^ {t} \int_ {\mathcal {L} _ {\text {f a r}} [ i ]} \kappa (x, R (x)) ^ {p} d \mathbb {P} (x) \leq \sum_ {i = 0} ^ {t} \int_ {\mathcal {L} _ {\text {f a r}} [ i ]} (2 ^ {- i + 1} \Delta) ^ {p} d \mathbb {P} (x) \\ = \frac {\varepsilon}{5} \sum_ {i = 0} ^ {t} \int_ {\mathcal {L} _ {\text {f a r}} [ i ]} \frac {5}{\varepsilon} (2 ^ {- i + 1} \Delta) ^ {p} d \mathbb {P} (x) \\ \leq \frac {\varepsilon}{5} \sum_ {i = 0} ^ {t} \int_ {\mathcal {L} _ {\text {f a r}} [ i ]} \kappa (x, F) ^ {p} d \mathbb {P} (x) \\ \leq \frac {\varepsilon}{5} \operatorname {c o s t} (F). \\ \end{array}
$$
Proposition D.10. The output  $\mathcal{L}_{\mathrm{near}}$  of Algorithm D.3 satisfies  $|\mathcal{L}_{\mathrm{near}}[i]|\leq kM$  where
$$
M := \left[ \frac {2 ^ {5}}{\varepsilon} \Delta \right] ^ {d}
$$
Proof. The furthest point  $x$  in a cell in  $\mathcal{L}_{\mathrm{near}}[i]$  can have a distance of at most
$$
2 ^ {- i} \Delta + \sqrt [ p ]{\frac {5}{\varepsilon}} (2 ^ {- i + 1} \Delta) \leq \left(1 + \frac {1 0}{\varepsilon}\right) (2 ^ {- i} \Delta)
$$
to the nearest center. Thus the points belonging to a particular center live in an  $\ell_{\infty}$  ball of that radius. Not including the cell containing the center, we can walk past at most
$$
\frac {1}{2 ^ {- i}} \cdot \left(1 + \frac {1 0}{\varepsilon}\right) (2 ^ {- i} \Delta) = \left(1 + \frac {1 0}{\varepsilon}\right) \Delta
$$
cells if we walk in the directions of the canonical basis of  $\mathbb{R}^d$ . It follows that there can be at most
$$
\left[ 1 + 2 \left(1 + \frac {1 0}{\varepsilon}\right) \Delta \right] ^ {d} \leq \left[ \frac {3 2}{\varepsilon} \Delta \right] ^ {d}
$$
cells in  $\mathcal{L}_{\mathrm{near}}[i]$  close to each center of  $F$ , and thus there are at most  $kM$  cells in total.
Proposition D.11. By choosing
$$
\gamma := \frac {\varepsilon}{5 t k M (2 \Delta) ^ {p}},
$$
The output  $\mathcal{L}_{\mathrm{near}}$  of Algorithm D.3 satisfies
$$
\sum_ {i = 0} ^ {t} \int_ {\mathcal {L} _ {\text {n e a r}} [ i ]} \kappa (x, R (x)) ^ {p} \leq \frac {\varepsilon \Lambda}{5}.
$$
Proof. Each light cell has measure at most  $\frac{\gamma \cdot \Lambda}{2^{-pi}}$ . By computation,
$$
\begin{array}{l} \sum_ {i = 0} ^ {t} \int_ {\mathcal {L} _ {\text {n e a r}} [ i ]} \kappa (x, R (x)) ^ {p} d \mathbb {P} (x) \leq \sum_ {i = 1} ^ {t} \int_ {\mathcal {L} _ {\text {n e a r}} [ i ]} (2 ^ {- i + 1} \Delta) ^ {p} d \mathbb {P} (x) \\ \leq \sum_ {i = 1} ^ {t} k M \cdot \frac {\gamma \cdot \Lambda}{2 ^ {- p i}} 2 ^ {- p i} (2 \Delta) ^ {p} \\ = t \cdot k M \cdot \gamma \cdot \Lambda \cdot (2 \Delta) ^ {p} \\ \leq \frac {\varepsilon \Lambda}{5}. \\ \end{array}
$$
Corollary D.12. Set
$$
\gamma := \frac {\varepsilon}{5 t k M (2 \Delta) ^ {p}}.
$$
Then for any set  $F$  of  $k$ -centers, the output  $R$  of Algorithm D.3 satisfies
$$
\int_ {\mathcal {X}} \kappa (x, R (x)) ^ {p} d \mathbb {P} (x) \leq \varepsilon \operatorname {c o s t} (F).
$$
Proof.  $\mathcal{X}$  is contained in the union of all  $\mathcal{L}[i]$ 's and  $\mathcal{S}[t]$ . Hence
$$
\begin{array}{l} \int_ {\mathcal {X}} \kappa (x, R (x)) ^ {p} d \mathbb {P} (x) \leq \frac {2 \varepsilon \Lambda}{5} + \frac {\varepsilon}{5} \operatorname {c o s t} (F) \\ \leq \frac {2 \varepsilon (1 + \varepsilon) \mathrm {O P T}}{5} + \frac {\varepsilon}{5} \operatorname {c o s t} (F) \\ \leq \frac {4 \varepsilon}{5} \operatorname {O P T} + \frac {\varepsilon}{5} \operatorname {c o s t} (F) \\ \leq \varepsilon \operatorname {c o s t} (F). \\ \end{array}
$$
D.3.4 Coreset Size
To determine the size of our coreset, we bound the number of marked heavy cells, as all our representative points are in marked heavy cells.
Fix  $F_{\mathrm{OPT}}$  to be a set of optimal centers for the  $(k,p)$ -clustering problem. Write  $\mathcal{M}[i]$  to denote the set of marked heavy cells on the  $i$ -th layer and partition  $\mathcal{M}[i]$  as
$$
\mathcal {M} _ {\text {c l o s e}} [ i ] := \left\{Z \in \mathcal {M} [ i ]: \min  _ {x \in Z} \kappa (x, F) \leq 2 ^ {- p i + 1} \right\}
$$
$$
\mathcal {M} _ {\mathrm {d i s t}} [ i ] := \left\{Z \in \mathcal {M} [ i ]: \min  _ {x \in Z} \kappa (x, F) > 2 ^ {- p i + 1} \right\}.
$$
Proposition D.13. The marked heavy cells  $\mathcal{M}_{\mathrm{dist}}$  outputted by Algorithm D.3 satisfy
$$
\left| \bigcup_ {0 \leq i \leq t} \mathcal {M} _ {\mathrm {d i s t}} [ i ] \right| \leq \frac {2 \beta}{\gamma}.
$$
Proof. Each cell in  $\mathcal{M}_{\mathrm{dist}}[i]$  has mass at least  $\frac{\gamma \cdot \Lambda}{2^{-pi+1}}$  and each  $x \in Z \in \mathcal{M}_{\mathrm{dist}}[i]$  is of distance at least  $2^{-pi+1}$  to its closest center. Thus each cell contributes at least
$$
\frac {\gamma \cdot \Lambda}{2 ^ {- p i + 1}} \cdot 2 ^ {- p i + 1} = \gamma \cdot \Lambda
$$
to the objective. If follows that there are at most
$$
\frac {\mathrm {O P T}}{\gamma \Lambda} \leq \frac {1}{\gamma} \beta (1 + \varepsilon) \leq \frac {2 \beta}{\gamma}.
$$
such cells.
Proposition D.14. The marked heavy cells  $\mathcal{M}_{\mathrm{close}}$  outputted by Algorithm D.3 satisfy
$$
\left. \left| \mathcal {M} _ {\text {c l o s e}} [ i ] \right| \leq k (7 \Delta) ^ {d}. \right.
$$
Proof. The furthest point  $x$  in a cell  $\mathcal{M}_{\mathrm{close}}[i]$  to its nearest center is
$$
2 ^ {- p i + 1} + 2 ^ {- i} \Delta .
$$
In other words, not including the cell containing the center, we can walk past at most
$$
2 ^ {i (1 - p) + 1} + \Delta \leq 2 + \Delta
$$
cells by walking along the direction of the canonical basis of  $\mathbb{R}^d$ . Thus there are at most
$$
k \left[ 1 + 2 (2 + \Delta) \right] ^ {d} \leq k [ 7 \Delta ] ^ {d}
$$
such cells on the  $i$ -th layer.
Proposition D.15. The image of the map  $R$  outputted by Algorithm D.3 satisfies
$$
N := | R (\mathcal {X}) | \leq \frac {2 \beta}{\gamma} + t k (7 \Delta) ^ {d}.
$$
In particular, for
$$
N := | R (\mathcal {X}) | \leq \frac {3 \beta}{\gamma} = 3 \beta \cdot \frac {5 t k M (2 \Delta) ^ {p}}{\varepsilon}.
$$
Proof. The image is contained in the marked heavy cells.
D.3.5  $k$ -Medians &  $k$ -Means
We now derive results for the case of  $p = 1$ .
Theorem D.16. Algorithm D.3 is  $\rho$ -replicable and outputs an  $\varepsilon$ -coreset of size
$$
O \left(\frac {k 2 ^ {5 d} \Delta^ {d + 1}}{\varepsilon^ {d + 1}} \log \frac {\Delta}{\varepsilon \cdot \mathrm {O P T}}\right)
$$
for statistical  $k$ -medians with probability at least  $1 - \delta$ . Moreover, it has sample complexity
$$
\tilde {O} \left(\frac {k ^ {3} 2 ^ {1 5 d} \Delta^ {3 d + 3}}{\rho^ {2} \varepsilon^ {3 d + 3} \cdot \mathrm {O P T} ^ {3}} \log \frac {1}{\delta}\right).
$$
Proof (Theorem D.16). By Corollary D.12, Algorithm D.3 outputs an  $\varepsilon$ -coreset. By Corollary D.6, Algorithm D.3 is  $\rho$ -replicable with sample complexity
$$
\frac {3 \beta}{\gamma} = O \left(\frac {k \left[ 2 ^ {5} \Delta / \varepsilon \right] ^ {d} (2 \Delta)}{\varepsilon} \cdot \log \frac {\Delta}{\varepsilon \cdot \mathrm {O P T}}\right) = O \left(\frac {k 2 ^ {5 d} \Delta^ {d + 1}}{\varepsilon^ {d + 1}} \log \frac {\Delta}{\varepsilon \cdot \mathrm {O P T}}\right)
$$
By Proposition D.15, the coreset has size at most this value.
Substituting the value of  $\gamma$  above and remarking that  $t$  is a polylogarithmic factor of the other terms, we thus conclude that the final sample complexity is
$$
\tilde {O} \left(\frac {k ^ {3} 2 ^ {1 5 d} \Delta^ {3 d + 3}}{\varepsilon^ {3 d + 3}} \cdot \frac {1}{\rho^ {2} \cdot \mathrm {O P T} ^ {3}} \log \frac {1}{\delta}\right) = \tilde {O} \left(\frac {k ^ {3} 2 ^ {1 5 d} \Delta^ {3 d + 3}}{\rho^ {2} \varepsilon^ {3 d + 3} \cdot \mathrm {O P T} ^ {3}} \frac {1}{\delta}\right).
$$
Now consider the case of  $p = 2$ . We can perform our analysis with the help of a standard inequality.
Proposition D.17. For any set of centers  $F$ ,
$$
\begin{array}{l} \mathbb {E} _ {x} \kappa (R (x), F) ^ {2} \leq \mathbb {E} _ {x} \kappa (x, F) ^ {2} + 2 \sqrt {\mathbb {E} _ {x} [ \kappa (x , F) ^ {2} ] \cdot \mathbb {E} _ {x} [ \kappa (x , R (x)) ^ {2} ]} + \mathbb {E} _ {x} \kappa (x, R (x)) ^ {2} \\ \mathbb {E} _ {x} \kappa (x, F) ^ {2} \leq \mathbb {E} _ {x} \kappa (R (x), F) ^ {2} + 2 \sqrt {\mathbb {E} _ {x} [ \kappa (R (x) , F) ^ {2} ] \cdot \mathbb {E} _ {x} [ \kappa (x , R (x)) ^ {2} ]} + \mathbb {E} _ {x} \kappa (x, R (x)) ^ {2}. \\ \end{array}
$$
Thus if  $\mathbb{E}_x\kappa (x,R(x))^2\leq \frac{\varepsilon^2}{2^6}\mathbb{E}_x\kappa (x,F)^2$
$$
\begin{array}{l} \mathbb {E} _ {x} \kappa (R (x), F) ^ {2} \leq \mathbb {E} _ {x} \kappa (x, F) ^ {2} + 2 \cdot \frac {1}{2 ^ {3}} \varepsilon \mathbb {E} _ {x} [ \kappa (x, F) ^ {2} ] + \frac {1}{2 ^ {6}} \varepsilon^ {2} \mathbb {E} _ {x} [ \kappa (x, F) ^ {2} ] \\ \leq \left(1 + \frac {\varepsilon}{4}\right) \mathbb {E} _ {x} \kappa (x, F) ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {x} \kappa (x, F) ^ {2} \leq \mathbb {E} _ {x} \kappa (R (x), F) ^ {2} + 2 \sqrt {(1 + \varepsilon) \mathbb {E} _ {x} [ \kappa (x , F) ^ {2} ] \cdot \frac {1}{2 ^ {6}} \varepsilon^ {2} \mathbb {E} _ {x} [ \kappa (x , F) ^ {2} ]} + \frac {1}{2 ^ {6}} \varepsilon^ {2} \mathbb {E} _ {x} \kappa (x, F) ^ {2} \\ \leq \mathbb {E} _ {x} \kappa (R (x), F) ^ {2} + \frac {1}{2} \varepsilon \mathbb {E} _ {x} [ \kappa (x, F) ^ {2} ] + \frac {1}{2 ^ {6}} \varepsilon^ {2} \mathbb {E} _ {x} \kappa (x, F) ^ {2} \\ \leq \mathbb {E} _ {x} \kappa (R (x), F) ^ {2} + \varepsilon \mathbb {E} _ {x} \kappa (x, F) ^ {2}. \\ \end{array}
$$
Proof. By Hölder's inequality,
$$
\begin{array}{l} \mathbb {E} _ {x} \kappa (R (x), F) ^ {2} \leq \mathbb {E} _ {x} [ \kappa (x, F) + \kappa (x, R (x)) ] ^ {2} \\ = \mathbb {E} _ {x} \kappa (x, F) ^ {2} + 2 \mathbb {E} _ {x} [ \kappa (x, F) \kappa (x, R (x)) ] + \mathbb {E} _ {x} \kappa (x, R (x)) ^ {2} \\ \leq \mathbb {E} _ {x} \kappa (x, F) ^ {2} + 2 \sqrt {\mathbb {E} _ {x} [ \kappa (x , F) ^ {2} ] \mathbb {E} _ {x} [ \kappa (x , R (x)) ^ {2} ]} + \mathbb {E} _ {x} \kappa (x, R (x)) ^ {2}. \\ \end{array}
$$
The other case is identical.
Theorem D.18. Algorithm D.3 is  $\rho$ -replicable and outputs an  $\varepsilon$ -coreset for statistical  $k$ -means of size
$$
O \left(\frac {k 2 ^ {1 1 d} \Delta^ {d + 2}}{\varepsilon^ {2 d + 2}} \log \frac {\Delta^ {2}}{\varepsilon \mathrm {O P T}}\right)
$$
with probability at least  $1 - \delta$ . Moreover, it has sample complexity
$$
\tilde {O} \left(\frac {k ^ {3} 2 ^ {3 3 d} \Delta^ {3 d + 6}}{\rho^ {2} \varepsilon^ {6 d + 6} \cdot \mathrm {O P T} ^ {3}} \log \frac {1}{\delta}\right).
$$
Proof (Theorem D.18). Similar to  $k$ -medians, Algorithm D.3 outputs an  $\varepsilon$ -coreset with probability at least  $1 - \delta$ , is  $\rho$ -replicable, and has sample complexity
However, we need to take  $\varepsilon^{\prime} := \varepsilon^{2}/2^{6}$ .
$$
\frac {3}{\gamma} = O \left(\frac {t k \left[ 2 ^ {5 + 6} \Delta / \varepsilon^ {2} \right] ^ {d} (2 \Delta) ^ {2}}{\varepsilon^ {2}}\right) = O \left(\frac {k 2 ^ {1 1 d} \Delta^ {d + 2}}{\varepsilon^ {2 d + 2}} \log \frac {\Delta^ {2}}{\varepsilon \mathrm {O P T}}\right).
$$
$t$  is a polylogarithmic factor of the rest of the parameters, thus we thus conclude that the final sample complexity is
$$
\tilde {O} \left(\frac {k ^ {3} 2 ^ {3 3 d} \Delta^ {3 d + 6}}{\varepsilon^ {6 d + 6}} \cdot \frac {1}{\rho^ {2} \cdot \mathrm {O P T} ^ {3}}\right) = \tilde {O} \left(\frac {k ^ {3} 2 ^ {3 3 d} \Delta^ {3 d + 6}}{\rho^ {2} \varepsilon^ {6 d + 6} \cdot \mathrm {O P T} ^ {3}} \log \frac {1}{\delta}\right).
$$
It is not clear how to compare our guarantees with that of Frahling and Sohler [2005] as we attempt to minimize the number of samples required from a distribution assuming only sample access. On the other hand, Frahling and Sohler [2005] attempted to minimize the running time of their algorithm assuming access to the entire distribution.
D.4 Replicable Multinomial Parameter Estimation
Now that we have reduced the problem to a finite distribution (coreset), we can think of it as a weighted instance of  $(k,p)$ -clustering where the weights are unknown but can be estimated using data.
Proposition D.19. Enumerate the coreset  $R(\mathcal{X}) = r^{(1)}, \ldots, r^{(N)}$  and let  $w^{(i)}$  be the probability mass at  $r^{(i)}$ . If we have estimates  $\hat{w}^{(i)}$  satisfying
$$
\left| \hat {w} ^ {(i)} - w ^ {(i)} \right| <   \frac {\varepsilon}{N},
$$
$$
\left| \sum_ {i = 1} ^ {N} w ^ {(i)} \kappa (r ^ {(i)}, F) ^ {p} - \sum_ {i = 1} ^ {N} \hat {w} ^ {(i)} \kappa (r ^ {(i)}, F) ^ {p} \right| \leq \varepsilon .
$$
for all centers  $F$ .
Proof. Indeed,
$$
\begin{array}{l} \left| \sum_ {i = 1} ^ {N} w ^ {(i)} \kappa (r ^ {(i)}, F) ^ {p} - \sum_ {i = 1} ^ {N} \hat {w} ^ {(i)} \kappa (r ^ {(i)}, F) ^ {p} \right| \leq \sum_ {i = 1} ^ {N} | w ^ {(i)} - \hat {w} ^ {(i)} | \kappa (r ^ {(i)}, F) ^ {p} \\ \leq \sum_ {i = 1} ^ {N} | w ^ {(i)} - \hat {w} ^ {(i)} | \cdot 1 \\ \leq \varepsilon . \\ \end{array}
$$
Let us formulate this as a replicable multinomial parameter estimation problem. Consider a multinomial distribution  $Z = (Z^{(1)},\ldots ,Z^{(N)})$  of dimension  $N$  with parameters  $p^{(1)},\dots,p^{(N)}$  and  $n$ . Note that this can be cast as a more general statistical query problem which we illustrate below. We allow simultaneous estimation of more general statistical queries  $g_{1},\ldots ,g_{N}:(\mathcal{X}^{n},\mathbb{P})\to \mathbb{R}$  compared to Impagliazzo et al. [2022], assuming they can be estimated with high accuracy and confidence from data, say each query  $g_{j}(x_{1},\ldots ,x_{n})$  concentrates about its true value  $G_{j}\in \mathbb{R}$  with high probability.
Algorithm D.4 Replicable Rounding
1: rRounding(statistical queries  $g_{1},\ldots g_{N}$  distribution  $\mathbb{P}$  number of samples  $n$  error  $\varepsilon$  , replicability  $\rho$  , confidence  $\delta$  ):
2: Sample  $x_{1},\ldots ,x_{n}\sim \mathbb{P}$
3:  $\alpha \leftarrow \frac{2\varepsilon}{\rho + 1 - 2\delta}$
4: for  $j\gets 1,\ldots ,N$  : do
5:  $\alpha^{(j)}\gets$  uniform random sample  $[0,\alpha ]$
6: Split  $\mathbb{R}$  into intervals  $I^{(j)} = \{[\alpha^{(j)} + z\alpha ,\alpha^{(j)} + (z + 1)\alpha):z\in \mathbb{Z}\}$
7: Output the midpoint  $\hat{G}_j$  of the interval within  $I^{(i)}$  that  $g_{j}(x_{1},\ldots ,x_{n})$  falls into
8: end for
Theorem D.20 (Replicable Rounding). Suppose we have a finite class of statistical queries  $g_1, \ldots, g_N$  and sampling  $n$  independent points from  $\mathbb{P}$  ensures that
Then Algorithm D.4 is  $\rho$ -replicable and outputs estimates  $\hat{G}_j$  such that
$$
| \hat {G} _ {j} - G _ {j} | \leq \varepsilon
$$
Proof. Outputting the midpoint of the interval can offset each  $g_{j}(x_{1},\ldots ,x_{n})$  by at most  $\frac{\alpha}{2}$ . Hence
$$
\left| \hat {G} _ {j} - G _ {j} \right| \leq \frac {\varepsilon (\rho - 2 \delta)}{\rho + 1 - 2 \delta} + \frac {\varepsilon}{\rho + 1 - 2 \delta} = \varepsilon .
$$
Consider two executions of our algorithm yielding estimates  $\hat{G}^{(1)},\hat{G}^{(2)}$ . The probability that any of the estimates fail to satisfy  $\ell_1$ -tolerance  $\varepsilon^{\prime}$  is at most  $2\delta$ . The two executions output different estimates only if some random offset "splits" some  $\hat{G}_j^{(1)},\hat{G}_j^{(2)}$ . Conditioning on the success to satisfy  $\ell_1$ -tolerance, this occurs with probability at most
$$
\begin{array}{l} \sum_ {j = 1} ^ {N} \frac {\left| \hat {G} _ {j} ^ {(1)} - \hat {G} _ {j} ^ {(2)} \right|}{\alpha} \leq \sum_ {j = 1} ^ {N} \frac {\left| \hat {G} _ {j} ^ {(1)} - G _ {j} \right| + \left| \hat {G} _ {j} ^ {(2)} - G _ {j} \right|}{\alpha} \\ \leq \frac {2 \varepsilon^ {\prime}}{\alpha} \\ \leq \rho - 2 \delta . \\ \end{array}
$$
Accounting for the probability of failure to satisfy the  $\ell_1$ -tolerance of  $2\delta$ , our algorithm is  $\rho$ -replicable.
We note that Theorem D.20 can be thought of as a generalization of the SQ oracle (cf. Theorem D.2) from Impagliazzo et al. [2022]. Indeed, simply using Theorem D.2 yields an extra factor of  $N$  in the sample complexity as we must take a union bound.
Corollary D.21. Let  $\varepsilon, \rho \in (0, 1)$  and  $\delta \in (0, \rho/3)$ . There is a  $\rho$ -replicable algorithm that outputs parameter estimates  $\bar{p}$  for a multinomial distribution of dimension  $N$  such that