Replicable Clustering
Authors: Hossein Esfandiari
Amin Karbasi
Vahab Mirrokni
Grigoris Velegkas
Felix Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Theory
Clustering Theory
Statistical Learning Theory
Reproducibility
Replicability
TL;DR: We design replicable algorithms for statistical k-medians, statistical k-means, and statistical k-centers.

Abstract: We design replicable algorithms in the context of statistical clustering under the recently introduced notion of replicability from Impagliazzo et al. [2022]. According to this definition, a clustering algorithm is replicable if, with high probability, its output induces the exact same partition of the sample space after two executions on different inputs drawn from the same distribution, when its internal randomness is shared across the executions. We propose such algorithms for the statistical $k$-medians, statistical $k$-means, and statistical $k$-centers problems by utilizing approximation routines for their combinatorial counterparts in a black-box manner. In particular, we demonstrate a replicable $O(1)$-approximation algorithm for statistical Euclidean $k$-medians ($k$-means) with $\operatorname{poly}(d)$ sample complexity. We also describe an $O(1)$-approximation algorithm with an additional $O(1)$-additive error for statistical Euclidean $k$-centers, albeit with $\exp(d)$ sample complexity. In addition, we provide experiments on synthetic distributions in 2D using the $k$-means++ implementation from sklearn as a black-box that validate our theoretical results.

Replicable Clustering"
Hossein Esfandiari Google Research esf esfandiarion saandiari@goggee.com
Amin Karbasi Yale University, Google Research amin karkaaarbbaaaae @yale.eealle. edu
Vahab Mirrokni
Google Research
mirrokni Mhirooni@googlee...c @google com
Grigoris Velegkas
Felix Zhou
Yale University
Yale University
arigoriss.ellegkasse edu
felix. wwwo@yalll.coooo edu
Abstract
We design replicable algorithms in the context of statistical clustering under the recently introduced notion of replicability from Impagliazzo et al. [2022]. Accord- ing to this definition, clustering algorithm is replicable if, with high probability, its output induces the exact same partition of the sample space after two exe- cutions different inputs drawn from the same distribution, when its internal randomness shared across the executions. We propose such algorithms for the statistical k-medians, statistical k-means, and statistical k-centers problems by utilizing approximation routines for their combinatorial counterparts in a black-box manner. In particular, we demonstrate replicable o(1)-apprxxiiation algo- rithm for statistical Euclidean k-medians (k-means) with Oppoly (k; d'j,log log sample complexity. We also describe an O(1) ((1)xappoxiimaion algorithm with an additional -addiiidt error for statistical Euclidean k-centers, albeit with (poly(A) (k) exp(d)) sample complexity. In addition, we provide experiments on synthetic distributions in 2D using the k-means++ implementation from sklearn a black-box that validate our theoretical results?.
Introduction
The unprecedented increase in the amount of data that is available to researchers across many different scientific areas has led to the study and development of automated data analysis methods. One fundamental category of such methods is unsupervised learning which aims to identify some inherent structure unlabeled data. Perhaps the most well-studied way to do that by grouping together data that share similar characteristics. As a result, clustering algorithms have become one of the central objects of study in unsupervised learning. Despite very long line work studying such algorithms, e.g. Jain and Dubes [1988], Hart et al. [2000], Anderberg [2014], there is not an agreed-upor definition that quantifies the quality of a clustering solution. Kleinberg [2002] showed that there is an inherent reason why this is the case: it is impossible to design clustering function that satisfies three natural properties, namely ecale-inaariace richness of solutions, and consistency. This means that the algorithm designer needs to balance several conflicting desiderata. As result, the radically different approaches that scientists use depending on their application domain can be sensitive to several factors such their random initialization, the measure of similarity of the data, the presence of noise in the measurements, and the existence of outliers in the dataset. All these issues give rise algorithms whose results are not replicable, i.e., when execute them on two different samples of the same population, they output solutions that vary significantly. This begs the
Authors are listed alphabetically. attops://anooymoouss http://wwww......ooo.o......atta
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

following question. Since the goal of clustering algorithms is to reveal properties of the underlying population, how can we trust and utilize their results when they fail to pass this simple test?
Replicability is imperative in making sure that scientific findings are both valid and reliable. Re- searchers have obligation to provide coherent results and conclusions across multiple repetitions of the same experiment. Shockingly, 2016 survey that appeared in Nature [Baker, 2016] revealed that 70% of researchers tried, but were unable to, replicate the findings of another researcher and more than 50% of them believe there is significant crisis in replicability. Unsurprisingly, similar worries have been echoed the subfields of machine learning and data science [Pineau et al., 2019, 2021]. In this work, we initiate the study of replicability in clustering, which is one of the canonical problems of unsupervised learning.
1.1 Related Works
Statistical Clustering. The most relevant previous results for (non-replicable) statistical clustering was established by Ben-David [2007], who designed D(p)-approinnation algorithms for statistical k-medians (k- (k-means) with (lk) sample complexity. However, their algorithm picks centers from within the samples and is therefore non-replicable.
Combinatorial Clustering. The flavor of clustering most studied in the approximation algorithms literature is the setting where we have uniform distribution over finite points and the algorithm has explicit access to the entire distribution. Our algorithms rely on having black-box access to combinatorial clustering oracle. See Byrka et al. [2017], Ahmadian et al. [2019] for the current best polynomial-ting approximation algorithms for combinatorial k-medians (k-means) in general metrics with approximation ratio 2.675 (9). Also, see Cohen-Addad et al. [2022] for a 2.406 (5.912) approximation algorithm for the combinatorial Euclidean k-medians (k--meas)).
Clustering Stability. Stability in clustering has been studied both from a practical and theoretical point of view [Ben-Hur et al., 2001, Lange et al., 2004, Von Luxburg and Ben-David, 2005, Ben-Davi et al., 2006, Rakhlin and Caponnetto, 2006, Ben-David et al., 2007, Von Luxburg et al., 2010]. In most applications, it is up to the algorithm designer to decide upon the value of k. i.e., the number of different clusters. Thus, it was proposed that a necessary condition it should satisfy is that it leads to solutions that not very far apart under resampling of the input data [Ben-Hur et al., 2001, Lange et al., 2004]. However, it was shown that this notion of stability for center- clustering is heavily based symmetries within the data which may be unrelated to clustering parameters (Ben-Davic et al., 2006). Our results differ from this line of work in that we require the output across two separate samples to be exactly the same with high probability, when the randomness is shared. Moreover, work reaffirms Ben-Dat David et al. [2006] in that their notion of stability can be perfectly attained no matter the choice of k.
Other notions of stability related to our work include robust hierarchical clustering [Balcan et al., 2014], robust online clustering [Lattanzi al., 2021]. average sensitivity [Yoshida and Ito, 2022], and differentially private (DP) clustering [Cohen et al., 2021, Ghazi et al., 2020J. The definition of replicability we use is statistical and relies on underlying data distribution while (DP) provides a worst-case combinatorial guarantee for two runs of the algorithm on neighboring datasets. Bun et al. [2023], Kalavasis et al. [2023] provide connections between DP and replicability for statistical learning problems. However, these transformations are not computationally efficient. would be interesting to come up with computationally efficient reductions between replicable and DP clustering algorithms.
Coresets for Clustering. A long line of work has focused on developing strong coresets for various flavors of centroid clustering problems. See Sohler and Woodruff [2018] for an overview of this rich line of work. The most relevant for our results include coresets for dynamic geometric streams through hierarchical grids [Frahling and Sohler, 2005] and sampling based techniques (Ben-David. 2007, Feldman and Langberg, 2011, Bachem al., 2018].
Dimensionality Reduction. Dimensionality reduction for clustering has been a popular area of study as it reduces both the time and space complexity of existing algorithms. The line of work on data-oblivious dimensionality reduction for k-means clustering was initiated by Boutsidis et al. [2010]. The goal is to approximately preserve the cost of all clustering solutions after passing the data through a dimensionality reduction map. This result was later improved and generalized to (k, p) cllusteringg [Cohen et al., 2015, Becchetti et al., 2019], culminating in the work of Makarychev
2

---

et al. [2019], whose bound on the target dimension is sharp up to factor of log 1/e. While Charikar and Waingarten [2022] overcome this factor, their result only preserves the cost across the optimal solution.
Replicability in ML. Our results extend the recently initiated line of work on designing provably replicable learning algorithms under the definition that was introduced by Impagliazzo et al. [2022]. Later, Esfandiari al. [2022] considered a natural adaption of this definition to the setting of bandits and designed replicable algorithms that have small regret. A slightly different notion of replicability in optimization was studied in Ahn al. [2022], where it is required that an optimization algorithm that uses noisy operations during its execution, e.g., noisy gradient evaluations, outputs solutions that are close when executed twice. Subsequently. Bun et al. [2023]. Kalavasis al. [2023] established strong connections between replicability and other notions of algorithmic stability. Recently. Dixon et al. [2023], Chase et [2023] proposed weaker notion of replicability where the algorithm not required to output the same solution across two executions, but its output needs to fall into small list of solutions.
2 Setting & Notation
Let X C Rd be the instance space endowed with metric K X' x X + R4 and IP be a distribution on X' which generates the i.i.d. samples that the learner observes.
For F - IRd and x E X, we overload the notation and write F(x) := argmin ser <(2; f) to be the closest point to x in F as well as 1((x, F) == k(z, F())) to be the shortest distance from x to a point in F.
We assume that X' a subset of Ba, the d-dimensional i--bal of diameter 1 centered about the origin". We also assume that is induced by some norm ||-|| on R2 that is sign-önvariant (invariant to changing the sign of a coordinate) and normalized (the canonical basis has unit length). Under these assumptions, the unit ball of is a subset of [-1,
Our setting captures a large family of norms, including the ép-norms, Top- norms (sum of largest coordinates in absolute value), and ordered norms (non-negative linear combinations of Top- norms) [Chakrabarty and Swamy, 2019]. Our results hold for more general classes of norms but for the sake of simplicity, we abide by these assumptions.
We define A == sup/x(x, y) y [0, 1)d} to be the k-diameter of the unit hypercube. Note that 1 < < d by assumption. Moreover, L& is the x-diameter of a hypercube with side length T.. For example, if ||-|| = ||2 is the Euclidean norm, then A =vd.
2.1 Clustering Methods and Generalizations
We now introduce the clustering objectives that we study in this work, which all fall in the category
of minimizing a cost function cost : + R,, where F == {F S Ba FF| = 1:). We write cost F)
to denote the objective the statistical setting and cost(F) for the combinatorial setting in order to distinguish the two.
Problem 2.1 (Statistical (k, p) Cluustterin Given i.i.d. samples from a distribution on X C Ba. minimize cost(F) E2
In other words, we need to partition the points into & clusters so that the expected distance of point to the center of its cluster, measured by k(.,)),, is minimized. This is closely related to the well-studied combinatorial variant of the (k: -llusteerii problem.
Problem 2.2 ((k, p)-CCusteriing) Given some points T1...., En X, minimize cost(F) = I 8(x;, F.P.
We note that (statistical) k-medians and (statistical) k-means special case of Problem 2.2 (Problem 2. with p 1, respectively. We also consider slight variant of the combinatorial problem, i.e., Problem 2.2, where we allow different points z; to participate with different weights w; in the objective. We refer to this problem as the weighted (k. pllusteriin problem.
OOu results can be generalized to r-balls of diameter L with an arbitrary center through translation and scaling.
3

---

We now shift our attention to the k-centers problem. Problem 2.3 (Statistical k-Centers), Given i..dd samples from a distribution on X' C B3, minimize cOst(F) maX k=2z, F).
Notice that the &oo norm the limit of the l norm as p tends to infinity, hence k-centers is, some sense, the limit of (Ae, -llluttrig k-means and k-medians in the sense that it has a min-max flavor, whereas the other two are concerned with minimizing some expected values. Due to this difference, need to treat k-centers separately from the other two problems, and we need to make some assumptions in order to be able to solve from samples (ef. Assumption F.1, Assumption F.2). We elaborate more that later.
tends to infinity. Also, notice that this problem differs from
Let us also recall the combinatorial version of k-centers. Problem 2.4 (k-Centers). Given max ( F).
points 21,-.., X, minimize cost(F)
We remark that clustering has mainly been studied from the combinatorial point of view, where the distribution the uniform distribution over some finite points and we are provided the entire distribution. The statistical clustering setting generalizes arbitrary distributions with only sample access. We emphasize that although we only have access to samples, our output should be a good solution for the entire distribution and not just the observed data.
We write FOPT to denote an optimal solution for the entire distribution and OPT := cost (OOP)). Similarly, we write FOPT denote optimal sample solution and OPT := cost(FoPt). Suppose we solve Problem 2.4 given sample of size n from Problem 2.3. Then OPT S OPT since we are optimizing over subset of the points.
Recall that a 3-approximate solution F' is one which has cost cost(F) < 30PT. Note this is with respect to the statistical version of our problems. An algorithm that outputs 3-approximate solutions known as --approximatiin algorithm. We also say that F is (3, -pppprxmiite solution cost(F) S OPT
2.1.1 Parameters p and K
Here we clarify the difference between p and k, which are two separate entities in the cost function Eg [m(x, F.) We denote by the distance metric used to measure the similarity between points. The most commonly studied and applied option is the Euclidean distance for which our algorithms are the most sample-efficient. On the other hand, p is the exponent to which we raise the distances when computing the cost of clustering. smaller choice of p puts less emphasis on points that are far away from centers and p seeks to control the average distance to the nearest center. A large choice of p puts emphasis on points that are further away from centers and as p tends to infinity, the objective is biased towards solutions minimizing the maximum distance to the nearest center. Thus we can think of k-centers as (k, p) cllusternn when p = oo. As concrete example, when x the Euclidean distance and p 5, the cost function becomes Ez || F(x)1
2.2 Replicability
Throughout this work, we study replicability as an algorithmic property using the definition of Impagliazzo al. [2022].
Definition 2.5 (Replicable Algorithm; [Impagliazzo et al., 2022]). Let (0, 1). A randomized algorithm is p-replicuble if for two sequences n i.i.d. samples X, Y generated from some distribution Ipn and a random binary string F R(X))
P PX.Y=P- (A(X;r) = A() 1-
In the above definition, we treat A as a randomized mapping to solutions of the clustering problem. Thus, even when X is fixed, A(X) should be thought of random variable, whereas A( the realization of this variable given the (fixed) X, F. We should think of f as the shared randomness
'Originally this definition was called reproducibility (Impagliazzo et al., 2022] but it was later pointed out that the correct term replicability [Ahn et al., 2022].

---

between the two executions. In practice, it can be implemented as a shared random seed. We underline that sharing the randomness across executions is crucial for the development of our algorithms. We also note that by doing that we couple the two random variables A(X) AYY). whose realization depends on ~ R()). Thus, if their realizations are equal with high probability under this coupling. it means that the distributions of ,, A(Y) are statistically close. This connection is discussed further in Kalavasis et al. [2023].
In the context of a clustering algorithm A, we interpret the output A(X;F) as clustering function f: X + [k] which partitions the support of IP. The definition of p-replicability demands that f is the same with probability at least 1 - P across two executions. We note that in the case of centroid- bassed clustering such as k-medians and k-means, the induced partition is a function of the centers and thus it is sufficient to output the exact same centers with probability 1 - across two executions. However, we also allow for algorithms that create partitions implicitly without computing their centers explicitly.
Our goal develop replicable clustering algorithms for k-medians, k-means, and k-centers, which necessitates that the centers we choose are arbitrary points within R and not only points among the samples. We underline that as in the case of differential privacy, it trivial design algorithms that satisfy the replicability property, e.g. we can let A be the constant mapping. The catch is that these algorithms do not achieve any utility. In this work, we are interested in designing replicable clustering algorithms whose utility is competitive with their non-replicable counterparts.
Main Results
In this section, we informally state results for replicable statistical k-medians ((k, l-clusttring k-means ((k, 2)-clustering). and k-centers under general distances. Unfortunately, generality comes at the cost of exponential dependency the dimension d. We also state results for replicable statistical k-medians and k-means specifically under the Euclidean distance, which has a polynomial dependency on d. Two key ingredients is the uniform convergence of (k.p))clustering costs (cf. Theorem C.9) as well as a data-oblivious dimensionality reduction technique for (k. p) -)lusttriiing in the distributional setting (ef. Theorem 10) These results may be of independent interest.
We emphasize that the Euclidean k-median and k-means are the most studied and applied flavors of clustering, thus the sample complexity for the general case and the restriction to p 1, 2 does not diminish the applicability our approach.
The main bottleneck in reducing the sample complexity for general norms is the lack of data- oblivious dimensionality reduction scheme. This bottleneck is not unique to replicability and such a scheme for general norms would be immediately useful for many distance-based problems including clustering. It may be possible to extend our results to general (k:, p)-clustering beyond p = 1,2. The main challenge is to develop an approximate triangle inequality for p-th powers of norms. Again, this limitation is not due to replicability but rather the technique of hicrarchical grids. a limitation shared by Frahling and Sohler [2005].
Before stating our results, we reiterate that the support of our domain X' is a subset of the unit-diameter R-ball Bg. In particular, we have that OPT < 1.
Theorem 3.1 (Informal). Let E, E 1). Given black-box access to a B 3-approximation oracle for weighted k-medians, respectively weighted k-means (cf. Problem 2.2), there is p-replicable algorithm for statistical k-medians, respectively k-means (cf. Problem 2. I),,, such that with probability at least 0.99, it outputs (1 + e) e)88-aprrximmatin. Moreover, the algorithm has sample complexity
ò
(pory
( poly ( k (POTT)
(?²)
2A
O(d)
When we are working in Euclidean space, we can get improved results for these problems.
Theorem 3.2 (Informal). Let p E 1). Suppose we are provided with black-box access to 3-aaproximation oracle for weighted Fuelidean k-medians (k-means)) Then there a p-replicable algorithm that partitions the input space with probability at least 0.99, the cost of the partition is
5

---

at most ((3 Moreover. the algorithm has sample complexity
δ poly (OPT)
d
h: (x)
Oflog (o))
We underline that in this setting we compute an implicit solution to Problem 2.1, since we do not output h centers. Instead, we output a function f that takes input a point z E X and outputs the label of the cluster it belongs to in polynomial time. The replicability guarantee states that, with probability 1 p. the function will be the same across two executions.
The combinatorial k-medians (k-means) problem where the centers are restricted be points of the imput is well-studied problem from the perspective of polynomial-ting -imeill-imme constant-facto approximation algorithms. See Byrka al. [2017], Ahmadian al. [2019] for the current best polynomial-tine approximation algorithms for combinatorial k-medians (k-means) in general metrics with approximation ratio 2.675 (9). Also, see Cohen- Addad et al. [2022] for 2.406 (5.912) approximation algorithm for the combinatorial Euclidean k-medians (k-means).
As we alluded before, in order to solve k-centers from samples, we need make an additional assumption. Essentially, Assumption F:2 states that there is a (3, -ppaproxxit solution F. such that, with some constant probability, e.g. 0.99, when we draw n samples from IP we will observe least one sample from each cluster of F.
Theorem 3.3 (Informal). Let E (0, 1). Given black-box access to a (3. B) -pppproxxiimaoonn oracle for k-centers (cf. Problem 2.4) and under Assumption F22, there is a p-replicable algorithm for statistical k-cceters (cf. Problem 2.3), that outputs a (O(B + 3), O(B + B + (B + 3 1)c)
solution with probability at least 0.99. Moreover, it has sample complexity
n 2 (1/2)
p²q2
Recall that there is simple greedy 2-approximation for the sample k-center problem whose approxi- mation ratio cannot be improved unless P = NP [Hochbaum and Shmoys, 1985].
We defer the discussion around k-centers to Appendix F, In particular, see Theorem F.8 in Appendix F for the formal statement of Theorem 3.3.
Overview of (ki
In this section, present approach to the (k, plcusterrin problem. First, replicably approximate the distribution with a finite set of points by extending the approach of Frahling and Sohler [2005] to the distributional setting. Then, solve the combinatorial (k, p)clusterngg problem on this coreset using an approximation oracle black-box manner. the following subsections, give more detailed overview for each step of our approach. For the full proofs and technical details, we kindly refer the reader to Appendix D.4 D.6. In summary:
1) Replicably build a variant of a quad tree [Finkel and Bentley, 1974] (cf. Section 4.2).
2) Replicably produce a weighted coreset using the quad tree (cf. Section 4. 4.
3) Apply the optimization oracle for the combinatorial problem on the coreset.
For general norms, this approach leads an exponential dependence d. However, are able to handle the case of Euclidean distances by extending existing dimensionality reduction techniques for sample Buclidean (k, -clustering [Makarychev et al., 2019] the distributional case (ef. Section 5). Thus, for the widely used Euclidean norm, our algorithm has poly(d (d) sample complexity.
4.1 Coresets
Definition 4.1 ((Strong) Coresets). For a distribution P with support X S B, and (0, a (strong) s-coreset for X' is a distribution IP' on X' C Ba which satisfies
(1 1 E) Ex-ph(s, FYP & EerF [M(2', F) < (1 + Ex-Pn(I,FF) for every set of centers F C Bd, Fl k.

---

Essentially, coresets help us approximate the true cost on the distribution P by considering another distribution IP' whose support X" can be arbitrarily smaller than the support of P..
Inspired by Frahling and Sohler [2005], the idea to replicably consolidate our distribution P through some mapping R: X' X whose image has small cardinality [R(X)] << that for any set of centers F.
(1 (1-E) (),,,, S (R( S (1+ E),
where e E 1) is some enror parameter. In other words, (PR, R(x)) is e-coreset. Note that given the function R, we can replicably estimate the probability mass at each point in R(x)) and then apply weighted (k, p) -clustering algorithm.
4.2 Replicable Quad Tree
We now explain how to replicably obtain the mapping R X X' by building upon the work of Frahling and Sohler [2005]. The pseudocode of the approach is provided in Algorithm 4.1. While Frahling and Sohler [2005] present their algorithm using hierarchical grids, we take an alternative presentation using the quad tree [Finkel and Bentley, 1974], which could be of independent interest.
First, we recall the construction of standard quad tree in dimension d. Suppose we have set of points in |-1/2, 3/2]d. The quad tree a tree whose nodes represent hypercubes containing points and can be built recursively as follows: The root represents the cell [-1/2, 1/2/ and contains all points. If node contains more than one point, we split the cell it represents into 2d disjoint, equally sized cells. For each non- non-empty cell, we add it as a child of the current node and recurse into it. The recursion stops when node contains only point. In the distributional setting, the stopping criterion is either when the diameter of a node is less than some length or when the node contains less than some mass, where both quantities are function of the depth of the node. See Algorithm 4.1.
quad tree implicitly defines a function R X' + X' as follows. Given a point x: E X' and the root node of our tree, while the current node has a child, go the child containing if such child exists, otherwise, go to any child. At a leaf node, output the center of the cell the leaf represents. Intuitively, the quad tree consolidates regions of the sample space into single points. The construction can be made replicable since the decision to continue the recursion or not is the only statistical operation and is essentially a heavy- operations which can be performed a replicable fasion.
Let G, denote the union of all 2nd possible cells at the i-th level. We write P. to denote the discretized
distribution to Gs. In other words, P. (g.) is the restriction of P the smallest o-algebra containing Gj. Moreover, we write A to denote a replicable estimate of OPT with relative error e, say +=) OPT < A S (1 + OPT for some absolute constant / 2 1. We demonstrate how to obtain such a replicable estimate in Appendix D.5.
Algorithm 4.1 Replicable Quad Tree
I: rQuadTree disttributinn IP, accuracy E, exponent p, replicability p. confidence 8):
2: Init the node on the first level Z [(0] + [1-1/2, 1/2]d
3: for depth i 1; Zii - 1] + Ø AND (2 - A)P > =A/s; i + i+1 do
4: (IP: is the discretized distribution over 2 cells on the i-th layer]
5: (y a parameter be determined later.)
6: is an upper bound on the number of layers.}
7: Compute H = HHHavyHitters
P.,
A v p 6
2- 21 t
8:
for node Z Zli 1] do
for heavy hitter cells Hi E H such that Hl C Z do 10: children(2 () + children(2) {H}
9:
11:
Z|i] + Z J (H}) 12: end for
13:
end for
14:
end for
15:
Output root node.
Our technique differs from that of Frahling and Sohler [2005] in least three ways. Firstly, they performed their analysis for finite, uniform distributions with access to the entire distribution, while
7

---

our results hold assuming only sample access to general bounded distribution'. Secondly, Frahling and Sohler [2005] bound the number of layers as a function of the cardinality of the support. For us, this necessitates the extra termination condition when the side lengths of our grids fall below a fraction of A as our distribution may have infinite support. Finally, Frahling and Sohler [2005] estimate OPT by enumerating powers of 2. This suffices for their setting since their distributions are discrete and bounded. However, we require a more nuanced approach (cf. Appendix D.5) as we do not have a lower bound for OPT. We tackle this by showing uniform convergence of the clustering solution costs, which we establish via metric entropy and Rademacher complexity (ef. Appendix C).
4.3 Putting it Together
Once we have produced the function R implicitly through a quad tree, there is still the matter of extracting a replicable solution from a finite distribution. We can accomplish this by replicably estimating the probability mass at each point of R(x²) and solving an instance of the weighted sample k-medians (k-means). This leads to the following results. For details, see Appendix D.4 D.6.
Theorem 4.2 (Theorem 3.1; Formal). Let E. E (0, 1) and ô E (0, p/s). Given black-box access to 3-approximation oracle for weighted k-medians (cf. Problem 2.2), there is a p-replicable algorithm for statistical k-medians (cf. Problem 2. such that, with probability at least 1 5, outputs (1 + =))Bapproximmation Moreover, it has sample complexity
δ (COA d2d22
3d43
log !)
1
OPT12 + p2g3d+5 3
Theorem 4.3 (Theorem 3.1; Formal). Given black-box access a 3-approximation oracle for weighted k-means (cf. Problem 2.2), there is p-replicable algorithm for statistical k-means (cf. Problem 2.1) l) such that, with probability at least 1 - 5, it replicably outputs a + ( aappppppoinntinonn Moreover, it has sample complexity
((
33239d 19239d3300 3d46
log 1
E 12 OPT +
p2g6d+8 .OPT3
5 The Euclidean Metric, Dimensionality Reduction, and (k, p)-Clusteri
In this section, we focus on the Euclidean metric x(y,z) := lyy 2/12 and show how the sample complexity for Euclidean (k, p)clusterig can be made polynomial in the ambient dimension d. Note that results for dimensionality reduction exist for the combinatorial Euclidean (kt plccustteingg problem [Charikar and Waingarten, 2022, Makarychev et al., 2019]. However, these do not extend trivially to the distributional case. We remark that we require our dimensionality reduction maps to be dala-oblivioos since we are constrained by replicability requirements.
The cornerstone result in dimensionality reduction for Euclidean distance the Johnson-LLnndentrene lemma (cf. Theorem E.I), which states that there is a distribution over linear maps Td,m from R + R. that approximately preserves the norm of any E Rd with constant probability for some target dimension m. Makarychev et al. [2019], the authors prove Theorem E.4, which roughly states that it suffices to take m = Ö(F / log()/s) in order to preserve (k, pclusterrn costs when the domain is finite. Firstly, we extend Theorem E.4 to the distributional setting by implicitly approximating the distribution with a weighted s-net. Then, we implicitly map the s-net onto the low-dimensional space and solve the clustering problem there. An important complication we need to overcome is that this mapping preserves the costs that correspond to partitions" of the data and not arbitrary solutions. Because of that, it is not clear how the dow-dimensional space to the original space. Thus, instead of outputting points that correspond algorithm outputs clustering function X + [k], which takes as
"lift" the solution from
to the centers of the clusters, input a point x E X' and returns the label of the cluster it belongs to. The replicability guarantees of the algorithm state that, with probability 1 p. it will output the same function across In Appendix F.4, we describe this function. We emphasize that for each & E X, the running time of f is polynomial in d, p, 2/s, log(1/)) Essentially, this function maps x onto the low-dimensional
executions.
5 Our results also generalize to unbounded distributions with sufficiently small tails 'Roughly speaking, solution corresponds to partition when the center of each cluster is its center of mass.
8

---

space using the same projection map our algorithm and then finds the nearest center of ((x) in the cow-dimensional space. For full details, we refer the reader Appendix E.3. We are now ready to state the result formally.
Theorem 5.1 (Theorem 3.2; Formal), s, p E (0, 1) and å E (0)) Given 8-approximation oracle for weighted Euclideun k-medians (k-means), there a p-replicable algorithm that outputs clustering function such that with probability at least 1 - 8, the cost of the partition is at most + E1B OPT. Moreover, the algorithm has sample complexity
Ô 0poly poly (POPT) ( (2
kd
O(m) 1 log
8)
where = 0 (+2 log *
6 Running Time for (k, (k,p)-Clusttring
All of our algorithms terminate in O(poly(n)) time where n denotes the sample complexity. See Table A.1 for more detailed time complexity of each stage of our approach. Moreover, we make (log("/(s OPT))) calls to the 3-approximation oracle for combinatorial (k: peclustrin problem within our OPT estimation subroutine and then one more call to the oracle in order to output solution on the coreset.
Replicable k-Centers
Due to space limitations, we briefly sketch our approach for the k-centers problem and kindly refer the reader Appendix F. As explained before, the assumptions make in this setting state that there exists some "good" solution, so that when we draw n i.i.d. samples from P we observe at least one sample from each cluster, with constant probability. We first take fixed grid of side e order to cover the unit-diameter ball. Then, we sample sufficiently many points from IP. Subsequently, we "round" all the points of the sample to the centers of the cells of the grid that they fall into and estimate the probability mass of every cell. In order to ensure replicability, we take a random threshold from a predefined interval and discard the points from all the cells whose mass falls below the threshold. Finally. we call the approximation oracle using the points that remain. Unlike the (k: -ellustrin problem (cf. Problem 2.1), to the best of our knowledge, there does not exist any dimensionality reduction techniques that apply to the k-centers problem. The main result is formally stated in Theorem F.8.
Experiments
We now provide a practical demonstration" of the replicability of our approach on synthetic data in 2D. In Figure 1. we leverage the sklearn [Pedregosa et al., 201 implementation of the popular k-means++ algorithm for & = 3 and compare the output across two executions on the two moons distribution. In the first experiment, we do not perform any preprocessing and run k-means++ as is, resulting in different centers across two executions. In the second experiment, we compute replicable coreset for the moons distribution before running k-means++ on the coreset. This leads to the same centers being outputted across the executions. Note that the computation for the coreset performed independently for each execution, albeit with shared random seed for the internal randomness. See also Figure 8.2 for the results of a similar experiment on a mixture of truncated Gaussian distributions.

---

Replicable K-Means on Two Moons Distribution
Execution
Execution 2
1.00
1.00
0.75
0.75
0.2
Nou Raplicable Centes
0.00
0.00
Repáiceble Coneers
Conout
0.5
0.25
0.50
0.50
-0.75
-0.50
-0.25
0.00
0.50
-1.00
-0.75
-0.50
-0.25
0.35
0.75
Figure 8.1: The results running vanilla vs replicable k-Means++ on the two moons distribution for k= == 3.
Replicable K-Means
Trumeated
Exccution
Execution 2
1.00
1.00
0.25
0.2
Sumples
Replicable Centers
0.00
0.00
Repáccbble Cemeer
0.25
0.25
-1.00
-1.00
1.00
0.75
-0.50
0.25
0.00
0.25
0.30
0.75
1.00
1.00
0.75
00.50
-0.55
0.00
0.25
0.50
0.75
1.00
Figure 8.2: The results of running vanilla vs replicable k-Means++ on mixture of truncated Gaussians distributions for & = 3.
Conclusion & Future Work
In this work, we designed replicable algorithms with strong performance guarantees using black-box access to approximation oracles for their combinatorial counterparts. There are many follow-up research directions that this work can lead to. For instance, our coreset algorithm adapts the coreset algorithm of Frahling and Sohler [2005] by viewing their algorithm as a series of heavy hitter estimations that can be made replicable. it may be possible to interpret more recent approaches for coreset estimation as series statistical operations to made replicable in order to get replicable algorithms in the statistical (k: p.clustriigg setting [Hu et al., 2018]. It would also be interesting to examine the sensitivity of (replicable) clustering algorithms to the choice parameters such as the choice of exponent in the cost function or the measure of similarity of the data. Another relevant direction is to explore sample complexity lower ounds for statistical clustering, where little is known in the non-replicable setting.
Acknowledgments and Disclosure of Funding
Amin Karbasi acknowledges funding direct support of this work from NSF (IS-1845032), ONR NN00014-19-1-2466) and the AI Institute for Learning-Enabled Optimization at Scale (TILOSS). Grigoris Velegkas is supported by TILOS, the Onassis Foundation, and the Bodossaki Foundation. Felix Zhou is supported by TILOS.
10

---

References
Sara Ahmadian, Ashkan Norouzi-Fard, Ola Svensson, and Justin Ward. Better guarantees for means and euclidean k-median by primal-dual algorithms. SIAM Journal on Computing, 49(4): FOCS17-97, 2019.
Kwangjun Ahn, Prateek Jain, Ziwei Ji, Satyen Kale, Praneeth Netrapalli, and Gil I Shamir. Repro- ducibility in optimization: Theoretical framework and limits. arXiv preprint arXiv:2202.04598. 2022.
Michael R Anderberg. Cluster analysis for applications: probability and mathematical statistics: series of monographs and textbooks, volume 19. Academic press, 2014.
Olivier Bachem, Mario Lucic, and Andreas Krause. Scalable k-means clustering via lightweight coresets. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1119-1127, 2018.
Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 2016.
Maria-Florina Balean, Yingyu Liang, and Pramod Gupta. Robust hierarchical clustering. The Journal of Machine Learning Research, (((1:3381-3877) 2014.
Luca Becchetti, Mare Bury, Vincent Cohen-Addad Fabrizio Grandoni, and Chris Schwiegelshohn. Oblivious dimension reduction for k-means: beyond subspaces and the cchnson-lindensiin lemma. In Proceedings of the 51st annual ACM SIGACT symposium on theory of computing, pages 1039-1050, 2019.
Shai Ben-David. A framework for statistical clustering with constant time approximation algorithms for k-median and k-means clustering. Machine Learning, 66(2):243-257, 2007.
Shai Ben-David, Ulrike Von Luxburg, and Dávid Pál. A sober look at clustering stability. In International conference computational learning theory, pages 5-19. Springer, 2006.
Shai Ben-David, Dávid Pál, and Hans Ulrich Simon. Stability of k-means clustering. In International conference on computational learning theory, pages 20-34. Springer, 2007.
Asa Ben-Hur, Andre Elisseeff, and Isabelle Guyon. A stability based method for discovering structure in clustered data. In Biocomputing 2002, pages 6-17. World Scientific, 2001.
Christos Boutsidis, Anastasios Zouzias, and Petros Drineas. Random projections for k-means clustering. Advances in neural information processing systems, 23, 2010.
Mark Bun, Marco Gaboardi. Max Hopkins, Russell Impagliazzo, Rex Lei, Toniann Pitassi, Jessica Sorrell, and Satchit Sivakumar. Stability is stable: Connections between replicability, privacy, and adaptive generalization. arXiv preprint arXiv:2303. /292/, 2023.
Jaroslaw Byrka, Thomas Pensyl. Bartosz Rybicki, Aravind Srinivasan, and Khoa Trinh. An improved approximation for k-median and positive correlation in budgeted optimization. ACM Transactions Algorithms (TALG), 13(2):1-31, 2017.
Bernd Carl and Irmtraud Stephani. Entropy, compactness and the approximation of operators. Cambridge University Press, 1990.
Deeparnab Chakrabarty and Chaitanya Swamy, Approximation algorithms for minimum norm and ordered optimization problems. In Proceedings of the 51st Anmual ACM SIGACT Symposium on Theory of Computing, pages 126-137, 2019.
Moses Charikar and Erik Waingarten. The minnsonninndesstrauss lemma for clustering and subspace approximation: From coresets to dimension reduction. arXiv preprint urXiv:2205.0037 2022.
Zachary Chase, Shay Moran, and Amir Ychudayoff. Replicability and stability in learning. arXiv preprint 23X4072304.03757 2023.
Edith Cohen, Haim Kaplan, Yishay Mansour, Uri Stemmer, and Eliad Tsfadia. Differentially-private clustering of easy instances. In International Conference on Machine Learning, pages 2049-2059.
PMLR, 2021.
11

---

Michael B Cohen, Sam Elder, Cameron Musco, Christopher Musco, and Madalina Persu. Dimen- sionality reduction for k-means clustering and low rank upproximation. In Proceedings of the Jorty-seventh annual ACM symposium on Theory of computing, pages 163-172, 2015.
Vincent Cohen-Addad, Hossein Esfandiari, Vahab Mirrokni, and Shyam Narayanan. Improved approximations for euclidean k-means and k --median, via nested quasi-indeppndent sets. arXiv preprint arXiv:2204.00 2444:2 2022.
Ronald A Ve and George G Lorentz. Constructive approximation, volume 303. Springer Science & Business Media, 1993.
Peter Dixon, A Pavan, Jason Vander Woude, and NV Vinodchandran. List and certificate complexities in replicable learning. arXiv preprint arXiv: arXiv:2304.0022400 2023.
Hossein Esfandiari, Alkis Kalavasis, Amin Karbasi, Andreas Krause, Vahab Mirrokni, and Grigoris Velegkas. Reproducible bandits. arXiv preprint arXiv:2210.01888, 2022.
Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Proceedings of the forty-third annual ACM symposium on Theory of computing, pages 569-578, 2011.
Raphael A Finkel and Jon Louis Bentley, Quad trees a data structure for retrieval on composite keys. Acta informatica, 4:1-9, 1974.
Gereon Frahling and Christian Sohler. Coresets in dynamic geometric data streams. In Proceedings of the thirty-seventh annual ACM symposium on Theory of computing, pages 209-217, 2005.
Badih Ghazi, Ravi Kumar, and Pasin Manurangsi. Differentially private clustering: Tight approxima- tion ratios. Advances in Neural Information Processing Systems, 33:4040-4054, 2020.
Peter E Hart, David G Stork, and Richard O Duda. Pattern classification. Wiley Hoboken, 2000.
Dorit S Hochbaum and David B Shmoys. A best possible heuristic for the k-center problem. Mathematics of operations research, 10(2):180-184, 1985.
Wei Hu, Zhao Song, Lin F Yang, and Peilin Zhong. Nearly optimal dynamic k-means clustering for high-dimensional data. arXiv preprint arXiv:1802.00455 2018.
Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell, Reproducibility in learning. arXiv preprint
2022.
Anil K Jain and Richard C Dubes. Algorithms for clustering data. Prentice-Hall, Inc., 1988.
Svante Janson. Tail bounds for sums of geometric and exponential variables. Statistics & Probability Letters, 135:1-6, 2018.
William B Johnson. Extensions of lipschitz mappings into a hilbert space. Contemp. Math., 26: 189-206, 1984.
Alkis Kalavasis, Amin Karbasi, Shay Moran, and Grigoris Velegkas. Statistical indistinguishabiiity of learning algorithms. arXiv preprint arXiv:2305. 14311, 2023.
Michael Kearns, Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM), 45(6):983-1006, 1998.
Jon Kleinberg. An impossibility theorem for clustering. Advances in neural information processing systems, 15, 2002.
Tilman Lange, Volker Roth, Mikio Braun, and Joachim M Buhmann. Stability-based validation of clustering solutions. Neural computation, 1296:112993333, 2004.
Silvio Lattanzi, Benjamin Moseley, Sergei Vassilvitskii, Yuyan Wang, and Rudy Zhou. Robust online correlation clustering. Advances in Neural Information Processing Systems, 34:4688-466 2021.
Michel Ledoux and Michel Talagrand. Probability Banach Spuces: isoperimetry and processes, volume 23, Springer Science & Business Media, 1991.
12

---

Konstantin Makarychev, Yury Makarychev, and Ilya Razenshteyn. Performance of johnson- lindenstrauss transform for k-means and k-medians clustering. In Proceedings of the 51st Anmual ACM SIGACT Symposium on Theory of Computing, pages 1027-1038, 2019.
Vitali D Milman and Gideon Schechtman. Asymptotic Theory of Finite Dimensional Normed Spaced, volume 1200. Springer Berlin, 1986.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Couraapaa,, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Joelle Pineau, Koustuv Sinha, Genevieve Fried, Rosemary Nan Ke, and Hugo Larochelle. Iclr reproducibility challenge 2019. ReScience C, 5(2):5, 2019.
Joelle Pineau, Philippe Vincent-Lamaree Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research: report from the neurips 2019 reproducibility program. Journal of Machine Learning Research, 22, 2021.
Gilles Pisier. The volume of convex bodies and Banach space geometry, volume 94, Cambridge University Press, 1999.
Alexander Rakhlin and Andrea Caponnetto. Stability of k-means clustering. Advances in neural information processing systems, 19, 2006.
Christian Sohler and David P Woodruff. Strong coresets for k-median and subspace approximation: Goodbye dimension. 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 802-813. IEEE, 2018.
AW van der Vaart and Jon A Wellner. Weak convergence and empirical processes with applications to statistics. Journal of the Royal Statistical Society-Series Statistics in Society, 160()))59--608, 1997.
Ulrike Von Luxburg and Shai Ben-David. Towards statistical theory of clustering. In Pascal workshop on statistics and optimization of clustering, pages 20-26. London, UK, 2005.
Ulrike Von Luxburg et al. Clustering stability: an overview. Foundations and Trends@ in Machine Learning, 2(3):235-274, 2010.
Martin J Wainwright. High--imenssioal statistics: non-asymptotic viewpoint, volume 48. Cam- bridge University Press, 2019.
Yuichi Yoshida and Shinji Ito. Average sensitivity of euclidean k-clustering. In Advances in Neural Information Processing Systems, 2022.
13

---

A Tables
We present the running times required for our sub-routines in order to ensure each stage p-replicable and succeeds with probability at least 0.99, The overall output is constant ratio approximation for Euclidean m-medians (k-means).
Table A.I: Running Time Overview for Euclidean k-Medians (k-Means)
ALGORITHM
RUNNING TIME
OPT ESTIMATION DIMENSIONALITY REDUCTION
O(poly(kd/ort.) o))
((poly(*/orT) (*/p) Ollog (og((//))) Ö (ppoly(//or) (*/p) Ollog (o/()))))) (ppoly(//or)) (4))) Ollog log (og()/o)))
CORESET
PROBABILITY MASS ESTIMATION
B Useful Facts
Proposition B.1 (Bretagnolle-Huber-Caaa Inequality; (Vaart and Wellner, 1997]). Suppose the
random vector (...)..., Z(N) multinomially distributed with parameters (p(1) ,..,,,,..... p(N) and
n. Let ((5) = z(1, Then
N pt
P
p001 2= 22" exp (-2E'n)
In particular, for any e, p E (0, 1), sampling
In Nin2
n
points from finite distribution implies that Σー=1 1710 p00 < 2s with probubility least p.
Remark B.2 (Weighted /-Means/Medians)). We remark that if we have access to a --approximation oracle for unweighted k-mmans//mddans, we can implement weighted one by considering multiple copies of the points. particular, in our applications, we get weights that are polynomials in the parameters of concern so this will not affect the stated runtime guarantees.
C Uniform Convergence of (k. (k Cluustern Costs
We would like to estimate OPT for statistical k-medians and statistical k-means by solving the combinatorial problem sufficiently large sample size. However, while the convergence of cost for a particular set of centers is guaranteed by standard arguments, e.g. Chernoff bounds, we require the stronger statement that convergence holds simultaneously for all possible choices of centers.
Similar results can found [Ben-David, 2007], with the limitation that centers are either chosen from the sample points, or chosen as centers of mass of the clusters they induce. While this suffices to achieve constant ratio approximations, we would like to choose our centers anywhere Bg. One reason for doing so is for replicability independent executions so we need to output centers that are not overly reliant on its specific input. Another is for dimensionality reduction, which we will see later.
have control over the location of samples from two
C.1 History of Uniform Laws of Large Numbers
Uniform laws of large numbers generalize convergence results for finite number of random variables to possibly uncountable classes of random variables. The Rademacher complexity and related Gaussian complexity are canonical techniques in developing these laws and also have lengthy history in the study of Banach spaces using probabilistic methods [Pisier, 1999, Milman and Schechtman,
14

---

1986, Ledoux and Talagrand, 1991]. Metric entropy, along with related notions of expressivity of various function classes, can be used control the Rademacher complexity and are also central objects of study in the field of approximation theory Voee and Lorentz, 1993, Carl and Stephani, 1990].
C.2 Rademacher Complexity
Let _0 denote a class of functions from IRd - R. For any fixed collection of n points 27 (21,,..,; We write
[(x)) (J(n)).. ((rn)) f })
i.e, the restriction of &0r onto the sample. Let 8 E 13" be a Rademacher random vector. That is,
& takes on values -1, 1 with equal probability independently of other components. Recall that the Rademacher complexity of f is given by
sup (s, S(XT)) [
Rn ($) = Ex,s
Theorem C.1 ([Wainwright, 2019]). For any b-uniformly bounded class of functions F integer n 2 1, and error e 2 0,
((X,) - E[((X) & (5) +
sup
fE
i=1
with probability at least exp
ng²
262
In particular, long (x o(1), have uniform convergence of the sample mean.
C.3 Metric Entropy
We write B(x,, r; (u) to denote the ball of radius r about point with respect to metric p. Recall that an e-cover of a set T with respect to metric is subset f,.... ON C T such that
......00y
TE U B(O,.;
iE[N]
The covering number N(e; T,,) cardinality of the smallest e-cover.
Proposition C.2 ([Wainwright, 2019]). Fix (0, Let ((R;) [|-]]) denote the d-dimensional ball of radius R with respect to ||-||. Then
2R
(R
N(e, B(RA & 1
<
We say collection of zero-mean random variables (Y respect to metric y on T if
T} is sub-Gaussine process with
\²p²(0000)
Eexp PNYY Yor)] s exp
2
for all 0,0' E T and À E R. Proposition C.3. The canonical Rademacher process
Yo = (s, 0)
is a zero-mean sub-Gaussian process with respect to the Euclidean norm on T C IR
Proof. Recall that a Rademacher variable sub-Gaussian with parameter Moreover, the sum of sub-Gaussian variables is sub-Gaussian with parameter equal to the Euclidean norm of the
parameters. It follows that Yo Yor = {8, 0 0') Gaussin with parameter ||0 0'|lz- The result follows.
D
15

---

Theorem C.4 (One-Step Discretization: [Wainwright, 2019]). Ler Yo. f E T be a zero-mean sub- Gaussian process with respect the metric μ.. Fix any E E [0, D] where D diam p)) such that (e;T,, p) 2 10. Then
sup Y OET
< E
sup Yo 0,0'E
Yg
2F
sup (T
Yo You +4Dylog N(e
In general, it may be possible to improve the bound in Theorem C.4 using more sophisticated techniques such as Dudley's entropy integral bound [Wainwright, 2019]. However, the simple inequality from Theorem C.4 suffices for our setting.
Corollary C.S. Fix any 10, D) where D = diam((; p)) such that Ne) 2 10. The canonical Rademacher process Yo (s, () for E T Rn satisfies
E
sup Yo
S 2 + 4D T(H;1,
Proof. By Proposition C.3, Ya zero-mean sub-Gaussian process and apply Theorem C.4 to conclude that
E
sup 0 11 Se
Yg
2
sup (u,
[Ivll
< 2B |||1||2 [10112]
< 22Vn.
Corollary C.5 gives way to control the Rademacher complexity of function class whose co-domain is well-behaved. We make this notion rigorous in the next section.
C.4 Uniform Convergence of (ks, p)-Cltsteringg Cost
For fixed set of centers wy Rdd + be given by
(z) == n(x, F)".
Define F= {F C Ba [F] = k}. We take our function class to be
= f F)
Let p: x F + be given by
MA,B)
x( b). aEA,bEB
max
Proposition C.6. μ is a metric on F.
Proof. It is clear that is symmetric and positive definite. We need only show that the triangle inequality holds. Fix A, B, F and suppose a E A, E such that k(a, c) p(A, C). Then
M(A,C) x(a,c) c)
< x(a, b) + k(b,c)
Vb E B
< u(A, B) P(B.
as desired.
Proposition C.7. F is p-Lipschitz parameterized with respect the metric p.
16

---

Proof. We have
(mb(2) xx)( In(a, F(x)) - F(),))))))
[n(a, +(F()) ((C =x(F(2)),
F1F,
Now, the function ((x) [0, 1] + R given by a + x p-Lipschitz by the mean value theorem:
[g(z) |(v)و v sup
(E) ir yl
vl-
cE(0.1]
The result follows by the fact that the composition of Lipschitz functions is Lipschitz with constant equal to the product of constants from the composed functions.
Theorem C.8. For any & E [0, vn) such that N(; (1+)) 2 10.
2e + 4
kd log
3ymp
Rn (3) S
o(1).
Proof. Remark that (x+) /np-Lipschitz with respect to the metric μ.. Indeed,
Hep <x:((x)) - nd (xx))? S np ('F,
Now, D diam (aaannF(*))))) (x+); 11112) n. We apply Corollary C.5 with T F(o)) to see that
R.
E
sup Yo
2:
log (xT))
<
+ +4V
Vn
Now, since (*+))) is Vnp-Lipschitz, a 7np -over for F yields an e-cover of $ ((+). Hence
N(e; (1))111)) (pp
F.
Note that the cross product of k e-covers of B is e-cover of F. Hence
E N
F
8
Ba, K6
Vnp
Vnp
3 3/ppp
kd
Proposition C.2
Substituting this bound on the covering number of F(x)) concludes the proof.
Note that the Lipschitz property was crucial to ensure that the exponent in the covering number does not contain Theorem C.9. Fix e, 8 E 1). Then with
k. 22222 c4
log p
8
i.i.d. samples from
cost cost(F) Se
with probability at least 1 8 for any set of centers F C Ba.
17

---

Proof. Choose E = = Vn. By Theorem C.8,
(5))
2.3
kd log
+
+1/
Vn
71
6
kid log Vnp
+
+4V
vn
72
6
kdlog
log vn
+
+
+
va+b v vb
Vn
n
n
6
kdd log
kd
5
4v/
log v S vn
Vn
71
n2
Fix some Er E (0, 1). We have
S1
36
Similarly, we have
kd logg
4
S E1
16kdlogp
2
Finally,
kd
4
E1
25
et
By taking the maximum of the three lower bounds, we conclude that
256 d2
2 logg
Rr( < 3
Fix 8 e (0. 1). Observe that & is 1-uniformly bounded. Thus by Theorem C.I, we require
2 log
1 256 d2
256h d2
max
8'
ot
log
<
c
log
samples in order guarantee that
sup N E
K (X,) (X)
-
2 +
71 i=1
S 781
with probability at least Choosing E & concludes the proof.
8.
D (k, (CussCCln
In this section, we provide the full proofs for Theorem 4.2 and Theorem 4.3, which state the guarantees for the replicable k-medians and k-means algorithms, respectively. First, describe
18

---

useful subroutine for replicable heavy hitters estimation in Appendix D.2. This subroutine is crucial the replicable coreset algorithm (cf. Algorithm D.3), which we analyze in Appendix D.3. Once we have coreset, it remains to solve the statistical (k, --llutterin problem on finite distribution. We describe a replicable algorithm for this in Appendix D.4. Our coreset algorithm assumes the knowledge of some constant ratio estimate of OPT. In Appendix D.5, we show how to output such an estimate replicably. Finally, we summarize our findings in Appendix D.6.
D.1 Warm-Up: Replicable SQ Oracle and s-Covers
In order to give some intuition to the reader, first show how developed impagliazzo et al. [2022] in order to derive some results the setting we are studying. We first need to define the statistical query model that was introduced in Kearns [1998]
subroutine that was
Definition D.1 (Statistical Query Oracle; [Kearns, 1998]). Let D be distribution over the domain X and Ø : X + I be a statistical query with true value
lim ((XI. XT,) E. n=000
Here X; ri..d. DD and the convergence is understood in probability distribution. Le 6 (0,1)? statistical query (SQ) oracle outputs value e such that [u o*1 I with probability at least
The simplest example of statistical query is the sample mean
6(X
Xn)
X
The idea is that by using sufficiently large number of samples, an SQ oracle returns an estimate of the expected value of a statistical query whose range is bounded. Impagliazzo al. [2022] provide replicable implementation of an SQ oracle with a mild blow-up in the sample complexity which we state below.
Theorem D.2 (Replicable SQ Oracle; [Impagliazzo et al., 2022)). Let E, E (0, and 8 E (0, p/3). Suppose & is statistical query with co-domain [0, 1]. There is p-replicable SQ oracle to estimate its true value with tolerance and failure rate Moreover, the oracle has sample complexity
ò ( log ;))
The interpretation of the previous theorem is that replicably estimate statistical queries whose range bounded.
We now explain how we can use this result for the statistical k-means problem under the Euclidean metric, but not hard to extend the approach to more general (k, -clusteringg problems. Consider replicably estimate the cost of this solution using Theorem D.2
fixed set of centers F. Then, within an additive accuracy E and confidence ó using 2=- log()/6)) samples. Thus, a natural approach is to consider E-cover of the unit-diameter ball and then exhaustively search among solutions whose centers coincide with elements the e-cover. This is outlined in Algorithm D. We now ready state results.
Algorithm D.I Replicable k-Means with c-Cover
rKMMaassCverr distribution P, error E, replicability p. confidence G - c/3-cover over the d-dimensional unit-diameter ball F G FF k} (We restrict the solutions the e-cover.)
Output argmin rSQ (roOOracl(coos(())) e//, 1/1 5/1=))
Lemma D.3. For any e, 6, p E (0, 1),1), < a/3, Algorithm D. is p-replicable and outputs a solution F whose cost is at most OPT +s with probability 8. Moreover, it has sample complexity
δ (9/2) 2kd
1 8 log
p2gz
19

---

Proof (Lemma D.3). We first argue about the replicability of Algorithm D.1. Since we make |F] calls the replicable SQ subroutine with parameter 9/1-7, the overall replicability of the algorithm follows by taking union bound.
Let us now focus on the correctness of the algorithm. Let F be the optimal solution. Consider the solution that we get when we move the centers of F* to the closest point of G and let us denote it by F Notice that the cost of F is at most OPT +5/s. Furthermore, by a union bound, all the calls to the SQ oracle will return an estimate that is within an additive s/s-error of the true cost. This happens with probability at least 1 - and we condition on this event for the rest of the proof. Thus the estimated cost of the solution F will be at most OPT +2e/3. Let F be the solution that we output. Its estimated cost is at most that of f and so its true cost will be most OPT + This concludes the proof of correctness.
Lastly, argue about the sample complexity of the algorithm. By Proposition C.2,
F S O ((%)))d..
By plugging this value into the sample complexity from Theorem D.2, we get
(%/) (%/)2ak
p²g² log (9/))md
2kd
Although Algorithm D.I provides some basic guarantees, there are several caveats with this approach. Namely, can only get additive approximations and the dependence of the sample and time complexity on both k, d is exponential.
In the following sections, we will explain how we can overcome these issues. As we alluded to before, our approach combines ideas from coresets estimation through hierarchical grids [Frahling and Sohler, 2005), uniform convergence through metric entropy (Wainwright, 2019], and dimensionality reduction techniques [Makarychev et al., 2019].
D.2 Replicable Heavy-Hitters
In this section, we present a replicable heavy hitters algorithm which is inspired by Impagliazzo et al. [2022] and has an improved sample complexity by factor of ((/(r=e)). We believe that this result could be of independent interest since the replicable heavy hitters algorithm has many applications as subroutine more complicated algorithms [Impagliazzo et al., 2022]. Intuitively, this algorithm consists of two phases. In the first phase, we estimate all the candidate heavy hitters and reduce the size of the domain. In the second phase, we estimate the mass these candidates. We present a new analysis of the second phase using Proposition B.I.
Algorithm D.2 Replicable Heavy Hitters r P. target v, error s, replicability confidence
X if
then
L
+ x
else
samples PP
n-m
end if
648 in 2/5 648 +1) l
S +-
samples ~ P
Choose of E (V - 2/35, 1/2s] uniformly randomly
Output all x E d such that Ps(x) 2 v' (empirical distribution
Theorem D.4. Fix u. E (0. 1) and E E (0, u). 8 E (0, p/3). Then Algorithm D.2 is p-replicable and returns a list L of elements x such that with probability at least 8:
(a) (x) S e. then & L.
(b) Pfx 2 then x L.
20

---

Moreover, it has sample complexity
(2) (w- (xx1((--))
1 22
log 1 //
Ô min
Proof. First, wish capture all u heavy hitters in A. If X' is sufficiently small, this is easy. Otherwise, we fail observe each (u =) heavy hitter with probability at most (1 v + s)(X1, By union bound over all (v s) possible heavy hitters, we fail to capture all e heavy hitters with probability most
(1 +)
1
exp
-(v E)
In
6(w-6)
VA
Moving forward, we condition on this step succeeding.
Next, consider Pl P2, the mass estimates over the course of two runs supported on the candidate sets &. both follow (possibly different) multinomial distributions of dimension at most | + 1 with unknown mean parameters pl (x) for E & U {vIJ, P2(2) for E X2 U {y2), respectively, and [S].
Here //, V/ are dummy elements for observations beyond & Ez. Suppose we draw (1.9++(11++))) In samples over each of two runs to yield estimates Pl, P2. By Proposition B.1,
2r
Σ xe rEXu((i)
(i(xx) - pı (x)) 2s'
E (2(()) - p2(x)) 2s'
each with probability least 4/2.. Moving forward, we condition on this step succeeding.
Now, consider choosing E [i - 2r/3, v - 5/s] uniformly at random.
Conrectness; By choosing 2s' < 5/3, any element æ with true mass at least will have empirical mass strictly more than 5/3 v'. Similarly, any element x with true mass most e will have empirical mass strictly less than 25/3 S v'. Thus we satisfy the correctness guarantees. Note that we satisfy this guarantee with probability least 1 5.
Replicability: v' lands in between some (1))), f2) for &g n n with total probability most
Ex (, pa(x)) e/3
Ex pi(x)l ExERinA (2(()) p2(x))
e/3
e/3
In addition, we wish for v' to avoid landing "below" any (2(x) where x $ & & This happens with probability at most
ExERIO n., (1(()) (x))
Mi (2(()) E
P2 (2(x)
e/3
E/3
The sum of the two expressions above is at most
1s'
Thus we set
5/3
4e
p
e/3
3
PE
E' <
36'
The probability of failing to be replicable is then at most 1 0/3 28 2
-
Sample Complexity: Recall that we set
min
36 6
36
21

---

If [X is sufficiently small, the sample complexity becomes
In 3 + (12)+
648 2 + 648 (122) + 1) In
2
1²ga
Otherwise, the sample complexity is
In
6(v-5)
648
648 In
0(0-2) P 8)
+ 1 In2
ppge
D.3 Replicable Coreset
We now prove that Algorithm 4.1 replicably yields an e-coreset. In Appendix D.3.1, we state Algorithm D.3, an equivalent description of Algorithm 4.1. Then, we show that is indeed replicable in Appendix D.3.2. Following this, we prove that the cost of any set of centers F Ba is preserved up to e-multiplicative error in Appendix D.3.3. In Appendix D.3.4, we show that the outputted coreset of modest size. Last but not least, we summarize our findings in Appendix D.3.5.
D.3.1 Pseudocode
Before presenting the analysis, we state an equivalent algorithm to Algorithm 4.1 which uses precise notation that is more suitable for analysis.
Starting with the grid that consists of single cell [-1/2, 1/2jd at layer O, we recursively subdivide each cell into 2d smaller cells of length 2- layer i. Each subdivided cell is child of the larger cell, the containing cell is the parent of a child cell, and cells sharing the same parent are siblings. Let Gi denote the union of cells at the i-th level. We write P, denote the discretized distribution to Gi- In other words, P. = P| (l.) is the restriction of IP to the smallest o-algebra containing G.. We say
cell the i-th level is heavy if the replicable heavy hitters algorithm (ef. Algorithm D.2) returns it with input distribution P., heavy hitter threshold e 2-m and error 7/2 for parameter y be determined later. A cell that not heavy is light. The recursion terminates either when there are no more heavy cells on the previous layer, the current grid length is sufficiently small fraction of OPT. In the second case, we say the last layer consists of special cells.
If no child cell of heavy parent is heavy, mark the parent. Otherwise, recurse its heavy children so that one its descendants will eventually be marked. The recursion must stop, either when there are more heavy children or when the side length is no more than fraction of A. Note that the light cells with heavy parents and special cells partition [-1/2, 1/2]d
Then, we build R in reverse as follows. For marked heavy cell Z, we set R(x) to be an arbitrary (but replicable) point Z, e.g. its center, for all x Z. This covers all light cells with no heavy siblings as well as special cells. Otherwise, cell Z is light with heavy siblings, and its parent is heavy but unmarked. Thus, the parent has some marked heavy descendent Z', and we choose R(x) = R(x') for all x E Z and x' E Z'.
D.3.2 Replicability
Proposition D.5. Algorithm D.3 terminates after at most
1033
log OPT
layers.
22

---

Algorithm D.3 Replicable Coreset; Algorithm 4.1 Equivalent
1: :Corrset(distribttion P, accuracy e, exponent p, replicability p. confidence 8):
2: Init H[i - 2 for all
3: Init [[i < for all i
4: Init S(i] - Ø for all
5:
6: H(() {1-1/2 }/2]
7:
for i 1: Hi 1] + 2 1 do
8:
if 12- < SA/5 then
9:
S[i] children 1])
10:
else
Hli < children (Hli 1J) n rHeevyHitters (Po,u= 7. 1/2, o/e, 8/1) {1 an upper
11:
bound on the number of layers.
12: C] children(H[i: 1]) \ H[j
13: end if
14:
end for
15:
16: for - 0: do
17:
for 2 E Híj do
18:
if (2) + rChoice
children)ZZ) Híj + Ø then
19:
20:
else
21:
(2) + R(CChoicc(childre/ (2) HD + 1)))
end if
23: end for
for Z Lij + 1] do
25:
R(2) < R' ((parent(Z)) (Z))
end
for
27: for Z E S + 1] do
28: R(2) + (parent (2)
29: end for
30:
end for
31:
32:
Output R
Proof. We have
S εΛ
(2-++1)) A)P A)"
EA
2p(-i+1) <
5A
EA
- p(-i+11 1) < log
5AP
5
log
+ 1.
A
But 1/A < 8(44e//Or,, concluding the proof.
Corollary D.6. Algorithm D.3 is p-replicable and succeeds with probability at least - 8. Moreover. has sample complexity
log 1 8 Ô 12g3 8PT
Proof. The only non-trivial random components Algorithm D.3 the heavy hitter estimation. By Theorem D.4, the heavy hitters subroutine call the i-th layer is e/trrellicblle and succeeds with
23

---

probability at least 1 - 8/e. Also, it has sample complexity
t 11 1
log
p2g A3
8)
Now,
t
3-
2
3pi
-
Σ
2 2-
8pi
2=0
1=0
=0.).
Thus all in all, Algorithm D.3 is p-replicable and succeeds with probability at least - 6. It also has sample complexity
t log /)
Ö
p2g3
A3
Since 1/A S #1+oo//PT, we conclude the proof.
D.3.3 Expected Shift
Proposition D.7. For any x E Z E C[i US[ outputted by Algorithm D.3,
<(z, R(x)) (2-++A))
Proof. z is represented by some point within its parent.
Proposition D.8. The output Sit) of Algorithm D.3 satisfies
f Jsd s(x, ((P)) s V
Proof. By computation,
/ 5/
<(z, d((x) <
(2 (yd+())
Proposition D.7
S[C
εA dP()))
definition of
STG]
eA dIP(x)
xx
sA
5
Let F' be an arbitrary set of & centers. Partition C[i]
into
5
Lnear [i] =
Z Cli]
min #EZ
k(x, F)P
<
22-1+1
i+1 A)
i+1 A) 5 (2
Ltar [i] =
Z
Cli
min #
n(x, F)?
E
Proposition D.9. The output Liar of Algorithm D.3 satisfies
Σ
n(x, P(x)) dPP()) cost(().
Crall))
i=0
24

---

Proof. Any x E Z E Lialli] contributes cost of at least (2 ++1 A). Hence
Σ Σ Crarl
Σ
n(2, R(z))(
2- PP
Coald
(2 i+1 A)))(>
(iar(1)
x(x, F))
==
Ltrld
cost(F) (F).
Proposition D.10. The output fenaar of Algorithm D.3 satisfies Luear [i]] S kM where
2
M ==
A
Proof. The furthest point x in a cell in Lucar [i] can have a distance of at most
2 2 + -2 (2 (1 (1+ (2-'A)
5
to the nearest center. Thus the points belonging to particular center live in an lo ball of that radius. Not including the cell containing the center, we can walk past at most
1
1+
10 (-
(1+ 10 ) A
2-
cells we walk the directions the canonical basis of Rd. follows that there can at most
[ +
1 10
d
32
A
<
cells in [i close to each center of F, and thus there are at most &M cells in total. Proposition D.11. By choosing
52kM
The output Cnear of Algorithu D.3 satisfies
Σ
n(x, R(x)) s EA
5 Cnearlil
Proof. Each light cell has measure at most
7 p By computation,
Σ
R(x)
(2-++1 d)))(x)
Coonely]
Lncar
kM 7.A p- (2A)²
2-pi
KM -7 - (2A)
A
EA
Corollary D.12. Set
y: 5tkM(2A))
Then for any set F of k-centers, the output R of Algorithm D.3 satisfies
n(x, dTP)) & e cost
25

---

Proof. X' is contained in the union of all L[d]ss and S(t). Hence
J n(a, R(x))
2
cost(F)
5
5
2= + OPT
cost(F)
5
4 OPT + cost (F)
cost(F).
D.3.4 Coreset Size
To determine the size our coreset, we bound the number of marked heavy cells, as all our representative points in marked heavy cells.
Fix ForT be a of optimal centers for the (ks pllusteriiin problem. Write M(i to denote the set of marked heavy cells on the i-th layer and partition M as
Melasa [i] = 7 E M[i
min s(x, F) S pi+1 #EZ
Maist == Z M[i
min <(x, F) > 2 pi+1
xEZ
Proposition D.13. The marked heavy cells Maist outputted by Algorithm D.3 satisfy
23 S
U OSiSt
Maist
A-A Proof. Each cell in Maist has mass at least 2-P and each x E Z E Maise least 2-pi-1 to its closest center. Thus each cell contributes at least
[i] is of distance at
7.A
2-+++1 2-
2-pi+1
to the objective. If follows that there are at most
OPT
74 S 3(1+6) 23
such cells.
Proposition D.14. The marked heavy cells Melose outpulted by Algorithm D.3 satisfy [Melose [i]l S k(7A)d
Proof. The furthest point æ in cell Metesn to nearest center
pi+1 + -
In other words, not including the cell containing the center, we can walk past most
20(1-))+1 + 2+ A
cells by walking along the direction of the canonical basis of Rd Thus there most
k +1 / < (
such cells on the i-th layer.
Proposition D.15. The image of the map R outputted by Algorithm D.3 satisfies
23 N = [R(x)) < + (77))))
26

---

In particular, for
7: 5thM(2A)
we have
38 38
(2A)
N: = R(2) - 7
Proof.
The image contained in the marked heavy cells.
D.3.5
k-Medians & k-Means
We now derive results for the case of 1.
Theorem D.16. Algorithm D.3 is p-replicable and outputs an e-coreset of size
k22d Ed+1
d+1
log
OPT
for statistical k-medians with probability at least 1 8. Moreover, it has sample complexity
ò
2 A 3d+3
1 8
gg3d44 OPT3 log
Proof (Theorem D.16). By Corollary D.12, Algorithm D.3 outputs an e-coreset. By Corollary D.6, Algorithm D.3 is p-replicable with sample complexity
(
log !!
ò
p2gs
OPT
We have
38 7
0 k: 12/ (2
A
1:25d Ad+1
A
log g OPT
0
log
gd+1
-OPT
By Proposition D.15, the coreset has size at most this value.
Substituting the value of y above and remarking that polylogarithmic factor of the other terms, we thus conclude that the final sample complexity is
(
R-3215d A 3d++3
1 13215d Add-3 1 1 ()) (peMtts. OPT3 8 1)
log 8
g3d+3
P² OPT
Now consider the case of p We can perform our analysis with the help of standard inequality. Proposition D.17. For any set of centers F.
E,x(R(z),F) < Ex F)² + 2 Eg [K( F) E. [<(2 R(x))² Ex R(x))² En 2)² R. (R))) 7)2 2 F. (x))) F)J] E. [n(x, R(x))2 Ean(x, R(x))²
Thus if Eyn(I, R(x))² < 3 E, F)2
1
E,n(R(z),F)) < E,K(I,F) F)² + 2 23 =E [)] + 20 =² [K(2, F)3]
$ (1+ ExK(I,FF2
E, < E.n(R(E),FF) F)² + 22/ (1+ Ex [n(x, F)2 1
26 E2E, [n(x, F)2 + 28 e e²Enn(x,FF) F)²
s {En(H(())FF) + Eby [K(I,, FT + 26 " F)²
<En((H()), F)2 EEEN(IIFF' F)?.
27

---

1 8 Proof. By Hölder's inequality,
Exx(H(x), F)2 s En | P) K(2,,
= =En((o, 2 F)((,, ((xx)(()))) + Ex R(x))
S Ran(a, F)² + 22 Ry [n(x, F) E [n(2, R(x)) EEN(s, R(x))²
The other case is identical.
Theorem D.18. Algorithm D.3 is p-replicable and outputs an E-coreset for statistical k-means of size
k.2
d+2
log
g2d+2
OPT
with probability at least 8. Moreover, has sample complexity
ò 1 A Ad4+6
1 8
log
pe0d00 OPT
Proof (Theorem D.18). Similar k-medians, Algorithm D.3 outputs an e-coreset with probability least 1 - 6, is p-replicable, and has sample complexity
(p log !)
ò p273 OPT
However, we need to take e' ;= =2/22.
We have
3 =
1k
(24)2
k2lld Ad2 d+2
A
=0
log
gdd+2
@PPT
By Proposition D.15, the coreset has size at most this value.
t a polylogarithmic factor of the rest of the parameters, thus we thus conclude that the final sample complexity is
1. 1-333d A 3d46
=ō
A-233d A 3346
δ
log
g6d+6
p² OPTS
p2-6d+6 OPT
is clear how compare our guarantees with that of Frahling and Sohler [2005] as we attempt to minimize the number of samples required from a distribution assuming only sample access. On the other hand, Frahling and Sohler [2005] attempted minimize the running time of their algorithm assuming access to the entire distribution.
D.4 Replicable Multinomial Parameter Estimation
Now that we have reduced the problem to a finite distribution (coreset), we can think of it as weighted instance of (k, clusstrrig where the weights unknown but be estimated using data.
Proposition D.19. Enumerate the coreset R(X') = r()), (N) and let w be the probability mass at ((), If we have estimates w(( satisfying
lw(0 - (0)) < N
then
N
N (4) He( (3) F)P Σ
(+)
(e(
F)P S
==1
for all centers F.
28

---

Proof. Indeed,
Σ
w 1
k(r
F)
-
k(r(0)
FF)
Σ w |x((1))
F))
S.
Let us formulate this as replicable multinomial parameter estimation problem. Consider multino- mial distribution Z = (2 2(N) of dimension N with parameters , p and n. Note that this can be cast more general statistical query problem which illustrate below. We allow simultaneous estimation of more general statistical queries ....... -N (A"",P) + R compared to Impagliazzo al. [2022], assuming they can be estimated with high accuracy and confidence from data, say each query (z Fn) concentrates about its true value G3 E R with high probability.
Algorithm D.4 Replicable Rounding
1: rRounding(statistica queries 91.-.-60 ON. distribution P, number of samples n, error e, replicabil-
ity p. confidence 8):
2: Sample ž1,
P
3: a p
4: for
N:
5: a <- uniform random sample [0, a)
6: Split IE into intervals 7() {[a + 2a, a3)) + (= + 1)a) E Z)
7: Output the midpoint G, of the interval within ((t) that 9j (Z....., Fn) falls into
8: end for
Theorem D.20 (Replicable Rounding). Suppose we have a finite class of statistical queries
.1..... IN and sampling independent points from IP ensures that
In) - G3l E' == c(p 26)
p 28
with probability at least
1-
Then Algorithum D.4 is p-replicable and outputs estimates G, such that J/ Gyl Se
with probability at least 1 5 for every j E [N]. Moreover, it requires at most n samples.
Proof. Outputting the midpoint of the interval can offset each gj (x......., xn) by at most 2. Hence
s(p - 26)
GG - GG
p+ 25 1+1- 25 =
Consider two executions of our algorithm yielding estimates (()), ((2) The probability that any of the estimates fail to satisfy f1-tolerance s' at most 26. The two executions output different
estimates only if some random offset "splits" some GGa)) G,,)). Conditioning on the success to satisfy --ttoeeaacc,, this occurs with probability at most
-(()
(2)
(0)) )
Ggl++ - -
Gyl
a
j=1
2x'
<
a
<
26.
Accounting for the probability of failure to satisfy the -oleraance of 25, our algorithm preppiccabl..
D
29

---

We note that Theorem D.20 can be thought of as generalization of the SQ oracle (cf. Theorem D.2) from Impagliazzo et [2022]. Indeed, simply using Theorem D.2 yields an extra factor of N in the sample complexity as we must take union bound.
Corollary D.21. Let E, (0.1) and ó E p/s).s). There is p-replicable algorithm that outputs parameter estimates for a multinomial distribution of dimension N such that
(a) p 1 S for every i E [M] with probability at least 1 8.
(b) p()) 2 0 for all i E [N].
(c) E 11
Moreover, the algorithm has sample complexity
In 1/8++N N
N 1 5
= Ô
log
(2 (p- 8)2
=2
Proof. Define
e' = e(p - 28) p 1 28
By Proposition B. sampling
2n 1/8 1/2N In 122 822 81n 1/6 +N s 28)2
points implies that Eimlo 1l1 <' with probability at least 6.
Thus running Algorithm D.4 with functions
N
( INN) =
Σ
11 = }}
N
yields p such that - |
1p() < e for each i [N]. If there are any negative p(0's, we can only
improve the approximation by setting them to O. If the sum of estimates is not equal to 1, we can normalize by taking
]
T t p
1 N
Σ 0
This introduces an additional error of S/N for each p(0) and maximum error 2s. Choosing E1 ;= 5/2 concludes the proof.
D.5 Replicable OPT Estimation for (k.p))-lusteriin
Our coreset algorithm assumes the knowledge provided us, show in this section that it is possible replicably compute such an estimate using a two-step approach. First, we are able to produce replicable estimates with additive error s simply by approximately solving the sample (k, cllustering problem on sufficiently large sample thanks to uniform convergence (cf. Theorem C.9). Then, we repeat this process with e =/2 until is but small fraction of the outputted estimate.
constant ratio estimation of OPT. If this is not
Theorem D.22 (OPT Estimation with Additive Error). Let > be an absolute constant and e E (0, 1),8 E (0, 9/3). Suppose we are provided with an algorithm A that, given an instance of sample (k, plccusttri outputs an estimate = of the value OPT such that
Ee OPT, OPT
3
30

---

Then there is p-replicable algorithm which produces an estimate A of OPT such that
OPT
A
+F
with probability at least Moreover, the algorithm has sample complexity
k d2
log
E6
We note that the algorithm A can be taken to be any B-approximation algorithm with some postpro- cessing, where we divide the value of the cost by 3. Alternativlly, we can take A to be some convex relaxation of our problem with constant integral gap.
Proof. Let 1,...., XN P be i.i.d. random variables. The output E A(1 Xw) E (0 is thus bounded random variable. Suppose we repeat this experiment times produce estimates E En. By an Hoeffding bound,
P
E
E
S 2cxp (-2ne?)
Thus with
2
In
2=
8
trials, the average estimate : " 2i=1= Ej satisfies
[
[E|
E with probability least 6.
Now, by Theorem C.9, choosing
k2dd
np
N 0
log
i..d. samples from P ensures that Thus conditioning on success, we have
OPT - OPT
< e for all trial with probability at least 1 6.
OPT B
E, E
OPT +e
for every trial E [n]] But then the average E also falls into this interval as well.
We have shown that there is algorithm that outputs estimate
[PP 3
E E E e, E[ + c
e, OPT
with probability at least - 8. Moreover, has sample complexity
nN 0
log E4
1 d2d
p log
log 1 )) 5
8
228
2222 g6
0
log²
E6
We can now apply the replicable rounding algorithm (ef. Algorithm D.4) to achieve the desired outcome. Indeed, by Theorem D.20, the output A after the rounding procedure is p-reproducible and offsets the average E by at most Hence we have
OPT 3
AE
25, OPT +2c.
with probability at least
8.8 Finally, the algorithm has sample complexity
k2
d2
P
O
log
6
6
Epō
Choosing g' = 5/2 completes the proof.
31

---

Theorem D.23 (OPT Estimation with Relative Error), E, E (0. 1) and å E (0, p/s). There is a p-replicable algorithm such that with probability least 6, it outputs an estimate A of OPT where
A E
OPT, (1
B(1
Moreover, it has sample complexity
32d 112 po OPT
log p
Proof.
We run the estimation algorithm with additive error with
E 2
P: = 2 p 6. : 88
for 1,2,3,... until obtain estimate A; such that e EA./2.
Remark that
1 2
2- A
1
OPT 3
= 2
2-1
2
E OPT
2-1
1+ _
2/
E CPP
= -i < log
43
B
i log
+ 2.
OPT
Thus the algorithm certainly terminates. Upon termination, we output some A such that [OPT// - 21/2, +c1/2]. follows that
1
A s OPT
2
A OPT
2
A < 1-
OPT
$ (1 + 8) OPT
and
A OPT
2 E
3
3 +
A 2 OPT
2
A 1
OPT.
B (1+$)
The probability of not being replicable in each iteration is 2 2 Hence the total probability is most
D 2 2 p
and similarly for 8.
32

---

OPT iterations and the sample complexity of each iteration is bounded above by the sample complexity of the final iteration where
Finally, there are O (og
E OPP
2
8
Hence the total sample complexity is
log
B
12222
log P
12d2,312 = E1 OPT log )))
EPP
[OPTT
<OPTT
D.6
Putting it Together
We are now ready prove the results on statistical i--medians and statistical k-means, which we restate below for convenience. Recall that 3 is an absolute constant and hence disappears under the big-O notation.
Theorem 4.2 (Theorem 3.1; Formal), Lete, E (0, 1) and 8 E (0, p/3). Given black-box access
3 3-approximation oracle for weighted k-medians (cf. Problem 2.2), there is a p-replicable algorithm for statistical k-medians (cf. Problem 1) such that, with probability at least 1 6, it outputs (1 + e)B cj8-approximation Moreover, it has sample complexity
(( k²d
1a218d 3d43
1 log
OPT
p2gdd45 OPT
8
Proof (Theorem 4.2). First, we compute A, an estimate of OPT with relative error E, confidence 6/3, and replicability parameter e/3. Then, produce an s/2-coreset for k-medians of cardinality N with confidence and replicability parameters $/a, 9/3. Finally, we estimate the mass at each point of the coreset with confidence and replicability parameters $/3, P/3 accuracy of
εA
8 OPT
4N
2N
By Proposition 1.19, running the approximation oracle on the coreset with the estimated weights yields B[ + apprroxximtioon.
By Theorem D.23, computing A requires
k2d log
1
E
12
1P
samples.
By Theorem D.16, the coreset construction yields a coreset of cardinality
N o
k-2dd d++1
d+1
log
A
2d
=0
k-26d
d+1
log
ed+1
--OPT
gd-1
OPT
and incurs a sample cost of
δ
1331 3d43
23d
Ō
1-32184 8d 3d43 1
OPT
p2g3d+3.
OPT 8
Finally, Corollary D.21 states that the probability mass estimation incurs sample cost of
δ
2222
N
A2
log
1
=
OPT .
N3
log
13218d A Ad+3
1
1
log
gdd+
c2
2
13218d A 3d+3
1
=
log
Pg3dd45
OPT
8
33

---

Theorem 4.3 (Theorem 3.1; Formal), Given black-box access to 3-approximation oracle for weighted k-means (ef. Problem 2.2), there p-replicable algorithm for statistical k-means (cf. Problem 2. such that, with probability at least 1 - 6, it replicably outputs a (1 + 2)8pppprroiimation Moreover, it has sample complexity
((
δ ((
d2
OPT + k A 3d+6
1 log
p2g OPT3
8
Proof (Theorem 4.3). Similar to k-median, we first compute A, an estimate of OPT with relative error confidence 8/3, and replicability parameter 9/3. Then, produce #/2-coreset for k-medians of cardinality N with confidence and replicability parameters 9/3, P/3. Finally, we estimate the mass at each point of the coreset with confidence and replicability parameters 8/3, P/3 to an accuracy of
εA
E OPT
4N
2N
By Proposition D.19, running the approximation oracle the coreset with the estimated weights yields (1 + -pappooximatinn.
By Theorem D.23, computing A requires
Ô
22d2 log
1 8
E p2
OPT 12
samples.
By Theorem D.18, the coreset construction yields a coreset of cardinality
k:2
A
1:2 A 4+2
A
= 0
log
.22d
log
g2d2
COP
c2d+2
E OPT
and incurs sample cost of
ò 2
A 3d+6
k92339 A 3d+6 1
26d
)
g2266 OPT
p2g6d46
OPT3
Finally, Corollary D.21 states that the probability mass estimation incurs a sample cost of
N
1
N3
Ô
p2
^2
e2p2 OPT
A 3446
δ
cld+6
13230d
3dd46
p2g6d48
OPT2
E The Euclidean Metric, Dimensionality Reduction, and (k, p..pulCCsserrnng
We now build towards proof for Theorem 5.1, which states the formal guarantees of the replicable k k-medians and k-meaas algorithms under the Euclidean distance. Let begin by recalling the Liimnsonniideestrauuss lemma.
Theorem E.1 ([Johnson, 1984]). There exists a family of random linear maps Vd.m from Rd R such that the following hold.
(i) For every d 1, (0,4/2), Rd. and O (3/²2 logg 1/8).
N:: 12 I ||=1|2 I (1 ->||
with probability at least 8. 8.
(ii) Wd.m is sub-Gaassian-tailed, thus for every unit vector a E Rdd and & >
|| 2 +
with probability most m)).
34

---

Furthermore, we can take Td.m to be the set of random orthogonal projections R - IR scaled by factor of V4mm
We say that fdm. is standard random dimensionn-eduutionn map and write P,, denote the projected distribution on I i.e., (=(() = (())
As mentioned in Section is not entirely clear how translate between sets of centers for high- dimensional data and solutions sets for the compressed data. For this reason, our goal is to produce a clustering function f X' + [k] which computes the label for any given point polynomial time. This also requires to consider another notion of cost which is translated more easily.
Definition E.2 (Partition Cost). Define the cost of a k-partition C = (CC,),..,,C.))) of as
cost(C)= == G JC(
min
E Σ 1 e -t0))//l,,elleee wl01)
i.e. we consider the expected cost of picking the optimal centers w.r.t. the underlying data-generating distribution. Similarly, define the following sample partition cost.
Definition E.3 (Sample Partition Cost). We define the sample cost a k- k-partition C of Æ1,..., In as
k
cost(C, w) = min at E
we
E 11 j=1
where w 0, E. ui 1. i.e. we consider the cost that is induced by the distribution specified by on the samples. We write cCst(C) to denote the cost with the uniform distribution on the samples.
Remark that if induced by F, then OPT S cost(C) S cost(() and similarly for the sample partition cost as well.
Theorem E.4 (Makarychev et al., 2019]). Let ; Fn Rd be instance of the Euclidean (k, de.p))clustering problem. Fix E E (0, 1/4), 5 E 10 and suppose æ RRd -> Rm is a standard random cimeension-reductiin map with
log k Ed
0
22
Then with probability at least
1
(Cs S cost((T(C) < (1 (C)
I+E
for every k-partition of .1:..., Fn-
In essence, this theorem tells that if let the centers of the clusters be the centers of the partition, then the (sample) cost is preserved in the lower dimensional space. To the best of our knowledge, this is the strongest result for dimensionality reduction in (sample) pcclutenn which preserves more than just the costs of optimal centers.
The rest of Appendix E serves generalize Theorem E.4 for preserving partition costs in the distributional setting. First, Appendix E. addresses the issue that a random orthogonal projection increase the diameter of our data in the worst case by scaling the original data by appropriate factor. Next, Appendix E.2 slightly generalizes Theorem E.4 to the weighted sample clustering setting. Finally, Appendix E.3 extends Theorem E.4 to the distributional setting, culminating with Theorem E. 10, which may be of independent interest beyond replicability.
Recall that P. is the push-forward measure (x{' induced by We write OPT, to denote the cost of an optimal solution to the statistical (k: p) clusttring problem on (m(X), and ÖPT, - for the (ks, p) cclusterin problem on TT ((21)...., ,I,)).,..,,,,,,,
35

---

E.1 Scaling
We may assume that a standard dimension
dimension-reduution reduction map satisfies π = Vmm for some orthogonal projection P. Hence has operator at most v and
x (()) S vdB...
In order to apply our existing analysis, we first scale down the samples in the input space by factor of vd so that the projected space still lives in x-ball of diameter 1. This does not affect our approximation guarantees as they are all multiplicative and hence scale-invariant. However, the analysis we perform on the scaled distribution with optimal cost OPT' d OPT. This extra term must be accounted for.
We proceed assuming now that S Ba and (x)) S Bmn-
E.2 Preserving Partition Costs
Let cost w), costottttt (C), w) be the weighted cost of the partition and projected partition, respec- tively. Recall this means that the points participate with weights w in the objective. Essentially, the next result shows that since the projection guarantees do not depend on the number of points we are projecting, we can simulate the weights by considering multiple copies of the points.
Proposition E.5. Let :1,.... Fn Rd be an instance of the Fuclidean (k. clustering problem and
Wi..... Wn IR+ be weights which sum to 1. Fix E (0, 2/4) 8 E (0, 1), and suppose т RR - R a standard random dimdussion-reductiion reduction map with
k log
62
cd
Then with probability at least 1 8,
1
cosis w) S cost(T(C),w)) S (1 + s)oosi w),
1+8
for every k-partition C of......... In-
Proof (Proposition E.5). First suppose that w; = m//b, for a,, bj E Z4 and consider w/ := lem(b,..... ..... bw),, Z+
obtained from w; by multiplying by the least common multiple of the denominators.
Let y1..... Ynr be the multiset obtained from .,,..., In by taking w/ multiples of xz. The cost of any partition of 21..... In with weights ....... In equal the cost of the induced partition of y1...., with uniform weights. Thus we can apply Theorem E.4 on yi..... Yn to conclude the proof.
Now, for general Wi E RR,, we remark by the density of the rationals in that there are some 2.....79m Q4 which sum to 1 and satisfy
min ÖPT, ÖPT,)
2
Then for any k-partition C and (1))
(R) Ba.
d
1x, 000 Jx(s, 60p
wi Σ
Mx E C00 500
4
1=1
{x E CuD wp
qi
W;
Wi
1=1
ÖPT.
2
36

---

In particular
1 cost(C, w) cost(C,,) S (1 E) cost(C,, w).
+ E
Similarly, for any v()). (()) Bm.
Σ 9 Σ E ( ub)))
Σ w Σ 1z, (0))((((2))) 2000p
Vn
OPT
m
which implies that
cost (TS)( w) S cost(T(C),g) 2 (1 +++e cost (T(C), (C),ww) w).
Finally, we conclude that
cost(C, w) $ (1 + cost :)
$ (1 + =)² cost (C), q)
$ (1 =) cost (T (C), w)
cost( w) s (1 + =) cost(C, w).
Choosing e' to be constant fraction of & concludes the proof.
Corollary E.6. Let ; En E Rd an instance of the weighted Euclidean (k, p) clusttrinng problem with non-negative weights .,...., Wn. R., which sum to 1. Fix E E (0, 1/4, (0, and suppose π Rd + m is a standard random dimension-reduction map with
p* E
k
log
gd
Then with probability at least 1 - 8,
ÖPT ÓPT, S (1+E)
1+
Proof (Corollary E.6). The optimal clustering cost coincides with the cost of the partition induced by the optimal centers.
E.3 Dimensionality Reduction
Let G (1(h) .9)....(())) be (-aaproximmte solution to the statistical Euclidean (k, p)clusteriinn problem on (x) We would like to argue that the partition of induced by G is a (1 + =) approximate partition with probability at least - 6. To do we first approximate X' with finite set &' in replicable fashion while preserving partition costs. This allows us to apply Proposition E.5.
Recall we write denote replicable estimate of OPT such that [OPT/a(++)) (1 + e) OPT] (cf. Theorem D.23). Consider grid of length <\/(4))). for point E X, we assign it to the center of the cell that it belongs to, say J. Let X² = ( x E X) denote this finite set and the distribution induced by the discretization. Remark that <( σ) & <A/(4p) for every æ X.
Note that a partition of X induces a partition of A and vice versa. We proceed without making any distinction between the two.
Proposition E.7. For any point u E Ba.
[<(x, u)" x(ā u) S pl)((,, n(ā, u)l-
Proof. By the mean value theorem, the function g [0, 1] + IR given by 2 + 2d is p-Liscchitz:
(g(v) - g(>)] < sup g (E)ly - =1
{E[0.1]
< ply 2|.
37

---

We write cost(C) to denote the cost of the partition C of X.
Proposition E.8. For any partition C of X'
cost(C) S cost(C) S (1+ cost(C).
1+8
Proof. We have
k
&
E
Σ
ff E clo
m(z, ul0)pp
Ex
Σ
I H E cbi).
n(3, ub03p
S Ex
Σ
1/ E chil. ul)p n(3, u 1
j=1
&
$ Ez
Σ
1fx Chi pln(x,, wb)) x(0, w(1i
j=
EA
S p
4p
OPT.
Corollary E.9. Fix E (04) 5 E (0,1) and suppose Rd - R is standard random coinension-reductiinnn map with
log k
d d= 0
E8
We have
(1 =) OPT & OPT,
with probability at least
6, where the probability is taken w.r.f. the choice of the random map.
Proof. The optimal clustering coincides with the cost of the partition induced by the optimal centers. An application Proposition E.8 yields the desired result.
Theorem E.10. Fix (0, (0, 1) and suppose E Td, a standard random dimension- reduction map with
m = 0
c2 log E6)
Let G- = (2 ..... (8) be 3-approximate solution the statistical Euclidean (k, cllusterii problem on (x Then the partition of induced by G is a + ( -apprroxiie partition with probability at least 6.
That is, for C (C(1) ((k) given by
( x E X gbو) argmin <( ((T(((()))))))))))))))) (
we have
cC) < (1 0PP
Proof. We have
cCst(C)
(1 (C)
Proposition E.8
< (1 + ()))
Proposition E.5
< (1 (G)
G' induces π (C)
< (1 + 3OPT
G is 3-approximmte
< (1 + 3)3 OPT.
Corollary E.9
Choosing E' to be constant fraction of concludes the proof.
38

---

E.4 Summary of Dimensionality Reduction
Before proving Theorem 5.1, us summarize the stages of our pipeline.
1) Scale the input data by l/va.
2) Estimate OPT of the scaled data within a multiplicative error.
3) Discretize the scaled data with fine grid.
4) Project the scaled data with a random map л E 7d.m-
5) Compute coreset the projected data.
6) Estimate the probability mass at each point of the coreset.
7) Call the 3-approximation oracle to produce & centers g()), (A))
8) Output the clustering function induced by the centers.
The clustering function proceeds as follows. Given point E X,
1) Scale by 1/va.
2) Discretize the scaled point with the same grid used in the algorithm.
3) Project the discretized point using the same x used the algorithm.
4) Output the label of the closest center g0) in the lower dimension for j E [k:]
We are now ready to prove Theorem 5.1, which we restate below for convenience.
Theorem 5.1 (Theorem 3.2; Formal), Let E, p E (0, and 5 E (0, 9/3). Given B-aaproximatiin oracle for weighted Euclidean k-medians (k-means), there is a p-replicable algorithm that outputs
clustering function such that with probability at least 8, the cost of the partition at most (1 + e)8 OPT. Moreover, the algorithm has sample complexity
δ (poly
poly PPPT)) )
( kd
(2VTT)
O(m)
1 log 8 8
where = 0 (12 log k).
Proof (Theorem 5.1). We randomly draw some и E Td.m for
( n² log cō
k
d= O
Then, we discretize sufficiently large sample after scaling and push it through и. We then run the corresponding algorithm (cf. Theorem 4.2, Theorem 4.3) to produce some centers for the compressed data. This allows to partition X' according to the closest member of G in the lower dimension. An application of Theorem E. 10 yields the performance guarantees this algorithm.
The sample complexity guarantee from Theorem 4.2, Theorem 4.3 is for the scaled distribution and the scaled optimal cost satisfies OPT', d OPT, Hence the sample complexity resolves to
kd Ô poly (OOPT)
(poly
22/m
((m)
log 8
We conclude the proof by remarking that for the Euclidean metric, A Va and that Ollog n) Ologg = n Olloglogm
k-Centers
In this section, we will explain our approach to solving the statistical k-centers problem (cf. Prob- lem 2.3). As we explained before, since this problem has a different flavor compared to (k, p)- clustering, we introduce some extra assumptions in order to be able to solve it from samples. To the best of our knowledge, this is the first work that considers model of statistical flavor for the k-centers problem.
39

---

F1 Assumptions
First, we describe an assumption that is necessary to obtain good clustering solutions even if we do not insist on using replicable algorithms.
Assumption F.1 (Clusterable). For some absolute constants B. B > O, i.i.d. sample S Ei..... EI....,22, F' of size n from P has the following property. With probability at least q. there exists some (3, appproxiiie solution F. to Problem 2.3 such that, for every f F. there is some xy E S with P. (2 = f. Here E [0, 1].
Assumption F.1 is necessary to obtain good clustering solutions even in the non-replicable case. Our distribution P must be sufficiently well-behaved so that solving the sample k-centers problem translates to good solution in the population case. Essentially, this assumption states that with high probability, the sample will contain point from every single cluster of some (3, B) approximaae solution. In order to design replicable algorithms, we need to make stronger assumption.
Assumption F.2 (Replicable). For some absolute constants B, B, there exists (3, B) -approxiimtee solution F. to Problem 2.3 with the following property. With probability at least q, in an i..d. sample
T1,.... In of size n from P, for every f F., there is some zy E S with F. (xy) = f. Here E [0. 1].
Note that Assumption F.1 guarantees that, with high probability, every sample will help us identify some good solution. However, these good solutions could be vastly different when we observe different samples, thus this assumption is not sufficient to design a replicable algorithm with high utility. Assumption F.2 necessary to attain replicable solutions. Indeed, this assumption essentially states that there is a fixed solution for which will observe samples from every cluster with high probability. If this does not hold and we observe points from clusters of solutions that vary signifi- cantly, cannot hope to replicably recover fixed approximate solution. Note that Assumption F.2 is stronger than Assumption F.I, since we have flipped the order of the existential quantifiers. One way satisfy this assumption is require that there is some (B, approooxxit solution that the total mass that is assigned to every cluster of it is at least I/n for some n E N. Then, by observing (nn) points, we will receive at least one from every cluster with high probability. This is made formal in Proposition F.3.
Proposition F.3 (Jannson, 2018]). Let 8 E (0, Under Assumption F.2, a sample of
0
nm
o
log
points contains at least m. points from each cluster of F. with probability least 1.8-
8.
Proof (Proposition F:3). Let Y; be a geometric variable with success rate g. which denotes the number of batches of samples draw until point from each cluster of F, is observed. Then Y := Σm Y, upper bounds the number of batches until we obtain m points from each cluster.
As shown in Janson [2018].
P(Y 2 )EY] I exp(1 - A)
m EY
The result follows by picking À = 1 log( (1/6).
F2 High-Level Overview of the Approach
We are now ready provide a high-level overview of our approach derive Theorem 3.3. We first take a fixed grid of side e in order to cover the unit ball. Then, we sample sufficiently many points from F. Subsequently, we round all the points of the sample the centers of the cells of the grid that they fall into. Using these points, we empirically estimate the density of the distribution on every cell of the grid. Next, in order to ensure that our solution replicable, we take a random threshold from a predefined interval and discard the points from all the cells whose density falls below the threshold. Finally, we call the approximation oracle that we have access to using the points that have survived after the previous step. We remark that unlike the (k: pccluscus problem (cf. Problem 2.1), to the best of our knowledge, there does not exist any dimensionality reduction techniques that apply to the k-centers problem. In the following subsections explain our approach in more detail.
40

---

F.3 Oracle on Sample
As we explained before, we require black-box access to an oracle for the combinatorial k-centers problem on sample which outputs (3, B) approoxiiate solutions F for Problem 2.4. Thus, we need to show that, with high probability, the output of this solution is good approximation to Problem 2.3. This is shown in Proposition F.4.
Proposition F.4. Suppose there a (3, -approximmaee solution F. for Problem 2.3 and that we are provided with sample of size O(n log(1/6)/)) containing an observation from each cluster of F., with probability at least 1 Then, the output F of the oracle O on the sample satisfies
k(z, F) (28 + 3) OPT +2B + B.
for all x E X, with probability at least 1 5.
In the previous result (), (3, B) are the approximation parameters that inherit due to the fact that we receive samples from IP and that we solve the combinatorial problem approximately using O, respectively.
Proof (Proposition F:4). Let F. denote the (3, -aaaproxxia solution from Assumption F.2. We condition the event that such solution exists in the sample, which occurs with probability at least 1 1-6. 6.
Fix E X, which is not necessarily observed in the sample. We pay cost of 8 OPT +B to travel to the nearest center of F.. Next, we pay another 33 OPT +B to travel to the observed sample from this center. Finally, pay cost of
cost < S0PT + B
3PPT +
arrive at the closest center in F.
F.4 Oracle with Grid
The issue with using directly the output of O on given sample is that the result will not be replicable, since we have no control over its behavior, In a similar way as with the (k:, plcusseeiiigg problem, the main tool that have in disposal to "stabilize" the output of O to the other problems, we cannot use the hierarchical grid approach in this one. The reason is that, due the min-max nature of k-centers, we need to cover the whole space and not just the points where the distribution is dense. Recall that so far we have established that the oracle O outputs (2 + B, 2B + 2B) approxiiate solutions F for Problem 2.3, given that Assumption F.2 holds. Algorithm F.1 "discretizes" the points of the sample using grid with side-length before applying the oracle. Proposition F.5 shows that by doing that, we have to pay extra additive term that is of the order (cA))
grid. Unlike
Algorithm F.1 Oracle with Grid
1: Oracle with (racle O, sample æi...., Zn, grid length c):
2: for i I...., do
3: Decompose = c x for some E 2d _ E [0, c)d
4: i. <- 2; c 1/22, 9/2) (Ø; is the center of the cell that contains ...))
5: end for
6: Return ($i,...,,
Proposition F.5. Suppose there is (8, B) approxiimae solution F. for Problem 2.3 and that provided with a sample of size n. containing an observation from each cluster of F.. Let G denote the output of Algorithm F.1 using (3, approxxiiiitai oracle O for Problem 2.4. Then, for any x E X.
<(z, G) < (23 3) OPT +2B + B (43 23 + 1)cA.
41

---

Proof (Proposition F.5). For the sake of analysis, imagine we used the grid to discretize all points of X' X. Then F. solution with cost 30PT B + cá and we observe a point from each of its clusters in our discretized sample. Here OPT is the cost of a solution to Problem 2.3 on X..
Proposition F.4 shows that the oracle returns a solution G with cost at most
(2/3 + 0)PT (2B cA) + B
on the discretized points.
Then for any X.
OPT = n(x, GOPT)
k(2, 1) n(ē GOPT)
< n(I, i) + k (z, FOPT)
2x(xr 3) + ,n,, FOPT)
s 2cA - OPT.
The conclusion follows by combining the two inequalities.
F,5 Replicable Active Cells
At first glance, we might be tempted to work towards replicability by arguing that, with high probability, every grid cell is non-empty in one execution of the algorithm if and only if it is non- empty in another exccution. However, since we have no control over F. it is not clear how one can prove such a statement. Our approach is to take more samples and ignore cells which do not have sufficient number of points using some random threshold. The idea is similar to the heavy-hitters algorithm (ef. Algorithm D.2) originally conceived by Impagliazzo et al. [2022] and is presented in Algorithm F.2. Then, we move all the points of active cells to the center the cell. Notice that the k-center objective, unlike the (k, cllusterii objective, placed in a cell, just whether it is positive or not.
do not care about how much mass is
We first derive bound on the number of cells that the distribution puts mass on. It is not hard to see that since we are working in a bounded domain, the grid contains at most (2/c)" cells in total. Thus there are most M S (1/2)d cells that have non-zero mass. Proposition F.6 shows how we can obtain such bound in the unbounded domain setting.
Proposition F6. Let be (B,B)-approee approximmaee solution for Problem 2.3. Let M denote the maximum number of cells which intersect some cluster of F. and P the discretized distribution that supported on centers of the cells of the grid of size c. Then, the support of IP is at most kM where
M (c+
cc (c+2cA 2cA + 2(3OPT+D
c
Proof (P (Proposition F.6. Fix a center if and consider the cell Z containing the center. The farthest point in the farthest cell in the cluster of f is of distance most OPT +B cA and thus the cells we are concerned with sit in an lx ball of that radius. Not including Z, we walk past at most
OPT +B c
cells along the canonical basis of Rdd hence there are at most
d
1 + 2cA
(8 +B)
such cells.
Notice that the previous discussion and Proposition F.5 reveal a natural trade-off in the choice of the parameter e: by reducing e, we decrease the additive error of our approximation algorithm but we increase the number of samples at a rate (1/))d. Since we have bounded the number of cells from which we can observe a point, the next step is to use a sufficiently large number of samples so that we can estimate replicably whether cell is active or not. This is demonstrated in Algorithm F.2.
42

---

Algorithm F.2 Replicable Active Cells
1: rActiveCells( grid length c):
2: <- O [AknM)//"
3: + AnmM
4: Sample 21....: NN ~ P.
5: Lazy initialize counter Z = 0 for every cell of the grid with length c
6: for 1,..... N do
7: G₁ < cell that samples x; falls into
8: Inerement Z(G,) t Z (G:) +1
9: end for
Choose E [0, m//] uniformly randomly
11: Output all G₁ such that 2(G.)/N 2
Proposition F.7. Suppose Assumption F:2 holds and let F. be the (3. B) appprxximate solution. Let 5, E (0. 1). Then Algorithm F.2 is p-replicable and returns cells of the grid such that least one sample from each cluster of F. falls into these cells with probability at least 1 - 5. Moreover, it has sample complexity
Ô ( n n2kM2 log(1/)
q2p²
Proof (Proposition F.7). Enumerate the k.M cells within the support of and let the counter Z denote the number of samples the j-th cell. Then Z (Z ..... (kM)) follows a multinomial
distribution with parameters (f(1), ..... (.M.))))) (RMM) and N, where p is unknown to us. Nevertheless, by Proposition B.1, for any ε E (0, 1), sampling
In 5/6 + kM In
N 2
2s with probability at least 8/5. From the definition of points implies that (0() pl
N, this equivalent
In 5/8/8 kM In2
m 2
n&M2
By Proposition F.3, we observe at least mM points from each cluster of F, with probability least 1- 8.
Let 5t) P2 be the empirical mean of Z across different executions of Algorithm F.2. Then by the triangle inequality, Σ} (66) [ < 4s.
Since we observe at least Mm points from each cluster of F., there least one point from each cluster with least m observations. Thus outputting the points j such that piJ) 2 for some E [0, m/N] means we always output one point from each cluster.
Now, the outputs between executions are not identical only if the point "splits" some pair P pz
This occurs with probability at most 4f/(m/N) 4Ne/m, To bound this value by 0/s, set
E Σ pm
20N P
20AnM
Plugging this back yields
400AnM In 5 4002nh:M2211
All all, Algorithm F.2 outputs the desired result with probability at least 8 and is p-replicable given
n k (2g)
222
samples from P.
43

---

F.6 Putting it Together
We now have all the ingredients in place to prove Theorem 3.3. The proof follows directly by combining Proposition F.5, and Proposition F.7. Essentially, we replicably estimate the active cells of our grid and then move all the points of each active cell to its center.
For completeness, we also state the formal version of Theorem 3.3.
Theorem F.8 (Theorem 3.3; Formal). Let ó, p E (0, 1). Suppose Assumption F.2 holds and let F. be the (3, B) -apppoxximetet solution. Then, given black-box access to a (3, B appproxxinntoo oracle for Problem 2.4, Algorithm F.2 is p-replicable and returns a solution F such that
<(x, F) < (23 3) OPT +2B + B + (43 23 + 1)cA
with probability at least
5. Moreover, the algorithm requires most n2 l (1/c) 3d log(16)
q²p²
samples from
44


Reviews:
[{"review_id": "DDIChnpaoH", "reviewer": "Reviewer_RdwG", "summary": "This paper initiates the study of formal replicability for clustering algorithms. Replicability is defined to be a property of an algorithm for a statistical clustering problem, requiring that fixing the internal randomness of the algorithm while resampling the input data will yield exactly the same (representation of a) mapping from points to clusters. Upper-bounds are given for the statistical k-means, k-medians, and k-centers problems, with poly(d) sample complexity proven via dimensionality reduction in the case of euclidian distance. The theoretical results are also empirically evaluated on synthetic data.", "questions": "See weaknesses. ", "limitations": "Yes, the authors address all limitations.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "This paper addresses an interesting problem by extending recently introduced formal notions of replicability to the task of statistical clustering. Clustering seems like a setting where replicability is particularly well-motivated, as there are many reasonable desirable properties for clustering solutions, and being able to ensure that solutions which demonstrate a particular balance of these properties are in fact a result of the algorithm generating solutions, and not just a fluke of the sample, is a natural objective.", "weaknesses": "The writing was very readable, but it would have been nice to have a more thorough comparison of this work to the related work. What is the cost of replicable clustering compared to non-replicable algorithms for the same problems? Does this notion of replicability solve any of the issues that motivated the \"stability of centers\" condition for selection of k that is mentioned in Section 1.1?\n\nI also found Section 4 a bit difficult to parse. I think some of the notation used in Algorithm 4.1 isn't defined, and though it's reasonably intuitive, using more precise language in the high-level exposition and fully defining all variables/notation would definitely improve the readability of this section. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the importance of replicability in clustering algorithms. We appreciate the reviewer for their time and effort in reading our paper and address their constructive comments as follows.\n\n> What is the cost of replicable clustering compared to non-replicable algorithms for the same problems? \n\nFor the case of (non-replicable) statistical k-medians/k-means, Ben-David’s work [Ben-David, 2007] implies randomized polynomial-time constant-ratio approximation algorithms with sample complexity $O(k / \\varepsilon^2)$.\nFor the case of (non-replicable) k-centers, ours is the first statistical formulation of the problem. Hence it is not clear how to compare our guarantees.\n\nWe will add these remarks to the next version of the manuscript.\n\n> Does this notion of replicability solve any of the issues that motivated the \"stability of centers\" condition for selection of k that is mentioned in Section 1.1?\n\nWe thank the reviewer for raising an interesting point.\n\nThe work [Ben-David et al., 2006] shows that the “stability of centers” is determined by the symmetry of data and does NOT reflect anything about the choice of k. Therefore, the stability of centers is perhaps not the right criterion for selecting k. Our work reaffirms the conclusions of [Ben-David et al., 2006] in that “stability” can essentially be achieved for any choice of k.\n\nWe believe that exploring more fine-grained notions of replicability, like list replicability and certificate replicability that were proposed recently in [Dixon et al., 2023], and understanding how these quantities vary as a function of k can shed light on this matter.\n\n> I also found Section 4 a bit difficult to parse. I think some of the notation used in Algorithm 4.1 isn't defined, and though it's reasonably intuitive, using more precise language in the high-level exposition and fully defining all variables/notation would definitely improve the readability of this section.\n\nWe thank the reviewer for this suggestion. We will fully define all variables/subroutines used in Algorithm 4.1 within the next version of the manuscript.\n\n[Ben-David, 2007]: Shai Ben-David. A framework for statistical clustering with constant time approximation algorithms for k-median and k-means clustering.\n\n[Ben-David et al., 2006]: Shai Ben-David, Ulrike Von Luxburg, and Da ́vid Pa ́l. A sober look at clustering stability.\n\n[Dixon et al., 2023]: Peter Dixon, A. Pavan, Jason Vander Woude, and N. V. Vinodchandran. List and certificate complexities in replicable learning."}, {"review_id": "J8S4J4gtzL", "reviewer": "Reviewer_Jm7K", "summary": "The topic of this paper is replicable clustering which in high-level asks for design of an algorithm that given two runs of the algorithm on different samples from the “same” input distribution. Replicability is a notion introduced recently in a work by Impagliazzo et al. [2022].  \nThe algorithm for statistical clustering with centroid-based objective with $p$-norm cost. For the case of $k$-means and $k$-medians, via corresponding centroid-based $k$-clustering on “finite” size (referred to as combinatorial clustering) in a black-box manner and design $O(1)$-approximation with poly($d$) sample complexity (but exponential in k?). However, the case of $k$-center is more complicated as it depends on the $\\ell_\\infty$-norm and very sensitive to the largest distance. The results on $k$-center has worse approximation guarantee and sample complexity and requires stronger assumptions. In particular, for the case of $k$-center, the guarantee is bicriterion.\n", "questions": "Please see the questions in the weakness section.", "limitations": "There is no discussion of limitation by the authors. Though, the paper does not seem to have potential negative societal impact.  ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "At high-level, the notion is interesting and related to several other important topics too. The technical contribution of the paper seems to be satisfactory. However, I'm less familiar with the literature and the technicality of the paper might not be significant for someone working on this area or differential privacy.", "weaknesses": "The paper is following the notion of [Impagliazzo et al., 20]; however, I still not quite convinced why the shared randomness assumption is meaningful. A related question is whether an assumption such as Assumption F. 2 can be sufficient for achieving replicability?\n\nThe result seems to heavily depend on the coreset construction of Frahling and Sohler [2005]. Can you elaborate why more recent coreset construction can be applied here?\n\nTheorem 3.1 and Theorem 3.2 provide upper bounds but there is no lowerbound. E.g., Any hope to show exponential dependence on k (or d) is needed (d, for the general metric space)?\n\nHow the algorithm in this paper is different from those for differentially private clustering? \n\nCan you elaborate on the proof of Corollary E. 9? I particular, how does the dimension show up?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in reading our paper and address their comments as follows.\n\n> The paper is following the notion of [Impagliazzo et al., 20]; however, I still not quite convinced why the shared randomness assumption is meaningful. A related question is whether an assumption such as Assumption F. 2 can be sufficient for achieving replicability?\n\nWe thank the reviewer for raising an important point.\n\nSharing the randomness can be thought of as a way to couple the two executions of the algorithm. Kalavasis et al. [2023] consider a notion of “replicability” where there is no need to share internal randomness and the two executions of the algorithm can be coupled in an arbitrary way. They also show that such algorithms can be converted to replicable algorithms by a specific implementation of the internal randomness. However, the conversion still requires sharing of the internal randomness and incurs an exponential computation time in the dimension of the data.\n\nFor our specific setting, shared randomness is crucial in order to achieve the exact same output in two executions of the algorithm (with high probability). For numerical queries such as mean estimation, this property may not mean much, as the utility of the output (distance to true mean) naturally aligns with replicability (distance between two executions). However, for clustering, the utility (objective function) does not correlate with the replicability (difference between outputted centers along two runs). In this case, sharing randomness seems to be required for replicability.\n\nWe show that Assumption F.2 is indeed sufficient for the specific case of k-centers clustering (Theorem F.8).\n\n> The result seems to heavily depend on the coreset construction of Frahling and Sohler [2005]. Can you elaborate why more recent coreset construction can be applied here?\n\nThe coreset construction of Frahling and Sohler [2005] can be adapted to the statistical setting since the algorithm can be viewed as a series of decisions based on heavy hitters estimation. This elementary statistical operation generalizes from the combinatorial setting to the statistical setting. We conjecture that more recent coreset constructions can also be adapted this way if they can be viewed as a series of elementary statistical operations.\n\n> Theorem 3.1 and Theorem 3.2 provide upper bounds but there is no lowerbound. E.g., Any hope to show exponential dependence on k (or d) is needed (d, for the general metric space)?\n\nTo the best of our knowledge, there is little known about lower bounds for statistical clustering, even in the non-replicable setting. We believe this would be an interesting line of future work and will mention this in the next version of our manuscript.\n\n> How the algorithm in this paper is different from those for differentially private clustering?\n\nWe thank the reviewer for raising an important point.\n\nDifferential privacy (DP) provides a worst-case combinatorial guarantee for two runs of the algorithm on neighboring datasets, i.e., two executions when the datasets differ in one element. Moreover, the definition of DP uses the max-divergence as a measure of statistical distance of the two posteriors of the algorithm. On the other hand, the definition of replicability we employ is statistical in the sense that it considers two executions where the inputs are (different) i.i.d. datasets from the same underlying distribution. Notice that the definition of DP does not require that the data come from a distribution, since it considers only one change in the dataset. Another difference is that instead of asking for small max-divergence between the posteriors of the algorithm on the two executions like DP, the definition of replicability requires that the outputs are exactly the same when the internal randomness is shared. Hence, these two definitions are not trivially comparable.\n\nNevertheless, the works of Bun et al. [2023] and Kalavasis et al. [2023] provide connections between DP and replicability for certain classes of problems. Although it is not immediate from these papers, this supports the plausible conjecture that there may be a way to connect the two lines of work in the clustering setting.\n\n> Can you elaborate on the proof of Corollary E. 9? In particular, how does the dimension show up?\n\nWe thank the reviewer for identifying a possible point of confusion. In general, we plan to make the appendix more self-contained in the next version of our manuscript.\n\nThe main basis of our dimensionality reduction scheme is a dimensionality reduction result for the combinatorial setting (Theorem E.1). The dimension shows up as part of the statement of this result. Section E.3 extends this result to the bounded distributional setting as follows: We first discretize the bounded domain to reduce it to a weighted version of the combinatorial setting with Proposition E.8 quantifying the estimation error of such a discretization. Then Corollary E.9 follows by applying Theorem E.1 to the weighted combinatorial case.\n\nAll in all, the dimension shows up since Corollary E.9 is derived from Theorem E.1 which contains the dimension.\n\n[Kalavasis et al., 2023]: Alkis Kalavasis, Amin Karbasi, Shay Moran, Grigoris Velegkas: Statistical indistinguishability of learning algorithms.\n\n[Frahling and Sohler, 2005]: Gereon Frahling and Christian Sohler. Coresets in dynamic geometric data streams.\n\n[Bun et al., 2023]: Mark Bun, Marco Gaboardi, Max Hopkins, Russell Impagliazzo, Rex Lei, Toniann Pitassi, Jessica Sorrell, and Satchit Sivakumar: Stability is stable: Connections between replicability, privacy, and adaptive generalization."}, {"review_id": "rJxdHHZYRC", "reviewer": "Reviewer_fyTj", "summary": "This paper studies the concept of replicability in clustering. This topic is important because replicability is what allows for other researchers to reproduce the results of a study to verify their correctness. This is a big problem these days, 50% of scientists saying that there is a replicability crisis. In the interest of improving the scientific verification process, designing replicable algorithms (i.e., algorithms that function the same when executed twice with the same random seed). This differs from the related notion of clustering stability in that its success is more dependent on the algorithmic design than structures in the data.\n\nIn their setting, points are sampled from a random distribution over the d-dimensional unit ball according to some given metric. It models many norms, including \\ell_p norms. They use Impagliazzo et al.’s notion of rho-replicability: The probability (over every two sampled sets of points and a single random seed) that the algorithm on each set but with the same random seed yields the same output is at least rho.  Outputs to these clustering problems are functions that, given a point, assigns it to a cluster (with no explicit center, just an identifier).\n\nThe main results are new algorithms for statistical k-means and k-medians, with extended improvements when using the Euclidean metric. Their approximation factors both near the approximation factor of a given black-box vanilla algorithm, and their sample complexity on general metrics has an exponential dependence on the dimension (it is sub-exponential in the Euclidean version). They also show an approximation algorithm for the replicable statistical k-centers algorithm. Given an approximation of the form a*OPT+b for k-centers, they give an approximation of the form c*OPT+d, where c and d are linear in both a and b. The query complexity is exponential in the dimensionality.\n\nTheir algorithm uses a quadtree decomposition, which creates a tree structure over all points. The leaves (which correspond to points) are then mapped to cluster centers. So when you receive a query for a point, you traverse the tree to find the leaf which contains it (as it is a partition of the space), and you map it to the corresponding cluster. However, in order to adequately depict the distribution of the space with their decomposition (which they are not given), they require access to an exponential number of queries in d. In the Euclidean case, they can first reduce the dimensionality via Johnson-Lindenstrauss, and map an epsilon-net from the original space to the low-dimensional space. There still is the difficulty of mapping the clustering between these two spaces, but it seems most of those details are in the appendix.\n\nFor replicability, they form a grid, sample points in the space, and map them to the centers of cells to estimate probability mass of each cell. On high-mass cells according to a random threshold, and then they apply a vanilla algorithm.\n\nFinally, they compare k-means++ and k-means++ run on the core produced for their replicability results. It is pretty clear that their approach leads to much greater replicability.\n", "questions": "1. I was a bit confused by the sentence “We emphasize that even though we only observe a sample from P we aim to solve the problem on the whole support of the distribution.” Can you clarify exactly what this means? Is it that the algorithm works on an actual given dataset P but needs to be able to work on all possible datasets? I would think that would be assumed.\n2. How do your algorithms compare to existing algorithms? Is there any baseline implied by any other work?\n", "limitations": "There was no limitations section, though it did not seem needed (but would be preferred).\n", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper is written extremely clearly. I really appreciate how slow and careful the introduction and preliminaries are, which I believe could be followed by someone unfamiliar with the field. For instance, while many of us take for granted that a replicable clustering algorithm optimizes for some utility notion, they take the time to explain this and how it is distinct from their replicability goal (which is a bigger focus of their preliminary section). They also take the time to point out to the reader that their metric function and the ell_p parameter p are different, and give an in-depth but understandable explanation of the intuition behind ell_p for different values of p. The informal presentation at the beginning of results in section 3 is also very nice.\n\nThe results are nice and interesting, but I wouldn’t say they are groundbreaking. The techniques have clear foundations in previous methods, but it also seems unique how they combined them together. I wasn’t sure by reading the paper how their replicability results compare to previous literature.\n", "weaknesses": "I found it very odd that the main focus of the paper introduction and title was replicable k-centers, but they spend most of their time on (nonreplicable) k-centers. It seems like if the highlight of the paper was replicability, they should have focused on the methods used to achieve this. I feel the details of this algorithm were insufficient for a reader invested in replicable k-centers. Additionally, I wasn’t sure how their results compared to any baseline results that exist, so it is difficult to judge the novelty.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort in reading our paper and address their comments as follows.\n\n> I wasn’t sure by reading the paper how their replicability results compare to previous literature.\n> How do your algorithms compare to existing algorithms? Is there any baseline implied by any other work?\n> I wasn’t sure how their results compared to any baseline results that exist, so it is difficult to judge the novelty.\n\nTo the best of our knowledge, we are the first to consider replicability as a theoretical property within the clustering setting. Thus it is not clear if there is any baseline against which we can compare our work.\n\n>I found it very odd that the main focus of the paper introduction and title was replicable k-centers, but they spend most of their time on (nonreplicable) k-centers. It seems like if the highlight of the paper was replicability, they should have focused on the methods used to achieve this. I feel the details of this algorithm were insufficient for a reader invested in replicable k-centers. \n\nAlthough k-centers is a well-studied clustering problem, it is not the only clustering formulation. We also study statistical k-medians/k-means clustering and provide replicable algorithms (Theorem 3.1, Theorem 3.2). Moreover, we derive replicable k-centers algorithms as well (Theorem 3.3). Our title refers to the general clustering problem and not just k-centers. We would appreciate it if the reviewer pointed out the specific parts of the introduction that lead to confusion since we do not explicitly refer to k-centers in the introduction.\n\n> I was a bit confused by the sentence “We emphasize that even though we only observe a sample from P we aim to solve the problem on the whole support of the distribution.” Can you clarify exactly what this means? Is it that the algorithm works on an actual given dataset P but needs to be able to work on all possible datasets? I would think that would be assumed.\n\nWe thank the reviewer for identifying a point of potential confusion. We will clarify this in the next version of our manuscript.\n\nClustering has mainly been studied from the combinatorial point of view, where the distribution is the uniform distribution over some finite points and we are provided the entire distribution. The statistical clustering setting generalizes to arbitrary distributions with only sample access. We wanted to clarify that although we only have access to samples, our output solution should be a good solution for the entire distribution, not just the observed data. \n\n> There was no limitations section, though it did not seem needed (but would be preferred).\n\nWe will add a limitations section to summarize the assumptions that we rely on. We remark that these assumptions are explicitly stated in other sections of the manuscript.\n"}, {"review_id": "dVo2EXXWUU", "reviewer": "Reviewer_nt3r", "summary": "This manuscript focuses on the concept of replicability in statistical clustering algorithms. It introduces the notion of replicability, which refers to the ability of an algorithm to produce consistent results when executed multiple times on different inputs from the same distribution. Replicability is seen as crucial for ensuring the validity and reliability of scientific findings. The manuscript proposes replicable algorithms for three common clustering problems: k-medians, k-means, and k-centers. These algorithms utilize approximation routines for their combinatorial counterparts. The replicable algorithm for statistical Euclidean k-medians achieves a replicable O(1)-approximation with polynomial complexity. Similarly, the algorithm for statistical Euclidean k-centers achieves replicable O(1)-approximation with an additional O(1)-additive error but with exponential sample complexity. The manuscript also discusses the importance of clustering algorithms in unsupervised learning and the lack of a universally agreed-upon definition for the quality of clustering solutions. It highlights the challenges and trade-offs that algorithm designers face due to factors such as random initialization, similarity measures, noise in measurements, and outliers in the dataset. These issues contribute to non-replicable results in clustering algorithms. Furthermore, the manuscript provides related work on norms and parameters used in clustering and presents experimental results on synthetic distributions to validate the proposed replicable algorithms.", "questions": "1] How does the proposed replicable algorithm compare to existing clustering algorithms in terms of performance and accuracy?  \n2] Are there any limitations or assumptions in the proposed replicable algorithms that could affect their applicability in real-world scenarios? \n3] How generalizable are the experimental results presented on synthetic distributions in 2D?  Can the replicable algorithms be applied to other types of datasets?\n4] How sensitive are the replicable algorithms to the choice of parameters such as the choice of exponent in the cost function or the measure of similarity of the data? \n5] Does the study discuss any potential implications or future research directions regarding the replicability of clustering algorithms?  \n", "limitations": "In the experiment, real-world data sets are not used, which is not very convincing.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscripts' innovation lies in proposing replicable clustering algorithms for statistical clustering problems.  It introduces the concept of replicability and highlights its significance in ensuring the validity and reliability of scientific findings.  The document aims to address concerns about replicability in clustering algorithms, which is a crucial topic in the subfields of machine learning and data science.  It presents replicable algorithms for k-medians, k-means, and k-centers problems and outlines their theoretical results.  Overall, the manuscripts' innovation lies in its contribution to the development of replicable algorithms for statistical clustering.", "weaknesses": "There are many theorems that are not well understood, and then there are relatively few experimental parts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the importance of replicability in clustering as well as their time and effort in reading our paper. We address their comments as follows.\n\n> 1] How does the proposed replicable algorithm compare to existing clustering algorithms in terms of performance and accuracy?\n\nOur algorithm relies on a reduction to (weighted) clustering algorithms in the combinatorial setting. For any existing $\\beta$-approximation algorithm, our techniques yield a $\\beta(1+\\varepsilon)$ solution for any $\\varepsilon > 0$.\n\nFor the case of (non-replicable) statistical k-medians/k-means, the work of Ben-David [2007] implies randomized polynomial-time constant-ratio approximation algorithms with sample complexity $O(k / \\varepsilon^2)$. However, these algorithms are non-replicable and we require greater sample complexity to achieve the replicability guarantee.\n\n> 2] Are there any limitations or assumptions in the proposed replicable algorithms that could affect their applicability in real-world scenarios? \n\nFor the case of k-medians/k-means, there is a blow-up in the sample complexity for non-Euclidean metrics that scales exponentially with the dimension. However, we remark that the majority of applications use the Euclidean distance where this exponential blow-up can be avoided using our dimensionality reduction method.\n\n> 3] How generalizable are the experimental results presented on synthetic distributions in 2D? Can the replicable algorithms be applied to other types of datasets?\n\n> In the experiment, real-world data sets are not used, which is not very convincing.\n\nThe experimental results can be generalized to higher dimensional datasets through our dimensionality reduction technique. We presented the experimental results in 2D for simplicity of visualization and implementation.\n\n> 4] How sensitive are the replicable algorithms to the choice of parameters such as the choice of exponent in the cost function or the measure of similarity of the data? \n\nWe thank the reviewer for raising an interesting question. Although we have not explored this explicitly, we expect the algorithm to be quite sensitive to the choice of parameters. For example, with regards to the exponent $p$, there is a non-trivial blow-up in the sample complexity just going from $p=1$ (k-medians) to $p=2$ (k-means). We acknowledge this is a potential direction for future research.\n\n> 5] Does the study discuss any potential implications or future research directions regarding the replicability of clustering algorithms?\n\nWe believe many future directions are possible. For example, determining some lower bounds for statistical clustering, even in the non-replicable case, would be very interesting. Also, it may be possible to improve our sample complexity bounds by considering more recent coreset algorithms. Moreover, It may be fruitful to consider connections with differential privacy and other stability measures. One could also consider the problem of hierarchical clustering. In general, we believe that we have scratched the surface of this problem and hope that it inspires future research. We will add this discussion to the next version of the manuscript.\n\n[Ben-David, 2007]: Shai Ben-David. A framework for statistical clustering with constant time approximation algorithms for k-median and k-means clustering."}, {"review_id": "qMClLNQWyw", "reviewer": "Reviewer_t2BD", "summary": "This paper studies the design of clustering approximation algorithms in the context of statistical clustering under the notion of replicability. In replicable clustering problem, it requires that with high probability, the output of the algorithms should have the exact same partition of the sample space after two executions on different inputs drawn from the same distribution. Given some black-box $\\beta$-approximation algorithm to the $k$-clustering problem, this paper gives an approximation framework that with constant probability, the output of the algorithm can achieves an $O(\\beta)$-approximation satisfying $\\rho$-replicable property.", "questions": "1. Can the authors offer some intuitive insights into how the proposed Replicable Quad Tree ensures replicable estimation?", "limitations": "Since this is a theoretical paper, I don't think this paper has any potential negative societal impact.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper is the first to initiate the study of replicable clustering, which is a recent hot topic in the field of machine learning. In this paper, a new framework is introduced for handling replicable clustering requirements. This paper also establishes sampling complexities for clustering problems in different norms with different metrics. In order to achieve replicability, a novel tree decomposition method called Replicable Quad Tree is designed such that replicable estimation can be obtained based on the tree decomposition constructed. The proposed Replicable Quad Tree is interesting and is quite different from the well-known HST-based tree decomposition method, which could have great potential for designing approximation algorithms for other replicable problems. ", "weaknesses": "1. The sample complexity has exponential dependence on the dimension $d$, which could be the main weakness for this paper when $d$ is large. Although in Euclidean metrics, this can be avoided by dimensionality reduction techniques, for general norms, there is no data-oblivious dimensionality reduction scheme.\n\n2. The main proofs of this paper are hard to follow. It is better to provide intuitive ideas in each section before the proofs such that one could easily follow the main ideas of the proofs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the extensibility of our work and their time and effort in reading our paper. We address their comments as follows.\n\n> The sample complexity has exponential dependence on the dimension \n, which could be the main weakness for this paper when \n is large. Although in Euclidean metrics, this can be avoided by dimensionality reduction techniques, for general norms, there is no data-oblivious dimensionality reduction scheme.\n\nWe thank the reviewer for raising an important point. Although the dimensionality reduction technique only works for the Euclidean distance, we remark that the majority of clustering applications use the Euclidean distance. Therefore, this does not limit the applicability of our algorithms.\n\n> The main proofs of this paper are hard to follow. It is better to provide intuitive ideas in each section before the proofs such that one could easily follow the main ideas of the proofs.\n\nWe thank the reviewer for the thoughtful suggestion and will carry it forward in the next version of our manuscript.\n\n> Can the authors offer some intuitive insights into how the proposed Replicable Quad Tree ensures replicable estimation?\n\nWe can view the quad tree construction algorithm as a series of decisions based on heavy-hitter estimations. This basic statistical operation can be made replicable. Once every decision is replicable, the algorithm also becomes replicable. We will add this discussion to the next version of our manuscript.\n"}]

Author rebuttal:

Meta-review: This work develops efficient clustering algorithms satisfying the recently introduced notion of replicability. Roughly speaking, replicability requires that fixing the internal randomness of the algorithm while resampling the input data will give the same mapping from points to clusters. The paper develops algorithms for $k$-means, $k$-median, and $k$-center, and also provides experimental evaluation. The reviewers agreed that this work is appropriate for publication at NeurIPS.
Decision: Accept (poster)