Title: Replicable Clustering
Venue: NeurIPS 2023
Accepted: yes
Keywords: Theory

Clustering Theory

Statistical Learning Theory

Reproducibility

Replicability
Abstract: We design replicable algorithms in the context of statistical clustering under the recently introduced notion of replicability from Impagliazzo et al. [2022]. According to this definition, a clustering algorithm is replicable if, with high probability, its output induces the exact same partition of the sample space after two executions on different inputs drawn from the same distribution, when its internal randomness is shared across the executions. We propose such algorithms for the statistical $k$-medians, statistical $k$-means, and statistical $k$-centers problems by utilizing approximation routines for their combinatorial counterparts in a black-box manner. In particular, we demonstrate a replicable $O(1)$-approximation algorithm for statistical Euclidean $k$-medians ($k$-means) with $\operatorname{poly}(d)$ sample complexity. We also describe an $O(1)$-approximation algorithm with an additional $O(1)$-additive error for statistical Euclidean $k$-centers, albeit with $\exp(d)$ sample complexity. In addition, we provide experiments on synthetic distributions in 2D using the $k$-means++ implementation from sklearn as a black-box that validate our theoretical results.

Replicable Clustering ∗

Abstract

We design replicable algorithms in the context of statistical clustering under the recently introduced notion of replicability from Impagliazzo et al. [2022]. According to this definition, a clustering algorithm is replicable if, with high probability, its output induces the exact same partition of the sample space after two executions on different inputs drawn from the same distribution, when its internal randomness is shared across the executions. We propose such algorithms for the statistical k -medians, statistical k -means, and statistical k -centers problems by utilizing approximation routines for their combinatorial counterparts in a black-box manner. In particular, we demonstrate a replicable O (1) -approximation algorithm for statistical Euclidean k -medians ( k -means) with ˜ O (poly( k, d ) k log log k ) sample complexity. We also describe an O (1) -approximation algorithm with an additional O (1) -additive error for statistical Euclidean k -centers, albeit with ˜ O (poly( k ) exp( d )) sample complexity. In addition, we provide experiments on synthetic distributions in 2D using the k -means++ implementation from sklearn as a black-box that validate our theoretical results 2 .

1 Introduction

The unprecedented increase in the amount of data that is available to researchers across many different scientific areas has led to the study and development of automated data analysis methods. One fundamental category of such methods is unsupervised learning which aims to identify some inherent structure in unlabeled data. Perhaps the most well-studied way to do that is by grouping together data that share similar characteristics. As a result, clustering algorithms have become one of the central objects of study in unsupervised learning. Despite a very long line of work studying such algorithms, e.g. Jain and Dubes [1988], Hart et al. [2000], Anderberg [2014], there is not an agreed-upon definition that quantifies the quality of a clustering solution. Kleinberg [2002] showed that there is an inherent reason why this is the case: it is impossible to design a clustering function that satisfies three natural properties, namely scale-invariance , richness of solutions , and consistency . This means that the algorithm designer needs to balance several conflicting desiderata. As a result, the radically different approaches that scientists use depending on their application domain can be sensitive to several factors such as their random initialization, the measure of similarity of the data, the presence of noise in the measurements, and the existence of outliers in the dataset. All these issues give rise to algorithms whose results are not replicable , i.e., when we execute them on two different samples of the same population, they output solutions that vary significantly. This begs the

following question. Since the goal of clustering algorithms is to reveal properties of the underlying population, how can we trust and utilize their results when they fail to pass this simple test?

Replicability is imperative in making sure that scientific findings are both valid and reliable. Researchers have an obligation to provide coherent results and conclusions across multiple repetitions of the same experiment. Shockingly, a 2016 survey that appeared in Nature [Baker, 2016] revealed that 70% of researchers tried, but were unable to, replicate the findings of another researcher and more than 50% of them believe there is a significant crisis in replicability. Unsurprisingly, similar worries have been echoed in the subfields of machine learning and data science [Pineau et al., 2019, 2021]. In this work, we initiate the study of replicability in clustering, which is one of the canonical problems of unsupervised learning.

1.1 Related Works

Statistical Clustering. The most relevant previous results for (non-replicable) statistical clustering was established by Ben-David [2007], who designed O (1) -approximation algorithms for statistical k -medians ( k -means) with O ( k ) sample complexity. However, their algorithm picks centers from within the samples and is therefore non-replicable.

Combinatorial Clustering. The flavor of clustering most studied in the approximation algorithms literature is the setting where we have a uniform distribution over finite points and the algorithm has explicit access to the entire distribution. Our algorithms rely on having black-box access to a combinatorial clustering oracle. See Byrka et al. [2017], Ahmadian et al. [2019] for the current best polynomial-time approximation algorithms for combinatorial k -medians ( k -means) in general metrics with approximation ratio 2.675 (9). Also, see Cohen-Addad et al. [2022] for a 2 . 406 ( 5 . 912 ) approximation algorithm for the combinatorial Euclidean k -medians ( k -means).

Clustering Stability. Stability in clustering has been studied both from a practical and a theoretical point of view [Ben-Hur et al., 2001, Lange et al., 2004, V on Luxburg and Ben-David, 2005, Ben-David et al., 2006, Rakhlin and Caponnetto, 2006, Ben-David et al., 2007, Von Luxburg et al., 2010]. In most applications, it is up to the algorithm designer to decide upon the value of k , i.e., the number of different clusters. Thus, it was proposed that a necessary condition it should satisfy is that it leads to solutions that are not very far apart under resampling of the input data [Ben-Hur et al., 2001, Lange et al., 2004]. However, it was shown that this notion of stability for center-based clustering is heavily based on symmetries within the data which may be unrelated to clustering parameters [Ben-David et al., 2006]. Our results differ from this line of work in that we require the output across two separate samples to be exactly the same with high probability, when the randomness is shared. Moreover, our work reaffirms Ben-David et al. [2006] in that their notion of stability can be perfectly attained no matter the choice of k .

Other notions of stability related to our work include robust hierarchical clustering [Balcan et al., 2014], robust online clustering [Lattanzi et al., 2021], average sensitivity [Yoshida and Ito, 2022], and differentially private (DP) clustering [Cohen et al., 2021, Ghazi et al., 2020]. The definition of replicability we use is statistical and relies on an underlying data distribution while (DP) provides a worst-case combinatorial guarantee for two runs of the algorithm on neighboring datasets. Bun et al. [2023], Kalavasis et al. [2023] provide connections between DP and replicability for statistical learning problems. However, these transformations are not computationally efficient. It would be interesting to come up with computationally efficient reductions between replicable and DP clustering algorithms.

Coresets for Clustering. A long line of work has focused on developing strong coresets for various flavors of centroid-based clustering problems. See Sohler and Woodruff [2018] for an overview of this rich line of work. The most relevant for our results include coresets for dynamic geometric streams through hierarchical grids [Frahling and Sohler, 2005] and sampling based techniques [Ben-David, 2007, Feldman and Langberg, 2011, Bachem et al., 2018].

Dimensionality Reduction. Dimensionality reduction for clustering has been a popular area of study as it reduces both the time and space complexity of existing algorithms. The line of work on data-oblivious dimensionality reduction for k -means clustering was initiated by Boutsidis et al. [2010]. The goal is to approximately preserve the cost of all clustering solutions after passing the data through a dimensionality reduction map. This result was later improved and generalized to ( k, p ) -clustering [Cohen et al., 2015, Becchetti et al., 2019], culminating in the work of Makarychev

et al. [2019], whose bound on the target dimension is sharp up to a factor of log 1 / ε . While Charikar and Waingarten [2022] overcome this factor, their result only preserves the cost across the optimal solution.

Replicability in ML. Our results extend the recently initiated line of work on designing provably replicable learning algorithms under the definition that was introduced by Impagliazzo et al. [2022]. Later, Esfandiari et al. [2022] considered a natural adaption of this definition to the setting of bandits and designed replicable algorithms that have small regret. A slightly different notion of replicability in optimization was studied in Ahn et al. [2022], where it is required that an optimization algorithm that uses noisy operations during its execution, e.g., noisy gradient evaluations, outputs solutions that are close when executed twice. Subsequently, Bun et al. [2023], Kalavasis et al. [2023] established strong connections between replicability and other notions of algorithmic stability. Recently, Dixon et al. [2023], Chase et al. [2023] proposed a weaker notion of replicability where the algorithm is not required to output the same solution across two executions, but its output needs to fall into a small list of solutions.

2 Setting & Notation

Let X ⊆ R d be the instance space endowed with a metric κ : X × X → R + and P be a distribution on X which generates the i.i.d. samples that the learner observes.

For F ⊆ R d and x ∈ X , we overload the notation and write F ( x ) := argmin f ∈ F κ ( x, f ) to be the closest point to x in F as well as κ ( x, F ) := κ ( x, F ( x )) to be the shortest distance from x to a point in F .

We assume that X is a subset of B d , the d -dimensional κ -ball of diameter 1 centered about the origin 3 . We also assume that κ is induced by some norm ∥·∥ on R d that is sign-invariant (invariant to changing the sign of a coordinate) and normalized (the canonical basis has unit length). Under these assumptions, the unit ball of κ is a subset of [ -1 , 1] d .

Our setting captures a large family of norms, including the ℓ p -norms, Topℓ norms (sum of ℓ largest coordinates in absolute value), and ordered norms (non-negative linear combinations of Topℓ norms) [Chakrabarty and Swamy, 2019]. Our results hold for more general classes of norms but for the sake of simplicity, we abide by these assumptions.

We define ∆ := sup { κ ( x, y ) : x, y ∈ [0 , 1) d } to be the κ -diameter of the unit hypercube. Note that 1 ≤ ∆ ≤ d by assumption. Moreover, L ∆ is the κ -diameter of a hypercube with side length L . For example, if ∥·∥ = ∥·∥ 2 is the Euclidean norm, then ∆ = √ d .

2.1 Clustering Methods and Generalizations

We now introduce the clustering objectives that we study in this work, which all fall in the category of minimizing a cost function cost : F → R + , where F := { F ⊆ B d : | F | = k } . We write cost( F ) to denote the objective in the statistical setting and ̂ cost( F ) for the combinatorial setting in order to distinguish the two.

Problem 2.1 (Statistical ( k, p ) -Clustering) . Given i.i.d. samples from a distribution P on X ⊆ B d , minimize cost( F ) := E x ∼ P κ ( x, F ) p .

In other words, we need to partition the points into k clusters so that the expected distance of a point to the center of its cluster, measured by κ ( · , · ) p , is minimized. This is closely related to the well-studied combinatorial variant of the ( k, p ) -clustering problem.

Problem 2.2 ( ( k, p ) -Clustering) . Given some points x 1 , . . . , x n ∈ X , minimize ̂ cost( F ) := 1 n ∑ n i =1 κ ( x i , F ) p .

We note that (statistical) k -medians and (statistical) k -means is a special case of Problem 2.2 (Problem 2.1) with p = 1 , 2 , respectively. We also consider a slight variant of the combinatorial problem, i.e., Problem 2.2, where we allow different points x i to participate with different weights w i in the objective. We refer to this problem as the weighted ( k, p ) -clustering problem.

We now shift our attention to the k -centers problem.

Problem 2.3 (Statistical k -Centers) . Given i.i.d. samples from a distribution P on X ⊆ B d , minimize cost( F ) := max x ∈X κ ( x, F ) .

Notice that the ℓ ∞ norm is the limit of the ℓ p norm as p tends to infinity, hence k -centers is, in some sense, the limit of ( k, p ) -clustering as p tends to infinity. Also, notice that this problem differs from k -means and k -medians in the sense that it has a min-max flavor, whereas the other two are concerned with minimizing some expected values. Due to this difference, we need to treat k -centers separately from the other two problems, and we need to make some assumptions in order to be able to solve it from samples (cf. Assumption F.1, Assumption F.2). We elaborate more on that later.

Let us also recall the combinatorial version of k -centers.

Problem 2.4 ( k -Centers) . Given some points x 1 , . . . , x n ∈ X , minimize ̂ cost( F ) := max i ∈ [ n ] κ ( x i , F ) .

We remark that clustering has mainly been studied from the combinatorial point of view, where the distribution is the uniform distribution over some finite points and we are provided the entire distribution. The statistical clustering setting generalizes to arbitrary distributions with only sample access. We emphasize that although we only have access to samples, our output should be a good solution for the entire distribution and not just the observed data.

We write F OPT to denote an optimal solution for the entire distribution and OPT := cost( F OPT ) . Similarly, we write ̂ F OPT to denote an optimal sample solution and ̂ OPT := ̂ cost( ̂ F OPT ) . Suppose we solve Problem 2.4 given a sample of size n from Problem 2.3. Then ̂ OPT ≤ OPT since we are optimizing over a subset of the points.

Recall that a β -approximate solution F is one which has cost cost( F ) ≤ β OPT . Note this is with respect to the statistical version of our problems. An algorithm that outputs β -approximate solutions is known as a β -approximation algorithm . We also say that F is a ( β, B ) -approximate solution if cost( F ) ≤ β OPT+ B .

2.1.1 Parameters p and κ

Here we clarify the difference between p and κ , which are two separate entities in the cost function E x [ κ ( x, F ( x )) p ] . We denote by κ the distance metric used to measure the similarity between points. The most commonly studied and applied option is the Euclidean distance for which our algorithms are the most sample-efficient. On the other hand, p is the exponent to which we raise the distances when computing the cost of a clustering. A smaller choice of p puts less emphasis on points that are far away from centers and p = 1 seeks to control the average distance to the nearest center. A large choice of p puts emphasis on points that are further away from centers and as p tends to infinity, the objective is biased towards solutions minimizing the maximum distance to the nearest center. Thus we can think of k -centers as ( k, p ) -clustering when p = ∞ . As a concrete example, when κ is the Euclidean distance and p = 5 , the cost function becomes E x [ ∥ x -F ( x ) ∥ 5 2 ] .

2.2 Replicability

Throughout this work, we study replicability 4 as an algorithmic property using the definition of Impagliazzo et al. [2022].

Definition 2.5 (Replicable Algorithm; [Impagliazzo et al., 2022]) . Let ρ ∈ (0 , 1) . A randomized algorithm A is ρ -replicable if for two sequences of n i.i.d. samples ¯ X , ¯ Y generated from some distribution P n and a random binary string ¯ r ∼ R ( X ) ,

In the above definition, we treat A as a randomized mapping to solutions of the clustering problem. Thus, even when ¯ X is fixed, A ( ¯ X ) should be thought of as random variable, whereas A ( ¯ X ; ¯ r ) is the realization of this variable given the (fixed) ¯ X, ¯ r. We should think of ¯ r as the shared randomness

between the two executions. In practice, it can be implemented as a shared random seed. We underline that sharing the randomness across executions is crucial for the development of our algorithms. We also note that by doing that we couple the two random variables A ( ¯ X ) , A ( ¯ Y ) , whose realization depends on r ∼ R ( X ) . Thus, if their realizations are equal with high probability under this coupling, it means that the distributions of A ( ¯ X ) , A ( ¯ Y ) are statistically close . This connection is discussed further in Kalavasis et al. [2023].

In the context of a clustering algorithm A , we interpret the output A ( ¯ X ; ¯ r ) as a clustering function f : X → [ k ] which partitions the support of P . The definition of ρ -replicability demands that f is the same with probability at least 1 -ρ across two executions. We note that in the case of centroid-based clustering such as k -medians and k -means, the induced partition is a function of the centers and thus it is sufficient to output the exact same centers with probability 1 -ρ across two executions. However, we also allow for algorithms that create partitions implicitly without computing their centers explicitly.

Our goal is to develop replicable clustering algorithms for k -medians, k -means, and k -centers, which necessitates that the centers we choose are arbitrary points within R d and not only points among the samples. We underline that as in the case of differential privacy, it is trivial to design algorithms that satisfy the replicability property, e.g. we can let A be the constant mapping. The catch is that these algorithms do not achieve any utility . In this work, we are interested in designing replicable clustering algorithms whose utility is competitive with their non-replicable counterparts.

3 Main Results

In this section, we informally state our results for replicable statistical k -medians ( ( k, 1) -clustering), k -means ( ( k, 2) -clustering), and k -centers under general distances. Unfortunately, generality comes at the cost of exponential dependency on the dimension d . We also state our results for replicable statistical k -medians and k -means specifically under the Euclidean distance, which has a polynomial dependency on d . Two key ingredients is the uniform convergence of ( k, p ) -clustering costs (cf. Theorem C.9) as well as a data-oblivious dimensionality reduction technique for ( k, p ) -clustering in the distributional setting (cf. Theorem E.10). These results may be of independent interest.

We emphasize that the Euclidean k -median and k -means are the most studied and applied flavors of clustering, thus the sample complexity for the general case and the restriction to p = 1 , 2 does not diminish the applicability of our approach.

The main bottleneck in reducing the sample complexity for general norms is the lack of a dataoblivious dimensionality reduction scheme. This bottleneck is not unique to replicability and such a scheme for general norms would be immediately useful for many distance-based problems including clustering. It may be possible to extend our results to general ( k, p ) -clustering beyond p = 1 , 2 . The main challenge is to develop an approximate triangle inequality for p -th powers of norms. Again, this limitation is not due to replicability but rather the technique of hierarchical grids. It is a limitation shared by Frahling and Sohler [2005].

Before stating our results, we reiterate that the support of our domain X is a subset of the unit-diameter κ -ball B d . In particular, we have that OPT ≤ 1 .

Theorem 3.1 (Informal) . Let ε, ρ ∈ (0 , 1) . Given black-box access to a β -approximation oracle for weighted k -medians, respectively weighted k -means (cf. Problem 2.2), there is a ρ -replicable algorithm for statistical k -medians, respectively k -means (cf. Problem 2.1), such that with probability at least 0 . 99 , it outputs a (1 + ε ) β -approximation. Moreover, the algorithm has sample complexity

When we are working in Euclidean space, we can get improved results for these problems.

Theorem 3.2 (Informal) . Let ρ ∈ (0 , 1) . Suppose we are provided with black-box access to a β -approximation oracle for weighted Euclidean k -medians ( k -means). Then there is a ρ -replicable algorithm that partitions the input space so with probability at least 0 . 99 , the cost of the partition is

at most O ( β OPT) . Moreover, the algorithm has sample complexity

We underline that in this setting we compute an implicit solution to Problem 2.1, since we do not output k centers. Instead, we output a function f that takes as input a point x ∈ X and outputs the label of the cluster it belongs to in polynomial time. The replicability guarantee states that, with probability 1 -ρ , the function will be the same across two executions.

The combinatorial k -medians ( k -means) problem where the centers are restricted to be points of the input is a well-studied problem from the perspective of polynomial-time constant-factor approximation algorithms. See Byrka et al. [2017], Ahmadian et al. [2019] for the current best polynomial-time approximation algorithms for combinatorial k -medians ( k -means) in general metrics with approximation ratio 2.675 (9). Also, see Cohen-Addad et al. [2022] for a 2 . 406 ( 5 . 912 ) approximation algorithm for the combinatorial Euclidean k -medians ( k -means).

As we alluded to before, in order to solve k -centers from samples, we need to make an additional assumption. Essentially, Assumption F.2 states that there is a ( β, B ) -approximate solution F , such that, with some constant probability, e.g. 0 . 99 , when we draw n samples from P we will observe at least one sample from each cluster of F .

Theorem 3.3 (Informal) . Let c ∈ (0 , 1) . Given black-box access to a ( ˆ β, ˆ B ) -approximation oracle for k -centers (cf. Problem 2.4) and under Assumption F .2, there is a ρ -replicable algorithm for statistical k -centers (cf. Problem 2.3), that outputs a ( O ( β + ˆ β ) , O ( B + ̂ B +( β + ˆ β +1) c )∆) -approximate solution with probability at least 0 . 99 . Moreover, it has sample complexity

Recall that there is a simple greedy 2-approximation for the sample k -center problem whose approximation ratio cannot be improved unless P = NP [Hochbaum and Shmoys, 1985].

We defer the discussion around k -centers to Appendix F. In particular, see Theorem F.8 in Appendix F for the formal statement of Theorem 3.3.

4 Overview of ( k, p ) -Clustering

In this section, we present our approach to the ( k, p ) -clustering problem. First, we replicably approximate the distribution with a finite set of points by extending the approach of Frahling and Sohler [2005] to the distributional setting. Then, we solve the combinatorial ( k, p ) -clustering problem on this coreset using an approximation oracle in a black-box manner. In the following subsections, we give a more detailed overview for each step of our approach. For the full proofs and technical details, we kindly refer the reader to Appendix D.4 - D.6. In summary:

Replicably build a variant of a quad tree [Finkel and Bentley, 1974] (cf. Section 4.2).

Replicably produce a weighted coreset using the quad tree (cf. Section 4.1).

Apply the optimization oracle for the combinatorial problem on the coreset.

For general norms, this approach leads to an exponential dependence on d . However, we are able to handle the case of Euclidean distances by extending existing dimensionality reduction techniques for sample Euclidean ( k, p ) -clustering [Makarychev et al., 2019] to the distributional case (cf. Section 5). Thus, for the widely used Euclidean norm, our algorithm has poly( d ) sample complexity.

4.1 Coresets

Definition 4.1 ((Strong) Coresets) . For a distribution P with support X ⊆ B d and ε ∈ (0 , 1) , a (strong) ε -coreset for X is a distribution P ′ on X ′ ⊆ B d which satisfies

for every set of centers F ⊆ B d , | F | = k .

Essentially, coresets help us approximate the true cost on the distribution P by considering another distribution P ′ whose support X ′ can be arbitrarily smaller than the support of P .

Inspired by Frahling and Sohler [2005], the idea is to replicably consolidate our distribution P through some mapping R : X → X whose image has small cardinality | R ( X ) | << ∞ so that for any set of centers F ,

where ε ∈ (0 , 1) is some error parameter. In other words, ( P R , R ( X )) is an ε -coreset. Note that given the function R , we can replicably estimate the probability mass at each point in R ( X ) and then apply a weighted ( k, p ) -clustering algorithm.

4.2 Replicable Quad Tree

We now explain how to replicably obtain the mapping R : X → X by building upon the work of Frahling and Sohler [2005]. The pseudocode of the approach is provided in Algorithm 4.1. While Frahling and Sohler [2005] present their algorithm using hierarchical grids, we take an alternative presentation using the quad tree [Finkel and Bentley, 1974], which could be of independent interest.

First, we recall the construction of a standard quad tree in dimension d . Suppose we have a set of n points in [ -1 / 2 , 1 / 2 ] d . The quad tree is a tree whose nodes represent hypercubes containing points and can be built recursively as follows: The root represents the cell [ -1 / 2 , 1 / 2 ] d and contains all points. If a node contains more than one point, we split the cell it represents into 2 d disjoint, equally sized cells. For each non-empty cell, we add it as a child of the current node and recurse into it. The recursion stops when a node contains only 1 point. In the distributional setting, the stopping criterion is either when the diameter of a node is less than some length or when the node contains less than some mass, where both quantities are a function of the depth of the node. See Algorithm 4.1.

A quad tree implicitly defines a function R : X → X as follows. Given a point x ∈ X and the root node of our tree, while the current node has a child, go to the child containing x if such a child exists, otherwise, go to any child. At a leaf node, output the center of the cell the leaf represents. Intuitively, the quad tree consolidates regions of the sample space into single points. The construction can be made replicable since the decision to continue the recursion or not is the only statistical operation and is essentially a heavy-hitters operations which can be performed in a replicable fasion.

Let G i denote the union of all 2 id possible cells at the i -th level. We write P i to denote the discretized distribution to G i . In other words, P i = P | σ ( G i ) is the restriction of P to the smallest σ -algebra containing G i . Moreover, we write Λ to denote a replicable estimate of OPT with relative error ε , say 1 / β (1+ ε ) OPT ≤ Λ ≤ (1 + ε ) OPT for some absolute constant β ≥ 1 . We demonstrate how to obtain such a replicable estimate in Appendix D.5.

̸

Our technique differs from that of Frahling and Sohler [2005] in at least three ways. Firstly, they performed their analysis for finite, uniform distributions with access to the entire distribution, while

our results hold assuming only sample access to a general bounded distribution 5 . Secondly, Frahling and Sohler [2005] bound the number of layers as a function of the cardinality of the support. For us, this necessitates the extra termination condition when the side lengths of our grids fall below a fraction of Λ as our distribution may have infinite support. Finally, Frahling and Sohler [2005] estimate OPT by enumerating powers of 2. This suffices for their setting since their distributions are discrete and bounded. However, we require a more nuanced approach (cf. Appendix D.5) as we do not have a lower bound for OPT . We tackle this by showing uniform convergence of the clustering solution costs, which we establish via metric entropy and Rademacher complexity (cf. Appendix C).

4.3 Putting it Together

Once we have produced the function R implicitly through a quad tree, there is still the matter of extracting a replicable solution from a finite distribution. We can accomplish this by replicably estimating the probability mass at each point of R ( X ) and solving an instance of the weighted sample k -medians ( k -means). This leads to the following results. For details, see Appendix D.4 - D.6.

Theorem 4.2 (Theorem 3.1; Formal) . Let ε, ρ ∈ (0 , 1) and δ ∈ (0 , ρ / 3 ) . Given black-box access to a β -approximation oracle for weighted k -medians (cf. Problem 2.2), there is a ρ -replicable algorithm for statistical k -medians (cf. Problem 2.1) such that, with probability at least 1 -δ , it outputs a (1 + ε ) β -approximation. Moreover, it has sample complexity

Theorem 4.3 (Theorem 3.1; Formal) . Given black-box access to a β -approximation oracle for weighted k -means (cf. Problem 2.2), there is a ρ -replicable algorithm for statistical k -means (cf. Problem 2.1) such that, with probability at least 1 -δ , it replicably outputs a (1 + ε ) β -approximation. Moreover, it has sample complexity

5 The Euclidean Metric, Dimensionality Reduction, and ( k, p ) -Clustering

In this section, we focus on the Euclidean metric κ ( y, z ) := ∥ y -z ∥ 2 and show how the sample complexity for Euclidean ( k, p ) -clustering can be made polynomial in the ambient dimension d. Note that results for dimensionality reduction exist for the combinatorial Euclidean ( k, p ) -clustering problem [Charikar and Waingarten, 2022, Makarychev et al., 2019]. However, these do not extend trivially to the distributional case. We remark that we require our dimensionality reduction maps to be data-oblivious since we are constrained by replicability requirements.

The cornerstone result in dimensionality reduction for Euclidean distance is the Johnson-Lindenstrauss lemma (cf. Theorem E.1), which states that there is a distribution over linear maps π d,m from R d → R m that approximately preserves the norm of any x ∈ R d with constant probability for some target dimension m . In Makarychev et al. [2019], the authors prove Theorem E.4, which roughly states that it suffices to take m = ˜ O ( p 4 / ε 2 log( 1 / δ )) in order to preserve ( k, p ) -clustering costs when the domain is finite. Firstly, we extend Theorem E.4 to the distributional setting by implicitly approximating the distribution with a weighted ε -net. Then, we implicitly map the ε -net onto the low-dimensional space and solve the clustering problem there. An important complication we need to overcome is that this mapping preserves the costs that correspond to partitions 6 of the data and not arbitrary solutions. Because of that, it is not clear how we can 'lift' the solution from the low-dimensional space to the original space. Thus, instead of outputting k points that correspond to the centers of the clusters, our algorithm outputs a clustering function f : X → [ k ] , which takes as input a point x ∈ X and returns the label of the cluster it belongs to. The replicability guarantees of the algorithm state that, with probability 1 -ρ , it will output the same function across two executions. In Appendix E.4, we describe this function. We emphasize that for each x ∈ X , the running time of f is polynomial in k, d, p, 1 / ε , log( 1 / δ ) . Essentially, this function maps x onto the low-dimensional

space using the same projection map π as our algorithm and then finds the nearest center of π ( x ) in the low-dimensional space. For full details, we refer the reader to Appendix E.3. We are now ready to state the result formally.

Theorem 5.1 (Theorem 3.2; Formal) . Let ε, ρ ∈ (0 , 1) and δ ∈ (0 , ρ / 3 ) . Given a β -approximation oracle for weighted Euclidean k -medians ( k -means), there is a ρ -replicable algorithm that outputs a clustering function such that with probability at least 1 -δ , the cost of the partition is at most (1 + ε ) β OPT . Moreover, the algorithm has sample complexity

where m = O ( 1 ε 2 log k δε ) .

6 Running Time for ( k, p ) -Clustering

All of our algorithms terminate in O (poly( n )) time where n denotes the sample complexity. See Table A.1 for more detailed time complexity of each stage of our approach. Moreover, we make O (log( β / ( ε OPT) )) calls to the β -approximation oracle for the combinatorial ( k, p ) -clustering problem within our OPT estimation subroutine and then one more call to the oracle in order to output a solution on the coreset.

7 Replicable k -Centers

Due to space limitations, we briefly sketch our approach for the k -centers problem and kindly refer the reader to Appendix F. As explained before, the assumptions we make in this setting state that there exists some 'good' solution, so that when we draw n i.i.d. samples from P we observe at least one sample from each cluster, with constant probability. We first take a fixed grid of side c in order to cover the unit-diameter ball. Then, we sample sufficiently many points from P . Subsequently, we 'round' all the points of the sample to the centers of the cells of the grid that they fall into and estimate the probability mass of every cell. In order to ensure replicability, we take a random threshold from a predefined interval and discard the points from all the cells whose mass falls below the threshold. Finally, we call the approximation oracle using the points that remain. Unlike the ( k, p ) -clustering problem (cf. Problem 2.1), to the best of our knowledge, there does not exist any dimensionality reduction techniques that apply to the k -centers problem. The main result is formally stated in Theorem F.8.

8 Experiments

We now provide a practical demonstration 7 of the replicability of our approach on synthetic data in 2D. In Figure 8.1, we leverage the sklearn [Pedregosa et al., 2011] implementation of the popular k -means++ algorithm for k = 3 and compare the output across two executions on the two moons distribution. In the first experiment, we do not perform any preprocessing and run k -means++ as is, resulting in different centers across two executions. In the second experiment, we compute a replicable coreset for the two moons distribution before running k -means++ on the coreset. This leads to the same centers being outputted across the executions. Note that the computation for the coreset is performed independently for each execution, albeit with a shared random seed for the internal randomness. See also Figure 8.2 for the results of a similar experiment on a mixture of truncated Gaussian distributions.

ReplicableK-Means++ on Two Moons Distribution

Execution 1

Execution 2

1.00

1.00

0.75

0.75

0.50

0.50

0.25

0.25

Samples

Non-Replicable Centers

0.00

0.00

Replicable Centers

0.25

0.25

Y

Coreset

0.50

0.50

0.75

0.75

-1.00 -

-1.00

-1.000.75 0.500.25

0.00

0.25

0.50

0.75

1.00

1.000.750.500.25

0.00

0.25

0.50

0.75

1.00

ReplicableK-Means++onTruncated GaussianMixture

Execution 1

Execution2

1.00

1.00

0.75

0.75

0.50

0.50

0.25

0.25

Samples

Non-Replicable Centers

0.00

0.00

Replicable Centers

0.25

0.25

Y

Coreset

0.50

0.50

0.75

0.75

-1.00

-1.00

-1.000.750.50-0.25

0.00

0.25

0.50

0.75

1.00

1.000.750.500.25

0.00

0.25

0.50

0.75

1.00

9 Conclusion & Future Work

In this work, we designed replicable algorithms with strong performance guarantees using black-box access to approximation oracles for their combinatorial counterparts. There are many follow-up research directions that this work can lead to. For instance, our coreset algorithm adapts the coreset algorithm of Frahling and Sohler [2005] by viewing their algorithm as a series of heavy hitter estimations that can be made replicable. it may be possible to interpret more recent approaches for coreset estimation as a series of statistical operations to be made replicable in order to get replicable algorithms in the statistical ( k, p ) -clustering setting [Hu et al., 2018]. It would also be interesting to examine the sensitivity of (replicable) clustering algorithms to the choice of parameters such as the choice of exponent in the cost function or the measure of similarity of the data. Another relevant direction is to explore sample complexity lower ounds for statistical clustering, where little is known even in the non-replicable setting.

Acknowledgments and Disclosure of Funding

Amin Karbasi acknowledges funding in direct support of this work from NSF (IIS-1845032), ONR (N00014-19-1-2406), and the AI Institute for Learning-Enabled Optimization at Scale (TILOS). Grigoris Velegkas is supported by TILOS, the Onassis Foundation, and the Bodossaki Foundation. Felix Zhou is supported by TILOS.

References

Sara Ahmadian, Ashkan Norouzi-Fard, Ola Svensson, and Justin Ward. Better guarantees for kmeans and euclidean k-median by primal-dual algorithms. SIAM Journal on Computing , 49(4): FOCS17-97, 2019.

Kwangjun Ahn, Prateek Jain, Ziwei Ji, Satyen Kale, Praneeth Netrapalli, and Gil I Shamir. Reproducibility in optimization: Theoretical framework and limits. arXiv preprint arXiv:2202.04598 , 2022.

Michael R Anderberg. Cluster analysis for applications: probability and mathematical statistics: a series of monographs and textbooks , volume 19. Academic press, 2014.

Olivier Bachem, Mario Lucic, and Andreas Krause. Scalable k-means clustering via lightweight coresets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1119-1127, 2018.

Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature , 533(7604), 2016.

Maria-Florina Balcan, Yingyu Liang, and Pramod Gupta. Robust hierarchical clustering. The Journal of Machine Learning Research , 15(1):3831-3871, 2014.

Luca Becchetti, Marc Bury, Vincent Cohen-Addad, Fabrizio Grandoni, and Chris Schwiegelshohn. Oblivious dimension reduction for k-means: beyond subspaces and the johnson-lindenstrauss lemma. In Proceedings of the 51st annual ACM SIGACT symposium on theory of computing , pages 1039-1050, 2019.

Shai Ben-David. A framework for statistical clustering with constant time approximation algorithms for k-median and k-means clustering. Machine Learning , 66(2):243-257, 2007.

Shai Ben-David, Ulrike Von Luxburg, and D´ avid P´ al. A sober look at clustering stability. In International conference on computational learning theory , pages 5-19. Springer, 2006.

Shai Ben-David, D´ avid P´ al, and Hans Ulrich Simon. Stability of k-means clustering. In International conference on computational learning theory , pages 20-34. Springer, 2007.

Asa Ben-Hur, Andre Elisseeff, and Isabelle Guyon. A stability based method for discovering structure in clustered data. In Biocomputing 2002 , pages 6-17. World Scientific, 2001.

Christos Boutsidis, Anastasios Zouzias, and Petros Drineas. Random projections for k -means clustering. Advances in neural information processing systems , 23, 2010.

Mark Bun, Marco Gaboardi, Max Hopkins, Russell Impagliazzo, Rex Lei, Toniann Pitassi, Jessica Sorrell, and Satchit Sivakumar. Stability is stable: Connections between replicability, privacy, and adaptive generalization. arXiv preprint arXiv:2303.12921 , 2023.

Jarosław Byrka, Thomas Pensyl, Bartosz Rybicki, Aravind Srinivasan, and Khoa Trinh. An improved approximation for k-median and positive correlation in budgeted optimization. ACM Transactions on Algorithms (TALG) , 13(2):1-31, 2017.

Bernd Carl and Irmtraud Stephani. Entropy, compactness and the approximation of operators . Cambridge University Press, 1990.

Deeparnab Chakrabarty and Chaitanya Swamy. Approximation algorithms for minimum norm and ordered optimization problems. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , pages 126-137, 2019.

Moses Charikar and Erik Waingarten. The johnson-lindenstrauss lemma for clustering and subspace approximation: From coresets to dimension reduction. arXiv preprint arXiv:2205.00371 , 2022.

Zachary Chase, Shay Moran, and Amir Yehudayoff. Replicability and stability in learning. arXiv preprint arXiv:2304.03757 , 2023.

Edith Cohen, Haim Kaplan, Yishay Mansour, Uri Stemmer, and Eliad Tsfadia. Differentially-private clustering of easy instances. In International Conference on Machine Learning , pages 2049-2059. PMLR, 2021.

Michael B Cohen, Sam Elder, Cameron Musco, Christopher Musco, and Madalina Persu. Dimensionality reduction for k-means clustering and low rank approximation. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing , pages 163-172, 2015.

Vincent Cohen-Addad, Hossein Esfandiari, Vahab Mirrokni, and Shyam Narayanan. Improved approximations for euclidean k -means and k -median, via nested quasi-independent sets. arXiv preprint arXiv:2204.04828 , 2022.

Ronald A DeVore and George G Lorentz. Constructive approximation , volume 303. Springer Science &Business Media, 1993.

Peter Dixon, A Pavan, Jason Vander Woude, and NV Vinodchandran. List and certificate complexities in replicable learning. arXiv preprint arXiv:2304.02240 , 2023.

Hossein Esfandiari, Alkis Kalavasis, Amin Karbasi, Andreas Krause, Vahab Mirrokni, and Grigoris Velegkas. Reproducible bandits. arXiv preprint arXiv:2210.01898 , 2022.

Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Proceedings of the forty-third annual ACM symposium on Theory of computing , pages 569-578, 2011.

Raphael A Finkel and Jon Louis Bentley. Quad trees a data structure for retrieval on composite keys. Acta informatica , 4:1-9, 1974.

Gereon Frahling and Christian Sohler. Coresets in dynamic geometric data streams. In Proceedings of the thirty-seventh annual ACM symposium on Theory of computing , pages 209-217, 2005.

Badih Ghazi, Ravi Kumar, and Pasin Manurangsi. Differentially private clustering: Tight approximation ratios. Advances in Neural Information Processing Systems , 33:4040-4054, 2020.

Peter E Hart, David G Stork, and Richard O Duda. Pattern classification . Wiley Hoboken, 2000.

Dorit S Hochbaum and David B Shmoys. A best possible heuristic for the k-center problem. Mathematics of operations research , 10(2):180-184, 1985.

Wei Hu, Zhao Song, Lin F Yang, and Peilin Zhong. Nearly optimal dynamic k -means clustering for high-dimensional data. arXiv preprint arXiv:1802.00459 , 2018.

Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell. Reproducibility in learning. arXiv preprint arXiv:2201.08430 , 2022.

Anil K Jain and Richard C Dubes. Algorithms for clustering data . Prentice-Hall, Inc., 1988.

Svante Janson. Tail bounds for sums of geometric and exponential variables. Statistics & Probability Letters , 135:1-6, 2018.

William B Johnson. Extensions of lipschitz mappings into a hilbert space. Contemp. Math. , 26: 189-206, 1984.

Alkis Kalavasis, Amin Karbasi, Shay Moran, and Grigoris Velegkas. Statistical indistinguishability of learning algorithms. arXiv preprint arXiv:2305.14311 , 2023.

Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM (JACM) , 45(6):983-1006, 1998.

Jon Kleinberg. An impossibility theorem for clustering. Advances in neural information processing systems , 15, 2002.

Tilman Lange, Volker Roth, Mikio L Braun, and Joachim M Buhmann. Stability-based validation of clustering solutions. Neural computation , 16(6):1299-1323, 2004.

Silvio Lattanzi, Benjamin Moseley, Sergei Vassilvitskii, Yuyan Wang, and Rudy Zhou. Robust online correlation clustering. Advances in Neural Information Processing Systems , 34:4688-4698, 2021.

Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes , volume 23. Springer Science & Business Media, 1991.

Konstantin Makarychev, Yury Makarychev, and Ilya Razenshteyn. Performance of johnsonlindenstrauss transform for k-means and k-medians clustering. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , pages 1027-1038, 2019.

Vitali D Milman and Gideon Schechtman. Asymptotic Theory of Finite Dimensional Normed Spaced , volume 1200. Springer Berlin, 1986.

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Joelle Pineau, Koustuv Sinha, Genevieve Fried, Rosemary Nan Ke, and Hugo Larochelle. Iclr reproducibility challenge 2019. ReScience C , 5(2):5, 2019.

Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivi` ere, Alina Beygelzimer, Florence d'Alch´ e Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research: a report from the neurips 2019 reproducibility program. Journal of Machine Learning Research , 22, 2021.

Gilles Pisier. The volume of convex bodies and Banach space geometry , volume 94. Cambridge University Press, 1999.

Alexander Rakhlin and Andrea Caponnetto. Stability of k -means clustering. Advances in neural information processing systems , 19, 2006.

Christian Sohler and David P Woodruff. Strong coresets for k-median and subspace approximation: Goodbye dimension. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) , pages 802-813. IEEE, 2018.

AWvan der Vaart and Jon A Wellner. Weak convergence and empirical processes with applications to statistics. Journal of the Royal Statistical Society-Series A Statistics in Society , 160(3):596-608, 1997.

Ulrike Von Luxburg and Shai Ben-David. Towards a statistical theory of clustering. In Pascal workshop on statistics and optimization of clustering , pages 20-26. London, UK, 2005.

Ulrike Von Luxburg et al. Clustering stability: an overview. Foundations and Trends® in Machine Learning , 2(3):235-274, 2010.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge University Press, 2019.

Yuichi Yoshida and Shinji Ito. Average sensitivity of euclidean k-clustering. In Advances in Neural Information Processing Systems , 2022.

A Tables

We present the running times required for our sub-routines in order to ensure each stage is ρ -replicable and succeeds with probability at least 0.99. The overall output is a constant ratio approximation for Euclidean k -medians ( k -means).

B Useful Facts

Proposition B.1 (Bretagnolle-Huber-Carol Inequality; [Vaart and Wellner, 1997]) . Suppose the random vector ( Z (1) , . . . , Z ( N ) ) is multinomially distributed with parameters ( p (1) , . . . , p ( N ) ) and n . Let ̂ p ( j ) := 1 n Z ( j ) . Then

In particular, for any ε, ρ ∈ (0 , 1) , sampling

points from a finite distribution implies that ∑ N j =1 | ̂ p ( j ) -p ( j ) | < 2 ε with probability at least 1 -ρ .

Remark B.2 (Weighted k -Means/Medians) . We remark that if we have access to a β -approximation oracle for unweighted k -means/medians, we can implement a weighted one by considering multiple copies of the points. In particular, in our applications, we get weights that are polynomials in the parameters of concern so this will not affect the stated runtime guarantees.

C Uniform Convergence of ( k, p ) -Clustering Costs

We would like to estimate OPT for statistical k -medians and statistical k -means by solving the combinatorial problem on a sufficiently large sample size. However, while the convergence of cost for a particular set of centers is guaranteed by standard arguments, e.g. Chernoff bounds, we require the stronger statement that convergence holds simultaneously for all possible choices of centers.

Similar results can be found [Ben-David, 2007], with the limitation that centers are either chosen from the sample points, or chosen as centers of mass of the clusters they induce. While this suffices to achieve constant ratio approximations, we would like to choose our centers anywhere in B d . One reason for doing so is for replicability as we have no control over the location of samples from two independent executions so we need to output centers that are not overly reliant on its specific input. Another is for dimensionality reduction, which we will see later.

C.1 History of Uniform Laws of Large Numbers

Uniform laws of large numbers generalize convergence results for a finite number of random variables to possibly uncountable classes of random variables. The Rademacher complexity and related Gaussian complexity are canonical techniques in developing these laws and also have a lengthy history in the study of Banach spaces using probabilistic methods [Pisier, 1999, Milman and Schechtman,

1986, Ledoux and Talagrand, 1991]. Metric entropy, along with related notions of expressivity of various function classes, can be used to control the Rademacher complexity and are also central objects of study in the field of approximation theory [DeVore and Lorentz, 1993, Carl and Stephani, 1990].

C.2 Rademacher Complexity

Let F denote a class of functions from R d → R . For any fixed collection of n points x n 1 := ( x 1 , . . . , x n ) , we write

i.e, the restriction of F onto the sample. Let s ∈ {-1 , 1 } n be a Rademacher random vector. That is, s i takes on values -1 , 1 with equal probability independently of other components. Recall that the Rademacher complexity of F is given by

Theorem C.1 ([Wainwright, 2019]) . For any b -uniformly bounded class of functions F , integer n ≥ 1 , and error ε ≥ 0 ,

with probability at least 1 -exp ( -nε 2 2 b 2 ) .

In particular, as long as R n ( F ) = o (1) , we have uniform convergence of the sample mean.

C.3 Metric Entropy

We write B ( x, r ; µ ) to denote the ball of radius r about a point x with respect to a metric µ . Recall that an ε -cover of a set T with respect to a metric µ is a subset θ 1 , . . . , θ N ⊆ T such that

The covering number N ( ε ; T, µ ) is the cardinality of the smallest ε -cover.

Proposition C.2 ([Wainwright, 2019]) . Fix ε ∈ (0 , 1) . Let B ( R ; ∥·∥ ) denote the d -dimensional ball of radius R with respect to ∥·∥ . Then

We say a collection of zero-mean random variables { Y θ : θ ∈ T } is a sub-Gaussian process with respect to a metric µ on T if

for all θ, θ ′ ∈ T and λ ∈ R .

Proposition C.3. The canonical Rademacher process

is a zero-mean sub-Gaussian process with respect to the Euclidean norm on T ⊆ R n .

Proof. Recall that a Rademacher variable is sub-Gaussian with parameter 1. Moreover, the sum of sub-Gaussian variables is sub-Gaussian with parameter equal to the Euclidean norm of the parameters. It follows that Y θ -Y θ ′ = ⟨ s, θ -θ ′ ⟩ is sub-Gaussian with parameter ∥ θ -θ ′ ∥ 2 . The result follows.

Theorem C.4 (One-Step Discretization; [Wainwright, 2019]) . Let Y θ , θ ∈ T be a zero-mean subGaussian process with respect to the metric µ . Fix any ε ∈ [0 , D ] where D = diam( T ; µ ) such that N ( ε ; T, µ ) ≥ 10 . Then

In general, it may be possible to improve the bound in Theorem C.4 using more sophisticated techniques such as Dudley's entropy integral bound [Wainwright, 2019]. However, the simple inequality from Theorem C.4 suffices for our setting.

Corollary C.5. Fix any ε ∈ [0 , D ] where D = diam( T ; µ ) such that N ( ε ; T, µ ) ≥ 10 . The canonical Rademacher process Y θ = ⟨ s, θ ⟩ for θ ∈ T ⊆ R n satisfies

Proof. By Proposition C.3, Y θ is a zero-mean sub-Gaussian process and we can apply Theorem C.4 to conclude that

Corollary C.5 gives us a way to control the Rademacher complexity of a function class whose co-domain is well-behaved. We make this notion rigorous in the next section.

C.4 Uniform Convergence of ( k, p ) -Clustering Cost

For a fixed set of centers F , let κ p F : R d → R be given by

Define F := { F ⊆ B d : | F | = k } . We take our function class to be

Proposition C.6. µ is a metric on F .

Proof. It is clear that µ is symmetric and positive definite. We need only show that the triangle inequality holds. Fix A,B,C ∈ F and suppose a ∈ A,c ∈ C are such that κ ( a, c ) = µ ( A,C ) . Then

as desired.

Proposition C.7. F is p -Lipschitz parameterized with respect to the metric µ .

Let µ : F × F → R be given by

Proof. We have

Now, the function g ( x ) : [0 , 1] → R given by x ↦→ x p is p -Lipschitz by the mean value theorem:

The result follows by the fact that the composition of Lipschitz functions is Lipschitz with a constant equal to the product of constants from the composed functions.

Theorem C.8. For any ε ∈ [0 , D = √ n ] such that N ( ε ; F ( x n 1 ) , ∥·∥ 2 ) ≥ 10 ,

Proof. Remark that κ p F ( x n 1 ) is √ np -Lipschitz with respect to the metric µ . Indeed,

Now, D := diam( F ( x n 1 ); ∥·∥ 2 ) = √ n . We can apply Corollary C.5 with T = F ( x n 1 ) to see that

Now, since κ p F ( x n 1 ) is √ np -Lipschitz, a ε √ np -cover for F yields an ε -cover of F ( x n 1 ) . Hence

Note that the cross product of k ε -covers of B is an ε -cover of F . Hence

Substituting this bound on the covering number of F ( x n 1 ) concludes the proof.

Note that the Lipschitz property was crucial to ensure that the exponent in the covering number does not contain n .

Theorem C.9. Fix ε, δ ∈ (0 , 1) . Then with

with probability at least 1 -δ for any set of centers F ⊆ B d .

i.i.d. samples from P ,

Proof. Choose ε := 3 ≤ D = √ n . By Theorem C.8,

Fix some ε 1 ∈ (0 , 1) . We have

Similarly, we have

Finally,

By taking the maximum of the three lower bounds, we conclude that

Fix δ ∈ (0 , 1) . Observe that F is 1 -uniformly bounded. Thus by Theorem C.1, we require

samples in order to guarantee that

with probability at least 1 -δ .

Choosing ε 1 = ε 7 concludes the proof.

D ( k, p ) -Clustering

In this section, we provide the full proofs for Theorem 4.2 and Theorem 4.3, which state the guarantees for the replicable k -medians and k -means algorithms, respectively. First, we describe a

useful subroutine for replicable heavy hitters estimation in Appendix D.2. This subroutine is crucial to the replicable coreset algorithm (cf. Algorithm D.3), which we analyze in Appendix D.3. Once we have a coreset, it remains to solve the statistical ( k, p ) -clustering problem on a finite distribution. We describe a replicable algorithm for this in Appendix D.4. Our coreset algorithm assumes the knowledge of some constant ratio estimate of OPT. In Appendix D.5, we show how to output such an estimate replicably. Finally, we summarize our findings in Appendix D.6.

D.1 Warm-Up: Replicable SQ Oracle and ε -Covers

In order to give some intuition to the reader, we first show how we can use a subroutine that was developed in Impagliazzo et al. [2022] in order to derive some results in the setting we are studying. We first need to define the statistical query model that was introduced in Kearns [1998]

Definition D.1 (Statistical Query Oracle; [Kearns, 1998]) . Let D be a distribution over the domain X and ϕ : X n → R be a statistical query with true value

Here X i ∼ i.i.d. D and the convergence is understood in probability or distribution. Let ε, δ ∈ (0 , 1) 2 . A statistical query (SQ) oracle outputs a value v such that | v -v ⋆ | ≤ ε with probability at least 1 -δ .

The simplest example of a statistical query is the sample mean

The idea is that by using a sufficiently large number of samples, an SQ oracle returns an estimate of the expected value of a statistical query whose range is bounded. Impagliazzo et al. [2022] provide a replicable implementation of an SQ oracle with a mild blow-up in the sample complexity which we state below.

Theorem D.2 (Replicable SQ Oracle; [Impagliazzo et al., 2022]) . Let ε, ρ ∈ (0 , 1) and δ ∈ (0 , ρ / 3 ) . Suppose ϕ is a statistical query with co-domain [0 , 1] . There is a ρ -replicable SQ oracle to estimate its true value with tolerance ε and failure rate δ . Moreover, the oracle has sample complexity

The interpretation of the previous theorem is that we can replicably estimate statistical queries whose range is bounded.

We now explain how we can use this result for the statistical k -means problem under the Euclidean metric, but it is not hard to extend the approach to more general ( k, p ) -clustering problems. Consider a fixed set of centers F. Then, we can replicably estimate the cost of this solution using Theorem D.2 within an additive accuracy ε and confidence δ using O ( ε -2 ρ -2 log( 1 / δ ) ) samples. Thus, a natural approach is to consider an ε -cover of the unit-diameter ball and then exhaustively search among solutions whose centers coincide with elements of the ε -cover. This is outlined in Algorithm D.1. We are now ready to state our results.

Algorithm D.1 Replicable k -Means with ε -Cover

Lemma D.3. For any ε, δ, ρ ∈ (0 , 1) , δ < ρ / 3 , Algorithm D.1 is ρ -replicable and outputs a solution F whose cost is at most OPT+ ε with probability 1 -δ . Moreover, it has sample complexity

Proof (Lemma D.3). We first argue about the replicability of Algorithm D.1. Since we make |F| calls to the replicable SQ subroutine with parameter ρ / |F| , the overall replicability of the algorithm follows by taking a union bound.

Let us now focus on the correctness of the algorithm. Let F ∗ be the optimal solution. Consider the solution that we get when we move the centers of F ∗ to the closest point of G and let us denote it by ˆ F ∗ . Notice that the cost of ˆ F ∗ is at most OPT+ ε / 3 . Furthermore, by a union bound, all the calls to the SQ oracle will return an estimate that is within an additive ε / 3 -error of the true cost. This happens with probability at least 1 -δ and we condition on this event for the rest of the proof. Thus the estimated cost of the solution ˆ F ∗ will be at most OPT+ 2 ε / 3 . Let ˜ F be the solution that we output. Its estimated cost is at most that of ˆ F ∗ and so its true cost will be at most OPT+ ε. This concludes the proof of correctness.

Lastly, we argue about the sample complexity of the algorithm. By Proposition C.2,

By plugging this value into the sample complexity from Theorem D.2, we get

Although Algorithm D.1 provides some basic guarantees, there are several caveats with this approach. Namely, we can only get additive approximations and the dependence of the sample and time complexity on both k, d is exponential.

In the following sections, we will explain how we can overcome these issues. As we alluded to before, our approach combines ideas from coresets estimation through hierarchical grids [Frahling and Sohler, 2005], uniform convergence through metric entropy [Wainwright, 2019], and dimensionality reduction techniques [Makarychev et al., 2019].

D.2 Replicable Heavy-Hitters

In this section, we present a replicable heavy hitters algorithm which is inspired by Impagliazzo et al. [2022] and has an improved sample complexity by a factor of O ( 1 / ( v -ε ) ) . We believe that this result could be of independent interest since the replicable heavy hitters algorithm has many applications as a subroutine in more complicated algorithms [Impagliazzo et al., 2022]. Intuitively, this algorithm consists of two phases. In the first phase, we estimate all the candidate heavy hitters and reduce the size of the domain. In the second phase, we estimate the mass of these candidates. We present a new analysis of the second phase using Proposition B.1.

Algorithm D.2 Replicable Heavy Hitters

rHeavyHitters: distribution P , target v , error ε , replicability ρ , confidence δ

Theorem D.4. Fix v, ρ ∈ (0 , 1) and ε ∈ (0 , v ) , δ ∈ (0 , ρ / 3 ) . Then Algorithm D.2 is ρ -replicable and returns a list L of elements x such that with probability at least 1 -δ :

Moreover, it has sample complexity

Proof. First, we wish to capture all v -ε heavy hitters in ˆ X . If X is sufficiently small, this is easy. Otherwise, we fail to observe each ( v -ε ) heavy hitter with probability at most (1 -v + ε ) | ˆ X| . By a union bound over all ( v -ε ) -1 possible heavy hitters, we fail to capture all v -ε heavy hitters with probability at most

Moving forward, we condition on this step succeeding.

Next, consider ˆ p 1 , ˆ p 2 , the mass estimates over the course of two runs supported on the candidate sets ˆ X 1 , ˆ X 2 . both follow (possibly different) multinomial distributions of dimension at most | ˆ X| +1 with unknown mean parameters p 1 ( x ) for x ∈ ˆ X 1 ∪{ y 1 } , p 2 ( x ) for x ∈ ˆ X 2 ∪{ y 2 } , respectively, and | S | . Here y 1 , y 2 are dummy elements for observations beyond ˆ X 1 , ˆ X 2 . Suppose we draw ln 2 / δ + ( | ˆ X| +1 ) ln 2 2 ε ′ 2 samples over each of two runs to yield estimates ˆ p 1 , ˆ p 2 . By Proposition B.1,

each with probability at least 1 -δ / 2 . Moving forward, we condition on this step succeeding.

Now, consider choosing v ′ ∈ [ v -2 ε / 3 , v -ε / 3 ] uniformly at random.

Correctness: By choosing 2 ε ′ < ε / 3 , any element x with true mass at least v will have empirical mass strictly more than v -ε / 3 ≥ v ′ . Similarly, any element x with true mass at most v -ε will have empirical mass strictly less than v -2 ε / 3 ≤ v ′ . Thus we satisfy the correctness guarantees. Note that we satisfy this guarantee with probability at least 1 -δ .

Replicability: v ′ lands in between some ˆ p 1 ( x ) , ˆ p 2 ( x ) for x ∈ ˆ X 1 ∩ ˆ X 2 with total probability at most

In addition, we wish for v ′ to avoid landing 'below' any ˆ p 1 ( x ) , ˆ p 2 ( x ) where x / ∈ ˆ X 1 ∩ ˆ X 2 . This happens with probability at most

The sum of the two expressions above is at most 4 ε ′ ε/ 3 . Thus we set

The probability of failing to be replicable is then at most 1 -ρ / 3 -2 δ ≥ 1 -ρ .

Sample Complexity: Recall that we set

If |X| is sufficiently small, the sample complexity becomes

Otherwise, the sample complexity is

D.3 Replicable Coreset

We now prove that Algorithm 4.1 replicably yields an ε -coreset. In Appendix D.3.1, we state Algorithm D.3, an equivalent description of Algorithm 4.1. Then, we show that it is indeed replicable in Appendix D.3.2. Following this, we prove that the cost of any set of centers F ⊆ B d is preserved up to an ε -multiplicative error in Appendix D.3.3. In Appendix D.3.4, we show that the outputted coreset is of modest size. Last but not least, we summarize our findings in Appendix D.3.5.

D.3.1 Pseudocode

Before presenting the analysis, we state an equivalent algorithm to Algorithm 4.1 which uses precise notation that is more suitable for analysis.

Starting with the grid that consists of a single cell [ -1 / 2 , 1 / 2 ] d at layer 0 , we recursively subdivide each cell into 2 d smaller cells of length 2 -i at layer i . Each subdivided cell is a child of the larger cell, the containing cell is the parent of a child cell, and cells sharing the same parent are siblings . Let G i denote the union of cells at the i -th level. We write P i to denote the discretized distribution to G i . In other words, P i = P ∣ ∣ σ ( G i ) is the restriction of P to the smallest σ -algebra containing G i . We say a cell on the i -th level is heavy if the replicable heavy hitters algorithm (cf. Algorithm D.2) returns it with input distribution P i , heavy hitter threshold v = γ · Λ 2 -pi , and error v / 2 for some parameter γ to be determined later. A cell that is not heavy is light . The recursion terminates either when there are no more heavy cells on the previous layer, or the current grid length is a sufficiently small fraction of OPT . In the second case, we say the last layer consists of special cells.

If no child cell of a heavy parent is heavy, we mark the parent. Otherwise, we recurse on its heavy children so that one of its descendants will eventually be marked. The recursion must stop, either when there are no more heavy children or when the side length is no more than a fraction of Λ . Note that the light cells with heavy parents and special cells partition [ -1 / 2 , 1 / 2 ] d .

Then, we build R in reverse as follows. For a marked heavy cell Z , we set R ( x ) to be an arbitrary (but replicable) point in Z , e.g. its center, for all x ∈ Z . This covers all light cells with no heavy siblings as well as special cells. Otherwise, a cell Z is light with heavy siblings, and its parent is heavy but unmarked. Thus, the parent has some marked heavy descendent Z ′ , and we choose R ( x ) = R ( x ′ ) for all x ∈ Z and x ′ ∈ Z ′ .

D.3.2 Replicability

Proposition D.5. Algorithm D.3 terminates after at most

layers.

̸

Proof. We have

But 1 / Λ ≤ β (1+ ε ) / OPT , concluding the proof.

Corollary D.6. Algorithm D.3 is ρ -replicable and succeeds with probability at least 1 -δ . Moreover, it has sample complexity

Proof. The only non-trivial random components of Algorithm D.3 is the heavy hitter estimation. By Theorem D.4, the heavy hitters subroutine call on the i -th layer is ρ / t -replicable and succeeds with

probability at least 1 -δ / t . Also, it has sample complexity

Thus all in all, Algorithm D.3 is ρ -replicable and succeeds with probability at least 1 -δ . It also has sample complexity

Since 1 / Λ ≤ β (1+ ε ) / OPT , we conclude the proof.

D.3.3 Expected Shift

Proposition D.7. For any x ∈ Z ∈ L [ i ] ∪ S [ i ] outputted by Algorithm D.3,

Proof. x is represented by some point within its parent.

Proposition D.8. The output S [ t ] of Algorithm D.3 satisfies

Now,

Proof. By computation,

Let F be an arbitrary set of k centers. Partition L [ i ] into

Proposition D.9. The output L far of Algorithm D.3 satisfies

Proof. Any x ∈ Z ∈ L far [ i ] contributes a cost of at least 5 ε (2 -i +1 ∆) p . Hence

Proposition D.10. The output L near of Algorithm D.3 satisfies |L near [ i ] | ≤ kM where

Proof. The furthest point x in a cell in L near [ i ] can have a distance of at most

to the nearest center. Thus the points belonging to a particular center live in an ℓ ∞ ball of that radius. Not including the cell containing the center, we can walk past at most

cells if we walk in the directions of the canonical basis of R d . It follows that there can be at most

cells in L near [ i ] close to each center of F , and thus there are at most kM cells in total.

Proposition D.11. By choosing

The output L near of Algorithm D.3 satisfies

Proof. Each light cell has measure at most γ · Λ 2 -pi . By computation,

Corollary D.12.

Then for any set F of k -centers, the output R of Algorithm D.3 satisfies

Proof. X is contained in the union of all L [ i ] 's and S [ t ] . Hence

D.3.4 Coreset Size

To determine the size of our coreset, we bound the number of marked heavy cells, as all our representative points are in marked heavy cells.

Fix F OPT to be a set of optimal centers for the ( k, p ) -clustering problem. Write M [ i ] to denote the set of marked heavy cells on the i -th layer and partition M [ i ] as

Proposition D.13. The marked heavy cells M dist outputted by Algorithm D.3 satisfy

Proof. Each cell in M dist [ i ] has mass at least γ · Λ 2 -pi +1 and each x ∈ Z ∈ M dist [ i ] is of distance at least 2 -pi +1 to its closest center. Thus each cell contributes at least

to the objective. If follows that there are at most

such cells.

Proposition D.14. The marked heavy cells M close outputted by Algorithm D.3 satisfy

Proof. The furthest point x in a cell M close [ i ] to its nearest center is

In other words, not including the cell containing the center, we can walk past at most

cells by walking along the direction of the canonical basis of R d . Thus there are at most

such cells on the i -th layer.

Proposition D.15. The image of the map R outputted by Algorithm D.3 satisfies

In particular, for we have

Proof. The image is contained in the marked heavy cells.

D.3.5 k -Medians & k -Means

We now derive results for the case of p = 1 .

Theorem D.16. Algorithm D.3 is ρ -replicable and outputs an ε -coreset of size

for statistical k -medians with probability at least 1 -δ . Moreover, it has sample complexity

Proof (Theorem D.16). By Corollary D.12, Algorithm D.3 outputs an ε -coreset. By Corollary D.6, Algorithm D.3 is ρ -replicable with sample complexity

We have

By Proposition D.15, the coreset has size at most this value.

Substituting the value of γ above and remarking that t is a polylogarithmic factor of the other terms, we thus conclude that the final sample complexity is

Now consider the case of p = 2 . We can perform our analysis with the help of a standard inequality. Proposition D.17. For any set of centers F ,

Thus if E x κ ( x, R ( x )) 2 ≤ ε 2 6 E x κ ( x, F ) 2

Proof. By H¨ older's inequality,

The other case is identical.

Theorem D.18. Algorithm D.3 is ρ -replicable and outputs an ε -coreset for statistical k -means of size

with probability at least 1 -δ . Moreover, it has sample complexity

Proof (Theorem D.18). Similar to k -medians, Algorithm D.3 outputs an ε -coreset with probability at least 1 -δ , is ρ -replicable, and has sample complexity

However, we need to take ε ′ := ε 2 / 2 6 .

We have

By Proposition D.15, the coreset has size at most this value.

t is a polylogarithmic factor of the rest of the parameters, thus we thus conclude that the final sample complexity is

It is not clear how to compare our guarantees with that of Frahling and Sohler [2005] as we attempt to minimize the number of samples required from a distribution assuming only sample access. On the other hand, Frahling and Sohler [2005] attempted to minimize the running time of their algorithm assuming access to the entire distribution.

D.4 Replicable Multinomial Parameter Estimation

Now that we have reduced the problem to a finite distribution (coreset), we can think of it as a weighted instance of ( k, p ) -clustering where the weights are unknown but can be estimated using data.

Proposition D.19. Enumerate the coreset R ( X ) = r (1) , . . . , r ( N ) and let w ( i ) be the probability mass at r ( i ) . If we have estimates ˆ w ( i ) satisfying

then for all centers F .

Proof. Indeed,

Let us formulate this as a replicable multinomial parameter estimation problem. Consider a multinomial distribution Z = ( Z (1) , . . . , Z ( N ) ) of dimension N with parameters p (1) , . . . , p ( N ) and n . Note that this can be cast as a more general statistical query problem which we illustrate below. We allow simultaneous estimation of more general statistical queries g 1 , . . . , g N : ( X n , P ) → R compared to Impagliazzo et al. [2022], assuming they can be estimated with high accuracy and confidence from data, say each query g j ( x 1 , . . . , x n ) concentrates about its true value G j ∈ R with high probability.

Algorithm D.4 Replicable Rounding

1: rRounding (statistical queries g 1 , . . . g N , distribution P , number of samples n , error ε , replicability ρ , confidence δ ):

2: Sample x 1 , . . . , x n ∼ P

3: α ← 2 ε ρ +1 -2 δ

4: for j ← 1 , . . . , N : do

5: α ( j ) ← uniform random sample [0 , α ]

6: Split R into intervals I ( j ) = { [ α ( j ) + zα, α ( j ) +( z +1) α ) : z ∈ Z }

7: Output the midpoint ˆ G j of the interval within I ( i ) that g j ( x 1 , . . . , x n ) falls into

8: end for

Theorem D.20 (Replicable Rounding) . Suppose we have a finite class of statistical queries g 1 , . . . , g N and sampling n independent points from P ensures that

with probability at least 1 -δ .

Then Algorithm D.4 is ρ -replicable and outputs estimates ˆ G j such that

with probability at least 1 -δ for every j ∈ [ N ] . Moreover, it requires at most n samples.

Proof. Outputting the midpoint of the interval can offset each g j ( x 1 , . . . , x n ) by at most α 2 . Hence

Consider two executions of our algorithm yielding estimates ˆ G (1) , ˆ G (2) . The probability that any of the estimates fail to satisfy ℓ 1 -tolerance ε ′ is at most 2 δ . The two executions output different estimates only if some random offset 'splits' some ˆ G (1) j , ˆ G (2) j . Conditioning on the success to satisfy ℓ 1 -tolerance, this occurs with probability at most

Accounting for the probability of failure to satisfy the ℓ 1 -tolerance of 2 δ , our algorithm is ρ -replicable.

We note that Theorem D.20 can be thought of as a generalization of the SQ oracle (cf. Theorem D.2) from Impagliazzo et al. [2022]. Indeed, simply using Theorem D.2 yields an extra factor of N in the sample complexity as we must take a union bound.

Corollary D.21. Let ε, ρ ∈ (0 , 1) and δ ∈ (0 , ρ / 3 ) . There is a ρ -replicable algorithm that outputs parameter estimates ¯ p for a multinomial distribution of dimension N such that

(a) | ¯ p ( i ) -p ( i ) | ≤ ε for every i ∈ [ N ] with probability at least 1 -δ .

(b) ¯ p ( i ) ≥ 0 for all i ∈ [ N ] .

(c) ∑ i ¯ p ( i ) = 1 .

Moreover, the algorithm has sample complexity

points implies that ∑ N i =1 | ̂ p ( i ) -p ( i ) | < ε ′ with probability at least 1 -δ .

yields ¯ p ( i ) 's such that | ¯ p ( i ) -p ( i ) | ≤ ε for each i ∈ [ N ] . If there are any negative ¯ p ( i ) 's, we can only improve the approximation by setting them to 0. If the sum of estimates is not equal to 1, we can normalize by taking

This introduces an additional error of ε / N for each ˆ p ( i ) and a maximum error of 2 ε . Choosing ε 1 := ε / 2 concludes the proof.

D.5 Replicable OPT Estimation for ( k, p ) -Clustering

Our coreset algorithm assumes the knowledge of a constant ratio estimation of OPT. If this is not provided to us, we show in this section that it is possible to replicably compute such an estimate using a two-step approach. First, we are able to produce replicable estimates with additive error ε simply by approximately solving the sample ( k, p ) -clustering problem on a sufficiently large sample thanks to uniform convergence (cf. Theorem C.9). Then, we repeat this process with ε ← ε / 2 until ε is but a small fraction of the outputted estimate.

Theorem D.22 (OPT Estimation with Additive Error) . Let β ≥ 1 be an absolute constant and ε, ρ ∈ (0 , 1) , δ ∈ (0 , ρ / 3 ) . Suppose we are provided with an algorithm A that, given an instance of sample ( k, p ) -clustering, outputs an estimate Ξ of the value ̂ OPT such that

Proof. Define

By Proposition B.1, sampling

Then there is a ρ -replicable algorithm which produces an estimate Λ of OPT such that

with probability at least 1 -δ . Moreover, the algorithm has sample complexity

We note that the algorithm A can be taken to be any β -approximation algorithm with some postprocessing, where we divide the value of the cost by β . Alternatively, we can take A to be some convex relaxation of our problem with constant integral gap.

Proof. Let X 1 , . . . , X N ∼ P be i.i.d. random variables. The output Ξ = A ( X 1 , . . . , X N ) ∈ [0 , 1] is thus a bounded random variable. Suppose we repeat this experiment n times to produce estimates Ξ 1 , . . . , Ξ n . By an Hoeffding bound,

Thus with trials, the average estimate ¯ Ξ := 1 n ∑ n i =1 Ξ i satisfies ∣ ∣ ¯ Ξ -E [Ξ] ∣ ∣ < ε with probability at least 1 -δ .

Now, by Theorem C.9, choosing

i.i.d. samples from P ensures that ∣ ∣ ∣ ̂ OPT -OPT ∣ ∣ ∣ ≤ ε for all n trial with probability at least 1 -δ . Thus conditioning on success, we have

for every trial i ∈ [ n ] . But then the average ¯ Ξ also falls into this interval as well.

We have shown that there is an algorithm that outputs an estimate

with probability at least 1 -δ . Moreover, it has sample complexity

We can now apply the replicable rounding algorithm (cf. Algorithm D.4) to achieve the desired outcome. Indeed, by Theorem D.20, the output Λ after the rounding procedure is ρ -reproducible and offsets the average ¯ Ξ by at most ε . Hence we have

with probability at least 1 -δ . Finally, the algorithm has sample complexity

Choosing ε ′ = ε / 2 completes the proof.

Theorem D.23 (OPT Estimation with Relative Error) . Fix ε, ρ ∈ (0 , 1) and δ ∈ (0 , ρ / 3 ) . There is a ρ -replicable algorithm such that with probability at least 1 -δ , it outputs an estimate Λ of OPT where

Moreover, it has sample complexity and

and similarly for δ .

Proof. We run the estimation algorithm with additive error with

for i = 1 , 2 , 3 , . . . until we obtain an estimate Λ i such that ε i ≤ ε Λ i / 2 .

Remark that

Thus the algorithm certainly terminates.

Upon termination, we output some Λ such that Λ ∈ [ OPT / β -ε Λ / 2 , OPT+ ε Λ / 2 ] . It follows that

The probability of not being replicable in each iteration is 2 -i ρ . Hence the total probability is at most

Finally, there are O ( log β ε OPT ) iterations and the sample complexity of each iteration is bounded above by the sample complexity of the final iteration where

Hence the total sample complexity is

D.6 Putting it Together

We are now ready to prove the results on statistical k -medians and statistical k -means, which we restate below for convenience. Recall that β is an absolute constant and hence disappears under the big-O notation.

Theorem 4.2 (Theorem 3.1; Formal) . Let ε, ρ ∈ (0 , 1) and δ ∈ (0 , ρ / 3 ) . Given black-box access to a β -approximation oracle for weighted k -medians (cf. Problem 2.2), there is a ρ -replicable algorithm for statistical k -medians (cf. Problem 2.1) such that, with probability at least 1 -δ , it outputs a (1 + ε ) β -approximation. Moreover, it has sample complexity

Proof (Theorem 4.2). First, we compute Λ , an estimate of OPT with relative error ε , confidence δ / 3 , and replicability parameter ρ / 3 . Then, we produce an ε / 2 -coreset for k -medians of cardinality N with confidence and replicability parameters δ / 3 , ρ / 3 . Finally, we estimate the mass at each point of the coreset with confidence and replicability parameters δ / 3 , ρ / 3 to an accuracy of

By Proposition D.19, running the approximation oracle on the coreset with the estimated weights yields a β (1 + ε ) -approximation.

samples.

By Theorem D.16, the coreset construction yields a coreset of cardinality

and incurs a sample cost of

Finally, Corollary D.21 states that the probability mass estimation incurs a sample cost of

Theorem 4.3 (Theorem 3.1; Formal) . Given black-box access to a β -approximation oracle for weighted k -means (cf. Problem 2.2), there is a ρ -replicable algorithm for statistical k -means (cf. Problem 2.1) such that, with probability at least 1 -δ , it replicably outputs a (1 + ε ) β -approximation. Moreover, it has sample complexity

Proof (Theorem 4.3). Similar to k -median, we first compute Λ , an estimate of OPT with relative error ε , confidence δ / 3 , and replicability parameter ρ / 3 . Then, we produce an ε / 2 -coreset for k -medians of cardinality N with confidence and replicability parameters δ / 3 , ρ / 3 . Finally, we estimate the mass at each point of the coreset with confidence and replicability parameters δ / 3 , ρ / 3 to an accuracy of

By Proposition D.19, running the approximation oracle on the coreset with the estimated weights yields a β (1 + ε ) -approximation.

samples.

By Theorem D.18, the coreset construction yields a coreset of cardinality

and incurs a sample cost of

Finally, Corollary D.21 states that the probability mass estimation incurs a sample cost of

E The Euclidean Metric, Dimensionality Reduction, and ( k, p ) -Clustering

We now build towards a proof for Theorem 5.1, which states the formal guarantees of the replicable k -medians and k -means algorithms under the Euclidean distance. Let us begin by recalling the Johnson-Lindenstrauss lemma.

Theorem E.1 ([Johnson, 1984]) . There exists a family of random linear maps π d,m from R d → R m such that the following hold.

(i) For every d ≥ 1 , ε, δ ∈ (0 , 1 / 2 ) , x ∈ R d , and m = O ( 1 / ε 2 log 1 / δ ) ,

with probability at least 1 -δ .

(ii) π d,m is sub-Gaussian-tailed , thus for every unit vector x ∈ R d and ε > 0 ,

with probability at most exp ( -Ω( ε 2 m ) ) .

Furthermore, we can take π d,m to be the set of random orthogonal projections R d → R m scaled by a factor of √ d / m .

We say that π ∈ π d,m is a standard random dimension-reduction map and write P π to denote the projected distribution on R m , i.e., P π ( · ) = P ( π -1 ( · )) .

As mentioned in Section 5, it is not entirely clear how to translate between sets of centers for highdimensional data and solutions sets for the compressed data. For this reason, our goal is to produce a clustering function f : X → [ k ] which computes the label for any given point in polynomial time. This also requires us to consider another notion of cost which is translated more easily.

Definition E.2 (Partition Cost) . Define the cost of a k -partition C = ( C (1) , . . . , C ( k ) ) of X as

i.e., we consider the expected cost of picking the optimal centers w.r.t. the underlying data-generating distribution. Similarly, we define the following sample partition cost.

Definition E.3 (Sample Partition Cost) . We define the sample cost of a k -partition C of x 1 , . . . , x n as

where w ≥ 0 , ∑ i w i = 1 . i.e., we consider the cost that is induced by the distribution specified by w on the samples. We write ̂ cost( C ) to denote the cost with the uniform distribution on the samples.

Remark that if C is induced by F , then OPT ≤ cost( C ) ≤ cost( F ) and similarly for the sample partition cost as well.

Theorem E.4 ([Makarychev et al., 2019]) . Let x 1 , . . . , x n ∈ R d be an instance of the Euclidean ( k, p ) -clustering problem. Fix ε ∈ (0 , 1 / 4 ) , δ ∈ (0 , 1) and suppose π : R d → R m is a standard random dimension-reduction map with

Then with probability at least 1 -δ ,

for every k -partition C of x 1 , . . . , x n .

In essence, this theorem tells us that if we let the centers of the clusters be the centers of the partition, then the (sample) cost is preserved in the lower dimensional space. To the best of our knowledge, this is the strongest result for dimensionality reduction in (sample) ( k, p ) -clustering which preserves more than just the costs of optimal centers.

The rest of Appendix E serves to generalize Theorem E.4 for preserving partition costs in the distributional setting. First, Appendix E.1 addresses the issue that a random orthogonal projection can increase the diameter of our data in the worst case by scaling the original data by an appropriate factor. Next, Appendix E.2 slightly generalizes Theorem E.4 to the weighted sample clustering setting. Finally, Appendix E.3 extends Theorem E.4 to the distributional setting, culminating with Theorem E.10, which may be of independent interest beyond replicability.

Recall that P π is the push-forward measure on π ( X ) induced by π . We write OPT π to denote the cost of an optimal solution to the statistical ( k, p ) -clustering problem on ( π ( X ) , P π ) and ̂ OPT π for the ( k, p ) -clustering problem on π ( x 1 ) , . . . , π ( x n ) .

E.1 Scaling

Wemayassume that a standard dimension-reduction map π satisfies π = √ d / m P for some orthogonal projection P . Hence π has operator norm at most √ d and √

In order to apply our existing analysis, we first scale down the samples in the input space by a factor of √ d so that the projected space still lives in a κ -ball of diameter 1. This does not affect our approximation guarantees as they are all multiplicative and hence scale-invariant. However, the analysis we perform is on the scaled distribution with optimal cost OPT ′ = d -p / 2 OPT . This extra term must be accounted for.

We proceed assuming now that X ⊆ 1 √ d B d and π ( X ) ⊆ B m .

E.2 Preserving Partition Costs

Let ̂ cost( C , w ) , ̂ cost( π ( C ) , w ) be the weighted cost of the partition and projected partition, respectively. Recall this means that the points participate with weights w in the objective. Essentially, the next result shows that since the projection guarantees do not depend on the number of points we are projecting, we can simulate the weights by considering multiple copies of the points.

Proposition E.5. Let x 1 , . . . , x n ∈ R d be an instance of the Euclidean ( k, p ) -clustering problem and w 1 , . . . , w n ∈ R + be weights which sum to 1. Fix ε ∈ (0 , 1 / 4 ) , δ ∈ (0 , 1) , and suppose π : R d → R m is a standard random dimension-reduction map with

Then with probability at least 1 -δ ,

for every k -partition C of x 1 , . . . , x n .

Proof (Proposition E.5). First suppose that w i = a i / b i for a i , b i ∈ Z + and consider w ′ i := lcm( b 1 , . . . , b n ) w i ∈ Z +

obtained from w i by multiplying by the least common multiple of the denominators.

Let y 1 , . . . , y n ′ be the multiset obtained from x 1 , . . . , x n by taking w ′ i multiples of x i . The cost of any partition of x 1 , . . . , x n with weights q 1 , . . . , q n is equal to the cost of the induced partition of y 1 , . . . , y n ′ with uniform weights. Thus we can apply Theorem E.4 on y 1 , . . . , y n ′ to conclude the proof.

Now, for general w i ∈ R + , we remark by the density of the rationals in R that there are some q 1 , . . . , q n ∈ Q + which sum to 1 and satisfy

Then for any k -partition C and u (1) , . . . , u ( k ) ∈ B d ,

In particular

Similarly, for any v (1) , . . . , v ( k ) ∈ B m ,

which implies that

Finally, we conclude that

′

Choosing ε to be a constant fraction of ε concludes the proof.

Corollary E.6. Let x 1 , . . . , x n ∈ R d be an instance of the weighted Euclidean ( k, p ) -clustering problem with non-negative weights w 1 , . . . , w n ∈ R + which sum to 1. Fix ε ∈ (0 , 1 / 4 ) , δ ∈ (0 , 1) and suppose π : R d → R m is a standard random dimension-reduction map with

Then with probability at least 1 -δ ,

Proof (Corollary E.6). The optimal clustering cost coincides with the cost of the partition induced by the optimal centers.

E.3 Dimensionality Reduction

Let G = { g (1) , . . . , g ( k ) } be a β -approximate solution to the statistical Euclidean ( k, p ) -clustering problem on ( π ( X ) , P π ) . We would like to argue that the partition of X induced by G is a (1 + ε ) β -approximate partition with probability at least 1 -δ . To do so, we first approximate X with a finite set ˜ X in a replicable fashion while preserving partition costs. This allows us to apply Proposition E.5.

Recall we write Λ to denote a replicable estimate of OPT such that Λ ∈ [ OPT / β (1+ ε ) , (1 + ε ) OPT] (cf. Theorem D.23). Consider a grid of length ε Λ / (4 p ∆) . for a point x ∈ X , we assign it to the center of the cell that it belongs to, say ˜ x . Let ˜ X := { ˜ x : x ∈ X} denote this finite set and ˜ P the distribution induced by the discretization. Remark that κ ( x, ˜ x ) ≤ ε Λ / (4 p ) for every x ∈ X .

Note that a partition of X induces a partition of ˜ X and vice versa. We proceed without making any distinction between the two.

Proposition E.7. For any point u ∈ B d ,

Proof. By the mean value theorem, the function g : [0 , 1] → R given by z ↦→ z p is p -Lipschitz:

We write ˜ cost( C ) to denote the cost of the partition C of ˜ X .

Proposition E.8. For any partition C of X

Proof. We have

Corollary E.9. Fix ε ∈ (0 , 1 / 4 ) , δ ∈ (0 , 1) and suppose π : R d → R m is a standard random dimension-reduction map with

We have with probability at least 1 -δ, where the probability is taken w.r.t. the choice of the random map.

Proof. The optimal clustering cost coincides with the cost of the partition induced by the optimal centers. An application of Proposition E.8 yields the desired result.

Theorem E.10. Fix ε ∈ (0 , 1 / 4 ) , δ ∈ (0 , 1) and suppose π ∈ π d,m is a standard random dimensionreduction map with

Let G = { g (1) , . . . , g ( k ) } be a β -approximate solution to the statistical Euclidean ( k, p ) -clustering problem on ( π ( ˜ X ) , ˜ P π ) . Then the partition of X induced by G is a (1 + ε ) β -approximate partition with probability at least 1 -δ .

That is, for C = { C (1) , . . . , C ( k ) } given by

we have

Proof. We have

Choosing ε ′ to be a constant fraction of ε concludes the proof.

E.4 Summary of Dimensionality Reduction

Before proving Theorem 5.1, let us summarize the stages of our pipeline.

Scale the input data by 1 / √ d .

Estimate OPT of the scaled data within a multiplicative error.

Discretize the scaled data with a fine grid.

Project the scaled data with a random map π ∈ π d,m .

Compute a coreset on the projected data.

Estimate the probability mass at each point of the coreset.

Call the β -approximation oracle to produce k centers g (1) , . . . , g ( k ) .

Output the clustering function induced by the centers.

The clustering function proceeds as follows. Given a point x ∈ X ,

Scale x by 1 / √ d .

Discretize the scaled point with the same grid used in the algorithm.

Project the discretized point using the same π used in the algorithm.

Output the label of the closest center g ( j ) in the lower dimension for j ∈ [ k ] .

We are now ready to prove Theorem 5.1, which we restate below for convenience.

Theorem 5.1 (Theorem 3.2; Formal) . Let ε, ρ ∈ (0 , 1) and δ ∈ (0 , ρ / 3 ) . Given a β -approximation oracle for weighted Euclidean k -medians ( k -means), there is a ρ -replicable algorithm that outputs a clustering function such that with probability at least 1 -δ , the cost of the partition is at most (1 + ε ) β OPT . Moreover, the algorithm has sample complexity

where m = O ( 1 ε 2 log k δε ) .

Proof (Theorem 5.1). We randomly draw some π ∈ π d,m for

Then, we discretize a sufficiently large sample after scaling and push it through π . We then run the corresponding algorithm (cf. Theorem 4.2, Theorem 4.3) to produce some centers G for the compressed data. This allows us to partition X according to the closest member of G in the lower dimension. An application of Theorem E.10 yields the performance guarantees of this algorithm.

The sample complexity guarantee from Theorem 4.2, Theorem 4.3 is for the scaled distribution and the scaled optimal cost satisfies OPT ′ π = d -p / 2 OPT π . Hence the sample complexity resolves to

We conclude the proof by remarking that for the Euclidean metric, ∆ = √ d and that O (log n ) O (log n ) = n O (log log n ) .

F k -Centers

In this section, we will explain our approach to solving the statistical k -centers problem (cf. Problem 2.3). As we explained before, since this problem has a different flavor compared to ( k, p ) -clustering, we introduce some extra assumptions in order to be able to solve it from samples. To the best of our knowledge, this is the first work that considers a model of statistical flavor for the k -centers problem.

F.1 Assumptions

First, we describe an assumption that is necessary to obtain good clustering solutions even if we do not insist on using replicable algorithms.

Assumption F.1 (Clusterable) . For some absolute constants β, B > 0 , an i.i.d. sample S = x 1 , . . . , x n of size n from P has the following property. With probability at least q , there exists some ( β, B ) -approximate solution F ∗ to Problem 2.3 such that, for every f ∈ F ∗ there is some x f ∈ S with F ∗ ( x f ) = f . Here q ∈ [0 , 1] .

Assumption F.1 is necessary to obtain good clustering solutions even in the non-replicable case. Our distribution P must be sufficiently well-behaved so that solving the sample k -centers problem translates to a good solution in the population case. Essentially, this assumption states that with high probability, the sample will contain a point from every single cluster of some ( β, B ) -approximate solution. In order to design replicable algorithms, we need to make a stronger assumption.

Assumption F.2 (Replicable) . For some absolute constants β, B , there exists a ( β, B ) -approximate solution F ∗ to Problem 2.3 with the following property. With probability at least q , in an i.i.d. sample S = x 1 , . . . , x n of size n from P , for every f ∈ F ∗ , there is some x f ∈ S with F ∗ ( x f ) = f . Here q ∈ [0 , 1] .

Note that Assumption F.1 guarantees that, with high probability, every sample will help us identify some good solution. However, these good solutions could be vastly different when we observe different samples, thus this assumption is not sufficient to design a replicable algorithm with high utility. Assumption F.2 is necessary to attain replicable solutions. Indeed, this assumption essentially states that there is a fixed solution for which we will observe samples from every cluster with high probability. If this does not hold and we observe points from clusters of solutions that vary significantly, we cannot hope to replicably recover a fixed approximate solution. Note that Assumption F.2 is stronger than Assumption F.1, since we have flipped the order of the existential quantifiers. One way to satisfy this assumption is to require that there is some ( β, B ) -approximate solution so that the total mass that is assigned to every cluster of it is at least 1 / n for some n ∈ N . Then, by observing ˜ O ( n ) points, we will receive at least one from every cluster with high probability. This is made formal in Proposition F.3.

Proposition F.3 ([Janson, 2018]) . Let δ ∈ (0 , 1) . Under Assumption F.2, a sample of

points contains at least m points from each cluster of F ∗ with probability at least 1 -δ .

Proof (Proposition F.3). Let Y i be a geometric variable with success rate q , which denotes the number of batches of n samples we draw until a point from each cluster of F ∗ is observed. Then Y := ∑ m i =1 Y i upper bounds the number of batches until we obtain m points from each cluster.

As shown in Janson [2018],

The result follows by picking λ = 1 + log( 1 / δ ) .

F.2 High-Level Overview of the Approach

We are now ready to provide a high-level overview of our approach to derive Theorem 3.3. We first take a fixed grid of side c in order to cover the unit ball. Then, we sample sufficiently many points from P . Subsequently, we round all the points of the sample to the centers of the cells of the grid that they fall into. Using these points, we empirically estimate the density of the distribution on every cell of the grid. Next, in order to ensure that our solution is replicable, we take a random threshold from a predefined interval and discard the points from all the cells whose density falls below the threshold. Finally, we call the approximation oracle that we have access to using the points that have survived after the previous step. We remark that unlike the ( k, p ) -clustering problem (cf. Problem 2.1), to the best of our knowledge, there does not exist any dimensionality reduction techniques that apply to the k -centers problem. In the following subsections we explain our approach in more detail.

F.3 Oracle on Sample

As we explained before, we require black-box access to an oracle O for the combinatorial k -centers problem on a sample which outputs ( ̂ β, ̂ B ) -approximate solutions ̂ F for Problem 2.4. Thus, we need to show that, with high probability, the output of this solution is a good approximation to Problem 2.3. This is shown in Proposition F.4.

Proposition F.4. Suppose there is a ( β, B ) -approximate solution F ∗ for Problem 2.3 and that we are provided with a sample of size O ( n log( 1 / δ ) /q ) containing an observation from each cluster of F ∗ , with probability at least 1 -δ . Then, the output ̂ F of the oracle O on the sample satisfies

for all x ∈ X , with probability at least 1 -δ .

In the previous result ( β, B ) , ( ̂ β, ̂ B ) are the approximation parameters that we inherit due to the fact that we receive samples from P and that we solve the combinatorial problem approximately using O , respectively.

Proof (Proposition F.4). Let F ∗ denote the ( β, B ) -approximate solution from Assumption F.2. We condition on the event that such a solution exists in the sample, which occurs with probability at least 1 -δ .

Fix x ∈ X , which is not necessarily observed in the sample. We pay a cost of β OPT+ B to travel to the nearest center of F ∗ . Next, we pay another β OPT+ B to travel to the observed sample from this center. Finally, we pay a cost of

to arrive at the closest center in ̂ F .

F.4 Oracle with Grid

The issue with using directly the output of O on a given sample is that the result will not be replicable, since we have no control over its behavior. In a similar way as with the ( k, p ) -clustering problem, the main tool that we have in our disposal to 'stabilize' the output of O is to use a grid. Unlike the other problems, we cannot use the hierarchical grid approach in this one. The reason is that, due to the min-max nature of k-centers, we need to cover the whole space and not just the points where the distribution is dense. Recall that so far we have established that the oracle O outputs (2 β + ̂ β, 2 B + 2 ̂ B ) -approximate solutions F for Problem 2.3, given that Assumption F.2 holds. Algorithm F.1 'discretizes' the points of the sample using a grid with side-length c before applying the oracle. Proposition F.5 shows that by doing that, we have to pay an extra additive term that is of the order O ( c ∆) .

Algorithm F.1 Oracle with Grid

1: Oracle with Grid (oracle O , sample x 1 , . . . , x n , grid length c ):

2: for i ← 1 , . . . , n do

3: Decompose x i = z i · c + x ′ i for some z i ∈ Z d , x ′ i ∈ [0 , c ) d

4: ˜ x i ← z i · c +( c / 2 , c / 2 , . . . , c / 2 ) { ˜ x i is the center of the cell that contains x i . }

5: end for

6: Return O (˜ x 1 , . . . , ˜ x n )

Proposition F.5. Suppose there is a ( β, B ) -approximate solution F ∗ for Problem 2.3 and that we are provided with a sample of size n containing an observation from each cluster of F ∗ . Let ˜ G denote the output of Algorithm F .1 using a ( ˆ β, ˆ B ) -approximation oracle O for Problem 2.4. Then, for any x ∈ X ,

Proof (Proposition F.5). For the sake of analysis, imagine we used the grid to discretize all points of X ↦→ ˜ X . Then F ∗ is a solution with cost β ˜ OPT+ B + c ∆ and we observe a point from each of its clusters in our discretized sample. Here ˜ OPT is the cost of a solution to Problem 2.3 on ˜ X .

Proposition F.4 shows that the oracle returns a solution ˜ G with cost at most

The conclusion follows by combining the two inequalities.

F.5 Replicable Active Cells

At a first glance, we might be tempted to work towards replicability by arguing that, with high probability, every grid cell is non-empty in one execution of the algorithm if and only if it is nonempty in another execution. However, since we have no control over P , it is not clear how one can prove such a statement. Our approach is to take more samples and ignore cells which do not have a sufficient number of points using some random threshold. The idea is similar to the heavy-hitters algorithm (cf. Algorithm D.2) originally conceived by Impagliazzo et al. [2022] and is presented in Algorithm F.2. Then, we move all the points of active cells to the center of the cell. Notice that in the k -center objective, unlike the ( k, p ) -clustering objective, we do not care about how much mass is placed in a cell, just whether it is positive or not.

We first derive a bound on the number of cells that the distribution puts mass on. It is not hard to see that since we are working in a bounded domain, the grid contains at most ( 1 / c ) d cells in total. Thus there are at most M ≤ ( 1 / c ) d cells that have non-zero mass. Proposition F.6 shows how we can obtain such a bound in the unbounded domain setting.

Proposition F.6. Let F ∗ be a ( β, B ) -approximate solution for Problem 2.3. Let M denote the maximum number of cells which intersect some cluster of F ∗ and ˜ P the discretized distribution that is supported on centers of the cells of the grid of size c . Then, the support of ˜ P is at most kM where

Proof (Proposition F.6). Fix a center f and consider the cell Z containing the center. The farthest point in the farthest cell in the cluster of f is of distance at most β OPT+ B + c ∆ and thus the cells we are concerned with sit in an ℓ ∞ ball of that radius. Not including Z , we walk past at most

cells along the canonical basis of R d , hence there are at most

on the discretized points.

Then for any x ∈ X , such cells.

Notice that the previous discussion and Proposition F.5 reveal a natural trade-off in the choice of the parameter c : by reducing c , we decrease the additive error of our approximation algorithm but we increase the number of samples at a rate ( 1 / c ) d . Since we have bounded the number of cells from which we can observe a point, the next step is to use a sufficiently large number of samples so that we can estimate replicably whether a cell is active or not. This is demonstrated in Algorithm F.2.

Algorithm F.2 Replicable Active Cells

1: rActiveCells (grid length c ):

2: m ← ˜ O ( λknM 2 / ρ 2 )

3: N ← λnmM

4: Sample x 1 , . . . , x N ∼ P ,

5: Lazy initialize counter Z = 0 for every cell of the grid with length c

6: for i ← 1 , . . . , N do

7: G i ← cell that samples x i falls into

8: Increment Z ( G i ) ← Z ( G i ) + 1

9: end for

10: Choose v ∈ [0 , m / N ] uniformly randomly

11: Output all G i such that Z ( G i ) / N ≥ v

Proposition F.7. Suppose Assumption F.2 holds and let F ∗ be the ( β, B ) -approximate solution. Let δ, ρ ∈ (0 , 1) . Then Algorithm F.2 is ρ -replicable and returns cells of the grid such that at least one sample from each cluster of F ∗ falls into these cells with probability at least 1 -δ. Moreover, it has sample complexity

Proof (Proposition F.7). Enumerate the kM cells within the support of P and let the counter Z ( j ) denote the number of samples at the j -th cell. Then Z = ( Z (1) , . . . , Z ( kM ) ) follows a multinomial distribution with parameters ( p (1) , . . . , p ( kM ) ) and N, where p is unknown to us. Nevertheless, by Proposition B.1, for any ε ∈ (0 , 1) , sampling

points implies that ∑ a j =1 | ̂ p ( j ) -p ( j ) | < 2 ε with probability at least 1 -δ / 5 . From the definition of N , this is equivalent to

By Proposition F.3, we observe at least mM points from each cluster of F ∗ with probability at least 1 -δ .

Let ̂ p ( j ) 1 , ̂ p ( j ) 2 be the empirical mean of Z ( j ) across two different executions of Algorithm F.2. Then by the triangle inequality, ∑ kM j =1 | ̂ p ( j ) 1 -̂ p ( j ) 2 | ≤ 4 ε .

Since we observe at least Mm points from each cluster of F ∗ , there is at least one point from each cluster with at least m observations. Thus outputting the points j such that ̂ p ( j ) ≥ v for some v ∈ [0 , m / N ] means we always output one point from each cluster.

Now, the outputs between two executions are not identical only if the point v 'splits' some pair ̂ p ( j ) 1 , ̂ p ( j ) 2 . This occurs with probability at most 4 ε / ( m/N ) = 4 Nε / m , To bound this value by ρ / 5 , set

Plugging this back yields

All in all, Algorithm F.2 outputs the desired result with probability at least 1 -δ and is ρ -replicable given

samples from P .

F.6 Putting it Together

We now have all the ingredients in place to prove Theorem 3.3. The proof follows directly by combining Proposition F.5, and Proposition F.7. Essentially, we replicably estimate the active cells of our grid and then move all the points of each active cell to its center.

For completeness, we also state the formal version of Theorem 3.3.

Theorem F.8 (Theorem 3.3; Formal) . Let δ, ρ ∈ (0 , 1) . Suppose Assumption F.2 holds and let F ∗ be the ( β, B ) -approximate solution. Then, given black-box access to a ( ˆ β, ˆ B ) -approximation oracle for Problem 2.4, Algorithm F.2 is ρ -replicable and returns a solution ˆ F such that

with probability at least 1 -δ . Moreover, the algorithm requires at most

samples from P .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the design of clustering approximation algorithms in the context of statistical clustering under the notion of replicability. In replicable clustering problem, it requires that with high probability, the output of the algorithms should have the exact same partition of the sample space after two executions on different inputs drawn from the same distribution. Given some black-box $\\beta$-approximation algorithm to the $k$-clustering problem, this paper gives an approximation framework that with constant probability, the output of the algorithm can achieves an $O(\\beta)$-approximation satisfying $\\rho$-replicable property.\n\nSTRENGTHS:\nThis paper is the first to initiate the study of replicable clustering, which is a recent hot topic in the field of machine learning. In this paper, a new framework is introduced for handling replicable clustering requirements. This paper also establishes sampling complexities for clustering problems in different norms with different metrics. In order to achieve replicability, a novel tree decomposition method called Replicable Quad Tree is designed such that replicable estimation can be obtained based on the tree decomposition constructed. The proposed Replicable Quad Tree is interesting and is quite different from the well-known HST-based tree decomposition method, which could have great potential for designing approximation algorithms for other replicable problems.\n\nWEAKNESSES:\n1. The sample complexity has exponential dependence on the dimension $d$, which could be the main weakness for this paper when $d$ is large. Although in Euclidean metrics, this can be avoided by dimensionality reduction techniques, for general norms, there is no data-oblivious dimensionality reduction scheme.\n\n2. The main proofs of this paper are hard to follow. It is better to provide intuitive ideas in each section before the proofs such that one could easily follow the main ideas of the proofs.\n\nQUESTIONS:\n1. Can the authors offer some intuitive insights into how the proposed Replicable Quad Tree ensures replicable estimation?\n\nLIMITATIONS:\nSince this is a theoretical paper, I don't think this paper has any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript focuses on the concept of replicability in statistical clustering algorithms. It introduces the notion of replicability, which refers to the ability of an algorithm to produce consistent results when executed multiple times on different inputs from the same distribution. Replicability is seen as crucial for ensuring the validity and reliability of scientific findings. The manuscript proposes replicable algorithms for three common clustering problems: k-medians, k-means, and k-centers. These algorithms utilize approximation routines for their combinatorial counterparts. The replicable algorithm for statistical Euclidean k-medians achieves a replicable O(1)-approximation with polynomial complexity. Similarly, the algorithm for statistical Euclidean k-centers achieves replicable O(1)-approximation with an additional O(1)-additive error but with exponential sample complexity. The manuscript also discusses the importance of clustering algorithms in unsupervised learning and the lack of a universally agreed-upon definition for the quality of clustering solutions. It highlights the challenges and trade-offs that algorithm designers face due to factors such as random initialization, similarity measures, noise in measurements, and outliers in the dataset. These issues contribute to non-replicable results in clustering algorithms. Furthermore, the manuscript provides related work on norms and parameters used in clustering and presents experimental results on synthetic distributions to validate the proposed replicable algorithms.\n\nSTRENGTHS:\nThe manuscripts' innovation lies in proposing replicable clustering algorithms for statistical clustering problems.  It introduces the concept of replicability and highlights its significance in ensuring the validity and reliability of scientific findings.  The document aims to address concerns about replicability in clustering algorithms, which is a crucial topic in the subfields of machine learning and data science.  It presents replicable algorithms for k-medians, k-means, and k-centers problems and outlines their theoretical results.  Overall, the manuscripts' innovation lies in its contribution to the development of replicable algorithms for statistical clustering.\n\nWEAKNESSES:\nThere are many theorems that are not well understood, and then there are relatively few experimental parts.\n\nQUESTIONS:\n1] How does the proposed replicable algorithm compare to existing clustering algorithms in terms of performance and accuracy?  \n2] Are there any limitations or assumptions in the proposed replicable algorithms that could affect their applicability in real-world scenarios? \n3] How generalizable are the experimental results presented on synthetic distributions in 2D?  Can the replicable algorithms be applied to other types of datasets?\n4] How sensitive are the replicable algorithms to the choice of parameters such as the choice of exponent in the cost function or the measure of similarity of the data? \n5] Does the study discuss any potential implications or future research directions regarding the replicability of clustering algorithms?\n\nLIMITATIONS:\nIn the experiment, real-world data sets are not used, which is not very convincing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the concept of replicability in clustering. This topic is important because replicability is what allows for other researchers to reproduce the results of a study to verify their correctness. This is a big problem these days, 50% of scientists saying that there is a replicability crisis. In the interest of improving the scientific verification process, designing replicable algorithms (i.e., algorithms that function the same when executed twice with the same random seed). This differs from the related notion of clustering stability in that its success is more dependent on the algorithmic design than structures in the data.\n\nIn their setting, points are sampled from a random distribution over the d-dimensional unit ball according to some given metric. It models many norms, including \\ell_p norms. They use Impagliazzo et al.’s notion of rho-replicability: The probability (over every two sampled sets of points and a single random seed) that the algorithm on each set but with the same random seed yields the same output is at least rho.  Outputs to these clustering problems are functions that, given a point, assigns it to a cluster (with no explicit center, just an identifier).\n\nThe main results are new algorithms for statistical k-means and k-medians, with extended improvements when using the Euclidean metric. Their approximation factors both near the approximation factor of a given black-box vanilla algorithm, and their sample complexity on general metrics has an exponential dependence on the dimension (it is sub-exponential in the Euclidean version). They also show an approximation algorithm for the replicable statistical k-centers algorithm. Given an approximation of the form a*OPT+b for k-centers, they give an approximation of the form c*OPT+d, where c and d are linear in both a and b. The query complexity is exponential in the dimensionality.\n\nTheir algorithm uses a quadtree decomposition, which creates a tree structure over all points. The leaves (which correspond to points) are then mapped to cluster centers. So when you receive a query for a point, you traverse the tree to find the leaf which contains it (as it is a partition of the space), and you map it to the corresponding cluster. However, in order to adequately depict the distribution of the space with their decomposition (which they are not given), they require access to an exponential number of queries in d. In the Euclidean case, they can first reduce the dimensionality via Johnson-Lindenstrauss, and map an epsilon-net from the original space to the low-dimensional space. There still is the difficulty of mapping the clustering between these two spaces, but it seems most of those details are in the appendix.\n\nFor replicability, they form a grid, sample points in the space, and map them to the centers of cells to estimate probability mass of each cell. On high-mass cells according to a random threshold, and then they apply a vanilla algorithm.\n\nFinally, they compare k-means++ and k-means++ run on the core produced for their replicability results. It is pretty clear that their approach leads to much greater replicability.\n\nSTRENGTHS:\nThis paper is written extremely clearly. I really appreciate how slow and careful the introduction and preliminaries are, which I believe could be followed by someone unfamiliar with the field. For instance, while many of us take for granted that a replicable clustering algorithm optimizes for some utility notion, they take the time to explain this and how it is distinct from their replicability goal (which is a bigger focus of their preliminary section). They also take the time to point out to the reader that their metric function and the ell_p parameter p are different, and give an in-depth but understandable explanation of the intuition behind ell_p for different values of p. The informal presentation at the beginning of results in section 3 is also very nice.\n\nThe results are nice and interesting, but I wouldn’t say they are groundbreaking. The techniques have clear foundations in previous methods, but it also seems unique how they combined them together. I wasn’t sure by reading the paper how their replicability results compare to previous literature.\n\nWEAKNESSES:\nI found it very odd that the main focus of the paper introduction and title was replicable k-centers, but they spend most of their time on (nonreplicable) k-centers. It seems like if the highlight of the paper was replicability, they should have focused on the methods used to achieve this. I feel the details of this algorithm were insufficient for a reader invested in replicable k-centers. Additionally, I wasn’t sure how their results compared to any baseline results that exist, so it is difficult to judge the novelty.\n\nQUESTIONS:\n1. I was a bit confused by the sentence “We emphasize that even though we only observe a sample from P we aim to solve the problem on the whole support of the distribution.” Can you clarify exactly what this means? Is it that the algorithm works on an actual given dataset P but needs to be able to work on all possible datasets? I would think that would be assumed.\n2. How do your algorithms compare to existing algorithms? Is there any baseline implied by any other work?\n\nLIMITATIONS:\nThere was no limitations section, though it did not seem needed (but would be preferred).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe topic of this paper is replicable clustering which in high-level asks for design of an algorithm that given two runs of the algorithm on different samples from the “same” input distribution. Replicability is a notion introduced recently in a work by Impagliazzo et al. [2022].  \nThe algorithm for statistical clustering with centroid-based objective with $p$-norm cost. For the case of $k$-means and $k$-medians, via corresponding centroid-based $k$-clustering on “finite” size (referred to as combinatorial clustering) in a black-box manner and design $O(1)$-approximation with poly($d$) sample complexity (but exponential in k?). However, the case of $k$-center is more complicated as it depends on the $\\ell_\\infty$-norm and very sensitive to the largest distance. The results on $k$-center has worse approximation guarantee and sample complexity and requires stronger assumptions. In particular, for the case of $k$-center, the guarantee is bicriterion.\n\nSTRENGTHS:\nAt high-level, the notion is interesting and related to several other important topics too. The technical contribution of the paper seems to be satisfactory. However, I'm less familiar with the literature and the technicality of the paper might not be significant for someone working on this area or differential privacy.\n\nWEAKNESSES:\nThe paper is following the notion of [Impagliazzo et al., 20]; however, I still not quite convinced why the shared randomness assumption is meaningful. A related question is whether an assumption such as Assumption F. 2 can be sufficient for achieving replicability?\n\nThe result seems to heavily depend on the coreset construction of Frahling and Sohler [2005]. Can you elaborate why more recent coreset construction can be applied here?\n\nTheorem 3.1 and Theorem 3.2 provide upper bounds but there is no lowerbound. E.g., Any hope to show exponential dependence on k (or d) is needed (d, for the general metric space)?\n\nHow the algorithm in this paper is different from those for differentially private clustering? \n\nCan you elaborate on the proof of Corollary E. 9? I particular, how does the dimension show up?\n\nQUESTIONS:\nPlease see the questions in the weakness section.\n\nLIMITATIONS:\nThere is no discussion of limitation by the authors. Though, the paper does not seem to have potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper initiates the study of formal replicability for clustering algorithms. Replicability is defined to be a property of an algorithm for a statistical clustering problem, requiring that fixing the internal randomness of the algorithm while resampling the input data will yield exactly the same (representation of a) mapping from points to clusters. Upper-bounds are given for the statistical k-means, k-medians, and k-centers problems, with poly(d) sample complexity proven via dimensionality reduction in the case of euclidian distance. The theoretical results are also empirically evaluated on synthetic data.\n\nSTRENGTHS:\nThis paper addresses an interesting problem by extending recently introduced formal notions of replicability to the task of statistical clustering. Clustering seems like a setting where replicability is particularly well-motivated, as there are many reasonable desirable properties for clustering solutions, and being able to ensure that solutions which demonstrate a particular balance of these properties are in fact a result of the algorithm generating solutions, and not just a fluke of the sample, is a natural objective.\n\nWEAKNESSES:\nThe writing was very readable, but it would have been nice to have a more thorough comparison of this work to the related work. What is the cost of replicable clustering compared to non-replicable algorithms for the same problems? Does this notion of replicability solve any of the issues that motivated the \"stability of centers\" condition for selection of k that is mentioned in Section 1.1?\n\nI also found Section 4 a bit difficult to parse. I think some of the notation used in Algorithm 4.1 isn't defined, and though it's reasonably intuitive, using more precise language in the high-level exposition and fully defining all variables/notation would definitely improve the readability of this section.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes, the authors address all limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-31"}