SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Replicable Clustering*
Hossein Esfandiari
Google Research esfandiari@google.com
Yale University, Google Research  
amin.karbasi@yale.edu
Google Research mirrokni@google.com
Yale University grigoris.velegkas@yale.edu
Yale University  
felix.zhou@yale.edu
We design replicable algorithms in the context of statistical clustering under the recently introduced notion of replicability from Impagliazzo et al. [2022]. According to this definition, a clustering algorithm is replicable if, with high probability, its output induces the exact same partition of the sample space after two executions on different inputs drawn from the same distribution, when its internal randomness is shared across the executions. We propose such algorithms for the statistical  $k$ -medians, statistical  $k$ -means, and statistical  $k$ -centers problems by utilizing approximation routines for their combinatorial counterparts in a black-box manner. In particular, we demonstrate a replicable  $O(1)$ -approximation algorithm for statistical Euclidean  $k$ -medians ( $k$ -means) with  $\tilde{O}(\mathrm{poly}(k,d)k^{\log \log k})$  sample complexity. We also describe an  $O(1)$ -approximation algorithm with an additional  $O(1)$ -additive error for statistical Euclidean  $k$ -centers, albeit with  $\tilde{O}(\mathrm{poly}(k)\exp(d))$  sample complexity. In addition, we provide experiments on synthetic distributions in 2D using the  $k$ -means++ implementation from sklearn as a black-box that validate our theoretical results<sup>2</sup>.
The unprecedented increase in the amount of data that is available to researchers across many different scientific areas has led to the study and development of automated data analysis methods. One fundamental category of such methods is unsupervised learning which aims to identify some inherent structure in unlabeled data. Perhaps the most well-studied way to do that is by grouping together data that share similar characteristics. As a result, clustering algorithms have become one of the central objects of study in unsupervised learning. Despite a very long line of work studying such algorithms, e.g. Jain and Dubes [1988], Hart et al. [2000], Anderberg [2014], there is not an agreed-upon definition that quantifies the quality of a clustering solution. Kleinberg [2002] showed that there is an inherent reason why this is the case: it is impossible to design a clustering function that satisfies three natural properties, namely scale-invariance, richness of solutions, and consistency. This means that the algorithm designer needs to balance several conflicting desiderata. As a result, the radically different approaches that scientists use depending on their application domain can be sensitive to several factors such as their random initialization, the measure of similarity of the data, the presence of noise in the measurements, and the existence of outliers in the dataset. All these issues give rise to algorithms whose results are not replicable, i.e., when we execute them on two different samples of the same population, they output solutions that vary significantly. This begs the
$^{2}$ https://anonymous.4open.science/r/replicable_clustering_experiments-E380
following question. Since the goal of clustering algorithms is to reveal properties of the underlying population, how can we trust and utilize their results when they fail to pass this simple test?
Replicability is imperative in making sure that scientific findings are both valid and reliable. Researchers have an obligation to provide coherent results and conclusions across multiple repetitions of the same experiment. Shockingly, a 2016 survey that appeared in Nature [Baker, 2016] revealed that  $70\%$  of researchers tried, but were unable to, replicate the findings of another researcher and more than  $50\%$  of them believe there is a significant crisis in replicability. Unsurprisingly, similar worries have been echoed in the subfields of machine learning and data science [Pineau et al., 2019, 2021]. In this work, we initiate the study of replicability in clustering, which is one of the canonical problems of unsupervised learning.
Statistical Clustering. The most relevant previous results for (non-replicable) statistical clustering was established by Ben-David [2007], who designed  $O(1)$ -approximation algorithms for statistical  $k$ -medians ( $k$ -means) with  $O(k)$  sample complexity. However, their algorithm picks centers from within the samples and is therefore non-replicable.
Combinatorial Clustering. The flavor of clustering most studied in the approximation algorithms literature is the setting where we have a uniform distribution over finite points and the algorithm has explicit access to the entire distribution. Our algorithms rely on having black-box access to a combinatorial clustering oracle. See Byrka et al. [2017], Ahmadian et al. [2019] for the current best polynomial-time approximation algorithms for combinatorial  $k$ -medians ( $k$ -means) in general metrics with approximation ratio 2.675 (9). Also, see Cohen-Addad et al. [2022] for a 2.406 (5.912) approximation algorithm for the combinatorial Euclidean  $k$ -medians ( $k$ -means).
Clustering Stability. Stability in clustering has been studied both from a practical and a theoretical point of view [Ben-Hur et al., 2001, Lange et al., 2004, Von Luxburg and Ben-David, 2005, Ben-David et al., 2006, Rakhlin and Caponnetto, 2006, Ben-David et al., 2007, Von Luxburg et al., 2010]. In most applications, it is up to the algorithm designer to decide upon the value of  $k$ , i.e., the number of different clusters. Thus, it was proposed that a necessary condition it should satisfy is that it leads to solutions that are not very far apart under resampling of the input data [Ben-Hur et al., 2001, Lange et al., 2004]. However, it was shown that this notion of stability for center-based clustering is heavily based on symmetries within the data which may be unrelated to clustering parameters [Ben-David et al., 2006]. Our results differ from this line of work in that we require the output across two separate samples to be exactly the same with high probability, when the randomness is shared. Moreover, our work reaffirms Ben-David et al. [2006] in that their notion of stability can be perfectly attained no matter the choice of  $k$ .
Other notions of stability related to our work include robust hierarchical clustering [Balcan et al., 2014], robust online clustering [Lattanzi et al., 2021], average sensitivity [Yoshida and Ito, 2022], and differentially private (DP) clustering [Cohen et al., 2021, Ghazi et al., 2020]. The definition of replicability we use is statistical and relies on an underlying data distribution while (DP) provides a worst-case combinatorial guarantee for two runs of the algorithm on neighboring datasets. Bun et al. [2023], Kalavasis et al. [2023] provide connections between DP and replicability for statistical learning problems. However, these transformations are not computationally efficient. It would be interesting to come up with computationally efficient reductions between replicable and DP clustering algorithms.
Coresets for Clustering. A long line of work has focused on developing strong coresets for various flavors of centroid-based clustering problems. See Sohler and Woodruff [2018] for an overview of this rich line of work. The most relevant for our results include coresets for dynamic geometric streams through hierarchical grids [Frahling and Sohler, 2005] and sampling based techniques [Ben-David, 2007, Feldman and Langberg, 2011, Bachem et al., 2018].
Dimensionality Reduction. Dimensionality reduction for clustering has been a popular area of study as it reduces both the time and space complexity of existing algorithms. The line of work on data-oblivious dimensionality reduction for  $k$ -means clustering was initiated by Boutsidis et al. [2010]. The goal is to approximately preserve the cost of all clustering solutions after passing the data through a dimensionality reduction map. This result was later improved and generalized to  $(k, p)$ -clustering [Cohen et al., 2015, Becchetti et al., 2019], culminating in the work of Makarychev
et al. [2019], whose bound on the target dimension is sharp up to a factor of  $\log^{1 / \varepsilon}$ . While Charikar and Waingarten [2022] overcome this factor, their result only preserves the cost across the optimal solution.
Replicability in ML. Our results extend the recently initiated line of work on designing provably replicable learning algorithms under the definition that was introduced by Impagliazzo et al. [2022]. Later, Esfandiari et al. [2022] considered a natural adaption of this definition to the setting of bandits and designed replicable algorithms that have small regret. A slightly different notion of replicability in optimization was studied in Ahn et al. [2022], where it is required that an optimization algorithm that uses noisy operations during its execution, e.g., noisy gradient evaluations, outputs solutions that are close when executed twice. Subsequently, Bun et al. [2023], Kalavasis et al. [2023] established strong connections between replicability and other notions of algorithmic stability. Recently, Dixon et al. [2023], Chase et al. [2023] proposed a weaker notion of replicability where the algorithm is not required to output the same solution across two executions, but its output needs to fall into a small list of solutions.
2 Setting & Notation
Let  $\mathcal{X} \subseteq \mathbb{R}^d$  be the instance space endowed with a metric  $\kappa: \mathcal{X} \times \mathcal{X} \to \mathbb{R}_+$  and  $\mathbb{P}$  be a distribution on  $\mathcal{X}$  which generates the i.i.d. samples that the learner observes.
For  $F \subseteq \mathbb{R}^d$  and  $x \in \mathcal{X}$ , we overload the notation and write  $F(x) \coloneqq \operatorname*{argmin}_{f \in F} \kappa(x, f)$  to be the closest point to  $x$  in  $F$  as well as  $\kappa(x, F) \coloneqq \kappa(x, F(x))$  to be the shortest distance from  $x$  to a point in  $F$ .
We assume that  $\mathcal{X}$  is a subset of  $\mathcal{B}_d$ , the  $d$ -dimensional  $\kappa$ -ball of diameter 1 centered about the origin<sup>3</sup>. We also assume that  $\kappa$  is induced by some norm  $\| \cdot \|$  on  $\mathbb{R}^d$  that is sign-invariant (invariant to changing the sign of a coordinate) and normalized (the canonical basis has unit length). Under these assumptions, the unit ball of  $\kappa$  is a subset of  $[-1,1]^d$ .
Our setting captures a large family of norms, including the  $\ell_p$ -norms, Top- $\ell$  norms (sum of  $\ell$  largest coordinates in absolute value), and ordered norms (non-negative linear combinations of Top- $\ell$  norms) [Chakrabarty and Swamy, 2019]. Our results hold for more general classes of norms but for the sake of simplicity, we abide by these assumptions.
We define  $\Delta := \sup \{\kappa(x, y) : x, y \in [0, 1)^d\}$  to be the  $\kappa$ -diameter of the unit hypercube. Note that  $1 \leq \Delta \leq d$  by assumption. Moreover,  $L\Delta$  is the  $\kappa$ -diameter of a hypercube with side length  $L$ . For example, if  $\| \cdot \| = \| \cdot \|_2$  is the Euclidean norm, then  $\Delta = \sqrt{d}$ .
2.1 Clustering Methods and Generalizations
We now introduce the clustering objectives that we study in this work, which all fall in the category of minimizing a cost function  $\mathrm{cost} : \mathcal{F} \to \mathbb{R}_+$ , where  $\mathcal{F} \coloneqq \{ F \subseteq \mathcal{B}_d : |F| = k \}$ . We write  $\mathrm{cost}(F)$  to denote the objective in the statistical setting and  $\widehat{\mathrm{cost}}(F)$  for the combinatorial setting in order to distinguish the two.
Problem 2.1 (Statistical  $(k,p)$ -Clustering). Given i.i.d. samples from a distribution  $\mathbb{P}$  on  $\mathcal{X} \subseteq \mathcal{B}_d$ , minimize  $\mathrm{cost}(F) \coloneqq \mathbb{E}_{x \sim \mathbb{P}} \kappa(x,F)^p$ .
In other words, we need to partition the points into  $k$  clusters so that the expected distance of a point to the center of its cluster, measured by  $\kappa (\cdot ,\cdot)^{p}$ , is minimized. This is closely related to the well-studied combinatorial variant of the  $(k,p)$ -clustering problem.
Problem 2.2  $((k,p)$ -Clustering). Given some points  $x_{1},\ldots ,x_{n}\in \mathcal{X}$ , minimize  $\widehat{\mathrm{cost}} (F):= \frac{1}{n}\sum_{i = 1}^{n}\kappa (x_i,F)^p$
We note that (statistical)  $k$ -medians and (statistical)  $k$ -means is a special case of Problem 2.2 (Problem 2.1) with  $p = 1, 2$ , respectively. We also consider a slight variant of the combinatorial problem, i.e., Problem 2.2, where we allow different points  $x_{i}$  to participate with different weights  $w_{i}$  in the objective. We refer to this problem as the weighted  $(k,p)$ -clustering problem.
3Our results can be generalized to  $\kappa$ -balls of diameter  $L$  with an arbitrary center through translation and scaling.
We now shift our attention to the  $k$ -centers problem.
Problem 2.3 (Statistical  $k$ -Centers). Given i.i.d. samples from a distribution  $\mathbb{P}$  on  $\mathcal{X} \subseteq \mathcal{B}_d$ , minimize  $\mathrm{cost}(F) := \max_{x \in \mathcal{X}} \kappa(x, F)$ .
Notice that the  $\ell_{\infty}$  norm is the limit of the  $\ell_p$  norm as  $p$  tends to infinity, hence  $k$ -centers is, in some sense, the limit of  $(k,p)$ -clustering as  $p$  tends to infinity. Also, notice that this problem differs from  $k$ -means and  $k$ -medians in the sense that it has a min-max flavor, whereas the other two are concerned with minimizing some expected values. Due to this difference, we need to treat  $k$ -centers separately from the other two problems, and we need to make some assumptions in order to be able to solve it from samples (cf. Assumption F.1, Assumption F.2). We elaborate more on that later.
Let us also recall the combinatorial version of  $k$ -centers.
Problem 2.4 (k-Centers). Given some points  $x_{1},\ldots ,x_{n}\in \mathcal{X}$ , minimize  $\widehat{\operatorname{cost}} (F)\coloneqq \max_{i\in [n]}\kappa (x_i,F)$ .
We remark that clustering has mainly been studied from the combinatorial point of view, where the distribution is the uniform distribution over some finite points and we are provided the entire distribution. The statistical clustering setting generalizes to arbitrary distributions with only sample access. We emphasize that although we only have access to samples, our output should be a good solution for the entire distribution and not just the observed data.
We write  $F_{\mathrm{OPT}}$  to denote an optimal solution for the entire distribution and OPT := cost(FOPT). Similarly, we write  $\widehat{F}_{\mathrm{OPT}}$  to denote an optimal sample solution and  $\widehat{\mathrm{OPT}} := \widehat{\mathrm{cost}}(\widehat{F}_{\mathrm{OPT}})$ . Suppose we solve Problem 2.4 given a sample of size  $n$  from Problem 2.3. Then  $\widehat{\mathrm{OPT}} \leq \mathrm{OPT}$  since we are optimizing over a subset of the points.
Recall that a  $\beta$ -approximate solution  $F$  is one which has cost  $\mathrm{cost}(F) \leq \beta$  OPT. Note this is with respect to the statistical version of our problems. An algorithm that outputs  $\beta$ -approximate solutions is known as a  $\beta$ -approximation algorithm. We also say that  $F$  is a  $(\beta, B)$ -approximate solution if  $\mathrm{cost}(F) \leq \beta \mathrm{OPT} + B$ .
2.1.1 Parameters  $p$  and  $\kappa$
Here we clarify the difference between  $p$  and  $\kappa$ , which are two separate entities in the cost function  $\mathbb{E}_x[\kappa (x,F(x))^p]$ . We denote by  $\kappa$  the distance metric used to measure the similarity between points. The most commonly studied and applied option is the Euclidean distance for which our algorithms are the most sample-efficient. On the other hand,  $p$  is the exponent to which we raise the distances when computing the cost of a clustering. A smaller choice of  $p$  puts less emphasis on points that are far away from centers and  $p = 1$  seeks to control the average distance to the nearest center. A large choice of  $p$  puts emphasis on points that are further away from centers and as  $p$  tends to infinity, the objective is biased towards solutions minimizing the maximum distance to the nearest center. Thus we can think of  $k$ -centers as  $(k,p)$ -clustering when  $p = \infty$ . As a concrete example, when  $\kappa$  is the Euclidean distance and  $p = 5$ , the cost function becomes  $\mathbb{E}_x\left[\| x - F(x)\|_2^5\right]$ .
Throughout this work, we study replicability<sup>4</sup> as an algorithmic property using the definition of Impagliazzo et al. [2022].
Definition 2.5 (Replicable Algorithm; [Impagliazzo et al., 2022]). Let  $\rho \in (0,1)$ . A randomized algorithm  $\mathcal{A}$  is  $\rho$ -replicable if for two sequences of  $n$  i.i.d. samples  $\bar{X}, \bar{Y}$  generated from some distribution  $\mathbb{P}^n$  and a random binary string  $\bar{r} \sim R(\mathcal{X})$ ,
$$
\mathbb {P} _ {\bar {X}, \bar {Y} \sim \mathbb {P} ^ {n}, \bar {r} \sim R (\chi)} \left\{\mathcal {A} (\bar {X}; \bar {r}) = \mathcal {A} (\bar {Y}; \bar {r}) \right\} \geq 1 - \rho .
$$
In the above definition, we treat  $\mathcal{A}$  as a randomized mapping to solutions of the clustering problem. Thus, even when  $\bar{X}$  is fixed,  $\mathcal{A}(\bar{X})$  should be thought of as random variable, whereas  $\mathcal{A}(\bar{X};\bar{r})$  is the realization of this variable given the (fixed)  $\bar{X},\bar{r}$ . We should think of  $\bar{r}$  as the shared randomness
<sup>4</sup>Originally this definition was called reproducibility [Impagliazzo et al., 2022] but it was later pointed out that the correct term is replicability [Ahn et al., 2022].
between the two executions. In practice, it can be implemented as a shared random seed. We underline that sharing the randomness across executions is crucial for the development of our algorithms. We also note that by doing that we couple the two random variables  $\mathcal{A}(\bar{X}),\mathcal{A}(\bar{Y})$ , whose realization depends on  $r\sim R(\mathcal{X})$ . Thus, if their realizations are equal with high probability under this coupling, it means that the distributions of  $\mathcal{A}(\bar{X}),\mathcal{A}(\bar{Y})$  are statistically close. This connection is discussed further in Kalavasis et al. [2023].
In the context of a clustering algorithm  $\mathcal{A}$ , we interpret the output  $\mathcal{A}(\bar{X};\bar{r})$  as a clustering function  $f:\mathcal{X}\to [k]$  which partitions the support of  $\mathbb{P}$ . The definition of  $\rho$ -replicability demands that  $f$  is the same with probability at least  $1 - \rho$  across two executions. We note that in the case of centroid-based clustering such as  $k$ -medians and  $k$ -means, the induced partition is a function of the centers and thus it is sufficient to output the exact same centers with probability  $1 - \rho$  across two executions. However, we also allow for algorithms that create partitions implicitly without computing their centers explicitly.
Our goal is to develop replicable clustering algorithms for  $k$ -medians,  $k$ -means, and  $k$ -centers, which necessitates that the centers we choose are arbitrary points within  $\mathbb{R}^d$  and not only points among the samples. We underline that as in the case of differential privacy, it is trivial to design algorithms that satisfy the replicability property, e.g. we can let  $\mathcal{A}$  be the constant mapping. The catch is that these algorithms do not achieve any utility. In this work, we are interested in designing replicable clustering algorithms whose utility is competitive with their non-replicable counterparts.
In this section, we informally state our results for replicable statistical  $k$ -medians ((k, 1)-clustering),  $k$ -means ((k, 2)-clustering), and  $k$ -centers under general distances. Unfortunately, generality comes at the cost of exponential dependency on the dimension  $d$ . We also state our results for replicable statistical  $k$ -medians and  $k$ -means specifically under the Euclidean distance, which has a polynomial dependency on  $d$ . Two key ingredients is the uniform convergence of  $(k, p)$ -clustering costs (cf. Theorem C.9) as well as a data-oblivious dimensionality reduction technique for  $(k, p)$ -clustering in the distributional setting (cf. Theorem E.10). These results may be of independent interest.
We emphasize that the Euclidean  $k$ -median and  $k$ -means are the most studied and applied flavors of clustering, thus the sample complexity for the general case and the restriction to  $p = 1,2$  does not diminish the applicability of our approach.
The main bottleneck in reducing the sample complexity for general norms is the lack of a data-oblivious dimensionality reduction scheme. This bottleneck is not unique to replicability and such a scheme for general norms would be immediately useful for many distance-based problems including clustering. It may be possible to extend our results to general  $(k,p)$ -clustering beyond  $p = 1,2$ . The main challenge is to develop an approximate triangle inequality for  $p$ -th powers of norms. Again, this limitation is not due to replicability but rather the technique of hierarchical grids. It is a limitation shared by Frahling and Sohler [2005].
Before stating our results, we reiterate that the support of our domain  $\mathcal{X}$  is a subset of the unit-diameter  $\kappa$ -ball  $\mathcal{B}_d$ . In particular, we have that  $\mathrm{OPT} \leq 1$ .
Theorem 3.1 (Informal). Let  $\varepsilon, \rho \in (0,1)$ . Given black-box access to a  $\beta$ -approximation oracle for weighted  $k$ -medians, respectively weighted  $k$ -means (cf. Problem 2.2), there is a  $\rho$ -replicable algorithm for statistical  $k$ -medians, respectively  $k$ -means (cf. Problem 2.1), such that with probability at least 0.99, it outputs a  $(1 + \varepsilon)\beta$ -approximation. Moreover, the algorithm has sample complexity
$$
\tilde {O} \left(\mathrm {p o l y} \left(\frac {k}{\rho \mathrm {O P T}}\right) \left(\frac {2 \Delta}{\varepsilon}\right) ^ {O (d)}\right).
$$
When we are working in Euclidean space, we can get improved results for these problems.
Theorem 3.2 (Informal). Let  $\rho \in (0,1)$ . Suppose we are provided with black-box access to a  $\beta$ -approximation oracle for weighted Euclidean  $k$ -medians ( $k$ -means). Then there is a  $\rho$ -replicable algorithm that partitions the input space so with probability at least 0.99, the cost of the partition is
at most  $O(\beta \mathrm{OPT})$ . Moreover, the algorithm has sample complexity
$$
\tilde {O} \left(\operatorname {p o l y} \left(\frac {d}{\mathrm {O P T}}\right) \left(\frac {k}{\rho}\right) ^ {O (\log \log (k / \rho))}\right).
$$
We underline that in this setting we compute an implicit solution to Problem 2.1, since we do not output  $k$  centers. Instead, we output a function  $f$  that takes as input a point  $x \in \mathcal{X}$  and outputs the label of the cluster it belongs to in polynomial time. The replicability guarantee states that, with probability  $1 - \rho$ , the function will be the same across two executions.
The combinatorial  $k$ -medians ( $k$ -means) problem where the centers are restricted to be points of the input is a well-studied problem from the perspective of polynomial-time constant-factor approximation algorithms. See Byrka et al. [2017], Ahmadian et al. [2019] for the current best polynomial-time approximation algorithms for combinatorial  $k$ -medians ( $k$ -means) in general metrics with approximation ratio 2.675 (9). Also, see Cohen-Addad et al. [2022] for a 2.406 (5.912) approximation algorithm for the combinatorial Euclidean  $k$ -medians ( $k$ -means).
As we alluded to before, in order to solve  $k$ -centers from samples, we need to make an additional assumption. Essentially, Assumption F.2 states that there is a  $(\beta, B)$ -approximate solution  $F$ , such that, with some constant probability, e.g. 0.99, when we draw  $n$  samples from  $\mathbb{P}$  we will observe at least one sample from each cluster of  $F$ .