SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For any dataset  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$ , a projection to a random lower-dimensional space affects the pairwise distance between the projected points in a predictable manner — the Johnson-Lindenstrauss lemma which projects to  $O(\log n)$  dimensions being a prime example of this fact. When projecting to just one dimension, however, pairwise distances will be significantly affected (by up to  $\mathrm{poly}(n)$ -factors). Thus, a naive analysis will give a  $\mathrm{poly}(n)$ -approximation for  $k$ -means. To improve a  $c$ -approximation to a  $O(\log k)$ -approximation, one needs a coreset of size roughly  $\mathrm{poly}(c / \log k)$ . This bound becomes vacuous when  $c$  is polynomial in  $n$  since there are at most  $n$  dataset points.
However, although many pairwise distances are significantly distorted, we show that the  $k$ -means cost is only affected by a poly  $(k)$ -factor. At a high level, this occurs because the  $k$ -means cost optimizes a sum of pairwise distances (according to a chosen clustering). The individual summands, given by pairwise distances, will change significantly, but the overall sum does not. Our proof follows the approach of [21], which showed that (roughly speaking) pairwise distortion of the  $k$  optimal centers suffices to argue about the  $k$ -means cost. The  $k$  optimal centers will incur maximal pairwise distortion poly  $(k)$  when projected to one dimension (because there are only  $O(k^2)$  pairwise distances among the  $k$  centers). This allows us to lift an  $r$ -approximate solution after the projection to an  $O(k^4 r)$ -approximate solution for the original points.
Lemma 2.5 (Informal). For any set  $X$  of points in  $\mathbb{R}^d$ , the following occurs with probability at least 0.9 over the choice of a standard Gaussian vector  $v \in \mathbb{R}^d$ . Letting  $X' \subset \mathbb{R}$  be the one-dimensional projection of  $X$  onto  $v$ , any  $r$ -approximate  $k$ -means clustering of  $X'$  gives an  $O(k^4 r)$ -approximate clustering of  $X$  with the same clustering partition.
In this section, we outline the experimental evaluation of our algorithm. The experiments evaluate the algorithms in two different ways. For each, we measure the running time and the  $k$ -means cost of the resulting solution (the sum of squares of point-to-center-assigned distances). (1) First, we evaluate our algorithm as part of a pipeline incorporating a coreset construction – the expected use case for our algorithm. (2) Second, we evaluate our algorithm by itself for approximate  $k$ -means clustering and compare it to  $k$ -means++ [6]. As per Theorems 2.1 and 2.2, we expect our algorithm to be much faster but output an assignment of higher cost. Our goal is to quantify these differences empirically.
All experiments were run on Linux using a notebook with a 3.9 GHz 12th generation Intel Core i7 six-core processor and 32 GiB of RAM. All algorithms were implemented in  $\mathrm{C + + }$  , using the blaze library for matrix and vector operations performed on the dataset unless specified differently below. We provide our code in the supplementary material for this submission.
Datasets. For our experiments, we use the following four datasets:
KDD [47]: Training data for the 2004 KDD challenge on protein homology. The dataset consists of 145751 observations with 77 real-valued features.
Song [12]: Timbre information for 515345 songs with 90 features each, used for year prediction.
Census [32]: 1990 US census data with 2458285 observations, each with 68 categorical features.
Gaussian: A synthetic dataset consisting of 240005 points of dimension 4. The points are generated by placing a standard normal distribution at a large positive distance from the origin on each axis and sampling 30000 points. The points are then mirrored so the center of mass remains at the origin. Finally, 5 points are placed on the origin. This is an adversarial example for lightweight coresets [10], which are unlikely to sample points close to the mean of the dataset.
3.1 Coreset Construction Comparison
Experimental Setup. Coreset constructions (with multiplicative approximation guarantees) always proceed by first finding an approximate clustering, which constitutes the bulk of the work. The approximate clustering defines a "sensitivity sampling distribution" (we expand on this in the appendix, see also [9]), and a coreset is constructed by repeatedly sampling from the sensitivity sampling distribution. In our first experiment, we evaluate the choice of initial approximation algorithm used to define the sensitivity sampling distribution. We compare the use of  $k$ -means++ and PRONE. In addition, we also compare the lightweight coresets of [10], which uses the distance to the center of mass as an approximation of the sensitivity sampling distribution. For the remainder of this section, we refer to sensitivity sampling using  $k$ -means++ as Sensitivity and lightweight coresets as Lightweight. All three algorithms produce a coreset, and the experiment will measure the running time of the three algorithms (Table 1) and the quality of the resulting coresets (Figure 2). We implemented all algorithms in C++.
Once a coreset is constructed for each of the algorithms, we evaluate the quality of the coreset by computing the cost of the centers found when clustering the coreset (see Definition A.1). We run a state-of-the-art implementation of Lloyd's  $k$ -means algorithm from the scikit-learn library [57] with the default configuration (repeating 15 times and reporting the mean cost to reduce the variance). The resulting quality of the coresets is compared to a (computationally expensive) baseline, which runs  $k$ -means++ from the scikit-learn library, followed by Lloyd's algorithm with the default configuration on the entire dataset (repeated 5 times to reduce variance).
We evaluate various choices of  $k$  ( $\{10, 100, 1000\}$ ) as well as coresets at various relative sizes,  $\{0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1\}$  times the size of the dataset. We use as performance metrics (1) a relative cost, which measures the average cost of the k-means solutions returned by Lloyd's algorithm on each coreset divided by the baseline, and (2) the running time of the coreset construction algorithm.
Results on Coreset Constructions. Relative cost. Figure 2 shows the coreset size ( $x$ -axis) versus the relative cost ( $y$ -axis). Each "row" of Figure 2 corresponds to a different value for  $k \in \{10, 100, 1000\}$ , and each "column" corresponds to a different dataset. Recall that the first three datasets (i.e., the first three columns) are real-world datasets, and the fourth column is the synthetic Gaussian dataset. We note our observations below:
- As expected, on all real-world data sets and all settings of  $k$ , the relative cost decreases as the coreset size increases.
- In real-world datasets, the specific relative cost of each coreset construction (Sensitivity, Lightweight, and ours) depends on the dataset<sup>4</sup>, but roughly speaking, all three share a similar trend. Ours and Sensitivity are very close and never more than twice the baseline (usually much better).
4The spike in relative cost for algorithm Sensitivity on the KDD data set for relative size  $5\cdot 10^{-3}$  is due to outliers.
Figure 2: Plot of relative cost versus coreset size on our four datasets. The shaded region indicates standard error. There is no data for relative size values where the coreset size is less than  $k$ .
- The big difference, distinguishing ours and Sensitivity from Lightweight, is the fourth column, the synthetic Gaussian dataset. For all settings of  $k$ , as the coreset size increases, Lightweight exhibits a minimal cost decrease and is a factor of 2.7-17x times worse than ours and Sensitivity (as well as the baseline). This is expected, as we constructed the synthetic Gaussian dataset to have arbitrarily high cost with Lightweight. Due to its multiplicative approximation guarantee, our algorithm does not suffer this degradation. In that sense, our algorithm is more "robust," and achieves worst-case multiplicative approximation guarantees for all datasets.
Running time. In (the first table in) Table 1, we show the running time of the coreset construction algorithms as  $k$  increases. Notice that as  $k$  increases, the relative speedup of our algorithm and Lightweight increases in comparison to Sensitivity. This is because our algorithm and Lightweight have running time which does not grow with  $k$ . In contrast, the running time of Sensitivity grows linearly in  $k$ . In summary, our coreset construction is between 33-192x faster than Sensitivity for large  $k$ . In addition, our algorithm runs about 3-5x slower than Lightweight, depending on the dataset. Our analysis also shows this; both algorithms make an initial pass over the dataset, using  $O(nd)$  time, but ours uses an additional  $O(n\log n)$  time to process.
3.2 Direct k-Means++ Comparison
Experimental Setup. This experiment compares our algorithm and  $k$ -means++ as a stand-alone clustering algorithm, as opposed to as part of a coreset pipeline. We implemented three variants of our algorithm. Each differs in how we sample the random one-dimensional projection. The first is a one-dimensional projection onto a standard Gaussian vector (zero mean and identity covariance). This approach risks collapsing an "important" feature, i.e. a feature with high variance. To mitigate this, we implemented two data-dependent variants that use the variance, resp. covariance of the data. Specifically, in the "variance" variant, we use a diagonal covariance matrix, where each entry in the diagonal is set to the empirical variance of the dataset along the corresponding feature. In the "covariance" variant, we use the empirical covariance matrix of the dataset. These variants aim to project along vectors that capture more of the variance of the data than when sampling a vector uniformly at random. Intuitively, the vectors sampled by the biased variants are more correlated with the first principal component of the dataset. For each of our algorithms, we evaluate the  $k$ -means cost of the output set  $C$  of centers when assigning points to the closest center  $(\mathrm{cost}_2(X,C)$  in Definition A.1) and when using our algorithm's assignment  $(\mathrm{cost}_2(X,C,\sigma)$  defined in Equation (1)).
We evaluated the algorithms for every  $k$  in  $\{10, 25, 50, 100, 250, 500, 1000, 2500, 5000\}$  and  $z = 2$ , for solving  $k$ -means with the  $\ell_2$ -metric. When evaluating the assignment cost, we ran each of our
Figure 3: Clustering cost of all our variants compared to  $k$ -means++. The top row shows the  $k$ -means cost, and the bottom row shows the cost of the assignment produced by our algorithm.
Dataset	k Algorithm	10	100	1000
Census	Lightweight	7.3	69.4	670.2
PRONE coreset	1.5	14.1	136.3
Sensitivity	1.0	1.0	1.0
Song	Lightweight	8.9	87.2	875.3
PRONE coreset	2.1	19.9	187.9
Sensitivity	1.0	1.0	1.0
KDD	Lightweight	6.4	63.0	642.8
PRONE coreset	2.1	19.6	192.6
Sensitivity	1.0	1.0	1.0
Gaussian	Lightweight	2.4	17.6	174.8
PRONE coreset	0.5	3.8	33.7
Sensitivity	1.0	1.0	1.0
Dataset	k Algorithm	50	500	5000
Census	PRONE	7.5	73.2	662.5
PRONE (variance)	2.2	22.2	214.7
PRONE (covariance)	1.1	10.7	117.4
k-means++	1.0	1.0	1.0
Song	PRONE	9.7	95.5	837.5
PRONE (variance)	2.3	23.1	217.2
PRONE (covariance)	0.8	8.2	82.4
k-means++	1.0	1.0	1.0
KDD	PRONE	6.9	68.3	727.5
PRONE (variance)	3.1	32.0	312.4
PRONE (covariance)	1.3	12.9	128.4
k-means++	1.0	1.0	1.0
Gaussian	PRONE	1.9	18.3	165.9
PRONE (variance)	2.0	17.7	162.9
PRONE (covariance)	1.7	16.1	152.6
k-means++	1.0	1.0	1.0
Table 1: Average speedup over sensitivity sampling across all relative sizes for constructing coresets (in the first table) and average speedup over  $k$ -means++ as a stand-alone clustering algorithm (in the second table). The tables with the full range of parameters can be found in the appendix.
algorithms 100 times for each  $k$  and five times when computing the nearest neighbor assignment, and we report the average cost of the solutions and the average running time. Due to lower variance and much higher runtime,  $k$ -means++ was run five times.
Results on Direct  $k$ -Means++ Comparison. Cost. Figure 3 (on top) shows the cost of the centers found by our algorithm compared to those found by the  $k$ -means++ algorithm after computing the optimal assignment of points to the centers computed by the algorithm (computing this takes time  $O(ndk)$ ). That is, we compare the values of  $\mathrm{cost}_2(X,C)$  in Definition A.1. In summary, the  $k$ -means cost of all three variants of our algorithm are roughly the same and closely match that of  $k$ -means++. On the Gaussian data set, one run of the biased algorithm failed to pick a center from the cluster at the origin, leading to a high "outlier" cost and a corresponding spike in the plot.
We also compared the k-means cost for the assignment computed by our algorithm (so that our algorithm only takes time  $O(nd + n\log n)$  and not  $O(ndk)$ ) with the cost of k-means++ (bottom row of Figure 3). That is, we compare the values of  $\mathrm{cost}_2(X,C,\sigma)$  defined in Equation (1). The clustering cost of our algorithms is higher than that of k-means++. This is the predicted outcome from our theoretical results; recall Theorem 2.2 gives a poly(k)-approximation, as opposed to  $O(\log k)$  from  $k$ -means++. On the real-world data sets, it is between one order of magnitude (for  $k = 10$ ) and two orders of magnitude (for  $k = 5000$ ) worse than k-means++ for our unbiased variant and between a factor 2 (for  $k = 10$ ) and one order of magnitude (for  $k = 5000$ ) worse than k-means++ for our biased and covariance variants.
Running time. Table 1 shows the relative running time of our algorithm compared to k-means++, assuming that no nearest-center assignment is computed. Our algorithms are designed to have a running time independent of  $k$ , so we can see, from the second table in Figure 1, all of our variants offer significant speedups.
- The running time of our algorithm stays almost constant as  $k$  increases while the running time of k-means++ scales linearly with  $k$ . Specifically for  $k = 25$ , even our slowest variants have about the same running time as k-means++, while for  $k = 5000$ , it is at least 82x faster, and our fastest version is up to 837x faster over k-means++.
- The two variants can affect the quality of the chosen centers by up to an order of magnitude, but they are also significantly slower. The "variance" and "covariance" variants are slower (between 2-4x slower and up to 10x slower, respectively) than the standard variant, and they also become slower as the dimensionality  $d$  increases. We believe these methods could be further sped up, as the blaze library's variance computation routine appears inefficient for our use case.
4 Conclusion and Limitations
To summarize, we present a simple algorithm that provides a new tradeoff between running time and approximation ratio. Our algorithm runs in expected time  $O(\mathsf{nnz}(X) + n\log n)$  to produce a poly  $(k)$ -approximation; with additional poly  $(kd)\cdot \log n$  time, we improve the approximation to  $O(\log k)$ . This latter bound matches that of  $k$ -means++ but offers a significant speedup.
Within a pipeline for constructing coresets, our experiments show that the quality of the coreset produced (when using our algorithm as the initial approximation) outperforms the sensitivity sampling algorithm. It is slower than the lightweight coreset algorithm, but it is more "robust" as it is independent of the diameter of the data set. It does not suffer from the drawback of having an additive error linear in the diameter of the dataset, which can arbitrarily increase the cost of the lightweight coreset algorithm. When computing an optimal assignment for the centers returned by our algorithm, its cost roughly matches the cost for k-means++. When directly using the assignment produced by one variant of our algorithm, its cost is between a factor 2 and 10 worse while being up to 300 times faster.
Our experiments and running time analysis show that our algorithm is very efficient. However, the clustering quality achieved by our algorithm is sometimes not as good as other, slower algorithms. We show that this limitation is insignificant when we use our algorithm to construct coresets. It remains an interesting open problem to understand the best clustering quality (e.g., in terms of approximation ratio) an algorithm can achieve while being as efficient as ours, i.e., running in time  $O(nd + n \log n)$ . Another interesting problem is whether other means of projecting the dataset into a  $O(1)$  dimensional space exist, which lead to algorithms with improved approximation guarantees and running time faster than  $O(ndk)$ .
Moses Charikar was supported by a Simons Investigator award.
Lunjia Hu was supported by Moses Charikar's and Omer Reingold's Simons Investigators awards, Omer Reingold's NSF Award IIS-1908774, and the Simons Foundation Collaboration on the Theory of Algorithmic Fairness. Part of this work was done while Erik Waingarten was a postdoc at Stanford University, supported by an NSF postdoctoral fellowship and by Moses Charikar's Simons Investigator Award.
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 101019564 "The Design of Modern Fully Dynamic Data
Structures (MoDynStruct) and the Austrian Science Fund (FWF) project Z 422-N, project "Static and Dynamic Hierarchical Graph Decompositions", I 5982-N, and project "Fast Algorithms for a Reactive Network Layer (ReactNet)", P 33775-N, with additional funding from the netidee SCIENCE Stiftung, 2020-2024.
[66] Harsha Vardhan Simhadri, George Williams, Martin Aumüller, Matthijs Douze, Artem Babenko, Dmitry Baranchuk, Qi Chen, Lucas Hosseini, Ravishankar Krishnaswamny, Gopal Srinivasa, et al. Results of the neurips'21 challenge on billion-scale approximate nearest neighbor search. In NeurIPS 2021 Competitions and Demonstrations Track, pages 177-189. PMLR, 2022.
[67] Christian Sohler and David P Woodruff. Strong coresets for k-median and subspace approximation: Goodbye dimension. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 802-813. IEEE, 2018.
[68] Kunal Talwar. Bypassing the embedding: Algorithms for low dimensional metrics. In Proceedings of the Thirty-Sixth Annual ACM Symposium on Theory of Computing, STOC '04, page 281-290, New York, NY, USA, 2004. Association for Computing Machinery.
[69] Mo Tiwari, Martin J Zhang, James Mayclin, Sebastian Thrun, Chris Piech, and Ilan Shomorony. Banditpam: Almost linear time k-medoids clustering via multi-armed bandits. Advances in Neural Information Processing Systems, 33:10211–10222, 2020.
[70] Erik Waingarten. Notes for algorithms for big data: Clustering. https://drive.google.com/file/d/1T5YYGrA3kdi4_QGvF3c_foOZvXMDaRPw/view.
[71] Jing Wang, Jingdong Wang, Qifa Ke, Gang Zeng, and Shipeng Li. Fast approximate k-means via cluster closures. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3037-3044, 2012.
A Problem Setup and Background
In this work, we always consider datasets  $X = \{x_{1},\ldots ,x_{n}\} \subset \mathbb{R}^{d}$  of high-dimensional vectors, and we will measure their distance using the Euclidean  $(\ell_2)$  distance. Below, we define  $(k,z)$ -clustering. This problem introduces a parameter  $z\geq 1$ , which measures the sensitivity to outliers (as  $z$  grows, the clusterings become more sensitive to points furthest from the cluster center). The case of  $k$ -means corresponds to  $z = 2$ , but other values of  $z$  capture other well-known clustering objectives, like  $k$ -median (the case of  $z = 1$ ).
Definition A.1  $((k,z)$ -Clustering). Consider a dataset  $X = \{x_{1},\ldots ,x_{n}\} \subset \mathbb{R}^{d}$ , a desired number of clusters  $k\in \mathbb{N}$ , and a parameter  $z\geq 1$ . For a set of  $k$  centers  $C = \{c_{1},\dots ,c_{k}\} \subset \mathbb{R}^{d}$ , let  $\mathrm{cost}_z(X,C)$  denote the cost of using the center set  $C$  to cluster  $X$ , i.e.,
$$
\operatorname {c o s t} _ {z} (X, C) = \sum_ {i = 1} ^ {n} \min  _ {j \in [ k ]} \| x _ {i} - c _ {j} \| _ {2} ^ {z}.
$$
We let  $\mathsf{opt}_{k,z}(X)$  denote the optimal cost over all choices of  $C = \{c_1,\dots ,c_k\} \subset \mathbb{R}^d$  ..
$$
\operatorname{opt}_{k,z}(X) = \inf_{\substack{C\subset \mathbb{R}^{d}\\ |C|\leq k}}\operatorname{cost}_{z}(X,C).
$$
A  $(k,z)$ -clustering algorithm has the following specifications. The algorithm receives as input a dataset  $X = \{x_{1},\ldots ,x_{n}\} \subset \mathbb{R}^{d}$ , as well as two parameters  $k\in \mathbb{N}$  and  $z\geq 1$ . After it executes, the algorithm should output a set of  $k$  centers  $C = \{c_1,\dots ,c_k\} \subset \mathbb{R}^d$  as well as an assignment  $\sigma :[n]\to [k]$  mapping each point  $x_{i}$  to a center  $c_{\sigma (i)}$ .
We measure the quality of the solution  $(C, \sigma)$  using the ratio between its  $(k, z)$ -clustering cost  $\mathrm{cost}_z(X, C, \sigma)$  and the optimal cost  $\mathrm{opt}_{k, z}(X)$ , where
$$
\operatorname {c o s t} _ {z} (X, C, \sigma) := \sum_ {i = 1} ^ {n} \| x _ {i} - c _ {\sigma \left(x _ {i}\right)} \| _ {2} ^ {z}. \tag {1}
$$
For any  $\mathsf{D} > 1$ , an algorithm that produces a D-approximation to  $(k,z)$ -clustering should guarantee that  $\mathrm{cost}_z(X,C,\sigma)$  is at most  $\mathsf{D} \cdot \mathrm{opt}_{k,z}(X)$ . For a randomized algorithm, the guarantee should hold with large probability (referred to as the success probability) for any input dataset.
A.1  $k$ -Means++ Seeding
The  $k$ -means++ seeding algorithm is a well-studied algorithm introduced in [6], and it is an important component of our algorithm. Below, we describe it for general  $z \geq 1$ , not necessarily  $z = 2$ .
Definition A.2 ( $k$ -means++ seeding, for arbitrary  $z \geq 1$ ). Given  $n$  data points  $x_1, \ldots, x_n \in \mathbb{R}^d$ , the  $k$ -means++ seeding algorithm produces a set of  $k$  centers,  $x_{\ell_1}, \ldots, x_{\ell_k}$  with the following procedure:
1. Choose  $\ell_1$  uniformly at random from  $[n]$ .
2. For  $t = 2, \ldots, k$ , sample  $\ell_t$  as follows. For every  $i \in [n]$ , let  $p_i$  denote the Euclidean distance from  $x_i$  to its closest point among  $x_{\ell_1}, \ldots, x_{\ell_{t-1}}$ . Sample  $\ell_t$  from  $[n]$  so that the probability  $\operatorname*{Pr}[\ell_t = i]$  is proportional to  $p_i^z$  for every  $i \in [n]$ . That is,
$$
\Pr \left[ \ell_ {t} = i \right] = \frac {p _ {i} ^ {z}}{\sum_ {i ^ {\prime} \in [ n ]} p _ {i} ^ {z}}.
$$
In the context of  $k$ -means (i.e., when  $z = 2$ ), the distribution is known as the  $D^2$  distribution.
3. Output  $x_{\ell_1}, \ldots, x_{\ell_k}$ .
In the above description, Step 2 of  $k$ -means++ needs to maintain, for each dataset point  $x_{i}$ , the Euclidean distance  $p_i$  to its closest center among the centers selected before the current iteration. This step is implemented by making an entire pass over the dataset for each of the  $k - 1$  iterations of Step 2, leading to an  $O(ndk)$  running time.
Theorem A.1 ([6], Theorem 3 in [70]). For  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$ , let  $X = \{x_{1},\dots,x_{n}\}$  be the input to the  $k$ -means++ seeding algorithm. For the output  $x_{\ell_1},\ldots ,x_{\ell_k}$  of the  $k$ -means++ seeding algorithm, define  $C = \{x_{\ell_1},\ldots ,x_{\ell_k}\}$ . Then
$$
\mathbb {E} \left[ \operatorname {c o s t} _ {z} (X, C) \right] = O \left(2 ^ {2 z} \log k\right) \cdot \operatorname {o p t} _ {k, z} (X).
$$
A.2 Coresetts via Sensitivity Sampling
One of our algorithm's applications is constructing coresets for  $(k,z)$ -clustering. We give a formal definition and describe the primary technique for building coresets.