Title: Simple, Scalable and Effective Clustering via One-Dimensional Projections
Venue: NeurIPS 2023
Accepted: yes
Keywords: clustering

k-means

random projection

massive datasets
Abstract: Clustering is a fundamental problem in unsupervised machine learning with many applications in data analysis. Popular clustering algorithms such as Lloyd's algorithm and $k$-means++ can take $\Omega(ndk)$ time when clustering $n$ points in a $d$-dimensional space (represented by an $n\times d$ matrix $X$) into $k$ clusters. On massive datasets with moderate to large $k$, the multiplicative $k$ factor can become very expensive. We introduce a simple randomized clustering algorithm that provably runs in expected time $O(\mathsf{nnz}(X) + n\log n)$ for arbitrary $k$. Here $\mathsf{nnz}(X)$ is the total number of non-zero entries in the input dataset $X$, which is upper bounded by $nd$ and can be significantly smaller for sparse datasets. We prove that our algorithm achieves approximation ratio $\widetilde{O}(k^4)$ on any input dataset for the $k$-means objective, and our experiments show that the quality of the clusters found by our algorithm is usually much better than this worst-case bound. We use our algorithm for $k$-means clustering and for coreset construction; our experiments show that it gives a new tradeoff between running time and cluster quality compared to previous state-of-the-art methods for these tasks. Our theoretical analysis is based on novel results of independent interest. We show that the approximation ratio achieved after a random one-dimensional projection can be lifted to the original points and that $k$-means++ seeding can be implemented in expected time $O(n\log n)$ in one dimension.

Simple, Scalable and Effective Clustering via One-Dimensional Projections

Abstract

Clustering is a fundamental problem in unsupervised machine learning with many applications in data analysis. Popular clustering algorithms such as Lloyd's algorithm and k -means++ can take Ω( ndk ) time when clustering n points in a d -dimensional space (represented by an n × d matrix X ) into k clusters. In applications with moderate to large k , the multiplicative k factor can become very expensive. We introduce a simple randomized clustering algorithm that provably runs in expected time O ( nnz ( X ) + n log n ) for arbitrary k . Here nnz ( X ) is the total number of non-zero entries in the input dataset X , which is upper bounded by nd and can be significantly smaller for sparse datasets. We prove that our algorithm achieves approximation ratio ˜ O ( k 4 ) on any input dataset for the k -means objective. We also believe that our theoretical analysis is of independent interest, as we show that the approximation ratio of a k -means algorithm is approximately preserved under a class of projections and that k -means++ seeding can be implemented in expected O ( n log n ) time in one dimension. Finally, we show experimentally that our clustering algorithm gives a new tradeoff between running time and cluster quality compared to previous state-of-the-art methods for these tasks.

1 Introduction

Clustering is an essential and powerful tool for data analysis with broad applications in computer vision and computational biology, and it is one of the fundamental problems in unsupervised machine learning. In large-scale applications, datasets often contain billions of high-dimensional points. Grouping similar data points into clusters is crucial for understanding and organizing datasets. Because of its practical importance, the problem of designing efficient and effective clustering algorithms has attracted the attention of numerous researchers for many decades.

One of the most popular algorithms for the k -means clustering problem is Lloyd's algorithm [52], which seeks to locate k centers in the space that minimize the sum of squared distances from the points of the dataset to their closest center (we call this the ' k -means cost'). While finding the centers minimizing the objective is NP-hard [3], in practice we can find high-quality sets of centers using Lloyd's iterative algorithm. Lloyd's algorithm maintains a set of k centers. It iteratively updates them by assigning points to one of k clusters (according to their closest center), then redefining the center as the points' center of mass. It needs a good initial set of centers to obtain a high-quality clustering and fast convergence. In practice, the k -means++ algorithm [6], a randomized seeding procedure, is used to choose the initial k centers. k -means++ achieves an O (log k ) -approximation ratio in expectation, upon which each iteration of Lloyd's algorithm improves. 1 Beyond their effectiveness, these algorithms are simple to describe and implement, contributing to their popularity.

The downside of these algorithms is that they do not scale well to massive datasets. A standard implementation of an iteration of Lloyd's algorithm needs to calculate the distance to each center for each point in the dataset, leading to a Θ( ndk ) running time. Similarly, the standard implementation of the k -means++ seeding procedure produces k samples from the so-called D 2 distribution (see Appendix A for details). Maintaining the distribution requires making a pass over the entire dataset after choosing each sample. Generating k centers leads to a Θ( ndk ) running time. Even for moderate values of k , making k passes over the entire dataset can be prohibitively expensive.

One particularly relevant application of large-scale k -means clustering is in approximate nearest neighbor search [66] (for example, in product quantization [44] and building inverted file indices [16]). There, k -means clustering is used to compress entire datasets by mapping vectors to their nearest centers, leading to billion-scale clustering problems with large k (on the order of hundreds or thousands). Other applications on large datasets requiring a large number of centers may be spam filtering [61, 65], near-duplicate detection [42], and compression or reconciliation tasks [62]. New algorithmic ideas are needed for these massive scales, and this motivates the following challenge:

Can we design a simple, practical algorithm for k -means that runs in time roughly O ( nd ) , independent of k , and produces high-quality clusters?

Given its importance in theory and practice, a significant amount of effort has been devoted to algorithms for fast k -means clustering. We summarize a few of the approaches below with the pros and cons of each so that we may highlight our work's position within the literature:

Standard k -means++ : This is our standard benchmark. Plus: Guaranteed to be an O (log k ) -approximation [6]; outputs centers, as well as the assignments of dataset points to centers. Minus: The running time is O ( ndk ) , which is prohibitively expensive in large-scale applications.

Using Approximate Nearest Neighbor Search : One may implement k -means++ faster using techniques from approximate nearest neighbor search (instead of a brute force search each iteration). Plus: The algorithms with provable guarantees, like [26], obtain an O ε (log k ) -approximation. Minus: The running time is ˜ O ( nd +( n log(∆)) 1+ ε ) , depending on a dataset dependent parameter ∆ , the ratio between the maximum and minimum distances between input points. The techniques are algorithmically sophisticated and incur extra poly-logarithmic factors (hidden in ˜ O ( · ) ), making the implementation significantly more complicated.

Approximating the D 2 -Distribution : Algorithms that speed up the seeding procedure for Lloyd's algorithm or generate fast coresets (we expand on this below) have been proposed in [8, 7, 10]. Plus: These algorithms are fast, making only one pass over the dataset in time O ( nd ) . (For [8, 7], there is an additional additive O ( k 2 d ) term in the running time). Minus: The approximation guarantees are qualitatively weaker than the approximation of k -means clustering. They incur an additional additive approximation error that grows with the entire dataset's variance (which can lead to an arbitrarily large error; see Section 3). These algorithms output a set of k centers but not the cluster assignments. Naively producing the assignments would take time O ( ndk ) . 2

Coresets. At a high level, coresets are a dataset-reduction mechanism. A large dataset X of n points in R d is distilled into a significantly smaller (weighted) dataset Y of m points in R d , called a 'coreset' which serves as a good proxy for X , i.e., the clustering cost of any k centers on Y is approximately the cost of the same centers on X . We point the reader to [9, 35] for a recent survey on coresets. Importantly, coreset constructions (with provable multiplicative-approximation guarantees) require an initial approximate clustering of the original dataset X . Therefore, any fast algorithm for k -means clustering automatically speeds up any algorithmic pipeline that uses coresets for clustering -looking forward, we will show how our algorithm can significantly speed up coreset constructions without sacrificing approximation.

Beyond those mentioned above, many works seek to speed up k -means++ or Lloyd iterations by maintaining some nearest neighbor search data structures [58, 54, 46, 45, 33, 60, 40, 59, 71, 30, 29, 13, 55, 28, 18], or by running some first-order methods [64]. These techniques do not give provable guarantees on the quality of the k -means clustering or on the running time of their algorithms.

Theoretical Results. We give a simple randomized clustering algorithm with provable guarantees on its running time and approximation ratio without making any assumptions about the data. It has the benefit of being fast (like the algorithms in Category C above) while achieving a multiplicative error guarantee without additional additive error (like the algorithms in Category B above).

The algorithm runs in time O ( nd + n log n ) irrespective of k . It passes over the dataset once to perform data reduction, which gives the nd factor plus an additive O ( n log n ) term to solve k -means on the reduced data, producing k centers and cluster assignments. On sparse input datasets, the nd term becomes nnz ( X ) , where nnz ( X ) is the number of non-zero entries in the dataset. Thus, our algorithm runs in O ( nnz ( X ) + n log n ) time on sparse matrices.

The algorithm is as simple as the k -means++ algorithm while significantly more efficient. The approximation ratio we prove is poly( k ) , which is worse than the O (log k ) -approximation achieved by k -means++ but it is purely multiplicative (see the remark below on improving this to O (log k ) ). It does not incur the additional additive errors from the fast algorithms in [8, 7, 10].

Our algorithm projects the input points to a random one-dimensional space and runs an efficient k -means++ seeding after the projection. For the approximation guarantee, we analyze how the approximation ratio achieved after the projection can be transferred to the original points (Lemma 2.5). We bound the running time of our algorithm by efficiently implementing the k -means++ seeding in one dimension and analyzing the running time via a potential function argument (Lemma 2.4). Our algorithm applies beyond k -means to other clustering objectives that sum up the z -th power of the distances for general z ≥ 1 , and our guarantees on its running time and approximation ratio extend smoothly to these settings.

Improving the Approximation from poly( k ) to O (log k ) . The approximation ratio of poly( k ) may seem significantly worse than the O (log k ) approximations achievable with k -means++. However, we can improve this to O (log k ) with an additional, additive O (poly( kd ) · log n ) term in the running time. Using previous results discussed in Appendix A.2 (specifically Theorem A.2), a multiplicative poly( k ) -approximation suffices to construct a coreset of size poly( kd ) and run k -means++ on the coreset. Constructing the coreset is simple and takes time poly( kd ) · log n (by sampling from an appropriate distribution); running k -means++ on the coreset takes poly( kd ) time (with no dependence on n ). Combining our algorithm with coresets, we get a O (log k ) -approximation in O ( nnz ( X )) + O ( n log n ) + poly( kd ) · log n time. Notably, these guarantees cannot be achieved with the additive approximations of [8, 7, 10].

Experimental Results. We implemented our algorithm, as well as the lightweight coreset of [10] and k -means++ with sensitivity sampling [15]. We ran two types of experiments, highlighting various aspects of our algorithm. We provide our code in the supplementary material. The two types of experiments are:

Coreset Construction Comparison : First, we evaluate the performance of our clustering algorithm when we use it to construct coresets. We compare the performance of our algorithm to k -means++ with sensitivity sampling [9] and lightweight coresets [10]. In real-world, high-dimensional data, the cost of the resulting clusters from the three algorithms is roughly the same. However, ours and the lightweight coresets can be significantly faster (ours is up to 190x faster than k -means++,

see Figure 2 and Table 1). The lightweight coresets can be faster than our algorithm (between 3-5x); however, our algorithm is 'robust' (achieving multiplicative approximation guarantees). 3 Additionally, we show that the clustering from lightweight coresets can have an arbitrarily high cost for a synthetic dataset. On the other hand, our algorithm achieves provable (multiplicative) approximation guarantees irrespective of the dataset (this is demonstrated in the right-most column of Figure 2).

Direct k-means++ comparison : Second, we compare the speed and cost of our algorithm to k-means++[6] as a stand-alone clustering algorithm (we also compare two other natural variants of our algorithm). Our algorithm can be up to 800x faster than k -means++ for k = 5000 and our slowest variant up to 100x faster (Table 1). The cost of the cluster assignments can be significantly worse than that of k -means++ (see Figure 3). Such a result is expected since our theoretical results show a poly( k ) -approximation. The other (similarly) fast algorithms (based on approximating the D 2 -distribution) which run in time O ( nd ) [8, 7] do not produce the cluster assignments (they only output k centers). These algorithms would take O ( ndk ) time to find the cluster assignments - this is precisely the computational cost our algorithm avoids.

We do not compare our algorithm with [26] nor implement approximate nearest neighbor search to speed up k -means++ for the following reasons. The algorithm in [26] is significantly more complicated, and there is no publicly available implementation. In addition, both [26] and approximate nearest neighbor search incur additional poly-logarithmic (or even n o (1) -factors for nearest neighbor search over ℓ 2 [4]) which add significant layers of complexity to the implementation and make a thorough evaluation of the algorithm significantly more complicated. Instead, our current implementation demonstrates that a simple, one-dimensional projection and k -means++ on the line enables dramatic speedups to coreset constructions without sacrificing approximation quality.

Related Work. Efficient algorithms for clustering problems with provable approximation guarantees have been studied extensively, with a few approaches in the literature. There are polynomial-time (constant) approximation algorithms (an exponential dependence on k is not allowed) (see [50, 17, 2, 39] for some of the most recent and strongest results), nearly linear time (1 ± ε ) -approximations with running time exponential in k which proceed via coresets (see [41, 19, 36, 37, 15, 9, 27, 25] and references therein, as well as the surveys [1, 35]), and nearly-linear time (1 ± ε ) -approximations in fixed / low-dimensional spaces [5, 48, 68, 38, 24, 22, 23]. Our O ( n log n ) -expected-time implementation of k -means++ seeding achieves an O (log k ) expected approximation ratio for k -median and k -means in one dimension. We are unaware of previous work on clustering algorithms running in time O ( n log n ) .

Another line of research has been on dimensionality reduction techniques for k -means clustering. Dimensionality reduction can be achieved via PCA based methods [31, 37, 21, 67], or random projection [21, 11, 53]. For random projection methods, it has been shown that the k -means objective is preserved up to small multiplicative factors when projecting onto O ε (log( k )) dimensional space. Additional work has shown that dimensionality reduction can be performed in O ( nnz ( A )) time [51]. To the best of our knowledge, we are the first to show that clustering objectives such as k -median and k -means are preserved up to a poly( k ) factor by one-dimensional projections.

Some works show that the O (log k ) expected approximation ratio for k -means++ can be improved by adding local search steps after the seeding procedure [49, 20]. In particular, Choo et al. [20] showed that adding εk local search steps achieves an O (1 /ε 3 ) approximation ratio with high probability.

Several other algorithmic approaches exist for fast clustering of points in metric spaces. These include density-based methods like DBSCAN [34] and DBSCAN++ [43] and the line of heuristics based on the Partitioning Around Medoids (PAM) approach, such as FastPAM [63], Clarans [56], and BanditPAM [69]. While these algorithms can produce high-quality clustering, their running time is at least linear in the number of clusters (DBSCAN++ and BanditPAM) or superlinear in the number of points (DBSCAN, FastPAM, Clarans).

2 Overview of Our Algorithm and Proof Techniques

Our algorithm, which we call PRONE (PRojected ONE-dimensional clustering), takes a random projection onto a one-dimensional space, sorts the projected (scalar) numbers, and runs the k -means++ seeding strategy on the projected numbers. By virtue of its simplicity, the algorithm is scalable and effective at clustering massive datasets. More formally, PRONE receives as input a dataset of n points in R d , a parameter k ∈ N (the number of desired clusters), and proceeds as follows:

Sample a random vector v ∈ R d from the standard Gaussian distribution and project the data points to one dimension along the direction of v . That is, we compute x ′ i = ⟨ x i , v ⟩ ∈ R in time O ( nnz ( X )) by making a single pass over the data, effectively reducing our dataset to the collection of one-dimensional points x ′ 1 , . . . , x ′ n ∈ R .

Run k -means++ seeding on x ′ 1 , . . . , x ′ n to obtain k indices j 1 , . . . , j k ∈ [ n ] indicating the chosen centers x ′ j 1 , . . . , x ′ j k and an assignment σ : [ n ] → [ k ] assigning point x ′ i to center x ′ j σ ( i ) . Even though k -means++ seeding generally takes O ( nk ) time in one dimension, we give an efficient implementation, leveraging the fact that points are one-dimensional, which runs in O ( n log n ) expected time, independent of k . A detailed algorithm description is in the appendix.

The one-dimensional k -means++ algorithm produces a collection of k centers x j 1 , . . . , x j k , as well as the assignment σ mapping each point x i to the center x j σ ( i ) . For each ℓ ∈ [ k ] , we update the cluster center for cluster ℓ to be the center of mass of all points assigned to x j ℓ .

While the algorithm is straightforward, the main technical difficulty lies in the analysis. In particular, our analysis (1) bounds the approximation loss incurred from the one-dimensional projection in Step 1 and (2) shows that we can implement Step 2 in O ( n log n ) expected time, as opposed to O ( nk ) time. We summarize the theoretical contributions in the following theorems.

Theorem 2.1. The algorithm PRONE has expected running time O ( nnz ( X ) + n log n ) on any dataset X = { x 1 , . . . , x n } ⊂ R d . Moreover, for any δ ∈ (0 , 1 / 2) and any dataset X , with probability at least 1 -δ , the algorithm runs in time O ( nnz ( X ) + n log( n/δ )) .

Theorem 2.2. The algorithm PRONE achieves an ˜ O ( k 4 ) approximation ratio for the k -means objective with probability at least 0 . 9 .

To our knowledge, PRONE is the first algorithm for k -means running in time O ( nd + n log n ) for arbitrary k . As mentioned in the paragraph on improving the competitive ratio, we obtain the following corollary of the previous two theorems using a two-stage approach with a coreset:

Corollary 2.3. By using PRONE as the α -approximation algorithm in Theorem A.2 and running k -means++ on the resulting coreset, we obtain an algorithm with an approximation ratio of O (log k ) that runs in time O ( nnz ( X ) + n log n +poly( kd ) log n ) , with constant success probability.

Due to space constraints, all proofs of our theoretical results are deferred to the appendix, where we also generalize them beyond k -means to clustering objectives that sum up the z -th power of Euclidean distances for general z ≥ 1 . The following subsections give a high-level overview of the main techniques we develop to prove our main theorems above.

2.1 Efficient Seeding in One Dimension

The k -means++ seeding procedure has k iterations, where a new center is sampled in each iteration. Since a new center may need to update Ω( n ) distances to maintain the D 2 distribution, which samples each point with probability proportional to its distance to its closest center, a naive analysis leads to a running time of O ( nk ) . A key ingredient in the proof of Theorem 2.1 is showing that, for one-dimensional datasets, k -means++ only needs to make O ( n log n ) updates, irrespective of k .

Lemma 2.4. The k -means++ seeding procedure can be implemented in expected time O ( n log n ) in one dimension. Moreover, for any δ ∈ (0 , 1 / 2) , with probability at least 1 -δ , the implementation runs in time O ( n log( n/δ )) .

The intuition of the proof is as follows: Since points are one-dimensional, we always maintain them in sorted order. In addition, each data point x i will maintain its center assignment and distance p i to the closest center. By building a binary tree over the sorted points (where internal nodes maintain sums of p 2 i 's), it is easy to sample a new center from the D 2 distribution in O (log n ) time. The difficulty is

that adding a new center may result in changes to p i 's of multiple points x i , so the challenge is to bound the number of times these values are updated (see Figure 1 below).

To bound the total running time, we leverage the one-dimensional structure. Observe that, for a new center, the updated points lie in a contiguous interval around the newly chosen center. Once a center is chosen, the algorithm scans the points (to the left and the right) until we reach a point that does not need to be updated. This point identifies that points to the other side of it need not be updated, so we can get away without necessarily checking all n points (see Figure 1). Somewhat surprisingly, when sampling centers from the D 2 -distribution, the expected number of times that each point will be updated is only O (log n ) , which implies a bound of O ( n log n ) on the total number of updates in expectation. The analysis of the fact that each point is updated O (log n ) times is non-trivial and uses a carefully designed potential function (Lemma C.5).

2.2 Approximation Guarantees from One-Dimensional Projections

Our proof of Theorem 2.2 builds on a line of work studying randomized dimension reduction for clustering problems [14, 21, 11, 53]. Prior work studied randomized dimension reduction for accurate (1 ± ϵ ) -approximations. Our perspective is slightly different; we restrict ourselves to one-dimensional projections and give an upper bound on the distortion.

For any dataset x 1 , . . . , x n ∈ R d , a projection to a random lower-dimensional space affects the pairwise distance between the projected points in a predictable manner - the Johnson-Lindenstrauss lemma which projects to O (log n ) dimensions being a prime example of this fact. When projecting to just one dimension, however, pairwise distances will be significantly affected (by up to poly( n ) -factors). Thus, a naive analysis will give a poly( n ) -approximation for k -means. To improve a c -approximation to a O (log k ) -approximation, one needs a coreset of size roughly poly( c/ log k ) . This bound becomes vacuous when c is polynomial in n since there are at most n dataset points.

However, although many pairwise distances are significantly distorted, we show that the k -means cost is only affected by a poly( k ) -factor. At a high level, this occurs because the k -means cost optimizes a sum of pairwise distances (according to a chosen clustering). The individual summands, given by pairwise distances, will change significantly, but the overall sum does not. Our proof follows the approach of [21], which showed that (roughly speaking) pairwise distortion of the k optimal centers suffices to argue about the k -means cost. The k optimal centers will incur maximal pairwise distortion poly( k ) when projected to one dimension (because there are only O ( k 2 ) pairwise distances among the k centers). This allows us to lift an r -approximate solution after the projection to an O ( k 4 r ) -approximate solution for the original points.

Lemma 2.5 (Informal) . For any set X of points in R d , the following occurs with probability at least 0 . 9 over the choice of a standard Gaussian vector v ∈ R d . Letting X ′ ⊂ R be the one-dimensional projection of X onto v , any r -approximate k -means clustering of X ′ gives an O ( k 4 r ) -approximate clustering of X with the same clustering partition.

3 Experimental Results

In this section, we outline the experimental evaluation of our algorithm. The experiments evaluate the algorithms in two different ways. For each, we measure the running time and the k -means cost of the resulting solution (the sum of squares of point-to-center-assigned distances). (1) First, we evaluate our algorithm as part of a pipeline incorporating a coreset construction - the expected use case for our algorithm. (2) Second, we evaluate our algorithm by itself for approximate k-means clustering and compare it to k-means++ [6]. As per Theorems 2.1 and 2.2, we expect our algorithm to be much faster but output an assignment of higher cost. Our goal is to quantify these differences empirically.

All experiments were run on Linux using a notebook with a 3.9 GHz 12th generation Intel Core i7 six-core processor and 32 GiB of RAM. All algorithms were implemented in C++, using the blaze library for matrix and vector operations performed on the dataset unless specified differently below. We provide our code in the supplementary material for this submission.

Datasets. For our experiments, we use the following four datasets:

KDD [47]: Training data for the 2004 KDD challenge on protein homology. The dataset consists of 145751 observations with 77 real-valued features.

Song [12]: Timbre information for 515345 songs with 90 features each, used for year prediction.

Census [32]: 1990 US census data with 2458285 observations, each with 68 categorical features.

Gaussian : A synthetic dataset consisting of 240005 points of dimension 4 . The points are generated by placing a standard normal distribution at a large positive distance from the origin on each axis and sampling 30000 points. The points are then mirrored so the center of mass remains at the origin. Finally, 5 points are placed on the origin. This is an adversarial example for lightweight coresets [10], which are unlikely to sample points close to the mean of the dataset.

3.1 Coreset Construction Comparison

Experimental Setup. Coreset constructions (with multiplicative approximation guarantees) always proceed by first finding an approximate clustering, which constitutes the bulk of the work. The approximate clustering defines a 'sensitivity sampling distribution' (we expand on this in the appendix, see also [9]), and a coreset is constructed by repeatedly sampling from the sensitivity sampling distribution. In our first experiment, we evaluate the choice of initial approximation algorithm used to define the sensitivity sampling distribution. We compare the use of k -means++ and PRONE . In addition, we also compare the lightweight coresets of [10], which uses the distance to the center of mass as an approximation of the sensitivity sampling distribution. For the remainder of this section, we refer to sensitivity sampling using k-means++ as Sensitivity and lightweight coresets as Lightweight . All three algorithms produce a coreset, and the experiment will measure the running time of the three algorithms (Table 1) and the quality of the resulting coresets (Figure 2). We implemented all algorithms in C++.

Once a coreset is constructed for each of the algorithms, we evaluate the quality of the coreset by computing the cost of the centers found when clustering the coreset (see Definition A.1). We run a state-of-the-art implementation of Lloyd's k -means algorithm from the scikit-learn library [57] with the default configuration (repeating 15 times and reporting the mean cost to reduce the variance). The resulting quality of the coresets is compared to a (computationally expensive) baseline , which runs k-means++ from the scikit-learn library, followed by Lloyd's algorithm with the default configuration on the entire dataset (repeated 5 times to reduce variance).

We evaluate various choices of k ( { 10 , 100 , 1000 } ) as well as coresets at various relative sizes, { 0 . 001 , 0 . 0025 , 0 . 005 , 0 . 01 , 0 . 025 , 0 . 05 , 0 . 1 } times the size of the dataset. We use as performance metrics (1) a relative cost, which measures the average cost of the k-means solutions returned by Lloyd's algorithm on each coreset divided by the baseline, and (2) the running time of the coreset construction algorithm.

Results on Coreset Constructions. Relative cost. Figure 2 shows the coreset size ( x -axis) versus the relative cost ( y -axis). Each 'row' of Figure 2 corresponds to a different value for k ∈ { 10 , 100 , 1000 } , and each 'column' corresponds to a different dataset. Recall that the first three datasets (i.e., the first three columns) are real-world datasets, and the fourth column is the synthetic Gaussian dataset. We note our observations below:

As expected, on all real-world data sets and all settings of k , the relative cost decreases as the coreset size increases.

In real-world datasets, the specific relative cost of each coreset construction ( Senstivity , Lightweight , and ours) depends on the dataset 4 , but roughly speaking, all three share a similar trend. Ours and Sensitivity are very close and never more than twice the baseline (usually much better).

10

3

10

2

10

1

1.0

1.2

1.4

1.6

Relative Cost

Dataset = KDD

10

3

10

2

10

1

1.000

1.025

1.050

1.075

1.100

1.125

Dataset = Song

10

3

10

2

10

1

1.00

1.01

1.02

1.03

1.04

1.05

1.06

Dataset = Census

10

3

10

2

10

1

1.0

1.5

2.0

2.5

3.0

k = 10

10

3

10

2

10

1

1.0

1.5

2.0

2.5

3.0

Relative Cost

10

3

10

2

10

1

1.0

1.1

1.2

1.3

10

3

10

2

10

1

1.00

1.02

1.04

1.06

1.08

1.10

1.12

10

3

10

2

10

1

1

2

3

4

5

6

7

k = 100

10

2

10

1

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Relative Cost

10

2

10

1

1.0

1.1

1.2

1.3

1.4

10

3

10

2

10

1

1.0

1.1

1.2

1.3

1.4

1.5

1.6

10

2

10

1

5

10

15

k = 1000

Algorithm

k-means++ + Sensitivity

Lightweight

PRONE + Sensitivity

Baseline

The big difference, distinguishing ours and Sensitivity from Lightweight , is the fourth column, the synthetic Gaussian dataset. For all settings of k , as the coreset size increases, Lightweight exhibits a minimal cost decrease and is a factor of 2.7-17x times worse than ours and Sensitivity (as well as the baseline). This is expected, as we constructed the synthetic Gaussian dataset to have arbitrarily high cost with Lightweight . Due to its multiplicative approximation guarantee, our algorithm does not suffer this degradation. In that sense, our algorithm is more 'robust,' and achieves worst-case multiplicative approximation guarantees for all datasets.

Running time. In (the first table in) Table 1, we show the running time of the coreset construction algorithms as k increases. Notice that as k increases, the relative speedup of our algorithm and Lightweight increases in comparison to Sensitivity . This is because our algorithm and Lightweight have running time which does not grow with k . In contrast, the running time of Sensitivity grows linearly in k . In summary, our coreset construction is between 33-192x faster than Sensitivity for large k . In addition, our algorithm runs about 3-5x slower than Lightweight , depending on the dataset. Our analysis also shows this; both algorithms make an initial pass over the dataset, using O ( nd ) time, but ours uses an additional O ( n log n ) time to process.

3.2 Direct k-Means++ Comparison

Experimental Setup. This experiment compares our algorithm and k -means++ as a stand-alone clustering algorithm, as opposed to as part of a coreset pipeline. We implemented three variants of our algorithm. Each differs in how we sample the random one-dimensional projection. The first is a one-dimensional projection onto a standard Gaussian vector (zero mean and identity covariance). This approach risks collapsing an 'important' feature, i.e. a feature with high variance. To mitigate this, we implemented two data-dependent variants that use the variance, resp. covariance of the data. Specifically, in the 'variance' variant, we use a diagonal covariance matrix, where each entry in the diagonal is set to the empirical variance of the dataset along the corresponding feature. In the 'covariance' variant, we use the empirical covariance matrix of the dataset. These variants aim to project along vectors that capture more of the variance of the data than when sampling a vector uniformly at random. Intuitively, the vectors sampled by the biased variants are more correlated with the first principal component of the dataset. For each of our algorithms, we evaluate the k -means cost of the output set C of centers when assigning points to the closest center ( cost 2 ( X,C ) in Definition A.1) and when using our algorithm's assignment ( cost 2 ( X,C,σ ) defined in Equation (1)).

We evaluated the algorithms for every k in { 10 , 25 , 50 , 100 , 250 , 500 , 1000 , 2500 , 5000 } and z = 2 , for solving k -means with the ℓ 2 -metric. When evaluating the assignment cost, we ran each of our

10 1

10 2

10 3

k

10 12

10 13

cost

Dataset = KDD

10 1

10 2

10 3

k

2×10 12

3×10 12

4×10 12

Dataset = Song

10 1

10 2

10 3

k

10 8

10 9

Dataset = Census

10 1

10 2

10 3

k

10 5

10 6

Algorithm

PRONE

PRONE (variance)

PRONE (covariance)

k-means++

10 1

10 2

10 3

k

10 12

10 13

10 14

cost

Dataset = KDD

10 1

10 2

10 3

k

2×10 12

3×10 12

4×10 12

6×10 12

Dataset = Song

10 1

10 2

10 3

k

10 8

10 9

Dataset = Census

10 1

10 2

10 3

k

10 6

10 8

10 10

Algorithm

PRONE

PRONE (variance)

PRONE (covariance)

k-means++

algorithms 100 times for each k and five times when computing the nearest neighbor assignment, and we report the average cost of the solutions and the average running time. Due to lower variance and much higher runtime, k-means++ was run five times.

Results on Direct k -Means++ Comparison. Cost. Figure 3 (on top) shows the cost of the centers found by our algorithm compared to those found by the k-means++ algorithm after computing the optimal assignment of points to the centers computed by the algorithm (computing this takes time O ( ndk ) ). That is, we compare the values of cost 2 ( X,C ) in Definition A.1. In summary, the k-means cost of all three variants of our algorithm are roughly the same and closely match that of k-means++. On the Gaussian data set, one run of the biased algorithm failed to pick a center from the cluster at the origin, leading to a high 'outlier' cost and a corresponding spike in the plot.

We also compared the k-means cost for the assignment computed by our algorithm (so that our algorithm only takes time O ( nd + n log n ) and not O ( ndk ) ) with the cost of k-means++ (bottom row of Figure 3). That is, we compare the values of cost 2 ( X,C,σ ) defined in Equation (1). The clustering cost of our algorithms is higher than that of k-means++. This is the predicted outcome from our theoretical results; recall Theorem 2.2 gives a poly( k ) -approximation, as opposed to O (log k ) from k -means++. On the real-world data sets, it is between one order of magnitude (for k = 10 ) and two orders of magnitude (for k = 5000 ) worse than k-means++ for our unbiased variant and between a factor 2 (for k = 10 ) and one order of magnitude (for k = 5000 ) worse than k-means++ for our biased and covariance variants.

Running time. Table 1 shows the relative running time of our algorithm compared to k-means++, assuming that no nearest-center assignment is computed. Our algorithms are designed to have a running time independent of k , so we can see, from the second table in Figure 1, all of our variants offer significant speedups.

The running time of our algorithm stays almost constant as k increases while the running time of k-means++ scales linearly with k . Specifically for k = 25 , even our slowest variants have about the same running time as k-means++, while for k = 5000 , it is at least 82x faster, and our fastest version is up to 837x faster over k-means++.

The two variants can affect the quality of the chosen centers by up to an order of magnitude, but they are also significantly slower. The 'variance' and 'covariance' variants are slower (between 2-4x slower and up to 10x slower, respectively) than the standard variant, and they also become slower as the dimensionality d increases. We believe these methods could be further sped up, as the blaze library's variance computation routine appears inefficient for our use case.

4 Conclusion and Limitations

To summarize, we present a simple algorithm that provides a new tradeoff between running time and approximation ratio. Our algorithm runs in expected time O ( nnz ( X ) + n log n ) to produce a poly( k ) -approximation; with additional poly( kd ) · log n time, we improve the approximation to O (log k ) . This latter bound matches that of k -means++ but offers a significant speedup.

Within a pipeline for constructing coresets, our experiments show that the quality of the coreset produced (when using our algorithm as the initial approximation) outperforms the sensitivity sampling algorithm. It is slower than the lightweight coreset algorithm, but it is more 'robust' as it is independent of the diameter of the data set. It does not suffer from the drawback of having an additive error linear in the diameter of the dataset, which can arbitrarily increase the cost of the lightweight coreset algorithm. When computing an optimal assignment for the centers returned by our algorithm, its cost roughly matches the cost for k-means++. When directly using the assignment produced by one variant of our algorithm, its cost is between a factor 2 and 10 worse while being up to 300 times faster.

Our experiments and running time analysis show that our algorithm is very efficient. However, the clustering quality achieved by our algorithm is sometimes not as good as other, slower algorithms. We show that this limitation is insignificant when we use our algorithm to construct coresets. It remains an interesting open problem to understand the best clustering quality (e.g., in terms of approximation ratio) an algorithm can achieve while being as efficient as ours, i.e., running in time O ( nd + n log n ) . Another interesting problem is whether other means of projecting the dataset into a O (1) dimensional space exist, which lead to algorithms with improved approximation guarantees and running time faster than O ( ndk ) .

Acknowledgments and Disclosure of Funding

Moses Charikar was supported by a Simons Investigator award.

Lunjia Hu was supported by Moses Charikar's and Omer Reingold's Simons Investigators awards, Omer Reingold's NSF Award IIS-1908774, and the Simons Foundation Collaboration on the Theory of Algorithmic Fairness. Part of this work was done while Erik Waingarten was a postdoc at Stanford University, supported by an NSF postdoctoral fellowship and by Moses Charikar's Simons Investigator Award.

This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 101019564 'The Design of Modern Fully Dynamic Data Structures (MoDynStruct)' and the Austrian Science Fund (FWF) project Z 422-N, project 'Static and Dynamic Hierarchical Graph Decompositions', I 5982-N, and project 'Fast Algorithms for a Reactive Network Layer (ReactNet)', P 33775-N, with additional funding from the netidee SCIENCE Stiftung, 2020-2024.

References

Pankaj K Agarwal, Sariel Har-Peled, Kasturi R Varadarajan, et al. Geometric approximation via coresets. Combinatorial and computational geometry , 52(1):1-30, 2005.

Sara Ahmadian, Ashkan Norouzi-Fard, Ola Svensson, and Justin Ward. Better guarantees for k-means and Euclidean k-median by primal-dual algorithms. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 61-72, 2017.

Daniel Aloise, Amit Deshpande, Pierre Hansen, and Preyas Popat. NP-hardness of Euclidean sum-of-squares clustering. Machine Learning , 75(2):245-248, 2009.

Alexandr Andoni, Piotr Indyk, and Ilya Razenshteyn. Approximate nearest neighbor search in high dimensions. In Proceedings of the International Congress of Mathematicians: Rio de Janeiro 2018 , pages 3287-3318. World Scientific, 2018.

Sanjeev Arora, Prabhakar Raghavan, and Satish Rao. Approximation schemes for Euclidean k-medians and related problems. In Proceedings of the Thirtieth Annual ACM Symposium on Theory of Computing , STOC '98, page 106-113, New York, NY, USA, 1998. Association for Computing Machinery.

David Arthur and Sergei Vassilvitskii. k-means++ : the advantages of careful seeding. In Proceedings of the Eighteenth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 1027-1035. ACM, New York, 2007.

Olivier Bachem, Mario Lucic, Hamed Hassani, and Andreas Krause. Fast and provably good seedings for k-means. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

Olivier Bachem, Mario Lucic, S. Hamed Hassani, and Andreas Krause. Approximate k-means++ in sublinear time. Proceedings of the AAAI Conference on Artificial Intelligence , 30(1), Feb. 2016.

Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv:1703.06476 , 2017.

Olivier Bachem, Mario Lucic, and Andreas Krause. Scalable k -means clustering via lightweight coresets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , KDD '18, page 1119-1127, New York, NY, USA, 2018. Association for Computing Machinery.

Luca Becchetti, Marc Bury, Vincent Cohen-Addad, Fabrizio Grandoni, and Chris Schwiegelshohn. Oblivious dimension reduction for k-means: Beyond subspaces and the johnson-lindenstrauss lemma. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , STOC 2019, page 1039-1050, New York, NY, USA, 2019. Association for Computing Machinery.

Thierry Bertin-Mahieux, Daniel P.W. Ellis, Brian Whitman, and Paul Lamere. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011) , 2011.

Thomas Bottesch, Thomas Bühler, and Markus Kächele. Speeding up k-means by approximating Euclidean distances via block vectors. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 2578-2586, New York, New York, USA, 20-22 Jun 2016. PMLR.

Christos Boutsidis, Anastasios Zouzias, Michael W Mahoney, and Petros Drineas. Randomized dimensionality reduction for k -means clustering. IEEE Transactions on Information Theory , 61(2):1045-1062, 2014.

Vladimir Braverman, Dan Feldman, and Harry Lang. New frameworks for offline and streaming coreset constructions. CoRR , abs/1612.00889, 2016.

James Briggs. Faiss: The missing manual. https://www.pinecone.io/learn/faiss/ .

Jarosław Byrka, Thomas Pensyl, Bartosz Rybicki, Aravind Srinivasan, and Khoa Trinh. An improved approximation for k-median, and positive correlation in budgeted optimization. In Proceedings of the Twenty-Sixth Annual ACM-SIAM Symposium on Discrete Algorithms , SODA '15, page 737-756, USA, 2015. Society for Industrial and Applied Mathematics.

Marco Capó, Aritz Pérez, and Jose A Lozano. An efficient k-means clustering algorithm for massive data. arXiv preprint arXiv:1801.02949 , 2018.

Ke Chen. On coresets for k-median and k-means clustering in metric and Euclidean spaces and their applications. SIAM Journal on Computing , 39(3):923-947, 2009.

Davin Choo, Christoph Grunau, Julian Portmann, and Vaclav Rozhon. k-means++: few more steps yield constant approximation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1909-1917. PMLR, 13-18 Jul 2020.

Michael B. Cohen, Sam Elder, Cameron Musco, Christopher Musco, and Madalina Persu. Dimensionality reduction for k-means clustering and low rank approximation. In Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing , STOC '15, page 163-172, New York, NY, USA, 2015. Association for Computing Machinery.

Vincent Cohen-Addad. A fast approximation scheme for low-dimensional k-means. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , SODA '18, page 430-440, USA, 2018. Society for Industrial and Applied Mathematics.

Vincent Cohen-Addad, Andreas Emil Feldmann, and David Saulpic. Near-linear time approximation schemes for clustering in doubling metrics. J. ACM , 68(6), oct 2021.

Vincent Cohen-Addad, Philip N. Klein, and Claire Mathieu. Local search yields approximation schemes for k-means and k-median in Euclidean and minor-free metrics. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS) , pages 353-364, 2016.

Vincent Cohen-Addad, Kasper Green Larsen, David Saulpic, and Chris Schwiegelshohn. Towards optimal lower bounds for k-median and k-means coresets. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1038-1051, 2022.

Vincent Cohen-Addad, Silvio Lattanzi, Ashkan Norouzi-Fard, Christian Sohler, and Ola Svensson. Fast and accurate k-means++ via rejection sampling. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 16235-16245. Curran Associates, Inc., 2020.

Vincent Cohen-Addad, David Saulpic, and Chris Schwiegelshohn. A new coreset framework for clustering. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 169-182, 2021.

Ryan R Curtin. A dual-tree algorithm for fast k-means clustering with large k. In Proceedings of the 2017 SIAM International Conference on Data Mining , pages 300-308. SIAM, 2017.

Yufei Ding, Yue Zhao, Xipeng Shen, Madanlal Musuvathi, and Todd Mytkowicz. Yinyang k-means: A drop-in replacement of the classic k-means with consistent speedup. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 579-587, Lille, France, 07-09 Jul 2015. PMLR.

Jonathan Drake. Faster k-means clustering. PhD thesis, Baylor University, 2013.

Petros Drineas, Alan Frieze, Ravi Kannan, Santosh Vempala, and Vishwanathan Vinay. Clustering large graphs via the singular value decomposition. Machine learning , 56:9-33, 2004.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.

Charles Elkan. Using the triangle inequality to accelerate k-means. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning , ICML'03, page 147-153. AAAI Press, 2003.

Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining , KDD'96, page 226-231. AAAI Press, 1996.

Dan Feldman. Introduction to core-sets: an updated survey. arXiv preprint arXiv:2011.09384 , 2020.

Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Proceedings of the forty-third annual ACM symposium on Theory of computing , pages 569-578, 2011.

Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constantsize coresets for k-means, pca, and projective clustering. SIAM Journal on Computing , 49(3):601-657, 2020.

Zachary Friggstad, Mohsen Rezapour, and Mohammad R. Salavatipour. Local search yields a ptas for k-means in doubling metrics. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS) , pages 365-374, 2016.

Fabrizio Grandoni, Rafail Ostrovsky, Yuval Rabani, Leonard J. Schulman, and Rakesh Venkat. A refined approximation for Euclidean k-means. Inf. Process. Lett. , 176(C), jun 2022.

Greg Hamerly. Making k-means even faster. In Proceedings of the 2010 SIAM international conference on data mining , pages 130-140. SIAM, 2010.

Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the Thirty-Sixth Annual ACM Symposium on Theory of Computing , STOC '04, page 291-300, New York, NY, USA, 2004. Association for Computing Machinery.

Oktie Hassanzadeh, Fei Chiang, Hyun Chul Lee, and Renée J. Miller. Framework for evaluating clustering algorithms in duplicate detection. Proc. VLDB Endow. , 2(1):1282-1293, aug 2009.

Jennifer Jang and Heinrich Jiang. Dbscan++: Towards fast and scalable density clustering. In International conference on machine learning , pages 3019-3029. PMLR, 2019.

Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence , 33(1):117-128, 2010.

T. Kanungo, D.M. Mount, N.S. Netanyahu, C.D. Piatko, R. Silverman, and A.Y. Wu. An efficient k-means clustering algorithm: analysis and implementation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 24(7):881-892, 2002.

Tapas Kanungo, David M. Mount, Nathan S. Netanyahu, Christine Piatko, Ruth Silverman, and Angela Y. Wu. The analysis of a simple k-means clustering algorithm. In Proceedings of the Sixteenth Annual Symposium on Computational Geometry , SCG '00, page 100-109, New York, NY, USA, 2000. Association for Computing Machinery.

2004 KDD Cup. Protein homology dataset. available at https://osmot.cs.cornell.edu/kddcup/datasets.html , 2004.

Stavros G. Kolliopoulos and Satish Rao. A nearly linear-time approximation scheme for the Euclidean k-median problem. In Jaroslav Nešetˇ ril, editor, Algorithms - ESA' 99 , pages 378-389, Berlin, Heidelberg, 1999. Springer Berlin Heidelberg.

Silvio Lattanzi and Christian Sohler. A better k-means++ algorithm via local search. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3662-3671. PMLR, 09-15 Jun 2019.

Shi Li and Ola Svensson. Approximating k-median via pseudo-approximation. In Proceedings of the Forty-Fifth Annual ACM Symposium on Theory of Computing , STOC '13, page 901-910, New York, NY, USA, 2013. Association for Computing Machinery.

Weiwei Liu, Xiaobo Shen, and Ivor Tsang. Sparse embedded k -means clustering. Advances in neural information processing systems , 30, 2017.

Stuart P. Lloyd. Least squares quantization in PCM. IEEE Transactions on Information Theory , 28(2):129-137, 1982.

Konstantin Makarychev, Yury Makarychev, and Ilya Razenshteyn. Performance of johnsonlindenstrauss transform for k-means and k-medians clustering. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , STOC 2019, page 1027-1038, New York, NY, USA, 2019. Association for Computing Machinery.

Andrew W. Moore. The anchors hierarchy: Using the triangle inequality to survive high dimensional data. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence , UAI'00, page 397-405, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc.

James Newling and Francois Fleuret. Fast k-means with accurate bounds. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 936-944, New York, New York, USA, 20-22 Jun 2016. PMLR.

Raymond T. Ng and Jiawei Han. Clarans: A method for clustering objects for spatial data mining. IEEE transactions on knowledge and data engineering , 14(5):1003-1016, 2002.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Dan Pelleg and Andrew Moore. Accelerating exact k-means algorithms with geometric reasoning. In Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '99, page 277-281, New York, NY, USA, 1999. Association for Computing Machinery.

James Philbin. Scalable object retrieval in very large image collections . PhD thesis, Oxford University, 2010.

James Philbin, Ondrej Chum, Michael Isard, Josef Sivic, and Andrew Zisserman. Object retrieval with large vocabularies and fast spatial matching. In 2007 IEEE Conference on Computer Vision and Pattern Recognition , pages 1-8, 2007.

Feng Qian, Abhinav Pathak, Yu Charlie Hu, Zhuoqing Morley Mao, and Yinglian Xie. A case for unsupervised-learning-based spam filtering. SIGMETRICS Perform. Eval. Rev. , 38(1):367-368, jun 2010.

MAli Rostami, Alieh Saeedi, Eric Peukert, and Erhard Rahm. Interactive visualization of large similarity graphs and entity resolution clusters. In EDBT , pages 690-693, 2018.

Erich Schubert and Peter J Rousseeuw. Faster k-medoids clustering: improving the pam, clara, and clarans algorithms. In Similarity Search and Applications: 12th International Conference, SISAP 2019, Newark, NJ, USA, October 2-4, 2019, Proceedings 12 , pages 171-187. Springer, 2019.

D. Sculley. Web-scale k-means clustering. In Proceedings of the 19th International Conference on World Wide Web , WWW'10, page 1177-1178, New York, NY, USA, 2010. Association for Computing Machinery.

Mina Sheikhalishahi, Andrea Saracino, Mohamed Mejri, Nadia Tawbi, and Fabio Martinelli. Fast and effective clustering of spam emails based on structural similarity. In Joaquin GarciaAlfaro, Evangelos Kranakis, and Guillaume Bonfante, editors, Foundations and Practice of Security , pages 195-211, Cham, 2016. Springer International Publishing.

Harsha Vardhan Simhadri, George Williams, Martin Aumüller, Matthijs Douze, Artem Babenko, Dmitry Baranchuk, Qi Chen, Lucas Hosseini, Ravishankar Krishnaswamny, Gopal Srinivasa, et al. Results of the neurips'21 challenge on billion-scale approximate nearest neighbor search. In NeurIPS 2021 Competitions and Demonstrations Track , pages 177-189. PMLR, 2022.

Christian Sohler and David P Woodruff. Strong coresets for k-median and subspace approximation: Goodbye dimension. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS) , pages 802-813. IEEE, 2018.

Kunal Talwar. Bypassing the embedding: Algorithms for low dimensional metrics. In Proceedings of the Thirty-Sixth Annual ACM Symposium on Theory of Computing , STOC '04, page 281-290, New York, NY, USA, 2004. Association for Computing Machinery.

Mo Tiwari, Martin J Zhang, James Mayclin, Sebastian Thrun, Chris Piech, and Ilan Shomorony. Banditpam: Almost linear time k-medoids clustering via multi-armed bandits. Advances in Neural Information Processing Systems , 33:10211-10222, 2020.

Erik Waingarten. Notes for algorithms for big data: Clustering. https://drive.google. com/file/d/1T5YYGrA3kdi4_QGvF3c_foOZvXMDaRPw/view .

Jing Wang, Jingdong Wang, Qifa Ke, Gang Zeng, and Shipeng Li. Fast approximate k-means via cluster closures. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 3037-3044, 2012.

A Problem Setup and Background

In this work, we always consider datasets X = { x 1 , . . . , x n } ⊂ R d of high-dimensional vectors, and we will measure their distance using the Euclidean ( ℓ 2 ) distance. Below, we define ( k, z ) -clustering. This problem introduces a parameter z ≥ 1 , which measures the sensivity to outliers (as z grows, the clusterings become more sensitive to points furthest from the cluster center). The case of k -means corresponds to z = 2 , but other values of z capture other well-known clustering objectives, like k -median (the case of z = 1 ).

Definition A.1 ( ( k, z ) -Clustering) . Consider a dataset X = { x 1 , . . . , x n } ⊂ R d , a desired number of clusters k ∈ N , and a parameter z ≥ 1 . For a set of k centers C = { c 1 , . . . , c k } ⊂ R d , let cost z ( X,C ) denote the cost of using the center set C to cluster X , i.e.,

We let opt k,z ( X ) denote the optimal cost over all choices of C = { c 1 , . . . , c k } ⊂ R d :

A ( k, z ) -clustering algorithm has the following specifications. The algorithm receives as input a dataset X = { x 1 , . . . , x n } ⊂ R d , as well as two parameters k ∈ N and z ≥ 1 . After it executes, the algorithm should output a set of k centers C = { c 1 , . . . , c k } ⊂ R d as well as an assignment σ : [ n ] → [ k ] mapping each point x i to a center c σ ( i ) .

We measure the quality of the solution ( C, σ ) using the ratio between its ( k, z ) -clustering cost cost z ( X,C,σ ) and the optimal cost opt k,z ( X ) , where

For any D > 1 , an algorithm that produces a D -approximation to ( k, z ) -clustering should guarantee that cost z ( X,C,σ ) is at most D · opt k,z ( X ) . For a randomized algorithm, the guarantee should hold with large probability (referred to as the success probability) for any input dataset.

A.1 k -Means++ Seeding

The k -means++ seeding algorithm is a well-studied algorithm introduced in [6], and it is an important component of our algorithm. Below, we describe it for general z ≥ 1 , not necessarily z = 2 .

Definition A.2 ( k -means++ seeding, for arbitrary z ≥ 1 ) . Given n data points x 1 , . . . , x n ∈ R d , the k -means++ seeding algorithm produces a set of k centers, x ℓ 1 , . . . , x ℓ k with the following procedure:

Choose ℓ 1 uniformly at random from [ n ] .

For t = 2 , . . . , k , sample ℓ t as follows. For every i ∈ [ n ] , let p i denote the Euclidean distance from x i to its closest point among x ℓ 1 , . . . , x ℓ t -1 . Sample ℓ t from [ n ] so that the probability Pr[ ℓ t = i ] is proportional to p z i for every i ∈ [ n ] . That is,

In the context of k -means (i.e., when z = 2 ), the distribution is known as the D 2 distribution.

Output x ℓ 1 , . . . , x ℓ k .

In the above description, Step 2 of k -means++ needs to maintain, for each dataset point x i , the Euclidean distance p i to its closest center among the centers selected before the current iteration. This step is implemented by making an entire pass over the dataset for each of the k -1 iterations of Step 2, leading to an O ( ndk ) running time.

Theorem A.1 ([6], Theorem 3 in [70]) . For x 1 , . . . , x n ∈ R d , let X = { x 1 , . . . , x n } be the input to the k -means++ seeding algorithm. For the output x ℓ 1 , . . . , x ℓ k of the k -means++ seeding algorithm, define C = { x ℓ 1 , . . . , x ℓ k } . Then

A.2 Coresets via Sensitivity Sampling

One of our algorithm's applications is constructing coresets for ( k, z ) -clustering. We give a formal definition and describe the primary technique for building coresets.

Definition A.3. Given a dataset X = { x 1 , . . . , x n } ⊂ R d , as well as parameters k ∈ N , z ≥ 1 and ε > 0 , a (strong) ε -coreset for ( k, z ) -clustering is specified by a set of points Y ⊂ R d and a weight function w : Y → R ≥ 0 , such that, for every set C = { c 1 , . . . , c k } ⊂ R d ,

Coresets are constructed via 'sensitivity sampling,' a technique that, given an approximate clustering of a dataset X , produces a probability distribution such that sampling enough points from this distribution results in a coreset.

Definition A.4 (Sensitivity Sampling) . Consider a dataset X = { x 1 , . . . , x n } ⊂ R d , as well as parameters k ∈ N , z ≥ 1 . For a centet set C = { c 1 , . . . , c k } ⊂ R d and assignment σ : [ n ] → [ k ] , let X j = { x i : σ ( i ) = j } . We let D be a distribution supported on X where

The main theorem that we will use is given below, which shows that given a center set and an assignment that gives an α -approximation to ( k, z ) -clustering, one may sample from the distribution D defined about in order to generate a coreset with high probability.

Theorem A.2 ([15]) . For any dataset X = { x 1 , . . . , x n } ⊂ R d and any parameters k ∈ N and z ≥ 1 , suppose that C = { c 1 , . . . , c k } ⊂ R d and σ : [ n ] → [ k ] is a α -approximation to ( k, z ) -clustering, i.e.,

Letting D denote the distribution specified in Definition A.4, the following occurs with high probability.

We let y 1 , . . . , y s denote independent samples from D , and w ( y i ) be the inverse of the probability that y i is sampled according to D . We set s ≥ poly( kd · α · 2 z /ε ) .

The set Y = { y 1 , . . . , y s } with weights w is an ε -coreset for ( k, z ) -clustering.

A.3 A Simple Lemma

We will repeatedly use the following simple lemma.

Lemma A.3. Let a, b ∈ R ≥ 0 be any two numbers and z ≥ 1 . Then, ( a + b ) z ≤ 2 z -1 a z +2 z -1 b z .

Proof. The function ϕ ( t ) = t z is convex for z ≥ 1 , so Jensen's inequality implies ϕ (( a + b ) / 2) ≤ (1 / 2) ϕ ( a ) + (1 / 2) ϕ ( b ) .

B Approximation Guarantees from One-Dimensional Projections

In this section, we prove Theorem 2.2 (rather, the generalization of Theorem 2.2 to any z ≥ 1 ) by analyzing the random one-dimensional projection step in our algorithm. In order to introduce some notation, let X = { x 1 , . . . , x n } ⊂ R d be a set of points, and for a partition of X into k sets, ( Y 1 , . . . , Y k ) , we let the ( k, z ) -clustering cost of X with the partition ( Y 1 , . . . , Y k ) be

and call the k points c selected as minima a set of centers realizing the ( k, z ) -clustering cost of ( Y 1 , . . . , Y k ) . We note that (2) is a cost function for ( k, z ) -clustering, but it is different from

Definition A.1. In Definition A.1, the emphasis is on the set of k centers C = { c 1 , . . . , c k } , and the induced set of clustering of X , i.e., the partition ( Y 1 , . . . , Y k ) given by assigning points to the closest center, is only implicitly specified by the set of centers. On the other hand, (2) emphasizes the clustering ( Y 1 , . . . , Y k ) , and the set of k centers implicitly specified by ( Y 1 , . . . , Y k ) . The optimal set of centers and the optimal clustering will achieve the same cost; however, our proof will mostly consider the clustering ( Y 1 , . . . , Y k ) as the object to optimize. Shortly, we will sample a (random) dimensionality reduction map Π : R d → R t and seek bounds for t = 1 . We will write cost z ( Π ( Y 1 ) , . . . , Π ( Y k )) for the cost of clustering the points after applying the dimensionality reduction map Π to the partition Y 1 , . . . , Y k . Namely, we write

Definition B.1. For a set of points X = { x 1 , . . . , x n } ⊂ R d , we use X ∗ 1 , . . . , X ∗ k of X to denote the partition of X with minimum ( k, z ) -clustering cost and we use C ∗ = { c ∗ 1 , . . . , c ∗ k } ⊂ R d to denote a set of k centers which realizes the ( k, z ) -clustering cost of X ∗ 1 , . . . , X ∗ k , i.e., the set of centers which satisfies

By slight abuse of notation, we also let c ∗ : X → C ∗ be the map which sends every point of X to its corresponding center (i.e., if x ∈ X ∗ i , then c ∗ ( x ) is the point c ∗ i ).

We prove the following lemma, which generalizes Lemma 2.5 from k -means to ( k, z ) -clustering (recall that k -means corresponds to the case of z = 2 ).

Lemma B.1 (Effect of One-Dimensional Projection on ( k, z ) -Clustering) . For n, d, k ∈ N and z ≥ 1 , let X = { x 1 , . . . , x n } ⊂ R d be an arbitrary dataset. We consider the (random) linear map Π : R d → R given by sampling g ∼ N (0 , I d ) and setting

With probability at least 0 . 9 over g , the following occurs:

We consider the projected dataset X ′ = { x ′ 1 , . . . , x ′ n } ⊂ R be given by x ′ i = Π ( x i ) , and

For any r ≥ 1 , we let ( Y 1 , . . . Y k ) denote any partition of X satisfying

Then,

By setting z = 2 , we obtain the desired bound from Lemma 2.5. We can immediately see that, from Lemma B.1, and the approximation guarantees of k -means++ (or rather, its generalization to z ≥ 1 ) in Theorem A.1, we obtain our desired approximation guarantees. Below, we state the generalization of Theorem 2.2 to all z ≥ 1 and, assuming Lemma B.1, its proof.

Theorem B.2 (Generalization of Theorem 2.2 to z ≥ 1 ) . For n, d, k ∈ N and z ≥ 1 , let X = { x 1 , . . . , x n } ⊂ R d be an arbitrary dataset. We consider the following generalization of our algorithm PRONE :

Sample a random Gaussian vector g ∼ N (0 , I d ) and consider the projection X ′ = { x ′ 1 , . . . , x ′ n } given by x ′ i = Π ( x i ) , for Π ( x ) = ⟨ x, g ⟩ ∈ R .

Execute the (generalization of the) k -means++ seeding strategy for z ≥ 1 of Definition A.2 with the dataset X ′ ⊂ R , and let x ′ j 1 , . . . , x ′ j k ∈ R denote the centers and ( Y 1 , . . . , Y k ) denote the partition of X specifying the k clusters found.

Output the clustering ( Y 1 , . . . , Y k ) , and the set of centers c 1 , . . . , c k ∈ R d where

Then, with probability at least 0 . 8 over the execution of the algorithm,

Proof of Theorem B.2 assuming Lemma B.1. We consider the case (over the randomness in the execution of the algorithm) that:

The conclusions of Lemma B.1 hold for the projected dataset X ′ (which happens with probability at least 0 . 9 ) by Lemma B.1.

The execution of the generalization k -means++ seeding strategy on X ′ (from Definition A.2) produces a set of centers { x ′ j 1 , . . . , x ′ j k } ⊂ R which cluster X ′ with cost at most O (2 2 z log k ) · opt k,z ( X ′ ) (which also happens with probability 0 . 9 by Markov's inequality).

By a union bound, both hold with probability at least 0 . 8 . We now use Lemma B.1 to upper bound the cost of the clustering ( Y 1 , . . . , Y k ) . The first inequality is trivial; suppose we let ˆ c ∗ 1 , . . . , ˆ c ∗ ℓ ∈ R d be the centers which minimize for each ℓ ∈ [ k ]

Then, we trivially have

Furthermore, we can also show a corresponding upper bound. For each ℓ ∈ [ k ] , recall that c ℓ ∈ R d is the center of mass of Y k , so we can apply the triangle inequality and Lemma A.3

where the second inequality is Jensen's inequality, since ϕ ( x ) = ∥ ˆ c ∗ ℓ -x ∥ z 2 is convex for z ≥ 1 . Thus, we have upper-bounded

The final step involves relating cost z ( Y 1 , . . . , Y k ) using the conclusions of Lemma B.1. Notice that our algorithm produces the clustering ( Y 1 , . . . , Y k ) of X ′ which is specified by letting

and by the event (2), we have cost z ( Π ( Y 1 ) , . . . , Π ( Y k )) ≤ O (2 2 z log k ) · opt k,z ( X ′ ) . By event (1), Lemma B.1 implies that cost z ( Y 1 , . . . , Y k ) ≤ 2 O ( z ) · k 2 z · O (2 2 z log k ) · opt k,z ( X ) . Combined with (3), we obtain our desired bound.

and therefore

B.1 Proof of Lemma B.1

We now turn to the proof of Lemma B.1, where our analysis will proceed in two steps. First, we assume a fixed dimensionality reduction map Π: R d → R t , which satisfies two geometrical conditions on Π . Under these conditions, we show how to 'lift' an approximate clustering of the mapped points in R t to an approximate clustering of the original dataset in R d at the cost of weakening the approximation ratio. Then, we show that a simple one-dimensional projection Π : R d → R given by Π ( x ) = ⟨ x, g ⟩ , for g being sampled from a d -dimensional standard Gaussian, satisfies the geometrical conditions of our lemma.

Lemma B.3. Let X = { x 1 , . . . , x n } ⊂ R d and Π: R d → R t be a linear map. Let C = { c ∗ 1 , . . . , c ∗ k } ⊂ R d denote the set of centers minimizing cost z ( X,C ) , and ( X ∗ 1 , . . . , X ∗ k ) denote the optimal ( k, z ) -clustering, and suppose that for the parameters D 1 , D 2 , D 3 ≥ 1 , the following conditions hold:

Centers Don't Contract : Every i, j ∈ [ k ] satisfies

Cost of ( X ∗ 1 , . . . , X ∗ k ) does not Increase : We have that

Approximately Optimal (Π( Y 1 ) , . . . , Π( Y k )) : The partition ( Y 1 , . . . , Y k ) of X is D 3 -approximately optimal for Π( X ) , i.e.,

Then,

Before starting the proof of Lemma B.3, we show that projecting points onto a random Gaussian vector gives the first two desired guarantees of the above lemma with D 1 := ( k 2 /δ ) and D 2 := 2 O ( z ) /δ with probability at least 1 -δ . The first lemma that we state below shows that the first condition of Lemma B.3 is satisfied with high probability, and the second lemma that the second condition of Lemma B.3 is satisfied with high probability.

Lemma B.4 (Centers Don't Contract) . Let C = { c 1 , . . . , c k } ⊂ R d denote any collection of k points and let Π : R d → R be a random map given by for a randomly chosen vector g ∼ N (0 , I d ) . Then, with probability at least 1 -δ over g , every i, j ∈ [ k ] satisfies

Lemma B.5 (Cost of ( X ∗ 1 , . . . , X ∗ k ) does not Increase) . Let X = { x 1 , . . . , x n } ⊂ R d and let X ∗ 1 , . . . , X ∗ k be the partition of X , and c ∗ 1 , . . . , c ∗ k ∈ R d be the centers which minimize the ( k, z ) -clustering cost of X . Then, with probability a least 1 -δ ,

Proof of Lemma B.1 assuming Lemma B.3, Lemma B.4 and Lemma B.5. We will apply Lemma B.3 by letting δ be a small enough constant (say, δ = 0 . 01 ) to take a union bound. Lemma B.4 implies the first condition with D 1 = O ( k 2 ) and Lemma B.5 implies the second condition with D 2 = 2 O ( z ) . Finally, the second assumption of Lemma B.1 sets r = D 3 , from which we derive the conclusion.

We now prove Lemma B.4, Lemma B.5. Lemma B.3 is proved in Subsection B.2.

Proof of Lemma B.4. The proof relies on the 2-stability property of the Gaussian distribution. Namely, if we let z ∈ R d be an arbitrary vector and we sample a standard Gaussian vector g ∼ N (0 , I d ) , the (scalar) random variable ⟨ z, g ⟩ is distributed like ∥ z ∥ 2 · g ′ , where g ∼ N (0 , 1) . Using the 2 -stability of the Gaussian distribution for every i, j ∈ [ k ] , we have that ∥ Π ( c i ) -Π ( c j ) ∥ 2 2 is distributed as ( g ′ ) 2 ∥ c i -c j ∥ 2 2 , where g ′ is distributed as a (one-dimensional) Gaussian N (0 , 1) . Thus, by a union bound, the probability that there exists a pair i, j ∈ [ k ] , which satisfies ∥ Π ( c i ) -Π ( c j ) ∥ 2 2 < α 2 · ∥ c i -c j ∥ 2 2 is at most k 2 times the probability that a Gaussian random variable lies in [ -α, α ] , and this probability is easily seen to be less than α . Setting α = δ/k 2 gives the desired lemma.

Proof of Lemma B.5. Similarly to the proof of Lemma B.4, we have that ∥ Π ( x ) -Π ( c ∗ i ) ∥ z 2 is distributed as | g ′ | z · ∥ x -c ∗ i ∥ z 2 , where g ′ is distributed as a (one-dimensional) Gaussian N (0 , 1) . By linearity of expectation,

To conclude, note that for z ≥ 1 , there is some α > 1 such that αz is an even integer and α ≤ 2 . Thus, by Jensen's inequality and the fact that f ( x ) = x 1 /α is concave we can write

Now note that all odd moments of the Gaussian distribution are zero by symmetry. Thus, for the moment generating function E [ e g ′ ] it holds that

Applying Markov's inequality now completes the proof.

B.2 Proof of Lemma B.3

Let { ˆ c i } i ∈ [ k ] be an optimal set of centers for the partition ( Y 1 , . . . , Y k ) for the ( k, z ) -clustering problem on Π( X ) , where ˆ c i ∈ Y i . Specifically, the points ˆ c 1 , . . . , ˆ c k ∈ R t are those which minimize

To quantize the cost difference between the centers c ∗ and the centers ˆ c we analyze the following value. We assume we mapped every point of X to Π c ∗ ( X ) , and we then compute the cost of the partition Y 1 , . . . , Y k on this set. Formally, we let

First, we prove the following simple claim.

Claim B.6. There exists a set of centers c ′ 1 , . . . , c ′ k (with possible repetitions) which are chosen among the points { c ∗ 1 , . . . , c ∗ k } such that

As E [ e g ′ ] ≤ e 1 / 2 it follows that

Proof. We will prove the claim using the probabilistic method. For every i ∈ [ k ] , consider the distribution over center { c ∗ 1 , . . . , c ∗ k } which samples a center c ′ i as

Then, we upper bound the expected cost of using the centers c ′ i . Using Lemma A.3,

We now upper bound cost z ( Y 1 , . . . , Y k ) in terms of cost z ( X ∗ 1 , . . . , X ∗ k ) . We do this by going through the centers chosen according to Claim B.6. This will allow us to upper bound the cost of clustering with ( Y 1 , . . . , Y k ) in terms of the cost z ( X ∗ 1 , . . . , X ∗ k ) as well as clustering cost involving only pairwise distances from { c ∗ 1 , . . . , c ∗ k } . Then, we relate to distances after applying the map Π . Specifically, first notice that if we consider the set of centers c ′ 1 , . . . , c ′ k chosen from Claim B.6

where the third inequality uses Lemma A.3 once more. Note that the right-most summation of (4) involves distances which are only among c ∗ 1 , . . . , c ∗ k , so by the first assumption of the map Π and Claim B.6, we may upper bound

Combining (4) and (5), we may upper bound

We continue upper bounding the right-most expression in (6) by applying the triangle inequality:

By the third assumption of the lemma, we note that

Summarizing by plugging (7) and (8) into (6), can upper bound

C Efficient Seeding in One Dimension

In this section, we prove Theorem 2.1, which shows an upper bound for the running time of our algorithm PRONE . As in Theorem B.2, we consider a generalized version of PRONE where we run k -means++ seeding for general z ≥ 1 (Definition A.2) in Step 2. We prove the following generalized version of Theorem 2.1:

Theorem C.1 (Theorem 2.1 for general z ≥ 1 ) . Let X = { x 1 , . . . , x n } ⊂ R d be a dataset consisting of n points in d dimensions. Assume that d ≤ nnz ( X ) , which can be ensured after removing redundant dimensions j ∈ [ d ] where the j -th coordinate of every x i is zero. For any z ≥ 1 , the algorithm PRONE (for general z as in Theorem B.2) has expected running time O ( nnz ( X )+2 z/ 2 n log n ) on X . For any δ ∈ (0 , 1 / 2) , with probability at least 1 -δ , the algorithm runs in time O ( nnz ( X )+2 z/ 2 n log( n/δ )) . Moreover, the algorithm always runs in time O ( nnz ( X ) + n log n + nk ) .

To prove Theorem 2.1, we show an efficient implementation (Algorithm 1) of the k -means++ seeding procedure that runs in expected time O (2 z/ 2 n log n ) for one-dimensional points (Lemma C.3). A naive implementation of the seeding procedure would take Θ( nk ) time in one dimension because we need Θ( n ) time to update p i and sample from the D 2 distribution to add each of the k centers. To obtain an improved and provable running time, we use a basic binary tree data structure to sample from the D 2 distribution more efficiently, and we use a potential argument to bound the number of updates to p i .

The data structure S we use in Algorithm 1 can be implemented as a basic binary tree, as described in more detail in Appendix D. The data structure S keeps track of n nonnegative numbers s 1 , . . . , s n corresponding to p z 1 , . . . , p z n and it supports the following operations:

Initialize ( a ) . Given an array a = ( a 1 , . . . , a n ) ∈ R n ≥ 0 , the operation Initialize ( a ) creates a data structure S that keeps track of the numbers s 1 , . . . , s n initialized so that ( s 1 , . . . , s n ) = ( a 1 , . . . , a n ) . This operation runs in O ( n ) time.

Sum ( S ) . The operation Sum ( S ) returns the sum s 1 + · · · + s n . This operation runs in O (1) time, as the value will be maintained as the data structure is updated.

Find ( S, r ) . Given a number r ∈ [0 , ∑ n i =1 s i ) , the operation Find ( S, r ) returns the unique index ℓ ∈ { 1 , . . . , n } such that

This operation runs in O (log n ) time.

Update ( S, a, i 1 , i 2 ) . Given an array a = ( a 1 , . . . , a n ) ∈ R n ≥ 0 and indices i 1 , i 2 satisfying 1 ≤ i 1 ≤ i 2 ≤ n , the operation Update ( S, a, i 1 , i 2 ) performs the updates s i ← a i for every i = i 1 , i 1 +1 , . . . , i 2 . This operation runs in O (( i 2 -i 1 +1) + log n ) time.

The following claim shows that Algorithm 1 correctly implements the k -means++ seeding procedure in one dimension.

Claim C.2. Consider the values of t, a 1 , . . . , a n and the data structure S at the beginning of each iteration of the for-loop (i.e., right before Line 6). Let s 1 , . . . , s n be the numbers the data structure S keeps track of. For every i = 1 , . . . , n , define p i := min t ′ =1 ,...,t -1 | x i -x ℓ t ′ | . Then s i = a i = p z i for every i = 1 , . . . , n . Consequently, the distribution of ℓ t at Line 7 conditioned on the execution history so far satisfies Pr[ ℓ t = i ] = p z i / ∑ n i ′ =1 p z i ′ for every i = 1 , . . . , n .

The claim follows immediately by induction over the iterations of the for-loop based on the description of the data structure S and its operations above. The following lemma bounds the running time of Algorithm 1:

Lemma C.3 (Lemma 2.4 for general z ≥ 1 ) . The expected running time of Algorithm 1 is O (2 z/ 2 n log n ) . For any δ ∈ (0 , 1 / 2) , with probability at least 1 -δ , Algorithm 1 runs in time O (2 z/ 2 n log( n/δ )) . Moreover, Algorithm 1 always runs in time O ( n log n + nk ) .

Before proving Lemma C.3, we first use it to prove Theorem C.1.

Proof of Theorem C.1. Lemma C.3 bounds the running time of Step 2 of our algorithm PRONE defined in Section 2. Now we show that Step 1 (random one-dimensional projection) can be performed in time O ( nnz ( X ) + n ) . Indeed, x ′ i can be computed as x ′ i = ∑ j x ij v j where the sum is over all the non-zero coordinates x ij of x i , and each v j is drawn independently from the one-dimensional standard Gaussian (the value of v j should be shared for all i ). The time needed to compute x ′ 1 , . . . , x ′ n in this way is O ( nnz ( X ) + n ) . In Step 3 of PRONE , we compute the center of mass for every cluster.

This can be done in time O ( nnz ( X ) + n ) by summing up the points in each cluster and dividing each sum by the number of points in that cluster.

The key step towards proving Lemma C.3 is to bound the number of updates to a at Lines 9 and 13. As the algorithm starts by sorting the n input points, which can be done in time O ( n log n ) , we can assume that the points are sorted such that x 1 ≤ · · · ≤ x n . For i = 1 , . . . , n and t = 2 , . . . , k , we define ξ ( i, t ) = 1 if a i is updated at Line 9 in iteration t and define ξ ( i, t ) = 0 otherwise. Here, we denote each iteration of the for-loop beginning at Line 5 by the value of the iterate t . We define u i := ∑ k t =2 ξ ( i, t ) to be the number of times a i gets updated at Line 9. The following lemma gives upper bounds on u i both in expectation and with high probability:

Lemma C.4. For every i = 1 , . . . , n , it holds that

Moreover, for some absolute constant B > 0 and for every δ ∈ (0 , 1 / 2) , it holds that

Before proving Lemma C.4, we first use it to prove Lemma C.3.

Proof of Lemma C.3. Recall that for i = 1 , . . . , n and t = 2 , . . . , k , we define ξ ( i, t ) = 1 if a i is updated at Line 9 in iteration t and define ξ ( i, t ) = 0 otherwise. Similarly, for j = 1 , . . . , n and t = 2 , . . . , k , we define ξ ′ ( j, t ) = 1 if a j is updated at Line 13 in iteration t and define ξ ′ ( j, t ) = 0 otherwise.

The computation at Lines 1-4 takes O ( n log n ) time. The computation at Lines 18-28 takes O ( k log k + n ) = O ( n log n ) time. For t = 2 , . . . , k , iteration t of the for-loop takes time O (log n + ∑ n i =1 ξ ( i, t ) + ∑ n i =1 ξ ′ ( i, t )) . Summing them up, the total running time of Algorithm 1 is

Also, for any δ ′ ∈ (0 , 1 / 2) , setting δ = δ ′ /n in Lemma C.4, by the union bound we have

Similarly to (10) and (11) we have

Plugging (10) and (11) into (9) proves that the expected running time of Algorithm 1 is O (2 z/ 2 n log n ) . Choosing δ ′ = δ/ 2 for the δ in Lemma C.3 and plugging (11) and (13) into (9), we can use the union bound to conclude that with probability at least 1 -δ Algorithm 1 runs in time O (2 z/ 2 n log( n/δ )) . Finally, plugging ξ ( i, t ) ≤ 1 and ξ ′ ( i, t ) ≤ 1 into (9), we get that Algorithm 1 always runs in time O ( n log n + nk ) .

To prove Lemma C.4, for i = 0 , . . . , n and u = 0 , . . . , u i , we define t ( i, u ) to be the smallest t ∈ { 1 , . . . , k } such that ∑ t t ′ =2 ξ ( i, t ′ ) = u . That is, a i gets updated at Line 9 for the u -th time in iteration t ( i, u ) . Our definition implies that t ( i, 0) = 1 and t ( i, u ) ∈ { 2 , . . . , k } for u = 1 , . . . , u i . We define a nonnegative potential function η ( i, u ) as follows and show that it decreases exponentially in expectation as u increases (Lemma C.5).

Potential Function η ( i, u ) . For t = 2 , . . . , k , we consider the value of ℓ t after Line 7 in iteration t . For u = 1 , . . . , u i , we define η ( i, u ) to be ℓ t ( i,u ) -i , which is guaranteed to be a positive integer by the definition of t ( i, u ) . Indeed, in the while-loop containing Line 9, i starts from ℓ t -1 and keeps decreasing, so whenever Line 9 is executed, i is smaller than ℓ t . In particular, a i is updated at Line 9 in iteration t ( i, u ) of the for-loop, so we have i < ℓ t ( i,u ) . We define η ( i, 0) = n , and for u = u i +1 , u i +2 , . . . , we define η ( i, u ) = 0 . See Figure 4 for an example illustrating the definition of η ( i, u ) .

Figure 4: An example illustrating the definition of the potential function η . Here, a i is updated at Line 9 for the first time in iteration t ( i, 1) when ℓ t ( i, 1) = i +4 . Then a i gets updated at Line 9 for the second time in iteration t ( i, 2) when ℓ t ( i, 2) = i +1 . We always have ℓ t > i whenever a i is updated at Line 9, and η is the difference between ℓ t and i . Thus in this example η ( i, 1) = 4 and η ( i, 2) = 1 . If a i is never updated at Line 9 after iteration t ( i, 2) , we define η ( i, u ) = 0 for u = 3 , 4 , . . . .

Lemma C.5 (Potential function decrease) . For any i ∈ { 1 , . . . , n } and u ∈ Z ≥ 0 ,

Before proving Lemma C.5, we first use it to prove Lemma C.4. Intuitively, Lemma C.5 says that η ( i, u ) decreases exponentially (in expectation) as a function of u . Since η ( i, u ) is always a nonnegative integer, we should expect η ( i, u ) to become zero as soon as u exceeds a small threshold. Moreover, our definition ensures η ( i, u i ) > 0 , so u i must be smaller than the threshold. This allows us to show upper bounds for u i and prove Lemma C.4.

Proof of Lemma C.4. Our definition of η ensures η ( i, u i ) ≥ 1 . By Lemma C.5 and Lemma C.7,

This implies E [ u i ] = O (2 z/ 2 log n ) . Moreover, by Lemma C.5,

and thus, by Markov's inequality,

For any δ ∈ (0 , 1 / 2) , choosing u = ln( n/δ ) / ln( 2 z/ 2 +1 2 z/ 2 ) = O (2 z/ 2 log( n/δ )) in the inequality above gives Pr[ u i ≥ u ] ≤ δ .

We need the following helper lemma to prove Lemma C.5.

Lemma C.6. The following holds at the beginning of each iteration of the for-loop in Algorithm 1, i.e., right before Line 6 is executed. Choose an arbitrary i = 1 , . . . , n and define

Then for ℓ, ℓ ′ ∈ L satisfying ℓ < ℓ ′ , it holds that a ℓ ′ ≤ 2 z a ℓ .

Proof. For t = 2 , . . . , k , at the beginning of iteration t , the values ℓ 1 , . . . , ℓ t -1 have been determined. For every x ∈ R , let ρ ( x ) denote the value among x ℓ 1 , . . . , x ℓ t -1 closest to x . By Claim C.2, a i = | x i -ρ ( x i ) | z for every i ∈ [ n ] . Now for a fixed i ∈ [ n ] , define L as in (14) and consider ℓ, ℓ ′ ∈ L satisfying ℓ < ℓ ′ . It is easy to see that x i ≤ ρ ( x ℓ ) ≤ x ℓ ′ cannot hold because otherwise a i ≤ | x i -ρ ( x ℓ ) | z ≤ | x i -x ℓ ′ | z < a i , a contradiction. For the same reason, the inequality x i ≤ ρ ( x ℓ ′ ) ≤ x ℓ ′ cannot hold. Thus there are only three possible orderings of x i , x ℓ , x ℓ ′ , ρ ( x ℓ ) , ρ ( x ℓ ′ ) :

<latexi sh

1_b

64="Nd87

cUk5

Du

O3pB

r

oHFK

z

>A

YX

ZL

EM

R9

+

V

Sn

v

IW

G0

/T

y

g

q

P

2

j

f

w

m

Q

<latexi sh

1_b

64="Sn5

F9GHvUXA70

I8

w

kf

Q

>

DZ

c

Ld

NB

M

R

Vu

g

y

E

T

3

Y

o

J

r

2

j

O+

z

/

m

q

p

C

K

W

P

<latexi sh

1_b

64="kUTEp

WBn

C

X

Ay

NfQ

>

DY

c

ZL

M

3

R9

r

V

v

OKHo

I

G0

/

d

g5

q

P

2

j

7

z

w

m

F

u

8

<latexi sh

1_b

64="3+

NMI

Dm

J0o5XdU

F/w

8zf2k

>A

c

ZL

9

RPr

O

C

p

Q

H

S

yq

Y

u

Wn

B

g

V

7

v

j

T

E

G

K

<latexi sh

1_b

64="Cy0TYu

SzfEOP

/

wm

Uro

>A

DdX

c

ZLN

M

I8

v

QJ

Rk

F

pB

q

K

H

V

G

72

W

g

j

5

n

9

3

+

<latexi sh

1_b

64="Cmwr5AJ3gXj q+

ZSy p2L

8

>

D

c

d

M

F

9

o

RGN

U

KO

P

W7T

n

u

E

k

B

f

/z

v

H

0

I

Y

Q

V

<latexi sh

1_b

64="

JH8rSgpTd

7q5

2

G

>A

D

3

c

ZL

MwF

9

o

Rvj

B

YV

mW

fu

n

EKN

0

Oy

I

+

X

C

k

/

Q

P

<latexi sh

1_b

64="gXTLr/

BkPw

H

ZNj

Jm nI

>A

Dd

c

M

E

p

FUKyC

So

z

9Q

R

5

q

3

V

2

f

u

W7

+

O

0

8

G

Y

v

<latexi sh

1_b

64="80qv

u

ULc

y

AGD

g

/

oH

>

ZX

d

MwF

9

R

OEC

JQV

3

p

P

W7Tm

n

N

k

K

B

f

r

z

5

S

+

2

j

I

Y

<latexi sh

1_b

64="Sn5

F9GHvUXA70

I8

w

kf

Q

>

DZ

c

Ld

NB

M

R

Vu

g

y

E

T

3

Y

o

J

r

2

j

O+

z

/

m

q

p

C

K

W

P

<latexi sh

1_b

64="I

W

5

Ku

STzDZ29gR

o

Oc

>A

YX

LN

EM

3

+

r

V

np

v

H

k

G0

/

d

y

q

P

j

f

7

B

w

m

F

Q

8

ρ ( x ℓ ) = ρ ( x ℓ ′ ) < x i ≤ x ℓ ≤ x ℓ ′ ;

ρ ( x ℓ ) < x i ≤ x ℓ ≤ x ℓ ′ < ρ ( x ℓ ′ ) ;

x i ≤ x ℓ ≤ x ℓ ′ < ρ ( x ℓ ) = ρ ( x ℓ ′ ) .

In scenario 3, it is clear that a ℓ ′ = | x ℓ ′ -ρ ( x ℓ ′ ) | z ≤ | x ℓ -ρ ( x ℓ ) | z = a ℓ . In the first two scenarios, for any t ′ = 0 , . . . , t -1 ,

This implies that ρ ( x ℓ ) is the closest point to x i among x ℓ 1 , . . . , x ℓ t -1 . Therefore, a i = | x i -ρ ( x ℓ ) | z . Jensen's inequality ensures (( g + h ) / 2) z ≤ ( g z + h z ) / 2 for any g, h ≥ 0 , which implies ( g + h ) z ≤ 2 z -1 g z +2 z -1 h z . Therefore,

whereas

Thus, we have a ℓ ′ ≤ 2 z a ℓ in all three scenarios.

Proof of Lemma C.5. Throughout the proof, we fix i ∈ { 1 , . . . , n } and u ∈ Z ≥ 0 so that they are deterministic numbers. Algorithm 1 is a randomized algorithm, and when we run it, exactly one of the following four events happens, and we define a random variable t ∗ accordingly:

Event E 1 : u i < u . That is, a i gets updated at Line 9 for less than u times. In this case we have η ( i, u +1) = 0 by our definition of η , and we define t ∗ = + ∞ .

Event E 2 : u i = u and i is never chosen as ℓ t at Line 7. In this case we also have η ( i, u +1) = 0 , and we also define t ∗ = + ∞ .

Event E 3 : u i = u and there exists t ∈ { 2 , 3 , . . . , k } such that i is chosen as ℓ t at Line 7 in iteration t . This t must satisfy t > t ( i, u ) , as all updates to a i in Line 9 must happen before x i is chosen as a center. We define t ∗ = t in this case. Again, we have η ( i, u +1) = 0 in this case.

Event E 4 : u i > u . We define t ∗ := t ( i, u +1) > t ( i, u ) in this case.

Define E ∗ := E 3 ∪ E 4 . Since η ( i, u +1) = 0 under E 1 and E 2 , it suffices to prove that 5

By our definition, the random variable t ∗ takes its value in { 2 , 3 , . . . , k } ∪ { + ∞} . Moreover, t ∗ = + ∞ if and only if E ∗ does not happen. Therefore, to prove (15), it suffices to prove the following for every t 0 = 2 , 3 , . . . , k :

Consider a fixed t 0 ∈ { 2 , 3 , . . . , k } . For t ∗ = t 0 to happen, the following must hold during the execution of Algorithm 1 before iteration t 0 : a i has been updated at Line 9 for exactly u times, and i has not been chosen as ℓ t at Line 7. Thus, the rest of the proof assumes that the execution history H of Algorithm 1 before iteration t 0 satisfies this property. Now we know that the values η ( i, 0) , . . . , η ( i, u ) are determined by the execution history H . Lines 6-7 guarantee that the distribution of ℓ t 0 satisfies

where we use the values a 1 , . . . , a n right before iteration t 0 is executed. Moreover, conditioned on H , we have t ∗ = t 0 if and only if ℓ t 0 ∈ L , where

Therefore, if we further condition on t ∗ = t 0 , we have ℓ t 0 ∈ L and

When t ∗ = t 0 , we have η ( i, u +1) = ℓ t 0 -i . Therefore, to prove (16), it suffices to show that

It is clear that we can write L as L = { i, i +1 , . . . , ℓ ∗ } for some integer ℓ ∗ ≥ i . If u > 0 , Claim C.2 implies a i ≤ | x i -x ℓ t ( i,u ) | z , and thus ℓ ∗ < ℓ t ( i,u ) and ℓ ∗ -i ≤ ℓ t ( i,u ) -i = η ( i, u ) . If u = 0 , we have η ( i, u ) = n , so it also holds that ℓ ∗ -i ≤ η ( i, u ) . By (17) and Lemma C.6, we can set γ = 2 z in Lemma C.8 to get

This proves (18) and thus proves the lemma.

C.1 Helper Lemmas

Lemma C.7. Let M ≥ 1 and λ ∈ (0 , 1) be parameters. Let α 0 , α 1 , . . . ∈ [0 , + ∞ ) be random variables satisfying α 0 = M and E [ α i +1 | α 1 , . . . , α i ] ≤ λα i for every i = 0 , 1 , . . . . Let t ≥ 0 be the smallest integer satisfying α t < 1 . Then

Proof. For every j = 0 , 1 , . . . , we define a random variable t j := min { t, j } . By the monotone convergence theorem, it suffices to show that

We prove (19) by induction on j . When j = 0 , we have t j = 0 , and the inequality above holds trivially. We assume that (19) holds for an arbitrary j ∈ Z ≥ 0 and show that it also holds with j replaced by j +1 . We have

By our definition of t j +1 , we have t j +1 -1 = min { t -1 , j } . Applying our induction hypothesis on the sequence α 1 , α 2 , . . . , we have

It is easy to check that f is an increasing concave function of α ∈ [0 , + ∞ ) and 1 + f ( λα ) ≤ f ( α ) holds for every α ≥ 1 . Plugging (21) into (20), we have

Lemma C.8. Let γ ≥ 1 be a real number. Let β 0 , . . . , β m -1 be non-negative real numbers such that for every i, j ∈ { 0 , . . . , m -1 } satisfying i ≤ j , it holds that β j ≤ γβ i . Then,

where

Proof. The lemma holds trivially if β 0 = 0 because in this case β j ≤ γβ 0 = 0 for every j = 0 , . . . , m -1 . We thus assume w.l.o.g. that β 0 > 0 . Define τ to be the unique real number satisfying

It is clear that τ ∈ [0 , m -1] . Our goal is to prove that

Define β ∗ := min 0 ≤ i ≤ τ β i . For every i = 0 , . . . , m -1 , we have β i ≥ β ∗ if i ≤ τ , and β i ≤ γβ ∗ if i > τ . Therefore, defining s := ⌊ τ ⌋ , we have

̸

Now, we show that β ∗ > 0 . For the sake of contradiction, assume β ∗ = 0 . We already assumed that β 0 > 0 , so β ∗ = β 0 . By the definition of β ∗ , this means that τ > 0 and inequality (23) is strict, leading to the false claim of

Therefore, β ∗ > 0 must hold. Now we know that (24) implies

Treating s as a real-valued variable, the left-hand side is minimized when s = τ -1 / 2 , giving us

The inequality above implies

Taking square root for both sides and solving for τ gives (22).

D Data Structure for Fast Sampling in Seeding

In Appendix C, our Algorithm 1 uses a binary tree data structure S that keeps track of n nonnegative numbers s 1 , . . . , s n and supports several operations. Here, we describe the implementation of this data structure. We assume that n = 2 q for some nonnegative integer q . This is without loss of generality because we can choose n ′ to be the number that satisfy n ≤ n ′ < 2 n and n ′ = 2 q for some q ∈ Z ≥ 0 and consider s 1 , . . . , s n , s n +1 , . . . , s n ′ with s n +1 = · · · = s n ′ = 0 . Under this assumption, the data structure S is a complete binary tree with q +1 layers indexed by 0 , . . . , q . In each layer ζ = 0 , . . . , q there are 2 ζ nodes each corresponding to a set of indices from { 1 , . . . , n } . The root, denoted by v (0) 1 , is the unique node in layer 0 and it corresponds to the entire set V (0) 1 := { 1 , . . . , n } . For ζ = 0 , . . . , q -1 , each node v ( ζ ) j in the ζ -th layer has two children v ( ζ +1) 2 j -1 , v ( ζ +1) 2 j in the ( ζ +1) -th layer corresponding to the sets V ( ζ +1) 2 j -1 , V ( ζ +1) 2 j , respectively, where V ( ζ +1) 2 j -1 is the smaller half of V ( ζ ) j and V ( ζ +1) 2 j is the larger half. Thus,

Each node v ( ζ ) j in the tree stores a sum s ( ζ ) j := ∑ i ∈ V ( ζ ) j s i .

The data structure supports the four types of operations needed in Appendix C as follows:

<latexi sh

1_b

64="k9M8R

P7KTIW

rvC

qgX

D

>A

c

ZLN

E

fd

S

Yp

F

V

2U

0

B

j

u

n

om

3

5+

G

/

H

z

y

J

O

w

Q

<latexi sh

1_b

64="9IumYLoUNV

y

Awzp3

STjP

Z

>

D

c

d

FMf

B

q

XQ

Rg

2

r0

H

C

EW

5

JK

v

+

/O8

n

k

G

7

<latexi sh

1_b

64="jgPfVEZ5S

N9Ow2

c

y

p

T

>A

D

X

L

M

d

I8

v

q

Y

FR

U

r0

B

K

7

u

n

om

3C

+

G

/

k

H

z

W

J

Q

<latexi sh

1_b

64="

OdAHcmzR0

n

MN8r

Y

KE9CU

>

D

3

ZL

wF

fV

B

uq

XQ

g

2S

p

T

W

5

PJ

v

+

/

k

I

j

o

y

7

G

<latexi sh

1_b

64="

Lup7

TjzSJM

IC

+

Y

>A

D

X

c

Z

N

E

d

0g

f

H2

q

V

GU

Q

yo

KR

r3

W

v

m

9

k

P

/

8

n

5

F

w

B

O

<latexi sh

1_b

64="GjTFE

m

OkB+I2VU

03LpKvA

g

>

D

X

c

Z

N

M

fd

8S

q

Y

R

r

9

7

u

n

o

C

5

/

H

z

P

W

y

J

Q

w

<latexi sh

1_b

64="3Nk

EIXO

c

mqGDUP

5

BjRW

QM

>A

Y

ZLd wF

fV

9

u

0

2

K

J

S8

o

y

r

p

n

H

v

C

/

+

T

z

7

g

<latexi sh

1_b

64="/Zq+9

LCTpWF

H

dmr

5uRw

>A

D

X

c

N

EM

0g

f

V

J

UB

G

z

v

y

o

Y

S

P

j

8

n

K

O

I

3

Q

7

<latexi sh

1_b

64="RAyW

Q

kuc

m

N75

q

>

D

X

ZL

M

EI

d

Jpw

Y

gF

V

2

T

U

S0

zr

v

K9

H

+

no

P

B

3

<latexi sh

1_b

64="kw9W/P

HZ

V

Cn

Bf

M

A

Yr

>

D

X

c

LN

E

d

I8Sv

q

p

gFR

2

0

K

3

7

j

G

z

u

J

U

mT

5

+

O

o

Q

y

<latexi sh

1_b

64="z mokqg

82N

R5

W

K7

OnD

/I

>A

Y3

c

B9

u

+

0

U

S

r

J

Xj

P

v

yp

H

C

T

E

GQ

<latexi sh

1_b

64="mq+R8

BUuwD

Y7X5MZ

QcP

>A

L

EI

d

Jp

N

gF

VS0

G

2z

W

v

K9

H

no

C

k

f

/

3

j

y

O

<latexi sh

1_b

64="

YGqumL

pNO

M9H

DZ

Sf

8

>A

X

c

E

d

I

v

gFRV

2U

r0

B

K

j

7

n

o

3C

5+

/

k

z

P

W

y

J

T

Q

w

<latexi sh

1_b

64="qB

ZgAuw

/

vMyPz

E

>

DY3

c

Ld

F

f

9

XQ

R

2S

r0

p

H

C

T

W

5

JK

+

O8

n

N

m

I

j

U

7

<latexi sh

1_b

64="5rp

Nkv97dm3q

PJDo

GA

>

X

c

ZL

M

E

f

I8S

Y

gFRV

2U

0

B

K

j

u

n

C

+

/

H

z

W

y

T

O

Q

w

<latexi sh

1_b

64="

0nyQYU5

v9/

mdO

EF

D

T

>A

X

c

ZLN

M

g+

fpH2

q

VCJ

G

o

KR

r3

W

u

k

S

P

jz

8

w

B

7

I

<latexi sh

1_b

64="G

w3gNvjd2ZSo

A

Y

CK

Qm+

>

D

X

c

L

M

EI

Jp

u

FRV

U

0r

P

7

n

z8

B

k

T

y

q/

9

f

H

O

5

W

<latexi sh

1_b

64="oCV

D

kcR

T3EYd

7

5

f

0

X8

>A

ZL

wFM

B9

uq

2

KW

U

m

J

S

Q

yP

r

pI

n

O

H

v

G

/

j

+

z

N

g

<latexi sh

1_b

64="wS2

/

Yz8op9J

g

XZ

B

>A

c

LN

M

E

f

I

v

r

u

VF

T

K

G

Wy

7

n

m

3C

5+

j

0

H

U

P

R

Q

O

q

<latexi sh

1_b

64="m gucVzZNX

9

2U

MOY

o

>A

D

L

EI

d

Jpw

FR

r0

K

P

j

v

7

n

3

8

B

k

C

T

y

q/G

S

f

+

H

5

W

Q

<latexi sh

1_b

64="WNnp8

E3

7uAVmrd5Z+/

KU

yc

>

DY

L

B9

q0

2

k

J

S

o

Q

P

R

I

O

H

v

C

G

j

T

X

z

g

<latexi sh

1_b

64="Yd op3WM2ZS

VFz

g

vQ

wN

>A

D

X

c

L

E

f

I8

q

R

U

r0

9

B

K

j

7

u

n

m

C

5+

G

/

k

H

P

y

J

T

O

<latexi sh

1_b

64="Z03TAc2E

JM

mn7H

y

5

Xo

>

DY

Ld wF

fV

B9

uq

Q

Rg

S

r

p

C

W

P

K

v

+

z

/O8

N

k

I

j

G

U

<latexi sh

1_b

64="

C

8BuX

Y

3m

P

Tv

Q

>

DZ

Ld wFM

V

9

2

H

j

0

E

kr

K

q

J

p

z

7

g

W

O

5

S

/n oR

+

U

N

y

G

<latexi sh

1_b

64="D

BcokI

9YO3

P

TF

g

U/

>A

Z

Ld

w

MfV

2

H

j

0

E

r

K

qQ

J8

p

z

X

7

W

n

R

+

N

y

m

u

G

C

<latexi sh

1_b

64="0

SY

XD

m

N

fOu

IKngG

c5

>A

Z3

Ld wFM

V

B9

2

TH

j

E

kv rqQ

J8

p

/

7P

W

zoR

+

U

y

C

<latexi sh

1_b

64="MW2wnVk

QE

GLP

A

J/g59c

U

>

DZ3

d

F

o

R

r

B

C

X

I

0qp zf

Tuy

Y

m

K

8

j

N

Ov

S

+

7

H

v

(0)

1

v

(3)

4

Layer 0

Layer 1

Layer 2

Layer 3

s

4

v

(1)

1

v

(1)

2

v

(2)

1

v

(2)

2

v

(2)

3

v

(2)

4

v

(3)

1

v

(3)

2

v

(3)

3

v

(3)

5

v

(3)

6

v

(3)

7

v

(3)

8

s

1

s

2

s

3

s

5

s

6

s

7

s

8

Initialize ( a ) . Recursively run Initialize on the first half ( a 1 , . . . , a n/ 2 ) and the second half ( a n/ 2+1 , . . . , a n ) to obtain the two subtrees rooted at v (1) 1 and v (1) 2 . Then add a root v (0) 1 that stores s (0) 1 ← s (1) 1 + s (1) 2 .

Sum ( S ) . Simply output s (0) 1 .

Find ( S, r ) . If r < s (1) 1 , recursively call Find on the left subtree rooted at v (1) 1 . Otherwise, recursively call Find on the right subtree rooted at v (1) 2 with r replaced by r -s (1) 1 . Once we reach a leaf v ( q ) ℓ , return ℓ .

Update ( S, a, i 1 , i 2 ) . If i 2 ≤ n/ 2 , recursively call Update on the left subtree rooted at v (1) 1 . If i 1 > n/ 2 , recursively call Update on the right subtree rooted at v (1) 2 . Otherwise, we have i 1 ≤ n/ 2 < i 2 and we call Update on the left subtree with indices i 1 , n/ 2 and call Update on the right subtree with indices n/ 2 + 1 , i 2 . In all cases, we update s (0) 1 ← s (1) 1 + s (1) 2 as the final step. The running time is proportional to the number of nodes we update. We need to update s ( ζ ) j stored at v ( ζ ) j only if V ( ζ ) j ∩{ i 1 , . . . , i 2 } ̸ = ∅ . For each ζ , the number of such j is at most ( i 2 -i 1 +1) / 2 q -ζ +2 . Summing up over ζ = 0 , . . . , q , the total number of nodes we need to update is O (( i 2 -i 1 +1) + q ) = O (( i 2 -i 1 +1) + log n ) .

E Additional Experiments and Data

In this section, we provide the tables for speedups and absolute running times of the experiments performed in Section 3. Additionally, we provide some experimental data for the algorithmic approach described in Theorem 2.3.

E.1 Improved Approximation Ratio

Experimental Setup. This experiment aims to compare the algorithmic approach outlined in Theorem 2.3 to the direct use of PRONE as a clustering algorithm as was done in Section 3.2. For this, we use PRONE as the approximation algorithm for sensitivity sampling and then cluster the coreset using a weighted variant of the k -means++ algorithm. This approach is termed PRONE (boosted) in the rest of this section. This pipeline requires as parameters the number of centers k and a hyperparameter α indicating the size of the coreset produced by sensitivity sampling. We aim to compare the clustering cost (see Definition A.1) and running time of our approach to that of k -means++.

We run both algorithms on the datasets described in Section 3 and choose k ∈ { 10 , 25 , 50 , 100 , 250 , 500 , 1000 , 2500 , 5000 } and α ∈ { 0 . 001 , 0 . 01 , 0 . 1 } . Each algorithm is run 5 times.

Results on Improved Approximation Ratio Costs. Figure 6 shows the costs of centers produced by this algorithm relative to the cost of centers produced by k -means++. It also contains the ( k, 2) -

10 1

10 2

10 3

k

1.0

1.5

2.0

2.5

3.0

cost

Dataset = KDD

10 1

10 2

10 3

k

1.00

1.05

1.10

1.15

1.20

Dataset = Song

10 1

10 2

10 3

k

1.00

1.25

1.50

1.75

2.00

Dataset = Census

10 1

10 2

10 3

k

2

4

6

8

10

Algorithm

PRONE (boosted,

= 0.001)

PRONE (boosted,

= 0.01)

PRONE (boosted,

= 0.1)

k-means++

PRONE, no pipeline

clustering costs of PRONE relative to k -means++. We can see that on all real datasets, PRONE (boosted) produces solutions of the same or better quality than k -means++, as long as αn ≪ k . This shows that although PRONE by itself produces centers of worse quality, the PRONE (boosted) variant produces centers of the same quality as vanilla k -means++. When αn ≈ k , we observe an uptick in cost before the end of the lines corresponding to α ∈ { 0 . 1 , 0 . 01 } in the plots for KDD, Song, and Gaussian. The boosted approach outperforms PRONE , which is usually worse by a constant factor compared to the other algorithms, and it helps to reduce significantly the amount of variance in the quality of solutions. On the Gaussian dataset, we observed a failure to sample a point from the central cluster, which explains the spike at k = 2500 for the line corresponding to α = 0 . 1 .

Running time. Table 2 shows the speedup of the boosted approach versus using plain k -means++, for the time taken to compute the centers. The running time of our algorithms now scales with k , but at a slower rate compared to k -means++, as we have to run it on a much smaller dataset. Once again, we observe significant speedups, especially as k grows large.

As expected, the speedup depends on the choice of the hyperparameter α . We observe diminishing returns for larger α as k scales, with the speedup remaining mostly constant for k ≥ 100 across datasets, except for Gaussian. This is due to the algorithm's running time being dominated by the time it takes to execute k -means++ on the coreset, which has O ( ndk ) asymptotic running time. The speedups we can achieve using this method are significant, up to 118x faster than k -means++. We expect that on massive datasets, even greater speedups can be achieved.

Interestingly, the speedup can come very close to or even out scale α , as observed on the KDD and Song datasets. The final stage of the boosted approach executes k -means++ on a coreset of size αn , so the running time of this step should be O ( αndk ) . The observed additional speedup may be due to better cache and memory utilization in the k -means++ step of the algorithm.

E.2 Additional Data

In this section, we provide the running time data for the full range of parameters for the experiments performed in Section 3.2. Table 3 shows the speedups over k -means++, analogous to the right-handside table in Table 1. Additionally, Table 4 provides absolute running times in milliseconds.

Centers

10

25

50

100

250

500

1000

2500

5000

Dataset

Algorithm

Census

PRONE

525.1

±

15.3

534.3

±

23.4

534.6

±

20.8

530.1

±

15.4

538.9

±

15.2

533.9

±

11.7

547.4

±

23.8

548.2

±

13.5

563.7

±

8.7

PRONE

(variance)

1750.6

±

71.7

1778.9

±

38.1

1780.3

±

23.9

1752.7

±

56.9

1768.6

±

28.2

1757.6

±

37.8

1778.9

±

27.5

1761.0

±

73.3

1739.6

±

10.9

PRONE

(covariance)

3769.0

±

152.2

3882.5

±

167.8

3743.9

±

316.7

3714.2

±

378.3

3766.4

±

220.2

3662.9

±

122.6

3708.7

±

372.4

3525.6

±

609.9

3182.2

±

288.7

k

-means++

812.5

±

20.9

2040.6

±

25.6

3994.3

±

25.3

7992.9

±

91.5

19519.0

±

161.5

39058.4

±

337.8

77823.6

±

527.5

192913.1

±

3657.0

373488.2

±

221.9

Song

PRONE

104.4

±

5.5

101.8

±

5.2

106.5

±

4.1

109.9

±

6.8

113.7

±

8.8

109.0

±

6.4

108.7

±

3.9

117.2

±

6.3

120.8

±

12.9

PRONE

(variance)

443.3

±

5.0

448.3

±

8.0

450.1

±

9.9

468.0

±

31.6

458.8

±

15.1

450.7

±

10.0

462.6

±

11.6

468.3

±

14.0

465.7

±

14.9

PRONE

(covariance)

1164.4

±

162.1

1266.5

±

136.6

1332.4

±

116.3

1381.8

±

108.4

1322.2

±

173.4

1263.0

±

175.6

1324.2

±

108.4

1296.8

±

177.8

1228.0

±

175.2

k

-means++

207.4

±

4.0

513.7

±

9.8

1036.6

±

17.0

2103.9

±

44.2

5245.9

±

143.6

10412.5

±

119.3

20464.1

±

196.9

51917.4

±

554.3

101149.0

±

1962.7

KDD

PRONE

31.2

±

5.2

32.2

±

7.4

34.1

±

1.8

28.9

±

4.9

29.2

±

9.1

34.6

±

6.3

30.8

±

8.6

28.8

±

5.2

33.4

±

6.6

PRONE

(variance)

72.0

±

1.4

71.8

±

1.4

76.5

±

7.2

75.1

±

7.1

71.8

±

1.2

73.9

±

2.1

77.0

±

4.6

74.7

±

3.2

77.7

±

5.9

PRONE

(covariance)

169.3

±

2.9

173.5

±

5.5

180.3

±

13.1

184.1

±

21.2

170.8

±

8.9

182.6

±

19.0

188.7

±

16.9

204.1

±

40.3

189.1

±

11.6

k

-means++

48.0

±

0.3

119.3

±

3.7

233.9

±

3.2

470.4

±

11.6

1153.5

±

2.1

2363.3

±

108.6

4878.1

±

171.1

11928.9

±

234.3

24285.9

±

243.8

PRONE

30.0

±

2.3

29.1

±

2.2

30.6

±

3.3

29.1

±

1.4

29.1

±

1.0

28.7

±

0.2

29.3

±

0.4

30.2

±

0.6

31.5

±

0.6

PRONE

(variance)

29.8

±

3.0

28.9

±

0.3

29.5

±

1.0

29.6

±

0.6

29.6

±

0.3

29.6

±

0.4

30.2

±

0.3

30.8

±

0.2

32.1

±

0.6

PRONE

(covariance)

31.8

±

2.2

31.5

±

2.1

34.6

±

6.6

30.8

±

0.3

32.7

±

2.8

32.6

±

2.4

32.8

±

2.3

32.2

±

0.5

34.3

±

0.4

30.0

0.4

59.5

2.2

111.5

2.8

268.6

1.5

525.3

1.9

1043.3

1.4

2575.3

5.0

5229.6

135.0

Peer reviews:
comments: SUMMARY:
$k$-means clustering is a classical and fundamental method in machine learning. This paper proposes a very simple method that projects the $n$ data points onto a one-dimensional space by generating a random Gaussian vector. Moreover, the one-dimensional projections can be used for coreset construction and the seeding in $k$-means++. The experimental results have proven the efficiency of the proposed algorithm.

STRENGTHS:
(1) The proposed method is very simple and easy to understand.  
(2) The idea of projecting the $n$ input points onto one-dimensional space is kind of novel.  
(3) This paper provides strict proofs for the running time and approximation ratio.  
(4) Experimental results show the efficiency for coreset construction and $k$-means++ implementation.

WEAKNESSES:
(1) The essence of the proposed method is dimension reduction. To be specific, the pairwise distances of the $n$ data points are roughly preserved by projecting onto one-dimensional space. The projected distances are used to $k$-means++ seeding and an approximate clustering. By the sorted $n$ scalars, the seeding can be implemented in time $O(n \log{n})$ independent of $k$ but at the cost of $\widetilde{O}(k^4)$ approximation ratio. For me, I think the technique is straightforward and the idea's novelty is a little limited.  
(2) The colors of the curves in Figure 2 and Figure 3 are not easy to distinguish.

QUESTIONS:
The approximation ratio $\widetilde{O}(k^4)$ can be improved to $O(\log{k})$ with running time $O(\text{nnz}(X) + n\log{n}) + \text{poly}(kd) \log{n}$. However, why does not this result show in the experiments?

LIMITATIONS:
See the Weaknesses part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new clustering algorithm that seeks to have the accuracy of k-means++ while avoiding the time complexity of O(ndk). In particular, the paper focuses on settings where k is large and seeks a O(nd+nlogn) time complexity. This is a practically relevant problem.\n\nThe main algorithm proposed in the paper is a simple algorithm based on projecting all data points onto a random one-dimensional subspace and performing the k-means++ seeding procedure on the dimensionality-reduced data. Interestingly, it is shown that this can be done in time O(nd+nlog(n)) and it has a poly(k) approximation guarantee with respect to the optimal k-means clustering. While the poly(k) guarantee is worse than the O(log k) guarantee of k-means++, a log(k) approximation guarantee can be obtained if we use the proposed algorithm to build a coreset of size poly(kd)*log(n), and then use k-means++ on the resulting coreset.\n\nWhile the techniques in [26] (based on approximate nearest neighbor search) achieve similar goals to those of the present paper (a fast alternative to k-means++ with a complexity that doesn’t depend on k), the algorithm in the present paper is attractive for its surprising simplicity.\n\nSTRENGTHS:\nThe proposed algorithm is natural and intuitive, but the theoretical analysis is quite involved.\n\nThe fact that a one-dimensional projection can be shown to provide an approximation guarantee (although a poly(k) one) is interesting and nontrivial, since a Johnson-Lindenstrauss type approach typically requires a projection onto log(n) dimensions. \n\nAfter the one-dimensional projection, a standard implementation of k-means++ seeding does not avoid the O(nk) dependence. As such, the paper proposes a careful data structure to perform k-means seeding on a one-dimensional dataset. The proof that this approach runs in time O(n logn) is technically sophisticated and could be of independent interest. The potential function must be carefully defined, and its use to show that each point is only updated O(log n) times is elegant.\n\nThe paper is well written and clear, and the results should be of interest to people working on the theory of clustering algorithms and to people looking for practical approaches for performing k-means clustering on large datasets with large k.\n\nWEAKNESSES:\nThe empirical results in the paper are generally positive, but perhaps a bit underwhelming, as the proposed approach doesn’t seem to have a strong advantage over the Lightweight coreset approach from [10]. In particular, the proposed algorithm is slower than the Lightweight approach, and the accuracy of the two approaches is similar (except on the Gaussian dataset case, which was designed to adversarially exploit a weakness of Lightweight).\n\nI found it a little confusing to distinguish between the different variants of the algorithm, and it may be helpful to give them clear names (rather than saying “Ours”). In particular, in Section 1, there is a brief discussion about how the approximation can be reduced from poly(k) to O(log k), by incurring an additional poly(nk)*log(n) time. Are the results in 3 and Table 1 for the variant with the O(log k) approximation? Also, in section 1, it says that this improvement is further discussed in the supplementary material, but it wasn’t very clear where. Is this just referring to the general discussion on coresets in Appendix A? It may be useful to have an explicit theorem about the algorithm that combines the proposed algorithm with k-means++ on a coreset (and that algorithm should have a name). I apologize if I missed this somewhere.\n\nIt may be useful to discuss some of the literature on fast clustering algorithms beyond k-means. For instance, there are “fast” implementations of PAM for k-medoids (such as FastPAM, Clarans and BanditPAM) and of dbscan (such as dbscan++). It may be the case that all those incur the O(ndk) time that this paper tries to avoid, but a mention of that would be useful to the community (specially as it relates to the claim that there are no other clustering algorithms with O(n*log(n)) complexity).\n\nSome other minor comments/typos:\n\n- line 144: k-measn -> k-means\n- line 278: “compared a” -> “compared to a”\n- line 279: “entre dataset” -> “entire dataset”\n- line 344: “running independent” -> “running time independent”\n\nQUESTIONS:\nA natural way to perform a one-dimensional reduction for clustering would be to take the first principal component of the dataset and then perform k-means++ on it. Would such a projection be expected to perform better than the random Gaussian projection used? I realize that computing the first PC would probably be slow, but it may be worth discussing. This may be connected with spectral clustering approaches as well, and with the “biased” versions of the proposed algorithm. \n\nA natural question that the proposed algorithm raises is whether one can use a constant number of random projections to improve the poly(k) approximation factor (maybe just reducing the polynomial degree). Could you comment on that?\n\nLIMITATIONS:\nThe limitations of the proposed approach are discussed in a fair manner.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submitted paper focuses on the problem of unsupervised clustering and aims at improving the $\\Omega(ndk)$ runtime of the ```kmeans++``` algorithm in the clustering of $n$ $d$-dimensional datapoints in $k$ clusters. The proposed algorithm relies on the intuition of running ```kmeans++``` on a one-dimensional projection of the dataset along a random direction $\\boldsymbol v$: the authors prove that such strategy leads to a clustering in $O(nd+n\\ln n)$ steps with an approximation ratio scaling as $\\tilde O(k^4)$ (to be compared with the $O(\\ln k)$ of ```kmeans++```). Numerical experiments compare the performance of the algorithm both with ```kmeans++``` and with other strategies for the production of coresets.\n\nSTRENGTHS:\nThe proposed algorithm is very simple to implement (reminiscent of sliced optimal transportation strategies, in which a low-dimensional projection is adopted to solve a high-dimensional problem with a subsequent polynomial gain in speed). The approach proposed by the authors is indeed compatible with a remarkable speed-up and, given its price in quality, suitable for adoption in a pipeline. One of the strengths of the paper is that the authors can provide theoretical guarantees on the performance obtained by this simple trick.\n\nWEAKNESSES:\nThe proposed algorithm pays the price of better performances producing clusterings of lower quality, as the authors themselves point out. This drawback can be mitigated by adopting the algorithm for the construction of coresets within a pipeline (as discussed in the paper). In the case in which the algorithm is adopted as a stand-alone solution, the (possibly deteriorating) effect of structure in the dataset is only partially investigated and addressed.\n\nQUESTIONS:\n* If I understand correctly, the algorithm relies on a single low-dimensional projection along a random vector $\\boldsymbol v\\in\\mathbb R^d$. I was wondering if the algorithm would benefit from the sampling of $m=O(1)$ vectors, or if instead, such strategy would lead to irrelevant benefits.\n* It is not very clear in which units the elements of Table 1 are given (from what I understand the table reports speed-up ratios). Please clarify the caption. Also, do the authors have an estimate of the error on the reported values?\n* Do the authors have an intuition about the fact that the biased and covariance version of the algorithm suffers a remarkable slowdown with respect to the unbiased version?\n* In section 3.2 it is commented that, when working on the Gaussian dataset, the algorithm *did not pick up one of the points close to the origin* so its output is associated with a particularly high cost. Could the author clarify what they mean with this sentence?\n\nMinor typos\n* Line 45: *in approximate* in place of *an approximate*.\n* Line 244: *measn* in place of *means*.\n* What does *with a few approaches in the literature*  refer to in line 147?\n* Line 293: *Senstivity* in place of *Sensitivity*.\n\nLIMITATIONS:\nLimitations of the contribution have been addressed, see also the *Weakness* section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper presents a new technique for clustering data such that the running time of the method is independent of the number of clusters, and the approximation error is polynomial only in the number of clusters and not the number of data points. 

The paper experimentally tests the method and finds that it is a good balance of speed and quality of the result.

STRENGTHS:
There are many strengths.


1) The runtime being independent of $k$ is a significant speed up. Further, the constants seem to be better as well. Since the method is not iterative and only makes a single pass over the data 
2) The experimental results on corsets are compelling. 
3) The presentation of the method is easy to follow.

WEAKNESSES:
The approximation of ratio of $O(k^4)$ is significantly worse than $O(\log k)$. I was hoping this was only theoretical and that we would see better results experimentally, but the experimental results are not clear. 

Specifically for Figure 3, I have no idea what the difference between the bottom and the top row is. This is partly due to the fact that the paper never actually writes down formally the problem or the metrics used. From my understanding, we have $n$ data points $x_1, \ldots, x_n$ in high dimensional space. We want to find $k$ clusters $C_1, \ldots, C_k$ where each $C_i$ is a subset of the $n$ data points such that if $c_i$ is the center of the mass of those points, then 

$$ \sum_{i=1}^k \sum_{j \in C_i} \|c_i - x_j\|^2 $$

is minimized. This is what I interpret to be the cost. Hence it is very unclear to me what Figure 3 is showing.

QUESTIONS:
What is the time complexity for the third step?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a randomized clustering algorithm that provably runs in expected time O(nnz(X) + n log n) for arbitrary k. In theoretical analysis, the proposed algorithm achieves approximation ratio O(k^4 ) on any input dataset for the k-means objective. In the experiments, the quality of the clusters found by the proposed algorithm is usually much better than this worst-case bound. They use the algorithm for k-means clustering and coreset construction. This work shows that the approximation ratio achieved after a random one-dimensional projection can be lifted to the original points and that k-means++ seeding can be implemented in expected time O(n log n) in one dimension.\n\nSTRENGTHS:\nThis paper has the following contributions: \nIt designs a simple, practical algorithm for k-means that runs in time roughly O(nd), independent of k, and produces high-quality clusters. In theoretical analysis, they give a randomized clustering algorithm with provable guarantees on its running time and approximation ratio without making any assumptions about the data. In experiments, they run two types of experiments, highlighting various aspects of the algorithm: Coreset Construction Comparison and Direct k-means++ comparison.\n\nWEAKNESSES:\n1.Insufficient research on related work. \nThere are many methods for accelerating clustering. This paper did not conduct comprehensive research. In the most relevant random projection clustering, this work is not the first random projection clustering work, such as paper [1]. It is not cited in this paper. The superiority of this work cannot be verified. Please compare it from the theoretical analysis, experimental results, algorithm complexity, etc.\n\n[1]Liu W, Shen X, Tsang I. Sparse Embedded $ k $-Means Clustering[J]. Advances in neural information processing systems, 2017.\n\n2.The organization and presentation of this paper can be further improved. The second half of the abstract is logically chaotic.\n\n3.The experiment is insufficient to verify the superiority of this work. The experimental results show that the performance of the proposed algorithm is not excellent, such as in Figure 2,3.\n\nQUESTIONS:\nSee “Weaknesses”.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}