Title: Multi-Swap k-Means++
Venue: NeurIPS 2023
Accepted: yes
Keywords: Clustering

k-means

approximation algorithms
Abstract: The $k$-means++ algorithm of Arthur and Vassilvitskii (SODA 2007) is often the practitioners' choice algorithm for optimizing the popular $k$-means clustering objective and is known to give an $O(\log k)$-approximation in expectation. To obtain higher quality solutions, Lattanzi and Sohler (ICML 2019) proposed augmenting $k$-means++ with $O(k \log \log k)$ local-search steps obtained through the $k$-means++ sampling distribution to yield a $c$-approximation to the $k$-means clustering problem, where $c$ is a large absolute constant. Here we generalize and extend their local-search algorithm by considering larger and more sophisticated local-search neighborhoods hence allowing to  swap multiple centers at the same time. Our algorithm achieves a $9 + \varepsilon$ approximation ratio, which is the best possible for local search. Importantly we show that our algorithm is practical, namely easy to implement and fast enough to run on a variety of classic datasets, and outputs solutions of better cost.

Multi-Swap k -Means++

Abstract

The k -means++ algorithm of Arthur and Vassilvitskii (SODA 2007) is often the practitioners' choice algorithm for optimizing the popular k -means clustering objective and is known to give an O (log k ) -approximation in expectation. To obtain higher quality solutions, Lattanzi and Sohler (ICML 2019) proposed augmenting k -means++ with O ( k log log k ) local search steps obtained through the k -means++ sampling distribution to yield a c -approximation to the k -means clustering problem, where c is a large absolute constant. Here we generalize and extend their local search algorithm by considering larger and more sophisticated local search neighborhoods hence allowing to swap multiple centers at the same time. Our algorithm achieves a 9 + ε approximation ratio, which is the best possible for local search. Importantly we show that our approach yields substantial practical improvements, we show significant quality improvements over the approach of Lattanzi and Sohler (ICML 2019) on several datasets.

1 Introduction

Clustering is a central problem in unsupervised learning. In clustering one is interested in grouping together 'similar' object and separate 'dissimilar' one. Thanks to its popularity many notions of clustering have been proposed overtime. In this paper, we focus on metric clustering and on one of the most studied problem in the area: the Euclidean k -means problem.

In the Euclidean k -means problem one is given in input a set of points P in R d . The goal of the problem is to find a set of k centers so that the sum of the square distances to the centers is minimized. More formally, we are interested in finding a set C of k points in R d such that ∑ p ∈ P min c ∈ C || p -c || 2 , where with || p -c || we denote the Euclidean distance between p and c .

The k -means problem has a long history, in statistics and operations research. For Euclidean k -means with running time polynomial in both n, k and d , a 5 . 912 -approximation was recently shown in Cohen-Addad et al. [2022a], improving upon Kanungo et al. [2004], Ahmadian et al. [2019], Grandoni et al. [2022] by leveraging the properties of the Euclidean metric. In terms of lower bounds, the first to show that the high-dimensional k -means problems were APX-hard were Guruswami and Indyk [2003], and later Awasthi et al. [2015] showed that the APX-hardness holds even if the centers can be placed arbitrarily in R d . The inapproximability bound was later slightly improved by Lee et al. [2017] until the recent best known bounds of Cohen-Addad and Karthik C. S. [2019], Cohen-Addad et al. [2022d, 2021] that showed that it is NP-hard to achieve a better than 1.06-approximation and hard to approximate it better than 1.36 assuming a stronger conjecture. From a more practical point of view, Arthur and Vassilvitskii [2009] showed that the widely-used popular heuristic of Lloyd Lloyd

[1957] can lead to solutions with arbitrarily bad approximation guarantees, but can be improved by a simple seeding strategy, called k -means++, so as to guarantee that the output is within an O (log k ) factor of the optimum Arthur and Vassilvitskii [2007].

Thanks to its simplicity k -means++ is widely adopted in practice. In an effort to improve its performances Lattanzi and Sohler [2019], Choo et al. [2020] combine k -means++ and local search to efficiently obtain a constant approximation algorithm with good practical performance. These two studies show that one can use the k -means++ distribution in combination with a local search algorithm to get the best of both worlds: a practical algorithm with constant approximation guarantees.

However, the constant obtained in Lattanzi and Sohler [2019], Choo et al. [2020] is very large (several thousands in theory) and the question as whether one could obtain a practical algorithm that would efficiently match the 9+ ε -approximation obtained by the n O ( d/ϵ ) algorithm of Kanungo et al. [2004] has remained open. Bridging the gap between the theoretical approach of Kanungo et al. [2004] and k -means++ has thus been a long standing goal.

Our Contributions. We make significant progress on the above line of work.

We adapt techniques from the analysis of Kanungo et al. [2004] to obtain a tighter analysis of the algorithm in Lattanzi and Sohler [2019]. In particular in Corollary 4, we show that their algorithm achieves an approximation of ratio of ≈ 26 . 64 .

We extend this approach to multi-swaps, where we allow swapping more than one center at each iteration of local search, improving significantly the approximation to ≈ 10 . 48 in time O ( nd · poly ( k )) .

Leveraging ideas from Cohen-Addad et al. [2021], we design a better local search swap that improves the approximation further to 9 + ε (see Theorem 12). This new algorithm matches the 9+ ε -approximation achieved by the local search algorithm in Kanungo et al. [2004], but it is significantly more efficient. Notice that 9 is the best approximation achievable through local search algorithms, as proved in Kanungo et al. [2004].

We provide experiments where we compare against k -means++ and Lattanzi and Sohler [2019]. We study a variant of our algorithm that performs very competitively with our theoretically sound algorithm. The variant is very efficient and still outperforms previous work in terms of solution quality, even after the standard postprocessing using Lloyd.

Additional Related Work. We start by reviewing the approach of Kanungo et al. [2004] and a possible adaptation to our setting. The bound of 9 + ε on the approximation guarantee shown by Kanungo et al. [2004] is for the following algorithm: Given a set S of k centers, if there is a set S + of at most 2 /ε points in R d together with a set S -of | S + | points in S such that S \ S -∪ S + achieves a better k -means cost than S , then set S := S \ S -∪ S + and repeat until convergence. The main drawback of the algorithm is that it asks whether there exists a set S + of points in R d that could be swapped with elements of S to improve the cost. Identifying such a set, even of constant size, is already non-trivial. The best way of doing so is through the following path: First compute a coreset using the state-of-the-art coreset construction of Cohen-Addad et al. [2022b] and apply the dimensionality reduction of Becchetti et al. [2019], Makarychev et al. [2019], hence obtaining a set of ˜ O ( k/ε 4 ) points in dimension O (log k/ε 2 ) . Then, compute grids using the discretization framework of Matousek [2000] to identify a set of ε -O ( d ) ∼ k O ( ε -2 log(1 /ε )) grid points that contains nearly-optimum centers. Now, run the local search algorithm where the sets S + are chosen from the grid points by brute-force enumeration over all possible subsets of grid points of size at most, say s . The running time of the whole algorithm with swaps of magnitude s , i.e.: | S + | ≤ s , hence becomes k O ( s · ε -2 log(1 /ε )) for an approximation of (1 + ε )(9 + 2 /s ) , meaning a dependency in k of k O ( ε -3 log(1 /ε )) to achieve a 9 + ε -approximation. Our results improves upon this approach in two ways: (1) it improves over the above theoretical bound and (2) does so through an efficient and implementable, i.e.: practical, algorithm.

Recently, Grunau et al. [2023] looked at how much applying a greedy rule on top of the k -means++ heuristic improves its performance. The heuristic is that at each step, the algorithm samples ℓ centers and only keeps the one that gives the best improvement in cost. Interestingly the authors prove that from a theoretical standpoint this heuristic does not improve the quality of the output. Local search algorithms for k -median and k -means have also been studied by Gupta and Tangwongsan [2008] who drastically simplified the analysis of Arya et al. [2004]. Cohen-Addad and Schwiegelshohn [2017]

demonstrated the power of local search for stable instances. Friggstad et al. [2019], Cohen-Addad et al. [2019] showed that local search yields a PTAS for Euclidean inputs of bounded dimension (and doubling metrics) and minor-free metrics. Cohen-Addad [2018] showed how to speed up the local search algorithm using kd -trees (i.e.: for low dimensional inputs).

For fixed k , there are several known approximation schemes, typically using small coresets Becchetti et al. [2019], Feldman and Langberg [2011], Kumar et al. [2010]. The state-of-the-art approaches are due to Bhattacharya et al. [2020], Jaiswal et al. [2014]. The best known coreset construction remains Cohen-Addad et al. [2022c,b].

If the constraint on the number of output centers is relaxed, then we talk about bicriteria approximations and k -means has been largely studied Bandyapadhyay and Varadarajan [2016], Charikar and Guha [2005], Cohen-Addad and Mathieu [2015], Korupolu et al. [2000], Makarychev et al. [2016].

2 Preliminaries

Notation. We denote with P ⊆ R d the set of input points and let n = | P | . Given a point set Q ⊆ P we use µ ( Q ) to denote the mean of points in Q . Given a point p ∈ P and a set of centers A we denote with A [ p ] the closest center in A to p (ties are broken arbitrarily). We denote with C the set of centers currently found by our algorithm and with O ∗ an optimal set of centers. Therefore, given p ∈ P , we denote with C [ p ] and O ∗ [ p ] its closest ALG-center and OPT-center respectively. We denote by cost ( Q,A ) the cost of points in Q ⊆ P w.r.t. the centers in A , namely

Weuse ALG and OPT as a shorthand for cost ( P, C ) and cost ( P, O ∗ ) respectively. When we sample points proportionally to their current cost (namely, sample q with probability cost ( q, C ) / cost ( P, C ) ) we call this the D 2 distribution. When using O ε ( · ) and Ω ε ( · ) we mean that ε is considered constant. We use ˜ O ( f ) to hide polylogarithmic factors in f . The following lemma is folklore.

Lemma 1. Given a point set Q ⊆ P and a point p ∈ P we have

Let O ∗ i be an optimal cluster, we define the radius of O ∗ i as ρ i such that ρ 2 i · | O ∗ i | = cost ( O ∗ i , o i ) , where o i = µ ( O ∗ i ) . We define the δ -core of the optimal cluster O ∗ i as the set of points p ∈ O ∗ i that lie in a ball of radius (1 + δ ) ρ i centered in o i . In symbols, core ( O ∗ i ) = P ∩ B ( o i , (1 + δ ) ρ i ) . Throughout the paper, δ is always a small constant fixed upfront, hence we omit it.

Lemma 2. Let O ∗ i be an optimal cluster and sample q ∈ O ∗ i according to the D 2 -distribution restricted to O ∗ i . If cost ( O ∗ i , C ) > (2 + 3 δ ) · cost ( O ∗ i , o i ) then Pr[ q ∈ core ( O ∗ i )] = Ω δ (1) .

Proof. Define α := cost ( O ∗ i , C ) / cost ( O ∗ i , o i ) > 2 + 3 δ . Thanks to Lemma 1, for each c ∈ C we have || c -o i || 2 ≥ ( α -1) ρ 2 i . Therefore, for each y ∈ core ( O ∗ i ) and every c ∈ C we have

Moreover, by a Markov's inequality argument we have | O ∗ i \ core ( O ∗ i ) | ≤ 1 1+ δ · | O ∗ i | and thus | core ( O ∗ i ) | ≥ Ω δ ( | O ∗ i | ) . Combining everything we get

3 Multi-Swap k -Means++

The single-swap local search (SSLS) k -means++ algorithm in Lattanzi and Sohler [2019] works as follows. First, k centers are sampled using k -means++ (namely, they are sampled one by one according to the D 2 distribution, updated for every new center). Then, O ( k log log k ) steps of local search follow. In each local search step a point q ∈ P is D 2 -sampled, then let c be the center among

the current centers C such that cost ( P, ( C \ { c } ) ∪ { q } ) is minimum. If cost ( P, ( C \ { c } ) ∪ { q } ) < cost ( P, C ) then we swap c and q , or more formally we set C ← ( C \ { c } ) ∪ { q } .

We extend the SSLS so that we allow to swap multiple centers simultaneously and call this algorithm multi-swap local search (MSLS) k -means++. Swapping multiple centers at the same time achieves a lower approximation ratio, in exchange for a higher time complexity. In this section, we present and analyse the p -swap local search (LS) algorithm for a generic number of p centers swapped at each step. For any constant δ > 0 , we obtain an approximation ratio ALG / OPT = η 2 + δ where

The Algorithm. First, we initialize our set of centers using k -means++. Then, we run O ( ndk p -1 ) local search steps, where a local search step works as follows. We D 2 -sample a set In = { q 1 . . . q p } of points from P (without updating costs). Then, we iterate over all possible sets Out = { c 1 . . . c p } of p distinct elements in C ∪ In and select the set Out such that performing the swap ( In, Out ) maximally improves the cost 2 . If this choice of Out improves the cost, then we perform the swap ( In, Out ) , else we do not perform any swap for this step.

Theorem 3. For any δ > 0 , the p -swap local search algorithm above runs in ˜ O ( ndk 2 p ) time and, with constant probability, finds an ( η 2 + δ ) -approximation of k -means, where η satisfies Equation (1) .

Notice that the SSLS algorithm of Lattanzi and Sohler [2019] is exactly the p -swap LS algorithm above for p = 1 .

Corollary 4. The single-swap local search in Lattanzi and Sohler [2019], Choo et al. [2020] achieves an approximation ratio < 26 . 64 .

Corollary 5. For p = O (1) large enough, multi-swap local search achieves an approximation ratio < 10 . 48 in time O ( nd · poly ( k )) .

3.1 Analysis of Multi-Swap k -means++

In this section we prove Theorem 3. Our main stepping stone is the following lemma.

Lemma 6. Let ALG denote the cost at some point in the execution of MSLS. As long as ALG / OPT > η 2 + δ , a local search step improves the cost by a factor 1 -Ω(1 /k ) with probability Ω(1 /k p -1 ) .

Proof of Theorem 3. First, we show that O ( k p log log k ) local steps suffice to obtain the desired approximation ratio, with constant probability. Notice that a local search step can only improve the cost function, so it is sufficient to show that the approximation ratio is achieved at some point in time. We initialize our centers using k -means++, which gives a O (log k ) -approximation in expectation. Thus, using Markov's inequality the approximation guarantee O (log k ) holds with arbitrary high constant probability. We say that a local-search step is successful if it improves the cost by a factor of at least 1 -Ω(1 /k ) . Thanks to Lemma 6, we know that unless the algorithm has already achieved the desired approximation ratio then a local-search step is successful with probability Ω(1 /k p -1 ) . To go from O (log k ) to η 2 + δ we need O ( k log log k ) successful local search steps. Standard concentration bounds on the value of a Negative Binomial random variable show that, with high probability, the number of trial to obtain O ( k log log k ) successful local-search steps is O ( k p log log k ) . Therefore, after O ( k p log log k ) local-search steps we obtain an approximation ratio of η 2 + δ .

To prove the running time bound it is sufficient to show that a local search step can be performed in time ˜ O ( ndk p -1 ) . This is possible if we maintain, for each point x ∈ P , a dynamic sorted dictionary 3 storing the pairs ( cost ( x, c i ) , c i ) for each c i ∈ C . Then we can combine the exhaustive search over all possible sizep subsets of C ∪ In and the computation of the new cost function using time O ( ndk p -1 log k ) . To do so, we iterate over all possible size-( p -1) subsets Z of C ∪ In and update all costs as if these centers were removed, then for each point x ∈ P we compute how much its cost increases if we remove its closest center c x in ( C ∪ In ) \ Z and charge that amount to c x . In the end, we consider Out = Z ∪ { c } where c is the cheapest-to-remove center found in this way.

The rest of this section is devoted to proving Lemma 6. For convenience, we prove that Lemma 6 holds whenever ALG / OPT > η 2 + O ( δ ) , which is wlog by rescaling δ . Recall that we now focus on

̸

a given step of the algorithm, and when we say current cost, current centers and current clusters we refer to the state of these objects at the end of the last local-search step before the current one. Let O ∗ 1 . . . O ∗ k be an optimal clustering of P and let O ∗ = { o i = µ ( O ∗ i ) | for i = 1 . . . k } be the set of optimal centers of these clusters. We denote with C 1 . . . C k the current set of clusters at that stage of the local search and with C = { c 1 . . . c k } the set of their respective current centers.

We say that c i captures o j if c i is the closest current center to o j , namely c i = C [ o j ] . We say that c i is busy if it captures more than p optimal centers, and we say it is lonely if it captures no optimal center. Let ˜ O = { o i | cost ( O ∗ i , C ) > δ · ALG /k } and ˜ C = C \ {C [ o i ] | o i ∈ O ∗ \ ˜ O} . For ease of notation, we simply assume that ˜ O = { o 1 . . . o h } and ˜ C = { c 1 . . . c h ′ } . Notice that h ′ > h .

Weighted ideal multi-swaps. Given In ⊆ P and Out ⊆ ˜ C of the same size we say that the swap ( In, Out ) is an ideal swap if In ⊆ ˜ O . We now build a set of weighted ideal multi-swaps S . First, suppose wlog that { c 1 . . . c t } is the set of current centers in ˜ C that are neither lonely nor busy. Let L be the set of lonely centers in ˜ C . For each i = 1 . . . t , we do the following. Let In be the set of optimal centers in ˜ O captured by c i . Choose a set L i of | In | -1 centers from L , set L ← L \ L i and define Out = L i ∪ { c i } . Assign weight 1 to ( In, Out ) and add it to S . For each busy center c i ∈ { c t +1 . . . c h ′ } let A be the set of optimal centers in ˜ O captured by c i , pick a set L i of | A | -1 lonely current centers from L (a counting argument shows that this is always possible). Set L ← L \ L i . For each o j ∈ A and c ℓ ∈ L i assign weight 1 / ( | A | -1) to ( o j , c ℓ ) and add it to S . Suppose we are left with ℓ centers o ′ 1 . . . o ′ ℓ ∈ ˜ O such that C [ o ′ i ] ̸∈ ˜ C . Apparently, we have not included any o ′ i in any swap yet. However, since | ˜ C| ≥ | ˜ O| , we are left with at least ℓ ′ ≥ ℓ lonely centers c ′ 1 . . . c ′ ℓ ′ ∈ ˜ C . For each i = 1 . . . ℓ we assign weight 1 to ( o ′ i , c ′ i ) and add it to S .

Observation 7. The process above generates a set of weighted ideal multi-swaps such that: (i) Every swap has size at most p ; (ii) The combined weights of swaps involving an optimal center o i ∈ ˜ O is 1 ; (iii) The combined weights of swaps involving a current center c i is at most 1 + 1 /p .

Consider an ideal swap ( In, Out ) . Let O ∗ In = ⋃ o i ∈ In O ∗ i and C Out = ⋃ c j ∈ Out C j . Define the reassignment cost Reassign ( In, Out ) as the increase in cost of reassigning points in C Out \ O ∗ In to centers in C \ Out . Namely,

We take the increase in cost of the following reassignment as an upper bound to the reassignment cost. For each p ∈ C Out \ O ∗ In we consider its closest optimal center O ∗ [ p ] and reassign p to the current center that is closest to O ∗ [ p ] , namely C [ O ∗ [ p ]] . In formulas, we have

Indeed, by the way we defined our ideal swaps we have C [ O ∗ [ p ]] ̸∈ Out for each p ̸∈ O ∗ In and this reassignment is valid. Notice that the right hand side in the equation above does not depend on In .

Proof. Deferred to the supplementary material.

Lemma 9. The combined weighted reassignment costs of all ideal multi-swaps in S is at most (2 + 2 /p ) · ( OPT + √ ALG √ OPT ) .

Proof. Denote by w ( In, Out ) the weight associated with the swap ( In, Out ) .

The second inequality uses ( iii ) from Observation 7. Applying Lemma 8 completes the proof.

Recall the notions of radius and core of an optimal cluster introduced in Section 2. We say that a swap ( In, Out ) is strongly improving if cost ( P, ( C ∪ In ) \ Out ) ≤ (1 -δ/k ) · cost ( P, C ) . Let In = { o 1 . . . o s } ⊆ ˜ O and Out = { c 1 . . . c s } ⊆ ˜ C we say that an ideal swap ( In, Out ) is good if for every q 1 ∈ core ( o 1 ) . . . q s ∈ core ( o s ) the swap ( Q , Out ) is strongly improving, where Q = { q 1 . . . q s } . We call an ideal swap bad otherwise. We say that an optimal center o i ∈ ˜ O is good if that's the case for at least one of the ideal swaps it belongs to, otherwise we say that it is bad. Notice that each optimal center in ˜ O is assigned to a swap in S , so it is either good or bad. Denote with G the union of cores of good optimal centers in ˜ O .

Lemma 10. If an ideal swap ( In, Out ) is bad, then we have

Proof. Let In = { o 1 . . . o s } , Q = { q 1 . . . q s } such that q 1 ∈ core ( o 1 ) . . . q s ∈ core ( o s ) . Then, by Lemma 1 cost ( O ∗ In , Q ) ≤ (2 + δ ) cost ( O ∗ In , O ∗ ) . Moreover, Reassign ( In, Out ) = cost ( P \ O ∗ In , C \ Out ) -cost ( P \ O ∗ In , C ) because points in P \ C Out are not affected by the swap. Therefore, cost ( P, ( C ∪ Q ) \ Out ) ≤ (2 + δ ) cost ( O ∗ In , O ∗ ) + Reassign ( In, Out ) + cost ( P \ O ∗ In , C ) . Suppose by contradiction that Equation (2) does not hold, then

Hence, ( Q , Out ) is strongly improving and this holds for any choice of Q , contradiction.

Lemma 11. If ALG / OPT > η 2 + δ then cost ( G, C ) = Ω δ ( cost ( P, C )) . Thus, if we D 2 -sample q we have P [ q ∈ G ] = Ω δ (1) .

Proof. First, we observe that the combined current cost of all optimal clusters in O ∗ \ ˜ O is at most k · δ ALG /k = δ ALG. Now, we prove that the combined current cost of all O ∗ i such that o i is bad is ≤ (1 -2 δ ) ALG. Suppose, by contradiction, that it is not the case, then we have:

The second and last inequalities make use of Observation 7. The third inequality uses Lemma 10.

Setting η 2 = ALG / OPT we obtain the inequality η 2 -(2 + 2 /p ± O ( δ )) η -(4 + 2 /p ± O ( δ )) ≤ 0 . Hence, we obtain a contradiction in the previous argument as long as η 2 -(2 + 2 /p ± O ( δ )) η -(4 + 2 /p ± O ( δ )) > 0 . A contradiction there implies that at least an δ -fraction of the current cost is due to points in ⋃ Good o i ∈ ˜ O O ∗ i . We combine this with Lemma 2 and conclude that the total current cost of G = ⋃ Good o i ∈ ˜ O core ( O ∗ i ) is Ω δ ( cost ( P, C )) .

Finally, we prove Lemma 6. Whenever q 1 ∈ G we have that q 1 ∈ core ( o 1 ) for some good o 1 . Then, for some s ≤ p we can complete o 1 with o 2 . . . o s such that In = { o 1 . . . o s } belongs to a good swap. Concretely, there exists Out ⊆ C such that ( In, Out ) is a good swap. Since In ⊂ ˜ O we have cost ( O ∗ i , C ) > δ OPT /k for all o i ∈ In , which combined with Lemma 2 gives that for i = 2 . . . s P [ q i ∈ core ( o i )] ≥ Ω δ (1 /k ) . Hence, we have P [ q i ∈ core ( o i ) for i = 1 . . . s ] ≥ Ω δ,p (1 /k p -1 ) . Whenever we sample q 1 . . . q s from core ( o 1 ) . . . core ( o s ) , we have that ( Q , Out ) is strongly improving. Notice, however, that ( Q , Out ) is a s -swap and we may have s < p . Nevertheless, whenever we sample q 1 . . . q s followed by any sequence q s +1 . . . q p it is enough to choose Out ′ = Out ∪ { q s +1 . . . q p } to obtain that ( { q 1 . . . q p } , Out ′ ) is an improving p -swap.

4 A Faster (9 + ε ) -Approximation Local Search Algorithm

The MSLS algorithm from Section 3 achieves an approximation ratio of η 2 + ε , where η 2 -(2 + 2 /p ) η -(4 + 2 /p ) = 0 and ε > 0 is an arbitrary small constant. For large p we have η ≈ 10 . 48 . On the other hand, employing p simultaneous swaps, Kanungo et al. [2004] achieve an approximation factor of ξ 2 + ε where ξ 2 -(2 + 2 /p ) ξ -(3 + 2 /p ) = 0 . If we set p ≈ 1 /ε this yields a (9 + O ( ε )) -approximation. In the same paper, they prove that 9 -approximation is indeed the best possible for p -swap local search, if p is constant (see Theorem 3 . 1 in Kanungo et al. [2004]). They showed that 9 is the right locality gap for local search, but they matched it with a very slow algorithm. To achieve a (9 + ε ) -approximation, they discretize the space reducing to O ( nε -d ) candidate centers and perform an exhaustive search over all size(1 /ε ) subsets of candidates at every step. As we saw in the related work section, it is possible to combine techniques from coreset and dimensionality reduction to reduce the number of points to n ′ = k · poly ( ε -1 ) and the number of dimensions to d ′ = log k · ε -2 . O ( ε -3 log ε -1 )

This reduces the complexity of Kanungo et al. [2004] to k .

In this section, we leverage techniques from Cohen-Addad et al. [2021] to achieve a (9 + ε ) -approximation faster 4 . In particular, we obtain the following.

Theorem 12. Given a set of n points in R d with aspect ratio ∆ , there exists an algorithm that computes a 9 + ε -approximation to k -means in time ndk O ( ε -2 ) log O ( ε -1 ) (∆) · 2 -poly ( ε -1 ) .

Notice that, besides being asymptotically slower, the pipeline obtained combining known techniques is highly impractical and thus it did not make for an experimental test-bed. Moreover, it is not obvious how to simplify such an ensemble of complex techniques to obtain a practical algorithm.

Limitations of MSLS. The barrier we need to overcome in order to match the bound in Kanungo et al. [2004] is that, while we only consider points in P as candidate centers, the discretization they employ considers also points in R d \ P . In the analysis of MSLS we show that we sample each point q i from core ( O ∗ i ) or equivalently that q i ∈ B ( o i , (1 + ϵ ) ρ i ) , where ρ i is such that O ∗ i would have the same cost w.r.t. o i if all its points were moved on a sphere of radius ρ i centered in o i . This allows us to use a Markov's inequality kind of argument and conclude that there must be Ω ϵ ( | O ∗ i | ) points in O ∗ i ∩ B ( o i , (1 + ϵ ) ρ i ) . However, we have no guarantee that there is any point at all in O ∗ i ∩ B ( o i , (1 -ε ) ρ i ) . Indeed, all points in O ∗ i might lie on ∂B ( o i , ρ i ) . The fact that potentially all our candidate centers q are at distance at least ρ i from o i yields (by Lemma 1) cost ( O ∗ i , q ) ≥ 2 cost ( O ∗ i , o i ) , which causes the zero-degree term in ξ 2 -(2+2 /p ) ξ -(3+2 /p ) = 0 from Kanungo et al. [2004] to become a 4 in our analysis.

Improving MSLS by taking averages. First, we notice that, in order to achieve (9 + ε ) -approximation we need to set p = Θ(1 /ε ) . The main hurdle to achieve a (9 + ε ) -approximation is that we need to replace the q i in MSLS with a better approximation of o i . We design a subroutine that computes, with constant probability, an ε -approximation ˆ o i of o i (namely, cost ( O ∗ i , ˆ o i ) ≤ (1 + ε ) cost ( O ∗ i , o i ) ). The key idea is that, if sample uniformly O (1 /ε ) points from O ∗ i and define ˆ o i to be the average of our samples then cost ( O ∗ i , ˆ o i ) ≤ (1 + ε ) cost ( O ∗ i , o i )

Though, we do not know O ∗ i , so sampling uniformly from it is non-trivial. To achieve that, for each q i we identify a set N of nice candidate points in P such that a poly ( ε ) /k fraction of them are from O ∗ i . We sample O (1 /ε ) points uniformly from N and thus with probability ( ε/k ) O (1 /ε ) we sample only points from O ∗ i . Thus far, we sampled O (1 /ε ) points uniformly from N ∩ O ∗ i . What about

the points in O ∗ i \ N ? We can define N so that all points in O ∗ i \ N are either very close to some of the ( q j ) j or they are very far from q i . The points that are very close to points ( q j ) j are easy to treat. Indeed, we can approximately locate them and we just need to guess their mass, which is matters only when ≥ poly ( ε ) ALG, and so we pay only a log O (1 /ε ) (1 /ε ) multiplicative overhead to guess the mass close to q j for j = 1 . . . p = Θ(1 /ε ) . As for a point f that is very far from q i (say, || f -q i ||≫ ρ i ) we notice that, although f 's contribution to cost ( O ∗ i , o i ) may be large, we have cost ( f, o ) ≈ cost ( f, o i ) for each o ∈ B ( q i , ρ i ) ⊆ B ( o i , (2 + ε ) ρ i ) assuming q i ∈ core ( o i ) .

5 Experiments

In this section, we show that our new algorithm using multi-swap local search can be employed to design an efficient seeding algorithm for Lloyd's which outperforms both the classical k -means++ seeding and the single-swap local search from Lattanzi and Sohler [2019].

Algorithms. The multi-swap local search algorithm that we analysed above performs very well in terms of solution quality. This empirically verifies the improved approximation factor of our algorithm, compared to the single-swap local search of Lattanzi and Sohler [2019].

Motivated by practical considerations, we heuristically adapt our algorithm to make it very competitive with SSLS in terms of running time and still remain very close, in terms of solution quality, to the theoretically superior algorithm that we analyzed. The adaptation of our algorithm replaces the phase where it selects the p centers to swap-out by performing an exhaustive search over ( k + p p ) subsets of centers. Instead, we use an efficient heuristic procedure for selecting the p centers to swap-out, by greedily selecting one by one the centers to swap-out. Specifically, we select the first center to be the cheapest one to remove (namely, the one that increases the cost by the least amount once the points in its cluster are reassigned to the remaining centers). Then, we update all costs and select the next center iteratively. After p repetitions we are done. We perform an experimental evaluation of the 'greedy' variant of our algorithm compared to the theoretically-sound algorithm from Section 3 and show that employing the greedy heuristic does not measurably impact performance.

The four algorithms that we evaluate are the following: 1) KM++: The k -means++ from Arthur and Vassilvitskii [2007], 2) SSLS: The Single-swap local search method from Lattanzi and Sohler [2019], 3) MSLS: The multi-swap local search from Section 3, and 4) MSLS-G: The greedy variant of multi-swap local search as described above.

We use MSLS-Gp = x and MSLSp = x , to denote MSLS-G and MSLS with p = x , respectively. Notice that MSLS-Gp = 1 is exactly SSLS. Our experimental evaluation explores the effect of p -swap LS, for p > 1 , in terms of solution cost and running time.

Datasets. We consider the three datasets used in Lattanzi and Sohler [2019] to evaluate the performance of SSLS: 1) KDD-PHY 100 , 000 points with 78 features representing a quantum physic task kdd [2004], 2) RNA 488 , 565 points with 8 features representing RNA input sequence pairs Uzilov et al. [2006], and 3) KDD-BIO 145 , 751 points with 74 features measuring the match between a protein and a native sequence kdd [2004]. We discuss the results for two or our datasets, namely KDD-BIO and RNA. We deffer the results on KDD-PHY to the appendix and note that the results are very similar to the results on RNA.

We performed a preprocessing step to clean-up the datasets. We observed that the standard deviation of some features was disproportionately high. This causes all costs being concentrated in few dimensions making the problem, in some sense, lower-dimensional. Thus, we apply min-max scaling to all datasets and observed that this causes all our features' standard deviations to be comparable.

Experimental setting. All our code is written in Python. The code will be made available upon publication of this work. We did not make use of parallelization techniques. To run our experiments, we used a personal computer with 8 cores, a 1 . 8 Ghz processor, and 15 . 9 GiB of main memory We run all experiments 5 times and report the mean and standard deviation in our plots. All our plots report the progression of the cost either w.r.t local search steps, or Lloyd's iterations. We run experiments on all our datasets for k = 10 , 25 , 50 . The main body of the paper reports the results for k = 25 , while the rest can be found in the appendix. We note that the conclusions of the experiments for k = 10 , 50 are similar to those of k = 25 .

Removing centers greedily. We first we compare MSLS-G with MSLS. To perform our experiment, we initialize k = 25 centers using k -means++ and then run 50 iterations of local search for both

KDD-BIO,k = 25

RNA,k = 25

1.05

1.00

1.00

0.95

0.95

0.90

Cost

t

0.85

0.90

0.80

0.85

MSLS-G-p=3

0.75

MSLS-G-p=3

MSLS-p=3

MSLS-p=3

KM++

0.70

+ +WX

0

10

20

30

40

50

0

10

20

30

40

50

LS Iterations

LS Iterations

algorithms, for p = 3 swaps. Due to the higher running of the MSLS we perform this experiments on 1% uniform sample of each of our datasets. We find out that the performance of the two algorithms is comparable on all our instances, while they both perform roughly 15%-27% at convergence. Figure 1 shows the aggregate results, over 5 repetitions of our experiment.

It may happen that MSLS, which considers all possible swaps of size p at each LS iteration, performs worse than MSLS-G as a sub-optimal swap at intermediate iterations may still lead to a better local optimum by coincidence. Given that MSLS-G performs very comparably to MSLS, while it is much faster in practice, we use MSLS-G for the rest of our experiments where we compare to baselines. This allows us to consider higher values of p , without compromising much the running time.

Results: Evaluating the quality and performance of the algorithms. In our first experiment we run KM++ followed by 50 iterations of MSLS-G with p = 1 , 4 , 7 , 10 and plot the relative cost w.r.t. KM++ at each iteration, for k = 25 . The first row of Figure 2 plots the results. Our experiment shows that, after 50 iterations MSLS-G for p = 4 , 7 , 10 achieves improvements of roughly 10% compared to MSLS-Gp = 1 and of the order of 20% -30% compared to KM++. We also report the time per iteration that each algorithm takes. For comparison, we report the running time of a single iteration of Lloyd's next to the dataset's name. It is important to notice that, although MSLS-Gp = 1 is faster, running more iterations MSLS-Gp = 1 is not sufficient to compete with MSLS-G when p > 1 .

Results: Evaluating the quality after postprocessing using Lloyd. In our second experiment, we use KM++ and MSLS-G as a seeding algorithm for Lloyd's and measure how much of the performance improvement measured in the first experiment is retained after running Lloyd's. First, we initialize our centers using KM++ and the run 15 iterations of MSLS-G for p = 1 , 4 , 7 . We measure the cost achieved by running 10 iterations of Lloyd's starting from the solutions found by MSLS-G as well as KM++. In Figure 2 (second row) we plot the results. Notice that, according to the running times from the first experiment, 15 iterations iterations of MSLS-G take less than 10 iterations of Lloyd's for p = 4 , 7 (and also for p = 10 , except on RNA). We observe that MSLS-G for p > 1 performs at least as good as SSLS from Lattanzi and Sohler [2019] and in some cases maintains non-trivial improvements.

Results: Evaluating the quality and performance of the algorithms against a fixed deadline.

In this experiment we run KM++ followed by MSLS-G with p = 1 , 4 , 7 , 10 , for a set of fixed amounts of time. This setting allows the versions of MSLS-G with smaller swap size to perform more iterations compared to the versions of the algorithm with a larger swap size, as smaller swap size leads to lower running time per iteration. Let τ be the average time that Lloyd's algorithm requires to complete a simple iteration on a specific instance. We plot the cost of the solution produced by each algorithm after running λ × τ for each λ ∈ { 1 , · · · , 20 } in Figure 3. Our experiment shows that MSLS-G for p = 4 , 7 , 10 achieves improvements of more than 5% compared to MSLS-Gp = 1 even when compared against a fixed running time, and of the order of 20% -30% compared to KM++.

Conclusion and Future Directions

We present a new algorithm for the k -means problem and we show that it outperforms theoretically and experimentally state-of-the-art practical algorithms with provable guarantees in terms of solution

KDD-BIO,k = 25 (0.19 s per Lloyd's iter)

RNA, k = 25 (0.56 s per Lloyd's iter)

KM++

1.10

MSLS-G-p=1, 0.03 s

1.00

MSLS-G-p=4, 0.05 s

1.05

MSLS-G-p=7, 0.08 S

MSLS-G-p=10, 0.13 s

1.00

0.95

S60

+ +WX

Cost

MSLS-G-p=1, 0.12 S

Cos

0.90

MSLS-G-p=4, 0.23 s

0.90

MSLS-G-p=7, 0.32 s

0.85

MSLS-G-p=10, 0.43 s

08°0

0.85

0.75

0.80

0

10

20

30

40

50

0

10

20

30

40

50

LS Iterations

LS Iterations

KDD-BIO,k = 25

RNA, k = 25

37000

KM++

24500

KM++

MSLS-G-p=1

MSLS-G-p=1

MSLS-G-p=4

24000-

MSLS-G-p=4

36500

MSLS-G-p=7

MSLS-G-p=7

MSLS-G-p=10

MSLS-G-p=10

23500-

36000

Cost

t

23000

0

35500

22500

35000

22000

21500 -

34500

2

3

4

5

6

7

8

9

10

2

3

4

5

6

7

8

9

10

Lloyd'sIterations

Lloyd'sIterations

KDD-PHY,k= 25

1.10

1.05

1.00

1.00

0.95

0.95

KM++

KM+ +

t

MSLS-G-p=1

t

0.90

MSLS-G-p=1

Cos

0.90

MSLS-G-p=4

MSLS-G-p=7

C

0.85

MSLS-G-p=7

MSLS-G-p=4

MSLS-G-p=10

MSLS-G-p=10

0.80

0.85

0.75

0.80

0.70

1

5

10

15

20

1

5

10

15

20

Timeout / (time for 1 Lloyd's iter.)

Timeout / (time for 1 Lloyd's iter.)

quality. A very interesting open question is to improve our local search procedure by avoiding the exhaustive search over all possible sizep subsets of centers to swap out, concretely an algorithm with running time ˜ O (2 poly (1 /ε ) ndk ) .

Acknowledgements. This work was partially done when Lorenzo Beretta was a Research Student at Google Research. Moreover, Lorenzo Beretta receives funding from the European Union's Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement No. 801199.

References

Kdd cup. 2004. URL http://osmot.cs.cornell.edu/kddcup/datasets.html .

Sara Ahmadian, Ashkan Norouzi-Fard, Ola Svensson, and Justin Ward. Better guarantees for kmeans and Euclidean k-median by primal-dual algorithms. SIAM Journal on Computing , 49(4): FOCS17-97-FOCS17-156, 2019.

David Arthur and Sergei Vassilvitskii. K-means++ the advantages of careful seeding. In Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms , pages 1027-1035, 2007.

David Arthur and Sergei Vassilvitskii. Worst-case and smoothed analysis of the ICP algorithm, with an application to the k-means method. SIAM J. Comput. , 39(2):766-782, 2009. doi: 10.1137/ 070683921. URL http://dx.doi.org/10.1137/070683921 .

Vijay Arya, Naveen Garg, Rohit Khandekar, Adam Meyerson, Kamesh Munagala, and Vinayaka Pandit. Local search heuristics for k-median and facility location problems. SIAM J. Comput. , 33(3):544-562, 2004. doi: 10.1137/S0097539702416402. URL https://doi.org/10.1137/ S0097539702416402 .

Pranjal Awasthi, Moses Charikar, Ravishankar Krishnaswamy, and Ali Kemal Sinop. The hardness of approximation of euclidean k-means. In Lars Arge and János Pach, editors, 31st International Symposium on Computational Geometry, SoCG 2015, June 22-25, 2015, Eindhoven, The Netherlands , volume 34 of LIPIcs , pages 754-767. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2015. doi: 10.4230/LIPIcs.SOCG.2015.754. URL https://doi.org/10.4230/LIPIcs.SOCG.2015.754 .

Sayan Bandyapadhyay and Kasturi Varadarajan. On variants of k-means clustering. In 32nd International Symposium on Computational Geometry (SoCG 2016) . Schloss Dagstuhl-LeibnizZentrum fuer Informatik, 2016.

Luca Becchetti, Marc Bury, Vincent Cohen-Addad, Fabrizio Grandoni, and Chris Schwiegelshohn. Oblivious dimension reduction for k -means: beyond subspaces and the johnson-lindenstrauss lemma. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, Phoenix, AZ, USA, June 23-26, 2019 , pages 1039-1050, 2019. doi: 10.1145/3313276. 3316318. URL https://doi.org/10.1145/3313276.3316318 .

Anup Bhattacharya, Dishant Goyal, Ragesh Jaiswal, and Amit Kumar. On sampling based algorithms for k-means. In Nitin Saxena and Sunil Simon, editors, 40th IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science, FSTTCS 2020, December 14-18, 2020, BITS Pilani, K K Birla Goa Campus, Goa, India (Virtual Conference) , volume 182 of LIPIcs , pages 13:1-13:17. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2020. doi: 10. 4230/LIPIcs.FSTTCS.2020.13. URL https://doi.org/10.4230/LIPIcs.FSTTCS.2020.13 .

Moses Charikar and Sudipto Guha. Improved combinatorial algorithms for facility location problems. SIAM J. Comput. , 34(4):803-824, 2005. doi: 10.1137/S0097539701398594. URL https: //doi.org/10.1137/S0097539701398594 .

Davin Choo, Christoph Grunau, Julian Portmann, and Vaclav Rozhon. k-means++: few more steps yield constant approximation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1909-1917. PMLR, 2020. URL https://proceedings.mlr.press/v119/ choo20a.html .

Vincent Cohen-Addad. A fast approximation scheme for low-dimensional k -means. In Artur Czumaj, editor, Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2018, New Orleans, LA, USA, January 7-10, 2018 , pages 430-440. SIAM, 2018. doi: 10.1137/1.9781611975031.29. URL https://doi.org/10.1137/1.9781611975031.29 .

Vincent Cohen-Addad and Karthik C. S. Inapproximability of clustering in lp metrics. In David Zuckerman, editor, 60th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2019, Baltimore, Maryland, USA, November 9-12, 2019 , pages 519-539. IEEE Computer Society, 2019. doi: 10.1109/FOCS.2019.00040. URL https://doi.org/10.1109/FOCS.2019.00040 .

Vincent Cohen-Addad and Claire Mathieu. Effectiveness of local search for geometric optimization. In 31st International Symposium on Computational Geometry, SoCG 2015, June 22-25, 2015, Eindhoven, The Netherlands , pages 329-343, 2015. doi: 10.4230/LIPIcs.SOCG.2015.329. URL http://dx.doi.org/10.4230/LIPIcs.SOCG.2015.329 .

Vincent Cohen-Addad and Chris Schwiegelshohn. On the local structure of stable clustering instances. In Chris Umans, editor, 58th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2017, Berkeley, CA, USA, October 15-17, 2017 , pages 49-60. IEEE Computer Society, 2017. doi: 10.1109/FOCS.2017.14. URL https://doi.org/10.1109/FOCS.2017.14 .

Vincent Cohen-Addad, Philip N. Klein, and Claire Mathieu. Local search yields approximation schemes for k-means and k-median in euclidean and minor-free metrics. SIAM J. Comput. , 48(2): 644-667, 2019. doi: 10.1137/17M112717X. URL https://doi.org/10.1137/17M112717X .

Vincent Cohen-Addad, Karthik C. S., and Euiwoong Lee. On approximability of clustering problems without candidate centers. In Dániel Marx, editor, Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms, SODA 2021, Virtual Conference, January 10 - 13, 2021 , pages 26352648. SIAM, 2021. doi: 10.1137/1.9781611976465.156. URL https://doi.org/10.1137/1. 9781611976465.156 .

Vincent Cohen-Addad, David Saulpic, and Chris Schwiegelshohn. Improved coresets and sublinear algorithms for power means in euclidean spaces. Advances in Neural Information Processing Systems , 34:21085-21098, 2021.

Vincent Cohen-Addad, Hossein Esfandiari, Vahab S. Mirrokni, and Shyam Narayanan. Improved approximations for euclidean k -means and k -median, via nested quasi-independent sets. In Stefano Leonardi and Anupam Gupta, editors, STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, Rome, Italy, June 20 - 24, 2022 , pages 1621-1628. ACM, 2022a. doi: 10.1145/3519935.3520011. URL https://doi.org/10.1145/3519935.3520011 .

Vincent Cohen-Addad, Kasper Green Larsen, David Saulpic, and Chris Schwiegelshohn. Towards optimal lower bounds for k-median and k-means coresets. In Stefano Leonardi and Anupam Gupta, editors, STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing, Rome, Italy, June 20 - 24, 2022 , pages 1038-1051. ACM, 2022b. doi: 10.1145/3519935.3519946. URL https://doi.org/10.1145/3519935.3519946 .

Vincent Cohen-Addad, Kasper Green Larsen, David Saulpic, Chris Schwiegelshohn, and Omar Ali Sheikh-Omar. Improved coresets for euclidean k-means. In NeurIPS , 2022c. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 120c9ab5c58ba0fa9dd3a22ace1de245-Abstract-Conference.html .

Vincent Cohen-Addad, Euiwoong Lee, and Karthik C. S. Johnson coverage hypothesis: Inapproximability of k -means and k -median in ℓ p metrics. In Proceedings of the 2022 ACM-SIAM Symposium on Discrete Algorithms, SODA 2022 . SIAM, 2022d.

Feldman and M. Langberg. A unified framework for approximating and clustering data. In STOC , pages 569-578, 2011.

Zachary Friggstad, Mohsen Rezapour, and Mohammad R. Salavatipour. Local search yields a PTAS for k-means in doubling metrics. SIAM J. Comput. , 48(2):452-480, 2019. doi: 10.1137/ 17M1127181. URL https://doi.org/10.1137/17M1127181 .

Fabrizio Grandoni, Rafail Ostrovsky, Yuval Rabani, Leonard J. Schulman, and Rakesh Venkat. A refined approximation for euclidean k-means. Inf. Process. Lett. , 176:106251, 2022. doi: 10.1016/j.ipl.2022.106251. URL https://doi.org/10.1016/j.ipl.2022.106251 .

Christoph Grunau, Ahmet Alper Özüdogru, Václav Rozhon, and Jakub Tetek. A nearly tight analysis of greedy k-means++. In Nikhil Bansal and Viswanath Nagarajan, editors, Proceedings of the 2023 ACM-SIAM Symposium on Discrete Algorithms, SODA 2023, Florence, Italy, January 2225, 2023 , pages 1012-1070. SIAM, 2023. doi: 10.1137/1.9781611977554.ch39. URL https: //doi.org/10.1137/1.9781611977554.ch39 .

Anupam Gupta and Kanat Tangwongsan. Simpler analyses of local search algorithms for facility location. CoRR , abs/0809.2554, 2008. URL http://arxiv.org/abs/0809.2554 .

Venkatesan Guruswami and Piotr Indyk. Embeddings and non-approximability of geometric problems. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, January 12-14, 2003, Baltimore, Maryland, USA. , pages 537-538, 2003. URL http://dl.acm.org/ citation.cfm?id=644108.644198 .

Mary Inaba, Naoki Katoh, and Hiroshi Imai. Applications of weighted voronoi diagrams and randomization to variance-based k-clustering. In Proceedings of the tenth annual symposium on Computational geometry , pages 332-339, 1994.

Ragesh Jaiswal, Amit Kumar, and Sandeep Sen. A simple D 2-sampling based PTAS for k-means and other clustering problems. Algorithmica , 70(1):22-46, 2014. doi: 10.1007/s00453-013-9833-9. URL https://doi.org/10.1007/s00453-013-9833-9 .

Tapas Kanungo, David M. Mount, Nathan S. Netanyahu, Christine D. Piatko, Ruth Silverman, and Angela Y. Wu. A local search approximation algorithm for k-means clustering. Computational Geometry , 28(2):89-112, 2004. ISSN 0925-7721. doi: https://doi.org/10.1016/ j.comgeo.2004.03.003. URL https://www.sciencedirect.com/science/article/pii/ S0925772104000215 . Special Issue on the 18th Annual Symposium on Computational Geometry - SoCG2002.

Madhukar R. Korupolu, C. Greg Plaxton, and Rajmohan Rajaraman. Analysis of a local search heuristic for facility location problems. J. Algorithms , 37(1):146-188, 2000. doi: 10.1006/jagm. 2000.1100. URL http://dx.doi.org/10.1006/jagm.2000.1100 .

Amit Kumar, Yogish Sabharwal, and Sandeep Sen. Linear-time approximation schemes for clustering problems in any dimensions. J. ACM , 57(2):5:1-5:32, 2010. doi: 10.1145/1667053.1667054. URL https://doi.org/10.1145/1667053.1667054 .

Silvio Lattanzi and Christian Sohler. A better k-means++ algorithm via local search. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 3662-3671. PMLR, 2019. URL https://proceedings.mlr.press/v97/lattanzi19a.html .

Euiwoong Lee, Melanie Schmidt, and John Wright. Improved and simplified inapproximability for k-means. Inf. Process. Lett. , 120:40-43, 2017. doi: 10.1016/j.ipl.2016.11.009. URL https: //doi.org/10.1016/j.ipl.2016.11.009 .

SP Lloyd. Least square quantization in pcm. bell telephone laboratories paper. published in journal much later: Lloyd, sp: Least squares quantization in pcm. IEEE Trans. Inform. Theor.(1957/1982) , 18, 1957.

Konstantin Makarychev, Yury Makarychev, Maxim Sviridenko, and Justin Ward. A bi-criteria approximation algorithm for k-means. In Klaus Jansen, Claire Mathieu, José D. P. Rolim, and Chris Umans, editors, Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM 2016, September 7-9, 2016, Paris, France , volume 60 of LIPIcs , pages 14:1-14:20. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2016. doi: 10.4230/LIPIcs. APPROX-RANDOM.2016.14. URL https://doi.org/10.4230/LIPIcs.APPROX-RANDOM. 2016.14 .

Konstantin Makarychev, Yury Makarychev, and Ilya P. Razenshteyn. Performance of johnsonlindenstrauss transform for k -means and k -medians clustering. In Moses Charikar and Edith Cohen, editors, Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC 2019, Phoenix, AZ, USA, June 23-26, 2019 , pages 1027-1038. ACM, 2019. doi: 10.1145/3313276. 3316350. URL https://doi.org/10.1145/3313276.3316350 .

Jirí Matousek. On approximate geometric k-clustering. Discrete & Computational Geometry , 24(1):61-84, 2000. doi: 10.1007/s004540010019. URL http://dx.doi.org/10.1007/ s004540010019 .

Andrew V Uzilov, Joshua M Keegan, and David H Mathews. Detection of non-coding rnas on the basis of predicted secondary structure formation free energy change. BMC bioinformatics , 7(1): 1-30, 2006.

Supplementary Material

Proofs from Section 3

Proof.

The second equality is due to Lemma 1 and the last inequality is due to Cauchy-Schwarz.

Proofs from Section 4

In this section, we prove the following.

Theorem 12. Given a set of n points in R d with aspect ratio ∆ , there exists an algorithm that computes a 9 + ε -approximation to k -means in time ndk O ( ε -2 ) log O ( ε -1 ) (∆) · 2 -poly ( ε -1 ) .

We start with a key lemma showing that a sample of size O (1 /ε ) is enough to approximate 1 -mean.

Lemma 13 (Form Inaba et al. [1994]) . Given an instance P ⊆ R d , sample m = 1 / ( εδ ) points uniformly at random from P and denote the set of samples with S . Then cost ( P, µ ( S )) ≤ (1 + ε ) cost ( P, µ ( P )) with probability at least 1 -δ .

Proof. We want to prove that with probability 1 -δ we have || µ ( S ) -µ ( P ) || 2 ≤ ε cost ( P, µ ( P )) / | P | . Then, applying Lemma 1 gives the desired result. First, we notice that µ ( P ) is an unbiased estimator of µ ( P ) , namely E [ µ ( S )] = µ ( P ) . Then, we have

where s i are uniform independent samples from P . Applying Markov's inequality concludes the proof.

The algorithm that verifies Theorem 12 is very similar to the MSLS algorithm from Section 3 and we use the same notation to describe it. The intuition is that in MSLS we sample Q = { q 1 . . . q p } hoping that q i ∈ core ( o i ) for each i ; here we refine q i to a better approximation ˆ o i of o i and swap the points (ˆ o i ) i rather than ( q i ) i . Our points ˆ o i are generated taking the average of some sampled point, thus we possibly have ˆ o i ̸∈ P while, on the other hand, q i ∈ P .

A (9+ ε ) -approximation MSLS algortihm. First, we initialize our set of centers using k -means++. Then, we run ndk O ( ε -2 ) · 2 poly ( ε -1 ) local search steps, where a local search step works as follows. Set p = Θ( ε -1 ) . We D 2 -sample a set Q = { q 1 . . . q p } of points from P (without updating costs). Then, we iterate over all possible sets Out = { c 1 . . . c p } of p distinct elements in C ∪ Q . We define the set of temporary centers T = ( C ∪ Q ) \ Out and run a subroutine APX-CENTERS ( T ) which returns a list of poly ( ε -1 ) · log O ( ε -1 ) (∆) sizes sets ̂ In = { ˆ o 1 . . . ˆ o s } (where s = |Q \ Out | ). We select the set ̂ In in this list such that the swap ( ̂ In, Out \ Q ) yields the maximum cost reduction. Then we select the set Out that maximizes the cost reduction obtained in this way. If ( ̂ In, Out \ Q ) actually reduces the cost then we perform that swap.

A subroutine to approximate optimal centers. Here we describe the subroutine APX-CENTERS ( T ) . Let Q\ Out = { q 1 . . . q s } . Recall that s ≤ p = O ( ε -1 ) . This subroutine outputs a list of 2 poly ( ε -1 ) · log O ( ε -1 ) (∆) sizes sets ̂ In = { ˆ o 1 . . . ˆ o s } . Here we describe how to find a list of 2 poly ( ε -1 ) · log(∆) values for ˆ o 1 . The same will apply for ˆ o 2 . . . ˆ o s and taking the Cartesian product yields a list of 2 poly ( ε -1 ) · log O ( ε -1 ) (∆) sizes sets. Assume wlog that the pairwise distances between points in P lie in [1 , ∆] . We iterate over all possible values of ρ 1 ∈ { 1 , (1 + ε ) . . . (1 + ε ) ⌈ log 1+ ε ∆ ⌉ } . We partition P in three sets: the set of far points F = { x ∈ P | cost ( x, q 1 ) > ρ 2 1 /ε 3 } , the set of close points C = { x ∈ P \ F | cost ( x, T ) ≤ ε 3 ρ 2 1 } and the set of nice points N = P \ ( C ∪ F ) . Then, we sample uniformly from N a set S of size Θ( ε -1 ) . For each ( s +1) -tuple of coefficients α 0 , α 1 . . . α s ∈ { 1 , (1 -ε ) , (1 -ε ) 2 , . . . (1 -ε ) ⌈ log 1 -ε ( ε 7 ) ⌉ } ∪{ 0 } we output the candidate solution given by the convex combination

so, for each value of ρ 1 , we output 2 poly ( ε -1 ) values for ˆ o 1 . Hence, 2 poly ( ε -1 ) · log(∆) values in total.

Analysis

The key insight in the analysis of the MSLS algorithm form Section 3 was that every q i was a proxy for o i because q i ∈ core ( o i ) , and thus q i provided a good center for O ∗ i . In the analysis of this improved version of MSLS we replace q i with ˆ o i which makes a better center for O ∗ i . Formally, fixed Out , we say that a point ˆ o i is a perfect approximation of o i when cost ( O ∗ i , ( C ∪ { ˆ o i } ) \ Out ) ≤ (1 + ε ) OPT i + ε OPT /k . We define ˜ O and ˜ C as in Section 3, except that we replace δ with ε (which here is not assumed to be a constant). Likewise, we build the set S of ideal multi-swaps as in Section 3. Recall that we say that a multi-swap ( In, Out ) is strongly improving if cost ( P, ( C ∪ In ) \ Out ) ≤ (1 -ε/k ) · cost ( P, C ) . Let In = { o 1 . . . o s } ⊆ ˜ O and Out = { c 1 . . . c s } ⊆ ˜ C , we overload the definition from Section 3 and say that the ideal multi-swap ( In, Out ) is good if for every ̂ In = { ˆ o 1 . . . ˆ o s } such that each ˆ o i is a perfect approximation of o i for each i = 1 . . . s the swap ( ̂ In, Out ) is strongly improving. We call an ideal swap bad otherwise. As in Section 3, we define the core of an optimal center; once again we replace δ with ϵ , which is no longer constant. The two following lemmas are our stepping stones towards Theorem 12.

Lemma 14. If ALG / OPT > 9 + O ( ε ) then, with probability k -O ( ε -1 ) · 2 -poly ( ε -1 ) , there exists Out ⊆ C ∪ Q such that:

(ii) If we define In = { o 1 . . . o s } then ( In, Out \ Q ) is a good ideal swap.

Lemma 15. If ( i ) from Lemma 14 holds, then with probability k -O ( ε -2 ) · 2 -poly ( ε -1 ) , the list returned by APX-CENTERS contains ̂ In = { ˆ o 1 . . . ˆ o s } such that ˆ o i is a perfect approximation of o i for each i = 1 . . . s .

Proof of Theorem 12. Here we prove that our improved MSLS algorithm achieves a (9 + O ( ε )) -approximation, which is equivalent to Theorem 12 up to rescaling ε . Combining Lemma 14 and

Lemma 15 we obtain that, as long as ALG / OPT > 9 + O ( ε ) , with probability at least k -O ( ε -2 ) · 2 -poly ( ε -1 ) , the list returned by APX-CENTERS contains ̂ In = { ˆ o 1 . . . ˆ o s } such that ( ̂ In, Out \ Q ) is strongly improving. If this happens, we call such a local step successful . Now the proof goes exactly as the proof of Theorem 3. Indeed, We show that k O ( ε -2 ) · 2 poly ( ε -1 ) local steps suffice to obtain Ω( k log log k/ε ) successful local steps, and thus to obtain the desired approximation ratio, with constant probability.

To prove the running time bound it is sufficient to notice that a local search step can be performed in time nd log O ( ε -1 ) (∆) · 2 poly ( ε -1 ) .

In the rest of this section, we prove Lemma 14 and Lemma 15.

Observation 16. If we assume δ = ε non-constant in Lemma 2, then performing the computations explicitly we obtain Pr[ q ∈ core ( O ∗ i )] ≥ poly ( ε ) .

In order to prove Lemma 14, we first prove the two lemmas. Lemma 17 is the analogous of Lemma 10 and Lemma 18 is the analogous of Lemma 11. Overloading once again the definition from Section 3, we define G as the union of cores of good optimal centers in ˜ O , where an optimal center is defined to be good if at least one of the ideal multi-swaps in S it belongs to is good (exactly as in Section 3).

Lemma 17. If an ideal swap ( In, Out ) is bad, then we have

Proof. Let In = { o 1 . . . o s } , ̂ In = { ˆ o 1 . . . ˆ o s } such that ˆ o i is a perfect approximation of o i for each i = 1 . . . s . Recall that O ∗ In := ⋃ s i =1 O ∗ i , then

Moreover, Reassign ( In, Out ) = cost ( P \ O ∗ In , C \ Out ) -cost ( P \ O ∗ In , C ) because points in P \ C Out are not affected by the swap. Therefore, cost ( P, ( C ∪ ̂ In ) \ Out ) ≤ (1 + ε ) cost ( O ∗ In , O ∗ ) + Reassign ( In, Out ) + cost ( P \ O ∗ In , C ) . Suppose by contradiction that Equation (4) does not hold, then

Hence, ( ̂ In, Out ) is strongly improving and this holds for any choice of ̂ In , contradiction.

Lemma 18. If ALG / OPT > 9 + O ( ε ) then cost ( G, C ) ≥ cost ( P, C ) · poly ( ε ) . Thus, if we D 2 -sample q we have P [ q ∈ G ] ≥ poly ( ε ) .

Proof. First, we observe that the combined current cost of all optimal clusters in O ∗ \ ˜ O is at most k · ε ALG /k = ε ALG. Now, we prove that the combined current cost of all O ∗ i such that o i is bad is ≤ (1 -2 ε ) ALG. Suppose, by contradiction, that it is not the case, then we have:

The second and last inequalities make use of Observation 7. The third inequality uses Lemma 17.

Setting η 2 = ALG / OPT we obtain the inequality η 2 -(2 + 2 /p ± O ( ε )) η -(3 + 2 /p ± O ( ε )) ≤ 0 . Hence, we obtain a contradiction in the previous argument as long as η 2 -(2 + 2 /p ± O ( ε )) η -(3 +

2 /p ± O ( ε )) > 0 , which holds for p = Θ( ε -1 ) and η 2 = 9+ O ( ε ) . Acontradiction there implies that at least an ε -fraction of the current cost is due to points in ⋃ Good o i ∈ ˜ O O ∗ i . Thanks to Observation 16, we have P q ∼ cost ( q, C ) [ q ∈ core ( O ∗ i ) | q ∈ O ∗ i ] ≥ poly ( ε ) . Therefore, we can conclude that the current cost of G = ⋃ Good o i ∈ ˜ O core ( O ∗ i ) is at least a poly ( ε ) -fraction of the total current cost.

Proof of Lemma 14. Thanks to Lemma 18, we have that P [ q 1 ∈ G ] ≥ poly ( ε ) . Whenever q 1 ∈ G we have that q 1 ∈ core ( o 1 ) for some good o 1 . Then, for some s ≤ p we can complete o 1 with o 2 . . . o s such that In = { o 1 . . . o s } belongs to a good swap. Concretely, there exists Out ⊆ C such that ( In, Out ) is a good swap. Since In ⊂ ˜ O we have cost ( O ∗ i , C ) > ε OPT /k for all o i ∈ In , which combined with Observation 16 gives that, for each i = 2 . . . s , P [ q i ∈ core ( o i )] ≥ poly ( ε ) /k . Hence, we have P [ q i ∈ core ( o i ) for i = 1 . . . s ] ≥ 2 -poly ( ε -1 ) k -O ( ε -1 ) . Notice, however, that ( ̂ In, Out ) is a s -swap and we may have s < p . Nevertheless, whenever we sample q 1 . . . q s followed by any sequence q s +1 . . . q p it is enough to choose Out ′ = Out ∪ { q s +1 . . . q p } to obtain that ( { q 1 . . . q p } , Out ′ ) is an improving p -swap.

In order to prove Lemma 15 we first need a few technical lemmas.

Lemma 19 (Lemma 2 from Lattanzi and Sohler [2019]) . For each x, y, z ∈ R d and ε > 0 , cost ( x, y ) ≤ (1 + ε ) cost ( x, z ) + (1 + 1 /ε ) cost ( z, y ) .

Lemma 20. Given q ∈ R d and Z ⊆ R d such that cost ( Z, q ) ≤ ε 2 Γ then, for each o ∈ R d

Proof. To obtain the first inequality, we apply Lemma 19 to bound cost ( z, o ) ≤ (1+ ε ) cost ( z, o ) + (1 + 1 /ε ) cost ( z, q ) for each z ∈ Z . To obtain the second inequality, we bound cost ( q, o ) ≤ (1 + ε ) cost ( z, o ) + (1 + 1 /ε ) cost ( z, q ) for each z ∈ Z .

Lemma 21. Let X = { x 1 . . . x ℓ } be a weighted set of points in R d such that x i has weight w i . Let µ be the weighted average of X . Let ˆ µ = ˆ µ ( α 1 . . . α ℓ ) be the weighted average of X where x i has weight α i . If w i ≤ α i ≤ w i / (1 -ε ) for each i = 1 . . . ℓ , then if we interpret cost ( X,C ) as ∑ x i ∈ X w i · cost ( x i , C ) we have cost ( X, ˆ µ ) ≤ (1 + O ( ε )) cost ( X,µ ) .

Proof. We note that µ minimizes the expression cost ( X,µ ) . Moreover, cost ( X,z ) ≤ ∑ ℓ i =1 α i · cost ( x i , z ) ≤ cost ( X,z ) / (1 -ε ) . Since ˆ µ minimizes the expression ∑ ℓ i =1 α i · cost ( x i , z ) it must be cost ( X, ˆ µ ) ≤ cost ( X,µ ) / (1 -ε ) .

Adopting the same proof strategy, we obtain the following.

Observation 22. Thanks to Lemma 20, we can assume that the points in Z are concentrated in q for the purpose of computing a (1 + O ( ε )) -approximation to the 1 -means problem on Z , whenever an additive error Γ is tolerable. Indeed, moving all points in Z to q introduces a 1 + O ( ε ) multiplicative error on cost ( Z, · ) and a O ( ε )Γ additive error.

The next lemma shows that a point z that is far from a center o experiences a small variation of cost ( z, o ) when the position of o is slightly perturbed.

Lemma 23. Given o, z ∈ R d such that || o -z || ≥ r/ε we have that for every o ′ ∈ B ( o, r ) , cost ( z, o ′ ) = (1 ± O ( ε )) cost ( z, o ) .

Proof. It is enough to prove it for all o ′ that lie on the line L passing through o and z , any other point in o ′′ ∈ B ( o, r ) admits a point o ′ ∈ B ( o, r ) ∩ L with || o ′ -z || = || o ′′ -z || . It is enough to compute the derivative of cost ( z, · ) with respect to the direction of L and see that ∂ cost ( z, · ) ∂L | B ( o,r ) = (1 ± O ( ε )) r/ε . Thus, cost ( z, o ′ ) = cost ( z, o ) ± (1 ± O ( ε )) r 2 /ε = (1 ± O ( ε )) cost ( z, o ) .

Proof of Lemma 15. Here we prove that for each o 1 . . . o s there exist coefficients α ( i ) 0 . . . α ( i ) s ∈ { 1 , (1 -ε ) . . . (1 -ε ) ⌈ log 1 -ε ( ε 7 ) ⌉ } ∪ { 0 } such that the convex combination ˆ o i = ˆ o i ( α ( i ) 0 . . . α ( i ) s ) is a perfect approximation of o i , with probability k -O ( ε -2 ) · 2 -poly ( ε -1 ) . Wlog, we show this

for o 1 only. Concretely, we want to show that, with probability k -O ( ε -1 ) · 2 -poly ( ε -1 ) , there exist coefficients α 0 . . . α s such that ˆ o 1 = ˆ o 1 ( α 0 . . . α s ) satisfies cost ( O ∗ 1 , ( C ∪ { ˆ o 1 } ) \ Out ) ≤ (1 + O ( ε )) OPT 1 + O ( ε ) OPT /k . Taking the joint probability of these events for each i = 1 . . . s we obtain the success probability k -O ( ε -2 ) · 2 -poly ( ε -1 ) . Note that we are supposed to prove that cost ( O ∗ 1 , ( C ∪ { ˆ o 1 } ) \ Out ) ≤ (1 + ε ) OPT 1 + ε OPT /k , however we prove a weaker version where ε is replaced by O ( ε ) , which is in fact equivalent up to rescaling ε .

Similarly to C [ · ] and O ∗ [ · ] define T [ p ] as the closest center to p in T . Denote with C 1 , F 1 and N 1 the intersections of O ∗ 1 with C, F and N respectively. In what follows we define the values of α 0 . . . α s that define ˆ o 1 = ˆ o 1 ( α 0 . . . α s ) and show an assignment of points in O ∗ 1 to centers in ( C ∪{ ˆ o 1 } ) \ Out with cost (1 + O ( ε )) OPT 1 + O ( ε ) OPT /k . Recall that we assume that q i ∈ core ( o i ) for each i = 1 . . . s .

In what follows, we assign values to the coefficients ( α i ) i . It is understood that if the final value we choose for α i is v then we rather set α i to the smallest power of (1 -ε ) which is larger than v , if v > ε 7 . Else, set α i to 0 . We will see in the end that this restrictions on the values of α i do not impact our approximation.

In what follows, we will assign the points in O ∗ 1 to C \ Out , if this can be done inexpensively. If it cannot, then we will assign points to ˆ o 1 . In order to compute a good value for ˆ o 1 we need an estimate of the average of points assigned to ˆ o 1 . For points in N 1 , computing this average is doable (leveraging Lemma 13) while for points in O ∗ 1 \ N 1 we show that either their contribution is negligible or we can collapse them so as to coincide with some q i ∈ Q without affecting our approximation. The coefficients ( α i ) i ≥ 1 represent the fraction of points in O ∗ i which is collapsed to q i . α 0 represents the fraction of points in O ∗ i which average we estimate as µ ( S ) . Thus, Equation (3) defines ˆ o i as the weighted average of points q i , where the weights are the (approximate) fractions of points collapsed onto q i , together with the the average µ ( S ) and its associated weight α 0 .

Points in C 1 . All points p ∈ C 1 such that T [ p ] ̸∈ Q can be assigned to T [ p ] ∈ C \ Out incurring a total cost of at most ε 6 OPT 1 , by the definition of C 1 . Given a point p ∈ C 1 with T [ p ] ∈ Q we might have T [ p ] ̸∈ C \ Out and thus we cannot assign p to T [ p ] . Denote with W the set of points p with T [ p ] ∈ Q . Our goal is now to approximate µ ( W ) . In order to do that, we will move each p ∈ W to coincide with q i = T [ p ] . We can partition W into W 1 . . . W s so that for each z ∈ W i T [ z ] = q i . If p ∈ Z i then we have || p -q i || 2 ≤ ε 3 ρ 2 1 . Hence, thanks to Observation 22, we can consider points in W i as if they were concentrated in q i while losing at most an additive factor O ( ε ) OPT 1 and a multiplicative factor (1 + ε ) on their cost. For i = 1 . . . s , set α i ←| W i | / | O ∗ 1 | . In this way, ∑ s i =1 α i · q i / ∑ s i =1 α i is an approximates solution to 1 -mean on W up to a multiplicative factor (1 + ε ) and an additive factor O ( ε ) OPT 1 .

Points in N 1 . Consider the two cases: ( i ) cost ( N 1 , T ) > ε 2 OPT /k ; ( ii ) cost ( N 1 , T ) ≤ ε 2 OPT /k .

Case ( i ) . We show that in this case µ ( S ) is a (1 + ε ) -approximation for 1 -mean on N 1 , with probability k -O ( ε -1 ) · 2 -poly ( ε -1 ) . First, notice that if we condition on S ⊆ N 1 then Lemma 13 gives that µ ( S ) is a (1 + ε ) -approximation for 1 -mean on N 1 with constant probability. Thus, we are left to prove that S ⊆ N 1 with probability k -O ( ε -1 ) · 2 -poly ( ε -1 ) . We have that the P p ∼ cost ( p, T ) [ p ∈ N 1 | p ∈ N ] ≥ ε 2 /k , however the costs w.r.t. T of points in N varies of at most a factor poly ( ε -1 ) , thus P p ∼ Unif [ p ∈ N 1 | p ∈ N ] ≥ poly ( ε ) /k . The probability of S ⊆ N 1 is thus ( poly ( ε ) /k ) | S | = k -O ( ε -1 ) · 2 -poly ( ε -1 ) . In this case, we set α 0 ← | N 1 | / | O ∗ 1 | because µ ( S ) approximates the mean of the entire set N 1 .

Case ( ii ) . Here we give up on estimating the mean of N 1 and set α 0 ← 0 . The point x ∈ N 1 such that T [ x ] ̸∈ Q can be assigned to T [ x ] incurring a combined cost of ε 2 OPT /k . We partition the remaining points in N 1 into Z 1 ∪ . . . Z s where each point x is placed in Z i if T [ x ] = q i . Now, we collapse the points in Z i so as to coincide with q i and show that this does not worsen our approximation factor. In terms of coefficients ( α i ) i , this translates into the updates α i ← α i + | Z i | / | O ∗ i | for each i = 1 . . . s .

Indeed, using Observation 22 we can move all points in Z i to q i incurring an additive combined cost of ε OPT /k and a multiplicative cost of 1 + O ( ε ) .

Points in F 1 . Points in F 1 are very far from q 1 and thus far from o 1 , hence even if their contribution to cost ( O ∗ 1 , o 1 ) might be large, we have cost ( F 1 , o 1 ) = (1 ± O ( ε )) cost ( F 1 , o ′ ) for all o ′ in a ball of radius ρ 1 /ε centered in o 1 , thanks to Lemma 23.

Let H be the set of points that have not been assigned to centers in C\ Out . In particular, H = W ∪ N 1 if points in N 1 satisfy case ( i ) and H = W ∪ Z 1 . . . Z s if points in N 1 satisfy case ( ii ) . We consider two cases.

If || µ ( H ) -q 1 || ≤ ρ/ε , then || µ ( H ) -o 1 || ≤ ρ (1 + ε + 1 /ε ) because q 1 ∈ core ( o 1 ) . Since for each f ∈ F 1 we have || f -o 1 || ≥ || f -q 1 || -(1 + ε ) ρ ≥ Ω( ρ/ε 3 ) then cost ( f, o ′ ) = (1 ± O ( ε )) cost ( f, o 1 ) for each o ′ in a ball of radius O ( ρ/ε ) centered in o 1 , and so in particular for o ′ = µ ( H ) . Thus in this case we can simply disregard all points in F 1 and computing ˆ o 1 according to the ( α i ) i defined above yields a perfect approximation of o i .

̸

Else, if || µ ( H ) -q 1 || > ρ/ε , a similar argument applies to show that cost ( H,o ′ ) = (1 ± ε ) cost ( H,o ) for each o ′ in ball of radius O ( ρ ) centered in o 1 . Indeed, we can rewrite cost ( H,o ′ ) as | H | · cost ( µ ( H ) , o ′ ) + cost ( µ ( H ) , H ) . If || µ ( H ) -q 1 || < ρ/ε the first term varies of at most a factor (1 + ε ) and the second term is constant. Thus in this case ˆ o 1 = q 1 is a perfect approximation of o 1 and we simply set α 1 = 1 and α j = 0 for j = 1 . In other words, here µ ( N 1 ∪ H ) is too far from q 1 (and thus o 1 ) to significantlyt influence the position of ˆ o 1 and the same holds for any point in F 1 . This works, of course, because we assumed q 1 ∈ core ( o 1 ) .

Discussing the limitations on the coefficients values. The proof above would work smoothly if we were allowed to set α i to exactly the values discussed above, representing the fractions of points from O ∗ i captured by different q i s. However, to make the algorithm efficient we limit ourselves to values in { 1 , (1 -ε ) . . . (1 -ε ) ⌈ log 1 -ε ( ε 7 ) ⌉ } ∪ { 0 } . Lemma 21 shows that as long as the values of ( α i ) i estimate the frequencies described above up to a factor 1 ± O ( ε ) then the approximation error is within a multiplicative factor 1 ± O ( ε ) .

We are left to take care of the case in which α i is set to a value < ε 7 . We set α i when dealing with points in C 1 ∪ N 1 and for each x ∈ C 1 ∪ N 1 we have, for each o ′ ∈ B ( q 1 , (1 + ε ) ρ ) , cost ( x, o ′ ) ≤ 2 cost ( q 1 , o ′ ) + 2 cost ( x, q 1 ) = O ( ρ 1 ε -6 ) . Thus, if we simply set α i ← 0 whenever we have α i < ε 7 then the combined cost of points in O ∗ 1 with respect to o ′ varies by ε 7 | O ∗ 1 | · ρ 1 ε -6 = O ( ε ) OPT 1 . Effectively, ignoring these points does not significantly impact the cost. hence solving 1 -mean ignoring these points finds a (1 + O ( ε )) -approximate solution to the original problem.

Additional Experimental Evaluation

In this section we report additional experiments which presentation did not fit in the main body. In particular, we run experiments on the dataset KDD-PHY and for k = 10 , 50 .

In Figure 4 we compare MSLS-G with MSLS. To perform our experiment, we initialize k = 25 centers using KM++ and then run 50 iterations of local search for both algorithms, for p ∈ { 2 , 3 } swaps. We repeat each experiment 5 times. For ease of comparison, we repeat the plot for the KDD-BIO and RNA datasets that we present in the main body of the paper. Due to the higher running of the MSLS we perform this experiments on 1% uniform sample of each of our datasets. We find out that the performance of the two algorithms is comparable on all our instances, while they both perform roughly 15%-27% better than k -means++ at convergence.

In Figure 5 we run KM++ followed by 50 iterations of MSLS-G with p = 1 , 4 , 7 , 10 and k = 10 , 25 , 50 (expcluding the degenerate case p = k = 10 ) and plot the relative cost w.r.t. KM++ at each iteration. The results for k = 25 on KDD-BIO and RNA can be found in Figure 2. We repeat each experiment 5 times. Our experiment shows that, after 50 iterations MSLS-G for p = 4 , 7 , 10 achieves improvements of roughly 5 -10% compared to MSLS-Gp = 1 and of the order of 20% -40% compared to KM++. These improvements are more prominent for k = 25 , 50 . We also report the time per iteration that each algorithm takes. For comparison, we report the running time of a single iteration of Lloyd's next to the dataset's name. Notice that the experiment on RNA for k = 50 is performed on a 10% uniform sample of the original dataset, due to the high running time.

In Figure 6, we use KM++ and MSLS-G as a seeding algorithm for Lloyd's and measure how much of the performance improvement measured is retained after running Lloyd's. First, we initialize

1.025

1.000

1.00

0.975

0.95

t

0.950

Cost

8

0.925

0.90

0.900

0.875

MSLS-G-p=2

0.85

MSLS-G-p=3

0.850

MSLS-p=2

MSLS-p=3

++WXI

+ +WX

0

10

20

30

40

50

0

10

20

30

40

50

LS Iterations

LS Iterations

KDD-PHY, k = 25

KDD-PHY, k = 25

1.05

MSLS-G-p=2

1.05

MSLS-G-p=3

MSLS-p=2

MSLS-p=3

++WX

++WX

1.00

1.00

0.95

Cost

0.95

t

8

0.90

0.90

0.85

0.85

0.80

0.80

0.75

0

10

20

30

40

50

0

10

20

30

40

50

LS Iterations

LS Iterations

RNA, k = 25

RNA, k = 25

1.05 -

1.05

1.00

1.00

0.95

0.95

Cost

06°0

Cost

0.90

0.85

0.85

0.80-

0.80

0.75

0.75

MSLS-G-p=2

MSLS-G-p=3

MSLS-p=2

MSLS-p=3

0.70

KM++

0.70

KM++

10

20

30

40

50

0

10

20

30

40

50

LS Iterations

LS Iterations

KDD-BI0, k = 10 (5.35 s per Lloyd's iter)

KDD-BI0, k = 50 (24.57 s per Lloyd's iter)

1.05 -

1.000

1.00

0.975

0.95

0.950

t

t

0.925

8

0.90

0060

0.85

0.875

++WX

KM++

MSLS-G-p=1, 12.07 s

0.80

MSLS-G-p=1, 4.83 s

0.850

MSLS-G-p=4, 18.65 s

MSLS-G-p=4, 7.15 s

MSLS-G-p=7, 21.05 s

MSLS-G-p=7, 11.83 s

0.825

MSLS-G-p=10, 23.68 s

0.75

10

20

30

40

50

0

10

20

30

40

50

LSIterations

LS Iterations

RNA, k = 10 (17.29 s per Lloyd's iter)

RNA, k = 50 (10.29 s per Lloyd's iter)

1.05

1.0

1.00

60

0.95

++WX

Cost

KM+ +

MSLS-G-p=1, 3.70 s

0.8

MSLS-G-p=4, 20.64 s

MSLS-G-p=1, 13.61 s

MSLS-G-p=4, 6.80 s

MSLS-G-p=7, 33.99 s

0.85

MSLS-G-p=7, 8.03 s

MSLS-G-p=10, 9.48 s

0.7

0.80

0.6

0.75

0.70

0

10

20

30

40

50

0

10

20

30

40

50

LS Iterations

LS Iterations

KDD-PHY,k=10 (5.97s per Lloyd's iter)

KDD-PHY, k = 25 (14.69 s per Lloyd's iter)

1.00

1.0

0.95

Cost

0.9

MSLS-G-p=7, 8.32 s

++WX

MSLS-G-p=1, 3.02 s

MSLS-G-p=4, 4.91 s

Cost

060

0.85

MSLS-G-p=7, 11.15 s

MSLS-G-p=1, 5.83 s

MSLS-G-p=4, 8.77 s

KM+ +

0.8

MSLS-G-p=10, 13.93 s

080

0.7

0.75

0.70

0

10

20

30

40

50

0

10

20

30

40

50

LS Iterations

LS Iterations

KDD-PHY,k =50 (25.51 s per Lloyd's iter)

KM++

1.00

MSLS-G-p=1, 8.62 S

MSLS-G-p=4, 13.50 s

MSLS-G-p=7, 15.59 s

S60

MSLS-G-p=10, 17.54 s

060

Cost

0.85

0.80

0.75

0.70

0

10

20

30

40

50

LS Iterations

our centers using KM++ and the run 15 iterations of MSLS-G for p = 1 , 4 , 7 . We measure the cost achieved by running 10 iterations of Lloyd's starting from the solutions found by MSLS-G as well as KM++. We run experiments for k = 10 , 25 , 50 and we repeat each experiment 5 times. We observe that for k = 25 , 50 MSLS-G for p > 1 performs at least as good as SSLS from Lattanzi and Sohler [2019] and in some cases maintains non-trivial improvements. These improvements are not noticeable for k = 10 ; however, given how Lloyd's behave for k = 10 we conjecture that k = 10 might be an 'unnatural' number of clusters for our datasets.

t

8

t

8

t

8

43000

42500

42000

41500

41000

40500

40000

44000

42000

40000

38000

36000

34000

32000

000000

195000

190000

185000

180000

2

2

Z

3

3

3

4

4

KDD-BIO, k = 10

5

6

Lloyd's Iterations

8

1

RNA, k = 10

5

6

7

8

Lloyd's Iterations

KDD-PHY, k = 10

4

7

5

6

Lloyd's Iterations

8

128000

126000

124000

Cost

122000

120000

118000

116000

114000

3

++WX

MSLS-G-p=4

MSLS-G-p=1

MSLS-G-p=7

9

10

+ +WX

MSLS-G-p=4

MSLS-G-p=1

MSLS-G-p=7

9

10

+ +WX

MSLS-G-p=1

MSLS-G-p=7

MSLS-G-p=4

9

00000

32800

32600

32400

32200

32000

31800 -

31600

17500

17000

16500

Cost

16000

15500

155000

152500

150000

8

147500

145000

142500

140000

137500

10

2

4

5

6

7

8

Lloyd's Iterations

4

5

7

Lloyd's Iterations

8

6

RNA, k = 50

4

3

3

3

5

7

8

6

Lloyd's Iterations

KDD-PHY, k = 25

4

7

5

6

8

Lloyd's Iterations

KM++

MSLS-G-p=1

MSLS-G-p=7

MSLS-G-p=4

9

10

Cost

2

2

++WX

MSLS-G-p=4

MSLS-G-p=1

MSLS-G-p=7

9

10

+ +WX

MSLS-G-p=4

MSLS-G-p=1

MSLS-G-p=7

9

10

++WX

MSLS-G-p=4

MSLS-G-p=1

MSLS-G-p=7

9

10

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies local-search algorithms for k-means clustering. The goal here is to obtain a local-search algorithm which (1) give a close to 9-approximation ratio (which is best possible for local search algorithms) and (2) is practical. In the past literature, there has been many local search algorithms developed for the k-means problem, but all of them either have a relatively big approximation ratio or have a prohibitive running time in practice due to the complexity of the local moves at each step. \n\nIn this paper, the authors give a simple local search algorithm which is practical and guarantees a 10.48-approximation. They also give a more complex local search which matches the approximation lower bound of 9, but with a significantly faster running time than in Kanungo et al. [2004]. However, this algorithm is significantly more complex and is unlikely to be used in practice. They also give experiments that show that their first algorithm and a slight variant of it perform well in practice.\n\nSTRENGTHS:\nIn my opinion, the strengths are as follows.\n1) The paper is clearly written.\n2) The first algorithm is a nice result since it combines practicality with an almost best possible theoretical guarantee. As far as I know (not being an expert), this is the first time the D^2-sampling is used and analyzed in combination with multi-swap local search. Overall, the techniques in the paper are nice.\n\nWEAKNESSES:\nI do not see an obvious weakness in this work, the results are tight or close. Overall, I think this is a nice paper that fits well in the context of NeurIPS.\n\nQUESTIONS:\nIt is unclear to me if the approximation ratio of 10.48 is tight for the algorithm. As well as the 26.64 for the single swap version.\n\nIn experiment section, I do not see any discussion on the running time of the compared algorithms? Maybe it would be nice just to mention it.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the standard $k$-means problem. Given a $k$-means instance $(P,k)$, the goal is to find a set $C$ of centers with size at most $k$ such that the sum of the squared  distances from $P$ to $C$ is minimized. For the $k$-means problem, Lattanzi and Sohler (ICML 2019) proposed an elegant combination of the local search and the $k$-means++ seeding methods. Instead of enumerating all the swap pairs constructed between the data points and the current centers, they showed that $k$-means++ sampling can be used for sampling a data point to serve as the candidate center for swapping in such that a $509$-approximation can be achieved in expectation in time $O(ndk^2loglogk)$. This paper extends the single-swap strategy to multi-swap strategy. In a single local search step, this paper shows that using $k$-means++ to sample $t$ data points induces a good $t$-swap for the local search process such that the clustering cost can be reduced significantly by at least a $\\Omega(1-1/k)$ fraction with certain probability. After $O(k^{t-1})$ local search steps, an improved approximation can be achieved related to the swap size $t$, where the total running time can be bounded by $\\tilde{O}(ndk^{2t})$. When $t$ is large enough, the approximation could be smaller than 10.48. By combining the techniques from Cohen Addad et al. (NeurIPS 2021), the papers shows that by using more local search steps (with exponential dependence on $\\epsilon$), the approximation guarantee can nearly match the lower bound of the $9+\\epsilon$ of the standard multi-swap local search method.\n\nSTRENGTHS:\nThe strengths of this paper can be summarized as follows:\n1. This paper proposes a fast multi-swap local search method for solving the standard $k$-means problem, which runs in linear time in the data size. The approximation ratio significantly improves the previous one with linear running time in the data size (i.e., 509-approximation)\n2. This paper gives another multi-swap local search method which better approximates the optimal clustering centers during the local search swaps using a subroutine called APX-CENTERS, where an improved approximation guarantee, i.e., $9+\\epsilon$, can be obtained with running time exponentially dependent on $poly(\\epsilon^{-1})$. The approximation ratio matches the approximation lower bound of local search methods for the $k$-means problem and significantly improves the running time of previous work based on direct enumeration of swap pairs constructed between the whole dataset and the current centers opened.\n3. In experiments, this paper proposes a heuristic method which avoids the exhaustive searching process for determining which subset of the sampled set of data points should be swapped in. The experiments show that the multi-swap local search method achieves better performance on clustering quality with fixed local search iterations compared with single-swap local search methods.\n\nWEAKNESSES:\n1. Although the authors show that sampling-based multi-swap local search can improve the approximation to a very small constant and even matches the lower bound of local search methods, the core idea behind is to use the successive $k$-means++ sampling strategy to construct candidate set of centers for swapping in, which is an extension of the sampling-based local search method (denoted as LS++ method )proposed by Lattanzi and Sohler. The claimed improvement in ratio seems to be minor as LS++ already guarantees constant approximation (in [1] the authors claimed that they did not attempt to optimize the constants and this paper verifies that LS++ can yield approximation ratio smaller than 26.4). The key idea behind the analysis for multi-swap local search method is as follows: (1) the authors further divide the optimal clusters into different groups according to their current clustering cost (i.e., $cost(P_h^*,C) \\ge \\delta ALG/k$ or $cost(P_h^*,C) < \\delta ALG/k$) (2) the authors give weights for swap pairs such that each current center can be used for at most $(1+1/t)$ times when performing a summation of all bad swap pairs. The analysis for probability lower bound that induces a good swap is similar to that of LS++ method.\n\n2. Some theoretical details is unclear to me. During the construction of ideal weighted multi-swaps, this paper partitions the optimal clustering centers and current clustering centers based on $\\tilde{O}=${\n$o_i \\ | \\ cost(O_i^*,C) \\ge \\delta ALG/k$} and $\\tilde{C}=$ $C\\backslash$ {$C[o_i] \\ | \\ o_i \\in O^* \\backslash \\tilde{O}$}. Then, in the construction of ideal multi-swaps, each swap is consist of $In$ and $Out$ such that $In \\subseteq \\tilde{O}$ and $Out \\subseteq \\tilde{C}$. If my understanding is not wrong, some of the optimal centers in $\\tilde{O}$ is not used for construction of ideal multi-swaps. If not all optimal centers is used at least once for constructing ideal multi-swaps, then the bound for summation of clustering cost for all \"bad'\" optimal clusters may not hold in Lemma 11. The ideal multi-swaps construction considers the two caes: (1) for each center $c_i \\in \\tilde{C}$ that is neither busy nor lonely, form $In$ with the optimal centers captured by $c_i$ (multi-swap) and borrow $|In|-1$ lonely centers from $L$ (2) for each center in $\\tilde{C}$ that is busy, form single swap using each of optimal centers captured by $c_i$ and the lonely centers in $L$. However, there may exist some case that for some optimal clusering center $o_i^* \\in \\tilde{O}$, the current center $c_j$ that captures $o_i^*$ is not in $\\tilde{C}$ and hence $o_i^*$ will no longer be used for constructing ideal multi-swaps. Here is an example where $o_i^*, o_2^*, o_3^*$ is the optimal clustering centers, and $c_1$, $c_2$, $c_3$ is the current centers with swap size $p=2$. In this instances, it holds that $c_1$ captures $o_1^*$ and $o_2^*$, $c_3$ captures $o_3^*$, $cost(O_1^*,C) \\ge \\delta ALG/k$, $cost(O_2^*,C)<\\delta ALG/k$ and $cost(O_3^*,C) \\ge \\delta ALG/k$. Hence, we have $\\tilde{O}=${$o_1^*,o_3^*$} and  $\\tilde{C} =${$c_2,c_3$}. During the construction of ideal multi-swaps, since $c_1$ is not in $\\tilde{C}$, $o_1^*$ will never be used during the construction. \n\n3. Another main weakness of this paper is the experiments. (1) As a local search algorithm, it’s important to validate the performance of the proposed algorithm within a certain time limitation, which is not presented in the experimental results. This paper mainly compares the experimental performances with fixed local search steps. It is unfair since multi-swap local search method takes more time than single-swap local search methods (i.e., the LS++ method). (2) In experiments, the maximum local search step is set to be 50, which is much smaller than the theoretical bounds (100000kloglogk) for obtaining good performance for LS++ algorithm. For this setting, even single-swap local search method may not converge to a good local optimal solution. The authors should present more evaluation for larger local search steps (i.e., several hundreds or thousands of local search steps) and fixed running time that is large enough for each algorithm to reach the convergence. (3) The size of the tested datasets is rather small (i.e., no larger than 500,000). However, in the recent results for clustering algorithms, instances of size over 1 million [1] or even 100 million [2] have been considered. Since multi-swap local search is much slower than the single-swap local search method (even with heuristic acceleration by avoiding the enumeration of all subsets of data points for swapping in), it is unclear that whether the proposed algorithm can scale well on large-scale datasets compared with single-swap local search method.\n\n[1] Ren J, Hua K, Cao Y. Global Optimal K-Medoids Clustering of One Million Samples[J]. Advances in Neural Information Processing Systems, 2022, 35: 982-994.\n[2] Matsui Y, Ogaki K, Yamasaki T, et al. Pqk-means: Billion-scale clustering for product-quantized codes[C]//Proceedings of the 25th ACM international conference on Multimedia. 2017: 1725-1733.\n\nQUESTIONS:\n1. During the construction of ideal multi-swaps, can the construction process described in the paper guarantee that all the optimal centers in $\\tilde{O}$ is used at least once for constructing ideal multi-swaps (see the example of weakness 2). If some of the optimal clustering centers   are not used for constructing ideal multi-swaps, will the bound for summation of clustering cost for all \"bad'\" optimal clusters in Lemma 11 still hold?\n\n2. In Lemma 2, this paper shows that if the current clustering cost for some optimal cluster $O_i^*$ is larger than $(2+3\\delta)$ times its optimal clustering cost, the clustering cost of data points close to $o_i^*$ should take a large fraction (related to $\\delta$) of the whole clustering cost induced by $O_i^*$. Hence, in each local search step, the success probability for performing a good swap should be related to parameter $\\delta$, for example $\\Omega(1/\\delta)$. However, the running time of the proposed multi-swap local search algorithm is independent of $\\delta$. I think there should be at least an $\\frac{1}{\\delta}$ term in the running time, or does it mean that $\\delta$ is a constant given as the input, please explain.\n\n3. In previous multi-swap local search approximation schemes, the approximation ratio is given as a function related to the swap size $t$ (i.e., in [1], the approximation is given as $(3+2/t)^2$). However, in this paper, the approximation ratio is determined by an inequality related to $\\eta$ and $t$. Can the author provide the approximation ratio in the form similar to that of [1] such that one could figure out how large that swap size is enough for obtaining an approximation guarantee of 10.48.\n\n4. Why are all the instances in the experimental setting small? In the recent results for clustering algorithms, instances of size over 1 million [2] or even 100 million [3] have been considered. How does the proposed multi-swap local search method scale on large datasets?\n\n5. What are the performances of different local search algorithms within a certain time limitation (large enough for reaching the convergence)?\n\n[1] Kanungo T, Mount D M, Netanyahu N S, et al. A local search approximation algorithm for k-means clustering[C]//Proceedings of the eighteenth annual symposium on Computational geometry. 2002: 10-18.\n\n[2] Ren J, Hua K, Cao Y. Global Optimal K-Medoids Clustering of One Million Samples[J]. Advances in Neural Information Processing Systems, 2022, 35: 982-994.\n\n[3] Matsui Y, Ogaki K, Yamasaki T, et al. Pqk-means: Billion-scale clustering for product-quantized codes[C]//Proceedings of the 25th ACM international conference on Multimedia. 2017: 1725-1733.\n\nLIMITATIONS:\nSince this is a theoretical paper, I don't think there is potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new k-means algorithm: multi-swap local search (MSLS) which combines local search and k-means++, to achieve a constant approximation guarantee with efficient time complexity. Specifically, the local search framework includes a step that selects alternative centers for optimizing the cost function, and this paper selects those centers using the $ D^2 $-sample from k-means++  (without updating costs). \n\nThe authors prove that the random candidate centers obtained in this way lead to an improvement over the cost. By iterating on this step, MSLS will return a constant approximation result.\n\nThe author also leverage ideas from coreset and  dimensionality reducion to propose a more efficent $ 9+\\varepsilon $-approximation algorithm within the local search framework.\n\nSTRENGTHS:\n- The paper is well written. All sources of motivations and techniques involved in this work are clearly presented, making it easy to follow the key ideas of the algoirthm and the related works. It does make a novel contribution to the classic problem.\n- The idea behind the MSLS algorithm is natural and the proofs are very clear. Although the author refer techniques from the previous work, the approximation analysis is non-trivial and results in very tight ratios.\n- The MSLS algorithm is simple and easy to implement. The experiments are comprehensive. This work is valuable in both theory and practice.\n\nWEAKNESSES:\n- the time complexity is $ \\tilde{O} (nd k^p) $, which grows exponentially w.r.t. $p$.\n\nQUESTIONS:\n- Both k-means++ and local search are not limited to the Euclidean space. Can thm3 be generalized to the metric space? I notice that the analysis are influenced by [1] and there exists mean computing ($ \\mu(\\cdot) $) that prevents such generalization. However, [2] gives a constant approximation for k-means in the metric space. Therefore, I am interested in what the current analysis method lacks for the generalization and whether such limitation can be overcomed? \n\n\n\n[1] A Local Search Approximation Algorithm for k-Means Clustering\n\n\n[2] Simpler Analyses of Local Search Algorithms for Facility Location\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe following results are given in the paper: \n1. A tighter analysis of the local search algorithm of Lattanzi and Sohler. The paper shows a constant approximation guarantee for their algorithm.\n2. The paper extends the approach of Lattanzi and Sohler to multi-swap local search (where more than one point is swapped in every iteration). Using this approach, an approximation guarantee of 10.48 is obtained.\n3. The paper also gives a (9+\\eps) approximation. Such a 9-approximation algorithm using local search is already known (https://www.sciencedirect.com/science/article/pii/S0925772104000215). The algorithm given in this paper has a slight running time advantage.\n\nThe algorithm is a local search algorithm. The justification of the title is that the swap-in points in the local search strategy are obtained by D^2 sampling (i.e., the sampling strategy used in k-means++).\n\nSTRENGTHS:\n1. Knowing a better analysis for an already published algorithm is good.\n2. Extension to multi-swap and its analysis giving better approximation guarantee is interesting.\n3. Matching the best possible approximation (possible using local search) using a slightly better running time is also interesting.\n4. The experimental section shows that the local search iterations do better than Lloyd's. This is interesting and can be of practical use. It would have been better if the running time comparison was also given to make the tradeoffs more visible.\n\nWEAKNESSES:\n1. The paper looks good from the point of view of extensions to the Lattanzi and Sohler work. However, from the point of view of developments in local search algorithms (which is essentially what the paper is), the improvement over (https://www.sciencedirect.com/science/article/pii/S0925772104000215) seems incremental. Even the proof techniques are similar.\n2. There are some discrepancies in the experimental section. For example, Figure 2 says \"first row...\", but there is none. I did not find the running time for the local search iterations (even MSLS-G).\n\nQUESTIONS:\n- Some of the other queries are mentioned above within the other fields.\n\nLIMITATIONS:\nThis work is mostly theoretical. There are no negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}