Title: Linear Time Algorithms for k-means with Multi-Swap Local Search
Venue: NeurIPS 2023
Accepted: yes
Keywords: Approximation Algorithms

k-means Clustering

Local Search
Abstract: The local search methods have been widely used to solve the clustering problems. In practice, local search algorithms for clustering problems mainly adapt the single-swap strategy, which enables them to handle large-scale datasets and achieve linear running time in the data size. However, compared with multi-swap local search algorithms, there is a considerable gap on the approximation ratios of the single-swap local search algorithms. Although the current multi-swap local search algorithms provide small constant approximation, the proposed algorithms tend to have large polynomial running time, which cannot be used to handle large-scale datasets. In this paper, we propose a multi-swap local search algorithm for the $k$-means problem with linear running time in the data size. Given a swap size $t$, our proposed algorithm can achieve a $(50(1+\frac{1}{t})+\epsilon)$-approximation, which improves the current best result 509 (ICML 2019) with linear running time in the data size. Our proposed method, compared with previous multi-swap local search algorithms, is the first one to achieve linear running time in the data size. To obtain a more practical algorithm for the problem with better clustering quality and running time, we propose a sampling-based method which accelerates the process of clustering cost update during swaps. Besides, a recombination mechanism is proposed to find potentially better solutions. Empirical experiments show that our proposed algorithms achieve better performances compared with branch and bound solver (NeurIPS 2022) and other existing state-of-the-art local search algorithms on both small and large datasets.

Linear Time Algorithms for k -means with Multi-Swap Local Search

Abstract

The local search methods have been widely used to solve the clustering problems. In practice, local search algorithms for clustering problems mainly adapt the singleswap strategy, which enables them to handle large-scale datasets and achieve linear running time in the data size. However, compared with multi-swap local search algorithms, there is a considerable gap on the approximation ratios of the single-swap local search algorithms. Although the current multi-swap local search algorithms provide small constant approximation, the proposed algorithms tend to have large polynomial running time, which cannot be used to handle large-scale datasets. In this paper, we propose a multi-swap local search algorithm for the k -means problem with linear running time in the data size. Given a swap size t , our proposed algorithm can achieve a (50(1 + 1 t ) + ϵ ) -approximation, which improves the current best result 509 (ICML 2019) with linear running time in the data size. Our proposed method, compared with previous multi-swap local search algorithms, is the first one to achieve linear running time in the data size. To obtain a more practical algorithm for the problem with better clustering quality and running time, we propose a sampling-based method which accelerates the process of clustering cost update during swaps. Besides, a recombination mechanism is proposed to find potentially better solutions. Empirical experiments show that our proposed algorithms achieve better performances compared with branch and bound solver (NeurIPS 2022) and other existing state-of-the-art local search algorithms on both small and large datasets.

1 Introduction

Clustering is a fundamental problem in the field of machine learning with many real-world applications. The goal of clustering is to partition a given set of data points into different clusters according to their similarity such that data points within the same cluster share high similarity as much as possible. Among different objective functions, the k -means clustering aims to minimize the sum of the squared distances between data points to their closest centers. More formally, given a set P ⊂ R d

of data points in a d -dimensional Euclidean space, the goal of the k -means clustering is to find a set C ⊂ R d of size at most k with the following objective: min C ⊂ R d ∑ p ∈ P min c ∈ C ∥ p -c ∥ 2 .

For the k -means problem, Lloyd's algorithm [12] is one of the most widely used heuristic in practice. However, there is no theoretical guarantee for Lloyd-type method unless certain data distribution assumptions are introduced. It is known that there are several constant approximation schemes based on primal-dual and randomized rounding techniques [2, 5, 8]. The current best approximation ratio in polynomial time is 5.912 [5], which is based on primal-dual method and nested quasi-independent set. For fixed dimension d or the number of clusters k , several (1 + ϵ ) -approximation algorithms were proposed [6, 7].

The k -means++ algorithm proposed by Arthur and Vassilvitskii [3] is a good seeding method that runs in linear time in the data size with O (log k ) -approximation. It is also known that the k -means++ algorithm gives a constant approximation by opening O ( k ) centers [1, 13, 16]. Lattanzi and Sohler [11] showed a combination of k -means++ seeding and the local search algorithm (named as LS++ algorithm), which yields a constant approximation with running time O ( ndk 2 log log k ) . In each round, LS++ algorithm samples a data point using k -means++ seeding and enumerates possible swap pairs to make improvements on clustering cost. They proved that after O ( k log log k ) rounds of sampling and swaps, one can obtain a 509 -approximate solution in expectation. Choo et al. [4] proved that one can achieve an O (1 /ϵ 3 ) -approximation using reduced O ( ϵk ) rounds of LS++ algorithm. Under the assumption that each optimal cluster has size Ω( n/k ) , Huang et al. [9] gave an improved approximation algorithm with ratio (100 + ϵ ) by random sampling methods.

However, there are still several issues for local search methods. Although current local search methods with multi-swap strategy can achieve good theoretical guarantee, the running time of them have polynomial dependence on the data size, which are hard to be used to handle large-scale datasets. Compared with the (9 + ϵ ) -approximation multi-swap local search algorithm given in [10], the approximation ratio 509 of LS++ is a large constant since it can only apply the single-swap strategy. Numerical experiments [9] showed that LS++ could easily fall into a poor local optimum when handling real-world datasets. An immediate idea is to apply the multi-swap strategy to LS++ algorithm for improvements. However, the swapping process of LS++ relies heavily on the one-to-one matched swap pairs defined in [11]. Thus, it is challenging to apply the multi-swap strategy to solve the k -means problem while maintaining a linear dependence of data size on the running time.

Secondly, in the process of local search swaps, the time and space complexities for clustering cost update during swaps have great impact on the efficiency of the algorithms. A direct way in [11] is to maintain the nearest and the second nearest centers for each data point such that picking the best swap pair and updating the clustering cost can be implemented in time O ( nd ) and O ( ndk ) , respectively. Maintaining the distances from data points to their centers requires an extra space complexity of O ( nd ) . To obtain faster implementation, as pointed out in [4], one can use binary search trees to store the distances from each data point to each of the clustering centers. By using this data structure, each local search step can be implemented in time O ( nd log k ) . However, the space complexity becomes O ( ndk ) . To get much practical algorithms for the k -means problem, it is necessary to further improve the time and space complexities for updating the clustering cost during swaps.

1.1 Our Contributions

In order to further narrow the gap between theory and practice, in this paper, we propose the first multi-swap local search algorithm for the k -means problem with linear running time in the data size. A common feature for the existing multi-swap local search methods is that Θ(( nk ) t ) candidate swaps should be enumerated for finding clustering cost improvements in a single local search step, which leads to at least quadratic running time. To overcome this challenge, our idea is to use a sampling-based strategy to construct a candidate set of centers that are close to the optimal clustering centers in linear time, which serves as the set of potentially good centers for swapping in. Based on the candidate set of centers constructed, enumerations on the current set of centers opened suffice to determine a good swap. Hence, the number of possible swaps can be reduced from Θ(( nk ) t ) to Θ( k t ) , where each local search step can be conducted in linear-time in the data size.

Sampling-based strategy has been used in [11] for designing single-swap local search algorithms. However, the theoretical analysis relies heavily on the one-to-one matched swap pairs. Thus, the approximation ratio is a large constant since there may exist some optimal clusters that cannot be well

approximated by performing matched swaps. In this paper, to obtain better approximation guarantee, we extend the notions of swap pairs and propose a new consecutive sampling method to construct candidate centers for swap such that data points close to a subset of optimal clustering centers can be swapped in simultaneously. A key challenge here is that there may exist some optimal clusters whose clustering costs only take a tiny fraction of the total clustering cost such that sampling methods may fail. To overcome this challenge, we propose new structures that divide optimal clusters into different groups for establishing a lower bound for the success probability of sampling.

By using the proposed multi-swap local search method (denoted as MLS algorithm for short), given a swap size t , an improved (50(1 + 1 t ) + ϵ ) -approximation can be obtained in time O ( ndk 2 t +1 log( ϵ -1 log k )) . To benefit more from the proposed multi-swap local search method when handling large-scale datasets, we propose a sampling-based method for accelerating the MLS algorithm. The proposed algorithm (denoted as MLSP algorithm) accelerates the updating of clustering cost during the swaps such that each local search iteration in MLS algorithm can be implemented in time O ( ndk + poly ( k ) d ) with extra space complexity of ˜ O ( kd ) . In order to obtain better clustering quality, we develop a recombination mechanism in MLSP which combines sampling and scoring methods to help the local search algorithm find better solutions when the search falls into a poor local optimum. By picking the topk data points with the highest scores as new initialization, the chance to get out of the local optimal solutions becomes large. Numerical experiments show that our proposed method achieves better performances compared with branch and bound solver and other local search algorithms. The main contributions of this paper are as follows.

We propose the first multi-swap local search algorithm (MLS algorithm) with running time linearly dependent on the data size. Given a swap size t with t ≥ 2 , our MLS algorithm achieves a (50(1 + 1 t ) + ϵ ) -approximation in time O ( ndk 2 t +1 log( ϵ -1 log k )) , which improves the current best approximation ratio 509 with linear running time in the data size.

We give a practical heuristic algorithm (MLSP algorithm) for better implementation of the proposed MLS algorithm, which accelerates the process of clustering cost update during swaps and provides better scalability to large-scale datasets. Besides, a recombination mechanism is proposed to prevent the local search algorithm falling into a poor local optimum too early.

Table 1 shows a detailed comparison of our results with the state-of-the-art ones.

Table 1: Comparison with related results on k -means clustering, where n is the size of the given dataset, d is the dimension, k is the number of clusters opened, t is the parameter representing the swap size of local search methods, and ∆ is the aspect ratio (aspect ratio is defined as the maximum pairwise distance of the given instance divided by the minimum pairwise distance).

2 Preliminaries

We use P ⊂ R d and k to denote the given dataset and the number of clusters, respectively. For any two points p , q ∈ P , we use d ( p, q ) = ∥ p -q ∥ 2 to denote the squared distance between them. Given two sets A , B ⊆ P , let ∆( A,B ) = ∑ p ∈ A min q ∈ B d ( p, q ) denote the sum of the squared distances from data points in A to their closest points in B . Let C ∗ = { c ∗ 1 , c ∗ 2 , ..., c ∗ k } be an optimal solution. We use P ( C ∗ ) = { P ∗ 1 , P ∗ 2 , ..., P ∗ k } to denote the corresponding optimal clusters by assigning data points in P to their closest centers in C ∗ . Let Opt be the cost of the optimal solution. For a subset Q ⊆ P ( C ∗ ) of optimal clusters, we use Z ( Q ) = ∪ P ∗ h ∈ Q P ∗ h to denote the set of data points in clusters of Q . Denote Z ′ ( Q ) = { c ∗ h : P ∗ h ∈ Q } as the set of optimal centers of clusters in Q . Given a subset S ⊆ C ∗ , let J ( S ) = { P ∗ h : c ∗ h ∈ S } be the collection of optimal clusters whose clustering centers are in G . For an integer t , let [ t ] = [1 , 2 , ..., t ] . The following lemma is a folklore for the k -means problem.

Lemma 1 [3] Let P ⊆ R d be a set of data points, and µ ( P ) = 1 | P | ∑ p ∈ P p denote the center of gravity. For any data point c ∈ R d , we can get ∆( P, { c } ) = | P | d ( µ ( P ) , c ) + ∆( P, { µ ( P ) } ) .

Theorem 1 [3] Algorithm 1 returns an O (log k ) -approximate solution in time O ( ndk ) .

Algorithm 1 k -means++

: An instance

Input

( P, k ) of the k -means problem.

Output : A set C ⊆ R d of centers with size at most k .

1: Randomly sample a point p ∈ P and set C = { p } .

2: for i = 1 to k -1 do

3: Pick a point p ∈ P with probability ∆( { p } , C ) / ∆( P, C ) , and add p to C .

4: return C .

3 Linear Time Local Search Algorithm with Multi-Swap Strategy

The general idea solving the k -means problem with multi-swap local search is that we propose a new consecutive sampling method to construct candidate centers for swap such that data points close to a subset of optimal clustering centers can be swapped in simultaneously. The multi-swap local search algorithm (denoted as MLS) is given in Algorithm 2. There are mainly two stages in each round of the MLS algorithm. Given a swap size t , in the first stage (steps 3 to 5), a candidate set of centers with size t will be sampled using the k -means++ method. This avoids enumerating all the data points for constructing the candidate sets for swapping in. In the second stage (steps 6 to 7), the algorithm enumerates all subsets (with size at most t ) of the candidate set of centers opened for swapping out. By extending the notions of swap pairs to swap set and carefully analyzing the structures of local optimal solutions, we prove that the clustering cost can be reduced significantly with certain probability in each iteration of the MLS Algorithm. The following is the main result of this paper.

Theorem 2 In the i -th iteration of Algorithm 2, let C ′ be the set of centers obtained in step 7. If the current clustering cost ∆( P, C ) is larger than 50(1 + 1 t ) Opt , then with probability at least Ω( k -t ) , we have ∆( P, C ′ ) ≤ (1 -1 100 k )∆( P, C ) . After O ( k O ( t ) log( ϵ -1 log k )) iterations, we get an approximate solution with ratio (50(1 + 1 t ) + ϵ ) in expectation 2 .

3.1 Analysis

In this subsection, we analyze our proposed Algorithm 2, where a candidate set of centers for swap is constructed by t independent sampling steps in each iteration. Our objective is to show that the clustering cost can be reduced with certain probability in each iteration. Due to space limit, all the detailed proofs are given in Appendix A.

In the following, we consider a single iteration of the proposed MLS algorithm. Assume ∆( P, C ) ≥ 50(1 + 1 t ) Opt holds within a single iteration in Algorithm 2. Otherwise, C is already a 50(1 + 1 t ) -approximate solution for P . Let C = { c 1 , c 2 , ..., c k } denote the set of centers before the swap (steps 6-7) of Algorithm 2. Let P ( C ) = { P 1 , P 2 , ..., P k } be the corresponding partition of clusters induced by C . For an optimal cluster P ∗ h , let c ∗ h be its clustering center. For each cluster P h ∈ P ( C ) , let c h ∈ C denote its clustering center. Following the ones in [11], we extend the definition of good clusters with respect to C ∗ as follows.

Definition 1 Good single cluster. A cluster P ∗ h ∈ P ( C ∗ ) is called good with a pair of points ( c ∗ h , c j ) such that c j ∈ C and ∆( P ∗ h , C ) -ζ ( P, C, c ∗ h , c j ) -9∆( P ∗ h , { c ∗ h } ) > 1 100 k ∆( P, C ) , where ζ ( P, C, c ∗ h , c j ) = ∆( P \ P ∗ h , C \{ c j } ) -∆( P \ P ∗ h , C ) is the reassignment cost by swapping c j out. Otherwise we say that P ∗ h is a bad single cluster with ( c ∗ h , c j ) .

Definition 2 Good t -Clusters. Given an integer t with t ≥ 2 , for a collection of optimal clusters Q ⊆ P ( C ∗ ) with | Q | ≤ t , Q is called good with a pair of sets ( Z ′ ( Q ) , V ) such that V ⊆ C , | V | = | Z ′ ( Q ) | and ∆( Z ( Q ) , C ) -ζ ( P, C, Z ′ ( Q ) , V ) -9∆( Z ( Q ) , C ∗ ) > 1 100 k ∆( P, C ) , where ζ ( P, C, Z ′ ( Q ) , V ) = ∆( P \ Z ( Q ) , C \ V ) -∆( P \ Z ( Q ) , C ) is the reassignment cost by swapping the points in V out. Otherwise we say that Q is a set of bad t -clusters with ( Z ′ ( Q ) , V ) .

Algorithm 2 MLS

The above definitions estimate the changes of clustering cost by replacing a set of centers V ⊆ C with a set Q ′ of data points that are close to a subset of optimal clustering centers, where a new clustering is constructed by reassigning data points in Z ( Q ) to Q ′ and data points in P \ Z ( Q ) to C \ V . The main idea behind Algorithm 2 is to iteratively find data points close enough to the optimal clustering centers for swap to make a reduction on clustering cost by at least (1 -Θ( 1 k )) . In the following, we will prove that with probability at least Ω( k -t ) , the sampling and swap process induces a significant reduction on clustering cost in each iteration.

We start by dividing the optimal clusters into several groups to give an upper bound of reassignment costs. Given a swap size t , for each optimal center c ∗ h ∈ C ∗ , we define Ψ( c ∗ h ) as a mapping function that maps c ∗ h to its closest center in C . For simplicity, we say that c ∗ h is captured by Ψ( c ∗ h ) . For a center c j ∈ C , let Ψ -1 ( c j ) be the set of optimal centers captured by c j . If | Ψ -1 ( c j ) | = 0 , then c j is called a lonely center. If | Ψ -1 ( c j ) | = 1 , let c ∗ h = Ψ -1 ( c j ) . Then it is called that ( c ∗ h , c j ) forms a type-1 matched swap pair. If 1 < | Ψ( c j ) | ≤ t , let σ h be an arbitrary set of unused lonely centers with | σ h | = | Ψ -1 ( c j ) | -1 . Let A = Ψ -1 ( c j ) and A ′ = { c j } ∪ σ h . Then, it is called that ( A,A ′ ) forms a type-1 matched swap set. Let M 1 = { ( c ∗ h , c j ) : ( c ∗ h , c j ) is a type-1 matched swap pair } and M 2 = { ( A,A ′ ) : ( A,A ′ ) is a type-1 matched swap set } be the collections of type-1 matched swap pairs and type-1 matched swap sets, respectively. If | Ψ -1 ( c j ) | > t , then find each lonely center c q ∈ C that has not been used for constructing type-1 matched swap pairs or sets. For each lonely center c q and each c ∗ h ∈ Ψ -1 ( c j ) , ( c ∗ h , c q ) forms a type-2 matched swap pair. We use M 3 = { ( c ∗ h , c q ) : ( c ∗ h , c q ) is a type-2 matched swap pair } to denote the set of type-2 matched swap pairs. For a set V ⊆ C of centers, let X ( V ) = ∪ c h ∈ V P h be the set of data points in P whose closest centers are in V . Given a subset S ⊆ C of clustering centers, we also use J ( S ) = { P h : c h ∈ G } to denote the set of clusters whose centers are in S . The following lemma gives upper bounds of reassignment cost by matched swap pairs or sets.

Lemma 2 Given a type-1 or type-2 matched swap pair ( c ∗ h , c j ) , it holds that ζ ( P, C, c ∗ h , c j ) ≤ 24∆( P j , C ∗ ) + 1 5 ∆( P j , C ) . Given a type-1 matched swap set ( Q,V ) , it holds that ζ ( P, C, Q, V ) ≤ 24∆( X ( V ) , C ∗ ) + 1 5 ∆( X ( V ) , C ) .

Let H 1 = { c ∗ h : ( c ∗ h , c j ) ∈ M 1 } be the set of optimal centers that participate in constructing type-1 matched swap pair. Let H 2 = { A : ( A,A ′ ) ∈ M 2 } be the collection of the subsets of optimal centers that participate in constructing type-1 matched swap set. Let L = { c ∗ h : ( c ∗ h , c q ) ∈ M 3 } be the set of optimal centers that participate in constructing type-2 matched swap pair. Let H ′ 1 = { c j : ( c ∗ h , c j ) ∈ M 1 } be the set of centers in C that participate in constructing type-1 matched swap pair. Let H ′ 2 = { A ′ : ( A,A ′ ) ∈ M 2 } be the collection of the subsets of centers in C that participate in constructing type-1 matched swap set. Let L ′ = { c q : ( c ∗ h , c q ) ∈ M 3 } denote the set of centers in C that participate in constructing type-2 matched swap pair.

During the sampling and swap process in steps 4-7 of Algorithm 2, there are two cases that may happen: (1) ∃ c ∗ h ∈ L such that P ∗ h is a good single cluster with a type-2 matched swap pair ( c ∗ h , c q ) ∈ M 3 ; (2) ∀ c ∗ h ∈ L , P ∗ h is a bad single cluster with any type-2 matched swap pair ( c ∗ h , c q ) ∈ M 3 . We will discuss the two cases separately in the following. If case (1) happens, we first show that with probability at least Ω( 1 k ) , the clustering cost can be reduced at least by 1 -Θ( 1 k ) . Let c ∗ h ∈ L be an optimal center such that P ∗ h is a good single cluster with a type-2 matched swap pair ( c ∗ h , c q ) ∈ M 3 . By the definition of good single cluster, we have ∆( P ∗ h , C ) ≥ 9∆( P ∗ h , { c ∗ h } ) .

Define χ ( P ∗ h ) = { p ∈ P ∗ h : d ( p, c ∗ h ) ≤ 1 . 5∆( P ∗ h , { c ∗ h } ) | P ∗ h | } as the set of data points in P ∗ h that are close to the optimal center c ∗ h . Observe that | χ ( P ∗ h ) | ≥ 1 3 | P ∗ h | . Otherwise, the clustering cost of data points in P ∗ h \ χ ( P ∗ h ) is at least ∆( P ∗ h , { c ∗ h } ) using c ∗ h as center, which contradicts with ∆( P ∗ h \ χ ( P ∗ h ) , { c ∗ h } ) < ∆( P ∗ h , { c ∗ h } ) .

Next, we will show that whenever an optimal cluster P ∗ h has large clustering cost with respect to C , i.e., ∆( P ∗ h , C ) = b ∆( P ∗ h , { c ∗ h } ) for a real number b ≥ 3 , it suffices to use data points in χ ( P ∗ h ) to approximate the clustering cost of P ∗ h .

Lemma 3 Let P ∗ h be an optimal cluster with ∆( P ∗ h , C ) = b ∆( P ∗ h , { c ∗ h } ) for a real number b ≥ 3 . Then, ∆( χ ( P ∗ h ) , C ) ≥ 1 200 ( b -1)∆( P ∗ h , { c ∗ h } ) .

We now argue that the clustering cost of each good single cluster takes a certain fraction of the total clustering cost. Then, by sampling according to the squared distances, with good probability, data points close to the center of a good single cluster can be sampled. According to the definition of good single cluster, we can assume that ∆( P ∗ h , C ) = b ∆( P ∗ h , { c ∗ h } ) for a real number b ≥ 9 . By Lemma 3, we know that ∆( χ ( P ∗ h ) , C ) ≥ b -1 200 ∆( P ∗ h , { c ∗ h } ) = b -1 200 b ∆( P ∗ h , C ) ≥ 1 300 ∆( P ∗ h , C ) . By the definition of good single cluster, we also have ∆( P ∗ h , C ) ≥ 1 100 k ∆( P, C ) , which implies that ∆( χ ( P ∗ h ) , C ) ≥ 1 30000 k ∆( P, C ) . Thus, in each round of the sampling process in steps 4-5 of Algorithm 2, with probability at least Ω( 1 k ) , we can sample a point q ∈ χ ( P ∗ h ) for a good single cluster P ∗ h with a type-2 matched swap pair ( c ∗ h , c q ) ∈ M 3 . The following lemma shows that, by swapping q with c q and assigning all the data points in P ∗ h to q , the clustering cost can be reduced at least by 1 -Θ( 1 k ) .

Lemma 4 By swapping q with c q , the clustering cost of ∆( P, C ) can be reduced at least by 1 -Θ( 1 k ) .

We have shown that if case (1) happens, we can sample a data point q close to the center of a good single cluster P ∗ h to make the clustering cost reduced significantly. Next, we assume that case (1) never happens and case (2) happens. In case (2), the idea behind is to sample data points close to the optimal centers in H 1 or sets of optimal centers in H 2 to reduce the clustering cost. We first bound the clustering cost of optimal clusters in J ( L ) .

Lemma 5 If case (2) happens, then ∆( X ( L ) , C ) ≤ (1 + 1 t )(9∆( X ( L ) , C ∗ ) + 24∆( X ( L ′ ) , C ∗ ) + 1 5 ∆( X ( L ′ ) , C ) + 1 100 ∆( P, C )) .

Now, consider a set of centers Q ∈ H 2 . We will divide the optimal clusters in J ( Q ) into two groups. Let Q L = { P ∗ h ∈ J ( Q ) : ∆( P ∗ h , C ) ≥ 1 30000 k 2 t -1 ∆( P, C ) } be the set of optimal clusters in J ( Q ) with large clustering cost with respect to C and Q S = { P ∗ h ∈ J ( Q ) : ∆( P ∗ h , C ) < 1 30000 k 2 t -1 ∆( P, C ) } be the set of optimal clusters in J ( Q ) with small clustering cost with respect to C , respectively. We define Q ′ S = { P ∗ h ∈ Q S : ∆( P ∗ h , C ) < 3∆( P ∗ h , { c ∗ h } ) } as the set of optimal clusters in Q S whose clustering centers are close to one of the centers in C . Let Q ′′ S = Q S \ Q ′ S and Q T = Q L ∪ Q ′′ S , respectively. We first show that, it suffices to only consider optimal clusters in Q T .

̸

Lemma 6 Let Q ∈ H 2 be a set of centers in H 2 , where J ( Q ) is a set of good t -clusters with a type-1 matched swap set ( Q,A ′ ) ∈ M 2 . Define V = A ′ \{ c j } , where c j is the center in A ′ with | Ψ -1 ( c j ) | > 1 . Let U ⊆ P be the set of data points with | U | = | V | such that U ∩ χ ( P ∗ h ) = ∅ holds for each P ∗ h ∈ Q T . Then, ∆( P, C \ V ∪ U ) ≤ (1 -1 100 k )∆( P, C ) .

̸

Note that there are t sampling iterations in each step 4 of Algorithm 2. Let H ∗ 2 = { P ∗ h : P ∗ h ∈ Q T , Q ∈ H 2 } and H t = J ( H 1 ) ∪ H ∗ 2 . We will define a mapping function m which maps each P ∗ h ∈ H t to an integer m ( P ∗ h ) ∈ [ t ] . For each P ∗ h ∈ H t such that c ∗ h ∈ H 1 , we define m ( P ∗ h ) = 1 . Then, consider each Q ∈ H 2 . For each P ∗ h ∈ Q T , we define m ( P ∗ h ) = i such that i ∈ [ | Q T | ] and m ( P ∗ h ) = m ( P ∗ j ) for any two optimal clusters P ∗ h , P ∗ j ∈ Q T . For each Q ∈ H 2 , let p s ( Q ) be the success probability that for each P ∗ h ∈ Q T , a data point is sampled from χ ( P ∗ h ) in the m ( P ∗ h ) -th iteration of Algorithm 2. For each c ∗ h ∈ H 1 , let p s ( c ∗ h ) be the success probability that a data point is sampled from χ ( P ∗ h ) in the first iteration (note that in this case m ( P ∗ h ) = 1 ) of step 4. Let H 1 G = { c ∗ h ∈ H 1 : P ∗ h is a good single cluster with the type-1 matched swap pair ( c ∗ h , c j ) ∈ M 1 } be the set of centers in H 1 whose optimal clusters are good single clusters with type-1 matched swap pairs in M 1 . Let H 2 G = { Q ∈ H 2 : J ( Q ) is a set of good t -clusters with the type-1 matched swap set ( Q,V ) ∈ M 2 } be the collection of the sets of centers in H 2 whose corresponding optimal clusters are good

t -clusters with type-1 matched swap sets in M 2 . Define p f s = ∑ c ∗ h ∈ H 1 G p s ( c ∗ h ) + ∑ Q ∈ H 2 G p s ( Q ) as the summation of the success probability. Since all the optimal clusters in J ( L ) belong to bad single cluster, and the events defined related to p s ( Q ) or p s ( c ∗ h ) for each Q ∈ H 2 and c ∗ h ∈ H 1 are mutually exclusive, p f s gives a lower bound success probability to get a (1 -1 100 k ) reduction on the clustering cost in each iteration of steps 2-7. In the following, we will show how to obtain a lower bound for p f s .

Consider a set Q ∈ H 2 of optimal centers such that J ( Q ) is a set of good t -clusters with a type-1 matched swap set ( Q,A ′ ) . There are two subcases that may happen: (1) ∃ P ∗ h ∈ J ( Q ) such that P ∗ h is a good single cluster with a swap pair ( c ∗ h , c j ) , where c j is a lonely center used for constructing type-1 matched swap set; (2) ∀ P ∗ h ∈ J ( Q ) , P ∗ h is a bad single cluster with any swap pair ( c ∗ h , c j ) , where c j is a lonely center used to construct type-1 matched swap set. In subcase (1), since t is usually a constant and could be much smaller than k , with probability at least Ω( k -1 ) , a data point q ∈ χ ( P ∗ h ) can be sampled in the first iteration of step 4 in Algorithm 2 for swap to make the clustering cost reduced at least by (1 -1 100 k ) according to Lemma 4. Next, we assume that subcase (1) never happens and subcase (2) happens. In subcase (2), our objective is to sample a set V of data points such that V contains at least one point from χ ( P ∗ h ) for each P ∗ h ∈ Q T . For each optimal cluster P ∗ h ∈ Q L , with probability at least Ω( k -1 ) , we can sample a data point q ∈ χ ( P ∗ h ) in the m ( P ∗ h ) -th iteration of step 4 in Algorithm 2. Thus, the probability can be bounded by Ω( k -| Q L | ) . Then, we only need to consider optimal clusters in Q S . By Lemma 6, it suffices to consider optimal clusters in Q ′′ S . The following lemma gives an upper bound of the failure probability of not sampling a data point from χ ( P ∗ h ) for each P ∗ h ∈ Q ′′ S in the m ( P ∗ h ) -th iteration in step 4 of Algorithm 2.

Lemma 7 Given a set Q ∈ H 2 of centers such that J ( Q ) is a set of good t -clusters, the probability that step 4 of Algorithm 2 fails to sample a data point q from χ ( P ∗ h ) for each P ∗ h ∈ Q ′′ S in the m ( P ∗ h ) -th iteration is at most (1 + 1 30000 k ) e -∆( Z ( Q ′′ S ) ,C ) / (300∆( P,C )) .

Now we can bound the success probabilities of p s ( Q ) and p s ( c ∗ h ) for each Q ∈ H 2 and c ∗ h ∈ H 1 . We will divide the optimal clusters in P ( C ∗ ) \ J ( L ) into two different groups H G and H B . Firstly, we consider the centers in H 1 . For a center c ∗ h ∈ H 1 , if P ∗ h is a good single cluster with the type-1 matched swap pair ( c ∗ h , c j ) ∈ M 1 , then add P ∗ h to H G . Otherwise add P ∗ h to H B . For a set Q ∈ H 2 of optimal clustering centers, if J ( Q ) is a set of good t -clusters with the type-1 matched swap set ( Q,V ) ∈ M 2 , then add each P ∗ h ∈ Q ′′ S to H G and each P ∗ h ∈ J ( Q ) \ Q ′′ S to H B . If J ( Q ) is a set of bad t -clusters with the type-1 matched swap set ( Q,V ) ∈ M 2 , then add each P ∗ h ∈ J ( Q ) to H B . The following lemma argues that the summation clustering cost of the optimal clusters in H G is large.

Lemma 8 For the optimal clusters in H G , we have ∆( Z ( H G ) , C ) ≥ 1 100 ∆( P, C ) .

For a good single cluster P ∗ h with the type-1 matched swap pair ( c ∗ h , c j ) ∈ M 1 where c ∗ h ∈ H 1 , define p f ( c ∗ h ) = 1 -p s ( c ∗ h ) as the probability that the first iteration in step 4 of Algorithm 2 fails to sample a data point q ∈ χ ( P ∗ h ) . Then, we have p f ( c ∗ h ) ≤ e -p s ( c ∗ h ) ≤ e -∆( P ∗ h ,C ) / 300∆( P,C ) by Lemma 3. For a set Q ∈ H 2 G of optimal centers, we have that p f ( Q ′′ S ) is the failure probability of not sampling a data point from χ ( P ∗ h ) for each P ∗ h ∈ Q ′′ S in the m ( P ∗ h ) -th iteration of step 4 in Algorithm 2. Recall that p f s = ∑ c ∗ h ∈ H 1 G p s ( c ∗ h ) + ∑ Q ∈ H 2 G p s ( Q ) is the summation of success probability. Then, we have p f s ≥ Ω( k -t )( ∑ c ∗ h ∈ H 1 G p s ( c ∗ h ) + ∑ Q ∈ H 2 G p s ( Q ′′ S )) . Let p f ′ s = ∑ c ∗ h ∈ H 1 G p s ( c ∗ h ) + ∑ Q ∈ H 2 G p s ( Q ′′ S ) . Observe that p f ′ s ≥ 1 -∏ c ∗ h ∈ H 1 G (1 -p s ( c ∗ h )) ∏ Q ∈ H 2 G (1 -p s ( Q ′′ S )) . Since there are at most k 2 good t -clusters, by Lemma 7, we have p f ′ s ≥ 1 -(1 + 1 30000 k ) k 2 e -∆( Z ( HG ) ,C ) 300∆( P,C ) ≥ 1 -e k 2 ln(1+ 1 30000 k ) -∆( Z ( HG ) ,C ) 300∆( P,C ) ≥ 1 -e 1 60000 -1 30000 ≥ 1 -e -1 60000 ≥ 1 60001 , where the third inequality follows from Lemma 8 and ln(1 + x ) ≤ x . Then, it holds that p f s ≥ Ω( k -t ) p f ′ s = Ω( k -t ) , which indicates that with probability at least Ω( k -t ) , we can sample data points close to a set of good t -clusters or a good single cluster for swap to make the clustering cost reduced at least by (1 -1 100 k ) according to Lemma 6. Putting all things together, Theorem 2 can be proved (Detailed proof of Theorem 2 is given in Appendix A).

Running Time Analysis. By Theorem 2, in order to obtain a (50(1 + 1 t ) + ϵ ) -approximate solution, the iteration rounds for Algorithm 2 should be O ( k t +1 log( ϵ -1 log k )) . In each iteration, it takes O ( ndk ) time to update the distances between data points to their closest centers. During the sampling process, t data points are sampled according to the D 2 -Sampling distribution to serve as the candidate set of centers for swapping in, which takes time O ( nt ) if the distances from data points to their

centers are already known. It takes O ( k t ) time to enumerate each subset with size at most t of the set of current centers opened. It takes O ( ndk t ) time to recalculate the clustering cost after each swap if t nearest centers of each data point are maintained during the whole process. Thus, the total running time of Algorithm 2 is O ( ndk 2 t +1 log( ϵ -1 log k )) .

3.2 Accelerating Multi-Swap Local Search for k -means

In this subsection, we provide a more practical algorithm for the k -means problem to accelerate the proposed multi-swap local search process. The algorithm is given in Algorithm 3. The main idea behind is to use sampling-based methods to obtain fast clustering cost updating during swaps. In step 7 of Algorithm 3, a small sample set S of size 2 k ϵ log k η is randomly taken from P . For clustering cost updating, instead of calculating the clustering cost of all the data points in P , we use the clustering cost of S ⊆ P as an estimation. This reduces the time for picking the best swap pair during a single local search iteration from O ( nd ) to O ( poly ( k ) d ) . Then, in steps 16-18, we design a recombination method to find better initialization which prevents the local search algorithm from falling into a poor local optimum too early. In step 16 of Algorithm 3, we randomly take a set D ⊆ P of centers with size O ( k log k ) . Let C 1 = C ∪ D be the set of the new candidate centers. For each center c h ∈ C 1 , we add a score of ∆( P h , C ) / ∆( P, C ) to it. Similarly, for each center c h ∈ C , we also add a score ∆( P h , C ) / ∆( P, C ) to it. Then, by giving each center in C 1 and C a score weight of 0.75 and 0.25, respectively, we pick the topk data points with the highest scores as a new initialization of clustering centers to find potentially better clustering costs until convergence.

Algorithm 3 MLSP

4 Experiments

In this section, we compare our proposed algorithms with the branch and bound solver and other local search methods. For hardware, all the experiments are conducted on 72 Intel Xeon Gold 6230 CPUs with 500GB memory.

Datasets We evaluate the performance of our algorithms on 8 datasets used in [15] with sizes over 50,000, two datasets SUSY (5,000,000 × 17) and HIGGS (11,000,000 × 27) from the UCI Machine

Learning Repository 3 , and one dataset SIFT (100,000,000 × 128) with size 100,000,000 used in [14]. In Appendix B, we give experiments on small datasets with sizes smaller than 50 , 000 used in [15].

Experimental Setup In our experiments, we choose centers from the datasets such that the problem can be solved by branch and bound solver (denoted as BB for short), which can serve as references of optimal solutions. We compare our MLSP and MLS algorithms with the BB method and other local search algorithms. Following the settings in [15], the centers obtained by different algorithms are projected to their closest data points in P , and the number of clusters k is set to be 3 , 5 and 10 . Each algorithm is executed for 10 times, and the average results with deviation, the best results and the average running time are given. We test the performance of different choices of parameters in Appendix B. We also conduct the experiments with fixed running time in Appendix B.

Algorithms In our experiments, we consider six algorithms. The first is the BB method in [15], which is the state-of-the-art solver for handling large-scale datasets. For local search algorithms, we compare our MLS and MLSP algorithms with the LS++ algorithm in [11] and FLS algorithm in [9], using the Lloyd's algorithm [12] as a baseline. For MLS algorithm, we use the sampling-based method designed for MLSP to accelerate the swapping process. For MLSP and MLS algorithms, we set the sampling rounds as T = 400 with a swap size t = 2 . For MLSP algorithm, the failure upper bound is set to be R ′ = 5 . For parameters ϵ and η , we fix them as 0 . 5 . For fair comparison, we also set the number of sampling rounds as 400 for LS++. Following the settings in [11], for all local search algorithms, the Lloyd's algorithm [12] is used as the final step to adjust the centers.

Method

Dataset

SampleSize

BB(Cost)

Best Cost

Average Cost

Time(s)

Dataset

SampleSize

BB(Cost)

Best Cost

Average Cost

Time(s)

LS++

rds

50,000*3

132.73

132.75

138.66

±

4.02

6.64

SPNET_3D

434,874*3

OOM

2.5720E+06

2.5801E+06

±

7.2E+03

55.28

FLS

131.74

136.19

±

2.78

27.25

2.5687E+06

2.5720E+06

±

1.8E+03

609.22

Lloyd

132.73

138.01

±

3.96

0.07

2.5702E+06

2.5816E+06

±

8.0E+03

0.62

MLSP

131.74

132.03

±

0.45

58.61

2.5677E+06

2.5688E+06

±

4.7E+02

561.06

MLS

131.78

136.71

±

2.76

5.12

2.5696E+06

2.5778E+06

±

7.3E+03

25.39

LS++

KEGG

53,413*23

6.1564E+07

6.1546E+07

6.2916E+07

±

1.5E+06

4.39

1,000,000*2

OOM

560502.36

561679.14

±

1177.73

123.86

FLS

6.1534E+07

6.5554E+07

±

1.1E+06

20.99

562756.29

561856.27

±

1034.52

1087.1

Lloyd

6.1564E+07

6.3870E+07

±

2.3E+06

0.21

562761.91

571251.23

±

5796.86

0.89

MLSP

6.1534E+07

6.1546E+07

±

3.7E+04

59.71

560502.36

560737.88

±

706.23

789.83

MLS

6.1546E+07

6.3076E+07

±

1.9E+06

5.68

560502.36

561444.42

±

1153.94

60.61

LS++

Urban_10

100,000*2

25123

24787

25209

±

575

10.39

USC_1990

2,458,685*68

OOM

2.7091E+08

2.7302E+08

±

2.9E+06

817.52

FLS

24786

24826

±

328

54.12

2.7082E+08

2.7083E+08

±

9.4E+03

7148.45

Lloyd

24663

25484

±

696

0.09

2.7100E+08

2.8497E+08

±

9.5E+06

40.32

MLSP

24659

24716

±

58

71.91

2.7073E+08

2.7079E+08

±

7.8E+03

5687.57

MLS

24659

24889

±

257

5.96

2.7082E+08

2.7082E+08

±

2.4E+03

619.16

LS++

RNG_AGR

199,843*7

1.3678E+14

1.3738E+14

1.3917E+14

±

2.2E+12

28.13

SUSY

5,000,000*17

OOM

3.2738E+07

3.2875E+07

±

1.1E+05

827.21

FLS

1.3663E+14

1.3745E+14

±

8.0E+11

206.67

3.1632E+07

3.1672E+07

±

2.8E+05

9287.26

Lloyd

1.3671E+14

1.3799E+14

±

2.7E+12

0.47

3.1639E+07

3.1668E+07

±

3.1E+05

31.87

MLSP

1.3663E+14

1.3685E+14

±

2.0E+11

201.21

3.1575E+07

3.1633E+07

±

3.8E+04

7462.57

MLS

1.3701E+14

1.3906E+14

±

1.8E+12

14.24

3.2219E+07

3.2424E+07

±

1.4E+05

534.11

LS++

Urban_GB

360,177*2

OOM

88720

89677

±

1370

30.78

HIGGS

11,000,000*27

OOM

1.8604E+08

1.8834E+08

±

1.4E+06

2424.97

FLS

88329

89233

±

890

183.4

1.8938E+08

1.8964E+08

±

1.5E+05

39826.29

Lloyd

88346

92595

±

4131

0.41

1.8461E+08

1.8568E+08

±

1.2E+06

171.06

MLSP

88329

88935

±

668

175.75

1.8373E+08

1.8410E+08

±

1.1E+05

21928.97

MLS

88329

89557

±

1333

13.07

1.8623E+08

1.8686E+08

±

5.5E+05

2037.68

LS++

SIFT

100,000,000*128

OOM

1.3994E+13

1.4014E+13

±

8.2E+10

121260

FLS

1.3905E+13

1.3985E+13

±

5.6E+10

140560

Lloyd

1.3718E+13

1.3803E+13

±

9.6E+10

566.99

MLSP

-

-

>

48h

MLS

1.3715E+13

1.3883E+13

±

9.0E+10

15179

Results Table 2 shows the results on 11 datasets with sizes over 50,000 using k = 10 . For clustering cost, on each dataset, the best clustering cost returned by our MLSP algorithm is smaller than BB method and other local search methods. For each dataset except for RNG_AGR, the average clustering cost returned by our MLSP algorithm outperforms the result of BB method. It can be seen that, as the sizes of datasets grow, BB method requires larger space complexity. When the size of dataset is over 360,000, it requires a memory of over 500GB, which is not practical for handling large-scale datasets. For LS++ method, it is more difficult to find high-quality solutions as the sizes of datasets grow. As for FLS, although the performance of FLS is better than that of LS++, our MLSP algorithm improves the performance of FLS on clustering cost with smaller deviation than FLS. By calculating the average values over all datasets, for MLSP algorithm, we can get that the clustering cost is reduced by 1.9%, 1.3% and 2.4% compared with LS++, FLS and Lloyd's algorithm, respectively. For MLS algorithm, on average, the clustering cost is reduced by 0.6% and 1.1% compared with LS++ and Lloyd's algorithm, respectively. As for running time, it can be seen that our proposed algorithm scales well as the sizes of datasets grow. By calculating the average values over all datasets, our proposed MLS algorithm is at least 1.79 times faster than LS++ algorithm.

The experimental results on the performances with varying T and R ′ (Appendix B.1) show that larger T and R ′ will not influence the results too much. In general, larger sampling rounds and larger failure upper bound can result in potentially better solutions with higher running time. For parameters ϵ and η , the results (Appendix B.1) show that the performances of our proposed algorithms are almost the same for different choices of ϵ and η . Tables 3, 4 and 5 show the comparison results of different algorithms with varying iteration rounds on dataset rds, KEGG and Urban_10, respectively. The results show that, our proposed MLSP algorithm always achieves the best clustering cost compared with LS++ and MLS algorithms. Tables 6, 7, and 8 show the results of MLSP algorithm with varying number of failure upper bound R ′ for fixed ϵ = 0 . 5 , η = 0 . 5 and T = 400 . It can be seen that a larger failure upper bound will lead to smaller deviation, and the running time becomes higher. Tables 9, 10, and 11 show the results of MLSP algorithm with varying parameters ϵ and η for fixed T = 400 and R ′ = 5 on dataset rds, KEGG, and Urban_10, respectively. It can be seen that smaller values of ϵ and η result in better performances on clustering cost with smaller deviation, and the running time becomes higher. The experimental results on small datasets (Appendix B.2) suggest that the proposed MLSP method not only outperforms other algorithms in terms of clustering quality but also runs much faster than the BB solver. The experimental results on the performances with different values of k (Appendix B.3) show that our proposed MLSP algorithm can still achieve the best clustering quality for smaller values of k . The experimental results on the performances with fixed time limit (Appendix B.4) show that our proposed MLSP algorithm achieves the best clustering quality within any given time constraints.

Tables 12 and 13 show the results on 18 datasets with sizes smaller than 50 , 000 using k = 10 . It can be seen that, for each dataset, the best clustering cost returned by our MLSP algorithm is smaller than BB method and other local search methods. For each dataset, the average clustering cost returned by our MLSP algorithm nearly matches the result of the BB method. As for FLS, although the performance of FLS is better than that of LS++, our proposed MLSP algorithm improves the performance of FLS on clustering cost with smaller deviation on most datasets. As for running time, there is no significant difference on running time among different local search algorithms for small datasets. Tables 14 and 15 present the performances of different local search algorithms with k = 3 on small datasets. Table 18 presents the performances of different local search algorithms with k = 3 on large datasets. Tables 16 and 17 present the performances of different local search algorithms with k = 5 on small datasets. Table 19 presents the performances of different local search algorithms with k = 5 on large datasets. It can be seen that, our proposed MLSP algorithm achieves the best clustering performance on most datasets with different values of k . On each dataset, the best clustering cost returned by our MLSP algorithm matches the clustering cost of BB method. On each dataset, the average clustering cost returned by our MLSP algorithm nearly matches the result of the BB method. As for running time, there is no significant difference on running time among all local search algorithms on small datasets. However, as the data sizes grow, our proposed MLS algorithm becomes much faster than other local search algorithms.

5 Conclusion

In this paper, we propose fast local search algorithms for the k -means problem with multi-swap strategy, which runs in linear time in the data size. We develop new sampling techniques, which accelerate the process of clustering cost update during swaps. By proposing a recombination mechanism, the proposed algorithm can find potentially better solutions. Experimental results show that our algorithms achieve better performance on both small and large datasets compared with the state-of-the-art algorithms. An interesting future direction is how to design fast local search approximation algorithms for handling high dimensional clustering datasets.

Acknowledgments

This work was supported by National Natural Science Foundation of China (62172446, 62350004, 62332020), Open Project of Xiangjiang Laboratory (22XJ02002, 22XJ03005), and Central South University Research Programme of Advanced Interdisciplinary Studies (2023QYJC023). This work was also carried out in part using computing resources at the High Performance Computing Center of Central South University.

References

Ankit Aggarwal, Amit Deshpande, and Ravi Kannan. Adaptive sampling for k -means clustering. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques , pages 15-28. 2009.

Sara Ahmadian, Ashkan Norouzi-Fard, Ola Svensson, and Justin Ward. Better guarantees for k -means and euclidean k -median by primal-dual algorithms. SIAM Journal on Computing , 49(4):17-97, 2019.

David Arthur and Sergei Vassilvitskii. k -means++: The advantages of careful seeding. In Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms , pages 10271035, 2007.

Davin Choo, Christoph Grunau, Julian Portmann, and Václav Rozhon. k -means++: Few more steps yield constant approximation. In Proceedings of the 37th International Conference on Machine Learning , volume 119, pages 1909-1917. PMLR, 2020.

Vincent Cohen-Addad, Hossein Esfandiari, Vahab S. Mirrokni, and Shyam Narayanan. Improved approximations for euclidean k -means and k -median, via nested quasi-independent sets. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1621-1628, 2022.

Vincent Cohen-Addad, Philip N Klein, and Claire Mathieu. Local search yields approximation schemes for k -means and k -median in euclidean and minor-free metrics. SIAM Journal on Computing , 48(2):644-667, 2019.

Zachary Friggstad, Mohsen Rezapour, and Mohammad R Salavatipour. Local search yields a ptas for k -means in doubling metrics. SIAM Journal on Computing , 48(2):452-480, 2019.

Fabrizio Grandoni, Rafail Ostrovsky, Yuval Rabani, Leonard J Schulman, and Rakesh Venkat. A refined approximation for euclidean k -means. Information Processing Letters , 176:106251, 2022.

Junyu Huang, Qilong Feng, Ziyun Huang, Jinhui Xu, and Jianxin Wang. FLS: A new local search algorithm for K-means with smaller search space. In Proceedings of the 31st International Joint Conference on Artificial Intelligence , pages 3092-3098, 2022.

Tapas Kanungo, David M Mount, Nathan S Netanyahu, Christine D Piatko, Ruth Silverman, and Angela Y Wu. A local search approximation algorithm for k -means clustering. In Proceedings of the 18th Annual Symposium on Computational Geometry , pages 10-18, 2002.

Silvio Lattanzi and Christian Sohler. A better k -means++ algorithm via local search. In Proceedings of the 36th International Conference on Machine Learning , volume 97, pages 3662-3671. PMLR, 2019.

Stuart Lloyd. Least squares quantization in PCM. IEEE Transactions on Information Theory , 28(2):129-137, 1982.

Konstantin Makarychev, Aravind Reddy, and Liren Shan. Improved guarantees for k -means++ and k -means++ parallel. In Proceedings of the 34th Conference on Neural Information Processing Systems , pages 16142-16152, 2020.

Yusuke Matsui, Keisuke Ogaki, Toshihiko Yamasaki, and Kiyoharu Aizawa. Pqk-means: Billionscale clustering for product-quantized codes. In Proceedings of the 25th ACM International Conference on Multimedia , pages 1725-1733, 2017.

Jiayang Ren, Kaixun Hua, and Yankai Cao. Global optimal K-medoids clustering of one million samples. In Proceedings of the 36th Conference on Neural Information Processing Systems , pages 982-994, 2022.

Dennis Wei. A constant-factor bi-criteria approximation guarantee for k -means++. In Proceedings of the 30th Conference on Neural Information Processing Systems , pages 604-612, 2016.

A Missing Proofs in Section 3

Lemma 2. Given a type-1 or type-2 matched swap pair ( c ∗ h , c j ) , it holds that ζ ( P, C, c ∗ h , c j ) ≤ 24∆( P j , C ∗ ) + 1 5 ∆( P j , C ) . Given a type-1 matched swap set ( Q,V ) , it holds that ζ ( P, C, Q, V ) ≤ 24∆( X ( V ) , C ∗ ) + 1 5 ∆( X ( V ) , C ) .

Proof We first consider a type-1 matched swap set ( Q,V ) . For a data point p ∈ P , we use s p to denote its closest center in C . Let o p be its closest optimal center in C ∗ . Observe that

where the first inequality follows from the fact that s o p is still in C \ V for each data point p ∈ X ( V ) \ Z ( J ( Q )) according to the definition of the swap set defined, the second and fourth steps follow from the triangle inequality, the third step follows from the fact that s o p is the nearest point to o p in C , and the last step follows from Cauchy Inequality. Let λ = 1 10 . Then ζ ( P, C, Q, V ) ≤ 24∆( X ( V ) , C ∗ ) + 1 5 ∆( X ( V ) , C ) . The cases for matched swap pair are similar to the cases for matched swap set. For a matched swap pair ( c ∗ h , c j ) , we can get that ζ ( P, C, c ∗ h , c j ) ≤ 24∆( P j , C ∗ )+ 1 5 ∆( P j , C ) . □

Lemma 3. Let P ∗ h be an optimal cluster with ∆( P ∗ h , C ) = b ∆( P ∗ h , { c ∗ h } ) for a real number b ≥ 3 . Then, ∆( χ ( P ∗ h ) , C ) ≥ 1 200 ( b -1)∆( P ∗ h , { c ∗ h } ) .

Proof Let s c ∗ h be the closest center in C to c ∗ h . Observe that d ( c ∗ h , s c ∗ h ) ≥ ( b -1)∆( P ∗ h , { c ∗ h } ) | P ∗ h | . Otherwise ∆( P ∗ h , { s c ∗ h } ) < b ∆( P ∗ h , { c ∗ h } ) holds according to Lemma 1. Consider an arbitrary data point p ∈ χ ( P ∗ h ) . By triangle inequality, we have

Since √ b -1 ≥ √ 2 , we have

Together with the fact that | χ ( P ∗ h ) | ≥ 1 3 | P ∗ h | , we have ∆( χ ( P ∗ h ) , C ) ≥ b -1 200 ∆( P ∗ h , { c ∗ h } ) , which proves the Lemma. □

Lemma 4. By swapping q with c q , the clustering cost of ∆( P, C ) can be reduced at least by 1 -Θ( 1 k ) .

Proof Observe that ∆( P ∗ h , { q } ) < 9∆( P ∗ h , { c ∗ h } ) holds by Lemma 1. Then, we can get that

where the second to the last inequality follows from the fact that ∆( P ∗ h , { q } ) < 9∆( P ∗ h , { c ∗ h } ) , and the last inequality follows from the definition of good single cluster. □

Lemma 5. If case (2) happens, then ∆( X ( L ) , C ) ≤ (1 + 1 t )(9∆( X ( L ) , C ∗ ) + 24∆( X ( L ′ ) , C ∗ ) + 1 5 ∆( X ( L ′ ) , C ) + 1 100 ∆( P, C )) .

Proof By the condition of case (2) , it holds that all the optimal clusters in J ( L ) belong to bad single cluster. For any bad single cluster P ∗ h ∈ J ( L ) , let ( c ∗ h , c m h ) ∈ M 3 be the type-2 matched swap pair with minimum reassignment cost, i.e., ( c ∗ h , c m h ) = arg min ( c ∗ h ,c n ) ∈ M 3 ζ ( P, C, c ∗ h , c n ) . According to the definition of bad single cluster, we have

Let κ ( M 3 ) be the set of lonely centers used for constructing type-2 matched swap pair. For an optimal center c ∗ h ∈ C ∗ , denote s c ∗ h as the center in C closest to c ∗ h . Let L 1 = ∪ c ∗ h ∈ L s c ∗ h . By the definitions of κ ( M 3 ) and L , for each s c ∗ h ∈ L 1 , we can find a set z ( s c ∗ h ) ⊆ κ ( M 3 ) with size | Ψ -1 ( s c ∗ h ) | -1 such that z ( a ) ∩ z ( b ) = ∅ for any a, b ∈ L 1 . For each s c ∗ h ∈ L 1 , since | Ψ -1 ( s c ∗ h ) | > t , it holds that | Ψ -1 ( s c ∗ h ) | / | z ( s c ∗ h ) | ≤ 1 + 1 t . By taking a summation over all the centers in L , we have | L | = ∑ c ∗ h ∈ L 1 = ∑ s c ∗ h ∈ L 1 | Ψ -1 ( s c ∗ h ) | ≤ ∑ s c ∗ h ∈ L 1 | z ( s c ∗ h ) | (1 + 1 t ) ≤ | κ ( M 3 ) | (1 + 1 t ) . Then, by considering all the optimal centers in L and taking a summation, we have

where the second to the last inequality follows from the fact that c m h is the center with minimum reassignment cost, and the last inequality follows from Lemma 2 and the fact that | L | / | κ ( M 3 ) | ≤ 1 + t -1 . □

̸

Lemma 6. Let Q ∈ H 2 be a set of centers in H 2 , where J ( Q ) is a set of good t -clusters with a type-1 matched swap set ( Q,A ′ ) ∈ M 2 . Define V = A ′ \{ c j } , where c j is the center in A ′ with | Ψ -1 ( c j ) | > 1 . Let U ⊆ P be the set of data points with | U | = | V | such that U ∩ χ ( P ∗ h ) = ∅ holds for each P ∗ h ∈ Q T . Then, ∆( P, C \ V ∪ U ) ≤ (1 -1 100 k )∆( P, C ) .

Proof Consider an arbitrary optimal cluster P ∗ h ∈ Q ′ S . We know that ∆( P ∗ h , C ) < 3∆( P ∗ h , { c ∗ h } ) by the definition of Q ′ S . Let s c ∗ h be the closest center in C to c ∗ h . We first argue that s c ∗ h is close to c ∗ h with d ( c ∗ h , s c ∗ h ) ≤ 8∆( P ∗ h , { c ∗ h } ) | P ∗ h | . For a data point p ∈ P ∗ h , let s p be its closest center in C . According

to the triangle inequality, we have

where the last inequality follows from ∆( P ∗ h , C ) < 3∆( P ∗ h , { c ∗ h } ) and feeding λ = √ 3 . By assigning all the data points in P ∗ h to s c ∗ h , we have ∆( P ∗ h , { s c ∗ h } ) ≤ 9∆( P ∗ h , { c ∗ h } ) by Lemma 1. Then, by swapping U with V , we have that

where the first inequality follows from the fact that s c ∗ h / ∈ V for each P ∗ h ∈ Q ′ S , the second inequality follows from ∆( P ∗ h , { s c ∗ h } ) ≤ 9∆( P ∗ h , { c ∗ h } ) for each P ∗ h ∈ Q ′ S , and the last inequality follows from the definition of good t -clusters and V ∪ { s c ∗ h } = A ′ . □

Lemma 7. Given a set Q ∈ H 2 of centers such that J ( Q ) is a set of good t -clusters, the probability that step 4 of Algorithm 2 fails to sample a data point q from χ ( P ∗ h ) for each P ∗ h ∈ Q ′′ S in the m ( P ∗ h ) -th iteration is at most (1 + 1 30000 k ) e -∆( Z ( Q ′′ S ) ,C ) / (300∆( P,C )) .

Proof We first consider an optimal cluster P ∗ h ∈ Q ′′ S . Observe that ∆( P ∗ h , C ) < 1 30000 k 2 t -1 ∆( P, C ) by the definition of Q ′′ S . Define p h f as the failure probability of not sampling data points from χ ( P ∗ h ) in the m ( P ∗ h ) -th iteration of step 4 in Algorithm 2. Then, we have that

Let p h s = 1 -p h f be the success probability. Then, we have

where the last inequality follows from 1 30000 < k . Let p f ( Q ′′ S ) denote the probability of not sampling a data point from χ ( P ∗ h ) for each P ∗ h ∈ Q ′′ S in the m ( P ∗ h ) -th iteration of step 4 in Algorithm 2. For each P ∗ h ∈ Q ′′ S , define X h = 1 if the m ( P ∗ h ) -th iteration of step 4 in Algorithm 2 samples a data point q ∈ χ ( P ∗ h ) , and X h = 0 if the m ( P ∗ h ) -th iteration fails to sample a data point q ∈ χ ( P ∗ h ) . Let E 1 be the event that X h = 0 for each P ∗ h ∈ Q ′′ S , and E 2 be the event that there are at least two clusters P ∗ h , P ∗ j ∈ Q ′′ S such that X h ∨ X j = 1 and X h ∧ X j = 0 . Observe that there are at most 2 t -2 subcases if event E 2 happens. For each subcase s , let ν ( s ) = { P ∗ h : P ∗ h ∈ Q ′′ S , X h = 1 } and ν ′ ( s ) = { P ∗ h : P ∗ h ∈ Q ′′ S , X h = 0 } , respectively. Define P r ( s ) as the probability that subcase s happens. Since the sampling iterations in step 4 of Algorithm 2 are mutually independent, we have P r ( s ) = ∏ P ∗ h ∈ ν ( s ) p h s ∏ P ∗ h ∈ ν ′ ( s ) p h f . By applying the inequality that p h s ≤ 1 30000 k (2 t -1 -1) p h f , we have

for each subcase s of E 2 , where the last inequality follows from 1 30000 k (2 t -1 -1) < 1 . Then, we have P r ( E 2 ) ≤ 1 30000 k ∏ P ∗ h ∈ Q ′′ S p h f by taking a probability summation over all 2 t -2 subcases of event

E 2 . Observe that the failure probability p f ( Q ′′ S ) = P r ( E 1 ) + P r ( E 2 ) , where P r ( E 1 ) = ∏ P ∗ h ∈ Q ′′ S p h f . Thus, we can get that

where the second to last inequality follows from Lemma 3, and the last inequality follows from the inequality 1 -x ≤ e -x . □

Lemma 8. For the optimal clusters in H G , we have ∆( Z ( H G ) , C ) ≥ 1 100 ∆( P, C ) .

Proof Given any type-1 matched swap set ( Q,V ) , if J ( Q ) is a set of good t -clusters with a type-1 matched swap set ( Q,V ) ∈ M 2 , according to the assumption of subcase (2), we know that each optimal cluster P ∗ h in Q L is a bad single cluster with a swap pair ( c ∗ h , c l ) , where c l is a lonely center in V . Since J ( Q ) is a set of good t -clusters, not all optimal clusters in J ( Q ) are bad single clusters. Hence, for each optimal cluster P ∗ h ∈ Q L , we can find a lonely center l ( c ∗ h ) ∈ V such that P ∗ h is a bad single cluster with the swap pair ( c ∗ h , l ( c ∗ h )) . By the definition of bad single cluster, we have

For each optimal cluster P ∗ h ∈ Q ′ S , by the definition of Q ′ S , it holds that ∆( P ∗ h , C ) ≤ 3∆( P ∗ h , { c ∗ h } ) . If J ( Q ) is a set of bad t -clusters, by the definition of bad t -clusters, we also have

Define H 1 B = { c ∗ h ∈ H 1 : P ∗ h is a bad single cluster with the swap pair ( c ∗ h , c j ) ∈ M 1 } , H 2 B = { Q ∈ H 2 : J ( Q ) is a set of bad t -clusters with the swap set ( Q,V ) ∈ M 2 } and H 2 B ′ = { Q ′ S ∪ Q L : Q ∈ H 2 , J ( Q ) is a set of good t -clusters with the swap set ( Q,V ) ∈ M 2 } , respectively. Putting all things together and taking a summation over all clusters in H B , we can get that

where the first inequality follows from Lemma 3. Then, together with Lemma 5, we can get that

where the second inequality follows from Lemma 5, the third inequality follows from the fact that H ′ 1 ∩ H ′ 2 ∩ L ′ = ∅ and Z ( H B ) ∩ J ( L ) = ∅ , and the fourth inequality follows from t ≥ 2 and the assumption that ∆( P, C ) ≥ 50(1 + 1 t ) Opt . □

Theorem 2. In the i -th iteration of Algorithm 2, let C ′ be the set of centers obtained in step 7. If the current clustering cost ∆( P, C ) is larger than 50(1 + 1 t ) Opt , then with probability at least Ω( k -t ) , we have ∆( P, C ′ ) ≤ (1 -1 100 k )∆( P, C ) . After O ( k O ( t ) log( ϵ -1 log k )) iterations, we get an approximate solution with ratio (50(1 + 1 t ) + ϵ ) in expectation.

Proof Let T = δk t +1 log(24 ϵ -1 log k ) , where δ is a sufficient large constant. Following the work in [11], we define another random process X with initial clustering cost ∆( P, C ′ ) for a set C ′ returned by the k -means++ algorithm such that for T iterations of sampling and swaps, it reduces the value of ∆( P, C ′ ) by at least (1 -1 100 k ) with probability λk -t , and it increases the final value of ∆( P, C ′ ) by 50(1 + 1 t ) OPT , where λ is a constant with λ < δ 100 . It is easy to see that E [∆( P, C )] < E [ X ] . Then, we have

This implies that E [∆( P, C ) | C ′ ] ≤ ϵ ∆( P,C ′ ) 24 log k +50(1 + 1 t ) Opt . Then, we can get that

Since the k -means++ algorithm provides an approximation ratio of 8(log k +2) in expectation by Theorem 1, we have E [∆( P, C )] ≤ (50(1 + 1 t ) + ϵ ) Opt . □

Running Time Analysis. By Theorem 2, in order to obtain a (50(1 + 1 t ) + ϵ ) -approximate solution, the iteration rounds for Algorithm 2 should be O ( k t +1 log( ϵ -1 log k )) . In each iteration, it takes O ( ndk ) time to update the distances between data points to their closest centers. During the sampling

process, t data points are sampled according to the D 2 -Sampling distribution to serve as the candidate set of centers for swapping in, which takes time O ( t ) if the distances from data points to their centers are already known. It takes O ( k t ) time to enumerate each subset with size at most t of the set of current centers opened. It takes O ( ndk t ) time to recalculate the clustering cost after each swap if t nearest centers of each data point are maintained during the whole process. Thus, the total running time of Algorithm 2 is O ( ndk 2 t +1 log( ϵ -1 log k )) .

B Complementary Experiments

B.1 Experiments with Different Parameter Settings

In this section, we present the experiments on the performances of our proposed MLSP algorithm with different parameters T , R ′ , ϵ and η . We also compare our MLS and MLSP algorithms with LS++ algorithm using different iteration rounds to show the parameter robustness of different algorithms.

Tables 3, 4 and 5 show the comparison results of different algorithms with varying iteration rounds on dataset rds, KEGG and Urban_10, respectively. The results show that, our proposed MLSP algorithm always achieves the best clustering cost compared with LS++ and MLS algorithms. It can be seen that the number of iteration rounds has little impact on our MLSP algorithm. For MLS algorithm, it performs better on clustering cost compared with LS++ algorithm for most cases, which indicates that larger iteration rounds will not influence the performance of our algorithms.

Tables 6, 7, and 8 show the results of MLSP algorithm with varying number of failure upper bound R ′ for fixed ϵ = 0 . 5 , η = 0 . 5 and T = 400 on dataset rds, KEGG and Urban_10, respectively. It can be seen that a larger failure upper bound will lead to smaller deviation, and the running time becomes higher. In general, the failure upper bound does not influence the performance of our proposed MLSP algorithm. Tables 9, 10, and 11 show the results of MLSP algorithm with varying parameters ϵ and η for fixed T = 400 and R ′ = 5 on dataset rds, KEGG, and Urban_10, respectively. It can be seen that smaller values of ϵ and η result in better performances on clustering costs with smaller deviation, and the running time becomes higher. There is no significant difference between different choices of parameters ϵ and η , which indicates that our proposed MLSP algorithm ensures the parameter robustness.

B.2 Experiments on Small Datasets

In this section, we present the experiments of the performances of our proposed MLSP algorithm on other datasets with sizes smaller than 50 , 000 used in [15].

Table 12 and Table 13 show the results on 18 datasets with sizes smaller than 50 , 000 using k = 10 . It can be seen that, on each dataset, the best clustering cost returned by our MLSP algorithm is smaller than BB method (note that BB method already guarantees a gap smaller than 0.1% to the optimal solutions on small datasets) and other local search methods. On each dataset, the average clustering cost returned by our MLSP algorithm nearly matches the result of the BB method. As for FLS, although the performance of FLS is better than that of LS++, our proposed MLSP algorithm improves the performance of FLS on clustering cost with smaller deviation for most datasets. As for running time, there is no significant difference on the running time among different local search algorithms for small datasets.

B.3 Experiments with Different Values of k

In this section, we present the experiments of the performances of our proposed MLSP and MLS algorithms on different datasets with different values of k . Tables 14 and 15 present the performances of different local search algorithms with k = 3 on small datasets. Table 18 presents the performances of different local search algorithms with k = 3 on large datasets. Tables 16 and 17 present the performances of different local search algorithms with k = 5 on small datasets. Table 19 presents the performances of different local search algorithms with k = 5 on large datasets.

It can be seen that, compared with other local search methods, our proposed MLSP algorithm achieves the best clustering performance on most datasets with different values of k . On each dataset, the best clustering cost returned by our MLSP algorithm matches the clustering cost of BB method (note that BB method already guarantees a gap of 0 . 1% to the optimal solution). On each dataset, the average clustering cost returned by our MLSP algorithm nearly matches the result of the BB method. As for running time, there is no significant difference on running time among all local search algorithms on small datasets. However, as the data sizes grow, our proposed MLS algorithm becomes much faster than other local search algorithms.

B.4 Experiments on Different Local Search Algorithms with Fixed Time Limits

In this section, we conduct some additional experiments on different datasets with fixed time limits. The experimental results in Table 20 demonstrate that our proposed MLSP algorithm achieves the best clustering quality with fixed time limits.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies local-search algorithms for k-means clustering. The goal here is to obtain a local-search algorithm which (1) give a constant factor approximation and (2) run in time linear in the dataset. In the past literature, one can distinguish essentially 2 types of local-search algorithms for this problem. Single-swap local search which have a simple swap procedure but can only guarantee approximation ratios around 500 at least. Multi-swap local search which use a richer swap structure to give better guarantees (the current best is 9+\\epsilon – approx) but are often impractical since finding a good swap often implies to enumerate over all subsets of some candidate set, which makes the algorithm slower than linear (in the dataset size) time. \n\nThis paper tries to reconcile these two directions by giving a new Multi-swap local search algorithm which gives and approximation factor of 50+\\epsilon and runs in essentially linear time. One of the main ideas in this work is to use sampling techniques in the spirit of the famous k-means++ algorithm and only after this try to swap current centers with the sampled candidates. Because the set of sampled candidates is much smaller than the actual dataset, the running time is significantly improved. They also give experiments that show that their algorithm performs well in practice.\n\nSTRENGTHS:\nIn my opinion, the strengths of the paper are as follows.\n1)\tInteresting result and techniques. Not being an expert, it seems that it is the first time that the sampling technique is used in the context of multi-swap local search.\n2)\tThe algorithm seems to perform well in practice. Experimental section is quite thorough.\n\nWEAKNESSES:\nSome minor weakness in my opinion is that there is still a significant gap between the lower bound of 9 for local search and the proven upper bounds.\n\nTypos : \nline 88: benefit\nline 105: much *more*\nline 145: I guess the OPT value is missing in the inequality\n\nQUESTIONS:\nThe authors actually propose 2 algorithms, first MLS for which they prove theoretical guarantees. And a second MLPS which they say is “more practical” but I do not see anywhere in the paper that the same approximation guarantee of 50 holds for MLPS as well. In the experiment sections, MLS seems to perform well, but it is unclear to me if this is the same MLS as the one that was analyzed in section 3 (line 336 the authors say they use the sampling method of MLPS for the MLS algorithm). Does this change the theoretical guarantee of MLS? It would be great of the authors could clarify these points.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the well-known k-means problem: Given a set of points in the Euclidean space, compute k centers such that the sum of squared distances between points and their closest center is minimized. A constant-factor approximation to this problem is achieved by local search: Start with arbitrary k centers and improve the solution by choosing a point which is currently not a center and replace a center in our current solution by this point. If we exchange at most t centers instead of one in the local search method Kanungo et al. showed that the algorithm computes a (3+2/t)^2-approximation.\n\nLattanzi and Sohler proposed a combination of k-means++ sampling and local search (LS++): Start with the k centers computed by the k-means++ solution and improve it via local search. In every step of local search the new center is sampled proportionally to its current cost in the solution. Lattanzi and Sohler showed that their algorithm computes a 509-approximation after O(k*log(log(k))) local search steps. \n\nThe authors extend the algorithm of Lattanzi and Sohler (MLS): Instead of sampling only one point, sample t points S simultaneously and search for a subset T of S such that replacing |T| points in the current solution by T reduces the cost significantly. The main result is the following: After O(k^O(t)*log(epsilon*log(k))) local search steps the algorithm returns a (50(1+1/t)+epsilon)-approximation in expectation. Each local search step takes time O(n*d*k^O(t)), where n is the number of points and d is the dimension of the euclidean space. \n\nIn the experimental part of the paper the authors implemented the MLS-algorithm for t=2 and proposed a second algorithm (MLSP): 1) Use k-means++ for an initial solution 2) For R rounds do the following in each round: Apply the MLS-algorithm, check whether the current solution C is the best observed so far. If yes, again apply MLS to improve it, if not exchange some (or all) centers from C according to a procedure developed by the authors and apply MLS with new center set.\nSo MLSP restarts MLS with a new set of centers if the local optimum which MLS computes is not good. This restart is performed R times. The authors compare the MLS and MLSP algorithm to three algorithms LS++, Lloyd and FLS on several data-sets (50 000- 11 000 000 points) for k=10. In most cases the MLSP algorithm computes the best solution but has a significantly higher run-time than MLS, Lloyd and LS++.\n\nSTRENGTHS:\nThe extension of LS++ to an algorithm which samples more than one center is natural and the resulting improvement in the approximation factor is significant. Even though some parts of the proof are similar to the approach of Lattanzi, Sohler and Kanungo et al. the proof involves some non-trivial steps and is original in my opinion.\n\nThe experiments show that the algorithm MLSP outperforms other algorithms with respect to the quality of the solution even in the setting where all algorithms run for a fixed amount of time. This may be due to the restart of the local search procedure when it runs into a bad local minimum, so it seems to be a good heuristic to implement in any local search procedure.\n\nWEAKNESSES:\nEspecially the technical part of the paper is very hard to read, which could have been prevented since it is mostly not due to the difficulty of the proofs but because of the complex notation and sometimes imprecise statements. While the result in the main theorem seems to be plausible I am not completely sure about the correctness of the proof.\n\nIt would be nice to have experiments for k>10, since the choice of k<=10 seems not to fit the considered data sizes of 50 000 to 10 000 000 points.  The run-time of MLS and MLSP rises faster with respect to k than the run-time of LS++ so it would be nice to see how their algorithms perform for larger values of k.   \n\nFurthermore the authors outline that the MLS algorithm is relatively fast, especially when compared to its easier version LS++ where only one center is exchanged in every step, which was confusing at first. However the speed-up in the run-time is achieved by an approximate computation of the clustering cost of a solution which could have also been implemented for LS++ for a fair comparison to MLS in my opinion.\n\nQUESTIONS:\nL 69: Should this be n^t instead of n*k^t? If not, please explain. (appears also in L 76, Table 1)  \nL 88: Missing O(...) in the run-time\nL 123: I feel that you should state Lemma 2 as a theorem\nL 136-138: You should mention that this only holds when the cost of the current solution is larger than ((50+1/t)+epsilon)*OPT.\nL 138: we can get an approximation -> we get an approximation\nL 145: In the lower bound on the cost of C there is missing the cost of an optimal solution.\nL 167: Maybe just introduce a map \\phi which assigns every center in C* the nearest center in C? Then you do not need the notation s_c and you can denote the set of optimal centers captured by a center c in C by \\phi^{-1}(c). I feel that the notation should be simplified at a lot of places throughout the paper.\nL  172: find a set \\sigma_h of unused lonely centers -> let \\sigma_h be an arbitrary set of unused lonely centers\nL 173: in the definition of A: don't we have c_h^* in \\phi(s_c_h^*) already by definition as c_h^* is captured by s_c_h^*?\nL 176-177: Please specify how you construct type 2 matched swap pairs. At first it seems that every unused lonely center from C should be in at most one type 2 matched swap pair, but this is not possible. I think what you later need is that every unused lonely center and every c_h^* with |\\phi(s_c_h^*)|>t should form a type 2 matched swap pair? \nL 179-182: For the union of clusters with centers in V you use X(V) while for the union of clusters with centers in Q you use Z(J(Q)), this is very confusing. I suggest to simplify the notation (maybe just directly write the union of clusters instead of introducing a new notation) and also don't change between center sets, clusters sets and union of cluster sets that much in the whole paper when it is avoidable.\nL 212-222: maybe it is better to state this as a lemma\nL 219-220: missing two \" ) \" in this chain of inequalities, also please put this into an align environment as it is hard to read if you include it directly in the text\nL 227:  cost of those optimal clusters -> cost of optimal clusters\nL 230: |Q|>1 : Isn't this automatically the case for sets in H_2? \nL 235: c_h^* can find clustering centers close to it -> c_h^* is close to a center from C\nL242: you use H_1,H_2  etc. are sets of centers, now H_2^* is some set of clusters, this is a little bit confusing\nL267: Shouldn‘t the probability Omega(1/k) also depend on t because of the definition in Line 231?\nL268-269: I belive that this should be independency of events, not the union bound\nL281: comma at beginning of line\nL281: Could you explain why we add Q_S‘‘ and not Q_T to H_G?\nAlgorithm 3 Line 11: Could you explain how this swapping exactly works? Also why don‘t use simply Lloyd here?\n\nAppendix:\nL428: In the following sequence of equations: first inequality remember the reader why s_{o_p} is not in V\nL446: rename one of the (c_h^*,c_h^m) in the argmin equation\nL448: \\mu is previously used as centroid of a set, maybe find an other letter\nL449: In the following sequence of equations, first inequality: should \\mu(L) be \\mu(M_3)? If not please define \\mu(L) \nL452: Please explain this upper bound\nL460: please state the bound for the relaxed triangle inequality as lemma somewhere\nL463: In the following sequence of equations: Z(Q_T) → Z(J(Q_T)), Z(Q‘_S)-Z(J(Q‘_S)), Z(Q)->Z(J(Q))\nL474-476: It is not clear if you mean the failure probability of not sampling at least one point correctly or all points correctly\nL484: In the following inequality: shouldn‘t „t“ be „|v(W)|“ in the exponent?\nL488 In the following inequalities, last inequality: \\Delta(P,C)→ 300\\Delta(P,C)\nL496: „unique lonely center“ misleading as one could think that l(c_h^*) is specific to P_h^*\nL515: \\lambda t^{-k} → \\lambda k^{-t}\nL520-528: this paragraph equals L512-520 → remove\n\nLIMITATIONS:\nn.a.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLocal search algorithm is a well studied technique for clustering problems. In the k-means problem, the simple swap heuristic states that start with an intiial chosen set of k centers  and then at each local search step, check if swapping an existing center with a new one leads to decrease in cost. It is known that this gives a constant factor approximation algorithm (with the constant being 25). The $t$-swap heuristic generalises this local search algorithm by swapping a subset of $t$ centers at each step. It is clear that the running time of such an algorithm will be exponential in $t$, though the approximation factor converges to 9 as $t$ increases. \n\nA related line of work on the k-means problem deals with the Lloyd's algorithm. In this algorithm, we choose an initial set of k seed centers and then run a different local search algorithm on it. A lot of work has happened on how to choose these initial centers. The k-means++ heuristic says that we choose the initial centers as follows: the first center is picked uniformly at random. After having picked $i$ initial centers, a point is chosen as the $(i+1)^{th}$ one with probability proportional to square of the distance from the $i$ centers chosen already. It is known that if we choose a set of $k$ centers in this manner, then it can lead to $O(\\log k)$-approximation algorithm for the k-means objective (and this bound is tight). Subsequent works have focussed on whether we can improve this initial set in a small amount of time. For example, reference [10] showed that if we run the single swap heuristic on the $k$ centers chosen by this random sampling procedure for about $O(k)$ steps, then we get a 509-approximation algorithm. \n\nThis paper is an extension of this last result by giving a fast implementation of the $t$-swap heuristic on an initial set of centers chosen using the random sampling procure described above. Each local search step chooses a set of $t$ new centers using a random sampling procedure and checks if it can replace an existing set of $t$ centers in the current solution. The paper shows that the running time now becomes $O(k^{O(t)} nd$, and hence is linear in $n$ for fixed $k,t$. They also show that the resulting set of centers has approximation ratio of about 50. The analysis has the same structure as that of [10], though analyzing $t$-swap is much more tricky, and requires new insights.\n\nSTRENGTHS:\n1. Usually $t$-swap heuristcs are expensive to implement, this paper gives a non-trivial implementation which is efficient in practice. \n2. The analysis of $t$-swap requires new ideas.\n\nWEAKNESSES:\n1. The improvement in experiments is only marginal (in most cases less than 5%). \n\n2. The usual local search heurstic (with single swap heuristic), ie.., reference [9] in table 1, has running time $O(nkd \\lof \\Delta)$ and has approximation ratio of 25. This running time almost matches the running time of the result in this paper (upto log \\Delta factor), and has better approximation ratio. So it is not clear if the theoretical contribution is significant. I also don't see a comparison with [9] (with single swap heuristic) in the implementation.\n\nQUESTIONS:\n1. Please compare with [9] in terms of theoretical and experimental results. \n2. It may be worth explaining what new ideas are needed in the analysis as compared to [10].\n\nLIMITATIONS:\nNo negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multi-swap local search algorithm for the k-means problem with linear running time in the data size, while also achieves a better approximation ratio when compared with other local search algorithms that adopt single-swap strategy. To benefit more from such algorithm when handling large-scale datasets, the authors further propose a sampling-based method to accelerate the updating of clustering cost during the swaps. The extensive empirical experiments validate the good numerical performance of the proposed algorithms.\n\nSTRENGTHS:\nThis paper studies the classic k-means clustering problem and gives a first linear time local search algorithm in literature that adopts the multi-swap strategy. Such algorithm enjoys the two advantage from both worlds: smaller approximation ratio and linear run-time in data size. The organization of this paper is clear and well-structured, the mathematical proof is very detailed and easy to follow, and the numerical experiments are also quite extensive.\n\nWEAKNESSES:\nFor Lloyd algorithm, according to your experiments it is able to save up to 99% of the run time (0.07 vs 5.12), while only having around 1% more average cost. Therefore, the merit carried by the work presented in this paper does not seem to be too impressive from the practical point of view. One related question is, considering the huge amount of time being saved from Lloyd algorithm, have the authors tried running Lloyd algorithm for multiple times (if Lloyd algorithm takes 0.07s for each initialization, then you can run 5.12/0.07 times of Lloyd algorithm) with different initialization and then simply take the best solution? How does that compare with MLS when they are set to have the same run time?\n\nQUESTIONS:\nLine 11-12, \"which improves the current best result 509\": It might be true for local search methods that take single-swap strategy, but It is unfair to call this the best result, since the current best approximation ratio is 6.357.\n\nLine 90, \"MLS algorithm\": Please refer it and write out the full name when mentioning this abbreviation for the first time. Please consider move the \"multi-swap local search algorithm\" at line 100 to here. \n\nAlgorithm 1: \"Output: ... of at most k centers\". I do not understand why you say \"at most\". Since in each iteration point p can be selected if and only if point p is not in C, isn't it guaranteed that C has size exactly k? Similar question to Algorithm 2.\n\nLine 3, Algorithm 1: $\\Delta$ is defined over two sets, here p is a point it should be {p}, and {C} should be C. This appears in other places as well, for instance Algorithm 2.\n\nIn the numerical experiment section, in Table 2 why is MLS able to outperform the single-swap local search methods LS++ and FLS by a big margin? It does not make sense to me since MLS algorithm is clearly more complex than LS++ in [10] and it has bigger runtime complexity.  \n\nLine 345-346, \"the average clustering ... nearly match the results of the BB method\": As far as I can see, except on one dataset, isn't MLSP algorithm always outperform BB method in terms of average clustering costs?\n\nLIMITATIONS:\nThe authors did not address the limitation of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}