SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Linear Time Algorithms for  $k$ -means with Multi-Swap Local Search
Junyu Huang $^{1,2}$ , Qilong Feng $^{1,2,*}$ , Ziyun Huang $^{3}$ , Jinhui Xu $^{4}$ , Jianxin Wang $^{1,2,5,*}$
$^{1}$ School of Computer Science and Engineering, Central South University,
Changsha 410083, China
$^{2}$ Xiangjiang Laboratory, Changsha 410205, China
<sup>3</sup>Department of Computer Science and Software Engineering, Penn State Erie,
The Behrend College
$^{4}$ Department of Computer Science and Engineering, State University of New York at Buffalo, NY, USA
<sup>5</sup>The Hunan Provincial Key Lab of Bioinformatics, Central South University,
junyuhuangcsu@foxmail.com, csufeng@mail.csu.edu.cn,
zxh201@psu.edu, jinhui@buffalo.edu, jxwang@mail.csu.edu.cn
The local search methods have been widely used to solve the clustering problems. In practice, local search algorithms for clustering problems mainly adapt the single-swap strategy, which enables them to handle large-scale datasets and achieve linear running time in the data size. However, compared with multi-swap local search algorithms, there is a considerable gap on the approximation ratios of the single-swap local search algorithms. Although the current multi-swap local search algorithms provide small constant approximation, the proposed algorithms tend to have large polynomial running time, which cannot be used to handle large-scale datasets. In this paper, we propose a multi-swap local search algorithm for the  $k$ -means problem with linear running time in the data size. Given a swap size  $t$ , our proposed algorithm can achieve a  $(50(1 + \frac{1}{t}) + \epsilon)$ -approximation, which improves the current best result 509 (ICML 2019) with linear running time in the data size. Our proposed method, compared with previous multi-swap local search algorithms, is the first one to achieve linear running time in the data size. To obtain a more practical algorithm for the problem with better clustering quality and running time, we propose a sampling-based method which accelerates the process of clustering cost update during swaps. Besides, a recombination mechanism is proposed to find potentially better solutions. Empirical experiments show that our proposed algorithms achieve better performances compared with branch and bound solver (NeurIPS 2022) and other existing state-of-the-art local search algorithms on both small and large datasets.
Clustering is a fundamental problem in the field of machine learning with many real-world applications. The goal of clustering is to partition a given set of data points into different clusters according to their similarity such that data points within the same cluster share high similarity as much as possible. Among different objective functions, the  $k$ -means clustering aims to minimize the sum of the squared distances between data points to their closest centers. More formally, given a set  $P \subset \mathbb{R}^d$
*Corresponding Authors
of data points in a  $d$ -dimensional Euclidean space, the goal of the  $k$ -means clustering is to find a set  $C \subset \mathbb{R}^d$  of size at most  $k$  with the following objective:  $\min_{C \subset \mathbb{R}^d} \sum_{p \in P} \min_{c \in C} \| p - c \|^2$ .
For the  $k$ -means problem, Lloyd's algorithm [12] is one of the most widely used heuristic in practice. However, there is no theoretical guarantee for Lloyd-type method unless certain data distribution assumptions are introduced. It is known that there are several constant approximation schemes based on primal-dual and randomized rounding techniques [2, 5, 8]. The current best approximation ratio in polynomial time is 5.912 [5], which is based on primal-dual method and nested quasi-independent set. For fixed dimension  $d$  or the number of clusters  $k$ , several  $(1 + \epsilon)$ -approximation algorithms were proposed [6, 7].
The  $k$ -means++ algorithm proposed by Arthur and Vassilvitskii [3] is a good seeding method that runs in linear time in the data size with  $O(\log k)$ -approximation. It is also known that the  $k$ -means++ algorithm gives a constant approximation by opening  $O(k)$  centers [1, 13, 16]. Lattanzi and Sohler [11] showed a combination of  $k$ -means++ seeding and the local search algorithm (named as  $\mathrm{LS}++$  algorithm), which yields a constant approximation with running time  $O(ndk^2\log \log k)$ . In each round,  $\mathrm{LS}++$  algorithm samples a data point using  $k$ -means++ seeding and enumerates possible swap pairs to make improvements on clustering cost. They proved that after  $O(k\log \log k)$  rounds of sampling and swaps, one can obtain a 509-approximate solution in expectation. Choo et al. [4] proved that one can achieve an  $O(1/\epsilon^3)$ -approximation using reduced  $O(\epsilon k)$  rounds of  $\mathrm{LS}++$  algorithm. Under the assumption that each optimal cluster has size  $\Omega(n/k)$ , Huang et al. [9] gave an improved approximation algorithm with ratio  $(100 + \epsilon)$  by random sampling methods.
However, there are still several issues for local search methods. Although current local search methods with multi-swap strategy can achieve good theoretical guarantee, the running time of them have polynomial dependence on the data size, which are hard to be used to handle large-scale datasets. Compared with the  $(9 + \epsilon)$ -approximation multi-swap local search algorithm given in [10], the approximation ratio 509 of  $\mathrm{LS}++$  is a large constant since it can only apply the single-swap strategy. Numerical experiments [9] showed that  $\mathrm{LS}++$  could easily fall into a poor local optimum when handling real-world datasets. An immediate idea is to apply the multi-swap strategy to  $\mathrm{LS}++$  algorithm for improvements. However, the swapping process of  $\mathrm{LS}++$  relies heavily on the one-to-one matched swap pairs defined in [11]. Thus, it is challenging to apply the multi-swap strategy to solve the  $k$ -means problem while maintaining a linear dependence of data size on the running time.
Secondly, in the process of local search swaps, the time and space complexities for clustering cost update during swaps have great impact on the efficiency of the algorithms. A direct way in [11] is to maintain the nearest and the second nearest centers for each data point such that picking the best swap pair and updating the clustering cost can be implemented in time  $O(nd)$  and  $O(ndk)$ , respectively. Maintaining the distances from data points to their centers requires an extra space complexity of  $O(nd)$ . To obtain faster implementation, as pointed out in [4], one can use binary search trees to store the distances from each data point to each of the clustering centers. By using this data structure, each local search step can be implemented in time  $O(nd\log k)$ . However, the space complexity becomes  $O(ndk)$ . To get much practical algorithms for the  $k$ -means problem, it is necessary to further improve the time and space complexities for updating the clustering cost during swaps.
In order to further narrow the gap between theory and practice, in this paper, we propose the first multi-swap local search algorithm for the  $k$ -means problem with linear running time in the data size. A common feature for the existing multi-swap local search methods is that  $\Theta((nk)^t)$  candidate swaps should be enumerated for finding clustering cost improvements in a single local search step, which leads to at least quadratic running time. To overcome this challenge, our idea is to use a sampling-based strategy to construct a candidate set of centers that are close to the optimal clustering centers in linear time, which serves as the set of potentially good centers for swapping in. Based on the candidate set of centers constructed, enumerations on the current set of centers opened suffice to determine a good swap. Hence, the number of possible swaps can be reduced from  $\Theta((nk)^t)$  to  $\Theta(k^t)$ , where each local search step can be conducted in linear-time in the data size.
Sampling-based strategy has been used in [11] for designing single-swap local search algorithms. However, the theoretical analysis relies heavily on the one-to-one matched swap pairs. Thus, the approximation ratio is a large constant since there may exist some optimal clusters that cannot be well
approximated by performing matched swaps. In this paper, to obtain better approximation guarantee, we extend the notions of swap pairs and propose a new consecutive sampling method to construct candidate centers for swap such that data points close to a subset of optimal clustering centers can be swapped in simultaneously. A key challenge here is that there may exist some optimal clusters whose clustering costs only take a tiny fraction of the total clustering cost such that sampling methods may fail. To overcome this challenge, we propose new structures that divide optimal clusters into different groups for establishing a lower bound for the success probability of sampling.
By using the proposed multi-swap local search method (denoted as MLS algorithm for short), given a swap size  $t$ , an improved  $(50(1 + \frac{1}{t}) + \epsilon)$ -approximation can be obtained in time  $O(ndk^{2t + 1}\log (\epsilon^{-1}\log k))$ . To benefit more from the proposed multi-swap local search method when handling large-scale datasets, we propose a sampling-based method for accelerating the MLS algorithm. The proposed algorithm (denoted as MLSP algorithm) accelerates the updating of clustering cost during the swaps such that each local search iteration in MLS algorithm can be implemented in time  $O(ndk + poly(k)d)$  with extra space complexity of  $\tilde{O} (kd)$ . In order to obtain better clustering quality, we develop a recombination mechanism in MLSP which combines sampling and scoring methods to help the local search algorithm find better solutions when the search falls into a poor local optimum. By picking the top- $k$  data points with the highest scores as new initialization, the chance to get out of the local optimal solutions becomes large. Numerical experiments show that our proposed method achieves better performances compared with branch and bound solver and other local search algorithms. The main contributions of this paper are as follows.
- We propose the first multi-swap local search algorithm (MLS algorithm) with running time linearly dependent on the data size. Given a swap size  $t$  with  $t \geq 2$ , our MLS algorithm achieves a  $(50(1 + \frac{1}{t}) + \epsilon)$ -approximation in time  $O(ndk^{2t + 1}\log (\epsilon^{-1}\log k))$ , which improves the current best approximation ratio 509 with linear running time in the data size.
- We give a practical heuristic algorithm (MLSP algorithm) for better implementation of the proposed MLS algorithm, which accelerates the process of clustering cost update during swaps and provides better scalability to large-scale datasets. Besides, a recombination mechanism is proposed to prevent the local search algorithm falling into a poor local optimum too early.
Table 1 shows a detailed comparison of our results with the state-of-the-art ones.
Result	Approximation Guarantee	Method	Assumption	Running Time
[3]	O(log k)	k-means++	-	O(ndk)
[10]	(3 + 2/t)2	Multi-Swap Local Search	-	O(nt+1kt'd log Δ)
[11]	509	Sampling + Single-Swap Local Search	-	O(ndk2loglogk)
[4]	O(1)	Sampling + Single-Swap Local Search	-	O(ndk log k)
[9]	100 + ε	Sampling + Single-Swap Local Search	|Ph*| ≥ nε/k	O(ndk2 log ε-1)
This Paper	50(1 + 1/t) + ε	Sampling + Multi-Swap Local Search	-	O(ndk2t+1 log(ε-1 log k))
Table 1: Comparison with related results on  $k$ -means clustering, where  $n$  is the size of the given dataset,  $d$  is the dimension,  $k$  is the number of clusters opened,  $t$  is the parameter representing the swap size of local search methods, and  $\Delta$  is the aspect ratio (aspect ratio is defined as the maximum pairwise distance of the given instance divided by the minimum pairwise distance).
We use  $P \subset \mathbb{R}^d$  and  $k$  to denote the given dataset and the number of clusters, respectively. For any two points  $p, q \in P$ , we use  $d(p, q) = \| p - q \|^2$  to denote the squared distance between them. Given two sets  $A, B \subseteq P$ , let  $\Delta(A, B) = \sum_{p \in A} \min_{q \in B} d(p, q)$  denote the sum of the squared distances from data points in  $A$  to their closest points in  $B$ . Let  $C^* = \{c_1^*, c_2^*, \ldots, c_k^*\}$  be an optimal solution. We use  $\mathbb{P}(C^*) = \{P_1^*, P_2^*, \ldots, P_k^*\}$  to denote the corresponding optimal clusters by assigning data points in  $P$  to their closest centers in  $C^*$ . Let  $Opt$  be the cost of the optimal solution. For a subset  $Q \subseteq \mathbb{P}(C^*)$  of optimal clusters, we use  $Z(Q) = \cup_{P_h^* \in Q} P_h^*$  to denote the set of data points in clusters of  $Q$ . Denote  $Z'(Q) = \{c_h^* : P_h^* \in Q\}$  as the set of optimal centers of clusters in  $Q$ . Given a subset  $S \subseteq C^*$ , let  $J(S) = \{P_h^* : c_h^* \in S\}$  be the collection of optimal clusters whose clustering centers are in  $G$ . For an integer  $t$ , let  $[t] = [1, 2, \ldots, t]$ . The following lemma is a folklore for the  $k$ -means problem.
Lemma 1 [3] Let  $P \subseteq \mathbb{R}^d$  be a set of data points, and  $\mu(P) = \frac{1}{|P|} \sum_{p \in P} p$  denote the center of gravity. For any data point  $c \in \mathbb{R}^d$ , we can get  $\Delta(P, \{c\}) = |P| d(\mu(P), c) + \Delta(P, \{\mu(P)\})$ .
Theorem 1 [3] Algorithm 1 returns an  $O(\log k)$ -approximate solution in time  $O(ndk)$ .
Algorithm 1  $k$ -means++
Input: An instance  $(P,k)$  of the  $k$ -means problem.
Output: A set  $C \subseteq \mathbb{R}^d$  of centers with size at most  $k$ .
1: Randomly sample a point  $p \in P$  and set  $C = \{p\}$ .
2: for  $i = 1$  to  $k - 1$  do
3: Pick a point  $p \in P$  with probability  $\Delta(\{p\}, C) / \Delta(P, C)$ , and add  $p$  to  $C$ .
4: return  $C$ .
3 Linear Time Local Search Algorithm with Multi-Swap Strategy
The general idea solving the  $k$ -means problem with multi-swap local search is that we propose a new consecutive sampling method to construct candidate centers for swap such that data points close to a subset of optimal clustering centers can be swapped in simultaneously. The multi-swap local search algorithm (denoted as MLS) is given in Algorithm 2. There are mainly two stages in each round of the MLS algorithm. Given a swap size  $t$ , in the first stage (steps 3 to 5), a candidate set of centers with size  $t$  will be sampled using the  $k$ -means++ method. This avoids enumerating all the data points for constructing the candidate sets for swapping in. In the second stage (steps 6 to 7), the algorithm enumerates all subsets (with size at most  $t$ ) of the candidate set of centers opened for swapping out. By extending the notions of swap pairs to swap set and carefully analyzing the structures of local optimal solutions, we prove that the clustering cost can be reduced significantly with certain probability in each iteration of the MLS Algorithm. The following is the main result of this paper.
Theorem 2 In the  $i$ -th iteration of Algorithm 2, let  $C'$  be the set of centers obtained in step 7. If the current clustering cost  $\Delta(P, C)$  is larger than  $50(1 + \frac{1}{t})\mathrm{Opt}$ , then with probability at least  $\Omega(k^{-t})$ , we have  $\Delta(P, C') \leq (1 - \frac{1}{100k})\Delta(P, C)$ . After  $O(k^{O(t)}\log(\epsilon^{-1}\log k))$  iterations, we get an approximate solution with ratio  $(50(1 + \frac{1}{t}) + \epsilon)$  in expectation<sup>2</sup>.
3.1 Analysis
In this subsection, we analyze our proposed Algorithm 2, where a candidate set of centers for swap is constructed by  $t$  independent sampling steps in each iteration. Our objective is to show that the clustering cost can be reduced with certain probability in each iteration. Due to space limit, all the detailed proofs are given in Appendix A.
In the following, we consider a single iteration of the proposed MLS algorithm. Assume  $\Delta(P, C) \geq 50(1 + \frac{1}{t})\mathcal{O}pt$  holds within a single iteration in Algorithm 2. Otherwise,  $C$  is already a  $50(1 + \frac{1}{t})$ -approximate solution for  $P$ . Let  $C = \{c_1, c_2, \dots, c_k\}$  denote the set of centers before the swap (steps 6-7) of Algorithm 2. Let  $\mathbb{P}(C) = \{P_1, P_2, \dots, P_k\}$  be the corresponding partition of clusters induced by  $C$ . For an optimal cluster  $P_h^*$ , let  $c_h^*$  be its clustering center. For each cluster  $P_h \in \mathbb{P}(C)$ , let  $c_h \in C$  denote its clustering center. Following the ones in [11], we extend the definition of good clusters with respect to  $C^*$  as follows.
Definition 1 Good single cluster. A cluster  $P_h^* \in \mathbb{P}(C^*)$  is called good with a pair of points  $(c_h^*, c_j)$  such that  $c_j \in C$  and  $\Delta(P_h^*, C) - \zeta(P, C, c_h^*, c_j) - 9\Delta(P_h^*, \{c_h^*\}) > \frac{1}{100k}\Delta(P, C)$ , where  $\zeta(P, C, c_h^*, c_j) = \Delta(P \setminus P_h^*, C \setminus \{c_j\}) - \Delta(P \setminus P_h^*, C)$  is the reassignment cost by swapping  $c_j$  out. Otherwise we say that  $P_h^*$  is a bad single cluster with  $(c_h^*, c_j)$ .
Definition 2 Good  $t$ -Clusters. Given an integer  $t$  with  $t \geq 2$ , for a collection of optimal clusters  $Q \subseteq \mathbb{P}(C^*)$  with  $|Q| \leq t$ ,  $Q$  is called good with a pair of sets  $(Z'(Q), V)$  such that  $V \subseteq C$ ,  $|V| = |Z'(Q)|$  and  $\Delta(Z(Q), C) - \zeta(P, C, Z'(Q), V) - 9\Delta(Z(Q), C^*) > \frac{1}{100k}\Delta(P, C)$ , where  $\zeta(P, C, Z'(Q), V) = \Delta(P \backslash Z(Q), C \backslash V) - \Delta(P \backslash Z(Q), C)$  is the reassignment cost by swapping the points in  $V$  out. Otherwise we say that  $Q$  is a set of bad  $t$ -clusters with  $(Z'(Q), V)$ .
2Note that the authors did not try to optimize the constants
Algorithm 2 MLS
Input: An instance  $(P,k)$  of the  $k$ -means problem, parameters  $T$  and  $t$ .
1: Initialize  $C = k$ -means++(P,k).
2: for  $i = 1$  to  $T$  do
3:  $I = \emptyset$
4: for  $j = 1$  to  $t$  do
5: Pick a point  $p \in P$  with probability  $\Delta(\{p\}, C) / \Delta(P, C)$  and add  $p$  to  $I$ .
6: if  $\exists U \subseteq I$  and  $V \subseteq C$  s.t.  $|U| = |V|$  and  $\Delta(P, C \setminus V \cup U) < (1 - \frac{1}{100k})\Delta(P, C)$  then
7:  $C = C\backslash V\cup U.$
8: return  $C$ .
The above definitions estimate the changes of clustering cost by replacing a set of centers  $V \subseteq C$  with a set  $Q'$  of data points that are close to a subset of optimal clustering centers, where a new clustering is constructed by reassigning data points in  $Z(Q)$  to  $Q'$  and data points in  $P \backslash Z(Q)$  to  $C \backslash V$ . The main idea behind Algorithm 2 is to iteratively find data points close enough to the optimal clustering centers for swap to make a reduction on clustering cost by at least  $(1 - \Theta(\frac{1}{k}))$ . In the following, we will prove that with probability at least  $\Omega(k^{-t})$ , the sampling and swap process induces a significant reduction on clustering cost in each iteration.
We start by dividing the optimal clusters into several groups to give an upper bound of reassignment costs. Given a swap size  $t$ , for each optimal center  $c_h^* \in C^*$ , we define  $\Psi(c_h^*)$  as a mapping function that maps  $c_h^*$  to its closest center in  $C$ . For simplicity, we say that  $c_h^*$  is captured by  $\Psi(c_h^*)$ . For a center  $c_j \in C$ , let  $\Psi^{-1}(c_j)$  be the set of optimal centers captured by  $c_j$ . If  $|\Psi^{-1}(c_j)| = 0$ , then  $c_j$  is called a lonely center. If  $|\Psi^{-1}(c_j)| = 1$ , let  $c_h^* = \Psi^{-1}(c_j)$ . Then it is called that  $(c_h^*, c_j)$  forms a type-1 matched swap pair. If  $1 < |\Psi(c_j)| \leq t$ , let  $\sigma_h$  be an arbitrary set of unused lonely centers with  $|\sigma_h| = |\Psi^{-1}(c_j)| - 1$ . Let  $A = \Psi^{-1}(c_j)$  and  $A' = \{c_j\} \cup \sigma_h$ . Then, it is called that  $(A, A')$  forms a type-1 matched swap set. Let  $M_1 = \{(c_h^*, c_j) : (c_h^*, c_j) \text{ is a type-1 matched swap pair}\}$  and  $M_2 = \{(A, A') : (A, A') \text{ is a type-1 matched swap set}\}$  be the collections of type-1 matched swap pairs and type-1 matched swap sets, respectively. If  $|\Psi^{-1}(c_j)| > t$ , then find each lonely center  $c_q \in C$  that has not been used for constructing type-1 matched swap pairs or sets. For each lonely center  $c_q$  and each  $c_h^* \in \Psi^{-1}(c_j)$ ,  $(c_h^*, c_q)$  forms a type-2 matched swap pair. We use  $M_3 = \{(c_h^*, c_q) : (c_h^*, c_q) \text{ is a type-2 matched swap pair}\}$  to denote the set of type-2 matched swap pairs. For a set  $V \subseteq C$  of centers, let  $X(V) = \cup_{c_h \in V} P_h$  be the set of data points in  $P$  whose closest centers are in  $V$ . Given a subset  $S \subseteq C$  of clustering centers, we also use  $J(S) = \{P_h : c_h \in G\}$  to denote the set of clusters whose centers are in  $S$ . The following lemma gives upper bounds of reassignment cost by matched swap pairs or sets.
Lemma 2 Given a type-1 or type-2 matched swap pair  $(c_h^*, c_j)$ , it holds that  $\zeta(P, C, c_h^*, c_j) \leq 24\Delta(P_j, C^*) + \frac{1}{5}\Delta(P_j, C)$ . Given a type-1 matched swap set  $(Q, V)$ , it holds that  $\zeta(P, C, Q, V) \leq 24\Delta(X(V), C^*) + \frac{1}{5}\Delta(X(V), C)$ .
Let  $H_{1} = \{c_{h}^{*}:(c_{h}^{*},c_{j})\in M_{1}\}$  be the set of optimal centers that participate in constructing type-1 matched swap pair. Let  $H_{2} = \{A:(A,A^{\prime})\in M_{2}\}$  be the collection of the subsets of optimal centers that participate in constructing type-1 matched swap set. Let  $L = \{c_h^*:(c_h^*,c_q)\in M_3\}$  be the set of optimal centers that participate in constructing type-2 matched swap pair. Let  $H_1^\prime = \{c_j:(c_h^*,c_j)\in M_1\}$  be the set of centers in  $C$  that participate in constructing type-1 matched swap pair. Let  $H_2^\prime = \{A^\prime :(A,A^\prime)\in M_2\}$  be the collection of the subsets of centers in  $C$  that participate in constructing type-1 matched swap set. Let  $L^{\prime} = \{c_{q}:(c_{h}^{*},c_{q})\in M_{3}\}$  denote the set of centers in  $C$  that participate in constructing type-2 matched swap pair.
During the sampling and swap process in steps 4-7 of Algorithm 2, there are two cases that may happen: (1)  $\exists c_h^* \in L$  such that  $P_h^*$  is a good single cluster with a type-2 matched swap pair  $(c_h^*, c_q) \in M_3$ ; (2)  $\forall c_h^* \in L$ ,  $P_h^*$  is a bad single cluster with any type-2 matched swap pair  $(c_h^*, c_q) \in M_3$ . We will discuss the two cases separately in the following. If case (1) happens, we first show that with probability at least  $\Omega\left(\frac{1}{k}\right)$ , the clustering cost can be reduced at least by  $1 - \Theta\left(\frac{1}{k}\right)$ . Let  $c_h^* \in L$  be an optimal center such that  $P_h^*$  is a good single cluster with a type-2 matched swap pair  $(c_h^*, c_q) \in M_3$ . By the definition of good single cluster, we have  $\Delta(P_h^*, C) \geq 9\Delta(P_h^*, \{c_h^*\})$ .
Define  $\chi(P_h^*) = \{p \in P_h^* : d(p, c_h^*) \leq \frac{1.5\Delta(P_h^*, \{c_h^*\})}{|P_h^*|}\}$  as the set of data points in  $P_h^*$  that are close to the optimal center  $c_h^*$ . Observe that  $|\chi(P_h^*)| \geq \frac{1}{3} |P_h^|$ . Otherwise, the clustering cost of data points in  $P_h^* \backslash \chi(P_h^*)$  is at least  $\Delta(P_h^*, \{c_h^*\})$  using  $c_h^*$  as center, which contradicts with  $\Delta(P_h^* \backslash \chi(P_h^), \{c_h^*\}) < \Delta(P_h^*, \{c_h^*\})$ .
Next, we will show that whenever an optimal cluster  $P_{h}^{*}$  has large clustering cost with respect to  $C$ , i.e.,  $\Delta(P_{h}^{*}, C) = b\Delta(P_{h}^{*}, \{c_{h}^{*}\})$  for a real number  $b \geq 3$ , it suffices to use data points in  $\chi(P_{h}^{*})$  to approximate the clustering cost of  $P_{h}^{*}$ .
Lemma 3 Let  $P_h^*$  be an optimal cluster with  $\Delta(P_h^*, C) = b\Delta(P_h^*, \{c_h^*\})$  for a real number  $b \geq 3$ . Then,  $\Delta(\chi(P_h^*), C) \geq \frac{1}{200} (b - 1)\Delta(P_h^*, \{c_h^*\})$ .