SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tables 12 and 13 show the results on 18 datasets with sizes smaller than 50,000 using  $k = 10$ . It can be seen that, for each dataset, the best clustering cost returned by our MLSP algorithm is smaller than BB method and other local search methods. For each dataset, the average clustering cost returned by our MLSP algorithm nearly matches the result of the BB method. As for FLS, although the performance of FLS is better than that of LS++, our proposed MLSP algorithm improves the performance of FLS on clustering cost with smaller deviation on most datasets. As for running time, there is no significant difference on running time among different local search algorithms for small datasets. Tables 14 and 15 present the performances of different local search algorithms with  $k = 3$  on small datasets. Table 18 presents the performances of different local search algorithms with  $k = 3$  on large datasets. Tables 16 and 17 present the performances of different local search algorithms with  $k = 5$  on small datasets. Table 19 presents the performances of different local search algorithms with  $k = 5$  on large datasets. It can be seen that, our proposed MLSP algorithm achieves the best clustering performance on most datasets with different values of  $k$ . On each dataset, the best clustering cost returned by our MLSP algorithm matches the clustering cost of BB method. On each dataset, the average clustering cost returned by our MLSP algorithm nearly matches the result of the BB method. As for running time, there is no significant difference on running time among all local search algorithms on small datasets. However, as the data sizes grow, our proposed MLSP algorithm becomes much faster than other local search algorithms.
In this paper, we propose fast local search algorithms for the  $k$ -means problem with multi-swap strategy, which runs in linear time in the data size. We develop new sampling techniques, which accelerate the process of clustering cost update during swaps. By proposing a recombination mechanism, the proposed algorithm can find potentially better solutions. Experimental results show that our algorithms achieve better performance on both small and large datasets compared with the state-of-the-art algorithms. An interesting future direction is how to design fast local search approximation algorithms for handling high dimensional clustering datasets.
This work was supported by National Natural Science Foundation of China (62172446, 62350004, 62332020), Open Project of Xiangjiang Laboratory (22XJ02002, 22XJ03005), and Central South University Research Programme of Advanced Interdisciplinary Studies (2023QYJC023). This work was also carried out in part using computing resources at the High Performance Computing Center of Central South University.
A Missing Proofs in Section 3
Lemma 2. Given a type-1 or type-2 matched swap pair  $(c_h^*, c_j)$ , it holds that  $\zeta(P, C, c_h^*, c_j) \leq 24\Delta(P_j, C^*) + \frac{1}{5}\Delta(P_j, C)$ . Given a type-1 matched swap set  $(Q, V)$ , it holds that  $\zeta(P, C, Q, V) \leq 24\Delta(X(V), C^*) + \frac{1}{5}\Delta(X(V), C)$ .
Proof We first consider a type-1 matched swap set  $(Q,V)$ . For a data point  $p\in P$ , we use  $s_p$  to denote its closest center in  $C$ . Let  $o_{p}$  be its closest optimal center in  $C^*$ . Observe that
$$
\begin{array}{l} \zeta (P,C,Q,V)\leq \sum_{p\in X(V)\backslash Z(J(Q))}d(p,s_{o _{p}}) - d(p,s_{p}) \\ \leq \sum_ {p \in X (V) \backslash Z (J (Q))} \left(\sqrt {d (p , o _ {p})} + \sqrt {d (o _ {p} , s _ {o _ {p}})}\right) ^ {2} - d (p, s _ {p}) \\ \leq \sum_ {p \in X (V) \backslash Z (J (Q))} \left(\sqrt {d (p , o _ {p})} + \sqrt {d (o _ {p} , s _ {p})}\right) ^ {2} - d (p, s _ {p}) \\ \leq \sum_ {p \in X (V) \backslash Z (J (Q))} \left(2 \sqrt {d (p , o _ {p})} + \sqrt {d (p , s _ {p})}\right) ^ {2} - d (p, s _ {p}) \\ \leq \sum_ {p \in X (V) \backslash Z (J (Q))} 4 d (p, o _ {p}) + 2 \sqrt {\frac {2}{\lambda}} \sqrt {2 \lambda d (p , o _ {p}) d (p , s _ {p})} \\ \leq \sum_ {p \in X (V) \backslash Z (J (Q))} \left(4 + \frac {2}{\lambda}\right) d \left(p, o _ {p}\right) + 2 \lambda d \left(p, s _ {p}\right), \\ \end{array}
$$
where the first inequality follows from the fact that  $s_{o_p}$  is still in  $C\backslash V$  for each data point  $p\in X(V)\backslash Z(J(Q))$  according to the definition of the swap set defined, the second and fourth steps follow from the triangle inequality, the third step follows from the fact that  $s_{o_p}$  is the nearest point to  $o_p$  in  $C$ , and the last step follows from Cauchy Inequality. Let  $\lambda = \frac{1}{10}$ . Then  $\zeta (P,C,Q,V)\leq 24\Delta (X(V),C^{*}) + \frac{1}{5}\Delta (X(V),C)$ . The cases for matched swap pair are similar to the cases for matched swap set. For a matched swap pair  $(c_h^*,c_j)$ , we can get that  $\zeta (P,C,c_h^*,c_j)\leq 24\Delta (P_j,C^*) + \frac{1}{5}\Delta (P_j,C)$ .
Lemma 3. Let  $P_h^*$  be an optimal cluster with  $\Delta(P_h^*, C) = b\Delta(P_h^*, \{c_h^*\})$  for a real number  $b \geq 3$ . Then,  $\Delta(\chi(P_h^*), C) \geq \frac{1}{200} (b - 1)\Delta(P_h^*, \{c_h^*\})$ .
Proof Let  $s_{c_h^*}$  be the closest center in  $C$  to  $c_h^*$ . Observe that  $d(c_h^*, s_{c_h^*}) \geq \frac{(b - 1)\Delta(P_h^*, \{c_h^*\})}{|P_h^*|}$ . Otherwise  $\Delta(P_h^*, \{s_{c_h^*}\}) < b\Delta(P_h^*, \{c_h^*\})$  holds according to Lemma 1. Consider an arbitrary data point  $p \in \chi(P_h^*)$ . By triangle inequality, we have
$$
\sqrt {\Delta (\{p \} , C)} \geq \sqrt {d (c _ {h} ^ {*} , s _ {c _ {h} ^ {*}})} - \sqrt {d (p , c _ {h} ^ {*})} \geq \sqrt {\frac {\Delta (P _ {h} ^ {*} , \{c _ {h} ^ {*} \})}{| P _ {h} ^ {*} |}} (\sqrt {b - 1} - \sqrt {1 . 5}).
$$
Since  $\sqrt{b - 1} \geq \sqrt{2}$ , we have
$$
\Delta (\{p \}, C) \geq \left(1 - \sqrt {\frac {3}{4}}\right) ^ {2} \frac {(b - 1) \Delta \left(P _ {h} ^ {*} , \left\{c _ {h} ^ {*} \right\}\right)}{\left| P _ {h} ^ {*} \right|}.
$$
Together with the fact that  $|\chi(P_h^*)| \geq \frac{1}{3} |P_h^*|$ , we have  $\Delta(\chi(P_h^*), C) \geq \frac{b - 1}{200} \Delta(P_h^*, \{c_h^*\})$ , which proves the Lemma.
Lemma 4. By swapping  $q$  with  $c_{q}$ , the clustering cost of  $\Delta(P, C)$  can be reduced at least by  $1 - \Theta\left(\frac{1}{k}\right)$ .
Proof Observe that  $\Delta(P_h^*, \{q\}) < 9\Delta(P_h^*, \{c_h^*\})$  holds by Lemma 1. Then, we can get that
$$
\begin{array}{l} \Delta (P, C \backslash \{c _ {q} \} \cup \{q \}) = \Delta (P, C) - (\Delta (P, C) - \Delta (P, C \backslash \{c _ {q} \} \cup \{q \})) \\ = \Delta (P, C) - \left(\Delta \left(P _ {h} ^ {*}, C\right) + \Delta \left(P \backslash P _ {h} ^ {*}, C\right) - \Delta \left(P _ {h} ^ {*}, C \backslash \left\{c _ {q} \right\} \cup \{q \}\right) \right. \\ - \Delta \left(P \backslash P _ {h} ^ {*}, C \backslash \{c _ {q} \} \cup \{q \}\right)) \\ \leq \Delta (P, C) - (\Delta \left(P _ {h} ^ {*}, C\right) - \zeta (P, C, c _ {h} ^ {*}, c _ {q}) - 9 \Delta \left(P _ {h} ^ {*}, \left\{c _ {h} ^ {*} \right\}\right)) \\ \leq (1 - \frac {1}{1 0 0 k}) \Delta (P, C), \\ \end{array}
$$
where the second to the last inequality follows from the fact that  $\Delta(P_h^*, \{q\}) < 9\Delta(P_h^*, \{c_h^*\})$ , and the last inequality follows from the definition of good single cluster.
Lemma 5. If case (2) happens, then  $\Delta(X(L), C) \leq (1 + \frac{1}{t})(9\Delta(X(L), C^*) + 24\Delta(X(L'), C^*) + \frac{1}{5}\Delta(X(L'), C) + \frac{1}{100}\Delta(P, C))$ .
Proof By the condition of case (2), it holds that all the optimal clusters in  $J(L)$  belong to bad single cluster. For any bad single cluster  $P_h^* \in J(L)$ , let  $(c_h^*, c_h^m) \in M_3$  be the type-2 matched swap pair with minimum reassignment cost, i.e.,  $(c_h^*, c_h^m) = \arg \min_{(c_h^*, c_n) \in M_3} \zeta(P, C, c_h^*, c_n)$ . According to the definition of bad single cluster, we have
$$
\Delta \left(P _ {h} ^ {*}, C\right) \leq \frac {1}{1 0 0 k} \Delta (P, C) + 9 \Delta \left(P _ {h} ^ {*}, \left\{c _ {h} ^ {*} \right\}\right) + \zeta \left(P, C, c _ {h} ^ {*}, c _ {h} ^ {m}\right).
$$
Let  $\kappa(M_3)$  be the set of lonely centers used for constructing type-2 matched swap pair. For an optimal center  $c_h^* \in C^*$ , denote  $s_{c_h^*}$  as the center in  $C$  closest to  $c_h^*$ . Let  $L_1 = \cup_{c_h^* \in L} s_{c_h^*}$ . By the definitions of  $\kappa(M_3)$  and  $L$ , for each  $s_{c_h^*} \in L_1$ , we can find a set  $z(s_{c_h^*}) \subseteq \kappa(M_3)$  with size  $|\Psi^{-1}(s_{c_h^*})| - 1$  such that  $z(a) \cap z(b) = \emptyset$  for any  $a, b \in L_1$ . For each  $s_{c_h^*} \in L_1$ , since  $|\Psi^{-1}(s_{c_h^*})| > t$ , it holds that  $|\Psi^{-1}(s_{c_h^*})| / |z(s_{c_h^*})| \leq 1 + \frac{1}{t}$ . By taking a summation over all the centers in  $L$ , we have  $|L| = \sum_{c_h^* \in L} 1 = \sum_{s_{c_h^*} \in L_1} |\Psi^{-1}(s_{c_h^*})| \leq \sum_{s_{c_h^*} \in L_1} |z(s_{c_h^*})|(1 + \frac{1}{t}) \leq |\kappa(M_3)|(1 + \frac{1}{t})$ . Then, by considering all the optimal centers in  $L$  and taking a summation, we have
$$
\begin{array}{l} \Delta (X (L), C) = \sum_ {c _ {h} ^ {*} \in L} \Delta \left(P _ {h} ^ {*}, C\right) \leq \sum_ {c _ {h} ^ {*} \in L} \frac {1}{1 0 0 k} \Delta (P, C) + 9 \Delta \left(P _ {h} ^ {*}, \left\{c _ {h} ^ {*} \right\}\right) + \zeta (P, C, c _ {h} ^ {*}, c _ {h} ^ {m}) \\ \leq \frac {1}{1 0 0} \Delta (P, C) + 9 \Delta (X (L), C ^ {*}) + \sum_ {c _ {h} ^ {*} \in L} \frac {\sum_ {c \in \kappa (M _ {3})} \zeta (P , C , c _ {h} ^ {*} , c)}{| \kappa (M _ {3}) |} \\ \leq \left(1 + \frac {1}{t}\right) \left(9 \Delta (Z (J (L)), C ^ {*}) + 2 4 \Delta \left(X \left(L ^ {\prime}\right), C ^ {*}\right) \right. \\ + \frac {1}{5} \Delta (X (L ^ {\prime}), C) + \frac {1}{1 0 0} \Delta (P, C)), \\ \end{array}
$$
where the second to the last inequality follows from the fact that  $c_h^m$  is the center with minimum reassignment cost, and the last inequality follows from Lemma 2 and the fact that  $|L| / |\kappa(M_3)| \leq 1 + t^{-1}$ .
Lemma 6. Let  $Q \in H_2$  be a set of centers in  $H_2$ , where  $J(Q)$  is a set of good  $t$ -clusters with a type-1 matched swap set  $(Q, A') \in M_2$ . Define  $V = A' \setminus \{c_j\}$ , where  $c_j$  is the center in  $A'$  with  $|\Psi^{-1}(c_j)| > 1$ . Let  $U \subseteq P$  be the set of data points with  $|U| = |V|$  such that  $U \cap \chi(P_h^*) \neq \emptyset$  holds for each  $P_h^* \in Q_T$ . Then,  $\Delta(P, C \setminus V \cup U) \leq (1 - \frac{1}{100k})\Delta(P, C)$ .
Proof Consider an arbitrary optimal cluster  $P_h^* \in Q_S'$ . We know that  $\Delta(P_h^*, C) < 3\Delta(P_h^*, \{c_h^*\})$  by the definition of  $Q_S'$ . Let  $s_{c_h^*}$  be the closest center in  $C$  to  $c_h^*$ . We first argue that  $s_{c_h^*}$  is close to  $c_h^*$  with  $d(c_h^*, s_{c_h^*}) \leq \frac{8\Delta(P_h^*, \{c_h^*\})}{|P_h^*|}$ . For a data point  $p \in P_h^*$ , let  $s_p$  be its closest center in  $C$ . According
to the triangle inequality, we have
$$
\begin{array}{l} d \left(c _ {h} ^ {*}, s _ {c _ {h} ^ {*}}\right) \leq \frac {\sum_ {p \in P _ {h} ^ {*}} d \left(c _ {h} ^ {*} , s _ {p}\right)}{\left| P _ {h} ^ {*} \right|} \leq \frac {1}{\left| P _ {h} ^ {*} \right|} \sum_ {p \in P _ {h} ^ {*}} (1 + \lambda) d \left(c _ {h} ^ {*}, p\right) + (1 + \frac {1}{\lambda}) d (p, s _ {p}) \\ \leq \frac {(1 + \lambda) \Delta (P _ {h} ^ {*} , \{c _ {h} ^ {*} \}) + (1 + \frac {1}{\lambda}) \Delta (P _ {h} ^ {*} , C)}{| P _ {h} ^ {*} |} \\ \leq \frac {8 \Delta (P _ {h} ^ {*} , \{c _ {h} ^ {*} \})}{| P _ {h} ^ {*} |}, \\ \end{array}
$$
where the last inequality follows from  $\Delta(P_h^*, C) < 3\Delta(P_h^*, \{c_h^*\})$  and feeding  $\lambda = \sqrt{3}$ . By assigning all the data points in  $P_h^*$  to  $s_{c_h^*}$ , we have  $\Delta(P_h^*, \{s_{c_h^*}\}) \leq 9\Delta(P_h^*, \{c_h^*\})$  by Lemma 1. Then, by swapping  $U$  with  $V$ , we have that
$$
\begin{array}{l} \Delta (P, C \backslash V \cup U) = \Delta (P, C) - (\Delta (P, C) - \Delta (P, C \backslash V \cup U)) \\ \leq \Delta (P, C) - (\Delta (Z (J (Q)), C) + \Delta (P \backslash Z (J (Q)), C) - \Delta (Z (Q _ {T}), U) \\ - \Delta \left(Z \left(Q _ {S} ^ {\prime}\right), \left\{s _ {c _ {h} ^ {*}} ^ {*} \right\}\right) - \Delta \left(P \backslash Z (J (Q)), C \backslash \left(V \cup \left\{s _ {c _ {h} ^ {*}} ^ {*} \right\}\right)\right) \\ \leq \Delta (P, C) - (\Delta (Z (J (Q)), C) - 9 \Delta (Z (J (Q)), C ^ {*}) - \zeta (P, C, Q, V)) \\ \leq (1 - \frac {1}{1 0 0 k}) \Delta (P, C), \\ \end{array}
$$
where the first inequality follows from the fact that  $s_{c_h^*} \notin V$  for each  $P_h^* \in Q_S'$ , the second inequality follows from  $\Delta(P_h^*, \{s_{c_h^*}\}) \leq 9\Delta(P_h^*, \{c_h^*\})$  for each  $P_h^* \in Q_S'$ , and the last inequality follows from the definition of good  $t$ -clusters and  $V \cup \{s_{c_h^*}\} = A'$ .
Lemma 7. Given a set  $Q \in H_2$  of centers such that  $J(Q)$  is a set of good  $t$ -clusters, the probability that step 4 of Algorithm 2 fails to sample a data point  $q$  from  $\chi(P_h^*)$  for each  $P_h^* \in Q_S''$  in the  $m(P_h^*)$ -th iteration is at most  $(1 + \frac{1}{30000k}) e^{-\Delta(Z(Q_S'', C) / (300\Delta(P,C))}$ .
Proof We first consider an optimal cluster  $P_h^* \in Q_S''$ . Observe that  $\Delta(P_h^*, C) < \frac{1}{30000k2^{t-1}} \Delta(P, C)$  by the definition of  $Q_S''$ . Define  $p_f^h$  as the failure probability of not sampling data points from  $\chi(P_h^*)$  in the  $m(P_h^*)$ -th iteration of step 4 in Algorithm 2. Then, we have that
$$
p _ {f} ^ {h} = 1 - \frac {\Delta (\chi (P _ {h} ^ {*} , C)}{\Delta (P , C)} \geq 1 - \frac {\Delta (P _ {h} ^ {*} , C)}{\Delta (P , C))} \geq 1 - \frac {1}{3 0 0 0 0 k 2 ^ {t - 1}}.
$$
Let  $p_s^h = 1 - p_f^h$  be the success probability. Then, we have
$$
p _ {s} ^ {h} = 1 - p _ {f} ^ {h} \leq (\frac {1}{1 - \frac {1}{3 0 0 0 0 k 2 ^ {t - 1}}} - 1) p _ {f} ^ {h} \leq \frac {\frac {1}{3 0 0 0 0}}{k 2 ^ {t - 1} - \frac {1}{3 0 0 0 0}} p _ {f} ^ {h} \leq \frac {1}{3 0 0 0 0 k (2 ^ {t - 1} - 1)} p _ {f} ^ {h},
$$
where the last inequality follows from  $\frac{1}{30000} < k$ . Let  $p_f(Q_h^{\prime \prime})$  denote the probability of not sampling a data point from  $\chi(P_h^*)$  for each  $P_h^* \in Q_S''$  in the  $m(P_h^*)$ -th iteration of step 4 in Algorithm 2. For each  $P_h^* \in Q_S''$ , define  $X_h = 1$  if the  $m(P_h^*)$ -th iteration of step 4 in Algorithm 2 samples a data point  $q \in \chi(P_h^*)$ , and  $X_h = 0$  if the  $m(P_h^*)$ -th iteration fails to sample a data point  $q \in \chi(P_h^*)$ . Let  $E_1$  be the event that  $X_h = 0$  for each  $P_h^* \in Q_S''$ , and  $E_2$  be the event that there are at least two clusters  $P_h^*$ ,  $P_j^* \in Q_S''$  such that  $X_h \vee X_j = 1$  and  $X_h \wedge X_j = 0$ . Observe that there are at most  $2^t - 2$  subcases if event  $E_2$  happens. For each subcase  $s$ , let  $\nu(s) = \{P_h^* : P_h^* \in Q_S'', X_h = 1\}$  and  $\nu'(s) = \{P_h^* : P_h^* \in Q_S'', X_h = 0\}$ , respectively. Define  $P_r(s)$  as the probability that subcase  $s$  happens. Since the sampling iterations in step 4 of Algorithm 2 are mutually independent, we have  $P_r(s) = \prod_{P_h^* \in \nu(s)} p_s^h \prod_{P_h^* \in \nu'(s)} p_f^h$ . By applying the inequality that  $p_s^h \leq \frac{1}{3000k(2^{t-1}-1)} p_f^h$ , we have
$$
P _ {r} (s) \leq \left(\frac {1}{3 0 0 0 0 k \left(2 ^ {t - 1} - 1\right)}\right) ^ {t} \prod_ {P _ {h} ^ {*} \in Q _ {S} ^ {\prime \prime}} p _ {f} ^ {h} \leq \frac {1}{3 0 0 0 0 k \left(2 ^ {t - 1} - 1\right)} \prod_ {P _ {h} ^ {*} \in Q _ {S} ^ {\prime \prime}} p _ {f} ^ {h}
$$
for each subcase  $s$  of  $E_{2}$ , where the last inequality follows from  $\frac{1}{30000k(2^{t - 1} - 1)} < 1$ . Then, we have  $P_{r}(E_{2}) \leq \frac{1}{30000k} \prod_{P_{h}^{*} \in Q_{S}^{\prime \prime}} p_{f}^{h}$  by taking a probability summation over all  $2^{t} - 2$  subcases of event
$E_{2}$ . Observe that the failure probability  $p_f(Q_S^{\prime \prime}) = P_r(E_1) + P_r(E_2)$ , where  $P_{r}(E_{1}) = \prod_{P_{h}^{*}\in Q_{S}^{\prime \prime}}p_{f}^{h}$ . Thus, we can get that
$$
\begin{array}{l} p _ {f} \left(Q _ {S} ^ {\prime \prime}\right) \leq \left(1 + \frac {1}{3 0 0 0 0 k}\right) \prod_ {P _ {h} ^ {*} \in Q _ {S} ^ {\prime \prime}} p _ {f} ^ {h} = \left(1 + \frac {1}{3 0 0 0 0 k}\right) \prod_ {P _ {h} ^ {*} \in Q _ {S} ^ {\prime \prime}} \left(1 - p _ {s} ^ {h}\right) \\ = \left(1 + \frac {1}{3 0 0 0 0 k}\right) \prod_ {P _ {h} ^ {*} \in Q _ {S} ^ {\prime \prime}} \left(1 - \frac {\Delta \left(\chi \left(P _ {h} ^ {*}\right) , C\right)}{\Delta (P , C)}\right) \leq \left(1 + \frac {1}{3 0 0 0 0 k}\right) \prod_ {P _ {h} ^ {*} \in Q _ {S} ^ {\prime \prime}} \left(1 - \frac {\Delta \left(P _ {h} ^ {*} , C\right)}{3 0 0 \Delta (P , C)}\right) \\ \leq \left(1 + \frac {1}{3 0 0 0 0 k}\right) e ^ {- \Delta \left(Z \left(Q _ {S} ^ {\prime \prime}, C\right)\right) / 3 0 0 \Delta (P, C)}, \\ \end{array}
$$
where the second to last inequality follows from Lemma 3, and the last inequality follows from the inequality  $1 - x \leq e^{-x}$ .
Lemma 8. For the optimal clusters in  $H_{G}$ , we have  $\Delta (Z(H_{G}),C)\geq \frac{1}{100}\Delta (P,C)$ .
Proof Given any type-1 matched swap set  $(Q,V)$ , if  $J(Q)$  is a set of good  $t$ -clusters with a type-1 matched swap set  $(Q,V) \in M_2$ , according to the assumption of subcase (2), we know that each optimal cluster  $P_h^*$  in  $Q_L$  is a bad single cluster with a swap pair  $(c_h^*, c_l)$ , where  $c_l$  is a lonely center in  $V$ . Since  $J(Q)$  is a set of good  $t$ -clusters, not all optimal clusters in  $J(Q)$  are bad single clusters. Hence, for each optimal cluster  $P_h^* \in Q_L$ , we can find a lonely center  $l(c_h^*) \in V$  such that  $P_h^*$  is a bad single cluster with the swap pair  $(c_h^*, l(c_h^*))$ . By the definition of bad single cluster, we have
$$
\Delta \left(P _ {h} ^ {*}, C\right) \leq \frac {1}{1 0 0 k} \Delta (P, C) + 9 \Delta \left(P _ {h} ^ {*}, \left\{c _ {h} ^ {*} \right\}\right) + \zeta \left(P, C, c _ {h} ^ {*}, l \left(c _ {h} ^ {*}\right)\right).
$$
For each optimal cluster  $P_h^* \in Q_S'$ , by the definition of  $Q_S'$ , it holds that  $\Delta(P_h^*, C) \leq 3\Delta(P_h^*, \{c_h^*\})$ . If  $J(Q)$  is a set of bad  $t$ -clusters, by the definition of bad  $t$ -clusters, we also have
$$
\Delta (Z (J (Q)), C) \leq \frac {1}{1 0 0 k} \Delta (P, C) + 9 \Delta (Z (J (Q)), C ^ {*}) + \zeta (P, C, Q, V).
$$
Define  $H_B^1 = \{c_h^* \in H_1 : P_h^*$  is a bad single cluster with the swap pair  $(c_h^*, c_j) \in M_1\}$ ,  $H_B^2 = \{Q \in H_2 : J(Q)$  is a set of bad  $t$ -clusters with the swap set  $(Q, V) \in M_2\}$  and  $H_{B'}^2 = \{Q_S' \cup Q_L : Q \in H_2, J(Q)$  is a set of good  $t$ -clusters with the swap set  $(Q, V) \in M_2\}$ , respectively. Putting all things together and taking a summation over all clusters in  $H_B$ , we can get that
$$
\begin{array}{l} \Delta (Z(H_{B}),C) = \sum_{P_{h}^{*}\in H_{B}}\Delta (P_{h}^{*},C) \\ = \sum_ {c _ {h} ^ {*} \in H _ {B} ^ {1}} \Delta (P _ {h} ^ {*}, C) + \sum_ {Q \in H _ {B} ^ {2}} \Delta (Z (J (Q)), C) + \sum_ {Q ^ {\prime} \in H _ {B ^ {\prime}} ^ {2}} \Delta (Z (Q ^ {\prime}), C) \\ \leq \sum_ {P _ {h} ^ {*} \in H _ {B}} \frac {1}{1 0 0 k} \Delta (P, C) + 9 \Delta \left(P _ {h} ^ {*}, \left\{c _ {h} ^ {*} \right\}\right) + 2 4 \Delta \left(X \left(H _ {1} ^ {\prime}\right), C ^ {*}\right) + \frac {1}{5} \Delta \left(X \left(H _ {1} ^ {\prime}\right), C\right) \\ + 2 4 \Delta \left(X \left(H _ {2} ^ {\prime}\right), C ^ {*}\right) + \frac {1}{5} \Delta \left(X \left(H _ {2} ^ {\prime}\right), C\right) \\ \leq \frac {1}{1 0 0} \Delta (P, C) + 9 \Delta \left(Z \left(H _ {B}\right), C ^ {*}\right) + 2 4 \Delta \left(X \left(H _ {1} ^ {\prime}\right), C ^ {*}\right) + \frac {1}{5} \Delta \left(X \left(H _ {1} ^ {\prime}\right), C\right) \\ + 2 4 \Delta (X (H _ {2} ^ {\prime}), C ^ {*}) + \frac {1}{5} \Delta (X (H _ {2} ^ {\prime}), C), \\ \end{array}
$$
where the first inequality follows from Lemma 3. Then, together with Lemma 5, we can get that
$$
\begin{array}{l} \Delta (Z (H _ {G}), C) \geq \Delta (P, C) - \Delta (Z (J (L)), C) - \Delta (Z (H _ {B}), C) \\ \geq \Delta (P, C) - \left(1 + \frac {1}{t}\right) \left(9 \Delta \left(Z (J (L)), C ^ {*}\right) + 2 4 \Delta \left(X \left(L ^ {\prime}\right), C ^ {*}\right) + \frac {1}{5} \Delta \left(X \left(L ^ {\prime}\right), C\right) \right. \\ + \frac {1}{1 0 0} \Delta (P, C) + \frac {1}{1 0 0} \Delta (P, C) + 9 \Delta \left(Z \left(H _ {B}\right), C ^ {*}\right) + 2 4 \Delta \left(X \left(H _ {1} ^ {\prime}\right), C ^ {*}\right) \\ + \frac {1}{5} \Delta \left(X \left(H _ {1} ^ {\prime}\right), C\right)) + 2 4 \Delta \left(X \left(H _ {2} ^ {\prime}\right), C ^ {*}\right) + \frac {1}{5} \Delta \left(X \left(H _ {2} ^ {\prime}\right), C\right)) \\ \geq \Delta (P, C) - (1 + \frac {1}{t}) (\frac {1}{5 0} \Delta (P, C) + \frac {1}{5} \Delta (P, C) + 3 3 O p t) \\ \geq \Delta (P, C) - \frac {3 3}{1 0 0} \Delta (P, C) - \frac {3 3}{5 0} \Delta (P, C) \\ \geq \frac {1}{1 0 0} \Delta (P, C), \\ \end{array}
$$
where the second inequality follows from Lemma 5, the third inequality follows from the fact that  $H_1' \cap H_2' \cap L' = \emptyset$  and  $Z(H_B) \cap J(L) = \emptyset$ , and the fourth inequality follows from  $t \geq 2$  and the assumption that  $\Delta(P, C) \geq 50(1 + \frac{1}{t})Opt$ .
Theorem 2. In the  $i$ -th iteration of Algorithm 2, let  $C'$  be the set of centers obtained in step 7. If the current clustering cost  $\Delta(P, C)$  is larger than  $50(1 + \frac{1}{t})\mathrm{Opt}$ , then with probability at least  $\Omega(k^{-t})$ , we have  $\Delta(P, C') \leq (1 - \frac{1}{100k})\Delta(P, C)$ . After  $O(k^{O(t)}\log(\epsilon^{-1}\log k))$  iterations, we get an approximate solution with ratio  $(50(1 + \frac{1}{t}) + \epsilon)$  in expectation.
Proof Let  $T = \delta k^{t + 1}\log (24\epsilon^{-1}\log k)$ , where  $\delta$  is a sufficient large constant. Following the work in [11], we define another random process  $X$  with initial clustering cost  $\Delta (P,C^{\prime})$  for a set  $C^\prime$  returned by the  $k$ -means++ algorithm such that for  $T$  iterations of sampling and swaps, it reduces the value of  $\Delta (P,C^{\prime})$  by at least  $(1 - \frac{1}{100k})$  with probability  $\lambda k^{-t}$ , and it increases the final value of  $\Delta (P,C^{\prime})$  by  $50(1 + \frac{1}{t})OPT$ , where  $\lambda$  is a constant with  $\lambda < \frac{\delta}{100}$ . It is easy to see that  $E[\Delta (P,C)] < E[X]$ . Then, we have
$$
\begin{array}{l} E [ X ] = 5 0 (1 + \frac {1}{t}) O p t \\ + \Delta (P, C ^ {\prime}) \sum_ {i = 1} ^ {T} \binom {T} {i} (\frac {1}{\lambda k ^ {t}}) ^ {i} (1 - \frac {1}{\lambda k ^ {t}}) ^ {T - i} (1 - \frac {1}{1 0 0 k}) ^ {i} \\ = \Delta (P, C ^ {\prime}) \left(1 - \frac {1}{1 0 0 \lambda k ^ {t + 1}}\right) ^ {T} + 5 0 \left(1 + \frac {1}{t}\right) O p t \\ \leq \frac {\epsilon \Delta (P , C ^ {\prime})}{2 4 \log k} + 5 0 (1 + \frac {1}{t}) O p t. \\ \end{array}
$$
This implies that  $E[\Delta(P, C) | C'] \leq \frac{\epsilon \Delta(P, C')}{24 \log k} + 50(1 + \frac{1}{t}) Opt$ . Then, we can get that
$$
\begin{array}{l} E [ \Delta (P, C) ] = \sum_ {C ^ {\prime}} E [ \Delta (P, C) | C ^ {\prime} ] P _ {r} \left(C ^ {\prime}\right) \\ \leq \sum_ {C ^ {\prime}} P _ {r} \left(C ^ {\prime}\right) \left(\frac {\epsilon \Delta (P , C ^ {\prime})}{2 4 \log k} + 5 0 \left(1 + \frac {1}{t}\right) O p t\right) \\ \leq \frac {\epsilon E [ \Delta (P , C ^ {\prime}) ]}{2 4 \log k} + 5 0 \left(1 + \frac {1}{t}\right) O p t. \\ \end{array}
$$
Since the  $k$ -means++ algorithm provides an approximation ratio of  $8(\log k + 2)$  in expectation by Theorem 1, we have  $E[\Delta(P, C)] \leq (50(1 + \frac{1}{t}) + \epsilon) Opt$ .
Running Time Analysis. By Theorem 2, in order to obtain a  $(50(1 + \frac{1}{t}) + \epsilon)$ -approximate solution, the iteration rounds for Algorithm 2 should be  $O(k^{t + 1}\log (\epsilon^{-1}\log k))$ . In each iteration, it takes  $O(ndk)$  time to update the distances between data points to their closest centers. During the sampling