SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Case (ii). Here we give up on estimating the mean of  $N_{1}$  and set  $\alpha_0 \gets 0$ . The point  $x \in N_{1}$  such that  $\mathcal{T}[x] \notin \mathcal{Q}$  can be assigned to  $\mathcal{T}[x]$  incurring a combined cost of  $\varepsilon^2\mathrm{OPT} / k$ . We partition the remaining points in  $N_{1}$  into  $Z_{1} \cup \ldots \cup Z_{s}$  where each point  $x$  is placed in  $Z_{i}$  if  $\mathcal{T}[x] = q_{i}$ . Now, we collapse the points in  $Z_{i}$  so as to coincide with  $q_{i}$  and show that this does not worsen our approximation factor. In terms of coefficients  $(\alpha_{i})_{i}$ , this translates into the updates  $\alpha_{i} \gets \alpha_{i} + |Z_{i}| / |O_{i}^{*}|$  for each  $i = 1\ldots s$ .
Indeed, using Observation 22 we can move all points in  $Z_{i}$  to  $q_{i}$  incurring an additive combined cost of  $\varepsilon \mathrm{OPT} / k$  and a multiplicative cost of  $1 + O(\varepsilon)$ .
Points in  $F_{1}$ . Points in  $F_{1}$  are very far from  $q_{1}$  and thus far from  $o_{1}$ , hence even if their contribution to  $\mathrm{cost}(O_1^*,o_1)$  might be large, we have  $\mathrm{cost}(F_1,o_1) = (1\pm O(\varepsilon))\mathrm{cost}(F_1,o')$  for all  $o^{\prime}$  in a ball of radius  $\rho_{1} / \varepsilon$  centered in  $o_{1}$ , thanks to Lemma 23.
Let  $H$  be the set of points that have not been assigned to centers in  $\mathcal{C} \backslash Out$ . In particular,  $H = W \cup N_1$  if points in  $N_1$  satisfy case (i) and  $H = W \cup Z_1 \ldots Z_s$  if points in  $N_1$  satisfy case (ii). We consider two cases.
If  $\| \mu (H) - q_1\| \leq \rho /\varepsilon$ , then  $\| \mu (H) - o_1\| \leq \rho (1 + \varepsilon +1 / \varepsilon)$  because  $q_{1}\in \mathrm{core}(o_{1})$ . Since for each  $f\in F_{1}$  we have  $\| f - o_1\| \geq \| f - q_1\| -(1 + \varepsilon)\rho \geq \Omega (\rho /\varepsilon^3)$  then  $\mathrm{cost}(f,o^{\prime}) = (1\pm O(\varepsilon))\mathrm{cost}(f,o_1)$  for each  $o^{\prime}$  in a ball of radius  $O(\rho /\varepsilon)$  centered in  $o_1$ , and so in particular for  $o^{\prime} = \mu (H)$ . Thus in this case we can simply disregard all points in  $F_{1}$  and computing  $\hat{o}_1$  according to the  $(\alpha_i)_i$  defined above yields a perfect approximation of  $o_i$ .
Else, if  $||\mu(H) - q_1|| > \rho / \varepsilon$ , a similar argument applies to show that  $\mathrm{cost}(H, o') = (1 \pm \varepsilon)\mathrm{cost}(H, o)$  for each  $o'$  in ball of radius  $O(\rho)$  centered in  $o_1$ . Indeed, we can rewrite  $\mathrm{cost}(H, o')$  as  $|H| \cdot \mathrm{cost}(\mu(H), o') + \mathrm{cost}(\mu(H), H)$ . If  $||\mu(H) - q_1|| < \rho / \varepsilon$  the first term varies of at most a factor  $(1 + \varepsilon)$  and the second term is constant. Thus in this case  $\hat{o}_1 = q_1$  is a perfect approximation of  $o_1$  and we simply set  $\alpha_1 = 1$  and  $\alpha_j = 0$  for  $j \neq 1$ . In other words, here  $\mu(N_1 \cup H)$  is too far from  $q_1$  (and thus  $o_1$ ) to significantly influence the position of  $\hat{o}_1$  and the same holds for any point in  $F_1$ . This works, of course, because we assumed  $q_1 \in \mathrm{core}(o_1)$ .
Discussing the limitations on the coefficients values. The proof above would work smoothly if we were allowed to set  $\alpha_{i}$  to exactly the values discussed above, representing the fractions of points from  $O_{i}^{*}$  captured by different  $q_{i}$ s. However, to make the algorithm efficient we limit ourselves to values in  $\left\{1,(1 - \varepsilon)\dots (1 - \varepsilon)^{\lceil \log_{1 - \varepsilon}(\varepsilon^{7})\rceil}\right\} \cup \{0\}$ . Lemma 21 shows that as long as the values of  $(\alpha_{i})_{i}$  estimate the frequencies described above up to a factor  $1\pm O(\varepsilon)$  then the approximation error is within a multiplicative factor  $1\pm O(\varepsilon)$ .
We are left to take care of the case in which  $\alpha_{i}$  is set to a value  $< \varepsilon^{7}$ . We set  $\alpha_{i}$  when dealing with points in  $C_1\cup N_1$  and for each  $x\in C_1\cup N_1$  we have, for each  $o^{\prime}\in B(q_{1},(1 + \varepsilon)\rho)$ ,  $\mathrm{cost}(x,o^{\prime})\leq 2\mathrm{cost}(q_1,o^{\prime}) + 2\mathrm{cost}(x,q_1) = O(\rho_1\varepsilon^{-6})$ . Thus, if we simply set  $\alpha_{i}\gets 0$  whenever we have  $\alpha_{i} < \varepsilon^{7}$  then the combined cost of points in  $O_{1}^{*}$  with respect to  $o^{\prime}$  varies by  $\varepsilon^7 |O_1^*\big|\cdot \rho_1\varepsilon^{-6} = O(\varepsilon)\mathrm{OPT}_1$ . Effectively, ignoring these points does not significantly impact the cost. hence solving 1-mean ignoring these points finds a  $(1 + O(\varepsilon))$ -approximate solution to the original problem.
Additional Experimental Evaluation
In this section we report additional experiments which presentation did not fit in the main body. In particular, we run experiments on the dataset KDD-PHY and for  $k = 10,50$ .
In Figure 4 we compare MSLS-G with MSLS. To perform our experiment, we initialize  $k = 25$  centers using  $\mathrm{KM}++$  and then run 50 iterations of local search for both algorithms, for  $p \in \{2,3\}$  swaps. We repeat each experiment 5 times. For ease of comparison, we repeat the plot for the KDD-BIO and RNA datasets that we present in the main body of the paper. Due to the higher running of the MSLS we perform this experiments on  $1\%$  uniform sample of each of our datasets. We find out that the performance of the two algorithms is comparable on all our instances, while they both perform roughly  $15\%-27\%$  better than  $k$ -means++ at convergence.
In Figure 5 we run  $\mathrm{KM} + +$  followed by 50 iterations of MSLS-G with  $p = 1,4,7,10$  and  $k = 10,25,50$  (expcluding the degenerate case  $p = k = 10$ ) and plot the relative cost w.r.t.  $\mathrm{KM} + +$  at each iteration. The results for  $k = 25$  on KDD-BIO and RNA can be found in Figure 2. We repeat each experiment 5 times. Our experiment shows that, after 50 iterations MSLS-G for  $p = 4,7,10$  achieves improvements of roughly  $5 - 10\%$  compared to MSLS-G- $p = 1$  and of the order of  $20\% - 40\%$  compared to  $\mathrm{KM} + +$ . These improvements are more prominent for  $k = 25,50$ . We also report the time per iteration that each algorithm takes. For comparison, we report the running time of a single iteration of Lloyd's next to the dataset's name. Notice that the experiment on RNA for  $k = 50$  is performed on a  $10\%$  uniform sample of the original dataset, due to the high running time.
In Figure 6, we use  $\mathrm{KM}++$  and MSLS-G as a seeding algorithm for Lloyd's and measure how much of the performance improvement measured is retained after running Lloyd's. First, we initialize
Figure 4: Comparison between MSLS and MSLS-G, for  $p = 2$  (left column) and  $p = 3$  (right column), for  $k = 25$ , on the datasets KDD-BIO (first row), KDD-PHY (second row) and RNA (third row). The  $y$  axis shows the mean solution cost, over the 5 repetitions of the experiment, divided by the means solution cost of  $\mathrm{KM}++$ .
Figure 5: We compare the cost of MSLS-G, for  $p \in \{1, 4, 7, 10\}$ , divided by the mean cost of KM++ at each LS step, for  $k \in \{10, 25, 50\}$ , excluding the degenerate case  $p = k = 10$ . The legend reports also the running time of MSLS-G per LS step (in seconds). The experiments were run on all datasets: KDD-BIO, RNA and KDD-PHY, excluding the case of  $k = 25$  for KDD-BIO and RNA which are reported in the main body of the paper.
our centers using  $\mathrm{KM}++$  and the run 15 iterations of MSLS-G for  $p = 1,4,7$ . We measure the cost achieved by running 10 iterations of Lloyd's starting from the solutions found by MSLS-G as well as  $\mathrm{KM}++$ . We run experiments for  $k = 10,25,50$  and we repeat each experiment 5 times. We observe that for  $k = 25,50$  MSLS-G for  $p > 1$  performs at least as good as SSLS from Lattanzi and Sohler [2019] and in some cases maintains non-trivial improvements. These improvements are not noticeable for  $k = 10$ ; however, given how Lloyd's behave for  $k = 10$  we conjecture that  $k = 10$  might be an "unnatural" number of clusters for our datasets.
Figure 6: We compare the cost after each of the 10 iterations of Lloyd with seeding from MSLS-G, for  $p \in \{1,4,7,10\}$  and 15 local search steps and  $\mathrm{KM}++$ , for  $k \in \{10,25,50\}$ . We excluded the degenerate case  $p = k = 10$ , and the experiments reported in the main body of the paper.