SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset. We generated our dataset  $X$  consisting of 200,001 points on two-dimensional space so that 100,000 points were drawn from a spherical Gaussian with standard deviation 2.75 centered at  $(-10,10)$  and 100,000 points were drawn from a spherical Gaussian with standard deviation 2.75 centered at  $(10,-10)$ . The final point of  $X$  was drawn from a spherical Gaussian with standard deviation 2.75 centered at  $(100000,100000)$ . Thus by construction of our synthetic dataset for  $k = 3$ , the optimal centers should be close to  $(-10,10)$ ,  $(10,-10)$ , and  $(100000,100000)$ . We then create the data stream  $S$  by prepending two additional points drawn from spherical Gaussians with standard deviation 2.75 centered at  $(-100000,100000)$  and  $(-100000,-100000)$  respectively. We set the window length to be 200,001 in accordance with the "true" data set, so that the first two points of the stream of length 200,003 will be expired by the data stream.
Experimental setup. For each of the instances of Lloyd's algorithm, either on the entire dataset  $X$  or the sampled coreset  $C$ , we use 3 iterations using the k-means++ initialization. In this case, the offline Lloyd's algorithm requires storing the entire dataset  $X$  in memory and thus its input size is 200,001 points. By comparison, we normalize the space requirement of the sublinear-space algorithms by permitting each algorithm to store  $m \in \{3,4,5,6,7,8,9,10,11,12\}$  points. Note that since  $k = 3$ , it would not be reasonable for  $C$  to have fewer than 3 points. We then run Lloyd's algorithm on the coreset  $C$ , with 3 iterations using the k-means++ initialization.
By construction of our dataset, we generally expect the uniform sampling algorithm uni to be stable across the various values of  $m$  but perform somewhat poorly, as it will sample points from the large clusters but it will miss the point generated from the Gaussian centered at (100000, 100000). Since in our construction the stream  $S$  only contains two more points than the dataset  $X$ , the histogram-based algorithm hist will not delete any points. Thus, the resulting coreset  $C$  generated by hist is somewhat likely contain the points generated from the Gaussians centered at  $(-100000, 100000)$
and  $(-100000, -100000)$  and can perform poorly on the synthetic dataset in these cases. Finally, since we allow the last point of the stream to be the single point of  $X$  far from the two large clusters, then the importance sampling based algorithm imp will sample the last point with high probability once any points of  $C$  have been expired. Hence by the construction of our stream, we expect imp to perform well.
(b) Comparisons for varying  $m$ .
Fig. 3: Comparison of average clustering costs made by uniform sampling, histogram-based algorithm, and our coreset-based algorithm across various settings of space allocated to the algorithm, given a synthetic dataset. For comparison, we also include the offline k-means++ algorithm as a baseline, though it is inefficient because it stores the entire dataset. Ranges are not plotted because they would not be visible.
Experimental results. For each choice of  $m$  and  $k$ , we ran each algorithm 50 times and tracked the resulting clustering cost. As expected by our construction, our algorithm performed significantly better than the other sublinear-space algorithms. In fact, even though our algorithm was only permitted memory size  $m \in \{3,4,5,6,7,8,9,10,11,12\}$ , our algorithm was quite competitive with the offline Lloyd's algorithm, which used memory size 200,001, i.e., the entire dataset. For  $k \geq 3$ , uniform sampling performed relatively poorly but quite stably, because although it never managed to sample the point generated from the Gaussian centered at (100000, 100000), the two other Gaussian distributions were sufficiently close that any sampled point would serve as a relatively good center for points generated from the two distributions. Similarly, for fixed  $k = 3$  in Figure 3b, the importance sampling approach used by histogram-based algorithms performed the worse, by multiple orders of magnitude. We expect this is because we did not delete the points in  $S \setminus X$  from  $C$  and thus the resulting Lloyd's algorithm on  $C$  moved the centers far away from the centers of the Gaussian distributions that induced  $X$ . A more optimized fine-tuned histogram-based algorithm would have searched for parameters that govern when to delete points from  $S \setminus X$ , which have reduced the algorithm down to our main algorithm. We plot our results in Figure 3.