Improved Coresets for Euclidean $k$-Means

Abstract: Given a set of $n$ points in $d$ dimensions, the Euclidean $k$-means problem consists of finding $k$ centers such that the sum of squared distances from every point to its closest center is minimized. The arguably most popular way of dealing with this problem in the big data setting is to first compress the data by computing a weighted subset known as a coreset and then run any algorithm on this subset. The guarantee of the coreset is that for any candidate solution, the ratio between coreset cost and the cost of the original instance is less than a $(1\pm \varepsilon)$ factor. The current state of the art  coreset size for Euclidean $k$-means is $\tilde O(\min(k^{2} \cdot \varepsilon^{-2},k\cdot \varepsilon^{-4}))$. This matches the lower bound of $\Omega(k \varepsilon^{-2})$ up to a $\min(k,\varepsilon^{-2})$ factor. In this paper, we improve this bound to $\tilde O(\min(k^{1.5} \cdot \varepsilon^{-2},k\cdot \varepsilon^{-4}))$. In the regime where $k \leq \varepsilon^{-2}$, this is a strict improvement over the state of the art. In particular, ours is the first provable bound that breaks through the $k^2$ barrier while retaining an optimal dependency on $\varepsilon$. 

Reviews and author responses:

 There is still value in those plots. The only thing that is unimportant is the variance in those costs. But we can remove them, if you so insist. We have other plots for the theoretical hard instance in the supplementary material with which we could replace the cost plots. 

As mentioned before, varying $\varepsilon$ does little other than determining constants. Varying $k$ and $\varepsilon$ together is very difficult to interpret, it is not clear when varying both parameters which causes an effect, and it is not clear what additional information we could gain from them other than what our plots already show.

 So the cost plots could as well be completely removed, right? Then one could conduct experiments for varying $(\epsilon, k) \in \{ 0.25, 0.5, 0.75 \} \times \{ 10, 20, \dots, 50 \} $ for example.

 I thank the authors for the clarifications. In light of this, my score has been raised.

 Distortion of 1.05 means an $\varepsilon$ of 0.05.
The difference in distortion is clearly visible, even at this scale. The difference in cost is indeed negligible and hard to see, but in our view and for the purpose of this paper also unimportant. The cost only provides a qualitative information that indeed, computing coresets is different from optimizing on these data sets. If this were not the case, one should not evaluate the data set to begin with, as it becomes easy to compute a coreset, as long as the cost substantially drops with increasing $k$. 

It is not clear what you mean by conducting the experiments with respect to different $\varepsilon$. 
There are three ways to conduct these experiments: Fix $k$, fix $\varepsilon$, or vary both. Varying both at the same time would be possible, but it is very hard to tell which parameter made a difference. Fixing $k$ and varying $\varepsilon$ only allows us to empirically measure the constants, but does not give any further insights. For this reason, we settled for a fixed $\varepsilon$ with varying $k$. This evaluation can at least give a hint as to what the correct dependency on $k$ is, which is also the main topic of this paper. Note that we are working under the assumption that the correct dependency on $\varepsilon$ is $\varepsilon^{-2}$.

Please note that the experiments are only supposed to support our theoretical results. We did not aim to provide a general experimental evaluation of coresets since this question is out of scope of this paper and has been studied previously in other works for a number of algorithms, including those presented here.

 Dear authors,

the work is indeed a bit incremental since it is based on ideas from previous works -- and improves a part of these. The result is very nice though, as I have already stated in my review. However, compared to a submission that introduces completely new ideas and directions, this is a small weakness. Nevertheless, I think that your work is worth publishing!

To the experiments: First, the cost plots need a scaling. As we are dealing with very large values there, one can barely observe differences between the different algorithms. (Maybe try a logarithmic scale?) Second, the sample size does only depend on $k$ and it is not completely clear what a distortion of e.g. $1.05$ means. Is that an empirical $epsilon$ of $0.05$? It would be nice if the experiments would also be conducted with respect to different $\epsilon$ and then instead of a distortion an empirical $\epsilon$ is plotted.

 I thank the authors for the detailed response. I have gone through the paper once again with the new details and explanations in mind.
I will change my score as all my questions have been answered.

 The reason why the analysis cannot be extended to arbitrary powers of $z$ is fairly in-depth in the analysis. Given that we could not add all the details for questions that we can prove within the page limit, we seriously question why we should use space on questions that we could not prove.

Furthermore, it is not easy to explain why things go wrong without a lot of familiarity, or without many explanations taken on faith. There are two instances where the analysis will fall short. We will focus on the more easily understood one. Here, the short answer is that even in constant dimension, it is not known how to obtain a coreset of size $k\varepsilon^{-2}$ for arbitrary powers of $z$ (whereas Group Sampling achieves this for k-median and k-means). Where this appears in the analysis is not easy to see.

For a slightly more indepth answer, hopefully the following comments will help. 
Any analysis of the cost estimator that results in a linear dependence on k has a variance of $\varepsilon^{-z}$ (both for Group Sampling and Sensitivity Sampling where this bound is even higher). This bound also appears in Lemma 3 where it is one part of the minimum over which we take trade off the values $k_i$ and $2^i$. Naively, this leads to a dependency of at least $k \cdot d\cdot \varepsilon^{-2} \cdot 2^i$, where $d$ is the dimension. $2^i$ can be bounded by $\varepsilon^{-z}$ for powers of z, using additional arguments. Thus, at this point, we have to find a way to remove $d$ and $\varepsilon^{-z}$ from this bound.

We remove $d$ via the chaining analysis. For the initial anchoring solution in the chaining argument (i.e. the first step in the chaining analysis), we require a coreset of size $k \cdot \varepsilon^{-2} \cdot d'$, where $d'$ is the dimension of the space containing the anchoring solutions. In particular $d'$ is not the same thing as the initial dimension of $d$. In this paper, we show that $d'$ can be chosen to be $\sqrt{k}$. Unfortunately, this still leaves a dependency of $k \cdot \sqrt{k} \cdot \varepsilon^{-2} \cdot 2^i \leq k^{1.5}\cdot \varepsilon^{-2} \cdot \varepsilon^{-z}.$ The dependency on $\varepsilon^{-z}$ can be reduced to $\max(\varepsilon^{-z+2},1)$, yielding a coreset of size $k^{1.5} \varepsilon^{-z}$. How this final step can be achieved technically is a fairly involved algebraic manipulation. Details appear on pages 20-22 in the supplementary pdf.

Since it is already known that Group Sampling yields coresets of size $\tilde{O}(k \varepsilon^{-z})$, the bound above yields no improvement. If the reviewer would like further details, we will provide them. 

 As for the supposed poor experimental performance of Group Sampling vs Sensitivity Sampling: We were open about the fact that we believe that a good analysis of Sensitivity Sampling will also yield an optimal coreset or at least a bound that matches Group Sampling. In fact, given that Sensitivity Sampling is arguably easier, we and presumably other coreset researchers would welcome this. Unfortunately this is not known, despite the algorithm being older and arguably better studied. Moreover, the main point of the experiments was not to compare the two algorithms to each other, to analyse a data set, or to evaluate secondary characteristics of these algorithms, but *only* to study whether the coreset size should depend linearly on k or not. In our view, it is not conductive towards obtaining optimal coreset bounds if we refuse to study Group Sampling simply because Sensitivity Sampling is better empirically, especially given that Group Sampling has the better theoretical bounds. For both algorithms, the experiments showed that a linear dependecy on k seems to be the right answer.

It is known that Group Sampling can be implemented efficiently, see for example the recent study https://arxiv.org/abs/2207.00966, though of course Sensitivity Sampling will always be slightly faster. Beyond that, running such experiments here does not add towards the important open question of obtaining optimal sized coresets. Running times, as long as they are nearly linear time (which Group Sampling supports) are not important for this paper.

The question of average distortion does not make sense. Coresets are a worst case guarantee. If even a single solution has high distortion, then that is the distortion of the coresets. Aside from this, obtaining average distortion bounds is an ill-posed problem in both theory and practise. Without any measure over the candidate solutions, defining the average distortion is impossible in theory. Certainly, a uniform measure over all solutions is meaningless as any candidate solution has density 0. In experiments, one could easily generate a large number of candidate solutions where the distortion is guaranteed to be small and articificially (and meaninglessly) decrease the average distortion. For example if the centers are sufficiently far away from a good constant factor approximation, the coreset will always have low distortion.

 We wished to give strong evidence that the Group Sampling algorithm will lead to an optimal coreset bound, at least asymptotically up to $polylog(k/\varepsilon)$ factors. Finding an optimal coreset algorithm is inarguably the most important open question in coreset research. Group Sampling is the best candidate algorithm, giving that it currently has the best theoretical bounds, and the only contribution of a paper making progress on the optimality of Group Sampling is the analysis. The reviewer may consider the improvement in the bound as not being a result that is strong enough. Beating the $k^2$ bound was not mentioned by the reviewer once and it is not clear if the reviewer understands the significance of it. If so, we might have made that point clearer. But all the other reasonings (Group Sampling not being a new algorithm, giving more explainations how the preprocessing in Group Sampling works, explaining how Group Sampling compares to other sampling distributions) are fallacious. It is not possible to give a reader unfamiliar with coresets an introduction. For an introduction, there exist dedicated surveys which we will happily add to the paper. 

Details missing:
For the specific complaint of group Sampling not being described: Given that Group Sampling was also used in two previous works, we decided to focus on the more novel aspects. While it might be preferable that all aspects of Group Sampling were given in this paper, the original paper by CSS'21 was over 50 pages, with roughly 20 pages describing how to canonize coreset constructions. Reviewing these assumptions in the main body is not possible if we are also to include our novel contributions, due to the page limit. 

Assumption 3 does not introduce any issues with bit complexity. Enforcing this assumption merely takes place in the analysis. In other words, an analysis for unweighted points implies an analysis with weighted points. The reference showing this was given in the supplementary material. 

$\ell_z$ clustering: see comment below.

Comparing the estimator to previous sampling distributions is not reasonable, especially given that none of the distributions are new, as the reviewer states. Experts can assess the relationship between these distributions. In no paper has such a comparison ever taken place. Reviewing all sampling distributions would qualify as a detailed technical survey paper itself. We cannot do this here, especially given the page limit. If the reviewer has specific questions, we will answer them.

Minor Comments:
The $p$ after Equation 2 should be in $C_i$.
$X$ in Lemma 5 should be the input point set $P$.

 1. $p$ is indeed in $C_j$
2. To expand on the proof of Lemma 5: If we set $t> 1/\varepsilon^2$ then the inequality in the third to last line becomes at least 1. Since the inequality is also always at most 1, as it is the absolute value of the inner product between two unit vectors, it means that it is equal to 1.
By construction, all of the vectors $(I-\Pi_{U_{i}}\Pi_{U_{i}}^T)p$ lie in the span of $U$ and moreover, these vectors are are pairwise orthogonal. Thus, $s$ can be written as linear combination of these vectors and the projection of $s$ onto the kernel of the space spanned by $U$, which is $(I-\Pi_U \Pi  _U^T)s$ must be the origin.
3. $\Pi_U$ is the orthogonal matrix whose rows span all the vectors contained in the subspace spanned by $U$. $\Pi_Us$ is the projection of $s$ onto that subspace.


 We respectfully disagree that the improvement is incremental. This is the first result that improves over the quadratic dependency on k while retaining an optimal dependency in k. No previous analysis in the entire coreset literature managed to do this. Aside from this, it also gives strong evidence that a future analysis of Group Sampling will show optimality (up to $polylog(k/\varepsilon)$ factors).

In terms of the analysis, it is known that the bound on the variance of the coreset estimator is tight. Therefore, the only challenge towards obtaining an optimal bound for coresets will be an improvement for the nets. The fact we improve the analysis of an existing estimator / coreset algorithm should not be taken as a lack of novelty, in our opinion, as if the Group Sampling algorithm is optimal, it is unclear why research should focus on a different algorithm. 

If there are specific questions regarding notation and technical details, we will be happy to answer them.

We would also appreciate details regarding the comment that the experiments are not meaningful.

 The analysis also works for Euclidean k-median (with some modifications), and we gave details how the analysis changes in the supplementary material. The bound on the coreset size still remains $O(k^{1.5}\varepsilon^{-2})$. As far as we are aware, the k-mediod objective is simply k-median with the centers being drawn from input points. As such a k-median coreset also implies a k-mediod coreset.

The bound $k^{1.5}$ is the best one can hope for using Lemma 3. The analysis does not really use the geometric mean, unless you are considering an interpolation between $k^{1.5}\varepsilon^{-2}$ and $k\varepsilon^{-4}$. If we have $k_i = 2^i = \sqrt{k} \ll \varepsilon^{-2}$ then the two tradeoff bounds $k \varepsilon^{-2} \left(\frac{k\cdot k_i\cdot 2^i}{(k+k_i\cdot 2^i)^2}\right)\cdot \min(k_i,2^i)$ agree at $k^{1.5} \cdot \varepsilon^{-2}$.

For the experiments: We can do our best to fit in more such details in the main body for the final version.


 
The paper considers the problem of providing a bound on the size of the coresets for the K-means problem which is closer the lower bound than the existing state of the art approaches. Some experimental results are also provided to support the analysis in the paper.
 The paper is highly compressed to fit into the confines of the conference submission. Though the writing is clear, it heavily relies on reading up on the results from Cohen-Addad et al's recent papers. 

The setting is of prime interest given that K-means shows up in various machine learning settings. The result of improving the min(k, 1/eps^2) to the \sqrt(k)
is pretty significant because now it is clearer that there will be a potential algorithm matching the lower bound of O(k/eps^2).

I checked over the main theorems and approach but did not work through all the details.

The paper is solving an important problem in clustering but the reduction from k to \sqrt(k), while theoretically very interesting, is a bit too narrow. 
 (A)Does this analysis follow through for other clustering problems such K-median, K-mediods etc? 

(B) It seems like you are able to get a better clustering nets because you can bound the error by the geometric mean (smaller) than the arthimetic mean. Is this the tightest one can get? 

(C) The experimental section is showing the cost of the various algorithms as well as the distortion results. However, it would be interesting to show the scaling of the size of the coresets. These are provided in the supplementary and would be better served if shown in the main paper? 
 The paper would be a nice contribution to the clustering literature but I wonder if it would be better suited to a primarily theoretical audience like COLT or similar. I provided an initial lower rating to reflect that and will adjust it based on the upcoming discussions. 


 The paper studies coresets for k-means clustering in d-dimensional Euclidean space. The paper is of theoretical nature and shows an improved upper bound of $O^\ast(\epsilon^{-2}k^{1.5})$ on the size of coresets that can be obtained by group sampling. This improves the current best bound of $O^\ast(\epsilon^{-2}k^2 \min\{\epsilon^{-2},k^2\})$.  The paper studies an important and interesting problem, which is under active research. The obtained upper bound is a further step towards closing the gap to the current lower bound.

The work is somewhat incremental since it heavily uses results from previous work and basically achieves the result by improving a part of it -- the cost vector net.

On the other hand, the obtained bound is nice and is certainly worth to be published.

Another weakness is that the notation and the technical details are not always clean and convincing. Also the experiments are not that meaningful. My suggestion is to improve the technical parts. Also the plots of the experiments may be improved (one can barely see the differences in cost.) No potential negative societal impact in sight.

 The paper presents a theoretical analysis leading to smaller coreset sizes for the problem of $k$-means. The coreset can then be extended also to the $k$-median problem.  

The authors provide an improvement over the work of Cohen-Addad et al. [2022], by finding nets (discretizes set of solutions) with better properties, to obtain their coreset size. This in term leads to the smallest coreset size for $k$-means with the emphasis that the coreset size should not be linearly in $k$.  * Strengths:
    1) A smaller coreset size for the k-means problem, leading to a subquadratic coreset size in $k$ (the number of means).
    2)  While the paper is based on a paper by Cohen-Addad et al. [2022], the authors present techniques that hinge upon finding better nets for obtaining smaller coreset sizes for both the $k$-means and $k$-median problems. In other words, the authors present an improvement on the techniques of Cohen-Addad et al. [2022], by generalizing the techniques to rather finding nets with better properties.

* Weaknesses:
The paper is somewhat hard to follow, and also contains many grammatical errors. 1) What is $p$ at the inequality below Eq. 2? I assume $p \in C_i$, right? 
2) Why does it imply that after $t = \frac{1}{\varepsilon^{-2}}$ rounds, $(I - \Pi_U\Pi_U^T)s = \overset{\to}{0}$? please clarify this point.
3) From what I understood, $Pi_U$ is the span of all previously added vectors from Lemma 5. So basically, this is a set, right? If so, what does it mean to have $\left\| Pi_Us - s^\prime \right\|$? Please do clarify. The authors clearly addressed the limitation of their work.

 The authors address the problem of coreset construction for the known k-means problem. They propose a coreset construction scheme and prove that it achieves coresets of size smaller than the state of the art, in some regimes.
To carry out the analysis, the authors propose a novel method that avoids terminal embeddings, which are common in the analysis of prior works. The authors claim this novel technical contribution may be of independent interest as well.
Empirical evaluations are then carried out to test the impact of the proposed approach against known techniques.
 Strengths:
- The proposed scheme indeed achieves state of the art coreset sizes for one of the cornerstone problems in the field.
- The analysis are novel and very interesting.
- The paper is relatively well written.

Weaknesses:
- Related work section is rushed through and lacks details. It does not contribute to the reader's knowledge, and should be better written in my opinion.
- Missing important discussions. Examples:
  * The validity of Assumptions 1-4 is placed in the appendix. At least a brief explanation should be placed in the main text. 
  * A discussion as to why this result cannot be extended to l_z clustering can be very helpful and enriching. 
  * It seems that the proposed algorithm, after ensuring the validity of the assumptions, samples each point according to some simple quantity which resembles the sensitivity distribution known from many prior works. A comparison or discussion is needed here.
- As of Assumption 3 regarding the input weights: to ensure integer weights (as stated in the appendix), some scaling and rounding is required. Doesnt this produce a dependency on the smallest weight, or, alternatively, on the model's precision? If so, this is an important note.
- The experimental results are not sufficient: 
  * Important details are missing.
  * The proposed method is not shown to achieve better results than the competing methods. 
  * Running time are not reported, which is crucial, as it seems that the proposed method would (theoretically) achieve much larger computational times. This is due to the computations required to enforce Assumptions 1-4 (which include a computation of an alternative coreset, e.g., the sensitivity coreset used as a competing method).
- At the end of the day, the proposed algorithm is not novel as it simply computes a (previously known) sampling probability for every input point, which is based on some (previously known) approximation for the problem, then it samples points from this distribution and does some reweighting.  Also, as the authors claim, enforcing Assumptions 1-4 is also not novel. The only novel part seems to be the analysis. Hence, in my opinion, the paper's main contribution is not sufficient.

Minor comments:
- The first paragraph on Page 2 is important and should be somehow better emphasized.
- Line 50: "may be reduced to log(k/\eps)\eps^-2 using dimension reduction techniques". some citation should be added at the end of this claim. 
- Line 50: "to technique" (typo).
- Equation at Line 50: missing closing bracket.
- Typos at Line 63.
- The equation after Eq. (2) utilizes the point on the left and on the right hand sides, but p does not seem to be defined.
- Lemma 5: where is X defined?
- Line 210: the maximum distortion is reported in the experiments. Do you have insights on the average distortion? The average distortion might be a more accurate indicator. See few question above. Yes

Meta-review: I actually really liked the paper despite the weak accept reviews - and decided to bump this up. Improving the known coreset bound for k-means is extremely important, as that is a fundamental problem. While it does indeed heavily build upon prior work and doesn't get optimal bounds, it is giving a novel analysis for a very important problem (with a huge line of prior work with worse bounds), and thus I recommend acceptance. 

Decision: Accept