Coresets for Kernel Clustering

Abstract: We devise the first coreset for kernel $k$-Means, and use it to obtain new, more efficient, algorithms. Kernel $k$-Means has superior clustering capability compared to classical $k$-Means particularly when clusters are separable non-linearly, but it also introduces significant computational challenges. We address this computational issue by constructing a coreset, which is a reduced dataset that accurately preserves the clustering costs.

Our main result is the first coreset for kernel $k$-Means, whose size is independent of the number of input points $n$, and moreover is constructed in time near-linear in $n$. This result immediately implies new algorithms for kernel $k$-Means, such as a $(1+\epsilon)$-approximation in time near-linear in $n$, and a streaming algorithm using space and update time $\mathrm{poly}(k \epsilon^{-1} \log n)$. 

We validate our coreset on various datasets with different kernels. Our coreset performs consistently well, achieving small errors while using very few points. We show that our coresets can speed up kernel $k$-Means++ (the kernelized version of the widely used $k$-Means++ algorithm), and we further use this faster kernel $k$-Means++ for spectral clustering. In both applications, we achieve up to 1000x speedup while the error is comparable to baselines that do not use coresets.

Reviews and author responses:

 Thanks for addressing my concerns. As for your question, "Turning Big data into tiny data: Constant-size coresets for $k$-means, PCA and projective clustering" can be used when the feature map is explicitly given which then a coreset can be obtained. However, the coreset size would be exponential in $k$, i.e., $O\left( k^{O\left( \frac{k}{\varepsilon}\right)}\right)$. For more details, see "k-Means for Streaming and Distributed Big Sparse Data" which discusses this point and improves the previous upper bound on the size of the coreset to $O\left( k^{\frac{1}{\varepsilon^2}}\right)$.

 We thank all reviewers for the valuable comments!

Apart from responding to your specific concerns in the following threads, we also revised our draft according to your suggestions, where the changes are highlighted in red.

 > 1)Less Novelty: The algorithm for construction of coresets itself is not novel. Existing coreset frameworks for classical k-means and (k,z) clusterings are extended to the kernelized setting.

Please see our response to BBxb (https://openreview.net/forum?id=1nlRIagHDUB&noteId=JmcU80pwzdw). In short, our contribution lies more on the conceptual rather than technical side. Conceptually, we significantly extend the scope of “coreset technology” from Euclidean spaces to non-Euclidean spaces, including even graph problems (we demonstrate it for spectral clustering, but another direction could be graph kernels). Technically, we are fortunate that several subtle details fit together smoothly (which is clear in hindsight). In the end, simplicity is a huge advantage. 

> 2) Clarity: Since the coreset construction algorithm is built up on previous works, a reader without the background in literature on coresets would find it hard to understand why the particular sampling probabilities are chosen and why they give particular guarantees. It would be useful rewrite the algorithm preview and to give at least a bit of intuition on how the importance sampling scores are chosen and how they can give the coreset guarantees

We appreciate this suggestion, and to improve clarity we added more intuitions and pointers to concrete steps of the algorithms.

> In the experiment section, other than uniform sampling, it would be interesting to use some other classical k-means coreset as baselines for comparison.

The main coreset algorithm that we evaluate in the experiment is already very close to a “classical” coreset for k-means.

Indeed, our Hilbert -> Euclidean dimension reduction step is only used in the analysis, and the coreset construction algorithm itself is the same as the non-kernelized version, except that distances are evaluated using the kernel trick. Furthermore, in the experiments, we omit the iterative size reduction of [Braverman et al., SODA 21], so the actual algorithm that we implement is essentially a classical importance sampling algorithm as in [Feldman and Langberg, STOC 2011] and [Feldman, Schmidt, and Sohler, SICOMP 2020]. The only difference is that we compute the importance score using an algorithm proposed in [Varadarajan and Xiao, FSTTCS 12], which is easy to implement and fast to run. However, this difference is minor, and we expect the performance impact to be minimal.

> Please highlight the technical challenges and contributions clearly when compared to coresets for classical k-means.

Please see our response to BBxb (https://openreview.net/forum?id=1nlRIagHDUB&noteId=JmcU80pwzdw), particularly items 1-4 in the response of the first comment.


 > The paper claims the first coreset for the problem of kernelized k-means problem. This claim is not true in my opinion due to the following. Even though the problem was not the main contribution of "A PTAS for k-Means Clustering Based on Weak Coresets" by Feldman et al., it was still mentioned in this paper. Another coreset paper mentions a coreset for kernelized k-means (for some type of kernels) where the coreset's size is exponential in k (see "Turning Big data into tiny data: Constant-size coresets for -means, PCA and projective clustering" by Feldman et al.). However, the best feature associated with your paper is the generation of coresets of smaller size with respect to any kernel function. A more proper claim would be the first coreset for -kernelized clustering for any kernel function.

Thanks for pointing this out, and we corrected our text as you suggested. We indeed missed the reference “A PTAS for k-Means Clustering Based on Weak Coresets" by Feldman et al. [FMS07] (where the kernel clustering application was only discussed in one paragraph), and we added this to our revision.

However, you also mention that "Turning Big data into tiny data: Constant-size coresets for k-means, PCA and projective clustering" by Feldman et al. [FSS20] discusses kernel clustering, but we could not find the keyword “kernel” in that paper. Could you please provide the exact place (and version) where a kernel is discussed?

> Throughout the experiments, there was no need for the feature map itself as it is not commonly explicitly defined. However, Algorithm 3 requires its explicit existence which forms a bit of a problem. Can you please justify this? 

We do not need this map explicitly. It is conventional to describe the algorithm in terms of the feature map \phi, however the algorithm only needs the kernelized distance between data points (i.e., distances in the feature space), which can be implemented efficiently using the standard “kernel trick”. We added this explanation in the revision.


 > The theoretical contributions here seem very slight, more like observations, and I don't see the paper having the type of underlying innovation I would expect from ICLR

We believe theoretical contributions should be measured from multiple aspects, particularly technical depth, and conceptual advance. This is in addition of course to the theory having strong connections to the application domain, which is crucial for venues like ICLR. 

Our contribution lies more on the conceptual side. We obtain the first coreset whose size is independent of the input size for kernel (k, z)-clustering under general kernels. In fact, this entire direction of using coresets for kernelized optimization is almost completely new (we say almost because of the slight exception of [FMS07], see discussion with Reviewer w9ag). Technically, an extension from “Euclidean spaces” to “spaces that embed into Euclidean” seems natural and straightforward. But for potential domain applications, it could mean a significant extension of the scope of “coreset technology”, from Euclidean spaces to non-Euclidean spaces, including for example graph problems (our paper demonstrates it for spectral clustering, and another direction could be graph kernels). Coresets have already proved to be a very powerful and practical tool, and thus our significant extension of the scope can be very useful, and presumably relevant to the broad audience of ICLR.

Technically, our result establishes a connection that may seem straightforward to experts. But apparently, it was not known, used or tested before (by theorists or practitioners), and is thus sorely missing in the literature. In fact, the simplicity of our result hinges on a few subtle points: 
1. Our discovery is timely because *dimension-independent* coresets were found only recently. 

2. Fortunately, we can rely on coresets that are constructed in *near-linear time* in the kernel setting (via the kernel trick). In contrast, one recent algorithm [Sohler and Woodruff, FOCS’18] is slow and uses the explicit representation in the feature space.

3. Our embedding from (infinite-dimensional) Hilbert space to finite-dimensional Euclidean eliminates any potential dependence on the ambient space (e.g., which points can serve as centers). The proof is easy but it does require a formal treatment, e.g., careful definitions. 

4. In fact, we crucially apply this embedding from Hilbert to Euclidean space (which could be computationally heavy) carefully so that it is used only in the analysis, avoiding any blowup in the running time.

In hindsight, all these technical details fit together very smoothly, but this might be less obvious apriori, even to experts. And in the end, simplicity is a huge advantage. 

> Also, I wasn't so impressed by an error of 10% using a coreset of 1000 points (although this application is somewhat outside my area of expertise).

We agree that 10% error is unsatisfactory for many applications, but we note that this is the worst-case error, since the evaluation is for a *strong* coreset, whose error is computed as the *maximum* over all possible center sets. Hence, this number need not show up in downstream applications; instead, the approximation ratio/error of the kernel k-means solution is usually more important.

Indeed, as can be observed in Figure 2 and Figure 3, our coreset-based algorithms maintain nearly the same accuracy, which is close to 1% instead of the worst case 10% using 1000 points, while achieving very significant speedup, up to 1000x, over k-means++ (for kernel k-means) and an implementation available in the well-known scikit-learn library (for spectral clustering).

The purpose of mentioning these numbers (“10% error”, “1000 points”) is simply to give an example of how to interpret our size-accuracy curve. Since our theory asserts that the coreset size should be independent of kernel type and dataset size, people should expect a similar curve in their own applications. 




The paper suggests a small coreset for the problem of clustering under kernel distances. Coresets allow geometric problems to be reduced to small size, and then an expensive algorithm run on the coreset instead of the full input. Here the authors consider the problem of k-means where the distance function is given as a kernel. They show that coresets exist for this formulation, and so a brute-force algorithm can be used to solve this problem.

The technical framework is as follows: The authors present a known embedding from Hilbert to Euclidean space, and then the coreset of Braverman et al. satisfies the required approximation bounds. 

The theoretical contributions here seem very slight, more like observations, and I don't see the paper having the type of underlying innovation I would expect from ICLR. Also, I wasn't so impressed by an error of 10% using a coreset of 1000 points (although this application is somewhat outside my area of expertise).

typo on page 6: "importane-sampling" Nice result, but in my opinion not innovative enough for ICLR.

The paper suggests a coreset construction technique to the kernelized $(k,z)$-clustering, whose size is independent of $n$ (the number of points in the input data), in time that is near-linear in $n$. The paper provides a coreset construction using the $D^z$ sampling technique to bound the probability of sampling any input point $p \in X$. The technique can be thought of as a generalization to the well-known technique $D^2$ sampling technique of David Arthur et al. due to the incorporation of the feature map.
While the paper is theoretical in nature and admits strong plausible results, I have some concerns:
1. The paper claims the first coreset for the problem of kernelized k-means problem. This claim is not true in my opinion due to the following. Even though the problem was not the main contribution of "A PTAS for k-Means Clustering Based on Weak Coresets" by Feldman et al., it was still mentioned in this paper. Another coreset paper mentions a coreset for kernelized k-means (for some type of kernels) where the coreset's size is exponential in k (see "Turning Big data into tiny data:
Constant-size coresets for $k$-means, PCA and projective clustering" by Feldman et al.). 
However, the best feature associated with your paper is the generation of coresets of smaller size with respect to any kernel function. A more proper claim would be the first coreset for $(k,z)$-kernelized clustering for any kernel function. 
2. Throughout the experiments, there was no need for the feature map itself as it is not commonly explicitly defined. However, Algorithm 3 requires its explicit existence which forms a bit of a problem. Can you please justify this? As far as I understood, Algorithm 3 uses the explicit feature map. How did you obtain this map? 
Practically speaking, I don't see why you need the explicit map, since the distance function can be slightly altered to apply the kernel trick also on points from $C^*$ (using $K$). This is due to the fact that $C^*$ is generated using points from $X$. Correct me if I am wrong. The paper is novel, well written, and admits theoretically and practically great results. I recommend acceptence. 

The paper claims to present the first coreset for kernel k-means and other (k,z) kernel clusterings given oracle access to the kernel. The size of coreset is independent of the number of points and construction time is near linear in k. The authors use the coreset to get a (1+\epsilon) approximation for the kernel k-means. The theory is supported with experimental results. Strengths:
1)Kernel k-means is an important problem in machine learning and since it is computationally expensive, the use of coresets to speed up the algorithm is an interesting and important problem.
2) Theoretically the paper uses a couple of nice small results (lemma 3.2 and corollary 3.3) to apply the coreset frame work in the kernelized setting. Overall the paper is not difficult to read.
3) Experimental results over large real datasets show significant speed ups for the kernelized k-means problem while preserving the accuracy. Code is also provided.

Weaknesses:
1)Less Novelty:  The algorithm for construction of coresets itself is not novel. Existing coreset frameworks for classical k-means and (k,z) clusterings  are  extended to the kernelized setting. 
2)Clarity:  Since the coreset construction algorithm is built up on previous works, a reader without the background in literature on coresets would find it hard to understand why the particular  sampling probabilities are chosen and why they give particular guarantees. It would be useful rewrite the algorithm preview and to give at least a bit of intuition on how the importance sampling scores are chosen and how they can give the coreset guarantees

Suggestions:
1) In the experiment section, other than uniform sampling, it would be interesting to use some other classical k-means coreset as baselines for comparison. 
2) Please highlight the technical challenges and contributions clearly when compared to coresets for classical k-means. Overall the paper discusses an important and interesting problem and the experiments performed on real datasets show the utitility of the method in practice. However the theory results are heavily based on existing coreset techniques for classical k-means and  I am not very sure if the paper has enough technical novelty for ICLR. 

Meta-review: Overall it was decided to reject this paper mainly because it seemed to be a minor extension of known constructions. The reviewers agreed that it certainly is valuable that the paper presents the best known results for kernel k-means, but the paper was viewed by the reviewers as more of an observation and primarily an off-the-shelf application of techniques in the coreset literature. Because of this, the novelty was thought to be a bit below the bar. 

One suggestion for improving the presentation is that in the thesis of Melanie Schmidt, there seems to be such a construction for kernel k-means which is exponential in 1/eps, and so much worse than what is in this submission. While that's great and certainly something to add and discuss in the paper, the reviewers still felt the technical novelty here was not quite enough to merit acceptance.

Decision: Reject