Coresets for Kernel Clustering

1 INTRODUCTION . We design the first coresets , and consequently new efficient algorithms , for kernel k-MEANS and related problems , like its generalization kernel ( k , z ) -CLUSTERING , under general kernels . The kMEANS problem has proved to be fundamental for unsupervised learning in numerous application domains . Vanilla k-MEANS fails to capture sophisticated cluster structures , e.g. , when the clusters are separable non-linearly , but this can be tackled by applying kernel methods ( Schölkopf et al. , 1998 ; Girolami , 2002 ) . This has led to kernel k-MEANS , where data points are first mapped to a high-dimensional feature space ( possibly implicitly via a kernel function ) , and then clustered in this richer space using a classical k-MEANS . Formally , a kernel for a dataset X is a function K : X ×X → R+ ( intended to measure similarity between elements in X ) that can be realized by inner products , i.e. , there exist a Hilbert space H and a map ϕ : X → H ( called feature space and feature map ) such that ∀x , y ∈ X , 〈ϕ ( x ) , ϕ ( y ) 〉 = K ( x , y ) . ( 1 ) In kernel k-MEANS , the input is a dataset X with weight function wX : X → R+ and a kernel function K : X × X → R+ as above , and the goal is to find a k-point center set C ⊆ H that minimizes the objective costϕ ( X , C ) = ∑ x∈X wX ( x ) ·min c∈C ‖ϕ ( x ) − c‖2 . ( 2 ) ( An equivalent formulation asks for a k-partitioning of X , keeping C implicit . ) This kernel version has superior clustering capability compared to classical k-MEANS ( Zhang & Rudnicky , 2002 ; Kim et al. , 2005 ) , and has proved useful in different application domains , such as pattern recognition ( Shawe-Taylor & Cristianini , 2004 ) , natural language processing ( Andrews & Fox , 2007 ) , biology ( Gönen & Margolin , 2014 ) and social networks ( van Laarhoven & Marchiori , 2016 ) . In fact , kernel k-MEANS is useful also for solving other clustering problems , such as normalized cut and spectral clustering ( Dhillon et al. , 2004 ; Ding et al. , 2005 ) . Computational challenges . As observed in previous work ( Girolami , 2002 ; Dhillon et al. , 2004 ) , the kernel trick can be applied to rewrite kernel k-MEANS using access only to the kernel K ( · , · ) and without computing the very high-dimensional map ϕ explicitly . However , this approach has outstanding computational challenges ( compared to classical k-MEANS ) , essentially because of the kernel trick . Consider the special case where k = 1 and the input is n unweighted points ( i.e. , 1-MEAN clustering ) . It is well known that the optimal center c ? has a closed form c ? : = 1n ∑ x∈X ϕ ( x ) . But the kernel trick requires Ω ( n2 ) accesses to K to evaluate costϕ ( X , c ? ) ,1 while in the classical setting such evaluation needs only O ( n ) distance computations . This Ω ( n2 ) barrier can be bypassed by allowing ( 1 + ) -approximation . In particular , let S be a uniform sample of poly ( −1 ) points from X , and let ĉ : = 1|S| ∑ x∈S ϕ ( x ) be its 1-MEAN ; then with high probability , costϕ ( X , ĉ ) ≤ ( 1 + ) costϕ ( X , c ? ) and evaluating costϕ ( X , ĉ ) takes only poly ( −1 ) n time . However , this uniform-sampling approach does not generally work for k ≥ 2 , because if the optimal clustering is highly imbalanced , a uniform sample is unlikely to include any point from a small cluster . Alternative approaches , such as dimension reduction , were also proposed to obtain efficient algorithms for kernel k-MEANS , but they too do not fully resolve the computational issue . We elaborate on these approaches in Section 1.2 . Our approach . To tackle this computational challenge , we adapt the notion of a coreset ( HarPeled & Mazumdar , 2004 ) to kernel k-MEANS . Informally , a coreset is a tiny reweighted subset of the original dataset on which the clustering cost is preserved within ( 1 ± ) factor for all candidate centers C ⊆ H. This notion has proved very successful for classical k-MEANS , e.g. , to design efficient near-linear algorithms . In our context of kernel k-MEANS , a coreset of size s for an input of size n = |X| has a huge advantage that its k optimal center points can all be represented as linear combinations of only s points in the feature space . Given these k optimal centers ( as linear combinations ) , evaluating the distance between a point ϕ ( x ) and such a center takes merely O ( s2 ) time , instead ofO ( n2 ) , and consequently the objective can be ( 1+ ) -approximated in timeO ( s2kn ) . Moreover , it suffices to use k centers ( again as linear combinations ) that are ( 1 + ) -approximately optimal for the coreset S. In addition , coresets are very useful in dealing with massive datasets , since an offline construction of coresets usually generalizes to the streaming setting ( Har-Peled & Mazumdar , 2004 ) , distributed computing ( Balcan et al. , 2013 ) and dynamic algorithms ( Henzinger & Kale , 2020 ) via the merge-and-reduce method ( Har-Peled & Mazumdar , 2004 ) , and existing ( offline ) algorithms can be efficiently applied to the coreset , instead of to the original dataset , with minor or no modifications . 1.1 OUR RESULTS . Our main result is the first coreset for kernel k-MEANS with a general kernel , that has size independent of the input size n = |X| ; moreover , our coreset can be constructed in near-linear time for small k. ( In fact , it generalizes to kernel ( k , z ) -CLUSTERING , see Section 2 for definitions . ) Formally , an -coreset for kernel k-MEANS with respect to weighted dataset X and kernel function K : X × X → R+ is a weighted subset S ⊆ X , such that for every feature space H and feature map ϕ that realize K , as defined in ( 1 ) , ∀C ⊆ H , |C| = k , costϕ ( S , C ) ∈ ( 1± ) · costϕ ( X , C ) . ( 3 ) Previously , only a weak coreset was known for kernel k-MEANS ( Feldman et al. , 2007 ) , meaning that the objective is preserved only for certain candidate centers ( whereas ( 3 ) guarantees this for all centers ) , and that coreset works only for certain kernels ( finite-dimensional ) . While we employ a similar approach , the technical differences make our bottom-line result much stronger . Throughout , we assume an oracle access to K takes unit time , and therefore our stated running times also bound the number of accesses to K. We denote Õ ( f ) = O ( f · polylog f ) to suppress logarithmic factors . Theorem 1.1 ( Informal version of Theorem 3.1 ) . Given n-point weighted dataset X , oracle access to a kernel K : X ×X → R+ , integer k ≥ 1 and 0 < < 1 , one can construct in time Õ ( nk ) , a 1In fact , evaluating ‖c ? − ϕ ( u ) ‖2 for a single point u ∈ X already requires Θ ( n2 ) accesses , since ‖c ? − ϕ ( u ) ‖2 = K ( u , u ) − 2 n ∑ x∈X K ( x , u ) + 1 n2 ∑ x , y∈X K ( x , y ) . reweighted subset S ⊆ X of size |S| = poly ( k −1 ) , that with high probability is an -coreset for kernel k-MEANS with respect to X and K. We can employ our coreset to devise a ( 1 + ) -approximation algorithm for kernel k-MEANS , that runs in time that is near-linear in n and parameterized by k. This is stated in Corollary 1.2 , whose proof follows by solving k-MEANS on S optimally , using straightforward enumeration over all kpartitions of S. To the best of our knowledge , such a fast ( 1+ ) -approximation for kernel k-MEANS was not known even for k = 2 ; for example , uniform sampling would fail in cases where the optimal clustering is very imbalanced , as mentioned earlier . Corollary 1.2 ( FPT-PTAS ) . Given n-point weighted dataset X , oracle access to a kernel K : X × X → R+ , integer k ≥ 1 and 0 < < 1 , one can compute in time O ( nk + kpoly ( k −1 ) ) , a center set C of k points , each represented as a linear combination of at most poly ( k −1 ) points from ϕ ( X ) , such that with high probability C is a ( 1 + ) -approximation for kernel k-MEANS on X and K. In particular , given such C , one can find for each x ∈ X its closest center in C in time poly ( k −1 ) . In fact , for the purpose of finding near-optimal solutions , it already suffices to preserve the cost for centers coming from span ( ϕ ( X ) ) ( see Fact 2.1 ) which is an n-dimensional subspace . However , our definition of coreset in ( 3 ) is much stronger , in that the objective is preserved even for centers coming from a possibly infinite-dimensional feature space . This stronger guarantee ensures that the coreset is composable , and thus the standard merge-and-reduce method can be applied . In particular , our coreset implies the first streaming algorithm for kernel k-MEANS . Corollary 1.3 ( Streaming kernel k-MEANS ) . There is a streaming algorithm that given a dataset X presented as a stream of n points , and oracle access to a kernel K : X ×X → R+ , constructs a reweighted subset S ⊆ X of poly ( k −1 ) points using poly ( k −1 log n ) words of space and update time , such that with high probability S is an -coreset for k-MEANS with respect to X and K. Experiments and other applications . We validate the efficiency and accuracy of our coresets on various data sets with polynomial and Gaussian kernels , which are frequently-used kernels . For every dataset , kernel , and coreset-size that we test , our coreset performs consistently better than uniform sampling which serves as a baseline . In fact , our coreset achieves less than 10 % error using only about 1000 points for every dataset . We also showcase significant speedup to several applications that can be obtained using our coresets . Specifically , we adapt the widely used k-MEANS++ ( Arthur & Vassilvitskii , 2007 ) to the kernel setting , and we compare the running time and accuracy of this kernelized k-MEANS++ with and without coresets . On a dataset of size 105 , we observe more than 1000x speedup of k-MEANS++ when using coresets , while achieving a very similar error . Furthermore , this new efficient version of kernelized k-MEANS++ ( based on coresets ) is applied to solve spectral clustering , using the connection discoverd by Dhillon et al . ( 2004 ) . Compared to the implementation provided by Scikitlearn ( Pedregosa et al. , 2011 ) , our algorithm often achieves a better result and uses significantly less time . Hence , our coreset-based approach can potentially become the leading method for solving spectral clustering in practice .

Summary: The paper claims to present the first coreset for kernel k-means and other (k,z) kernel clusterings given oracle access to the kernel. The size of coreset is independent of the number of points and construction time is near linear in k. The authors use the coreset to get a (1+\epsilon) approximation for the kernel k-means. The theory is supported with experimental results.