Refined Learning Bounds for Kernel and Approximate $k$-Means

√ n ) , which do not match with the stated lower bound Ω ( √ k/n ) in terms of k , where k is the number of clusters and n is the size of the training set . In this paper , we study the statistical properties of kernel k-means and Nyström-based kernel k-means , and obtain optimal clustering risk bounds , which improve the existing risk bounds . Particularly , based on a refined upper bound of Rademacher complexity [ 21 ] , we first derive an optimal risk bound of rate O ( √ k/n ) for empirical risk minimizer ( ERM ) , and further extend it to general cases beyond ERM . Then , we analyze the statistical effect of computational approximations of Nyström kernel k-means , and prove that it achieves the same statistical accuracy as the original kernel k-means considering only Ω ( √ nk ) Nyström landmark points . We further relax the restriction of landmark points from Ω ( √ nk ) to Ω ( √ n ) under a mild condition . Finally , we validate the theoretical findings via numerical experiments . 1 Introduction . Clustering , a fundamental data mining task , is used in numerous applications including web search , medical imaging , gene expression analysis , social network analysis and recommendation systems [ 56 , 55 , 23 , 42 ] . k-means is arguably one of the most popular approaches to clustering , producing clusters with piece-wise linear boundaries . Its kernel version , which employs a nonlinear distance function , has the ability to find clusters of varying densities and distributions , greatly improving the flexibility of the approach [ 18 , 53 , 38 , 37 , 39 , 60 , 29 , 36 , 35 , 61 ] . To understand ( kernel ) k-means and guide the development of new clustering algorithms , researchers have investigated its theoretical properties for decades . The consistency of the empirical minimizer was demonstrated by [ 45 , 47 , 1 ] . Rates of convergence and non-asymptotic performance bounds were considered by [ 46 , 13 , 33 , 7 , 32 , 14 , 20 ] . Most of the proposed risk bounds are dependent upon the dimension of the hypothesis space . For example , Bartlett et al . [ 7 ] provided , under certain mild assumptions , a clustering risk bound of orderO ( √ kd/n ) , where d is the dimension of the hypothesis space and n is the size of the training set . However , the hypothesis space of kernel k-means is typically an infinite-dimensional Hilbert space , such as the reproducing kernel Hilbert space ( RKHS ) associated with Gaussian kernels [ 50 ] . Thus , the existing theoretical analysis of k-means are not usually suitable for explaining its kernel version . Recently , [ 20 , 16 , 10 , 41 , 3 , 27 , 24 , 9 ] extended the previous results , and provided dimension-independent bounds for kernel k-means . As shown in [ 16 ] , if the feature map associated with the kernel function satisfies ‖Φ‖ ≤ 1 , then the clustering risk bounds are of order O ( k/ √ n ) . These clustering risk bounds for kernel k-means are usually linearly 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . dependent on the number of clusters k. However , the number of clusters k may be very large in some domains , such as social networks and recommendation systems . Thus , from a theoretical perspective , these existing bounds of O ( k/ √ n ) do not match with the stated lower bound Ω ( √ k/n ) in k [ 7 ] . Although kernel k-means is one of the most popular clustering methods , it requires the computation of a n × n kernel matrix . As for other kernel methods , this becomes unfeasible for large-scale problems , and thus deriving approximate computations , such as partial decompositions [ 6 , 28 ] , random projection [ 16 , 15 ] , Nyström approximations [ 19 , 11 , 9 , 43 , 53 , 58 , 57 ] , and random feature approximations [ 48 , 12 , 5 , 49 , 34 , 31 ] , has become the subject of numerous recent works . However , few of these optimization-based methods focused on the underlying excess risk problem . To the best of our knowledge , the only two results providing excess risk guarantees for approximate kernel k-means are [ 16 ] and [ 9 ] . In [ 16 ] , Devroye and Lugosi considered the excess clustering risk when the approximate Hilbert space is obtained using Gaussian projections . In [ 9 ] , Calandriello and Rosasco showed that , when sampling Ω ( √ n ) Nyström landmarks , the excess risk bound can reach O ( k/ √ n ) . The excess risk bounds of [ 9 ] and [ 16 ] are both linearly dependent on k and thus do not match with the theoretical lower bound [ 7 ] . In the recent work [ 21 ] , the authors showed that the Rademacher complexity of the k-valued function class of Lipschitz continuity with respect to the L∞ norm can be bounded by the maximum Rademacher complexity of the restriction of the function class along each coordinate , times a factor of O ( √ k ) . Although it may be not very difficult to use the result of [ 21 ] for kernel k-means , the optimal bound of O ( √ k/n ) for kernel k-means has never been given before . Moreover , we creatively extend the results of kernel k-means to the approximate one . Our major contributions include two parts : 1 ) A ( nearly ) optimal excess clustering risk bound of rate Õ ( √ k/n ) 1 is proposed for empirical risk minimization ( ERM ) ( see Theorem 1 ) . To the best of our knowledge , this is the first ( nearly ) optimal excess risk bound for kernel k-means in terms of both k and n. Beyond ERM , we further extend the result of Theorem 1 to general cases ( see Theorem 2 and Theorem 3 ) . 2 ) A ( nearly ) optimal excess risk bound for Nyström kernel k-means is also obtained when sampling Ω ( √ nk ) points ( see Theorem 4 ) . We further relax the restriction of landmark points from Ω ( √ nk ) to Ω ( √ n ) ( see Theorem 5 ) and extend it to general cases ( see Theorem 6 and Theorem 7 ) . This result shows that we can use the Nyström method to improve the effectiveness of kernel k-means , while guaranteeing the optimal generalization performance . The rest of the paper is organized as follows . In Section 2 , we introduce some notations and provide an overview of kernel k-means . In Section 3 , we provide nearly optimal excess risk bounds . In Section 4 , we quantify the statistical effect of computational approximations of the Nyström-based kernel k-means . In Section 5 , we validate our theoretical findings by performing experiments on both simulated and real data . We end in Section 6 with conclusion . All the detailed proofs are deferred to the Appendix . 2 Background . In this section , we will introduce some notations and provide a brief introduction of kernel k-means . Please refer to [ 18 , 9 ] for more details . 2.1 Notations . Assume P is a ( unknown ) distribution on X , and S = { xi } ni=1 ∈ X is a set of n samples drawn i.i.d . from P. The empirical distribution Pn is denoted as Pn ( x ) = 1n if x ∈ S , otherwise 0 . Let κ : X ×X → R be a mercer kernel [ 51 ] , andH be its associated RKHS [ 52 ] , which is the completion of the linear span of the set of functions : H = span { κ ( x , · ) , x ∈ X } . We denote the Cartesian product of H by Hk = ⊗ki=1H . We use the feature map ψ : X → H to map X into the Hilbert spaceH , and assume thatH is separable , such that for any x ∈ X , we have 1Õ hides logarithmic terms . Φx = ψ ( x ) . Intuitively , in the rest of the paper , the reader can assume that Φx ∈ Rd with d n or even infinite . From here on , we will denote the inner product ofH by 〈· , ·〉 , and the associated norm by ‖ · ‖ , and assume that ‖Φx‖ ≤ 1 for any x ∈ X . We let D = { Φi = ψ ( xi ) } ni=1 , and denote [ K ] i , j = κ ( xi , xj ) = 〈Φi , Φj〉 as the kernel matrix . The notations µ = O ( ν ) and µ = Ω ( ν ) mean that there exist constants c , c1 , c2 such that µ ≤ cν and c1ν ≤ µ ≤ c2ν , respectively . We use Õ and Ω̃ to hide logarithmic terms . 2.2 Kernel k-Means In this paper , we aim at partitioning the given dataset into k disjoint clusters , each characterized by its centroid cj . The Voronoi cell associated with a centroid cj is defined as [ 9 ] Cj : = { i : j = arg min s=1 , ... , k ‖Φi − cs‖2 } . Let C = [ c1 , . . . , ck ] be a collection of k centroids fromHk . In this paper , we focus on the so-called kernel k-means clustering , by minimizing the empirical squared norm criterion W ( C , Pn ) : = 1 n n∑ i=1 min j=1 , ... , k ‖Φi − cj‖2 ( 1 ) over all possible choices of cluster centers C ∈ Hk . From [ 18 , 9 ] , we know thatW ( C , Pn ) can be written as W ( C , Pn ) : = 1 n min C∈Hk k∑ j=1 ∑ i∈Cj ∥∥∥∥∥∥Φi − 1|Cj | ∑ t∈Cj Φt ∥∥∥∥∥∥ 2 = 1 n min C∈Hk k∑ j=1 ∑ i∈Cj κ ( xi , xi ) − 2|Cj |∑ t∈Cj κ ( xi , xt ) + 1 |Cj |2 ∑ t , t′∈Cj κ ( xt , xt′ ) . The empirical risk minimizer ( ERM ) is defined as Cn : = arg min C∈Hk W ( C , Pn ) . ( 2 ) The performance of a clustering scheme given by the collection C = [ c1 , . . . , ck ] ∈ Hk of cluster centers is usually measured by the expected squared norm criterion or expected clustering risk W ( C , P ) : = ∫ min j=1 , ... , k ‖Φx − cj‖2dP ( x ) . Given a C ∈ Hk , let fC = ( fc1 , . . . , fck ) be a k-valued function of the collection C with fcj ( x ) = ‖Φx − cj‖2 . Let ϕ : Rk → R be a minimum function . From the definition of ϕ ( fC ( x ) ) = min ( fc1 ( x ) , . . . , fck ( x ) ) , one can see that the empirical and expected squared norm criteria can be respectively written as W ( C , Pn ) : = 1 n n∑ i=1 ϕ ( fC ( xi ) ) andW ( C , P ) : = ∫ ϕ ( fC ( x ) ) dP ( x ) . In this paper , we consider bounding the excess clustering risk E ( Cn ) of the empirical risk minimizer [ 16 ] : E ( Cn ) : = ED [ W ( Cn , P ) ] −W∗ ( P ) , whereW∗ ( P ) = infC∈HkW ( C , P ) is the optimal clustering risk . In the following , we will ignore the subscript D if the input dataset D is clear .

Summary: In this paper, the authors provide a near-optimal risk bound for kernel $k$-means, namely $\tilde{O}(\sqrt{k/n})$, which improves upon the existing risk bound $O(k/\sqrt{n})$, and nearly matches the lower bound $\Omega(\sqrt{k/n})$, where $k$ is the number of clusters and $n$ is the size of the training set. They first derive a near-optimal risk bound for the empirical risk minimizer (ERM) and then extend it to general cases beyond ERM and $k$-means++. For efficiency, the authors also provide near-optimal risk bounds for Nyström kernel $k$-means for the case of using $\tilde{\Omega}(\sqrt{n})$ sampling points. The authors have performed numerical experiments to validate their theoretical findings.