Label consistency in overfitted generalized $k$-means

1 Introduction . Consider the problem of clustering data points sampled according to some probability measure µ from a normed space X with norm ‖ · ‖X . In the ideal setting , the generalized k-means clustering minimizes the population cost function Q ( ξ ; µ ) : = ( ∫ min 1≤ ` ≤L ‖x− ξ ` ‖pXdµ ( x ) ) 1/p ( 1 ) where ξ = ( ξ1 , . . . , ξL ) ∈ XL is a set of L vectors in X , for some fixed integer L. In practical data analysis , we are given a sample { x1 , . . . , xn } drawn from µ and solve an empirical version of ( 1 ) , namely , Q̂ ( ξ ) = Q ( ξ ; Pn ) : = ( 1 n n∑ i=1 min 1≤ ` ≤L ‖xi − ξ ` ‖pX ) 1/p . ( 2 ) Here , Pn : = 1n ∑n i=1 δxi is the empirical measure associated with the sample and δx is the point mass measure at x . The minimizer of Q̂ ( · ) over XL is denoted as ξ̂ = ( ξ̂1 , . . . , ξ̂L ) and each point xi is assigned a cluster label ẑi : = argmin ` ‖xi − ξ̂ ` ‖X . Meanwhile , we assume that each data point xi also has a true cluster label zi ∈ [ K ] : = { 1 , . . . , K } which is determined solely by an unknown data-generating process . These true labels are not necessarily related to the optimal solutions of ( 1 ) or ( 2 ) . To distinguish the two , we refer to the clustering induced by ( zi ) as the true clustering , while a clustering that minimizes the generalized k-means cost function ( 2 ) , i.e. , the clustering induced by ( ẑi ) , is referred to as an optimal k-means clustering . In this paper , we consider the label consistency problem , that is , how close the labels ( ẑi ) estimated by k-means clustering are to the true labels ( zi ) . Note that we allow the number of k-means clusters L to be different from the true number of clusters K. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In the above formulation , the case where p = 2 , X = Rd and ‖ · ‖X is the Euclidean norm leads to the classical and widely used k-means problem . Much of the theoretical analysis of k-means has been performed in this case . Early work has focused on how close the optimization problems based on the empirical and ideal cost functions ( 2 ) and ( 1 ) are to each other , where the closeness is measured in terms of the recovered centers ( i.e. , ξ̂ and ξ ) or the optimal value of the objective function . Such consistency results are proved , for the global minimizers of ( 2 ) , in the early work of [ 22 , 29 ] and also in [ 30 , 19 ] from the vector quantization perspective . These classical results do not directly apply to the performance of the k-means in practice , mainly because solving ( 2 ) is NP-hard and approximation methods are usually applied to deal with it . Also , considerations of the label consistency problem are absent from this line of work since no true clustering , external to the k-means problem , is assumed to exist . More recently , there has been more interest in the consistency of practical k-means algorithms [ 14 , 21 ] as well as the label consistency problem . Lu and Zhou [ 21 ] obtain sharp bounds on the label consistency of the Lloyd ’ s algorithm [ 20 ] under a sub-Gaussian mixture model . Semidefinite programming ( SDP ) relaxation is another popular technique for deriving polynomial-time approximations to the k-means problem [ 28 ] . Its label consistency has been studied when data is generated from the stochastic ball model [ 4 , 10 ] , sub-Gaussian mixtures [ 25 , 8 , 9 ] , the Stochastic Block Model ( SBM ) [ 9 ] and general models [ 18 ] . Convex clustering is another relaxation method whose label consistency has been discussed in [ 34 , 27 , 11 , 31 ] . The literature on community detection in SBM , a network clustering problem , is also mainly focused on label consistency and inspires our work here ; see [ 1 , 33 ] for a review of those results . For label consistency in kernel spectral clustering , see [ 2 ] . In this paper , we study the label consistency of approximate solutions of the generalized k-means problem ( 2 ) when L ≥ K. Our focus will be on the overfitted case where L > K. This is often relevant in practice since the data-generating process may have a natural number of clusters K that is unknown a priori . An example is the sub-Gaussian mixture with K components . More interesting examples are given in Section 3 . All the aforementioned works on label consistency exclusively consider the correctly-fitted case L = K. We show that when the data-generating process admits a set of centers that satisfy certain separation conditions , estimated labels with L ≥ K clusters , are close to a refinement of the true labels . These bounds reduce to the label consistency criteria for L = K , but have no counterpart in the literature for L > K. Overfitting in k-means is considered in [ 32 , 23 ] where it is shown to improve the approximation factor ( see Assumption 1 ( b ) ) of certain polynomial-time k-means algorithms . Analysis of the approximation factor is concerned with how close one can get to the optimal value of the k-means objective function . In contrast , we are concerned with the label recovering problem and not directly concerned with how well the objective function is approximated . Our work is also aligned with the recent trend of beyond worst case analysis of the NP-hard problems [ 6 ] , where the performance of the algorithms are considered assuming that there are some meaningful structures in the data ( e.g. , true clusters ) . We refer to Remark 1 for a more detailed comparison with this literature . Our results are algorithm-free in the sense that they apply to any algorithm that achieves a constantfactor approximation to the optimal objective . They are also model-free in the sense that we do not make any explicit assumption on the data-generating process . This is important in practice , since many common data models , such as sub-Gaussian mixtures , are often too simplified to capture real clustering problems . We provide examples of more complicated data models in Section 3 and show how our general results can provide new insights for these models . Since k-means clustering often appears as a building block in many sophisticated clustering algorithms , we believe our results will be of broad interest in understanding the performance of clustering algorithms in overfitted settings . Notation . Q ( ξ ; µ ) is only dependent on the set of values among the coordinates of ξ . Although we view ξ as a vector ( for which the order of elements matter ) , with some abuse of notation , we view Q ( · ; µ ) as a set function ( mapping 2X to R ) that is only sensitive to the set of values represented by ξ . This justifies using the the same symbol for the function irrespective of the number of coordinates of ξ , i.e. , the number of clusters . The reason to keep ξ as an ( ordered ) vector is to make the cluster labels well-defined . For simplicity , let ‖ · ‖ = ‖ · ‖X . For the case where X ⊂ Rd , one often takes ‖ · ‖ to be the Euclidean norm , but our results are valid for any norm on Rd , and more broadly any normed space X . 2 Main Results . We first state assumptions about the k-means clustering algorithm . Assumption 1 . Consider an algorithm for the generalized k-means problem ( 2 ) , referred to as ALG ( p ) hereafter , and let ξ̂ ( L ) ∈ XL and ξ̂ ( K ) ∈ XK be its estimated centers when applied with L and K clusters , respectively . Let L ≥ K. Assume that ALG ( p ) has the following properties , for all input sequences ( xi ) : ( a ) Efficiency : The Voronoi cell of every estimated center ξ̂ ( L ) ` contains at least one of ( xi ) . ( b ) κ-approximation : Q̂ ( ξ̂ ( K ) ) ≤ κ ·minξ∈XK Q̂ ( ξ ) , and similarly with K replaced by L. Efficiency can be achieved by substituting centers whose Voronoi cells have an empty intersection with { xi } , by those having the opposite property . For κ-approximation , the factor κ can depend on the number of clusters K ( or L ) . For example , the k-means++ algorithm has κ = O ( logK ) , with high probability over the initialization [ 3 ] . However , there are also constant-factor approximation algorithms for k-means where κ = O ( 1 ) independent of K ( or L ) [ 24 , 13 , 15 ] . For example , with local search , k-means++ can achieve a constant-factor approximation [ 16 ] . In addition , κapproximation is not required for all inputs . That is , we are not concerned with the worst-case approximation factor . The κ in Assumption 1 ( b ) is the approximation factor of the algorithm on the specific data under consideration . It is enough for an algorithm to achieve good approximation only on the data of interest . For some of the results , Assumption 1 ( b ) can be replaced with the following modified version : ( b′ ) κ-approximation only for K clusters plus a mononoticity assumption : Q̂ ( ξ̂ ( L ) ) ≤ Q̂ ( ξ̂ ( K ) ) . Mononoticity is also a reasonable requirement and obviously true for the exact k-means solutions . Next , we extend the definition of the misclassification rate to the overfitted case . Definition 1 . The ( generalized ) misclassification rate between two label vectors z ∈ [ K ] n and ẑ ∈ [ L ] n , with K ≤ L , is Miss ( z , ẑ ) = min ω 1 n n∑ i=1 1 { zi 6= ω ( ẑi ) } , where the minimization ranges over all surjective maps ω : [ L ] → [ K ] . When L = K , a surjective map ω is necessarily a bijection and the above becomes the usual definition of misclassification rate when the number of clusters is correctly identified . In this case , Miss ( z , ẑ ) = 0 means that there is a one-to-one correspondence between the estimated and true clusters . The generalized definition above allows us to extend this notion of exact recovery to the case L > K. In particular , Miss ( z , ẑ ) = 0 when L > K , if and only if ẑ is a refinement of z . To see this , note that Miss ( z , ẑ ) = 0 implies the existence of a map ω : [ L ] → [ K ] such that ω ( ẑi ) = zi for all i . This in turn is equivalent to : ẑi = ẑi′ =⇒ zi = zi′ , which is the refinement relation for the associated clusters . In general , Miss ( z , ẑ ) is small if ẑ is close to a refinement of z . We also use the ( optimal ) matching distances between elements of two vectors viewed as sets . Definition 2 . For ξ ∈ XL and ξ∗ ∈ XK , define the ` ∞ and ` p optimal matching distances as d∞ ( ξ , ξ ∗ ) = min σ max 1≤k≤K ‖ξσ ( k ) − ξ∗k‖ , dp ( ξ , ξ∗ ) = min σ ( K∑ k=1 ‖ξσ ( k ) − ξ∗k‖p ) 1/p , where σ : [ K ] → [ L ] ranges over all injective maps . For K = L , d∞ is an upper bound on the Hausdorff distance between the two sets . Obviously , we have d∞ ≤ dp for any p ≥ 1 .

Summary: This paper considers the problem of label consistency in the generalized k-means problem in an overfit situation where the number of clusters $L$ used by a given clustering algorithm is more than the ground truth $K$.  The main contributions are:  1. Theoretical guarantees the above problem, under certain assumptions on the clustering algorithm, for both  exact and approximate label recovery. To the best of my knowledge, this is the first result in the context of the k-means problem.  2. The results are model-free, but strongly depend on the maximum or average distance of the points to true cluster center.