Label consistency in overfitted generalized $k$-means

1 Introduction . Consider the problem of clustering data points sampled according to some probability measure µ from a normed space X with norm ‖ · ‖X . In the ideal setting , the generalized k-means clustering minimizes the population cost function Q ( ξ ; µ ) : = ( ∫ min 1≤ ` ≤L ‖x− ξ ` ‖pXdµ ( x ) ) 1/p ( 1 ) where ξ = ( ξ1 , . . . , ξL ) ∈ XL is a set of L vectors in X , for some fixed integer L. In practical data analysis , we are given a sample { x1 , . . . , xn } drawn from µ and solve an empirical version of ( 1 ) , namely , Q̂ ( ξ ) = Q ( ξ ; Pn ) : = ( 1 n n∑ i=1 min 1≤ ` ≤L ‖xi − ξ ` ‖pX ) 1/p . ( 2 ) Here , Pn : = 1n ∑n i=1 δxi is the empirical measure associated with the sample and δx is the point mass measure at x . The minimizer of Q̂ ( · ) over XL is denoted as ξ̂ = ( ξ̂1 , . . . , ξ̂L ) and each point xi is assigned a cluster label ẑi : = argmin ` ‖xi − ξ̂ ` ‖X . Meanwhile , we assume that each data point xi also has a true cluster label zi ∈ [ K ] : = { 1 , . . . , K } which is determined solely by an unknown data-generating process . These true labels are not necessarily related to the optimal solutions of ( 1 ) or ( 2 ) . To distinguish the two , we refer to the clustering induced by ( zi ) as the true clustering , while a clustering that minimizes the generalized k-means cost function ( 2 ) , i.e. , the clustering induced by ( ẑi ) , is referred to as an optimal k-means clustering . In this paper , we consider the label consistency problem , that is , how close the labels ( ẑi ) estimated by k-means clustering are to the true labels ( zi ) . Note that we allow the number of k-means clusters L to be different from the true number of clusters K. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In the above formulation , the case where p = 2 , X = Rd and ‖ · ‖X is the Euclidean norm leads to the classical and widely used k-means problem . Much of the theoretical analysis of k-means has been performed in this case . Early work has focused on how close the optimization problems based on the empirical and ideal cost functions ( 2 ) and ( 1 ) are to each other , where the closeness is measured in terms of the recovered centers ( i.e. , ξ̂ and ξ ) or the optimal value of the objective function . Such consistency results are proved , for the global minimizers of ( 2 ) , in the early work of [ 22 , 29 ] and also in [ 30 , 19 ] from the vector quantization perspective . These classical results do not directly apply to the performance of the k-means in practice , mainly because solving ( 2 ) is NP-hard and approximation methods are usually applied to deal with it . Also , considerations of the label consistency problem are absent from this line of work since no true clustering , external to the k-means problem , is assumed to exist . More recently , there has been more interest in the consistency of practical k-means algorithms [ 14 , 21 ] as well as the label consistency problem . Lu and Zhou [ 21 ] obtain sharp bounds on the label consistency of the Lloyd ’ s algorithm [ 20 ] under a sub-Gaussian mixture model . Semidefinite programming ( SDP ) relaxation is another popular technique for deriving polynomial-time approximations to the k-means problem [ 28 ] . Its label consistency has been studied when data is generated from the stochastic ball model [ 4 , 10 ] , sub-Gaussian mixtures [ 25 , 8 , 9 ] , the Stochastic Block Model ( SBM ) [ 9 ] and general models [ 18 ] . Convex clustering is another relaxation method whose label consistency has been discussed in [ 34 , 27 , 11 , 31 ] . The literature on community detection in SBM , a network clustering problem , is also mainly focused on label consistency and inspires our work here ; see [ 1 , 33 ] for a review of those results . For label consistency in kernel spectral clustering , see [ 2 ] . In this paper , we study the label consistency of approximate solutions of the generalized k-means problem ( 2 ) when L ≥ K. Our focus will be on the overfitted case where L > K. This is often relevant in practice since the data-generating process may have a natural number of clusters K that is unknown a priori . An example is the sub-Gaussian mixture with K components . More interesting examples are given in Section 3 . All the aforementioned works on label consistency exclusively consider the correctly-fitted case L = K. We show that when the data-generating process admits a set of centers that satisfy certain separation conditions , estimated labels with L ≥ K clusters , are close to a refinement of the true labels . These bounds reduce to the label consistency criteria for L = K , but have no counterpart in the literature for L > K. Overfitting in k-means is considered in [ 32 , 23 ] where it is shown to improve the approximation factor ( see Assumption 1 ( b ) ) of certain polynomial-time k-means algorithms . Analysis of the approximation factor is concerned with how close one can get to the optimal value of the k-means objective function . In contrast , we are concerned with the label recovering problem and not directly concerned with how well the objective function is approximated . Our work is also aligned with the recent trend of beyond worst case analysis of the NP-hard problems [ 6 ] , where the performance of the algorithms are considered assuming that there are some meaningful structures in the data ( e.g. , true clusters ) . We refer to Remark 1 for a more detailed comparison with this literature . Our results are algorithm-free in the sense that they apply to any algorithm that achieves a constantfactor approximation to the optimal objective . They are also model-free in the sense that we do not make any explicit assumption on the data-generating process . This is important in practice , since many common data models , such as sub-Gaussian mixtures , are often too simplified to capture real clustering problems . We provide examples of more complicated data models in Section 3 and show how our general results can provide new insights for these models . Since k-means clustering often appears as a building block in many sophisticated clustering algorithms , we believe our results will be of broad interest in understanding the performance of clustering algorithms in overfitted settings . Notation . Q ( ξ ; µ ) is only dependent on the set of values among the coordinates of ξ . Although we view ξ as a vector ( for which the order of elements matter ) , with some abuse of notation , we view Q ( · ; µ ) as a set function ( mapping 2X to R ) that is only sensitive to the set of values represented by ξ . This justifies using the the same symbol for the function irrespective of the number of coordinates of ξ , i.e. , the number of clusters . The reason to keep ξ as an ( ordered ) vector is to make the cluster labels well-defined . For simplicity , let ‖ · ‖ = ‖ · ‖X . For the case where X ⊂ Rd , one often takes ‖ · ‖ to be the Euclidean norm , but our results are valid for any norm on Rd , and more broadly any normed space X . 2 Main Results . We first state assumptions about the k-means clustering algorithm . Assumption 1 . Consider an algorithm for the generalized k-means problem ( 2 ) , referred to as ALG ( p ) hereafter , and let ξ̂ ( L ) ∈ XL and ξ̂ ( K ) ∈ XK be its estimated centers when applied with L and K clusters , respectively . Let L ≥ K. Assume that ALG ( p ) has the following properties , for all input sequences ( xi ) : ( a ) Efficiency : The Voronoi cell of every estimated center ξ̂ ( L ) ` contains at least one of ( xi ) . ( b ) κ-approximation : Q̂ ( ξ̂ ( K ) ) ≤ κ ·minξ∈XK Q̂ ( ξ ) , and similarly with K replaced by L. Efficiency can be achieved by substituting centers whose Voronoi cells have an empty intersection with { xi } , by those having the opposite property . For κ-approximation , the factor κ can depend on the number of clusters K ( or L ) . For example , the k-means++ algorithm has κ = O ( logK ) , with high probability over the initialization [ 3 ] . However , there are also constant-factor approximation algorithms for k-means where κ = O ( 1 ) independent of K ( or L ) [ 24 , 13 , 15 ] . For example , with local search , k-means++ can achieve a constant-factor approximation [ 16 ] . In addition , κapproximation is not required for all inputs . That is , we are not concerned with the worst-case approximation factor . The κ in Assumption 1 ( b ) is the approximation factor of the algorithm on the specific data under consideration . It is enough for an algorithm to achieve good approximation only on the data of interest . For some of the results , Assumption 1 ( b ) can be replaced with the following modified version : ( b′ ) κ-approximation only for K clusters plus a mononoticity assumption : Q̂ ( ξ̂ ( L ) ) ≤ Q̂ ( ξ̂ ( K ) ) . Mononoticity is also a reasonable requirement and obviously true for the exact k-means solutions . Next , we extend the definition of the misclassification rate to the overfitted case . Definition 1 . The ( generalized ) misclassification rate between two label vectors z ∈ [ K ] n and ẑ ∈ [ L ] n , with K ≤ L , is Miss ( z , ẑ ) = min ω 1 n n∑ i=1 1 { zi 6= ω ( ẑi ) } , where the minimization ranges over all surjective maps ω : [ L ] → [ K ] . When L = K , a surjective map ω is necessarily a bijection and the above becomes the usual definition of misclassification rate when the number of clusters is correctly identified . In this case , Miss ( z , ẑ ) = 0 means that there is a one-to-one correspondence between the estimated and true clusters . The generalized definition above allows us to extend this notion of exact recovery to the case L > K. In particular , Miss ( z , ẑ ) = 0 when L > K , if and only if ẑ is a refinement of z . To see this , note that Miss ( z , ẑ ) = 0 implies the existence of a map ω : [ L ] → [ K ] such that ω ( ẑi ) = zi for all i . This in turn is equivalent to : ẑi = ẑi′ =⇒ zi = zi′ , which is the refinement relation for the associated clusters . In general , Miss ( z , ẑ ) is small if ẑ is close to a refinement of z . We also use the ( optimal ) matching distances between elements of two vectors viewed as sets . Definition 2 . For ξ ∈ XL and ξ∗ ∈ XK , define the ` ∞ and ` p optimal matching distances as d∞ ( ξ , ξ ∗ ) = min σ max 1≤k≤K ‖ξσ ( k ) − ξ∗k‖ , dp ( ξ , ξ∗ ) = min σ ( K∑ k=1 ‖ξσ ( k ) − ξ∗k‖p ) 1/p , where σ : [ K ] → [ L ] ranges over all injective maps . For K = L , d∞ is an upper bound on the Hausdorff distance between the two sets . Obviously , we have d∞ ≤ dp for any p ≥ 1 .

Summary: The paper considers the problem of label recovery from data clustering. Generally speaking, these two tasks are unrelated: the label recovery is a supervised learning problem, where each data point has a unknonwn true label, and the goal is to infer the label for each data point; while the clustering problem is to group data together to minimize the generalized $k$-means cost (i.e., replace $\ell_2$-error with $\ell_p$-error in traditional $k$-means), which usually has nothing to do with labelings. Although the former task also induces a clustering (in which data points with the same label belongs to the same cluster), it can be a very bad solution for the later task, and vice versa. To distinguish, for a given data set, we say a clustering is a **true clustering** if it recovers the unknown labeling (up to permutations); while we say it is an **optimal $k$-means clustering** if it solves the second task, i.e., minimizes the generalized $k$-means cost function.  The paper shows that, when the *true clustering* satisfy certain regularity conditions, it can be recovered (up to "refinement") by (an $O(1)$ approximation to) the \emph{optimal $k$-means clustering}. The paper defines a **refinement** of labeling $z\in[K]^n$ to be another labeling $\tilde{z}\in[L]^n$ together with a mapping $\omega:[L]\mapsto[K]$, such that $\omega(\tilde{z}_i)=z_i$ for all $i\in[n]$. Intuitively one can think of this as breaking the *true clusters* further into multiple smaller clusters.  Per my understanding the main result can be summarized as follows. When the true clustering is itself a good clustering with respect to the generalized $K$-means cost (Theorem 1 and 2), or if the true clustering can be broken into $L>K$ smaller clusters which have small $L$-means cost (Theorem 3), then an $O(1)$-approximation for the optimal $L$-means clustering is also a good approximation for the true clustering, in the sense that it approximately induces a refinement of the true labeling.