Uniform Concentration Bounds toward a Unified  Framework for Robust Clustering

1 Introduction . Clustering is a fundamental task in unsupervised learning , which seeks to discover naturally occurring groups within a dataset . Among a plethora of algorithms in the literature , center-based methods remain widely popular , with k-means [ 35 , 33 ] as the most prominent example . Given n data points X = { Xi : i = 1 , . . . , n } ⊂ Rp , k-means seeks to partition the data into k mutually exclusive and exhaustive groups by minimizing the within cluster variance . Representing the cluster centroids Θ = { θ1 , . . . , θk } ⊂ Rp , the k-means problem is formulated as the minimization of the objective function fk-means ( Θ ) = ∑n i=1 min1≤j≤k d ( Xi , θj ) , where d ( · , · ) is a dissimilarity measure . Taking the squared Euclidean distance d ( x , y ) = ‖x− y‖22 yields the classical k-means formulation . Clustering via k-means is well-documented to be sensitive to initialization [ 54 ] , prone to getting stuck in poor local optima [ 57 , 56 , 17 ] , and fragile against linearly non-separable clusters [ 39 ] or ∗Joint first authors contributed equally ‡ Corresponding author 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in the presence of even a single outlying observation [ 26 ] . Researchers continue to tackle these drawbacks of k-means while preserving its simplicity and interpretability [ 4 , 1 , 48 , 40 , 58 , 49 ] . Although often successful in practice , few of these approaches are grounded in rigorous theory or provide finite-sample statistical guarantees . The available consistency results are mostly asymptotic in nature [ 17 , 51 , 52 ] , lacking convergence rates or operating under restrictive assumptions on the relation between p and n [ 42 ] . For example , the recent large sample analysis [ 42 ] of the hard Bregman k-means algorithm [ 4 ] obtains an asymptotic error rate of O ( √ log n/n ) under restrictive assumptions on the relation between n and p. The approach by [ 48 ] focuses on a unified framework from an optimization perspective , while [ 3 , 49 ] focus on different divergence-based methods to better understand the underlying feature-space . The presence of outliers only further complicates matters . Outlying data are common in real applications , but many of the aforementioned approaches are fragile to deviations from the assumed data generating mechanism . On the other hand , recent work on robust clustering methods [ 18 , 21 , 26 ] do not integrate the practical advances surveyed above , and similarly lack finite-sample guarantees . To bridge this gap , the Median of Means ( MoM ) literature provides a promising and attractive framework to robustify center-based clustering methods against outliers . MoM estimators are not only insensitive to outliers , but are also equipped with exponential concentration results under the mild condition of finite variance [ 34 , 29 , 6 , 31 , 27 ] . Recently , near-optimal results for mean estimation [ 37 ] , classification [ 30 ] , regression [ 36 , 34 ] , clustering [ 26 , 13 ] , bandits [ 14 ] and optimal transport [ 46 ] have been established from this perspective . Under the MoM lens , we propose a unified framework for robust center-based clustering . Our treatment considers a family of Bregman loss functions not restricted to the classical squared Euclidean loss . We explore the exact sample error bounds for a general class of algorithms under mild regularity assumptions that apply to popular existing approaches , which we show to be special cases . The proposed framework allows for the data to be divided into two categories : the set of inliers ( I ) and the set of outliers ( O ) . The inliers are assumed to be independent and identically distributed ( i.i.d . ) , while absolutely no assumption is required on O , allowing outliers to be unboundedly large , dependent on each other , sampled from a heavy-tailed distribution and so on . Our contributions within the MoM framework make use of Rademacher complexities [ 5 , 7 ] and symmetrization arguments [ 53 ] , powerful tools that often find use in the empirical process literature but , in our view , are underexplored in the context of robust clustering . The paper is organized as follows . In Section 2 , we identify a general centroid-based clustering framework which encompasses k-means [ 35 ] , k-harmonic means [ 57 ] , and power k-means [ 56 ] as special cases , to name a few . We show how this framework is made robust via Median of Means estimation , yielding an array of center-based robust clustering methods . Within this framework , we derive uniform deviation bounds and concentration inequalities under standard regularity conditions through bounding Rademacher complexity by metric entropy via Dudley ’ s chaining argument in Section 3 . The analysis newly reveals the convergence rate for popularly used clustering methods such as k-harmonic means and power k-means , matching the known rate results for k-means , and elegantly carries over to their MoM counterparts . We then implement and empirically assess the resulting algorithms through simulated and real data experiments . In particular , we find that power k-means [ 56 ] under the MoM paradigm outperforms the state-of-the-art in the presence of outliers . 1.1 Related theoretical analyses of clustering . In seminal work , Pollard [ 43 ] , proved the strong consistency of k-means under a finite second moment assumption , spurring the large sample analysis of unsupervised clustering methods . This result has been extended to separable Hilbert spaces [ 10 , 32 ] and for related algorithms [ 15 , 51 , 52 ] , but these do not provide guarantees on the number of samples required so that the excess risk falls below a given threshold . Towards finding probabilistic error bounds , following research derived uniform concentration results for k-means and its variants [ 49 ] , sub-Gaussian distortion bounds for the kmedians problem [ 12 ] , and aO ( √ log n/n ) bound on k-means with Bregman divergences [ 41 ] . More recently , concentration inequalities for k-means under the MoM paradigm have been established [ 26 , 13 ] under the restriction that sample cluster centroids ( Θ̂n in Section 3 ) are assumed to be bounded . This paper shows how a number of center-based clustering methods can be brought under the same umbrella and can be robustified using a general-purpose scheme . The theoretical analyses of this broad spectrum of methods is conducted via Dudley ’ s chaining arguments and through the aid of Rademacher complexity-based uniform concentration bounds . This approach enables us to replace assumptions on the sample cluster centroids by a bounded support assumption of the ( inlying ) data points , which yields an intuitive way of ensuring the boundedness of the cluster centroids through the obtuse angle property of Bregman divergences ( Lemma 3.1 below ) . In contrast to the prior results , our analyses are not asymptotic in nature , so that the derived bounds hold for all values of the model parameters . 2 Problem Setting and Proposed Method . We consider the problem of partitioning a set of n data points X = { Xi : i = 1 , . . . , n } ⊂ Rp into k mutually exclusive clusters . In a center-based clustering framework , we represent the jth cluster by its centroid θj ∈ Rp for each j ∈ { 1 , . . . , k } . To quantify the notion of “ closeness '' , we allow the dissimilarity measure to be any Bregman divergence . Recall any differentiable , convex function φ : Rp → R generates the Bregman divergence dφ : Rp×Rp → R≥0 ( R≥0 denoting the set of non-negative reals ) defined as dφ ( x , y ) = φ ( x ) − φ ( y ) − 〈∇φ ( y ) , x− y〉 . For instance , φ ( u ) = ‖u‖22 generates the Euclidean distance . Without loss of generality , one may assume φ ( 0 ) = ∇φ ( 0 ) = 0 . In this paradigm , clustering is achieved by minimizing the objective 1 n n∑ i=1 Ψα ( dφ ( Xi , θ1 ) , . . . , dφ ( Xi , θk ) ) : = fΘ ( X ) . ( 1 ) Here Ψα : Rk≥0 → R≥0 is a component-wise non-decreasing function ( such as a generalized mean ) of the dissimilarities { dφ ( X , Θj ) } kj=1 which satisfies Ψ ( 0 ) = 0 . The hyperparameter α ∈ A ⊆ Rq is specified by the user , and we will additionally assume that Ψ is Lipschitz . For intuition , we begin by showing how this setup includes several popular clustering methods . Examples : Suppose φ ( u ) = ‖u‖22 and Ψ ( x ) = ( ∑k j=1 x −1 j ) −1 . Then the objective ( 1 ) becomes 1 n ∑n i=1 ( ∑k j=1 ‖Xi − θj‖ −2 2 ) −1 , which is the objective function of k-harmonic means clustering [ 57 ] . Now consider other generalized means : take Ψs ( x ) = Ms ( x ) where we denote the power mean Ms ( x ) = ( k −1∑k i=1 x s i ) 1/s for s ∈ ( −∞ , −1 ] . Then objective ( 1 ) coincides with the recent power k-means method [ 56 ] , 1n ∑n i=1Ms ( ‖Xi − θ1‖22 , . . . , ‖Xi − θk‖22 ) . When Ψ ( x ) = min1≤j≤k xj , ( 1 ) recovers Bregman hard clustering 1n ∑n i=1 min1≤j≤k dφ ( Xi , θj ) proposed in [ 4 ] for any valid φ , while the special case of φ ( u ) = ‖u‖22 yields the familiar Euclidean k-means problem [ 35 ] . In what follows , we derive concentration bounds that establish new theoretical guarantees such as consistency and convergence rates for clustering algorithms in this framework . These analyses lead us to a unified , robust framework by embedding this class of methods within the Median of Means ( MoM ) estimation paradigm . Via elegant connections between the properties of MoM estimators and Vapnik-Chervonenkis ( VC ) theory , our MoM estimators too will inherit uniform concentration inequalities from the preceding analysis , extending convergence guarantees to the robust setting . Median of Means Instead of directly minimizing the empirical risk ( 1 ) , MoM begins by partitioning the data into L sets B1 , . . . , BL which each contain exactly b many elements ( discarding a few observations when n is not divisible by L ) . The partitions can be assigned uniformly at random , or can be shuffled throughout the algorithm [ 30 ] . MoM then entails a robust version of the estimator defined under ( 1 ) by instead minimizing the following objective with respect to Θ : MoMnL ( Θ ) = Median ( 1 b ∑ i∈B1 fΘ ( Xi ) , . . . , 1 b ∑ i∈BL fΘ ( Xi ) ) . ( 2 ) Intuitively , MoM estimators are more robust than their ERM counterparts since under mild conditions , only a subset of the partitions is contaminated by outliers while others are outlier-free . Taking the median over partitions negates the influence of these spurious partitions , thus reducing the effect of outliers . Formal analysis of robustness via breakdown points is also available for MoM estimators [ 28 , 44 ] ; we will make use of the nice concentration properties of MoM estimators in Section 3.4 . Algorithm 1 MoM Clustering via Adagrad Input : X ∈ Rn×p , k , L , fΘ ( · ) , η , . Output : The cluster centroids Θ. Initialization : Randomly partition { 1 , . . . , n } into L many partitions of equal length . Randomly choose k points without replacement from X to initialize Θ0 . repeat Step 1 : Find ` t ∈ { 1 , . . . , L } , such that MoMnL ( Θt ) = 1b ∑ i∈B ` t fΘt ( Xi ) . Step 2 : g ( t ) j ← 1b ∑ i∈B ` t ∇θjfΘ ( Xi ) Step 3 : Update Θ by θ ( t+1 ) j ← θ ( t ) j − η√ + ∑ ( t ) t′=1 ‖g ( t′ ) j ‖22 g ( t ) j . until objective ( 2 ) converges Furthermore , optimizing ( 2 ) is made tractable via gradient-based methods . We advocate the Adagrad algorithm [ 19 , 22 ] , whose updates are given by θ ( t+1 ) j ← θ ( t ) j − η√ + ∑ ( t ) t′=1 ‖g ( t′ ) j ‖22 g ( t ) j for hyperparameter > 0 , learning rate η > 0 , and g ( t ) j denoting a subgradient of MoM n L ( Θ ) at Θt . That is , if ` t denotes the median partition at step t , then ∇Θj MoMnL ( Θt ) = 1 b ∑ i∈B ` t ∇θjfΘ ( Xi ) |Θ=Θt . For any Ψα differentiable , ∇θjfΘ ( x ) = ∂jΨα ( dφ ( x , θ ( t ) 1 ) , . . . , dφ ( x , θ ( t ) k ) ) ∇θdφ ( x , θ ) by the chain rule . As a concrete illustration of the general formula , consider the power k-means objective fΘ ( x ) = Ms ( ‖x− θ1‖22 , . . . , ‖x− θk‖22 ) : upon partial differentiation , we obtain ∇θjfΘ ( x ) = 2 k 1 k k∑ j′=1 ‖x− θj′‖2s2 1/s−1 ‖x− θj‖2 ( s−1 ) 2 ( θj − x ) . As another example , the classic k-means objective fΘ ( x ) = minθ∈Θ ‖x − θ‖22 requires the subgradient∇θjfΘ ( x ) = 2 ( θj−x ) 1 { j ∈ J ( Θ , x ) } , where J ( Θ , x ) = argmin1≤j≤k ‖x−θj‖22 and 1 { · } denotes the indicator function . A pseudocode summary appears in Algorithm 1 . This method tends to find clusters efficiently : for instance , we incur O ( npk ) per-iteration complexity applied to both the k-means and power k-means instances , which matches that of their original algorithms without robustifying via MoM [ 35 , 33 , 56 ] . Of course , here we update θj by an adaptive gradient step rather than by a closed form expression . As emphasized by an anonymous reviewer , one should note that the algorithms under the proposed framework may converge to sub-optimal local solutions under the proposed optimization scheme . As is the case for k-means and its variants , this possibility arises due to the non-convexity of the objective functions ( 1 ) and ( 2 ) . A complete theoretical understanding from an optimization perspective for such methods is not yet fully developed and global results are in general notoriously difficult to obtain . Having said that , it is worth noting that recent empirical analysis show that techniques such as annealing [ 56 , 17 ] may be effective to circumvent this difficulty . Indeed , our experimental analysis ( see Section 4 ) suggests that a robust version of power k-means using this annealing technique best overcomes this difficulty even in the presence of outliers .

Summary: The authors consider the minimization of a Median-of-Means (MoM) objective, which is suitable to yield minimizers of a centroid-based clustering which is more robust to outliers. The type of the centroid-based clustering is defined over the employed measurement of distances, which can be any  Bregman divergence. The considered MoM objective is hence suitable to encompass various $k$-means formalizations.  The authors formulate an Adagrad method to minimize the MoM objective, and analyze properties of the global minimizer of the MoM objective with respect to robustness to outliers. They apply concentration inequalities to compare the MoM global minimizer to the global minimizer of the expected value of the generalized distance of points to their centroid. Since outliers do not influence the expected value, a small distance of the MoM minimizer to the one of the expected value indicates robustness to outliers.   The authors provide an experimental analysis of the power $k$-means objective, optimized by the proposed optimization scheme on synthetic data and one real-world dataset.