Extreme Value k-means Clustering

1 INTRODUCTION . Clustering is a fundamental and important task in the unsupervised learning ( Jain , 2010 ; Rui Xu & Wunsch , 2005 ) . It aims at clustering data samples of high similarity into the same cluster . The most well-known clustering algorithm is the k-means , whose objective is to minimize the sum of squared distances to their closest centroids . k-means has been extensively studied in the literature , and some heuristics have been proposed to approximate it ( Jain , 2010 ; Dubes & Jain , 1988 ) . The most famous one is Lloyd ’ s algorithm ( Lloyd , 1982 ) . The k-means algorithm is widely used due to its simplicity , ease of use , geometric intuition ( Bottesch et al. , 2016 ) . Unfortunately , its bottleneck is that computational complexity reaches O ( nkd ) ( Rui Xu & Wunsch , 2005 ) , since it requires computing the Euclidean distances between all samples and all centroids . The data is embedded in the Euclidean space ( Stemmer & Kaplan , 2018 ) , which causes the failure on clustering non-convex clusters ( Beliakov & King , 2006 ) . Even worse , k-means is highly sensitive to the initial centroids , which usually are randomly initialized . Thus , it is quite possible that the objective of k-means converges to a local minimum , which causes the instability of k-means , and is less desirable in practice . Despite a stable version – k-means++ ( Arthur & Vassilvitskii , 2007 ) gives a more stable initialization , fundamentally it is still non-trivial to extend k-means in clustering data samples of non-convex shape . To solve these problems , this paper improves the clustering ability of k-means by measuring the similarity between samples and centroids by EVT ( Coles et al. , 2001 ) . In particular , we consider the generalized extreme value ( GEV ) ( Jenkinson , 1955 ) distribution or generalized Pareto distribution ( GPD ) ( Pickands III et al. , 1975 ; DuMouchel , 1975 ) to transform the Euclidean space into a probability space defined as , extreme value space . GEV and GPD are employed to model the maximum distance and output the probability that a distance is an extreme value , which indicates the similarity of a sample to a centroid . Further , we adopt the Block Maxima Method ( BMM ) ( Gumbel , 2012 ) to choose the maximal distance for helping GEV fit the data . The Peaks-Over-Thresh ( POT ) method ( Leadbetter , 1991 ) is utilized to model the excess of distance exceeding the threshold , and thus very useful in fitting the data for GPD . Formally , since both GEV and GPD can measure the similarity of samples and centroids , they can be directly utilized in k-means , i.e. , GEV k-means and GPD k-means , which are uniformly called Extreme Value k-means ( EV k-means ) algorithm . In contrast to k-means , EV k-means is a probability-based clustering algorithm that clusters samples according to the probability output from GEV or GPD . Furthermore , to accelerate the computation of Euclidean distance , We expand the samples and the centroids into two tensors of the same shape , and then accelerate with the high performance parallel computing of GPU . For clustering steaming data , we propose online Extreme Value k-means based on Mini Batch kmeans ( Sculley , 2010 ) . When fit the GEV distribution , we use mini batch data as a block . For the fitting of GPD , we dynamically update the threshold . The parameters of GEV or GPD are learned by stochastic gradient descent ( SGD ) ( LeCun et al. , 1998 ) . The main contributions are described as follows . ( 1 ) This paper utilizes EVT to improve k-means in addressing the problem of clustering data of non-convex shape . We thus propose the novel Extreme Value k-means , including GEV k-means and GPD k-means . A method for accelerating Euclidean distance computation has also been proposed to solve the bottleneck of k-means . ( 2 ) Under the strong theoretical support provided by EVT , we use GEV and GPD to transform Euclidean space into extreme value space , and measure the similarity between samples and centroids . ( 3 ) Based on Mini Batch k-means , We propose online Extreme value k-means for clustering streaming data , which can learn the parameters of GEV and GPD online . We corroborate the effectiveness of EV k-means and online EV k-means by conducting experiments on synthetic datasets and real datasets . Experimental results show that EV k-means and online EV k-means significantly outperform compared algorithms consistently across all experimented datasets . 2 RELATED WORKS . k-means and EVT have been extensively studied in the literature in many aspects ( Jain , 2010 ; Rui Xu & Wunsch , 2005 ; De Haan & Ferreira , 2007 ) . Previous work on k-means focused on the following aspects , such as determining the optimal k , initializing the centroids , and accelerating k-means . Bandyopadhyay & Maulik ( 2002 ) ; Lin et al . ( 2005 ) ; Van der Merwe & Engelbrecht ( 2003 ) ; Omran et al . ( 2005 ) propose to select the optimal k value based on the genetic algorithm . Initializing the centroids is a hot issue in k-means ( Celebi et al. , 2013 ) . k-means++ ( Arthur & Vassilvitskii , 2007 ) is the most popular initialization scheme . Katsavounidis et al . ( 1994 ) ; Khan & Ahmad ( 2004 ) ; Redmond & Heneghan ( 2007 ) proposed density-based initial centroid selection method , that is , selecting the initial cluster center according to the density distribution of the samples . Recently , Bachem et al . ( 2016 ) propose using Markov chain Monte Carlo to accelerate k-means++ sampling . There is also a lot of work focused on solving the computational complexity of k-means . Hamerly ( 2010 ) argued that using triangle inequality can accelerate k-means . Sinha ( 2018 ) showed that randomly sparse the original data matrix can significantly speed up the computation of Euclidean distance . EVT is widely used in many area , such as natural phenomena , finance , and traffic prediction . In recent years , EVT has many applications in the field of machine learning . However , far too little attention has been paid to the combination of k-means and EVT . Li et al . ( 2012 ) proposes using generalized extreme value distribution for feature learning based on k-means . However , our method is significant different from this method . First , they compute the squared distance from a point to the nearest centroid and form a GEV regarding to each point , while we compute the distance from a centroid all data points and then fit the GEV or GPD regarding to each centroid . Second , their algorithm adds the likelihood function as a penalty term into the objective function of k-means , but our algorithm changes the objective function by fitting the GEV or GPD for each centroid and assign the data point to the one with the highest probabilities they belong to . Finally , this paper also presents GPD k-means and online Extreme Value k-means , which is not stated in this paper . 3 PRELIMINARIES . 3.1 k-MEANS CLUSTERING ALGORITHM Denote X = { x1 , x2 , . . . , xn } ⊆ Rd as the dataset and C = { C1 , C2 , . . . , Ck } as a partition of X satisfying Ci ∩ Cj = ∅ , i 6= j . Let Θ = { µ1 , µ2 , . . . , µk } with µi ⊆ Rd be the centroid of cluster Ci , i = 1 , 2 , . . . , k , that is , µi = 1|Ci| ∑ x∈Ci x . The sum squared error is defined as J ( C ; Θ ) = k∑ i=1 ∑ x∈Ci ‖x− µi‖22 ( 1 ) Eq . ( 1 ) indicates that the smaller J is , the higher degree of closeness between the samples and their centroids in the clusters , so the similarity of the samples in the clusters is higher . To find the global minimum of Eq . 1 , we need to compute all possible cluster partitions , so k-means is an NP-hard problem ( Aloise et al. , 2009 ) . Lloyd ’ s algorithm ( Lloyd , 1982 ) uses a greedy strategy to approximate the Eq . ( 1 ) by iteratively optimizing between assigning cluster labels and updating centroids . Specifically , in assigning cluster labels , a cluster label is assigned to each sample according to the closest centroid . When the centroids are being updated , each centroid is updated to the mean of all samples in the cluster . These two steps loop iteratively until the centroids no longer change . 3.2 EXTREME VALUE THEORY . In this subsection , we first introduce the statistical aspects of a sample maximum in Extreme Value Theory ( EVT ) , which is a branch of statistics dealing with the stochastic behavior of extreme events found in the tail of probability distribution . LetX1 , X2 , . . . , Xn be a sample of independent copy of X with distribution F . It is theoretically interesting to consider the asymptotic behavior of sample maximum and upper order statistics . More specifically , denote Mn = max1≤i≤nXi as the sample maximum , whose distribution is Pr ( Mn 6 x ) = F n ( x ) , x ∈ R. ( 2 ) On the other hand , the upper order statistics of the sample is related to the survival function over a threshold u , which is Pr ( X > u+ x|X > u ) = Pr ( X > u+ x ) Pr ( X > u ) = 1− F ( u+ x ) 1− F ( u ) , x > 0 . ( 3 ) EVT considers the non-degenerated limit when n→∞ in Eq . ( 2 ) and u ↑ x∗ in Eq . ( 3 ) by re-scaling the objects , which is presented as the conditions of the maximum domain of attraction for F . Theorem 3.1 ( Fisher-Tippett Theorem ( Fisher & Tippett , 1928 ) ) A distribution function F satisfis the condition of a maximum domain of attraction : if there exists a constant ξ ∈ R and sequences an > 0 , bn , n ∈ N such that lim n→∞ Fn ( anx+ bn ) = exp { − ( 1 + ξx ) −1/ξ } , 1 + ξx > 0 . ( 4 ) The shape parameter ξ is called the extreme value index . Theorem 3.1 motivates the Block Maxima Method ( BMM ) ( Gumbel , 2012 ) : for the block size s ∈ { 1 , 2 , . . . , n } , divide the sample into m = xn/sy blocks of length s. Since the data is independent , each block maxima has distribution F s and can be approximated by a three-parametric generalized extreme value distribution ( GEV ) GGEV ( · ; µ , σ , ξ ) when the block size s is large enough and the number of blocks m is sufficient . The class of GEV distributions is defined as GGEV ( x ; µ , σ , ξ ) = exp { − ( 1 + ξ x− µ σ ) −1/ξ } , 1 + ξ x− µ σ > 0 . ( 5 ) We treat the case of ξ = 0 as the limit of ξ → 0 . An equivalent representation of the maximum domain of attraction condition is as follows : Theorem 3.2 ( Pickands-Balkema-de Haan Theorem ( Balkema & De Haan , 1974 ) ) A distribution function F satisfies the condition of maximum domain of attraction : if there exists a constant ξ ∈ R and a positive function σ ( t ) such that lim u↑x∗ 1− F ( u+ σ ( u ) x ) 1− F ( u ) = ( 1 + ξx ) −1/ξ , 1 + ξx > 0 . ( 6 ) where x∗ denotes the right end-point of the support of F . The upper order statistics of a sample usually provides useful information about the tail of the distribution F . Then Theorem 3.2 gives rise to an alternative peak-over-threshold ( POT ) approach ( Pickands III et al. , 1975 ) : given sufficient large threshold u in Eq . ( 7 ) , we have that , for anyXi > u , its conditional distribution can be approximated by a two-parametric generalized Pareto distribution ( GPD ) GGPD ( · ; σ , ξ ) , which is defined as GGPD ( x ; σ , ξ ) = 1− ( 1 + ξ x σ ) −1/ξ , x > 0 . ( 7 ) Similarly , we treat the case of ξ = 0 as the limit of ξ → 0 . The POT approach focuses on the excess over the threshold u to fit the GPD and asymptotically characterize the tail features of the distribution , while the BMM only approximates the GEV distribution when m is large enough . The BMM only uses a very small amount of dataset , and there may be cases where the submaximal value of one block is larger than the maximum value of the other block , which can not be utilized . In contrast , POT method uses all data beyond the threshold to fit the GPD , making full use of the extreme data . However , there is no winner in theory . 4 THE EXTREME VALUE k-MEANS ALGORITHM

Summary: The paper considers extending the k-means algorithm to allow for finding clusters with non-convex shapes. Particularly, it uses an existing theoretical framework (Extreme Value Theory) to maps a Euclidean space into what it calls the extreme value space, and proposes two Extreme Value k-means algorithms: GEV k-means and GPD k-means. It then provides some empirical results demonstrating their approach.