Parallel and Efficient Hierarchical k-Median Clustering

1 Introduction . Clustering is a central tool in any large scale machine learning library . The goal of clustering is to group objects into subsets so that similar objects are in the same groups while dissimilar objects are in different groups . There are many different definitions of clustering depending on the description of the objects , the similarity measure between objects , as well as the application . While it is impossible to find the perfect formulation for a clustering problem ( [ 29 ] , see also [ 13 ] ) , the metric version of the problem has received a lot of attention over the last decades and is a central topic in unsupervised learning research . Here we focus on one fundamental metric clustering problem : the hierarchical Euclidean k-median problem . In this formulation , the input objects are described by vectors and the distance ( dissimilarity ) between objects is measured by their Euclidean distance . More formally , given a set P ⊆ Rd , the objective of the hierarchical k-median problem in its classic formulation [ 33 ] is to return an ordering ∗Equal contribution †Work was partially done while author was visiting researcher at Google Research , Switzerland . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . c1 , . . . , cn of points and a set of n nested partitions P1 , . . . , Pn of P ( i.e. , for any 1 ≤ i ≤ n , the ( i− 1 ) -partition is obtained by merging two clusters in the i-partition ) such that the following objective is minimized : the maximum over all 1 ≤ k ≤ n of the ratio of ∑k i=1 ∑ p∈Ci ‖p − ci‖2 to the optimum unconstrained k-median cost , where Ck = { c1 , . . . , ck } is the set of centers and Pk = { C1 , . . . , Ck } a k-partition . In this paper , we will provide a slightly different guarantee : instead of minimizing the maximum ratio , we show that for every fixed k this ratio isO ( min { d , log n } log ∆ ) in expectation . Our results extend to a slightly weaker bound that holds simultanuously for all k. We also remark that the above problem formulation implies a similar bound for the variant of the problem where we do not require an ordering of the centers but instead choose the optimal center for each cluster of the partition . Hierarchical clustering , thanks to its ability in explaining the nested structures in real world data , has been extensively studied in the computer science literature [ 27 , 26 , 22 , 43 , 9 , 14 , 41 , 44 , 5 ] . The main focus of previous work has been on sequential solutions for the problem and so they are difficult to apply on large data sets . For this reason , several papers recently proposed scalable hierarchical clustering algorithms [ 27 , 26 , 22 , 43 , 5 , 39 , 34 ] . Nevertheless most prior work has focused only on scaling the single-linkage algorithm ( in fact , efficient MapReduce and Spark algorithms are known for this problem [ 27 , 26 , 5 , 46 ] ) for which no objective function is clearly specified . Another closely related area of research focuses on the k-median and k-means “ flat ” clustering problems in the distributed setting where several important results are known [ 3 , 6 , 18 , 35 , 1 , 10 , 23 ] . The main idea behind these algorithms is to compute in a distributed way some form of small summary of the data ( such as coresets ) . However , all algorithms based on this framework use at least k + n memory per machine ( often significantly more ) to solve the problem in O ( 1 ) rounds . Unfortunately , this is too much in our setting where we want to solve the problem also for k ∈ Θ ( n ) ( in fact to solve the hierarchical clustering problem we need to compute a solution even for k = n − 1 ) . To overcome this problem , in a recent related work [ 31 ] Lattanzi et al . provide a parallel algorithm for the hierarchical Euclidean k-median problem by dividing the data in two clusters iteratively . Unfortunately , this approach does not provide any theoretical guarantees for the problem ( in the experimental section we actually show that it may return low quality solutions ) and its round complexity is O ( log n ) . Our Results . In this work we design the first distributed algorithm for the hierarchical Euclidean k-median problem with provable guarantees . In particular , our algorithm returns an approximate solution to the problem using memory s ∈ Ω ( log2 ( ∆ + n+ d ) ) , O ( logs ( nd log ( n+ ∆ ) ) ) parallel rounds , and O ( nd/s ) machines , where ∆ is the ratio between the maximum and minimum distance of two points . Note that this implies that our algorithm runs in a constant number of rounds if s ∈ nδ for any constant δ > 0 . Interestingly , our algorithm is easily implementable in standard parallel frameworks such as MapReduce [ 16 ] , Hadoop [ 45 ] , Spark [ 47 ] and Dryad [ 25 ] etc . and we analyze it in the standard Massive Parallel Computation ( a. k. a. MPC ) model [ 28 , 21 , 7 ] . Furthermore , we complement our theoretical analysis with an experimental study in distributed setting where we show that our algorithm is significantly faster than parallel hierarchical [ 31 ] and flat solutions [ 3 , 10 , 18 ] for the k-median problem . Properties , extensions and limitations . Work efficiency . An interesting aspect of our algorithm is that the total running time of our algorithm across all machines is almost linear . In particular , the total running time of our algorithm is Õ ( nd ) so it has almost optimal work efficiency3 . To the best of our knowledge , our algorithm is also the first approximation algorithm having almost linear running time for the high-dimensional Euclidean k-median problem ( at the expense of having an expected O ( min { d , log k } log ∆ ) -approximation factor ) .4 . Round Lower Bound . We note here that Theorem 4 in [ 8 ] can be easily extended to the Euclidean k-median problem5 and so to the hierarchical Euclidean k-median problem . Thus , no distributed 3In this section , for the sake of simplicity , we use Õ notation to hide both logn and log ∆ factors . The exact term are provided in the following sections . 4We remark that for general metric spaces there is a lower bound of Ω ( nk ) total work to achieve a constant approximation guarantee for our problem [ 37 ] . Furthermore , we note that finding a constant-factor approximation algorithm for the Euclidean k-median problem in almost linear time would imply an asymptotically better approximation algorithm for the closest pair problem which is a long standing open problem in algorithm design . For low-dimensional Euclidean space , near-linear time algorithms are known [ 11 , 12 , 30 ] 5The exact same construction and proof work also for the Euclidean k-median problem . algorithm with memory s can output any approximate solution in less than logs n rounds . So our memory-round complexity trade-off is asymptotically optimal . Simplicity and Efficiency . Our algorithm is extremely simple to implement . In fact , our results show that it is possible to approximate the hierarchical Euclidean k-median problem just by using a sorting algorithm and few aggregation operations . For the same reason , our algorithm is also very parallelizable and efficient . Practicality . Besides our theoretical guarantees our algorithm is also very efficient in practice . We run distributed experiments and we show that the algorithm is more accurate and efficient than previous parallel algorithms . Furthermore , for k = 10 our algorithm already outperforms the classic distributed algorithms for flat k-median6 [ 3 , 10 , 18 ] by a factor of 3 . For larger k the gap becomes even larger ( for example for k = 1000 our algorithm is an order of magnitude faster ) 7 . The large k setting . In a recent paper [ 8 ] the authors showed for the first time that it is possible to compute an approximate solution for the Euclidean k-means problem using parallel memory significantly smaller than k. In particular they provide a bi-criteria algorithm that approximates the solution of the k-means problem within a factor O ( ( log n log log n ) 2 ) using O ( k log k log n ) centers , memory-per-machine s ∈ Ω ( d log n ) , and O ( logs n ) parallel rounds . Our algorithm proves that this is also possible for the k-median problem . Furthermore our algorithm returns a solution that is not bi-criteria solving an open question in [ 8 ] . Limitations . Our result has two main limitations . First , it relies on the existence of a good tree embedding that can be computed in a few MPC rounds . As we show in this paper , it is indeed possible to compute a tree embedding of a d-dimensional Euclidean point set with expected distortion O ( d log ∆ ) in a constant number of rounds . While there exists a sequential O ( log n ) -distortion tree embedding construction of point sets of size n of arbitrary metric spaces [ 20 ] , it remains an interesting open problem as to how this could be implemented in a constant number of rounds . Second , the expected approximation factor of our algorithm is O ( min { d , log n } log ∆ ) and the expectation is for every fixed k. Nevertheless , in our experimental analysis , we show that in practice the quality of our solution is comparable to state-of-the-art algorithms while being much faster . For additional related work we refer the reader to the full version . 2 Preliminaries . For two points p , q we use ‖p− q‖ to denote their Euclidean distance . For a point p and set C ⊆ Rd we define DIST ( p , C ) = minc∈C ‖p− c‖ , i.e. , the distance from p to its closest point in C. For two partitions P1 , P2 we say that P1 is nested in P2 if P2 can be obtained from P1 by merging two or more parts of P1 . Given a point set P ⊆ Rd , a set of centers C = { c1 , . . . , ck } , and a partition P of P into k parts { P1 , . . . , Pk } , its k-median cost is COST ( P , C , P ) = ∑k i=1 ∑ p∈Pi DIST ( p , ci ) . The points in C are called the cluster centers . We let OPT ( P , k ) = minC⊆Rd , |C|=k minP , |P|=k COST ( P , C , P ) denote the cost of an optimal k-median solution . In this paper , we focus on a slight variation of the classic Hierarchical Euclidean k-median problem8 , which for a given point set P ⊆ Rd asks to find an ( ordered ) sequence of centers C = { c1 , . . . , cn } together with a collection of n nested partitions Π = { P1 , . . . , Pn } of P , such that Pi partition the entire space and contains i parts . We say that an algorithm for the Hierarchical Euclidean k-median problem has an approximation guarantee of α if , for every k = 1 , . . . , n , the output C , Π defines an α-approximate solution for k-median . That is , for every k ∈ { 1 , 2 , . . . , n } COST ( P , { c1 , . . . , ck } , Pk ) ≤ α · OPT ( P , k ) . Similarly , if the algorithm is randomized then we say that it has an approximation guarantee of α if the guarantee holds in expectation over the random hierarchical clustering C , Π output by the algorithm : for every k ∈ { 1 , 2 , . . . , n } , E [ COST ( P , { c1 , . . . , ck } , Pk ) ] ≤ α · OPT ( P , k ) . 6Note that flat clustering is a sub-problem of hierarchical Euclidean k-median problem , in fact our algorithm solve the problem for all k at the same time 7Note that the large k has many practical applications in spam and abuse [ 38 , 42 ] , near-duplicate detection [ 24 ] , compression or reconciliation tasks [ 40 ] 8We note here that our results naturally extend to the Prefix Euclidean k-median problem [ 36 ] , where clusters are not required to be nested . For simplicity of presentation , here we assume that P ⊆ { 0 , . . . , ∆ } d and then we explain how to remove this assumption in our theoretical analysis in Appendix . The k-median problem can also be formulated for discrete metric spaces M = ( P , DISTM ) , where P is a finite set of points and DISTM is a metric . For a subset C ⊆ P of the points , we also let DISTM ( p , C ) = minc∈C DISTM ( p , c ) be the distance from the point p to its closest point in C. The objective of the metric k-median problem is then to find a subset C ⊆ P of size k such that COSTM ( P , C ) = ∑ p∈P DISTM ( p , C ) is minimized . The hierarchical k-median problem is analogously defined . A ( discrete ) metric space ( P , DISTT ) is called a tree metric , if there exists a positively weighted tree T = ( P , E , w ) such that for all pairs p , q ∈ P we have that DISTT ( p , q ) equals the shortest path distance between p and q in T . We will be mostly interested in tree metrics that are defined by restricted hierarchically well separated trees . 9 Definition 2.1 A restricted ` -hierarchically well separated tree ( RHST ) is a positively weighted rooted tree such that all the leafs are at the same level , all edges at the same level have the same weight , and the length of the edges decreases by a factor of ` on every root to leaf path . Throughout the paper , we consider RHSTs with ` = 2 , i.e. , 2-RHSTs . We therefore simplify notation and sometimes write RHST for 2-RHST . In the next section , we describe how to embed our point set P in Euclidean space into the leaf of a RHST . Formally , a metric embedding between two metric spaces ( P , DIST ) , ( P ′ , DIST′ ) is an injective mapping f : P → P ′ . To simplify notation and to be consistent with this definition ( and the definition of a metric ) we consider instances that contain no two points within distance 0 . However , all our arguments and algorithms generalize to instances that may have several identical points . MPC model . We design algorithms for the MPC model [ 28 , 21 , 7 ] that is considered de-facto the standard theoretical model for large-scale parallel computing . Computation in MPC proceeds in synchronous parallel rounds over multiple machines . Each machine has memory s. At the beginning of a computation , data is partitioned across the machines . During each round , machines process data locally . At the end of a round , machines exchange messages with a restriction that each machine is allowed to send messages and also receive messages of total size s. The efficiency of an algorithm in this model is measured by the number of rounds it takes for the algorithm to terminate and by the size of the memory of every machine . In this paper we focus on the most challenging and practical regime of small memory . In particular , we only assume to have s ∈ Ω ( log2 ( n+ ∆ + d ) ) .

Summary: This paper studies the hierarchical k-medians clustering problem and develops a new parallel algorithm. The algorithm is based on building a tree embedding of the dataset parallelly first and then the partitions of the dataset are computing using this tree. A rigorous theoretical analysis is provided for the communication complexity, memory, and the approximation guarantees of the algorithm for the k-medians problem for all k. The provided experiments complement the theoretical results.