k-Median Clustering via Metric Embedding: Towards Better Initialization with Privacy

1 INTRODUCTION . Clustering is an important problem in unsupervised learning that has been widely used in social network analysis , sensor networks , etc . ( Punj & Stewart , 1983 ; Dhillon & Modha , 2001 ; Banerjee et al. , 2005 ; Abbasi & Younis , 2007 ) . The goal of clustering is to partition a set of data points into clusters such that items in the same cluster are expected to be similar , while items in different clusters should be different . This is concretely measured by the sum of distances ( or squared distances ) between each point to its nearest cluster center . One conventional notion to evaluate a clustering algorithms is : with high probability , cost ( C , D ) ≤ γOPTk ( D ) + ξ , where C is the centers output by the algorithm and cost ( C , D ) is some cost function defined for C on the dataset D. OPTk ( D ) is the cost of optimal ( oracle ) k-median solution on D. When everything is clear from context , we will use OPT for short . Here , γ is called multiplicative error and ξ is called additive error . Alternatively , we may also use the notion of expected cost . Two popularly studied clustering problems are 1 ) the k-median problem , and 2 ) the k-means problem . The origin of k-median dates back to the 1970 ’ s ( e.g. , Kaufman et al . ( 1977 ) ) , where one tries to find the best location of facilities that minimizes the cost measured by the distance between clients and facilities . Formally , given a set of points D and a distance measure , the goal is to find k center points minimizing the sum of absolute distances of each sample point to its nearest center . In k-means , the objective is to minimize the sum of squared distances instead . There are two popular clustering algorithms . One heuristic is the Lloyd ’ s algorithm ( Lloyd , 1982 ) , which is built upon an iterative distortion minimization approach . In most cases , this method can only be applied to numerical data , typically in the Euclidean space . Clustering in general metric spaces ( discrete spaces ) is also important and useful when dealing with , for example , the graph data , where Lloyd ’ s method is no longer applicable . A more broadly applicable approach , the local search method ( Kanungo et al. , 2002 ; Arya et al. , 2004 ) , has also been widely studied . It iteratively finds the optimal swap between the center set and non-center data points to keep lowering the cost . Local search can achieve a constant approximation ratio ( γ = O ( 1 ) ) to the optimal solution for k-median ( Arya et al. , 2004 ) . Initialization of cluster centers . It is well-known that the performance of clustering can be highly sensitive to initialization . If clustering starts with good initial centers ( i.e. , with small approximation error ) , the algorithm may use fewer iterations to find a better solution . The k-median++ algorithm ( Arthur & Vassilvitskii , 2007 ) iteratively selects k data points as initial centers , favoring distant points in a probabilistic way ( see Appendix A for more details ) . Intuitively , the initial centers tend to be well spread over the data points ( i.e. , over different clusters ) . The produced initial center is proved to have O ( log k ) multiplicative error . Followup works of k-means++ further improved its efficiency and scalability , e.g. , Bahmani et al . ( 2012 ) ; Bachem et al . ( 2016 ) ; Lattanzi & Sohler ( 2019 ) . In this work , we propose a new initialization framework based on metric embedding methods , with comparable approximation error and running time as k-median++ . Importantly , our initialization scheme can be conveniently combined with the notion of differential privacy . Clustering with Differential Privacy . The concept of differential privacy ( Dwork , 2006 ; McSherry & Talwar , 2007 ) has been popular to rigorously define and resolve the problem of keeping useful information for model learning , while protecting privacy for each individual . Private k-means problem has been widely studied , e.g. , Feldman et al . ( 2009 ) ; Nock et al . ( 2016 ) ; Feldman et al . ( 2017 ) , mostly in the Euclidean space . The paper ( Balcan et al. , 2017 ) considered identifying a good candidate set ( in a private manner ) of centers before applying private local search , which yieldsO ( log3 n ) multiplicative error and O ( ( k2 + d ) log5 n ) additive error . Later on , the Euclidean k-means errors are further improved to γ = O ( 1 ) and ξ = O ( k1.01 ·d0.51+k1.5 ) by Stemmer & Kaplan ( 2018 ) , with more advanced candidate set selection . Huang & Liu ( 2018 ) gave an optimal algorithm in terms of minimizing Wasserstein distance under some data separability condition . For private k-median clustering , Feldman et al . ( 2009 ) considered the problem in high dimensional Euclidean space . However , it is rather difficult to extend their analysis to more general metrics in discrete spaces ( e.g. , on graphs ) . The strategy of Balcan et al . ( 2017 ) to form a candidate center set could as well be adopted to k-median , which leads to O ( log3/2 n ) multiplicative error and O ( ( k2+d ) log3 n ) additive error in high dimensional Euclidean space . Gupta et al . ( 2010 ) proposed a private method for the classical local search heuristic in discrete space , which applies to both kmedians and k-means . In order to cast privacy on each swapping step , the authors applied the exponential mechanism of McSherry & Talwar ( 2007 ) . Their method produced an -differentially private solution with cost 6OPT +O ( 4k2 log2 n/ ) , where4 is the diameter of point set . The main contributions of this work include : • We introduce Hierarchically Well-Separated Tree ( HST ) , a metric embedding tree structure , to the k-median clustering problem . Once the HST is constructed using the data samples , we provide an efficient sampling strategy to select the initial center set , with an approximation factor O ( log min { k,4 } ) in the non-private setting , which is O ( log min { k , d } ) when 4 = O ( d ) ( e.g. , bounded data ) . This improves the O ( log k ) error of k-means/median++ in e.g. , the lower dimensional Euclidean space . • The main strength of our HST initialization method is that it could be effectively adapted to differential privacy ( DP ) . The so-called DP-HST algorithm is -DP and outputs initial centers withO ( log n ) multiplicative error andO ( −14k2 log2 n ) additive error . Moreover , running DP-local search starting from this initialization givesO ( 1 ) multiplicative error and O ( −14k2 ( log log n ) log n ) additive error , which improves previous results towards the well-known lower bound O ( −14k log ( n/k ) ) on the additive error ( Gupta et al. , 2010 ) within a small O ( k log log n ) factor . To our knowledge , this is the first center initialization method with differential privacy guarantee and improved error rate in general metrics . • We conduct experiments on simulated and real-world datasets to demonstrate the effectiveness of our methods . In both non-private and private settings , our proposed HST-based approach achieves smaller cost at initialization than k-median++ , which may also lead to improvements in the final clustering quality . Our private algorithm can also save computational costs by reaching a good solution with fewer iterations . 2 PRELIMINARIES . Definition 2.1 ( Differential Privacy ( DP ) ( Dwork , 2006 ) ) . If for any two adjacent data sets D and D′ with symmetric difference of size one , for any O ⊂ Range ( A ) , an algorithmA satisfies Pr [ A ( D ) ∈ O ] ≤ e Pr [ A ( D′ ) ∈ O ] , then algorithmA is said to be -differentially private . Intuitively , differential privacy requires that after removing any observation , the output of D′ should not be too different from that of the original dataset D. Smaller indicates stronger privacy , which , however , usually sacrifices utility . Thus , one of the central topics in differential privacy literature is to balance the utility-privacy trade-off . To achieve DP , it is common to introduce noise to the data or the algorithm output . The Laplace mechanism adds Laplace ( η ( f ) / ) noise to the output , which is known to achieve -DP . The exponential mechanism is also a tool for many DP algorithms . Let O be the set of feasible outputs . The utility function q : D × O → R is what we aim to maximize . The exponential mechanism outputs an element o ∈ O with probability P [ A ( D ) = o ] ∝ exp ( q ( D , o ) 2η ( q ) ) , where D is the input dataset and η ( f ) = sup|D−D′|=1 |f ( D ) − f ( D′ ) | is the sensitivity of f . Both mechanisms will be used in the design of our proposed DP approach . k-Median Clustering . Following the framework of Arya et al . ( 2004 ) ; Gupta et al . ( 2010 ) , the problem of k-median clustering ( DP and non-DP ) studied in our paper is stated as below . Definition 2.2 ( k-median ) . Given a universe point set U and a metric ρ : U × U → R , let D ⊆ U be a set of demand points . The goal of DP k-median is to pick F ⊆ U with |F | = k to minimize Private k-median : cost ( F , D ) = ∑ v∈D min f∈F ρ ( v , f ) . ( 1 ) At the same time , the output F is required to be -differentially private to D. For standard nonprivate k-median , we assume U = D and minimize k-median : cost ( F , U ) = ∑ v∈U min f∈F ρ ( v , f ) . ( 2 ) We may drop “ F ” and use “ cost ( D ) ” or “ cost ( U ) ” if there is no risk of ambiguity . To better understand the DP clustering problem , we provide an real-world example as follows . Example 2.1 . Consider U to be the universe of all users in a social network ( e.g. , Twitter ) . Each user ( account ) is public , but also has some private information that can only be seen by the data holder . Let D be users grouped by some information that might be set as private . Suppose a third party plans to collaborate with the most influential users in D for e.g. , commercial purposes , thus requesting the cluster centers of D. In this case , we need a strategy to safely release the centers , while protecting the individuals in D from being identified ( since the membership of D is private ) . The local search procedure for k-median proposed by Arya et al . ( 2004 ) is summarized in Algorithm 1 . First we randomly pick k points in U as the initial centers . In each iteration , we search over all x ∈ F and y ∈ U , and do the swap F ← F − { x } + { y } such that F − { x } + { y } improves the cost of F the most ( if more than factor ( 1−α/k ) where α > 0 is a hyper-parameter ) . We repeat the procedure until no such swap exists . Arya et al . ( 2004 ) showed that the output centers F achieves 5 approximation error to the optimal solution , i.e. , cost ( F ) ≤ 5OPT . Algorithm 1 : Local search for k-median clustering ( Arya et al. , 2004 ) Input : Data points U , parameter k , constant α Initialization : Randomly select k points from U as initial center set F while ∃ x ∈ F , y ∈ U s.t . cost ( F − { x } + { y } ) ≤ ( 1− α/k ) cost ( F ) do Select ( x , y ) ∈ Fi × ( D \ Fi ) with arg minx , y { cost ( F − { x } + { y } ) } Swap operation : F ← F − { x } + { y } Output : Center set F

Summary: The paper suggests an algorithm for the metric k-median problem using ideas from Metric embedding theory. The suggested use of the algorithm is as an initialization routine for the local search based algorithm for k-median. The differentially private version of the algorithm is also given along with bounds on k-median approximation factor. Experiments are conducted over datasets such as MNIST and results compared against the k-means++ algorithm (a popular initialisation algorithm).