k-Median Clustering via Metric Embedding: Towards Better Initialization with Privacy

1 INTRODUCTION . Clustering is an important problem in unsupervised learning that has been widely used in social network analysis , sensor networks , etc . ( Punj & Stewart , 1983 ; Dhillon & Modha , 2001 ; Banerjee et al. , 2005 ; Abbasi & Younis , 2007 ) . The goal of clustering is to partition a set of data points into clusters such that items in the same cluster are expected to be similar , while items in different clusters should be different . This is concretely measured by the sum of distances ( or squared distances ) between each point to its nearest cluster center . One conventional notion to evaluate a clustering algorithms is : with high probability , cost ( C , D ) ≤ γOPTk ( D ) + ξ , where C is the centers output by the algorithm and cost ( C , D ) is some cost function defined for C on the dataset D. OPTk ( D ) is the cost of optimal ( oracle ) k-median solution on D. When everything is clear from context , we will use OPT for short . Here , γ is called multiplicative error and ξ is called additive error . Alternatively , we may also use the notion of expected cost . Two popularly studied clustering problems are 1 ) the k-median problem , and 2 ) the k-means problem . The origin of k-median dates back to the 1970 ’ s ( e.g. , Kaufman et al . ( 1977 ) ) , where one tries to find the best location of facilities that minimizes the cost measured by the distance between clients and facilities . Formally , given a set of points D and a distance measure , the goal is to find k center points minimizing the sum of absolute distances of each sample point to its nearest center . In k-means , the objective is to minimize the sum of squared distances instead . There are two popular clustering algorithms . One heuristic is the Lloyd ’ s algorithm ( Lloyd , 1982 ) , which is built upon an iterative distortion minimization approach . In most cases , this method can only be applied to numerical data , typically in the Euclidean space . Clustering in general metric spaces ( discrete spaces ) is also important and useful when dealing with , for example , the graph data , where Lloyd ’ s method is no longer applicable . A more broadly applicable approach , the local search method ( Kanungo et al. , 2002 ; Arya et al. , 2004 ) , has also been widely studied . It iteratively finds the optimal swap between the center set and non-center data points to keep lowering the cost . Local search can achieve a constant approximation ratio ( γ = O ( 1 ) ) to the optimal solution for k-median ( Arya et al. , 2004 ) . Initialization of cluster centers . It is well-known that the performance of clustering can be highly sensitive to initialization . If clustering starts with good initial centers ( i.e. , with small approximation error ) , the algorithm may use fewer iterations to find a better solution . The k-median++ algorithm ( Arthur & Vassilvitskii , 2007 ) iteratively selects k data points as initial centers , favoring distant points in a probabilistic way ( see Appendix A for more details ) . Intuitively , the initial centers tend to be well spread over the data points ( i.e. , over different clusters ) . The produced initial center is proved to have O ( log k ) multiplicative error . Followup works of k-means++ further improved its efficiency and scalability , e.g. , Bahmani et al . ( 2012 ) ; Bachem et al . ( 2016 ) ; Lattanzi & Sohler ( 2019 ) . In this work , we propose a new initialization framework based on metric embedding methods , with comparable approximation error and running time as k-median++ . Importantly , our initialization scheme can be conveniently combined with the notion of differential privacy . Clustering with Differential Privacy . The concept of differential privacy ( Dwork , 2006 ; McSherry & Talwar , 2007 ) has been popular to rigorously define and resolve the problem of keeping useful information for model learning , while protecting privacy for each individual . Private k-means problem has been widely studied , e.g. , Feldman et al . ( 2009 ) ; Nock et al . ( 2016 ) ; Feldman et al . ( 2017 ) , mostly in the Euclidean space . The paper ( Balcan et al. , 2017 ) considered identifying a good candidate set ( in a private manner ) of centers before applying private local search , which yieldsO ( log3 n ) multiplicative error and O ( ( k2 + d ) log5 n ) additive error . Later on , the Euclidean k-means errors are further improved to γ = O ( 1 ) and ξ = O ( k1.01 ·d0.51+k1.5 ) by Stemmer & Kaplan ( 2018 ) , with more advanced candidate set selection . Huang & Liu ( 2018 ) gave an optimal algorithm in terms of minimizing Wasserstein distance under some data separability condition . For private k-median clustering , Feldman et al . ( 2009 ) considered the problem in high dimensional Euclidean space . However , it is rather difficult to extend their analysis to more general metrics in discrete spaces ( e.g. , on graphs ) . The strategy of Balcan et al . ( 2017 ) to form a candidate center set could as well be adopted to k-median , which leads to O ( log3/2 n ) multiplicative error and O ( ( k2+d ) log3 n ) additive error in high dimensional Euclidean space . Gupta et al . ( 2010 ) proposed a private method for the classical local search heuristic in discrete space , which applies to both kmedians and k-means . In order to cast privacy on each swapping step , the authors applied the exponential mechanism of McSherry & Talwar ( 2007 ) . Their method produced an -differentially private solution with cost 6OPT +O ( 4k2 log2 n/ ) , where4 is the diameter of point set . The main contributions of this work include : • We introduce Hierarchically Well-Separated Tree ( HST ) , a metric embedding tree structure , to the k-median clustering problem . Once the HST is constructed using the data samples , we provide an efficient sampling strategy to select the initial center set , with an approximation factor O ( log min { k,4 } ) in the non-private setting , which is O ( log min { k , d } ) when 4 = O ( d ) ( e.g. , bounded data ) . This improves the O ( log k ) error of k-means/median++ in e.g. , the lower dimensional Euclidean space . • The main strength of our HST initialization method is that it could be effectively adapted to differential privacy ( DP ) . The so-called DP-HST algorithm is -DP and outputs initial centers withO ( log n ) multiplicative error andO ( −14k2 log2 n ) additive error . Moreover , running DP-local search starting from this initialization givesO ( 1 ) multiplicative error and O ( −14k2 ( log log n ) log n ) additive error , which improves previous results towards the well-known lower bound O ( −14k log ( n/k ) ) on the additive error ( Gupta et al. , 2010 ) within a small O ( k log log n ) factor . To our knowledge , this is the first center initialization method with differential privacy guarantee and improved error rate in general metrics . • We conduct experiments on simulated and real-world datasets to demonstrate the effectiveness of our methods . In both non-private and private settings , our proposed HST-based approach achieves smaller cost at initialization than k-median++ , which may also lead to improvements in the final clustering quality . Our private algorithm can also save computational costs by reaching a good solution with fewer iterations . 2 PRELIMINARIES . Definition 2.1 ( Differential Privacy ( DP ) ( Dwork , 2006 ) ) . If for any two adjacent data sets D and D′ with symmetric difference of size one , for any O ⊂ Range ( A ) , an algorithmA satisfies Pr [ A ( D ) ∈ O ] ≤ e Pr [ A ( D′ ) ∈ O ] , then algorithmA is said to be -differentially private . Intuitively , differential privacy requires that after removing any observation , the output of D′ should not be too different from that of the original dataset D. Smaller indicates stronger privacy , which , however , usually sacrifices utility . Thus , one of the central topics in differential privacy literature is to balance the utility-privacy trade-off . To achieve DP , it is common to introduce noise to the data or the algorithm output . The Laplace mechanism adds Laplace ( η ( f ) / ) noise to the output , which is known to achieve -DP . The exponential mechanism is also a tool for many DP algorithms . Let O be the set of feasible outputs . The utility function q : D × O → R is what we aim to maximize . The exponential mechanism outputs an element o ∈ O with probability P [ A ( D ) = o ] ∝ exp ( q ( D , o ) 2η ( q ) ) , where D is the input dataset and η ( f ) = sup|D−D′|=1 |f ( D ) − f ( D′ ) | is the sensitivity of f . Both mechanisms will be used in the design of our proposed DP approach . k-Median Clustering . Following the framework of Arya et al . ( 2004 ) ; Gupta et al . ( 2010 ) , the problem of k-median clustering ( DP and non-DP ) studied in our paper is stated as below . Definition 2.2 ( k-median ) . Given a universe point set U and a metric ρ : U × U → R , let D ⊆ U be a set of demand points . The goal of DP k-median is to pick F ⊆ U with |F | = k to minimize Private k-median : cost ( F , D ) = ∑ v∈D min f∈F ρ ( v , f ) . ( 1 ) At the same time , the output F is required to be -differentially private to D. For standard nonprivate k-median , we assume U = D and minimize k-median : cost ( F , U ) = ∑ v∈U min f∈F ρ ( v , f ) . ( 2 ) We may drop “ F ” and use “ cost ( D ) ” or “ cost ( U ) ” if there is no risk of ambiguity . To better understand the DP clustering problem , we provide an real-world example as follows . Example 2.1 . Consider U to be the universe of all users in a social network ( e.g. , Twitter ) . Each user ( account ) is public , but also has some private information that can only be seen by the data holder . Let D be users grouped by some information that might be set as private . Suppose a third party plans to collaborate with the most influential users in D for e.g. , commercial purposes , thus requesting the cluster centers of D. In this case , we need a strategy to safely release the centers , while protecting the individuals in D from being identified ( since the membership of D is private ) . The local search procedure for k-median proposed by Arya et al . ( 2004 ) is summarized in Algorithm 1 . First we randomly pick k points in U as the initial centers . In each iteration , we search over all x ∈ F and y ∈ U , and do the swap F ← F − { x } + { y } such that F − { x } + { y } improves the cost of F the most ( if more than factor ( 1−α/k ) where α > 0 is a hyper-parameter ) . We repeat the procedure until no such swap exists . Arya et al . ( 2004 ) showed that the output centers F achieves 5 approximation error to the optimal solution , i.e. , cost ( F ) ≤ 5OPT . Algorithm 1 : Local search for k-median clustering ( Arya et al. , 2004 ) Input : Data points U , parameter k , constant α Initialization : Randomly select k points from U as initial center set F while ∃ x ∈ F , y ∈ U s.t . cost ( F − { x } + { y } ) ≤ ( 1− α/k ) cost ( F ) do Select ( x , y ) ∈ Fi × ( D \ Fi ) with arg minx , y { cost ( F − { x } + { y } ) } Swap operation : F ← F − { x } + { y } Output : Center set F

Summary: This paper considers the problem of finding good initial centers for the fundamental problem of $k$-median clustering using a randomized embedding of the original metric into a tree metric.  After setting the initial centers, a standard local search algorithm is applied to produce an improved solution.  This is explored in both the standard context of $k$-median clustering, as well as in the relevant context of differentially private clustering.  In the latter setting, the goal is to minimize the amount of additive error introduced by the algorithm subject to being $\epsilon$-differentially private.  An extension to $k$-means is given in the appendix.  In the standard setting of $k$-median clustering, the main theoretical result is an initialization algorithm which is an $O(\log(\min(\Delta,k)))$-approximation to the optimal k-median clustering.  This is an improvement over k-median++ (which gives $O(\log k)$) when $\Delta$ is small, e.g. for $\Delta = O(d)$ and $d$ is small.  Using this as a seed results for a local search method results in an $O(1)$-approximation overall.  At a high level, their algorithm first constructs an embedding of the original metric into a hierarchically well-separated tree (HST).  From there, the initialization can be seen as finding an $O(1)$-approximate solution on the HST efficiently.  The overall guarantee follows from standard results about HST's  In  the differentially private setting, the main result is a similar guarantee on the quality of the initial solution and also a bound on the quality of the final solution when using a known private local search algorithm.  The quality of the final solution has $O(1)$-multiplicative error and $O(\epsilon^{-1}k^2\Delta\log(n)\log\log(n))$ additive error.  This is an improvement over the additive error of $O(\epsilon^{-1}k^2\Delta\log^2(n)$ due to Gupta et al. 2010.  The number of local search iterations is also improved from $O(k\log n)$ to $O(k\log\log n)$.  The main idea for the initialization is similar to the standard setting, but here they use the structure of the HST to ensure the initial solution is private by injecting a different amount of noise at each level of the tree.  An empirical study is done on a class of synthetic graphs as well as the MNIST dataset.  For the synthetic graphs, the metric space is given by the weighted shortest path distance in each graph, while for the MNIST dataset the metric is given by either $\ell_1$ or $\ell_2$.  The authors compare both the initial costs and the final costs (after running a local search method) for several initialization methods in both the standard and differentially private settings.  The main observation is that the proposed initialization methods tend to have better initial cost and the proposed differentially private method often outperforms the other methods in both initial cost and final cost.