Median DC for Sign Recovery: Privacy can be Achieved by Deterministic Algorithms

1 INTRODUCTION . With the development of technology for data acquisition and storage , the modern dataset has a larger scale , more complex structure , and more practical considerations , which addresses new challenges for data analysis . In recent years , large quantities of sensitive data are collected by individuals and companies . While one wants to extract more accurate statistical information from the distributed dataset , we must also beware of the leakage of this sensitive personal information during the training process . This calls for the study of distributed learning under privacy constraints ( Pathak et al. , 2010 ; Hamm et al. , 2016 ; Jayaraman et al. , 2018 ) . In privacy literature , differential privacy , which was firstly proposed in Dwork et al . ( 2006 ) , has been the most widely adopted definition of privacy tailored to statistical data analysis . It has achieved tremendous success in real-world applications . Denote the data universe to be X , for the dataset X = { Xi } ni=1 ∈ Xn where Xi ’ s are the data observations . The ( , δ ) -differentially privacy can be defined as follows . Definition 1 . ( Differential Privacy in Dwork et al . ( 2006 ) ) A randomized algorithm A : Xn → Θ gives ( , δ ) -differentially private if for any pair of adjacent datasets X ∈ Xn and X′ ∈ Xn , there always holds P { A ( X1 : n ) ∈ U } ≤ e · P { A ( X′1 : n ) ∈ U } + δ , ( 1 ) for every subset U ⊆ Θ . Here two datasets X and X′ are adjacent if and only if the Hamming distance ( Lei , 2011 ) of these two datasets of same size H ( X , X′ ) = 1 . As we can see , the quantities and δ measure the level of privacy loss . There are also several relaxations of differential privacy ( see , e.g. , Bun & Steinke ( 2016 ) ; Dwork & Rothblum ( 2016 ) ; Mironov ( 2017 ) ; Dong et al . ( 2019 ) ) designed for the ease of analysis . However , in these definitions , the dataset X is always assumed to be fixed , and the probability in ( 1 ) only takes over the randomness of the algorithm A . Therefore , it is impossible to achieve strict differential privacy without adding auxiliary perturbations in the algorithm . On the other hand , the statistical performance of the output is inevitably deteriorated by the additional randomness . This lead to a large body of works discussing the tradeoff between accuracy and privacy ( Wasserman & Zhou , 2010 ; Bassily et al. , 2014 ; Bun et al. , 2018 ; Duchi et al. , 2018 ; Cai et al. , 2019 ) . In this paper , we consider the private sign recovery problem in the distributed system . To be more precise , assume the parameter of interest is a sparse vector , which has many zeros in its entries . The task is to identify the signs of the parameter from the observations stored in multiple machines while protecting each individual ’ s privacy . The sign recovery problem , as an extension of sparsity pattern recovery , has found its significance in a broad variety of contexts , including variable selection ( Tibshirani , 1996 ; Miller , 2002 ) , graphical models ( Meinshausen & Bühlmann , 2006 ; Cai et al. , 2011 ) , compressed sensing ( Candes & Tao , 2005 ; Donoho , 2006 ) , and signal denoising ( Chen et al. , 2001 ) . However , this problem is rarely considered in the privacy community . To address the sign recovery problem , we propose the Median Divide-and-Conquer ( Med-DC ) method , a simple two-step procedure . Firstly , each local machine estimates the sparse parameter and sends the sign-vectors back to the server ; Secondly , the server aggregates these sign-vectors using coordinate-wise median and output the final sign estimator . While mean based divide-andconquer ( also referred to as Mean-DC ) approaches have been widely analyzed in distributed learning literature ( Mcdonald et al. , 2009 ; Zhang et al. , 2013 ; Lee et al. , 2017 ; Battey et al. , 2018 ) , the median-based counterpart has not yet been well explored . It is well-known that naively averaging the local estimators behaves badly for nonlinear and penalized optimization problems . This is because averaging can not reduce the bias in the local sub-problems . In particular , for the distributed Lasso problem , as mentioned in Lee et al . ( 2017 ) , the estimation error of averaged local Lasso estimator is of the same order as that of local estimators . However , when only considering the sign recovery problem , we found that the Med-DC method perfectly fits the nature of the distributed private setup . ( See Section 2.2 for more detailed discussions ) For the sake of clarity , we only consider the sign recovery problem for sparse mean estimation and sparse linear regression , the two fundamental models in statistics . The proposed Med-DC method has the following advantages : • Consistent recovery . For both sparse mean estimation and sparse linear regression , the Med-DC method consistently recovers the signs of the true parameter with theoretical guarantees . Under some constraints , we can prove that our approach can identify signals larger thanC √ log n/N for some constant C > 0 ( where N is the full sample size and n is the local sample size ) , which coincides with the minimal signal level in the single machine setting ( all data stored in one machine ) . • Efficient communication . To recover the signs of the parameter of interest in the distributed setup , a naive approach is to estimate the parameter using existing private distributed estimation methods and take the signs of the estimators . However , these methods usually involve multi-round aggregation of gradient information or local estimators , which seems costly for the simple sign recovery problem . Instead , our approach only aggregates the vectors of signs ( bits information ) in one shot , which is much more communicationally efficient . •Weak privacy . By relaxing the differential privacy to high-probability sense , our deterministic Med-DC method can be proved to be weakly ‘ private ’ . We also extend this concept to group privacy . To the best of our knowledge , this is the first deterministic algorithm that has a provable high-probability privacy guarantee . Moreover , as each machine only needs to transmit the vectors of signs , instead of the local estimators or gradient vectors , our proposed method also protects the privacy of each local machine , since gradient sharing can also result in privacy leakage ( Zhu et al. , 2019 ) . •Wide applicability . We believe the Med-DC approach deserves more attention due to its excellent practical performance and ease of implementation . For example , it is promising to apply the Med-DC method to wider classes of models , ( e.g . , generalized linear model , M -estimation , etc . ) or hybridize this method with many sophisticated distributed algorithms like averaged de-biased estimator in Lee et al . ( 2017 ) and Communication-Efficient Accurate Statistical Estimator ( CEASE ) in Fan et al . ( 2019 ) . Notations . For every vector v = ( v1 , ... , vp ) T , denote |v|2 = √∑p l=1 v 2 l , |v|1 = ∑p l=1 |vl| , and |v|∞ = max1≤l≤p |vl| . Moreover , we use supp ( v ) = { 1 ≤ l ≤ p | vl 6= 0 } as the support of the vector v , and v−l = ( v1 , . . . , vl−1 , vl+1 , . . . , vp ) T. For every matrix A ∈ Rp1×p2 , define ‖A‖ = sup|v|2=1 |Av|2 , ‖A‖∞ = max1≤l1≤p1,1≤l2≤p2 |Al1 , l2 | , ‖A‖L∞ = sup|v|∞=1 |Av|∞ as various matrix norms , Λmax ( A ) and Λmin ( A ) as the largest and smallest eigenvalues ofA respectively . We will use I ( · ) as the indicator function and sgn ( · ) as the sign function . For two sequences an , bn , we say an bn when an = O ( bn ) and bn = O ( an ) hold at the same time . For simplicity , we denote Sp−1 and Bp as the unit sphere and unit ball in Rp centered at 0 . For a sequence of vectors { vi } ni=1 ⊆ Rp , we denote med ( · ) as the coordinate-wise median . Lastly , the generic constants are assumed to be independent of m , n , and p . 2 PRIVATE SIGN RECOVERY OF MEAN VECTOR . 2.1 MEDIAN BASED DIVIDE-AND-CONQUER . Let µ∗ = ( µ∗1 , . . . , µ ∗ p ) T be the true parameter of interest . We assume the vector is sparse in the sense that many entries µ∗l are zero . There are N i.i.d . observations Xi ’ s satisfying E [ Xi ] = µ∗ , and they are evenly stored in m different machines Hj ( where 1 ≤ j ≤ m ) . Denote X = { X1 , . . . , XN } as the full dataset . For simplicity we assume N = mn so that each machine has equally n samples . Our task is to identify the signs of µ∗ on all coordinates ( denoted as sgn ( µ∗ ) ) in this distributed setup while protecting the privacy of every elementXi on each machineHj . To recover the signs of the true mean vector µ∗ privately , the most direct way is to estimate the mean by some existing differentially private algorithms and take the signs of the estimator coordinatewisely . By post-processing property of differential privacy ( Proposition 2.1 in Dwork & Roth ( 2014 ) ) , we know this sign recovery method is also differentially private . Private mean estimation is a fundamental problem in private statistical analysis and has been studied intensively ( Dwork et al. , 2006 ; Lei , 2011 ; Bassily et al. , 2014 ; Cai et al. , 2019 ) . The standard approach is to project the data onto a known bounded domain , and then apply the noises according to the diameter of the feasible domain and the privacy level . However , this method requires input data or the true parameter lies in a known bounded domain , which seems unsatisfactory in practice . Moreover , since we only want to estimate the signs , which take value in the discrete set { −1 , 0 , 1 } , it seems unnecessary to perturb the mean directly . Indeed , to recover the signs of the true parameter , there is no need to obtain an accurate mean with all data . Instead , we propose a Median based Divide-and-Conquer ( Med-DC ) approach . To be more precise , we can estimate the signs on each local machine Hj , and aggregate these vectors of signs by taking median to produce a more accurate sign estimator . To present our method more clearly , we define the following quantization function Qλ ( · ) , Qλ ( x ) = sgn { sgn ( x ) · ( |x| − λ ) +︸ ︷︷ ︸ shrinkage operator } = { sgn ( x ) if |x| > λ , 0 if |x| ≤ λ . ( 2 ) Here λ is a thresholding parameter . When x is a vector , Qλ ( x ) performs the above operation coordinate-wisely . In particular , when λ = 0 , the function Q0 ( · ) acts the same as the sign function sgn ( · ) . Then we present our method in Algorithm 1 . Algorithm 1 Median divide-and-conquer ( Med-DC ) for sparse mean estimator . Input : Dataset X = { X1 , . . . , XN } evenly divided into m local machines Hj ( where j = 1 , . . . , m ) , the universal thresholding parameter λN . 1 : for j = 1 , . . . , m do 2 : The j-th machine Hj computes the local sample mean X̄j = n−1 ∑ i∈Hj Xi . Then Hj sendsQj = QλN ( X̄j ) to the server . 3 : end for 4 : The server takes coordinate-wise median Q̂ ( X ) = med ( QλN ( X̄j ) | 1 ≤ j ≤ m ) , ( 3 ) Output : The vector of signs Q̂ ( X ) . The choice of thresholding parameter will be discussed after Theorem 1 in Section 2.3 . Especially mention that there are some cases when the median is not uniquely determined . For example , it is possible that there are the same numbers of 0 ’ s and 1 ’ s , then the median can be arbitrary value in [ 0 , 1 ] . To avoid ambiguity , we simply take Q̂l ( X ) = 0 ( where l denotes the coordinate index ) whenever the median Q̂l ( X ) is not unique . Another important remark is that , the proposed sign recovery algorithm Q̂ ( · ) is deterministic . More precisely , as there is no additional random perturbation in Algorithm 1 , the output Q̂ ( X ) is completely determined by the input dataset X . It is able to protect data privacy in a weaker sense .

Summary: The paper considers the sign recovery problem in a distributed setting with privacy constraints. The paper proposes an algorithm “median divide-and-conquer (Med-DC)” which takes the sign locally in each machine and then takes the median globally. The paper shows that in the sparse mean estimation setting, Med-DC is correct with high probability under some assumptions and Med-DC satisfies a weaker notion of differential privacy proposed by the paper. The paper then extends this algorithm to the sparse linear regression setting.