Median DC for Sign Recovery: Privacy can be Achieved by Deterministic Algorithms

1 INTRODUCTION . With the development of technology for data acquisition and storage , the modern dataset has a larger scale , more complex structure , and more practical considerations , which addresses new challenges for data analysis . In recent years , large quantities of sensitive data are collected by individuals and companies . While one wants to extract more accurate statistical information from the distributed dataset , we must also beware of the leakage of this sensitive personal information during the training process . This calls for the study of distributed learning under privacy constraints ( Pathak et al. , 2010 ; Hamm et al. , 2016 ; Jayaraman et al. , 2018 ) . In privacy literature , differential privacy , which was firstly proposed in Dwork et al . ( 2006 ) , has been the most widely adopted definition of privacy tailored to statistical data analysis . It has achieved tremendous success in real-world applications . Denote the data universe to be X , for the dataset X = { Xi } ni=1 ∈ Xn where Xi ’ s are the data observations . The ( , δ ) -differentially privacy can be defined as follows . Definition 1 . ( Differential Privacy in Dwork et al . ( 2006 ) ) A randomized algorithm A : Xn → Θ gives ( , δ ) -differentially private if for any pair of adjacent datasets X ∈ Xn and X′ ∈ Xn , there always holds P { A ( X1 : n ) ∈ U } ≤ e · P { A ( X′1 : n ) ∈ U } + δ , ( 1 ) for every subset U ⊆ Θ . Here two datasets X and X′ are adjacent if and only if the Hamming distance ( Lei , 2011 ) of these two datasets of same size H ( X , X′ ) = 1 . As we can see , the quantities and δ measure the level of privacy loss . There are also several relaxations of differential privacy ( see , e.g. , Bun & Steinke ( 2016 ) ; Dwork & Rothblum ( 2016 ) ; Mironov ( 2017 ) ; Dong et al . ( 2019 ) ) designed for the ease of analysis . However , in these definitions , the dataset X is always assumed to be fixed , and the probability in ( 1 ) only takes over the randomness of the algorithm A . Therefore , it is impossible to achieve strict differential privacy without adding auxiliary perturbations in the algorithm . On the other hand , the statistical performance of the output is inevitably deteriorated by the additional randomness . This lead to a large body of works discussing the tradeoff between accuracy and privacy ( Wasserman & Zhou , 2010 ; Bassily et al. , 2014 ; Bun et al. , 2018 ; Duchi et al. , 2018 ; Cai et al. , 2019 ) . In this paper , we consider the private sign recovery problem in the distributed system . To be more precise , assume the parameter of interest is a sparse vector , which has many zeros in its entries . The task is to identify the signs of the parameter from the observations stored in multiple machines while protecting each individual ’ s privacy . The sign recovery problem , as an extension of sparsity pattern recovery , has found its significance in a broad variety of contexts , including variable selection ( Tibshirani , 1996 ; Miller , 2002 ) , graphical models ( Meinshausen & Bühlmann , 2006 ; Cai et al. , 2011 ) , compressed sensing ( Candes & Tao , 2005 ; Donoho , 2006 ) , and signal denoising ( Chen et al. , 2001 ) . However , this problem is rarely considered in the privacy community . To address the sign recovery problem , we propose the Median Divide-and-Conquer ( Med-DC ) method , a simple two-step procedure . Firstly , each local machine estimates the sparse parameter and sends the sign-vectors back to the server ; Secondly , the server aggregates these sign-vectors using coordinate-wise median and output the final sign estimator . While mean based divide-andconquer ( also referred to as Mean-DC ) approaches have been widely analyzed in distributed learning literature ( Mcdonald et al. , 2009 ; Zhang et al. , 2013 ; Lee et al. , 2017 ; Battey et al. , 2018 ) , the median-based counterpart has not yet been well explored . It is well-known that naively averaging the local estimators behaves badly for nonlinear and penalized optimization problems . This is because averaging can not reduce the bias in the local sub-problems . In particular , for the distributed Lasso problem , as mentioned in Lee et al . ( 2017 ) , the estimation error of averaged local Lasso estimator is of the same order as that of local estimators . However , when only considering the sign recovery problem , we found that the Med-DC method perfectly fits the nature of the distributed private setup . ( See Section 2.2 for more detailed discussions ) For the sake of clarity , we only consider the sign recovery problem for sparse mean estimation and sparse linear regression , the two fundamental models in statistics . The proposed Med-DC method has the following advantages : • Consistent recovery . For both sparse mean estimation and sparse linear regression , the Med-DC method consistently recovers the signs of the true parameter with theoretical guarantees . Under some constraints , we can prove that our approach can identify signals larger thanC √ log n/N for some constant C > 0 ( where N is the full sample size and n is the local sample size ) , which coincides with the minimal signal level in the single machine setting ( all data stored in one machine ) . • Efficient communication . To recover the signs of the parameter of interest in the distributed setup , a naive approach is to estimate the parameter using existing private distributed estimation methods and take the signs of the estimators . However , these methods usually involve multi-round aggregation of gradient information or local estimators , which seems costly for the simple sign recovery problem . Instead , our approach only aggregates the vectors of signs ( bits information ) in one shot , which is much more communicationally efficient . •Weak privacy . By relaxing the differential privacy to high-probability sense , our deterministic Med-DC method can be proved to be weakly ‘ private ’ . We also extend this concept to group privacy . To the best of our knowledge , this is the first deterministic algorithm that has a provable high-probability privacy guarantee . Moreover , as each machine only needs to transmit the vectors of signs , instead of the local estimators or gradient vectors , our proposed method also protects the privacy of each local machine , since gradient sharing can also result in privacy leakage ( Zhu et al. , 2019 ) . •Wide applicability . We believe the Med-DC approach deserves more attention due to its excellent practical performance and ease of implementation . For example , it is promising to apply the Med-DC method to wider classes of models , ( e.g . , generalized linear model , M -estimation , etc . ) or hybridize this method with many sophisticated distributed algorithms like averaged de-biased estimator in Lee et al . ( 2017 ) and Communication-Efficient Accurate Statistical Estimator ( CEASE ) in Fan et al . ( 2019 ) . Notations . For every vector v = ( v1 , ... , vp ) T , denote |v|2 = √∑p l=1 v 2 l , |v|1 = ∑p l=1 |vl| , and |v|∞ = max1≤l≤p |vl| . Moreover , we use supp ( v ) = { 1 ≤ l ≤ p | vl 6= 0 } as the support of the vector v , and v−l = ( v1 , . . . , vl−1 , vl+1 , . . . , vp ) T. For every matrix A ∈ Rp1×p2 , define ‖A‖ = sup|v|2=1 |Av|2 , ‖A‖∞ = max1≤l1≤p1,1≤l2≤p2 |Al1 , l2 | , ‖A‖L∞ = sup|v|∞=1 |Av|∞ as various matrix norms , Λmax ( A ) and Λmin ( A ) as the largest and smallest eigenvalues ofA respectively . We will use I ( · ) as the indicator function and sgn ( · ) as the sign function . For two sequences an , bn , we say an bn when an = O ( bn ) and bn = O ( an ) hold at the same time . For simplicity , we denote Sp−1 and Bp as the unit sphere and unit ball in Rp centered at 0 . For a sequence of vectors { vi } ni=1 ⊆ Rp , we denote med ( · ) as the coordinate-wise median . Lastly , the generic constants are assumed to be independent of m , n , and p . 2 PRIVATE SIGN RECOVERY OF MEAN VECTOR . 2.1 MEDIAN BASED DIVIDE-AND-CONQUER . Let µ∗ = ( µ∗1 , . . . , µ ∗ p ) T be the true parameter of interest . We assume the vector is sparse in the sense that many entries µ∗l are zero . There are N i.i.d . observations Xi ’ s satisfying E [ Xi ] = µ∗ , and they are evenly stored in m different machines Hj ( where 1 ≤ j ≤ m ) . Denote X = { X1 , . . . , XN } as the full dataset . For simplicity we assume N = mn so that each machine has equally n samples . Our task is to identify the signs of µ∗ on all coordinates ( denoted as sgn ( µ∗ ) ) in this distributed setup while protecting the privacy of every elementXi on each machineHj . To recover the signs of the true mean vector µ∗ privately , the most direct way is to estimate the mean by some existing differentially private algorithms and take the signs of the estimator coordinatewisely . By post-processing property of differential privacy ( Proposition 2.1 in Dwork & Roth ( 2014 ) ) , we know this sign recovery method is also differentially private . Private mean estimation is a fundamental problem in private statistical analysis and has been studied intensively ( Dwork et al. , 2006 ; Lei , 2011 ; Bassily et al. , 2014 ; Cai et al. , 2019 ) . The standard approach is to project the data onto a known bounded domain , and then apply the noises according to the diameter of the feasible domain and the privacy level . However , this method requires input data or the true parameter lies in a known bounded domain , which seems unsatisfactory in practice . Moreover , since we only want to estimate the signs , which take value in the discrete set { −1 , 0 , 1 } , it seems unnecessary to perturb the mean directly . Indeed , to recover the signs of the true parameter , there is no need to obtain an accurate mean with all data . Instead , we propose a Median based Divide-and-Conquer ( Med-DC ) approach . To be more precise , we can estimate the signs on each local machine Hj , and aggregate these vectors of signs by taking median to produce a more accurate sign estimator . To present our method more clearly , we define the following quantization function Qλ ( · ) , Qλ ( x ) = sgn { sgn ( x ) · ( |x| − λ ) +︸ ︷︷ ︸ shrinkage operator } = { sgn ( x ) if |x| > λ , 0 if |x| ≤ λ . ( 2 ) Here λ is a thresholding parameter . When x is a vector , Qλ ( x ) performs the above operation coordinate-wisely . In particular , when λ = 0 , the function Q0 ( · ) acts the same as the sign function sgn ( · ) . Then we present our method in Algorithm 1 . Algorithm 1 Median divide-and-conquer ( Med-DC ) for sparse mean estimator . Input : Dataset X = { X1 , . . . , XN } evenly divided into m local machines Hj ( where j = 1 , . . . , m ) , the universal thresholding parameter λN . 1 : for j = 1 , . . . , m do 2 : The j-th machine Hj computes the local sample mean X̄j = n−1 ∑ i∈Hj Xi . Then Hj sendsQj = QλN ( X̄j ) to the server . 3 : end for 4 : The server takes coordinate-wise median Q̂ ( X ) = med ( QλN ( X̄j ) | 1 ≤ j ≤ m ) , ( 3 ) Output : The vector of signs Q̂ ( X ) . The choice of thresholding parameter will be discussed after Theorem 1 in Section 2.3 . Especially mention that there are some cases when the median is not uniquely determined . For example , it is possible that there are the same numbers of 0 ’ s and 1 ’ s , then the median can be arbitrary value in [ 0 , 1 ] . To avoid ambiguity , we simply take Q̂l ( X ) = 0 ( where l denotes the coordinate index ) whenever the median Q̂l ( X ) is not unique . Another important remark is that , the proposed sign recovery algorithm Q̂ ( · ) is deterministic . More precisely , as there is no additional random perturbation in Algorithm 1 , the output Q̂ ( X ) is completely determined by the input dataset X . It is able to protect data privacy in a weaker sense .

Summary: The paper gives "almost private" algorithms for problem of sign recovery of mean vector and of linear regression. The techniques follow their general framework of Median DC, which is similar to the well-known median-of-means approach. They give theoretical guarantees for the same, along with empirical results comparing with known differentially private algorithms and their non-private counterparts.