Practical Locally Private Federated Learning with Communication Efficiency

1 INTRODUCTION . 1.1 BACKGROUND . Federated learning ( FL ) Kairouz et al . ( 2019 ) ; Konečnỳ et al . ( 2016 ) is a rapidly evolving application of distributed optimization to large-scale learning or estimation scenarios where multiple entities . called clients , collaborate in solving a machine learning problem , under the coordination of a central server . Each client ’ s raw data is stored locally and not exchanged or transferred . To achieve the learning objective , the server collects minimal information from the clients for immediate aggregation . FL is particularly suitable for mobile and edge device applications since the ( sensitive ) individual data never directly leave the device and has seen deployments in industries ( ? Hard et al. , 2019 ; Leroy et al. , 2019 ) . While FL offers significant practical privacy improvements over centralizing all the training data , it lacks a formal privacy guarantee . As discussed in Melis et al . ( 2018 ) , even if only model updates ( i.e . gradient updates ) are transmitted , it is easy to compromise the privacy of individual clients . Differential privacy ( DP ) ( Dwork et al. , 2014 ) is the state-of-the-art approach to address information disclosure . Differentially private algorithms fuse participation of any individual via injecting algorithm-specific random noise . In FL setting , DP is suitable for protecting against external adversaries , i.e . a malicious analyst that tries to infer individual data via observing final or intermediate model results . However , DP paradigms typically assume a trusted curator , which corresponds to the server in the FL setting . This assumption is often not satisfied in practical cases , under which users that act as clients may not trust the service provider that acts as the server . Local differential privacy ( LDP ) ( Kasiviswanathan et al. , 2011 ; Dwork et al. , 2014 ) provides privacy protection on the individual level via applying randomized mechanisms that obfuscate the data before leaving the client , and is a more natural privacy model for distributed learning scenarios like FL , allowing easier compliance with regulatory strictures ( Bhowmick et al. , 2018 ) . 1.2 METHODOLOGY . We will be focusing on distributed learning procedures that aim to solve an empirical risk minimization problem in a decentralized fashion : min θ∈Θ L ( θ ) , L ( θ ) = 1 N M∑ m=1 Lm ( θ ) ( 1 ) where M denotes the number of clients . Let Yi denotes training data that belongs to the m-th client consisting Nm data points , and N = ∑M m=1Nm . The term Lm ( θ ) = ∑ y∈Ym ` ( θ , y ) stands for locally aggregated loss evaluated at parameter θ , where ` : Θ × Y 7→ R+ is the loss function depending on the problem context and we use Θ and Y to describe parameter and data domain respectively . We will seek to optimize ( 1 ) via gradient-based methods . In what follows we will use [ n ] , n ∈ N+ to denote the set { 1 , 2 , . . . , n } . At round t , the procedure iterates as : 1 . Server distribute the current value θt among a subset S of clients . 2 . For each client s ∈ S , a local update is computed ∆s = g ( θt , Ys ) and transmitted to the server . 3 . The server aggregates all { ∆s } s∈Ss to obtain a global update ∆ and updates the parameter as θt+1 = θt + ∆ . We distinguish 2 practical scenarios as cross-silo FL and cross-device FL ( Kairouz et al. , 2019 ) . In cross-silo FL , M is relatively small ( i.e . M ≤ 100 ) and usually each client has a moderate or large amount of data ( i.e . minmNm 1 ) . As a consequence , all the clients participate in the learning process ( i.e . S = [ M ] ) . In each iteration a client computes locally a negative stochastic gradient g ( θt , Ys ) = − 1R ∑ y∈Rs ∇ ` ( θt , y ) of Ls ( θt ) /Ns , based on a uniform random subsample Rs ⊂ Ys of size R. In cross-device FL S is a uniformly random subset of [ M ] , and g ( θt , Ys ) = − 1Ns ∑ y∈Ys ∇ ` ( θt , y ) is the average negative gradient over Ys evaluated at θt . In this paper we will be focusing on the FedSgd aggregation rule corresponding to a stochastic gradient step with learning rate η , i.e. , ∆ = η ∑ s∈S Ns N g ( θt , Ys ) . Generally speaking , there are two sources of data that need privacy protection : the parameter θt and the updates { ∆s } s∈S . In this paper we will develop algorithms for protecting { ∆s } s∈S against inference attack or reconstruction attack , which will be defined later . Note that the protection of θts could be done via applying standard central DP techniques as in Bhowmick et al . ( 2018 ) . To begin our discussion on suitably defined privacy models , we first review the local model of privacy . Local differential privacy Kasiviswanathan et al . ( 2011 ) we say a randomized algorithm A that maps the private data X ∈ X to some value Z = A ( X ) ∈ Z is -locally differentially private , if the induced conditional probability measure P ( ·|X = x ) satisfies that for any x , x′ ∈ X and any Z ∈ Z : e− ≤ P ( Z|X = x ) P ( Z|X = x′ ) ≤ e ( 2 ) In the FL setting , the private data X is typically not the raw data Y but the gradient of the loss function ` evaluated at Y . Hereafter we will assume the private data to be a d dimensional euclidean vector , while d could be very large ( i.e . to the order of millions ) . LDP is a very strong privacy protection model that protects individual data against inference attacks that aims at inferring the membership of arbitrary data from the data universe . According to the discussion in Bhowmick et al . ( 2018 ) , allowing adversaries with such strength may be overly pessimistic : to conduct an effective inference attack the adversary shall come up with `` the true data '' and only use the privatized output to verify his/her belief about the membership . If the prior information is reasonably constrained for the adversary , we may adopt a weaker , but still elegant type of privacy protection paradigm that protects against reconstruction attacks . Heuristically , reconstruction attack aims at recovering individual data with respect to a well-defined criterion , given a prior over the data domain that is not too concentrated . Formally , we adopt the definition in Bhowmick et al . ( 2018 ) : Reconstruction attack Bhowmick et al . ( 2018 ) Let π be a prior distribution over the data domain Y that encodes the adversary ’ s prior belief . We describe the generation process of privatized user data as Y → X → Z = A ( X ) for some ( privacy-protecting ) mechanism A. Additionally let f : Y 7→ Rk be the target of reconstruction ( i.e. , the adversary wants to evaluate the value f ( Y ) ) and Lrec : Rk × Rk 7→ R+ be the reconstruction loss serving as the criterion . Then an estimator ζ : X 7→ Rk provides an ( α , p , f ) -reconstruction breach for the loss Lrec if there exists some z ∈ Z such that : P ( Lrec ( f ( X ) , ζ ( z ) ) |A ( X ) = z ) > p ( 3 ) The existence of a reconstruction breach provides an attack that effectively breaks the privatized algorithm for some output z . Hence to provide protection against reconstruction attacks , we need the following to hold : sup ζ sup z∈Z P ( Lrec ( f ( X ) , ζ ( z ) ) |A ( X ) = z ) ≤ p ( 4 ) When ( 4 ) holds , the mechanism A is said to be ( α , p , f ) -protected against reconstruction for the loss Lrec . In ( Bhowmick et al. , 2018 , Lemma 2.2 ) , the authors relates LDP mechanisms with protection against reconstruction attacks , and the result for several scenarios suggest that using LDP mechanisms with large may still provide decent protection against reconstruction . As a consequence , we identify two sorts of threats used in this paper : Inference attacks by powerful adversaries this is the standard setup in LDP scenarios under which the adversary can observe the final learned model , as well as the information about potential participants except for the precise membership list . For such cases , only low LDP mechanisms provide sufficient protection against inference attacks . Reconstruction attacks by curious onlookers in this relaxed scenario we assume the adversary is able to observe all model updates and individual communications to conduct reconstruction attacks that target the evaluation of some function f over private individual data . But knowledge about individual data is restricted some prior distribution πf . For such cases , we will discover mechanisms that are based upon LDP algorithms with large but provides decent protection against reconstruction . Moreover , we assume the adversaries to be semi-honest ( Lyu et al. , 2020 ) , i.e . they are curious about individual data but follow the FL protocol correctly . In the seminal work ( Duchi et al. , 2018 ) , the authors developed minimax optimal LDP mean estimators under i.i.d generative constraints . Bhowmick et al . ( 2018 ) used separate mechanisms to privatize both the direction and the scale of the individual gradients , and was shown to achieve comparable performance with respect to non-private counterparts using high s. Despite the effectiveness of the LDP approach , the communication cost 1 is prohibitive . For models involving deep neural architectures , the number of parameters explodes to the order of millions . For each round , each client transmit O ( dδ ) bits to the server , where δ is the minimum number of bits to represent a real number with desired precision . The resulting communication cost is usually not affordable for mobile scenarios . In fact , the primary bottleneck for FL is usually communication , especially for deep neural architectures ( Kairouz et al. , 2019 ) . It is thus of significant importance to reduce the communication cost to a reasonable level . We identify two main challenges in building practical privacy-preserving FL algorithms : Communication Efficiency The algorithm shall be communication efficient , measured in terms of bits transferred per client in a single round . High-dimensional compatibility It was shown in Duchi et al . ( 2018 ) that upon estimating a ddimensional vector , the locally private constraint incurs a multiplicative penalty of O ( d/ 2 ) in terms of minimax ` 2 risk . This linear dependence on the dimension of gradients may result in overly noisy gradient estimates that destroy the learning process . Hence the algorithm shall be able to handle models with high-dimensional inputs with tolerable performance degradation . 1In general , there are two sources of communication cost in a distributed learning scenario that requires sequential interactions between the server and the clients : downlink cost that happens during clients downloading the updated model from the central server , and uplink cost that happens when clients sending their updates to the server . In was previously noted in Konečnỳ et al . ( 2016 ) that uplink cost usually dominates downlink costs in FL settings . Hence in the scope of this work , we will refer to communication cost as uplink communication cost . Summary of contributions In this paper , we propose a practical solution to locally private FL setting that addresses the above concerns which we term selective quantized stochastic gradient descent ( sqSGD ) . Our contributions are summarized as follows : 1 . We propose a novel algorithm for locally private multivariate mean estimation . The proposed algorithm stochastically quantizes each dimension of each sample to K equally spaced levels and serves as the base algorithm for our gradient-based learning framework . 2 . We make several improvements to the basic algorithm under the gradient-based learning scenario . Specifically , we attribute the estimation error of the base algorithm to perturbation error caused by privacy constraints , and quantization error . Then we utilize a gradient subsampling strategy to simultaneously reduce communication cost and improve private estimation utility . We also apply randomized rotation to reduce quantization error . 3 . We verify the performance of sqSGD on benchmark datasets using standard neural architectures like LeNet ( Lecun et al. , 1998 ) and ResNet ( He et al. , 2016 ) , and make systematic studies on the impact of both communication and privacy constraints . Specifically , under the same privacy and communication constraints , our model outperforms baseline algorithms that do not involve quantization by a significant margin .

Summary: This paper studies a low communication algorithm for multivariate mean estimation in the federated learning setting with differentially private communication. The algorithm uses quantization and dimension subsampling (only reporting some coordinates of the vector) to lower communication and randomized rotation (essentially applying a random orthogonal matrix) to reduce quantization error. They then apply this algorithm to ERM, using it as a subroutine in SGD. They experimentally explore the behavior of their algorithm on a number of benchmark datasets. They consider how the performance changes as they vary epsilon, the discretization parameter and the number of epochs (in SGD).