Certified Robustness for Free in Differentially Private Federated Learning

1 INTRODUCTION . Federated Learning ( FL ) , which aims to jointly train a global model with distributed local data , has been widely applied in different applications , such as finance ( Yang et al. , 2019b ) , medical analysis ( Brisimi et al. , 2018 ) , and user behavior prediction ( Hard et al. , 2018 ; Yang et al. , 2018 ; 2019a ) . However , the fact that the local data and the training process are entirely controlled by the local users who may be adversarial raises great concerns from both security and privacy perspectives . In particular , recent studies show that FL is vulnerable to different types of training-time attacks , such as model poisoning ( Bhagoji et al. , 2019 ) , backdoor attacks ( Bagdasaryan et al. , 2020 ; Xie et al. , 2019 ; Wang et al. , 2020 ) , and label-flipping attacks ( Fung et al. , 2020 ) . Further , privacy concerns have motivated the need to keep the raw data on local devices without sharing . However , sharing other indirect information such as gradients or model updates as part of the FL training process can also leak sensitive user information ( Zhu et al. , 2019 ; Geiping et al. , 2020 ; Bhowmick et al. , 2018 ; Melis et al. , 2019 ) . As a result , approaches based on differential privacy ( DP ) ( Dwork & Roth , 2014 ) , homomorphic encryption ( Bost et al. , 2015 ; Rouhani et al. , 2018 ; Gilad-Bachrach et al. , 2016 ) , and secure multiparty computation ( Ben-Or et al. , 1988 ; Bonawitz et al. , 2017 ) have been proposed to protect privacy of users in federated learning . In particular , differentially private federated learning ( DPFL ) provides strong information theoretic guarantees on user privacy , while causing relatively low performance overhead ( Li et al. , 2020b ) . Several defenses have been proposed to defend against poisoning attacks in FL . For instance , various robust aggregation methods ( Fung et al. , 2020 ; Pillutla et al. , 2019 ; Blanchard et al. , 2017 ; El Mhamdi et al. , 2018 ; Chen et al. , 2017b ; Yin et al. , 2018 ; Fu et al. , 2019 ; Li et al. , 2020a ) identify and down-weight the malicious updates during aggregation or estimate a true “ center ” of the received updates rather than taking a weighted average . Other methods include robust federated training protocols ( e.g. , clipping ( Sun et al. , 2019 ) , noisy perturbation ( Sun et al. , 2019 ) , and additional evaluation during training ( Andreina et al. , 2020 ) ) and post-training strategies ( e.g. , fine-tuning and pruning ( Wu et al. , 2020 ) ) that repair the poisoned global model . However , as these works mainly focus on providing empirical robustness for FL , they have been shown to be vulnerable to newly proposed strong adaptive attacks ( Wang et al. , 2020 ; Xie et al. , 2019 ; Baruch et al. , 2019 ; Fang et al. , 2020 ) . Hence , in this paper , we aim to develop certified robustness guarantees for FL against different poisoning attacks . Further , as differentially private federated learning ( DPFL ) is often used to protect user privacy , we also aim to ask : Can we leverage the innate privacy property of DPFL to provide robustness certification against poisoning attacks for free ? Can we further improve the privacy of FL so as to improve its certified robustness ? Recent studies suggest that differential privacy ( DP ) is inherently related with robustness of ML models . Intuitively , DP is designed to protect the privacy of individual data , such that the output of an algorithm remains essentially unchanged when one individual input point is modified . Hence , the prediction of a DP model will be less impacted by a small amount of poisoned training data . Consequently , DP has been used to provide both theoretical and empirical defenses against evasion attacks ( Lecuyer et al. , 2019a ) and data poisoning attacks ( Ma et al. , 2019 ; Hong et al. , 2020 ) on centralized ML models . It has also been used as an empirical defense against backdoor attacks ( Gu et al. , 2019 ) in federated learning ( Bagdasaryan et al. , 2020 ; Sun et al. , 2019 ) , although no theoretical guarantee is provided . To the best of our knowledge , despite of the wide application of DPFL , there is no work providing certified robustness for DPFL leveraging its privacy property . In this paper , we aim to leverage the inherent privacy property of DPFL to provide robustness certification for FL against poisoning attacks for free . Our challenges include : ( 1 ) performing privacy analysis over training rounds in DPFL algorithms and ( 2 ) theoretically guaranteeing certified robustness based on DP properties under a given privacy budget . We propose two robustness certification criteria for FL : certified prediction and certified attack cost under different attack constraints . We consider both user-level DP ( Agarwal et al. , 2018 ; Geyer et al. , 2017 ; McMahan et al. , 2018 ; Asoodeh & Calmon , 2020 ; Liang et al. , 2020 ) which is widely guaranteed in FL , and instance-level DP ( Malekzadeh et al. , 2021 ; Zhu et al. , 2021 ) which is less explored in FL . We prove that a FL model satisfying user-level DP is certifiably robust against a bounded number of adversarial users . In addition , we propose InsDP-FedAvg algorithm to improve instance-level DP in FL , and prove that instance-level DPFL is certifiably robust against a bounded number of adversarial instances . We also study the correlation between privacy guarantee and certified robustness of FL . While stronger privacy guarantees result in greater attack cost , overly strong privacy can hurt the certified prediction by introducing too much noise in the training process . Thus , the optimal certified prediction is often achieved under a proper balance between privacy protection and utility loss . Key Contributions . Our work takes the first step to provide certified robustness in DPFL for free against poisoning attacks . We make contributions on both theoretical and empirical fronts . • We propose two criteria for certified robustness of FL against poisoning attacks ( Section 4.2 ) . • Given a FL model satisfying user-level DP , we prove that it is certifiably robust against arbitrary poisoning attacks with a bounded number of adversarial users ( Section 4.2 ) . • We propose InsDP-FedAvg algorithm to improve FL instance-level privacy guarantee ( Sec- tion 5.1 ) . We prove that instance-level DPFL is certifiably robust against the manipulation of a bounded number of instances during training ( Section 5.2 ) . • We conduct extensive experiments on image classification of MNIST , CIFAR-10 and sentiment analysis of tweets to verify our proposed certifications of two robustness criteria , and compare the certified results of different DPFL algorithms ( Section 6 ) . 2 RELATED WORK . Differentially Private Federated Learning . Different approaches are proposed to guarantee the user-level privacy for FL . ( Geyer et al. , 2017 ; McMahan et al. , 2018 ) clip the norm of each local update , add Gaussian noise on the summed update , and characterize its privacy budget via moment accountant ( Abadi et al. , 2016 ) . ( McMahan et al. , 2018 ) extends ( Geyer et al. , 2017 ) to language models . In CpSGD ( Agarwal et al. , 2018 ) , each user clips and quantizes the model update , and adds noise drawn from Binomial distribution , achieving both communication efficiency and DP . ( Bhowmick et al. , 2018 ) derive DP for FL via Rényi divergence ( Mironov , 2017 ) and study its protection against data reconstruction attacks . ( Liang et al. , 2020 ) utilizes Laplacian smoothing for each local update to enhance the model utility . Instead of using moment accountant to track privacy budget over FL rounds as previous work , ( Asoodeh & Calmon , 2020 ) derives the DP parameters by interpreting each round as a Markov kernel and quantify its impact on privacy parameters . All these works only focus on providing user-level privacy , leaving its robustness property unexplored . In terms of instance-level privacy for FL , there are only a few work ( Malekzadeh et al. , 2021 ; Zhu et al. , 2021 ) . Dopamine ( Malekzadeh et al. , 2021 ) provides instance-level privacy guarantee when each user only performs one step of DP-SGD ( Abadi et al. , 2016 ) at each FL round . However , it can not be applied to multi-step SGD for each user , thus it can not be extended to the general FL setting FedAvg ( McMahan et al. , 2017 ) . ( Zhu et al. , 2021 ) privately aggregate the labels from users in a voting scheme , and provide DP guarantees on both user level and instance level . However , it is also not applicable to standard FL , since it does not allow aggregating the gradients or updates . Differential Privacy and Robustness . In standard ( centralized ) learning , Pixel-DP ( Lecuyer et al. , 2019a ) is proposed to certify the model robsutness against evasion attacks . However , it is unclear how to leverage it to certify against poisoning attacks . To certify the robustness against poisoning attacks , ( Ma et al. , 2019 ) show that private learners are resistant to data poisoning and analyze the lower bound of attack cost against poisoning attacks for regression models . Here we certify the robustness in DPFL setting with such lower bound as one of our certification criteria and additionally derive its upper bounds . ( Hong et al. , 2020 ) show that the off-the-shelf mechanism DP-SGD ( Abadi et al. , 2016 ) , which clips per-sample gradients and add Guassian noises during training , can serve as a defense against poisoning attacks empirically . In federated learning , empirical work ( Bagdasaryan et al. , 2020 ; Sun et al. , 2019 ) show that DPFL can mitigate backdoor attacks ; however , none of these work provides certified robustness guarantees for DPFL against poisoning attacks . 3 PRELIMINARIES . We start by providing some background on differential privacy ( DP ) and federated learning ( FL ) . Differential Privacy ( DP ) . DP is a formal , mathematically rigorous definition ( and standard ) of privacy that intuitively guarantees that a randomized algorithm behaves similarly on similar inputs and that the output of the algorithm is about the same whether or not an individual ’ s data is included as part of the input ( Dwork & Roth , 2014 ) . Definition 1 ( ( , δ ) -DP ( Dwork et al. , 2006 ) ) . A randomized mechanismM : D → Θ with domain D and range Θ satisfies ( , δ ) -DP if for any pair of two adjacent datasets d , d′ ∈ D , and for any possible ( measurable ) output set E ⊆ Θ , it holds that Pr [ M ( d ) ∈ E ] ≤ e Pr [ M ( d′ ) ∈ E ] + δ . In Definition 1 , whenM is a training algorithm for ML model , domain D and range Θ represent all possible training datasets and all possible trained models respectively . Group DP for ( , δ ) -DP mechanisms follows immediately from Definition 1 where the privacy guarantee drops with the size of the group . Formally , it says : Lemma 1 ( Group DP ) . For mechanismM that satisfies ( , δ ) -DP , it satisfies ( k , 1−e k 1−e δ ) -DP for groups of size k. That is , for any d , d′ ∈ D that differ by k individuals , and any E ⊆ Θ it holds that Pr [ M ( d ) ∈ E ] ≤ ek Pr [ M ( d′ ) ∈ E ] + 1−e k 1−e δ. Federated Learning . FedAvg was introduced by ( McMahan et al. , 2017 ) for FL to train a shared global model without direct access to training data of users . Specifically , given a FL system with N users , at round t , the server sends the current global model wt−1 to users in the selected user set Ut , where |Ut| = m = qN and q is the user sampling probability . Each selected user i ∈ Ut locally updates the model for E local epochs with its dataset Di and learning rate η to obtain a new local model . Then , the user sends the local model updates ∆wit to the server . Finally , the server aggregates over the updates from all selected users into the new global model wt = wt−1 + 1m ∑ i∈Ut ∆w i t .

Summary: ## Update after rebuttal and discussions  I thank the authors for taking the time to discuss the issues pointed out in the reviews at length. Unfortunately, I am still not convinced that the paper is ready for publication. My main concerns:  1) There are now experiments in the updated paper claimed to be DP which are not (median clipping).  2) I continue to have doubts about the subsampling amplification. Simply stating that the sampling is random is not good enough, since the key issue is the added uncertainty due to the subsampling: if the sampling does not increase the adversary's uncertainty, there is no amplification. As an immediate remedy, I suggest the authors state the threat model more clearly.  3) I still think the paper can be improved a lot by taking the time to rewrite it focusing on the main contribution of certified robustness under DP and clarity of the presentation.  ---  The paper looks at the robustness properties of differentially private (DP) federated learning (FL), focusing on learning classification models from labeled data. The main idea is to turn DP privacy guarantees into certifiable robustness properties. The authors look at 2 certifiable properties, namely, certified prediction (data poisoning does not alter most likely label), and certified attack cost (there is a lower bound on the loss the given attack tries to minimize). They continue to show that DP models in general guarantee these on some level that depends on the privacy bounds. The paper also presents several DPFL learning algorithms for user and instance-level DP.