Differential Privacy in Adversarial Learning with Provable Robustness

1 INTRODUCTION . The pervasiveness of machine learning exposes new vulnerabilities in software systems , in which deployed machine learning models can be used ( a ) to reveal sensitive information in private training data ( Fredrikson et al. , 2015 ) , and/or ( b ) to make the models misclassify , such as adversarial examples ( Carlini & Wagner , 2017 ) . Efforts to prevent such attacks typically seek one of three solutions : ( 1 ) Models which preserve differential privacy ( DP ) ( Dwork et al. , 2006 ) , a rigorous formulation of privacy in probabilistic terms ; ( 2 ) Adversarial training algorithms , which augment training data to consist of benign examples and adversarial examples crafted during the training process , thereby empirically increasing the classification accuracy given adversarial examples ( Kardan & Stanley , 2017 ; Matyasko & Chau , 2017 ) ; and ( 3 ) Provable robustness , in which the model classification given adversarial examples is theoretically guaranteed to be consistent , i.e. , a small perturbation in the input does not change the predicted label ( Cisse et al. , 2017 ; Kolter & Wong , 2017 ) . On the one hand , private models , trained with existing privacy-preserving mechanisms ( Abadi et al. , 2016 ; Shokri & Shmatikov , 2015 ; Phan et al. , 2016 ; 2017b ; a ; Yu et al. , 2019 ; Lee & Kifer , 2018 ) , are unshielded under adversarial examples . On the other hand , robust models , trained with adversarial learning algorithms ( with or without provable robustness to adversarial examples ) , do not offer privacy protections to the training data . That one-sided approach poses serious risks to machine learning-based systems ; since adversaries can attack a deployed model by using both privacy inference attacks and adversarial examples . To be safe , a model must be i ) private to protect the training data , and ii ) robust to adversarial examples . Unfortunately , there has not yet been research on how to develop such a model , which thus remains a largely open challenge . Simply combining existing DP-preserving mechanisms and provable robustness conditions ( Cisse et al. , 2017 ; Kolter & Wong , 2017 ; Raghunathan et al. , 2018 ) can not solve the problem , for many reasons . ( a ) Existing sensitivity bounds ( Phan et al. , 2016 ; 2017b ; a ) and designs ( Yu et al. , 2019 ; Lee & Kifer , 2018 ) have not been developed to protect the training data in adversarial training . It is obvious that using adversarial examples crafted from the private training data to train our models introduces a previously unknown privacy risk , disclosing the participation of the benign examples ( Song et al. , 2019 ) . ( b ) There is an unrevealed interplay among DP preservation , adversarial learning , and robustness bounds . ( c ) Existing algorithms can not be readily applied to address the trade-off among model utility , privacy loss , and robustness . Therefore , theoretically bounding the robustness of a model ( which both protects the privacy and is robust against adversarial examples ) is nontrivial . Our Contributions . Motivated by this open problem , we propose to develop a novel differentially private adversarial learning ( DPAL ) mechanism to : 1 ) preserve DP of the training data , 2 ) be provably and practically robust to adversarial examples , and 3 ) retain high model utility . In our mech- anism , privacy-preserving noise is injected into inputs and hidden layers to achieve DP in learning private model parameters ( Theorem 1 ) . Then , we incorporate ensemble adversarial learning into our mechanism to improve the decision boundary under DP protections . To do this , we introduce a concept of DP adversarial examples crafted using benign examples in the private training data under DP guarantees ( Eq . 9 ) . To address the trade-off between model utility and privacy loss , we propose a new DP adversarial objective function to tighten the model ’ s global sensitivity ( Theorem 3 ) ; thus , we significantly reduce the amount of noise injected into our function , compared with existing works ( Phan et al. , 2016 ; 2017b ; a ) . In addition , ensemble DP adversarial examples with a dynamic perturbation size µa are introduced into the training process to further improve the robustness of our mechanism under different attack algorithms . An end-to-end privacy analysis shows that , by slitting the private training data into disjoint and fixed batches across epochs , the privacy budget in our DPAL is not accumulated across training steps ( Theorem 4 ) . After preserving DP in learning model parameters , we establish a solid connection among privacy preservation , adversarial learning , and provable robustness . Noise injected into different layers is considered as a sequence of randomizing mechanisms , providing different levels of robustness . By leveraging the sequential composition theory in DP ( Dwork & Roth , 2014 ) , we derive a novel generalized robustness bound , which essentially is a composition of these levels of robustness ( Theorem 5 and Proposition 1 ) . To our knowledge , our mechanism establishes the first connection between DP preservation and provable robustness against adversarial examples in adversarial learning . Rigorous experiments conducted on MNIST and CIFAR-10 datasets ( Lecun et al. , 1998 ; Krizhevsky & Hinton , 2009 ) show that our mechanism notably enhances the robustness of DP deep neural networks , compared with existing mechanisms . 2 BACKGROUND . In this section , we revisit adversarial learning , DP , and our problem definition . Let D be a database that contains N tuples , each of which contains data x ∈ [ −1 , 1 ] d and a ground-truth label y ∈ ZK , withK possible categorical outcomes . Each y is a one-hot vector ofK categories y = { y1 , . . . , yK } . A single true class label yx ∈ y given x ∈ D is assigned to only one of the K categories . On input x and parameters θ , a model outputs class scores f : Rd → RK that maps d-dimensional inputs x to a vector of scores f ( x ) = { f1 ( x ) , . . . , fK ( x ) } s.t . ∀k ∈ [ 1 , K ] : fk ( x ) ∈ [ 0 , 1 ] and∑K k=1 fk ( x ) = 1 . The class with the highest score value is selected as the predicted label for the data tuple , denoted as y ( x ) = maxk∈K fk ( x ) . A loss function L ( f ( x ) , y ) presents the penalty for mismatching between the predicted values f ( x ) and original values y . For the sake of clarity , the notations and terminologies frequently used in this paper are summarized in Table 1 ( Appendix A ) . Let us briefly revisit DP-preserving techniques in deep learning , starting with the definition of DP . Definition 1 ( , δ ) -DP ( Dwork et al. , 2006 ) . A randomized algorithm A fulfills ( , δ ) -DP , if for any two databases D and D′ differing at most one tuple , and for all O ⊆ Range ( A ) , we have : Pr [ A ( D ) = O ] ≤ e Pr [ A ( D′ ) = O ] + δ ( 1 ) A smaller enforces a stronger privacy guarantee . Here , controls the amount by which the distributions induced by D and D′ may differ , δ is a broken probability . DP also applies to general metrics ρ ( D , D′ ) ≤ 1 , where ρ can be lp-norms ( Chatzikokolakis et al. , 2013 ) . DP-preserving algorithms in deep learning can be categorized into two lines : 1 ) introducing noise into gradients of parameters ( Abadi et al. , 2016 ; Shokri & Shmatikov , 2015 ; Abadi et al. , 2017 ; Yu et al. , 2019 ; Lee & Kifer , 2018 ; Phan et al. , 2019 ) , 2 ) injecting noise into objective functions ( Phan et al. , 2016 ; 2017b ; a ) , and 3 ) injecting noise into labels ( Papernot et al. , 2018 ) . In Lemmas 2 and 4 , we will show that our mechanism achieves better sensitivity bounds compared with existing works ( Phan et al. , 2016 ; 2017b ; a ) . Adversarial Learning . For some target model f and inputs ( x , yx ) , the adversary ’ s goal is to find an adversarial example xadv = x + α , where α is the perturbation introduced by the attacker , such that : ( 1 ) xadv and x are close , and ( 2 ) the model misclassifies xadv , i.e. , y ( xadv ) 6= y ( x ) . In this paper , we consider well-known lp∈ { 1,2 , ∞ } -norm bounded attacks ( Goodfellow et al. , 2014b ) . Let lp ( µ ) = { α ∈ Rd : ‖α‖p ≤ µ } be the lp-norm ball of radius µ . One of the goals in adversarial learning is to minimize the risk over adversarial examples : θ∗ = arg minθ E ( x , ytrue ) ∼D [ max‖α‖p≤µ L ( f ( x + α , θ ) , yx ) ] , where an attack is used to approximate solutions to the inner maximization problem , and the outer minimization problem corresponds to training the model f with parameters θ over these adversarial examples xadv = x + α . There are two basic adversarial example attacks . The first one is a single-step algorithm , in which only a single gradient computation is required . For instance , FGSM algorithm ( Goodfellow et al. , 2014b ) finds adversarial examples by solving the inner maximization max‖α‖p≤µ L ( f ( x + α , θ ) , yx ) . The second one is an iterative algorithm , in which multiple gradients are computed and updated . For instance , in ( Kurakin et al. , 2016a ) , FGSM is applied multiple times with Tµ small steps , each of which has a size of µ/Tµ . To improve the robustness of models , prior work focused on two directions : 1 ) Producing correct predictions on adversarial examples , while not compromising the accuracy on legitimate inputs ( Kardan & Stanley , 2017 ; Matyasko & Chau , 2017 ; Wang et al. , 2016 ; Papernot et al. , 2016b ; a ; Gu & Rigazio , 2014 ; Papernot & McDaniel , 2017 ; Hosseini et al. , 2017 ) ; and 2 ) Detecting adversarial examples ( Metzen et al. , 2017 ; Grosse et al. , 2017 ; Xu et al. , 2017 ; Abbasi & Gagné , 2017 ; Gao et al. , 2017 ) . Among existing solutions , adversarial training appears to hold the greatest promise for learning robust models ( Tramèr et al. , 2017 ) . One of the well-known algorithms was proposed in ( Kurakin et al. , 2016b ) . At every training step , new adversarial examples are generated and injected into batches containing both benign and adversarial examples . The typical adversarial learning in ( Kurakin et al. , 2016b ) is presented in Alg . 2 ( Appendix B ) . DP and Provable Robustness . Recently , some algorithms ( Cisse et al. , 2017 ; Kolter & Wong , 2017 ; Raghunathan et al. , 2018 ; Cohen et al. , 2019 ; Li et al. , 2018 ) have been proposed to derive provable robustness , in which each prediction is guaranteed to be consistent under the perturbation α , if a robustness condition is held . Given a benign example x , we focus on achieving a robustness condition to attacks of lp ( µ ) -norm , as follows : ∀α ∈ lp ( µ ) : fk ( x+ α ) > max i : i 6=k fi ( x+ α ) ( 2 ) where k = y ( x ) , indicating that a small perturbation α in the input does not change the predicted label y ( x ) . To achieve the robustness condition in Eq . 2 , Lecuyer et al . ( Lecuyer et al. , 2018 ) introduce an algorithm , called PixelDP . By considering an input x ( e.g. , images ) as databases in DP parlance , and individual features ( e.g. , pixels ) as tuples , PixelDP shows that randomizing the scoring function f ( x ) to enforce DP on a small number of pixels in an image guarantees robustness of predictions against adversarial examples . To randomize f ( x ) , random noise σr is injected into either input x or an arbitrary hidden layer , resulting in the following ( r , δr ) -PixelDP condition : Lemma 1 ( r , δr ) -PixelDP ( Lecuyer et al. , 2018 ) . Given a randomized scoring function f ( x ) satisfying ( r , δr ) -PixelDP w.r.t . a lp-norm metric , we have : ∀k , ∀α ∈ lp ( 1 ) : Efk ( x ) ≤ e rEfk ( x+ α ) + δr ( 3 ) where Efk ( x ) is the expected value of fk ( x ) , r is a predefined budget , δr is a broken probability . At the prediction time , a certified robustness check is implemented for each prediction . A generalized robustness condition is proposed as follows : Êlbfk ( x ) > e2 r max i : i 6=k Êubfi ( x ) + ( 1 + e r ) δr ( 4 ) where Êlb and Êub are the lower and upper bounds of the expected value Êf ( x ) = 1n ∑ n f ( x ) n , derived from the Monte Carlo estimation with an η-confidence , given n is the number of invocations of f ( x ) with independent draws in the noise σr . Passing the check for a given input guarantees that no perturbation up to lp ( 1 ) -norm can change the model ’ s prediction . PixelDP does not preserve DP in learning private parameters θ to protect the training data . That is different from our goal .

Summary: This paper focus on providing both differential privacy and adversarial robustness to machine learning models. The authors propose an algorithm called differentially private adversarial learning (DPAL) to achieve such goal. DPAL consists two sub-models: (1) An auto-encoder to extract feature representation; and (2) A classifier takes the embedding of encoder and return the predict logits. The auto encoder uses the reconstruction loss and the classifier uses the similar loss as in adversarial learning.