R-GAP: Recursive Gradient Attack on Privacy

1 INTRODUCTION . Distributed and federated learning have become common strategies for training neural networks without transferring data ( Jochems et al. , 2016 ; 2017 ; Konečný et al. , 2016 ; McMahan et al. , 2017 ) . Instead , model updates , often in the form of gradients , are exchanged between participating nodes . These are then used to update at each node a copy of the model . This has been widely applied for privacy purposes ( Rigaki & Garcia , 2020 ; Cristofaro , 2020 ) , including with medical data ( Jochems et al. , 2016 ; 2017 ) . Recently , it has been demonstrated that this family of approaches is susceptible to attacks that can in some circumstances recover the training data from the gradient information exchanged in such federated learning approaches , calling into question their suitability for privacy preserving distributed machine learning ( Phong et al. , 2018 ; Wang et al. , 2019 ; Zhu et al. , 2019 ; Zhao et al. , 2020 ; Geiping et al. , 2020 ; Wei et al. , 2020 ) . To date these attack strategies have broadly fallen into two groups : ( i ) an analytical attack based on the use of gradients with respect to a bias term ( Phong et al. , 2018 ) , and ( ii ) an optimization-based attack ( Zhu et al. , 2019 ) that can in some circumstances recover individual training samples in a batch , but that involves a difficult nonconvex optimization that doesn ’ t always converge to a correct solution ( Geiping et al. , 2020 ) , and that provides comparatively little insights into the information that is being exploited in the attack . The development of privacy attacks is most important because they inform strategies for protecting against them . This is achieved by perturbations to the transferred gradients , and the form of the attack can give insights into the type of perturbation that can effectively protect the data ( Fan et al. , 2020 ) . As such , the development of novel closed-form attacks is essential to the analysis of privacy in federated learning . More broadly , the existence of model inversion attacks ( He et al. , 2019 ; Wang et al. , 2019 ; Yang et al. , 2019 ; Zhang et al. , 2020 ) calls into question whether transferring a fully trained model can be considered privacy preserving . As the weights of a model trained by ( stochastic ) gradient descent are the summation of individual gradients , understanding gradient attacks can assist in the analysis of and protection against model inversion attacks in and outside of a federated learning setting . In this work , we develop a novel third family of attacks , recursive gradient attack on privacy ( RGAP ) , that is based on a recursive , depth-wise algorithm for recovering training data from gradient information . Different from the analytical attack using the bias term , R-GAP utilizes much more information and is the first closed-form algorithm that works on both convolutional networks and fully connected networks with or without bias term . Compared to optimization-based attacks , it is not susceptible to local optima , and is orders of magnitude faster to run with a deterministic running time . Furthermore , we show that under certain conditions our recursive attack can fully recover training data in cases where optimization attacks fail . Additionally , the insights gained from the closed form of our recursive attack have lead to a refined rank analysis that predicts which network architectures enable full recovery , and which lead to provable noisy recovery due to rankdeficiency . This explains well the performance of both closed-form and optimization-based attacks . We also demonstrate that using rank analysis we are able to make small modifications to network architectures to increase the network ’ s security without sacrificing its accuracy . 1.1 RELATED WORK . Bias attacks : The original discovery of the existence of an analytical attack based on gradients with respect to the bias term is due to Phong et al . ( 2018 ) . Fan et al . ( 2020 ) also analyzed the bias attack as a system of linear equations , and proposed a method of perturbing the gradients to protect against it . Their work considers convolutional and fully-connected networks as equivalent , but this ignores the aggregation of gradients in convolutional networks . Similar to our work , they also perform a rank analysis , but it considers fewer constraints than is included in our analysis ( Section 4 ) . Optimization attacks : The first attack that utilized an optimization approach to minimize the distance between gradients appears to be due to Wang et al . ( 2019 ) . In this work , optimization is adopted as a submodule in their GAN-style framework . Subsequently , Zhu et al . ( 2019 ) proposed a method called deep leakage from gradients ( DLG ) which relies entirely on minimization of the difference of gradients ( Section 2 ) . They propose the use of L-BFGS ( Liu & Nocedal , 1989 ) to perform the optimization . Zhao et al . ( 2020 ) further analyzed label inference in this setting , proposing an analytic way to reconstruct the one-hot label of multi-class classification in terms of a single input . Wei et al . ( 2020 ) show that DLG is sensitive to initialization and proposed that the same class image is an optimal initialization . They proposed to use SSIM as image similarity metric , which can then be used to guide optimization by DLG . Geiping et al . ( 2020 ) point out that as DLG requires second-order derivatives , L-BFGS actually requires third-order derivatives , which leads to challenging optimzation for networks with activation functions such as ReLU and LeakyReLU . They therefore propose to replace L-BFGS with Adam ( Kingma & Ba , 2015 ) . Similar to the work of Wei et al . ( 2020 ) , Geiping et al . ( 2020 ) propose to incorporate an image prior , in this case total variation , while using PSNR as a quality measurement . 2 OPTIMIZATION-BASED GRADIENT ATTACKS ON PRIVACY ( O-GAP ) Optimization-based gradient attacks on privacy ( O-GAP ) take the real gradients as its ground-truth label and utilizes optimization to decrease the distance between the real gradients ∇W and the dummy gradients ∇W′ generated by a pair of randomly initialized dummy data and dummy label . The objective function of O-GAP can be generally expressed as : arg min x′ , y′ ‖∇W−∇W′‖2 = arg min x′ , y′ d∑ i=1 ‖∇Wi −∇W′i‖2 , ( 1 ) where the summation is taken over the layers of a network of depth d , and ( x′ , y′ ) is the dummy training data and label used to generate ∇W ′ . The idea of O-GAP was proposed by Wang et al . ( 2019 ) . However , they have adopted it as a part of their GAN-style framework and did not realize that O-GAP is able to preform a more accurate attack by itself . Later in the work of Zhu et al . ( 2019 ) , O-GAP has been proposed as a stand alone approach , the framework has been named as Deep Leakage from Gradients ( DLG ) . The approach is intuitively simple , and in practice has been shown to give surprisingly good results ( Zhu et al. , 2019 ) . However , it is sensitive to initialization and prone to fail ( Zhao et al. , 2020 ) . The choice of optimizer is therefore important , and convergence can be very slow ( Geiping et al. , 2020 ) . Perhaps most importantly , Equation 1 gives little insight into what information in the gradients is being exploited to recover the data . Analysis in Zhu et al . ( 2019 ) is limited to empirical insights , and fundamental open questions remain : What are sufficient conditions for arg minx′ , y′ ∑d i=1 ‖∇Wi − ∇W ′ i‖2 to have a unique minimizer ? We address this question in Section 4 , and subsequently validate our findings empirically . 3 CLOSED-FORM GRADIENT ATTACKS ON PRIVACY . The first attempt of closed-form GAP was proposed in a research of privacy-preserving deep learning by Phong et al . ( 2018 ) . Theorem 1 ( Phong et al . ( 2018 ) ) . Assume a layer of a fully connected network with a bias term , expressed as : Wx + b = z , ( 2 ) where W , b denote the weight matrix and bias vector , and x , z denote the input vector and output vector of this layer . If the loss function ` of the network can be expressed as : ` = ` ( f ( x ) , y ) where f indicates a nested function of x including activation function and all subsequent layers , y is the ground-truth label . Then x can be derived from gradients w.r.t . W and gradients w.r.t . b , i.e . : ∂ ` ∂W = ∂ ` ∂z x > , ∂ ` ∂b = ∂ ` ∂z x > = ∂ ` ∂Wj / ∂ ` ∂bj ( 3 ) where j denotes the j-th row , note that in fact from each row we can compute a copy of x > . When this layer is the first layer of a network , it is possible to reconstruct the data , i.e . x , using this approach . In the case of noisy gradients , we can make use of the redundancy in estimating x by averaging over noisy estimates : x̂ > = ∑ j ∂ ` ∂Wj / ∂ ` ∂bj . However , simply removing the bias term can disable this attack . Besides , this approach does not work on convolutional neural networks due to a dimension mismatch in Equation 3 . Both of these two problems have been resolved in our approach . 3.1 RECURSIVE GRADIENT ATTACK ON PRIVACY ( R-GAP ) For simplicity we derive the R-GAP in terms of binary classification with a single image as input . In this setting we can generally describe the network and loss function as : µ = ywd = : fd−1 ( x ) ︷ ︸︸ ︷ σd−1 Wd−1 σd−2 ( Wd−2φ ( x ) ) ︸ ︷︷ ︸ = : fd−2 ( x ) ( 4 ) ` = log ( 1 + e−µ ) ( 5 ) where y ∈ { −1 , 1 } , d denotes the d-th layer , φ represents all layers previous to d− 2 , and σ denotes the activation function . Note that , although our notation omits the bias term in our approach , with an augmented matrix and augmented vector it is able to represent both of the linear map and the translation , e.g . Equation 2 , using matrix multiplication as shown in Equation 4 . So our formulation also includes the approach proposed by Phong et al . ( 2018 ) . Moreover , if the i-th layer is a convolutional layer , then Wi is an extended circulant matrix representing the convolutional kernel ( Golub & Van Loan , 1996 ) , and data x as well as input of each layer are represented by a flattened vector in Equation 4 . 3.1.1 RECOVERING DATA FROM GRADIENTS . From Equation 4 and Equation 5 we can derive following gradients : ∂ ` ∂wd = y ∂ ` ∂µ f > d−1 ( 6 ) ∂ ` ∂Wd−1 = ( ( w > d ( y ∂ ` ∂µ ) ) σ′d−1 ) f > d−2 ( 7 ) ∂ ` ∂Wd−2 = ( ( W > d−1 ( ( w > d ( y ∂ ` ∂µ ) ) σ′d−1 ) ) σ′d−2 ) φ > ( 8 ) where σ′ denotes the derivative of σ , for more details of deriving the gradients refer to Appendix H. The first observation of these gradients is that : ∂ ` ∂wd · wd = ∂ ` ∂µ µ ( 9 ) Additionally , if σ1 , ... , σd−1 are ReLU or LeakyRelu , the dot product of the gradients and weights of each layer will be the same , i.e . : ∂ ` ∂wd · wd = ∂ ` ∂Wd−1 ·Wd−1 = ... = ∂ ` ∂W1 ·W1 = ∂ ` ∂µ µ ( 10 ) Since gradients and weights of each layer are known , we can obtain ∂ ` ∂µµ . If loss function ` is logistic loss ( Equation 5 ) , we obtain : ∂ ` ∂µ µ = −µ 1 + eµ . ( 11 ) In order to perform R-GAP , we need to derive µ from ∂ ` ∂µµ . As we can see , ∂ ` ∂µµ is non-monotonic , which means knowing ∂ ` ∂µµ does not always allow us to uniquely recover µ . However , even in the case that we can not uniquely recover µ , there are only two possible values to consider . Figure 1 illustrates ∂ ` ∂µµ of logistic , exponential , and hinge losses , showing when we can uniquely recover µ from ∂ ` ∂µµ . The non-uniqueness of µ inspires us to find a sort of data that can trigger exactly the same gradients as the real data , which we name twin data , denoted by x̃ . The existence of twin data demonstrates that the objective function of DLG could have more than one global minimum , which explains at least in part why DLG is sensitive to initialization , for more information and experiments about the twin data refer to Appendix B . The second observation on Equations 6-8 is that the gradients of each layer have a repeated format : ∂ ` ∂wd = kdf > d−1 ; kd : = y ∂ ` ∂µ ( 12 ) ∂ ` ∂Wd−1 = kd−1f > d−2 ; kd−1 : = ( w > d kd ) σ′d−1 ( 13 ) ∂ ` ∂Wd−2 = kd−2φ > ; kd−2 : = ( W > d−1kd−1 ) σ′d−2 ( 14 ) In Equation 12 , the value of y can be derived from the sign of the gradients at this layer if the activation function of previous layer is ReLU or Sigmoid , i.e . fd−1 > 0 . For multi-class classification , y can always be analytically derived as proved by Zhao et al . ( 2020 ) . From Equations 12-14 we can see that gradients are actually linear constraints on the output of the previous layer , also the input of the current layer . We name these gradient constraints , which can be generally described as : Kixi = flatten ( ∂ ` ∂Wi ) , ( 15 ) where i denotes i-th layer , xi denotes the input and Ki is a coefficient matrix containing all gradient constraints at the i-th layer .

Summary: This paper studies the problem of gradient attack in deep learning models. In particular,  this paper tries to form a system of linear equations to find a training data point when the gradient of the deep learning model with respect to that data point is available. The algorithm for finding the data point is called R-GAP.