Bayesian Framework for Gradient Leakage

1 INTRODUCTION . Federated learning ( McMahan et al. , 2017 ) has become a standard paradigm for enabling users to collaboratively train machine learning models . In this setting , clients compute updates on their own devices , send the updates to a central server which aggregates them and updates the global model . Because user data is not shared with the server or other users , this framework should , in principle , offer more privacy than simply uploading the data to a server . However , this privacy benefit has been increasingly questioned by recent works ( Melis et al. , 2019 ; Zhu et al. , 2019 ; Geiping et al. , 2020 ; Yin et al. , 2021 ) which demonstrate the possibility of reconstructing the original input from shared gradient updates . The reconstruction works by optimizing a candidate image with respect to a loss function that measures the distance between the shared and candidate gradients . The attacks typically differ in their loss function , their regularization , and how they solve the optimization problem . Importantly , the success of these attacks raises the following key questions : ( i ) what is the theoretically worst-case attack ? , and ( ii ) how do we evaluate defenses against gradient leakage ? This work In this work , we study and address these two questions from both theoretical and practical perspective . Specifically , we first introduce a theoretical framework which allows us to measure the expected risk an adversary has in reconstructing an input , given the joint probability distribution of inputs and their gradients . We then analyze the Bayes optimal adversary , which minimizes this risk and show that it solves a specific optimization problem involving the joint distribution . Further , we phrase existing attacks ( Zhu et al. , 2019 ; Geiping et al. , 2020 ; Yin et al. , 2021 ) as approximations of this optimal adversary , where each attack can be interpreted as implicitly making different assumptions on the distribution of gradients and inputs , in turn yielding different loss functions for the optimization . In our experimental evaluation , we compare the Bayes optimal adversary with other attacks , those which do not leverage the probability distribution of gradients , and we find that the Bayes optimal adversary performs better , as explained by the theory . We then turn to practical evaluation and experiment with several recently proposed defenses ( Sun et al. , 2021 ; Gao et al. , 2021 ; Scheliga et al. , 2021 ) based on different heuristics and demonstrate that they are not sufficient for protecting from gradient leakage against stronger attacks we design specifically for each defense . Interestingly , we find that models are especially vulnerable to attacks early in training , thus we advocate that defense evaluation should take place during and not only at the end of training . Overall , our findings suggest that creation of effective defenses and their evaluation is a challenging problem , and that our insights and contributions can substantially advance future research in the area . Main contributions Our main contributions are : • Formulation of the gradient leakage problem in a Bayesian framework which enables phrasing Bayes optimal adversary as an optimization problem . • Interpretation of several prior attacks as approximations of the Bayes optimal adversary , each using different assumptions for the distributions of inputs and their gradients . • Practical implementation of the Bayes optimal adversary for several defenses , showing higher reconstruction success than previous attacks , confirming our theoretical results . • Evaluation of several existing heuristic defenses demonstrating that they do not effectively protect from strong attacks , especially early in training , thus suggesting better ways for evaluating defenses in this space . 2 RELATED WORK . We now briefly survey some of the work most related to ours . Federated Learning Federated learning ( McMahan et al. , 2017 ) was introduced as a way to train machine learning models in decentralized settings with data coming from different user devices . This new form of learning has caused much interest in its theoretical properties ( Konečný et al. , 2016a ) and ways to improve training efficiency ( Konečný et al. , 2016b ) . More specifically , besides decentralizing the computation on many devices , the fundamental promise of this approach is privacy , as the user data never leaves their devices . Gradient Leakage Attacks Recent work ( Zhu et al. , 2019 ; Geiping et al. , 2020 ) has shown that such privacy assumptions , in fact , do not hold in practice , as an adversarial server can reliably recover an input image given gradient updates . They phrase the attacks as a minimization problem over the ` 2-distance between the gradients of a randomly initialized input image x′ and a target image x on a neural network hθ : ( x∗ , y∗ ) = argmin ( x′ , y′ ) ||∇l ( hθ ( x ) , y ) −∇l ( hθ ( x′ ) , y′ ) ||2 ( 1 ) Here l denotes a loss used to train the network , usually cross-entropy , while y and y′ correspond to the original and reconstructed label , respectively . Follow-up works improve on these results by using different distance metrics such as cosine similarity and input-regularization ( Geiping et al. , 2020 ) , smarter initialization ( Wei et al. , 2020 ) , normalization ( Yin et al. , 2021 ) , and others ( Mo et al. , 2021 ; Jeon et al. , 2021 ) . A significant improvement proposed by Zhao et al . ( 2020 ) showed how to recover the target label y from the gradients alone , reducing Eq . ( 1 ) to an optimization over x′ only . In Section 4 we show how these existing attacks can be interpreted as different approximations of the Bayes optimal adversary . Defenses In response to the rise of privacy-violating attacks on federated learning , many defenses have been proposed ( Abadi et al. , 2016 ; Sun et al. , 2021 ; Gao et al. , 2021 ) . Except for DP-SGD ( Abadi et al. , 2016 ) , a version of SGD with clipping and adding Gaussian noise , which is differentially private , they all provide none or little theoretical privacy guarantees . This is partly due to the fact that no mathematically rigorous attacker model exists , and defenses are empirically evaluated against known attacks . This also leads to a wide variety of proposed defenses : Soteria ( Sun et al. , 2021 ) prunes the gradient for a single layer , ATS ( Gao et al. , 2021 ) generates highly augmented input images that train the network to produce non-invertible gradients , and PRECODE ( Scheliga et al. , 2021 ) uses a VAE to hide the original input . Here we do not consider defenses that change the communication and training protocol ( Lee et al. , 2021 ; Wei et al. , 2021 ) . 3 BACKGROUND . Let X ⊆ Rd be an input space , and let hθ : X → Y be a neural network with parameters θ classifying an input to a label in the label space Y . We assume that inputs ( x , y ) are coming from a distribution D with a marginal distribution p ( x ) . In standard federated learning , there are n clients with loss functions l1 , ... , ln , who are trying to collaboratively solve the following optimization problem : min θ 1 n n∑ i=1 E ( x , y ) ∼D [ li ( hθ ( x ) , y ) ] To ease notation , we will assume a single client throughout the paper , but the same reasoning can be applied to the general n-client case . Additionally , each client could have a different distribution D , but the approach is again easy to generalize to this case . In a single training step , each client i first computes∇θli ( hθ ( xi ) , yi ) on a batch of data ( xi , yi ) , then sends these to the central server that performs a gradient descent step to obtain the new parameters θ′ = θ − αn ∑n i=1∇θli ( hθ ( xi ) , yi ) , where α is a learning rate . We will consider a scenario where each client reports , instead of the true gradient∇θli ( hθ ( xi ) , yi ) , a noisy gradient g sampled from a distribution p ( g|x ) , which we call a defense mechanism . The purpose of the defense mechanism is to add enough noise to hide the sensitive user information from the gradients while retaining high enough informativeness so that it can be used for training . Thus , given some noisy gradient g , the central server would update the parameters as θ′ = θ−αg ( assuming n = 1 as mentioned above ) . Typical examples of defenses used in our experiments in Section 6 , each inducing p ( g|x ) , include adding Gaussian or Laplacian noise to the original gradients , as well as randomly masking some components of the gradient . Naturally , p ( x ) and p ( g|x ) together induce a joint distribution p ( x , g ) . Note that a network that has no defense corresponds to the distribution p ( g|x ) , which is concentrated only on the true gradient at x . 4 BAYESIAN ADVERSARIAL FRAMEWORK . Here we describe our theoretical framework for gradient leakage in federated learning . Adversarial risk We first define the adversarial risk for gradient leakage and then derive the Bayes optimal adversary that minimizes this risk . The adversary can observe the gradient g and is trying to reconstruct the input x that produced this gradient . Formally , the adversary is a function f : Rk → X mapping gradients to inputs . Given some ( x , g ) sampled from the joint distribution p ( x , g ) , the adversary outputs the reconstruction f ( g ) and incurs loss L ( x , f ( g ) ) , which is a function L : X × X → R. Typically , we will consider a binary loss that evaluates to 0 if the adversary ’ s output is close to the original input , and 1 otherwise . If the adversary wants to reconstruct the exact input x , we can define the loss to be L ( x , x′ ) : = 1x 6=x′ , denoting with 1 the indicator function . If the adversary only wants to get to some δ-neighbourhood of the input x in the input space , a more appropriate definition of the loss is L ( x , x′ ) : = 1d ( x , x′ ) > δ . In this section , we will assume that the distance d is the ` 2-distance , but the approach can be generalized to other notions as well . This definition is well suited for image data , where ` 2 distance captures our perception of visual closeness , and for which the adversary can often obtain a reconstruction that is very close to the original image , both visually and in the ` 2 space . Note that if we let δ approach 0 in our second definition of the loss , we essentially recover the first loss . We can now define the risk R ( f ) of the adversary f as R ( f ) : = Ex , g [ L ( x , f ( g ) ) ] = Ex∼p ( x ) Eg∼p ( g|x ) [ L ( x , f ( g ) ) ] . Bayes optimal adversary We consider white-box adversary who knows the joint distribution p ( x , g ) , as opposed to the weaker alternative based on security through obscurity , where adversary does not know what defense is used , and therefore does not have a good estimate of p ( x , g ) . We also do not consider adversaries that can exploit other vulnerabilities in the system to obtain extra information . Let us consider the second definition of the loss for which L ( x , f ( g ) ) : = 1||x−f ( g ) ||2 > δ . We can then rewrite the definition of the risk as follows : R ( f ) = Ex , g [ L ( x , f ( g ) ) ] = EgEx|g [ 1||x−f ( g ) ||2 > δ ] = Eg ∫ X p ( x|g ) · 1||x−f ( g ) ||2 > δ dx = Eg ∫ X\B ( f ( g ) , δ ) p ( x|g ) dx = 1− Eg ∫ B ( f ( g ) , δ ) p ( x|g ) dx . Here B ( f ( g ) , δ ) denotes the ` 2-ball of radius δ around f ( g ) . Thus , an adversary which wants to minimize their risk has to maximize Eg ∫ B ( f ( g ) , δ ) p ( x|g ) dx , meaning that the adversarial function f can be defined as f ( g ) : = argmaxx0 ∫ B ( x0 , δ ) p ( x|g ) dx . Intuitively , the adversary predicts x0 which has the highest likelihood that one of the inputs in its δ-neighborhood was the original input that produced gradient g. Note that if we would let δ → 0 , then f ( g ) → argmaxx p ( x|g ) , which would be the solution for the loss that requires the recovered input to exactly match the original input . As we do not have the closed form for p ( x|g ) , it can be rewritten using Bayes ’ rule : f ( g ) = argmax x0∈X ∫ B ( x0 , δ ) p ( x|g ) dx = argmax x0∈X ∫ B ( x0 , δ ) p ( g|x ) p ( x ) p ( g ) dx = argmax x0∈X ∫ B ( x0 , δ ) p ( g|x ) p ( x ) dx = argmax x0∈X [ log ∫ B ( x0 , δ ) p ( g|x ) p ( x ) dx ] ( 2 ) The last steps follow from multiplying by the constant p ( g ) and taking the logarithm , both not affecting the argmax . Computing the optimal reconstruction now requires evaluating both the input prior p ( x ) and the conditional probability p ( g|x ) , which is determined by the used defense mechanism . Given these two ingredients , Eq . ( 2 ) then provides us with a way to compute the output of the Bayes optimal adversary by solving an optimization problem involving distributions p ( x ) and p ( g|x ) . Algorithm 1 Approximate Bayes optimal adversary x← attack init ( ) for i = 1 to m do Sample x1 , ... , xk uniformly from B ( x , δ ) x← x+ α∇x 1k ∑k i=1 log p ( g|xi ) + log p ( xi ) end for return x Approximate Bayes optimal adversary While Eq . ( 2 ) provides a formula for the optimal adversary in the form of an optimization problem , using this adversary for practical reconstruction is difficult due to three main challenges : ( i ) we require knowledge of the prior distribution p ( x ) , ( ii ) computing the integral over the δ-ball around x0 is generally not tractable , and ( iii ) we need to solve the optimization problem over X . However , we can address each of these challenges by introducing appropriate approximations . We first apply Jensen ’ s inequality to the logarithm function : max x0∈X [ logC ∫ B ( x0 , δ ) p ( g|x ) p ( x ) dx ] ≥ max x0∈X [ C ∫ B ( x0 , δ ) ( log p ( g|x ) + log p ( x ) ) dx ] . Here C = 1/δd is a normalization constant . For image data , we approximate the prior distribution using the total variation image prior , which has already worked well for gradient inversion ( Geiping et al. , 2020 ) . Alternatively , we could estimate it from data using density estimation models such as PixelCNN ( van den Oord et al. , 2016 ) or Glow ( Kingma & Dhariwal , 2018 ) . We then approximate the integral over the δ-ball via Monte Carlo integration by sampling k points x1 , ... , xk uniformly inside the ball to obtain the objective : 1 k k∑ i=1 log p ( g|xi ) + log p ( xi ) . ( 3 ) Finally , as the objective is differentiable , we can use gradient-based optimizer such as Adam ( Kingma & Ba , 2015 ) , and obtain the attack in Algorithm 1 . Fig . 1 shows a single run of this adversary which is initialized randomly , and gets closer to the original image at every step . Existing attacks as approximations of the Bayes optimal adversary We now describe how existing attacks can be viewed as different approximations of the Bayes optimal adversary . Recall that the optimal adversary f searches for the δ-ball with the maximum value of the integral of log p ( g|x ) + log p ( x ) . Previously proposed attacks can in fact be recovered by plugging in different approximations for log p ( g|x ) and log p ( x ) into Algorithm 1 , with estimating the integral using k = 1 sample located at the center of the ball . For example , suppose that the defense mechanism adds Gaussian noise to the original gradient , which corresponds to the conditional probability p ( g|x ) = N ( ∇θl ( hθ ( x ) , y ) , σ2I ) . This implies that log p ( g|x ) = C − 12σ2 ||g − ∇θl ( hθ ( x ) , y ) || 2 2 , where C is a constant . Assuming a uniform prior ( where log p ( x ) is constant ) , the optimization problem in Eq . ( 2 ) boils down to minimizing ` 2 distance between g and ∇θl ( hθ ( x ) , y ) , which is exactly the optimization problem solved by Deep Leakage from Gradients ( DLG ) ( Zhu et al. , 2019 ) . Inverting Gradients ( Geiping et al. , 2020 ) uses a total variation ( TV ) image prior for log p ( x ) and cosine similarity instead of ` 2 to measure similarity between gradients . Cosine similarity corresponds to the distribution log p ( g|x ) = C − g T∇θl ( hθ ( x ) , y ) ||g||2||∇θl ( hθ ( x ) , y ) ||2 which also requires the support of the distribution to be bounded ( this happens , e.g. , when gradients are clipped ) . GradInversion ( Yin et al. , 2021 ) introduces a more complex prior based on a combination of the total variation , ` 2 norm of the image , and a DeepInversion prior while using l2 norm to measure distance between gradients , which corresponds to Gaussian noise , as observed before . Note that we can work with unnormalized densities , as multiplying by the normalization constant does not change the argmax in Eq . ( 2 ) . We summarize these observations in Table 1 , showing for each of the mentioned attacks their respective conditional and prior probability .

Summary: The paper proposes a theoretical Bayesian framework for the problem of gradient leakage in federated learning. It shows that recent gradient leakage attacks are approximations of the Bayesian framework with different prior distributions for the input and conditional distributions of the gradients given the input. The paper also claims that recent defense mechanisms are not good enough, especially during the early stages of training.