Bayesian Framework for Gradient Leakage
Authors: Mislav Balunovic
Dimitar Iliev Dimitrov
Robin Staab
Martin Vechev
Venue: ICLR 2022 Poster (2022)
Keywords: federated learning
privacy
gradient leakage

Abstract: Federated learning is an established method for training machine learning models without sharing training data. However, recent work has shown that it cannot guarantee data privacy as shared gradients can still leak sensitive information. To formalize the problem of gradient leakage, we propose a theoretical framework that enables, for the first time, analysis of the Bayes optimal adversary phrased as an optimization problem. We demonstrate that existing leakage attacks can be seen as approximations of this optimal adversary with different assumptions on the probability distributions of the input data and gradients. Our experiments confirm the effectiveness of the Bayes optimal adversary when it has knowledge of the underlying distribution. Further, our experimental evaluation shows that several existing heuristic defenses are not effective against stronger attacks, especially early in the training process. Thus, our findings indicate that the construction of more effective defenses and their evaluation remains an open problem.


Published as a conference paper at ICLR 2022
BAYESIAN FRAMEWORK FOR GRADIENT LEAKAGE
Mislav Balunović, Dimitar Dimitrov, Robin Staab, Martin Vechev Department of Computer Science
ETH Zurich
{mi s slav.av.o balunovic, dimitar, dimitrov,
robin. staab, martin. vechev/@inf..o ctt...c.eee..cchh
ABSTRACT
Federated learning is an established method for training machine learning models without sharing training data. However, recent work has shown that it cannot guarantee data privacy as shared gradients can still leak sensitive information. To formalize the problem of gradient leakage, we propose theoretical framework that enables, for the first time, analysis of the Bayes optimal adversary phrased as an optimization problem. We demonstrate that existing leakage attacks can be seen approximations of this optimal adversary with different assumptions on the probability distributions of the input data and gradients. Our experiments confirm the effectiveness of the Bayes optimal adversary when it has knowledge of the underlying distribution. Further, our experimental evaluation shows that several existing heuristic defenses are not effective against stronger attacks, especially early in the training process. Thus, our findings indicate that the construction of more effective defenses and their evaluation remains an open problem.
INTRODUCTION
Federated learning (McMahan et al. 2017 has become a standard paradigm for enabling users to collaboratively train machine learning models. In this setting, clients compute updates on their own devices, send the updates to central server which aggregates them and updates the global model. Because user data is not shared with the server or other users, this framework should, in principle, offer more privacy than simply uploading the data to a server. However, this privacy benefit has been increasingly questioned by recent works (Melis al. 2019 Zhu al. 2019 Geiping al., 2020 Yin et al. (021)) which demonstrate the possibility of reconstructing the original input from shared gradient updates. The reconstruction works by optimizing candidate image with respect to loss function that measures the distance between the shared and candidate gradients. The at- tacks typically differ in their loss function, their regularization, and how they solve the optimization problem. Importantly, the success of these attacks raises the following key questions: (i) what is the theoretically worst-cuse attack?, and (ii) how do we evaluate defenses against gradient leakage?
This work In this work, study and address these two questions from both theoretical and prac- tical perspective. Specifically, we first introduce a theoretical framework which allows us to measure the expected risk adversary has in reconstructing an input, given the joint probability distribution of inputs and their gradients. We then analyze the Bayes optimal adversary. which minimizes this risk and show that solves specific optimization problem involving the joint distribution. Further, we phrase existing attacks (Zhu et al.] 2019 Geiping et al., 2020; Yin et al. 202T) as approxima- tions of this optimal adversary, where each attack can interpreted as implicitly making different assumptions on the distribution of gradients and inputs, in turn yielding different loss functions for the optimization. In our experimental evaluation, we compare the Bayes optimal adversary with other attacks, those which do not leverage the probability distribution of gradients, and we find that the Bayes optimal adversary performs better, as explained by the theory. We then turn to practical evaluation and experiment with several recently proposed defenses (Sun al.) 2021 Gao et al.) 2021 Scheliga et al.) 202T) based on different heuristics and demonstrate that they do not protect from gradient leakage against stronger attacks that design specifically for each defense. Interest- ingly, we find that models are especially vulnerable to attacks early in training, and thus we advocate that defense evaluation should take place during and not only the end of training. Overall, our

---

Published as a conference paper at ICLR 2022
findings suggest that creation of effective defenses and their evaluation is a challenging problem, and that our insights and contributions can substantially advance future research the area.
Main contributions Our main contributions are:
Formulation of the gradient leakage problem in a Bayesian framework which enables phras- ing Bayes optimal adversary as an optimization problem.
Interpretation of several prior attacks as approximations of the Bayes optimal adversary, each using different assumptions for the distributions of inputs and their gradients.
Practical implementation of the Bayes optimal adversary for several defenses, show- ing higher reconstruction success than previous attacks, confirming our theoretical re- sults. We make our code publicly available at
bayes-framewon fwamework- Leaakaa
Evaluation of several existing heuristic defenses demonstrating that they do not effectively protect from strong attacks, especially early in training, thus suggesting better ways for evaluating defenses in this space.
RELATED WORK
We now briefly survey some of the work most related to ours.
Federated Learning Federated learning (MeMahan et 2017) was introduced way train machine learning models in decentralized settings with data coming from different user devices. This new form of learning has caused much interest in its theoretical properties (Konečný et al.] 2016a) and ways to improve training efficiency (Konečný et al. (0166). More specifically, besides decentralizing the computation on many devices, the fundamental promise of this approach pri- vacy, as the user data never leaves their devices.
Gradient Leakage Attacks Recent work (Zhu et al. 2019 Geiping et al. 2020) has shown that adversarial server can reliably
such privacy assumptions, in fact, do not hold in practice, recover an input image from gradient updates. Given the gradient produced from an image æ and corresponding label y. they phrase the attack between the original gradient and the gradient of randomly initialized input image x' and label y/:
minimization problem over the ly-distance
(x" 1²) arg min VV(ho(z)))) Vi(ho (2)) (z)),
(w'
Here he is a neural network and denotes a loss used to train the network, usually cross-entropy. Follow-up works improve on these results by using different distance metrics such cosine similar- ity and input-regularization (Geiping et al., 2020), smarter initialization (Wei et al., 2020), normal- ization (Yin al. (202)) and others (Mo al. 2022 Jeon et al.) 2021). A significant improvement proposed by Zhao et al. (2020) showed how to recover the target label y from the gradients alone, reducing Eq. I to optimization over x' only. In Section 4 we show how these existing attacks can be interpreted as different approximations of the Bayes optimal adversary.
Defenses In response the rise of privacy-violating attacks on federated learning, many defenses have been proposed (Abadi et al.) 2016); Sun et al., [021; Gao et al.) [22T]) Except for DP-SGD (Abadi al. 2016). a version of SGD with clipping and adding Gaussian noise, which differen- tially private, they all provide none or little theoretical privacy guarantees. This is partly due to the fact that no mathematically rigorous attacker model exists, and defenses are empirically evaluated against known attacks. This also leads to wide variety of proposed defenses: Soteria (Sun et al.) 2021) prunes the gradient for a single layer, ATS (Gao et al., (02T) generates highly augmented input images that train the network to produce non-invertible gradients, and PRECODE (Scheligal al., 2021) uses a VAE to hide the original input. Here we do not consider defenses that change the communication and training protocol (Lee et al. 2021
Wei et al. 2021.
2

---

Published as a conference paper at ICLR 2022
3 BACKGROUND
Let X C Rdd be input space, and let he X' y be neural network with parameters classifying an input x to a label y in the label space y. We assume that inputs (x, are coming from distribution D with marginal distribution p(a)). In standard federated learning, there are clients with loss functions 4 4,..., In. who are trying to jointly solve the optimization problem:
min Σ B( (i(ho(z))
To ease notation, we will assume single client throughout the paper, but the same reasoning can be applied to the general n-client case. Additionally, each client could have different distribution D, but the approach is again easy to generalize to this case. In a single training step, each client i first computes Vol, (ho((ho(3; ), yi) on batch of data (zz, V/). then sends these to the central server that performs a gradient descent step to obtain the new parameters 8' = a E Vol:(ho(z;) (x;4, y:), where is learning rate. We will consider a scenario where each client reports, instead of the true gradient Vol, (ho,(ho(z;)) (x;). vt). a noisy gradient sampled from distribution p(g(z)) which we call defense mechanism. The purpose of the defense mechanism is to add enough noise hide the sensitive user information from the gradients while retaining high enough informativeness so that can be used for training. Thus, given some noisy gradient g. the central server would update the parameters as l = 0 og (assuming = mentioned above). Typical examples of defenses used in our experiments in Section 6 each inducing p(g|æ)) include adding Gaussian or Laplacian noise to the original gradients, as well as randomly masking some components of the gradient. This setup also captures the common DP-SGD defense (Abadi et al. (0166) where p(g[æ) is a Gaussian centered around the clipped true gradient. Naturally, p(x) and plg ( together induce joint distribution p(z, g). Note that a network that has no defense corresponds to the distribution p(g|z)) which is concentrated only on the true gradient at
4 BAYESIAN ADVERSARIAL. FRAMEWORK
Here describe our theoretical framework for gradient leakage in federated learning. As show, our problem setting captures many commonly used defenses such as DP-SGD (Abadi et al. 2016) and many attacks from prior work (Zhu al. 2019 Geiping al., 2020).
Adversarial risk We first define the adversarial risk for gradient leakage and then derive the Bayes optimal adversary that minimizes this risk. The adversary can only observe the gradient g and tries to reconstruct the input x that produced g. Formally, the adversary is a function f R²* + X mapping gradients inputs. Given some (I,,9) (9) sampled from the joint distribution p(x, g). the adversary
outputs the reconstruction f(g) and incurs loss [(x, (g)), which is a function L : X x X' + R. Typically, we will consider a binary loss that evaluates
if the adversary's 's output close to the original input, and otherwise. If the adversary wants to reconstruct the exuc! input x, we can define the loss to be C(x,, x(x=x')== := 1 1zz'' denoting with the indicator function. If the adversary only wants to get to some 8- 5-neighbourhood of the input x in the input space, a more appropriate definition of the loss is f(x, x²) := =d(=,=")>> In this section, we will assume that the distance d is the fg-distance, but the approach can be generalized to other notions well. This definition is well suited for image data, where & distance captures our perception of visual closeness, and for which the adversary often obtain reconstruction that is very close the original image, both visually and in the & space. Note that if we let ó approach O in our second definition of the loss, we essentially recover the first loss. We can now define the risk R(f) of the adversary
R(f) = Ez.g [C(x, f(g)) = Ex-p(a) E. Egrriatz) f()))
Bayes optimal adversary We consider white-box adversary who knows the joint distribution p(z, g). as opposed to the weaker alternative based on security through obscurity, where adversary does not know what defense is used, and therefore does not have a good estimate of p(z; g). We also do not consider adversaries that can exploit other vulnerabilities in the system to obtain extra infor- mation. Let consider the second definition of the loss for which L(x, f()) == 1| 2-6-
3

---

Published as a conference paper at ICLR 2022
(8)
6
xx(2)
6
Forig
((1)
[ZEoiggz 6)
Figure 1: Example of gradient leakage attack. Bayes optimal adversary randomly initializes image 2((1) and then optimizes for the image with the highest log p(g[z) log in its 6-neighborhood. The adversary has loss if the final reconstruction is outside the ball B (x 6), and O otherwise.
We can then rewrite the definition of the risk as follows:
R() = Ex, [f(x, [()))
= Eylo A11=-fia)laad
= 1g / p(rlg)) - 1())l(0) dx
BEy [
JX(B(S(O))))) p(xlg) dz
1 Eg B(F()).))
p(xlg) dr.
Here (gf 8) denotes the fz-ball of radius 5 around f()). Thus, an adversary which wants to minimize their risk has maximize Egg JBC(()),)) p(xlg) dx, meaning that the adversarial function can be defined as f(g) arg max J p(x (g] dz. Intuitively, the adversary predicts zo which has the highest likelihood that one of the ipputs in its 6-neighborhood was the original input that produced gradient g. Note that if we would let 5 + O, then f(g) - arg max, p(zlg), which would be the solution for the loss that requires the recovered input exactly match the original input. As we do not have the closed form for p(xlg), it can be rewritten using Bayes' rule:
f(g) = arg max ang m
p(xlg) dr
B(
p(g]=)p(x) dr p(g)
arg max X
B(
arg max x
p(g(x)p(() dr
B(
arg max m
log
p(glz)p(z) dx
(2)
B(
Computing the optimal reconstruction now requires evaluating both the input prior p(x) and the conditional probability p(g[z), which is determined by the used defense mechanism. Given these two ingredients, Eq. 2 then provides us with a way to compute the output of the Bayes optimal adversary by solving optimization problem involving distributions p(z) and p(g]z))
Approximate Bayes optimal adversary While Eq. (2) provides a formula for the optimal ad- versary in the form an optimization problem, using this adversary for practical reconstruction is difficult due three main challenges: (i) need to know the exact prior distribution p(z), (ii) in general computing the integral over the 8-ball around 2o is intractable, and (iii) we need to solve the optimization problem over X. However, address each of these challenges by introducing
4

---

Published as a conference paper at ICLR 2022
Attack
Prior p(x)
Conditional p(glz)
DLG (Zhu et al. 2019) Inverting Gradients (Geiping et al.
Uniform
Gaussian
2020
TV
Cosine
GradInversion (Yin èt al. 2021)
TV + Gaussian + DeepInv
Gaussian
Table I: Several existing attacks be interpreted instances of Bayesian framework. We show prior and conditional distribution for corresponding losses that each attack uses.
appropriate approximations. We first apply Jensen's inequality to the logarithm function:
log log C / p(gl=)p(z) da
max #0E
max 20EX
C B(x0.5)
(log + log p(x)) da
B(x0.5)
Here C 1/8d is normalization constant. For image data, approximate log p(x) using the total variation image prior, which has already worked well for Geiping et al. (2020). Alternatively, we could estimate it from data using density estimation models such as PixeICNN (van den Oord al., 2016) or Glow (Kingma & Dhariwal 2018). We then approximate the integral over the d-ball via Monte Carlo integration by sampling points 21, m., Ty uniformly in the ball obtain the objective:
1 logr(gla;) log p(:)).
Finally, as the objective is differentiable, we can gradient-based optimizer such as Adam (Kingma & Ba 2015), and obtain the attack in Algorithm [ Fig. [ shows a single run of this adversary which is initialized randomly, and gets closer to the original image at every step.
Existing attacks as approximations of the Bayes optimal adversary We now describe how existing at- tacks can be viewed as different ap- proximations of the Bayes optimal adversary. Recall that the optimal ad- versary f searches for the 8-ball with the maximum value for the integral of log p(g]z) + log p(x). Next we show previously proposed attacks can in fact be recovered by plugging in dif- ferent approximations for p(glt) and log p(x) into Algorithm estimating the integral using l samples located the center of the ball. For example, suppose that the defense mechanism adds Gaussian noise to the gradient, which corresponds to the conditional probability p(g[z) oV(Vol(he v), o² I). This implies log p(g]z) C 1 (where log p(z) constant), the problem Eq. ( simplifies to minimizing the & distance be- tween g and V Vol(ha y). recovering exactly the optimization problem solved by Deep Leakage from Gradients (DLG) (Zhu et 2019) Inverting Gradients (Geiping et al. (2020) uses total vari- ation (TV) image prior for log p( and cosine similarity instead of l2 to measure similarity between "" (el(hh(m)))) (
Algorithm Approximate Bayes optimal adversary
x01)) attack init() for = to m do
Sample 3i, .... 26 uniformly from ((x(1,
(i+1) < (() + +av Σ logp(gla,) + log p(x,)
end for return (m)
202 l19 Vol(he y) 12. where C' is constant. Assuming a uniform prior
gradients. Cosine similarity corresponds to the distribution logl(glæ) =
which also requires the support of the distribution to be bounded. This happens, e.g., when gradients are clipped. GradInversion (Yin et al., 202I) introduces more complex prior based on combi- nation of the total variation, Ea norm of the image, and DeepInversion prior while using le norm to measure distance between gradients, which corresponds to Gaussian noise, as observed before. work with unnormalized densities, multiplying by the normalization constant
Note that does not change the argmax in Eq. (2). We summarize these observations in Table [[ showing the respective conditional and prior probabilities for each of the discussed attacks.
5

---

Published as a conference paper at ICLR 2022
5 BREAKING EXISTING DEFENSES
In this section we provide practical attacks against three recent defenses, showing they are not able to withstand stronger adversaries early in training. While in Section we have shown that Bayes op- timal ad adversary is the optimal attack for any defense (each with different p(g[z)) we now show how practical approximations of this attack can be used attack existing defenses (more details in Ap- pendix A. In Section 6 we show experimental results obtained using our attacks, demonstrating the need to evaluate defenses against closest possible approximation to the optimal attack.
Soteria The Soteria defense (Sun al. 2021 perturbs the intermediate representation of the input at a chosen defended layer/ of the attacked neural network H _n + I with L layers of size B1 ..... nn and input size ng. Let X and X'' E Io denote the original and reconstructed images on H and ha. : R + R= denote the function between the input of the 1th layer of H and the output of the jth For the chosen layer L. Sun et al. 2 denotes the inputs to that layer for the images X and X' with r (X)) and r' 1o,1- (X'). respectively, and aims to solve
max |X -x s.t. ||r lo S.
(4)
Intuitively, Eq. (4) is searching for a minimal perturbation of the input to layer that results in maximal perturbation of the respective reconstructed input X'. Despite the optimization being over the intermediate representation r', for an attacker who observes neither r nor r' the defense amounts to a perturbed gradient at layer I. In particular let VW = (VW, VW2.... VW } be the set of gradients for the variables in the different layers of H. [an et al. (2021) first solves Eq. (4) to obtain r'. It afterwards uses this r' generate perturbed gradient at layer denoted with VW/ Notably r' is not propagated further through the network and hence the gradient perturbation stays local to the defended layer Hence, to defend the data X, Soteria's 's clients send the perturbed set of gradients VW" = {VW,.... V VW, VW VWL} in place of VW. Sun et al. (2021) show
that Soteria is safe against several attacks from prior work (Zhu et al. 2019 Geiping et al.) 2020).
In this work, we propose to circumvent this limitation by dropping the perturbed gradients VW from VW to obtain VW = (VW1.... VWi-1, VWi+1.... WWL]. As long as VW* contains enough gradients, this allows an attacker to compute an almost perfect reconstruction of X. Note that the attacker does not know which layer defended, but can simply run the attack for all.
Automated Transformation Search The Automatic Transformation Search (ATS) (Gao et al. 2021) attempts to hide sensitive information from input images by augmenting the images during training. The key idea is to score sequences of roughly 3 to 6 augmentations from AutoAugment library (Cubuk et al. 2019) based on the ability of the trained network to withstand gradient- based reconstruction attacks and the overall network accuracy. Similarly to Soteria, Gao et al. (2021) also
demonstrate that ATS is safe against attacks proposed by ZZuu et al. (2019) and Geiping et al. (202))
In this work we show that, even though the ATS defense works well in later stages of training, in ini- tial communication rounds we can easily reconstruct the input images using Geiping et al. (2020)ss 's attack. Our experiments in Section 6 on the network architecture and augmentations introduced in Gao et al.] (202T) indicate that attacker can successfully extract large parts of the input despite heavy image augmentation. The main observation is that the gradients of a randomly initialized network allow casier reconstruction independent of the input. Only after training, which changes gradient distribution, can defenses such as ATS empirically defend against gradient leakage.
PRECODE PRECODE (Scheliga et al. 2021) is a proposed defense which inserts a variational bottleneck between two layers in the network. Given an input æ, PRECODE first encodes the input into a representation
(bb2) from a latent Gaussian distribution. Then, they compute new latent representation á = D(b), and finally obtain the output & = ( For this defense we focus on an MLP network, which Scheliga et al. (021) evaluates against several attacks and shows that images cannot be recovered.
Fxx) then samples bottleneck features
Our attack is based on the attack presented in Phong et al. (017)), later extended by Geiping et al. 2020) it shows that in most cases the inputs to an MLP can be perfectly reconstructed from the gradients. Next we detail this attack as presented in Phong et al. (2017):/Geipiin et al. (2020). Let
the output of first layer of the MI. (before the activation) be y/o Agæ bo. with x Ra0

---

Published as a conference paper at ICLR 2022
No Augmeetations
Segera
Augmentaations
40
Augmentations Hybrid
PSNR
10
PSNR
20
20
15
10
2
10 50
100 200 500 1000
10
Training Sleps
Training Epechs
(a) ATS
(b) Soteria
Figure 2: PSNR obtained by reconstruction attacks on ATS and Soteria during the first 1000 steps and 10 epochs, respectively, demonstrating high gradient leakage early in training.
yo: by E p and Ag _" Assume a client sends the unperturbed gradients aAA that input layer. Let (vo), and (bo); denote the hh entry in these vectors. By the chain rule we have:
di and di for
Tbe
dl dl d(ya)s
dl
d(bo); d(yo); d(bo); d(yo)+ di Combining this result with the chain rule for a(Ao).. we further obtain:
d(yo)s d(Ao).. d(yo); d(Ao).. d(yo); x²
di
dl d(bo);
di where (Au)a,: denotes the i-th line of the matrix Ag. Assuming a(ha); * 0, we can precisely calculate
xT
dl as there is typically at least one non-zero entry in bp-
d(bo);
d(A.)).
6 EXPERIMENTAL EVALUATION
We now evaluate existing defenses against strong approximations of the Bayes optimal adversary.
Evaluating existing defenses In this experiment, we evaluate the three recently proposed defenses described in Section 5 Soteria, ATS and PRECODE, on the CIFAR-I0 10 dataset (Krizhevsky) (2009). For ATS and Soteria, we use the code from their respective papers. In particular, we evaluated AT' on their ConvNet implementation, a network with 7 convolutional layers, batch-norm, and ReLU activations followed by a single linear layer. We consider 2 different augmentation strategies for ATS, well as a hybrid strategy. For Soteria, we evaluate our own network architecture with 2 convolutional and 3 linear layers. The defense is applied on the largest linear layer, directly after the convolutions. Both attacks implemented using the code from Geiping et al. (2020), and using cosine similarity between gradients, the Adam optimizer (Kingma & Bal 2015) with a learning rate 0.1 total variation regularization of 10-5 for ATS and 4 x 10 for Soteria, as well 2000 and 4000 attack iterations respectively. We perform the attack on both networks using batch size 1. For PRECODE, which has no public code, we use our own implementation. We use MLP architecture consisting of layers with 500 neurons each and ReLU activations. Following Scheliga et al. (202T) apply the variational bottleneck before the last fully connected layer of the network.
The results of this experiment are shown in Fig. 2 and Soteria for the first 10 epochs, measuring peak signal-to-noise ratio (PSNR) of the reconstruc- tion obtained using our attack at every step. In all of our PRECODE experiments we obtain perfect reconstruction with PSNR values > 150 so we do not show PRECODE on the plot. We can ob- serve that, generally, each network becomes less susceptible to the attack with the increased number of training steps. However, early in training, networks are very vulnerable, and images can be re- constructed almost perfectly. Fig. 3 visualizes the first 40 reconstructed images obtained using our attacks on Soteria, ATS and PRECODE. We can see that for all defenses, our reconstructions are very close to their respective inputs. This indicates that proposed defenses do not reliably protect privacy under gradient leakage, especially in the earlier stages of training. Our findings suggest that creating effective defenses and properly evaluating them remains key challenge.
We attack ATS for the first 1000 steps of training
7

---

Published as a conference paper at ICLR 2022
(a) Original
(b) Soteria
(c) ATS
(d) PRECODE
Figure 3: Images obtained by running attacks on Soteria, ATS, and PRECODE on the CIFAR-10 dataset after the 1hth training step. We can observe that reconstructed images are very close the original ones, meaning that these defenses do not protect privacy early in the training.
Table 2: PSNR of reconstructed images using approximate Bayes optimal attack, and attacks based on E2, l,, and cosine distance between the gradients. We confirm our theoretical results that Bayes optimal attack performs significantly better when it knows probability distribution of the gradients.
Train step 0
Train step 500
Defense
Bayes
f2
f
Cos
Bayes
f2
6
Cos
Gaussian
19.63
19.95
19.39
19.70
16.06
16.18
15.86
15.39
Laplacian
19.48
18.86
19.99
18.64
15.95
15.47
16.31
15.14
MNIST
Prune Gauss
18.40
14.44
13.95
16.72
16.42
13.15
13.24
14.47
Prune Lap
18.27
14.01
13.86
15.21
16.52
13.60
13.54
14.57
Gaussian
21.86
21.81
21.49
21.53
17.94
18.64
18.00
18.34
Laplacian
21.87
21.02
21.89
20.87
18.41
18.37
19.11
18.02
CIFAR-10
Prune + Gauss
20.73
16.60
16.20
18.73
18.67
15.61
15.27
16.74
Prunc + Lap
20.54
16.25
16.13
17.40
18.51
15.32
15.24
15.51
Bayes optimal adversary in practice In the following experiment, we compare the approxima- tion of Bayes optimal adversary described in Section 4 with three variants of Inverting Gradients attack (Geiping et al. 2020), based on b2, f1, and cosine distance, on the MNIST (Lecun et al., 1998) and CIFAR-IO datasets, Recall from Table [ that the l2, E and cosine attacks make different assumptions on the probability distribution of the gradients, which are not optimal for every de- fense. To this end, we evaluate the performance of these attacks on diverse set of defenses. For both MNIST and CIFAR-IO. we train CNN with ReLU activations, and attack gradients created on image batches of size 1 both at the initialization and step 500 of the training. For all attacks, we use anisotropic total variation image prior, and we initialize the images with random Gaussian noise. Further experiments using prior based on pixel value range are presented in Appendix A.44 We optimize the loss using Adam (Kingma & Ba 2015) with exponential learning rate decay. For this experiment, we modified Eq. (3) to add an additional weighting parameter , for the prior as Et= log p(g]z,) + 8 l(r)(rr) which compensates for the imperfect image prior selected.
Experimentallly we observed that the success the different attacks is very sensitive to the choice of their parameters. Therefore, to accurately compare the attacks, we use grid search that selects the optimal parameters for each of them individually. In particular, for all attacks we tune their initial learning rates and learning rate decay factors well as the weighting parameter B. Further, for the l2, & and cosine attacks the grid search also selects whether exponential layer weighting, introduced by [Geiping et al. (2020), is applied on the gradient reconstruction loss. In this experiment, the approximation the Bayes optimal adversary uses & Monte Carlo samples. Further ablation study on the effects of l is provided in Appendix A.6 Our grid search explores 480 combinations of parameters for the &,, & and cosine attacks, and 540 combinations for the Bayesian attacks, respectively. We provide more details this parameter search in Appendix A.3
8

---

Published as a conference paper at ICLR 2022
For this experiment, we consider defenses for which exactly compute conditional probability distribution of the gradients for a given imput, denoted earlier as p(g (z). First we consider two defenses which add Gaussian noise with standard deviation a 0.1 and Laplacian noise of scale b = 0.1 to the original gradients, thus corresponding to the probability distributions p(g(æ) = Nig (ho(ho y), of I) and p(g]=)) Lap(g Vol(ho(z), y), bT). respectively. Note that the Gaussian defense with additional gradient clipping corresponds to the well known DP-SGD algorithm (Abadi et al. 2016). We also consider defenses based on soft pruning that set random 50% entries of the gradient to 0, and then add Gaussian or Laplacian noise as in previous defenses. In Table [ we report the average PSNR values on the first 100 images of the training set for each combination of attack and defense. First, we observe, as in Fig. 2 that network is significantly more vulnerable early in training. We can observe that Bayes optimal adversary generally performs best, showing that the optimal attack needs to leverage structure of the probability distribution of the gradients induced by the defense. Note that, in the case of Gaussian defense, lz and Bayes attacks equivalent up constant factor (as explained in Section D. and it is expected that they achieve a similar result. In all other cases, Bayes optimal adversary outperforms the other two attacks. Overall, this experiment provides empirical evidence for our theoretical results from Section 4
Approximations of Bayes optimal adversary In this experiment, we compare the Bayes optimal ad- versary with attacks that have suboptimal approxi- mations of p(g(æ) and p(æ). As vision datasets have complex priors p(x) which we cannot compute actly, we now consider a synthetic dataset where p(z) is 20-dimensional unit Gaussian. We define the true label y == arg max(Wx) where W is a fixed random matrix, and perform attack
pixi-Lap. pigiki-Gauss
plxt=G Gauss
pixi-Lap. pigixi-iap ptxl=Gauss pig
Distamcc
2-layer
0
25
50
100
125
150
175
200
Step
MLP defended by adding Laplacian noise with a 0.1 scale. Thus, p(x) is a Gaussian, and p(g] (z) is Lapla- cian. In this study, we consider 4 different variants of the attack, obtained by choosing Laplacian or Gaussian for prior and conditional. Fig. 4 shows, for each attack, the distance from the original input for 200 steps. We can observe that Bayes optimal attack (with Gaussian prior and Laplacian conditional) converges significantly closer to the original input than the other attacks, providing empirical evidence for our theoretical results in Section 4
Figure Ablation with the Bayes attack.
Discussion and future work Our experimental evaluation suggests the evaluation of the existing heuristic defenses against gradient leakage is inadequate. As these defenses are significantly more vuinerable at the beginning of the training, we advocate for their evaluation throughout the entire training. and not only at the end. Ideally, defenses should be evaluated against the Bayes optimal adversary, and if such adversary cannot be computed, against properly tuned approximations such as the ones outlined in Section 4. Some interesting future work includes designing better approxima- tions of the Bayes optimal adversary, e.g. by using better priors than total variation, and designing an effective defense for which is tractable to compute the distribution p(glz) (glz) so that can be eval- uated using Bayes optimal adversary similarly to what we did in Table We believe our findings can facilitate future progress in this area, both in terms of attacks and defenses.
7 CONCLUSION
We proposed a theoretical framework to formally analyze the problem of gradient leakage, which has recently emerged as an important privacy issue for federated learning. Our framework enables us to analyze the Bayes optimal adversary for this setting and phrase it as an optimization problem. We interpreted several previously proposed attacks as approximations of the Bayes optimal adver- sary, each approximation implicitly using different assumptions on the distribution over inputs and gradients. Our experimental evaluation shows that the Bayes optimal adversary is effective in prac- tical scenarios in which it knows the underlying distributions. We additionally experimented with several proposed defenses based on heuristics and found that they do not offer effective protection against stronger attacks. Given our findings, believe that formulating an effective defense that balances accuracy and protection against gradient leakage during all stages of training remains an exciting open challenge.
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (ACM CCS), pp. 308-318, 2016, 2016. doi: 121144//976749.2978833
Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. Autoaugment: Learning augmentation strategies from data. In CVPR, pp. 113-123. Computer Vision Foundation / IEEE, 2019.
Wei Gao, Shangwei Guo, Tianwei Zhang, Han Qiu, Yonggang Wen, and Yang Liu. Privacy- preserving collaborative learning with automatic transformation search. In CVPR, pp. 114-123. Computer Vision Foundation IEEE, 2021.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients how casy is it to break privacy in federated learning? In NeurlPS, 2020.
Jiahui Geng, Yongli Mou, Feifei Li, Qing Li, Oya Beyan, Stefan Decker, and Chunming Rong. Towards general deep leakage in federated learning. arXiv preprint 1rX1v2210000004. 2021.
Jinwoo Jeon, Jaechang Kim, Kangwook Lee, Sewoong Oh, and Jungseul Ok. Gradient inversion with generative image prior. In International Workshop on Federated Learning for User Privacy and Data Confidentiality. 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In NeurIPS, 2018
Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimiza- tion: Distributed machine learning for on-device intelligence. CoRR, abs/1610.02227, 2016a.
Jakub Konečný, H. Brendan McMahan, Felix X. Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. CoRR, abs/1610.05492, 2016b.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- bbased learning applied document recog- nition. Proceedings of the IEEE, 86(1 1):2278--23, 1998. doi: 10. (0.1109/5.726791.
Hongkyu Lee, Jeehyeong Kim, Seyoung Ahn, Rasheed Hussain, Sunghyun Cho, and Junggab Son. Digestive neural networks: novel defense strategy against inference attacks in federated learn- ing. Computers and Security, 109; 10237,, 2021.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communictiion-fficieent learning of deep networks from decentralized data. In AISTATS, vol- ume 54 of Proceedings of Machine Learning Research, pp. 1273-1282. PMLR, 2017.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In IEEE Symposium on Security and Privacy, pp. 691- 706. IEEE, 2019.
Fan Mo, Anastasia Borovykh, Mohammad Malekzadeh, Hamed Haddadi, and Soteris Demetriou. Quantifying information leakage from gradients. CoRR, abs/2105.13929, 2021.
Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy- preserving deep learning: Revisited and enhanced. In ATIS, volume 719 of Communications in Computer and Information Science, pp. 100-110. Springer, 2017.
Daniel Scheliga, Patrick Mder, and Marco Secland. Precode a generic model extension to prevent deep gradient leakage, 2021.
10

---

Published as a conference paper at ICLR 2022
Jingwei Sun, Ang Li, Binghui Wang. Huanrui Yang. Hai Li, and Yiran Chen. Soteria: Provable defense against privacy leakage in federated learning from representation perspective. In CVPR, pp. 9311 199999. Computer Vision Foundation IEEE, 2021.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves, Conditional image generation with pixelenn decoders. In NIPS, 2016.
Wenqi Wei, Ling Liu, Margaret Loper, Ka Ho Chow, Mehmet Emre Gursoy, Stacey Truex, and Yanzhao Wu. A framework for evaluating client privacy leakages in federated learning. In ES- ORICS (I), volume 12308 of Lecture Notes in Computer Science, pp. 545-566. Springer, 2020.
Wenqi Wei, Ling Liu, Yanzhao Wu, Gong Su, and Arun Iyengar. Gradient-leakage resilient federated learning. CoRR, abs/2117.011555 2021.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In CVPR. Computer Vision Founda- tion / IEEE, 2021.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients, 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In NeurlPS, 2019.

---

Published as a conference paper at ICLR 2022
(onv2d(in_caannelse out_channels=32, kerne (trre stride=,, padding=1) paddin.
ReLUO
AvgPoo12d(kernel size=2, siz= stride=2)
Conv2d(in channels=32, out channels=64, kernel size=1, padding=1)
ReLUU)
AvgPool2@(kernelsiiee stride=2)
Linear(in (eaturrsnffauurs=5111 out (0utt)
ReLUO
Linear(in.features wew.furesuuee00....... out out_ffetuees=1000 1000))
ReI.U()
Linear(in features=1000, out features=10)
Table 3: ConvBig architecture.
(a) Original
(b) Direct attack defended largest layer
(c) Our attack on the defended largest layer
(d) Our attack on defended second largest layer
Figure 5: We compare the results of direct attack on the defended network 5b with our attack 5c Furthermore we show our attack on a network in which the second largest layer is defended 5d All networks have been trained for 10 steps.
A
APPENDIX
Here we provide additional details for our work.
A.1 SOTERIA
For Soteria, we built directly on the Sun et al. (022) repository using their implementation of the defense and the included Inverting Gradient (Geiping al. 2020 library. Testing was conducted primarily on the ConvBig architecture presented below:
The architecture consists of convolutional "feature extractor" followed three linear layers, shown in Table We chose this architecture because (i) it is simple in structure while providing rea- sonable accuracies on datasets such CIFARIO and (ii) because (unlike many small convolutional networks) it has more than one layer with a significant fraction of the overall network parameters. In particular, the first linear layer roughly contains around 80% of the network weights and the second one 20%. This is particularly relevant for our attack, as cutting out large layer of weights will negatively affect the reconstruction quality. As in practice uncommon to have the majority of weights in single layer, we believe our architecture provides a reasonable abstraction. To justify this, we show in Fig. 5 that no matter which the larger two layers is defended by Soteria, we can attack the network successfully. For all Soteria defenses set the pruning rate (refer to (Sun al.) 2021) for details) to 80%. Note however that our attack works independent of the pruning rate, as always remove the entire layer.
12

---

Published as a conference paper at ICLR 2022
Conv2d(3, width, http://eeee..adiiniiiiiaaahhhaao ReLU0
Conv2d(1 width, 2 width, gemmel_sizee33,padddngg---- BatchNorrm200,
Bachhooom ReLUO
Conv2d(2 width, 2 * width, kernel.size=3, pa.ddigg) BatchNon22d) ReLUO
Conv2d(2 * width, 4 width, kernel size=3, padding=1), BatchNorm2d)), ReLUO
Conv2d(4 width, 4 width, kernel size=3, ) BatchNorm220) ReLUO
Conv2d(4 width, 4 width, kernel Batpe:e_aaadiigggg_aaaaaiooonoe
ReLUO
MaxPool2d(3),
Conv2d(4 width, 4 * width, kemel size=3,, padding=1), Batchhohhhhoodd ReLUO
Conv2d(4 width, 4 width, kemel padding==).. BatchNor22... ReLUO
Conv2d(4 * width, 4 width, kernel_size=3, padding=1), BatchNorm2d(), ReLUU
MaxPool2d(3)
Linear(36 * #with, 3)
Table 4: ConvNet architecture. Our benchmark instatiation uses a width of 64.
We use the Adam optimizer with learning rate of 0.1 with decay. As similarity measure, cosine similarity. Besides cutting one layer of weights, we weigh all gradients equally. The total variation regularization constant is 4 10-1, We initialize the initial guess randomly and only try reconstruction once per image, attacking one image at a time. For training, we used a batch size of 32. We trained the network without defense and applied the defense at inference time speed up the training.
A.2 AUTOMATED TRANSFORMATION SEARCH
For ATS, we built upon the repository released alongside (Gao et al., 2021). We use the ConvNet architecture with width of 64 also proposed in (Gao et al.) (021)) and train with the augmentations "7-4-15", "21-13-3", "21-13-3+7-4-15" which perform the best on ConvNet with CIFARI00. We present the ConvNet architecture in Table
For reconstruction, we use the Adam optimizer with a learning rate of 0.1 with decay. As similarity measure,
cosine similarity weighing all gradients equally. The total variation regularization constant is X 10-5, We initialize the initial guess randomly and only try reconstruction once per image, attacking one image at time. For training, used batch size of 32 and trained individually for every set of augmentations.
In Figure 6 we show how the quality of the reconstructed inputs degrades during training. Neverthe- less we can recover high-quality inputs during the first 10 to 20 training steps.
A.3 PARAMETER SEARCH FOR THE ATTACKS
Since the range of for which the different attacks perform well is wide, prior to the grid search we need to calculate a range of reasonable values for 3 for each of the attacks. We do this by searching for values of [ for which the respective attack is optimal. The rest of the parameters of these attacks are set to the same set of initial values. The values of B considered are in the range [1 x 10-7, x 10a] and are tested on logarithmic scale. The final range for 3 used the grid search is given by the range [0.5,,, 2Ba.), where 3, is the values that produced the highest PSNR
A.4 EXPERIMENT WITH STRONGER PRIORS
In this section, we compare the effects of different image priors on the gradient leakage attack. In particular, compared a simple anisotropic total variation image prior and prior that combination of the same anisotropic total variation and an error term first introduced in Geng et al. (202 that encourages the reconstructed image's pixels to be in the [0, 1] range:
x) log p(x) = x- Liv + (1 111-00 Φ) L
L = | clip(a, 0, 1)112+
where o E [0, 1] balances the total variation eтror term Liv and the pixel range enror term Lp. and clip(ær, 0, 1) clips the values of x in the [0, range. We compared both priors the MNiST
13

---

Published as a conference paper at ICLR 2022
(a) Reconstruction after 5 training step
(b) Reconstruction after 10 training steps
(c) Reconstruction after 20 training steps
(d) Reconstruction after 50 training steps
Figure 6: Our reconstruction results after several training steps with ATS, batch_size 32, and aug- mentations 7-4-15. We can see how the visual quality starts decline after 10 steps and at 50 steps one can no longer reliably recover the input.
Table 5: Attack using la distance metric on MNIST trained for 500 steps with and without using stronger prior that takes into account that pixels should in the [0, 1] range.
Defense
l + LTv
l2 + LTv + Lqu.a]
Gaussian
16.18
16.30
Laplacian
15.47
15.76
Prune Gauss
13.15
13.63
Prune Lap
13.60
14.10
network trained for 500 steps from Table 2 usiig l conditional distribution. We compared on defenses considered in Table [ We used the same grid search procedure, but extended it to search for optimal value of o as well. The results are presented in Table 3 From the results, we that the addition of the pixel range prior term improved the attack for all defenses, showing that despite its simplicity the pixel range prior is very effective.
A.5 OUR PRACTICAL ATTACKS AS APPROXIMATIONS OF THE BAYES OPTIMAL ADVERSARY
In this subsection we interpret our attacks on Soteria, ATS and PRECODE different approxima- tions of Bayes optimal adversary. We summarize this in Table 6, and also describe details of each approximation in separate paragraphs.
Attack on Soteria In this section, we interpret the attack Soteria that we presented in Section 5 as an approximation to the Bayes optimal attack this setting. As described in Section 5 Soteria takes the original network gradient VW (VW1. VW2.... .... VWz} } and defends it by substituting the gradient at layer with the modified gradient VW7 producid
VW VW
m #1
Ok
ifk=
where Im is the subvector of the client gradient update vector y. corresponding to the mht layer of the network.
In Section 4 we derived the Bayes optimal attacker objective as
&
(log p(gla:))) + log
k
14

---

Published as a conference paper at ICLR 2022
Attack
Prior p(a)
Conditional p(g[z)
Soteria attack
Layerwise Gaussian
ATS attack
TV
Cosine
PRECODE attack
Arbitrary
Dirac delta mixture
Table 6: Our attacks heuristic defenses can be interpreted instances of our Bayesian framework. We show prior and conditional distribution for corresponding losses that each attack uses.
Next, show how approximate this objective for the Soteria defense. Since Soteria modifies the network gradients per-layer, we model the gradient probability distribution p(g(z) of our Bayes optimal attack as per-layer separable:
p()l))
II p(gml (z)). m=1
To model the design choice of Soteria to not change gradients of layers different than 1, we set
p(gmla (r)) NVVW Fm) with om O, for all layers * 1. With that choice, the Bayes objective becomes
(log((g(z)) + (log((>)))
k
((gl(()))) + Blogp(a;) +
Σ log logg((mla:))
(5)
k
m3H1
k k
= C +
[log((a(z;) + + Blogp(x1)
1
Σ
o2 11mm (ol(ha(z)..))13
2
As 0m + O, the contribution of log p(g](z) to the overall objective becomes exceedingly smaller. Therefore, one can choose om = for some a > O, such that the optimal solution of the Bayes objective in Eq. (5) is very close to the optimal solution of the surrogate objective:
(Blog((x;)
ΣΣ
Im
Vol(ho(z:). v)1
(6)
2
The objective in Eq. 6 the exact objective that we optimize to produce the Soteria attacks in Sec- tion 5
we attack ATS using the attack from Geiping et al. (020). This attack corresponds to the proor with total variation, and conditional distribution is based on cosine similarity.
Attack on ATS As described in Section 5
Attack on PRECODE Our attack on PRECODE is based on the observation that the distribution p(zlg) for PRECODE corresponds is always unique input Hp that produced this gradient, and the density p(zlg) is then Dirac delta centered at do (uniqueness of the solution follows from the derivation Phong et al. (2017) for linear layers). As shown in Eq. P this corresponds to maximizing the value of p( g)) In this case, we can in fact solve for the maximum of p(zlg) exactly using attack described in Section 5 and we do not have rewrite the objective using Bayes rule and apply optimization. In terms of p(x) and p(g(æ), we can have arbitrary p(x), while p(g[æ) is mixture Dirac delta distributions where the mixture comes from the random samples of the VAE used by PRECODE.
Dirac delta distribution. Namely, given gradient g. there
A.6 ABLATION STUDY OF THE MONTE CARLO ESTIMATION
In this experiment we perform an ablation study for the Monte Carlo estimation of the adversarial objective. More specifically, vary the number of samples & used for the Monte Carlo estimate.
15

---

Published as a conference paper at ICLR 2022
19.75
19.50
19.25
PSNR
19.00
18.75
18.50
18.25
18.00
10
20
30
70
Steps
Figure 7: Ablation for the number of samples in the Monte Carlo estimate.
We show the PSNR value averaged over 100 different sampled images for each step of the opti- mization process in Figure 7 We used convolutional network on the MNIST dataset, at the initial training step, defended by Gaussian noise of standard deviation I and using 6 = 9.0 in the defini- tion of the adversarial risk. The results indicate that higher number of samples in the Monte Carlo estimate results faster convergence of the attacker towards the reconstruction image closer to the original. This is expected estimate using larger number of samples results in an Monte Carlo estimator that has lower variance.
A.7 RELATIONSHIP BETWEEN BAYES ATTACK AND LANGEVIN DYNAMICS
In this section, we discuss the relationship between our proposed Bayes optimal attack and Langevin updates proposed by Yin et al. (202T) We focus on the case where the number of samples in the Monte Carlo estimate h
shown Algorithm | our update
(log p(g logp(z1)),
where zı is point sampled from the ball B (x, 8) uniformly at random. At the same time, the update with Langevin dynamics proposed by Yin et al. (2021) given by
x レ (a,,7 +
( (logg((g)))) logp(z)))
where y (0, I) noise sampled from unit Gaussian, and 0, is noise scaling coefficient. As Yin et al. (202)) explain, we can view Langevin updates as a method to encourage exploration and diversity. The main differences between the updates are: (i) we sample a point 3ı from the ball B(x,8) uniformly at random, while Langevin update in Yin et al. (2021) samples noise 1) from unit Gaussian, (ii) we compute gradient the sampled point 21 B x. 8). while Langevin update evaluates gradient directly at and adds noise to afterwards. Note that Yin et al. (2021) do not provide ablation study investigate the effect of Langevin update, so do know important is in the overall optimization. Overall, both Monte Carlo estimation and Langevin updates can be seen as different ways to add noise to the optimization process, though currently we do not see way to formally view Langevin updates Bayesian prior.
16


Reviews:
[{"review_id": "m1PohuuzLMG", "reviewer": "Reviewer_CRtt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "After reading author's response: review remains unaltered.\n\nStrengths:\n1. The unification of several existing attacks is interesting. \n2. The paper is well-written.\n\n\nWeaknesses:\n1. It would be better if the authors could introduce more details of how to approximate p(x) and p(g|x). Estimating it from data is known to be a hard problem. Does this mean the attack may fail on a large-scale dataset?\n", "summary_of_the_paper": "This paper provides a gradient leakage attack via Bayes optimal adversary. The authors also demonstrate that existing attacks can be seen as approximations of Bayes optimal adversary. Empirically, this paper shows some heuristic defenses fail under the proposed attack.", "summary_of_the_review": "This paper is overall well-written. The proposed method is simple and empirically effective.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "j7HAUfNZOEO", "reviewer": "Reviewer_DWMa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) The theoretical formulation proposed in the paper is intuitive and appears to be mostly correct. \n\n2) The key limitation of the work is the lack of much practical impact, due to the following reasons.\n\n(a) The paper admits that for most sophisticated defense mechanisms, it is not possible to define a Bayes optimal adversary. Instead the paper falls back on designing custom attacks for each of these defense mechanisms.\n\n(b) It is also not surprising that most existing defense mechanisms are ineffective against early stages of training, which are most sensitive to the input data. As training progresses and the models converge, the inputs are expected to have little impact on the gradients. So, this cannot be considered as a significant finding.\n\n(c) From Table 2, it appears that the Bayes optimal attack does not produce significantly better reconstructions compared to existing attacks (l_2 and cos), except in the case where pruning is involved (last two rows). Furthermore, the paper does not claim reduction in the computational complexity of the attack using the Bayes optimal attack. Thus, there appears to be no major benefit from using the Bayesian framework.\n\n(d) The ablation study (Figure 4) shows that even for the simplest distributions, mis-specification of the prior and conditional distributions can lead to sub-optimal reconstruction. For real-world inputs, it may be simply impossible to precisely define the prior and conditional distributions to carry out a Bayes optimal attack.", "summary_of_the_paper": "The paper proposes a theoretical Bayesian framework for the problem of gradient leakage in federated learning. It shows that recent gradient leakage attacks are approximations of the Bayesian framework with different prior distributions for the input and conditional distributions of the gradients given the input. The paper also claims that recent defense mechanisms are not good enough, especially during the early stages of training. ", "summary_of_the_review": "The proposed Bayesian formulation for gradient leakage is reasonable, but has little practical significance because it neither provides an easier way to break sophisticated defense mechanisms nor does it lead to better reconstructions. The design of attacks against recent defense mechanisms (Soteria, ATS, and Precode) may be a useful contribution, but does not merit a publication in itself. \n\nSince the authors have managed to show that the proposed attacks can be derived as approximations of the Bayes optimal adversary and are not purely custom designed, I have updated the rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "as7ZXjVmbGG", "reviewer": "Reviewer_cPAG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ This paper is well-written, easy to follow. Provides sufficient background and literature review.\n+ The framework is straightforward yet makes sense. The mathematical results seem to be derived correctly. \n+ Linking the existing attacks to their framework is interesting. \n-  As mentioned in the paper the Bayesian adversary loss is intractable, thus providing some kind of approximation for it would have been better. \n- It is not clear to me even though as mentioned by the authors we do not have any sort of effective defense, what are the authors' suggestions in that regard as the paper is based on that. Could the authors please elaborate on that?\n- Section 6, the existing attack section contains interesting results but I fail to see the relevance to this paper. It has taken most of the section without justifying its importance to the proposed framework. \n- This paper suffers from the lack of a wide range of experimental results to back up the claims. The experimental results must cover compassion to other techniques in addition to accenting the importance of each experiment. \n-  The lack of provided details in the paper, however, the code for experimental results must not be considered due to the late submission of their codes. Authors should provide more details regarding their experiments. \n\n", "summary_of_the_paper": "This paper introduces a framework to address issues in the leakage of information through gradients. This paper claims to provide Bayes optimal adversary as an optimization problem which other attacks can be considered as an approximation of this framework. Some experiments are provided to support their claims. ", "summary_of_the_review": "- Even though the assumptions seem to be correct, the contribution of the paper is extermley simple and straightforward. \n- The experimental results seem to be insufficient as it is only compared to two other methods. The experiment section is written poorly and the importance of each section is not justified. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "FZ1kPsj0lBe", "reviewer": "Reviewer_6AUa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In summary I think this is an interesting submission. The unification of existing attacks in a Bayesian sense (while maybe not thrilling from a purely theoretical point of view) is certainly a good idea and clearly described and executed and, to me, the strongest part of this work. The experiments in Sec. 6 then show that this viewpoint can have tangible benefits. However I am torn regarding the section on breaking existing defenses. While these are great attacks and I liked to read this section, it does feel disconnected from the rest of the paper. None of the insights from the Bayesian framework seem to help in creating these adaptive attacks.\n\nSome more comments and questions regarding the experimental evaluation below:\n\n* After the introduction of the Bayesian framework in Sec. 4 it does seem a bit disappointing that the only Bayesian attack evaluated in Sec. 6 operates with k=1 Monte-Carlo samples. It would have been great to see an ablation of the effects (or the lack thereof) of a range of values of k.\n* If k=1 and p(g|x) is Gaussian, how does the proposed attack differ from the attack of Yin et al. with Langevin noise (from a random Gaussian distr.)? The order of gradient computation and noise addition differs, but it is not clear to me if this is a meaningful distinction in the Bayesian sense.\n* For Sec. 6, Table 2 it would be interesting to complete the selection of objectives and also show results for an $\\ell^1$ objective, which would be the optimal adversary for Laplacian noise when $\\delta\\to0$.\n* The data distribution $p(x)$ appears somewhat under-utilized after its introduction in Sec. 4. For example, for the TV regularization, the optimal $\\alpha$ could have been tuned by fitting the distribution to measured data instead of grid-searching all possible alpha in terms of final PSNR of the attack. Also, would the results in Table 2 be strengthened or weakened when using a stronger data prior such as e.g. the  DeepInversion prior from Yin et al.?\n* In terms of discussions of priors, Table 1 should also contain a mention of Jeon et al., who do spend time on searching for closer approximations to $p(x)$ in several ways.\n* One particular prior effect that appears in some previous work but sometime falls under the radar is the prior knowledge that all real images are drawn from the bounded set  $[0,1]^n$  (for $n$ the number of pixels). Did the authors experiment with this (although arguably relatively weak) prior as well?\n* Also in Table 2, the authors also compare to pruning. However it is not clear to me whether $p(g|x)$ takes the pruning into account in those rows?", "summary_of_the_paper": "The submission \"Bayesian Framework for Gradient Leakage\" discusses privacy attacks against federated learning based on gradient inversion. In the first part of this work, a range of existing attacks is unified in a Bayesian setting. In the middle part of this work, three existing defenses are analyzed and broken by adaptive attacks. In the final part of this work, an attack based on estimation of the optimal Bayesian adversary is evaluated against several classical defenses and compared to other objectives.", "summary_of_the_review": "I have a positive outlook on this submission. Classifying and unifying previous attacks in a single framework and clarifying the optimal choice of regularizer and objective is a helpful contribution to the community. \n\nI do have several questions regarding the experimental evaluation that I would like for the authors to answer and I would be interested in a clarification why the inclusion of Sec. 5 should be part of this submission and its connection to the overall Bayesian framework discussed in this work.\n\n--------------------------------------------- After Author Responses: --------------------------------------------\n\nThe authors have adressed my concerns during the response period. This lead me to increase my score from 6 to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper formalizes the problem of gradient leakage through a Bayesian framework. They show that existing attacks can be viewed as approximations of a Bayesian optimal adversary. The empirical results show that heuristic defences are not good against stronger attacks and that the early part of the training is particularly vulnerable. There was a lively discussion in the reviews and rebuttal and the outstanding questions of the reviewers have been answered.
Decision: Accept (Poster)