Evaluating Gradient Inversion Attacks and Defenses in Federated Learning
Authors: Yangsibo Huang
Samyak Gupta
Zhao Song
Kai Li
Sanjeev Arora
Venue: NeurIPS 2021 Oral (2021)
Keywords: Federated learning
gradient inversion
deep leakage from gradients

Abstract: Gradient inversion attack (or input recovery from gradient) is an emerging threat to the security and privacy preservation of Federated learning, whereby malicious eavesdroppers or participants in the protocol can recover (partially) the clients' private data. This paper evaluates existing attacks and defenses. We find that some attacks make strong assumptions about the setup. Relaxing such assumptions can substantially weaken these attacks. We then evaluate the benefits of three proposed defense mechanisms against gradient inversion attacks. We show the trade-offs of privacy leakage and data utility of these defense methods, and find that combining them in an appropriate manner makes the attack less effective, even under the original strong assumptions. We also estimate the computation cost of end-to-end recovery of a single image under each evaluated defense. Our findings suggest that the state-of-the-art attacks can currently be defended against with minor data utility loss, as summarized in a list of potential strategies.

Evaluating Gradient Inversion Attacks and Defenses in Federated Learning
Yangsibo Huang Princeton University Princeton, NJ 08540 cangsslo@pprincetnn edu
Samyak Gupta Princeton University Princeton, NJ 08540
Zhao Song Adobe Research San Jose, CA 95110
samyakgåcs wwwppetoo..neeeeeeeeeeeeee samyakg@ss.princeton.cc
zsongdadobe.com com
Kai Li
Sanjeev Arora
Princeton University Princeton, NJ 08540
Princeton University Princeton, NJ 08540
1i0cs. u.8cs.princeton. princeton. edu
arora@cs
wworaacs.prinetttoo wwwnceton.....m. edu
Abstract
Gradient inversion attack (or input recovery from gradient) an emerging threat to the security and privacy preservation of Federated learning, whereby malicious eavesdroppers or participants in the protocol can recover (partially) the clients' private data. This paper evaluates existing attacks and defenses. We find that some attacks make strong assumptions about the setup. Relaxing such assumptions can substantially weaken these attacks. We then evaluate the benefits of three proposed defense mechanisms against gradient inversion attacks. We show the trade-offs privacy leakage and data utility of these defense methods, and find that combining them in an appropriate manner makes the attack less effective, even under the original strong assumptions. We also estimate the computation cost of end-to-end recovery of a single image under each evaluated defense. Our findings suggest that the state-of of-the-art attacks can currently be defended against with minor data utility loss, as summarized in a list of potential strategies.
Introduction
Federated learning [McMahan et al., 2016, Kairouz et al., 2021] a framework that allows multiple clients in distributed environment to collaboratively train neural network model at central server, without moving their data to the central server. At every training step, each client computes a model update -ic., gradient- on its local data using the latest copy of the global model, and then sends the gradient to the central server. The server aggregates these updates (typically by averaging) to construct global model, and then sends the model parameters to all clients. By allowing clients to participate in training without directly sharing their data, such protocols align better with data privacy regulations such as Health Insurance Portability and Accountability Act (HIPPA) [Act, 1 1996], California Consumer Privacy Act (CCPA) [Legislature, 2018), and General Data Protection Regulation [Commission, 2018).
While sharing gradients was thought to leak little information about the client's private data, recent papers [Zhu et al., 2019, Zhao et al., 2020, Geiping et al., 2020, Yin al., 2021] developed "gradient inversion attack" by which an attacker eavesdropping on a client's 's communications with the server begin reconstruct the client's private data. The attacker can also be malicious participant in the Federated Learning scheme, including a canest-but-curtg but-risos client who wishes reconstruct private data of other
server who wishes to reconstruct
private data of clients, clients. These attacks have been shown to work with batch sizes only up to 100 but even so they have
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

created doubts about the level of privacy ensured in Federated Learning. The current paper seeks to evaluate the risks and suggest ways to minimize them.
Several defenses against gradient inversion attacks have been proposed. These include perturbing gradients [Zhu al., 2019, Wei et al., 2020] and using transformation for training data that clients can apply on the fly [Zhang et al., 2018a, Huang et al.. 2020J. More traditional cryptographic ideas including secure aggregation [Bonawitz al., 2016] or homomorphic encryption [Phong al., 2018] for the gradients can also be used and presumably stop any eavesdropping attacks completely. They will not be studied here due to their special setups and overhead.
We are not aware of a prior systematic evaluation of the level of risk arising from current attacks and the level of security provided by various defenses, as well as the trade-off (if any) between test accuracy, computation overhead, and privacy risks.
The paper makes two main contributions. First, we draw attention to two strong assumptions that current gradient inversion attack [Geiping et al., 2020] implicitly makes. We show that by nullifying these assumptions, the performance of the attack drops significantly and can only work for low- resolution images. The findings are explored in Section 3 and already imply some more secure configurations in Federated Learning (Section 6).
Second, we summarize various defenses (Section 4) and systematically evaluate (Section 5) some of their performance of defending against a state-of-the-ar gradient inversion attack, and present their data utility and privacy leakage trade-offs. We estimate the computation cost of end-to-end recovery of single image under each evaluated defense. We also experimentally demonstrate the feasibility and effectiveness of combined defenses. Our findings are summarized as strategies to further improve Federated Learning's s security against gradient inversion attacks (Section 6).
In Appendix B, we provide theoretical insights for mechanism of each evaluated defense.
2 Gradient Inversion Attacks
Previous studies have shown the feasibility of recovering input from gradient (i.e. gradient inversion) for image classification tasks, by formulating as an optimization problem: given neural network with parameters 0, and the gradient V VoLe(x", ,y""",""""""""")"" computed with a private data batch (x", .y" e Raxd x R (b, being the batch size, image size), the attacker tries to recover x E Rixd an approximation of x":
arg min Caraa( O, VoLo(o", + aRaux())
(1)
The optimization goal consists of two parts: Lgrad (2; 0, VoLo (x,,""")) enforces matching of the gradient of recovered batch x with the provided gradients Lo yy) and Raux (x) regularizes the recovered image based on image prior(s).
[Phong et al., 2017] brings theoretical insights on this task by proving that such reconstruction is possible with a single-layer neural network. [Zhu et al., 2019] is the first to show that accurate pixel- level reconstruction is practical for maximum batch size of 8. Their formulation uses Ey-distance as (...) but no regularization term Raxx(z). The approach works for low-resolution CIFAR datasets [Krizhevsky et al., 2009], with simple neural networks with sigmoid activations, but cannot scale up to high-resolution images, or larger models with ReLU activations. A follow-up [Zhao al., 2020] proposes simple approach extract the ground-truth labels from the gradient, which improves the attack but still cannot overcome its limitations.
With careful choice of Egead and Raux (x). [Geiping et al., 2020] substantially improves the attack and succeeds in recovering a single ImageNet [Deng et al., 2009] image from gradient: their approach uses cosine distance as Lgrad. and the total variation as Raxx (z). Their approach is able reconstruct low-resolution images with a maximum batch size of 100 or a single high-resolution image. Based on [Geiping et al., 2020], [Wei et al., 2020] analyzes how different configurations in the training may affect attack effectiveness.
A more recent work [Yin et al., 2021] further improves the attack on high-resolution images, by introducing to (xux(x) a new image prior term based on batch normalization [loffe and Szegedy, 2015] statistics, and regularization term which enforces consistency across multiple attack trials.

---

An orthogonal line of work [Zhu and Blaschko, 2021] proposes to formulate the gradient inversion attack a closed-form recursive procedure, instead of an optimization problem. However, their implementation can recover only low-resolution images under the setting where batch size 1.
3 Strong Assumptions Made by SOTA Attacks
3.1 The state-of-the-art attacks
Two recent attacks [Geiping et al., 2020, Yin et al., 2021] achieve best recovery results. Our analysis focuses on the former as the implementation of the latter is not available the time of writing this paper. We plan to include the analysis for the latter attack in the final version of this paper if its implementation becomes available.
[Geiping et al., 2020]'ss attack optimizes the following objective function:
(VoCo(x, y), VeLo "e""""") = RTV(T
arg min VoCola, v) VaCo (xt w*)l
(2)
where ( is the inner-product between vectors, and Rrv()) is the total variation of images.
We notice that Geiping et al. has made two strong assumptions (Section 3.2). Changing setups to
invalidate those assumptions will substantially weaken the attacks (Section 3.3). We also summarize whether other attacks have made similar assumptions in Table 1.
3.2 Strong assumptions
We find that previous gradient inversion attacks have made different assumptions about whether the attacker knows Batch normalization statistics or private labels, as shown in Table 3,2. Note that [Geiping et al.. 2020]'s attack makes both strong assumptions.
Assumption 1: Knowing BatchNorm statistics. Batch normalization (BatchNorm) [loffe and Szegedy, 2015] is a technique for training neural networks that normalizes the inputs to a layer for every mini-batch. It behaves differently during training and evaluation. Assume the model has L batch normalization layers. Given x², a batch of input images, we use xf denote the input features to the (-th BatchNorm layer, where E [L]. During training, the 1-th BatchNorm layer normalizes zî based on the batch's 's mean mean mean(2f) and variance var(="). ), and keeps running estimate of mean and variance of all training data points, denoted by 14 and of. During inference, {1))/2 and {o?)= are used normalize test images. In the following descriptions, we leave out {-)/, for simplicity (i.c. use μl, a² to denote {MHEE {o M1=1 and mean(x"). var(z*) denote (mean(z))) (var(zt)][
We notice that [Geiping et al., 2020]'s implementation' assumes that BatchNorm statistics of the private batch, i.e., mean(x*) var(**). ), are jointly provided with the gradient. Knowing BatchNorm statistics would enable the attacker to apply the same batch normalization used by the private batch on his recovered batch, to achieve better reconstruction. This implicitly increases the power of the attacker, as sharing private BatchNorm statistics are not necessary in Federated learning [Andreux al., 2020, Li et al., 2021].
Note that this assumption may be realistic in some settings: 1) the neural network shallow, thus does require using BatchNorm layers, 2) the neural network is deep, but adapts approaches that normalize batch inputs with a fixed mean and variance (as alternative to BatchNorm), e.g. Fixup initialization [Zhang et al., 2019].
Assumption 2: Knowing or able to infer private labels. Private labels are not intended to be shared in Federated learning, but knowing them would improve the attack. [Zhao et al., 2020] finds that label information of a single private image can be inferred from the gradient (see Section 3.3 for details). Based on this, [Geiping et al., 2020] assumes the attacker knows private labels (see remark at the end of Section 4 in their paper). However, this assumption may not hold true when multiple will show in the next section.
images in batch share the same label,
'The official implementation of [Geiping et al., 2020J: https//dthilloooommmmmSigggnnnnt

---

Assumptions
[Zhu et al., 2019]
[Zhao et al., 2020]
[Geiping al., 2020]
[Yin al., 2021]
Knowing BN statisties Knowing private labels
N/A
N/A No
Yes"
No
Yes
Nof
Table 1: Assumptions of gradient inversion attacks. i; its evaluation uses a simple model without a BatchNorm layer; F; it proposes method to infer private labels, which works when images in a batch have unique labels (see Section 3.3); ": although the paper discusses a setting where BatchNorm statistics unknown, its main results assume knowing BatchNorm statistics.
3.3 Re-evaluation under relaxed assumptions
We re-evaluate the performance of the gradient inversion attack in settings where two assumptions above relaxed. For each relaxation, re-design the attack (if needed) based on the knowledge that the attacker has.
Relaxation 1: Not knowing BatchNorm statistics. We refer to the previous threat model as BNexact. where the attacker knows exact BatchNorm statistics of the private batch. We consider more realistic threat model where these statistics are not exposed, and re-design the attack based on it.
Threat model. In each training step, the client normalizes its private batch x using the batch's 's mean mean(x" (xx) and variance var(x"), keeps the running estimate of mean and variance locally as in [Li al., 2021], and shares the gradient. The client releases the final aggregated variance o² all training data points at the end of training. Same as before, the attacker has access the model and the gradient during training.
μt, and aggregated
Re-design A: Brprosys attacker naively uses μ and o². A simple idea that the attacker uses (J,, a²) the proxy for (mean(x*) ), var(x*)) )), and uses them to normalize x, his guesses of the private batch. Other operations of the gradient inversion attack remain the same as before. However, Figure 1.d and 1.h show quor-quality reconstruction with this re-design.
Re-design B: infifers attacker infers (mean var(x")) based on (Js, o2). more reasonable attacker will try to infer (mean(x") var(x")) while updating x, his guesses of the private batch, and uses (mean(x). vx)( to normalize the batch. this case, (μ, o²) could be used as a prior of BatchNorm statistics to regularize the recovery, suggested in [Yin et al., 2021]:
arg min 1 (VoLolx, y). VoCa .²)))
aTVRRT(()) + RBNREN(Z)
(3)
VooLo v) VoLo (x*, w²)l
where RBN (x) = E, ||maa((r))) mill + Σ, var(()) -
We tune OBN and present the best result in Figure 1.c and 1.g (see results of different OBNN in Appendix A). As shown, for a batch of low-resolution images, BNinfer gives a much better reconstruction result than BN, proxy, but still cannot recover details of the private batch when compared with BNxxact- The result for a single high-resolution image is worse: the attacker fails to return recognizable reconstruction with BN This suggests not having access to BatchNorm statistics of the private batch already weakens the state-of-the-an gradient inversion attack.
(a) Original (b) BNowaet. (c) BN (d) BN
(c) Original
(f) BN
(g) BNuffer (h) BN
Figure Attacking batch of 16 low-resolution images from CIFAR-10 (a-d) and single high- resolution image from ImageNet (e-h) with different knowledge of BatchNorm statistics. Attack weakened when NatchNorm statisties are not available (c, versus b, and g, h versus f). See Appendix A for more examples and quantitative results.

---

Original
Private labels known
Private labels unknown
30%
batch
batch size 32
Percentage of batch
13%4
batch size 64
batch size 128
20%
1.%
Class Index
(a) Distribution of labels in batch
(b) Reconstructions with and without private labels
Figure 2: Attack is weakened when private labels are not available. (a) shows that for CIFAR- 10, when the batch size is large, many images in the batch belong to the same class, which essentially weakens label restoration [Zhao et al., 2020, Yin et al., 2021]. (b) visualizes reconstructed batch of 16 images with and without private labels known. The quality of the reconstruction drops without knowledge of private labels.
Not knowing private labels. [Zhao et al., 2020] notes that label information of single private image can be computed analytically from the gradients of the layer immediately before the output layer. [Yin et al., 2021] further extends this method to support recovery of labels for batch of images. However, if multiple images in the private batch belong to a same label, neither approach can tell how many images belong to that label, let alone which subset of images belong to that label. Figure 2a demonstrates that with CIFAR- 10, for batches of various sizes many of the training samples to be have the same label, and the distribution of labels is not uniform and hence, inferring labels becomes harder and the attack would be weakened. Figure 2b we evaluate the worst-case for attacker in this setting by comparing recoveries where the batch labels are simultancously reconstructed alongside the training samples.
Relaxation 2:
possible for
4 Defenses Against the Gradient Inversion Attack
Several defense ideas have been proposed to mitigate the risks of gradient inversion.
4.1 Enerypt gradients
Cryptography-bett bbsed approaches encrypt gradient to prevent gradient inversion. [Bonawitz et al., 2016) presents a secure aggregation protocol for Federated learning by computing sum of gradient vectors based secret sharing [Shamir, 1979]. [Phong et al., 2018] proposes using homomorphic encryption to encrypt the gradients before sending. These approaches require special setup and can be costly to implement.
Moreover, with secure aggregation protocol, an honest-but-curious server can still launch the gradient inversion attack the summed gradient vector. Similarly, an caness-but-curiouu client can launch the gradient inversion attack on the model returned by the server to reconstruct other clients' private data, even with homomorphic encryption.
As alternatives, two other types of defensive mechanisms have been proposed to mitigate the risks of attacks on plain-text gradient.
4.2 Perturbing gradients
Gradient pruning. When proposing the first practical gradient inversion attack. [Zhu et al., 2019] also suggests a defense by setting gradients of small magnitudes to zero (i.c. gradient pruning). Based on their attack, they demonstrate that pruning more than 70% of the gradients would make the recovered images no longer visually recognizable. However, the suggested prune ratio is determined based weaker attacks, and may not remain safe against the state-of oftt-----he-ant art attack.
Adding noise to gradient. Motivated by DPSGD [Abadi et al., 2016] which adds noise to gradients to achieve differential privacy [Dwork, 2009, Dwork and Roth, 2014], [Zhu et al., 2019, Wei et al.,
5

---

2020] also suggests defending by adding Gaussian or Laplacian noise to gradient. They show that successful defense requires adding high noise level such that its accuracy drops by more than 30% with CIFAR-10 10 tasks. Recent works [Papernot et al., 2020, Tramèr and Boneh, 2021] suggest using better pre-training techniques and large batch size (e.g. 4, 096) to achieve a better accuracy for DPSGD training.
Since most DPSGD implementations for natural image classification tasks [Abadi et al., 2016, Papernot et al., 2020, Tramèr and Boneh, 2021] use a pre-training and fine-tuning pipeline, it is hard to fairly compare with other defense methods that can directly apply when training the model from scratch. Thus, we leave the comparison with DPSGD to future work.
4.3 Weak encryption of inputs (i.e. encoding inputs)
MixUp. MixUp data augmentation [Zhang et al., 2018a] trains neural networks on composite images created via linear combination of image pairs. It has been shown to improve the generalization of the neural network and stabilizes the training. Recent work also suggests that Mix increases the model's 's robustness to adversarial examples [Pang et al., 2020, Lamb al., 2019].
InstaHide. Inspired by MixUp, [Huang al., 2020] proposes InstaHide as light-weight instance- encoding scheme for private distributed learning. To encode an image x E R from a private dataset, InstaHide first picks k other images $2, $3, ......... 8k from that private dataset, or a large public dataset, and k random nonnegative coefficients {A, that sum to 1, and creates composite image 21 Σ X,8; (k is typically small, e.g., 4). A composite label is also created using the same set of coefficients. Then it adds another layer of security: pick random sign-flipping pattern E 1- and output the encryption ă o (Ayx + + Aos;), where coordinate- wise multiplication of vectors. The neural network is then trained on encoded images, which look like random pixel vectors to the human eye and yet lead good classification accuracy (< 6% accuracy loss on CIFAR-10, CIFAR- 100, and ImageNet).
Recently, [Carlini et al., 2020] gives an attack to recover private images of a small dataset, when the InstaHide encodings are revealed to the attacker (not in a Federated learning setting). Their first step to train neural network on public dataset for similarity annotation, to infer whether a pair of InstaHide encodings contain the same private image. With the inferred similarities of all pairs of encodings, the attacker then runs a combinatorial algorithm (cubic time in size of private dataset) to cluster all encodings based on their original private images, and finally uses a regression algorithm (with the help of composite labels) to recover the private images.
Neither [Huang et al., 2020] or [Carlini et al., 2020] has evaluated their defense or attack in the Federated learning setting, where the attacker observes gradients of the encoded images instead of original encoded images. This necessitates the systematic evaluation in our next section.
Evaluation of defenses
The main goal of our experiments is to understand the trade-off's between data utility (accuracy) and securely defending the State-of-the-aa gradient inversion attack even in its strongest setting, without any relaxation of its implicit assumptions. Specifically, we grant the attacker the knowledge of 1) BatchNorm statistics of the private batch, and 2) labels of the private batch.
We vary key parameters for each defense, and evaluate their performance in terms of the test accuracy, computation overhead, and privacy risks (Section 5.2). We then investigate the feasibility of combining defenses (Section 5.3). We also estimate the computation cost of end-to-end recovery of a single image under evaluated defenses (Section 5.4).
As the feasibility of the state-of-the-a attack [Geiping et al., 2020] on a batch of high-resolution images remain elusive when its implicit assumptions no longer hold (see Figure 1), we focus on the evaluation with low-resolution in trying to understand whether current attacks can be mitigated.
Only the labels of examples from the private dataset will get combined. See [Huang et al., 2020] for details.

---

5.1 Experimental setup
Key parameters of defenses. We evaluate following defenses CIFAR-10 dataset [Krizhevsky ct al., 2009] with ResNet-18 architecture [He et al., 2016].
GradPrune (gradient pruming): gradient pruning set gradients of small magnitudes to zero. We vary the pruning ratio in {0.5, 0.7, 0.9, 0.95, 0.99, 0.999).
MixUp encodes private image by linearly combining with other images from the training set. Following [Huang et al., 2020), we vary l in {4, 6), and set the upper bound of a single coefficient to 0.65 (coefficients sum to 1).
MixUp:
Intra-InstaHiee InstaHide [Huang et al., 2020] proposes two versions: Inter-InstaHide and Intra- Ittaa-nntaHHie. The only difference is that at the mixup step, Instaldet mixes up an image with images from public dataset, whereas Intra-InstaHide only mixes with private images. Both versions apply a random sign flipping pattern on each mixed image. We evaluate Intra-InstaHide in our experiments, which is a weaker version of InstaHide. Similar to the evaluation of Mix we vary i in (4, 6), and set the upper bound of single coefficient to 0.65. Note that InstaHide flips signs of pixels in the image, which destroys the total variation prior. However, the absolute value of adjacent pixels should still be close. Therefore, for the InstaHide defense, we apply the total variation regularizer on |.c], i.e. taking absolute value each pixel in the reconstruction.
We train the ResNet- 18 architecture on CIFAR-10 using different defenses, and launch the attack. We provide more details of the experiments in Appendix A.
The attack. We use a subset of 50 CIFAR-I0 images to evaluate the attack performance. Note that attacking MixUp and InstaHide involves another step to decode private images from the encoded images. We apply [Carlini al., 2020]'s 's attack here as the decode step. where the attacker needs to eavesdrop T epochs of training, instead of a single training step. We set T = 20 in our evaluation. We also grant the attacker the strongest power for the decode step to evaluate the upper bound of privacy leakage. Given MixUp or Intra-Instal-ide image which encodes k private images, we assume the attacker knows:
1. The indices of & images in the private dataset. In a realistic scenario, the attacker of [Carlini et al., 2020] would need to train neural network to detect similarity of encodings, and run a combinatorial algorithm to solve an approximation of this mapping.
2. The mixing coefficients for each of the k private image. In real Federated learning, this information is not available.
hyyper-parameter of the attack. The attack minimize the objective function given in Eq.3. We search arv in (0, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5] for all defenses, and apply the best choice for each defense: 0.05 for GradPrune, 0.1 for MixUp, and 0.01 for Intra-InstaHide. We apply OBN 0.001 for all defenses after searching it in (0, 0.0005, 0.001, 0.01, 0.05, 0.01). We optimize the attack for 10, 000 iterations using Adam [Kingma and Ba, 2015], with initial learning rate 0.1. We decay the learning rate by a factor of 0.1 at 3/8,5/8 7 of the optimization.
Batch size of the attack. [7hu et al., 2019, Geiping et al., 2020] have shown that a small batch size is important for the success of the attack. We intentionally evaluate the attack with three small batch sizes to test the upper bound of privacy leakage, including the minimum (and unrealistic) batch size and two small but realistic batch sizes, 16 and 32.
Metries for reconstruction quality. We visualize reconstructions obtained under different de- fenses. Following (Yin et al., 2021], we also use the learned perceptual image patch similarity (LPIPS) score [Zhang et al., 2018b] to measure mismatch between reconstruction and original images: higher values suggest more mismatch (less privacy lcakage).
7

---

Original
No defense
GradPrune
MixUp
Intra-Inelaffide
lesé
kmd,
Figure 3: Reconstruction results under different defenses with batch size being 1, 16 and 32. When batch size is 32, combining gradient pruning and Intra-InstaHic makes the reconstruction almost unrecognizable (the last column). See Figure 7 in Appendix for the full version.
GradPrune (p +MixUp + intra-InstaHidd
None
GradPrune (p)
MixUp (8)
Intra-InstaHide (k)
Parameter
0.5
0.7
0.9
0.95
0.99
0.999
4
6
6
==
k:
Test Acc.
93.37
93.19
93.01
90.57 89.92
88.61
83.58
92.31
90.41
90.04
88.20
91.37
86.10
Time (train)
1x
1.04x
1.06x
1.08%
1.10x
Attack batch
Avg. LPIPS
0.46
Best LPIPS
0.02
0.02
0.05
0.14
0.22
0.32
0.36
0.12
0.25
0.41
0.42
0.21
0.43
(LPIPS 0.16
0.08
0.06
0.08
0.07
0.09
0.07
0.09
Attack
Avg. LPIPS 1
0.45
0.46
0.47
0.51
0.55
0.58
0.61
0.34
0.31
0.62
0.63
0.46
0.68
Best LPTPS
0.18
0.19
0.19
0.31
0.43
0.47
0.51
0.11
0.13
0.41
0.44
0.22
0.54
(LPIPS std.)
0.12
0.03
0.07
Atlack batch size
32
Avg. LPIPS 1
0.45
0.46
0.48
0.52
0.54
0.58
0.63
0.50
0.49
0.69
0.69
0.62
0.73
Best LPIPS
0.18
0.18
0.22
0.31
0.43
0.48
0.54
0.31
0.28
0.56
0.56
0.37
0.65
(LPIPS std.)
0.11
0.11
0.09
0.07
0.05
0.04
0.04
0.10
0.10
0.06
0.07
0.10
0.05
leverage-cade (eakage)
leakage)
0.8
Avg. LPIPS
0.999
0.99
4m4, pm0.9
0.5
LPiPS
0.6
p.999
1g-4, 4, p-0.9
0.99
0.4
0.4
5 p=0.9
Gradirunciet
MaUUpp00
0.2
GradPrunc(p)
MixUpiki
Instatiiduk)
Mon-deferded
InstaHidudk
Hon-dalarded
88
92
82
86
90
94
Test Accuracy
Accuracy
Table 2: Utilit-security trade-off of different defenses. We train the ResNet- 18 model on the whole CIFAR- We evaluate the attack
dataset, and report the averaged test accuracy and running time of independent a subset of 50 CIFAR-10 images, report the LPIPS score (L: lower values suggest more privacy leakage). We mark the least-leakage defense measured by the metric in green.
5.2 Performance of defense methods
We summarize the performance of each defense Table 2, and visualize reconstructed images in Figure 3. We report the averaged and the best results for the metric of reconstruction quality, proxy for average-case and worst-case privacy leakage.
No defense. Without any defense, when batch size is 1, the attack can recover images well from the gradient. Increasing the batch size makes it difficult to recover well, but the recovered images are visually similar to the originals (see Figure 3).
Gradient pruning (GradPrune). Figure 3 shows that as the pruning ratio p increases, there are more artifacts in the reconstructions. However, the reconstructions are still recognizable even when pruning ratio p = 0.9, thus the previous suggestion of using 0.7 by [7hu et al., 2019] no longer safe against the state-of-the-a attack. Our results suggest that, for CIFAR- 10, defending the

---

strongest attack with gradient pruning may require the pruning ratio p= 2 0.999. As a trade-off, such high pruning ratio would introduce an accuracy loss of around 10% (see Table 2).
MixUp. MixUp introduces small computational overhead to training. MixUp with k: = 4 only has minor impact (-2%) on test accuracy. but is not sufficient to defend the gradient inversion attack (see Figure 3). Increasing k from 4 to 6 slightly reduces the leakage, however, the reconstruction is still highly recognizable. This suggests that MixUp alone may not be practical defense against the state-of-the-art gradient inversion attack.
Intra-InstaHide. Intra-InstaHide with k incurs an extra -2% accuracy loss compared with MixUp, but it achieves better defense performance: when batch size is 32, there are obvious artifacts and color shift in the reconstruction (see Figure 3). However, with batch size 32, Intra- Hida- alone also cannot defend the tiee-oofthhearr gradient inversion, as structures of private images are still vaguely identifiable reconstructions.
Appendix A provides the whole reconstructed dataset under MixUp and Intra-InstaHide.
5.3 Performance of combined defenses
We notice that two types of defenses (i.e perturbing gradient and encoding inputs) are complementary to each other, which motivates an evaluation of combining gradient pruning with MixUp or Intra- InstaHide.
As shown in Figure 3. when the batch size is 32, combining Intra-InstaHide (k: 4) with gradient pruning (p- 0.9) makes the reconstruction almost unrecognizable. The combined defense yields higher LPIPS score than using gradient pruning with p = 0.999, but introduces a smaller accuracy loss (-7% compared with a no-defense pipeline).
Note that our evaluation uses the strongest attack and relatively small batch sizes. As shown in Appendix A, invalidating assumptions in Section 3 or increasing the batch size may hinder the attack with an even weaker defense (e.g. with a lower p, or smaller k), which gives a better accuracy.
5.4 Time estimate for end-to-end recovery of single image
Table 3 shows time estimates for the end-to-end recovery of a single image in a Federated learning setting with GradPrune or InstaHide defense. We do not estimate for MixUp since it has been shown to be a weak defense (see Section 5.2).
Our time estimates consider three fairly small dataset sizes. The largest size in our estimate is a small fraction a dataset of ImageNet scale. We consider a client holds a dataset of N private images and participates in Federated learning, which trains a ResNet-18 model with batch size b = 128. Assumes that the resolution of the client's 's data is 32 x 32 x 3. If the attacker uses a single NVIDIA GefFrce RTX 2080 Ti GPU his computation resource, and runs gradient inversion with 10,000 iterations of optimization, then f, the running time for attacking a single batch is -0.25 GPU hours (batch size b has little impact on the attack's 's running time, but a larger makes the attack less effective).
Non-defended and gradient pruning. Recovering single image in a non-defended pipeline (or pipeline that applies gradient pruning alone as the defense) only requires the attacker to invert gradient of a single step of training, which takes time t.
InstaHide. When InstaHide is applied, the current attack [Carlini et al., 2020] suggests that recovering a single image would involve recovering the whole dataset first. As discussed in Section 4, Carlini et al.' 's attack consists of two steps: 1) recover InstaHide images from gradient of T epochs. This would take (NT /b) x GPU hours. 2) Run the decode attack [Carlini et al., 2020] on InstaHide images to recover the private dataset, which involves:
2a Train a neural network to detect similarity in recovered InstaHide images. Assume that training the network requires at least n recovered InstaHide images, then collecting these images by running gradient inversion would take (n/b) x GPU hours. The training takes 10 GPU hours according to [Carlini et al., 2020], so training the similarity network would take (n/6) t + 10 GPU hours in total.

---

Size of client's dataset (N)
No defense
GradPrune
Inst
5,000
934.48 46,579.01 (= 5.5 GPU years)
50,000
0.25
0.25
500,000
4,215,524.32 (As 493.4 GPU years)
Table 3: Time estimates (NVIDIA GeForce RTX 2080 Ti GPU hours) of recovering single image from the client's dataset using the state-of-the-ar gradient inversion attack [Geiping et al., 2020] under different defenses. We assume image resolution of the client's 8 data is 32 x 32 x 3.
2b Run the combinotorial algorithm to recover original images. Running time of this step has been shown be at least quadratic in m, the number of InstaHide encodings [Chen et al., 2021]. This step takes 1/6 GPU hours with = 5 x 10 Therefore for NT, the running time is at least 1/6 x 5x
NT )² GPU hours.
In total, attack InstaHide in this real-world setting would take (NT/b) L (n/b) I NT ) GPU hours. We use T = 50 (used by [Carlini et al., 2020J,, 10, 000 and give estimate in Table 3. As shown, when InstaHide is applied on small dataset (N 5, 000), the end-to-end recovery of a single image takes >> 3, 000 longer than in a no-defense pipeline or GradPrune pipeline; when InstaHide is applied on a larger dataset (N 500, 000), the computation cost for end-to-end recovery is enormous.
10 1/6 ( 10s
Conclusions
This paper first points out that some state-of-the-ar gradient inversion attacks have made strong assumptions about knowing BatchNorm statistics and private labels. Relaxing such assumptions can significantly weaken these attacks.
The paper then reports the performance of a set of proposed defenses against gradient inversion attacks, and estimates the computation cost of an end-to-end recovery of single image in different dataset sizes. Our evaluation shows that InstaHide without mixing with data from a public dataset combined with gradient pruning can defend the state-of-the-art attack, and the estimated time to recover a single image a medium-size client dataset (e.g. of 500,000 images) enormous.
Based on our evaluation of the attack by [Geiping et al., 2020] and multiple defenses for plain-text gradients, we have the following observations:
Using BatchNorm layers in your deep net but don't I share BatchNorm statistics of the private batch during Federated learning weakens the atlack. We have demonstrated in Section that exposing BatchNorm statistics to the attacker significantly improves the quality of gradient inversion. So a more secure configuration of Federated Learning would be to use BatchNorm layers, but do not share BatchNorm statistics in training, which has been shown feasible in [Andreux al., 2020, Li et al., 2021].
Using large batch size weakens the attack; a batch size smaller than 32 is not safe. We have shown that a larger batch size hinders the attack by making it harder to guess the private labels (Section 3) and to recover the private images (Section 5). Our experiments suggest that even with some weak defenses applied, a batch size smaller than 32 is not safe against the strongest gradient inversion attack.
with correct private labels
Combining multiple defenses may achieve better ment, for batch size of 32, combining InstaHide = 4) with gradient pruning (p 0.9) achieves the best quility-prvaacy trade-off, by making the reconstruction almost unrecogniz- able at a cost of ~7% accuracy loss (using InstaHide also makes the end-to-end recovery of a single image more computationally expensive). Best parameters would vary for different deep learning tasks, but we strongly encourage Federated learning participants to explore the possibility of combining multiple defensive mechanisms, instead of only using of them.
trade-off. In our experi-
We hope to extend our work by including evaluation of defenses for high-resolution images, the attack by Yin et al., 2021] (when its implementation becomes available), and more defense mechanisms including those rely adding noise to gradients.
10


Reviews:
[{"review_id": "waSdX8cTo-", "reviewer": "Reviewer_g1LK", "summary": "This paper has empirical study of federated learning with the following contributions:\n1. It evaluates the state-of-the-art gradient attack algorithms under weaker assumptions. For example, attackers do not have access to batch norm statistics and some private labels. Under these weaker assumptions, these attack algorithms perform significantly worse.\n2. It experiments various defense algorithms. For example, gradient pruning, MixUp, Weak-InstaHide, with the current state-of-the-art Carlini et al. 2020 attack algorithm. Further, it designs and evaluates a new method which combines various defense algorithms. Through empirical evaluations, it is shown that when attack batch size is 1, W-InstaHide is highly useful, and when attack batch size increases, the combined gradient pruning+W-InstaHide gives the best result.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. This paper demonstrates some practical perspective of the state-of-the-art gradient attack algorithms by removing certain strong assumptions and experimenting in more general conditions. Its empirical result is strong and clear that as soon as these strong assumptions are removed, the SOTA attack algorithms no longer perform that well. That sheds lights on how to improve the security of a federated learning system, i.e., do not share statistics like batch norm or certain private labels.\n2. From a defense perspective, this paper considers a novel scheme which combines various different defense algorithms, such as gradient pruning, MixUp and InstaHide to achieve a better defense performance compared to any single defense algorithm, especially when attack batch size is large.\n3. Previous works either focus on encoding private dataset directly or encoding the gradient information. This paper is a unified review of these two different methods, and combines them to improve the overall defense performance.\n\nCons:\nThe line 233 is a bit confusing: “InstaHide further encodes a MixUp image by randomly flipping its pixel signs. We are only mixing private images in our experiments, which is a weaker version of InstaHide”. Does this mean that W-InstaHide does not use random sign flips? If so, what’s the difference between MixUp and W-InstaHide? I believe there should be a difference between MixUp and W-InstaHide. However, based on the write-up, it’s a bit unclear about such differences and hence confusing. It should be further clarified.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I did not realize any potential negative societal impact of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "Fjg8k_nb53", "reviewer": "Reviewer_PVFu", "summary": "The paper presents an empirical investigation of existing gradient inversion attacks and defenses in federated learning setting. The authors point out that current attacks make strong assumptions such as knowing batch norm stats or private labels, whose absence would weaken the attacks. The authors also conclude with a list of best practices of doing conducting safe federated learning via comprehensive review and experiments on current defense methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well written and organized. Advancing state-of-the-art is important, but checking caveats in current methods is critical as well. Summarizing current assumptions made in current gradient inversion attacks makes it easier for future research to design fair and reasonable experiments. The practices lead by experiment results of various defenses are also valuable. However, although the conclusions are interesting, I wonder if the results are strong enough. Instead of doing experiments on a single dataset (CIFAR-10) with a single architecture (ResNet18), I would like to see more diverse settings, either easier (MNIST with MLP) or more complex (ImageNet with deeper ResNets). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, the limitations and potential negative societal impact are not clear in Section 6. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "CThAPUiPNXH", "reviewer": "Reviewer_5LCL", "summary": "This paper evaluates existing attacks and defenses to gradient inversion attack, where eavesdropper of the protocol could steal private data from the clients. It re-evaluates attacks under relaxed assumptions, the effectiveness of the defenses, and proposes actionable ways of combining defenses under realistic assumptions to have more robust FL. The paper clarifies the state of the art in gradient inversion attacks and sheds light on best practices for good defense deployment", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- Understanding and systematically evaluating strengths and weaknesses in the state of the art is crucial for arising fields\n- Authors adjust threat model to support the idea that gradient inversion attacks are much less realistic in practice\n- The paper remains always very practical, and contains many actionable points well-summarized in the conclusions.\n- Code is planned of being released\n\n# Weaknesses\n- Focused only on image classification tasks \n- The experimental testbed is a bit unclear, which datasets are used in which parts\n- Only 50 images on CIFAR-10 used for the re-evaluation of the defenses (line 241)\n\n# Detailed comments\n\nThe paper studies a very important problem and takes time to re-evaluate the current state of attacks and defenses in gradient inversion attacks. Privacy is of utmost importance, and it is somehow hard to realize what are the pros/cons of existing methods. The authors conduct a very systematic and thorough evaluation, with very good and convincing arguments regarding the revision of existing attack assumptions, to help reason better about more realistic scenarios in-the-wild. Nevertheless, the paper also does a great job at finding interesting properties about more secure deployments, and greatly summarized actionable points for both practitioners and researcher. \n\nNevertheless, I do have a couple of major concerns that I'd like the authors to discuss a bit better. \n\n## Experimental testbed\n\nThe datasets used in the experiments are not always fully clear. Of course authors are using CIFAR-10 (mostly) and a bit of ImageNet, but it is not always clear *how much* of it they are using for the evaluation. \n\nFor example, are Figures 1 and 2 contain some images used for the actual attacks and proof-of-concept evaluation of effectiveness in presence of relaxed assumptions. Are these the only adversarial examples generated? This sounds a bit 'small' in terms of chosen images. \n\nOn line 241, you mention:\n> We use a subset of 50 CIFAR-10 images to evaluate the attack performance.\n\nThis seems quite 'limited', and possibly introducing some unintentional sampling bias in the evaluation that may threaten the empirical validity of the results.\n\nI appreciate that the Appendix has additional settings, but I wonder how much of the study may be affected/threatened by the limited sampling of the dataset for attacks. \n\n\n## Task\n\nThe paper is focused only in the **image classification task**, so it would be interesting to understand whether the same properties/conclusions would hold also in other domains where deep neural networks are successful, for example speech recognition in smart home devices. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I feel the authors should emphasize more limitations related to the number of adversarial examples generated.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "0LVYS7pkUxj", "reviewer": "Reviewer_XbmS", "summary": "The authors of this paper provide a systematical analysis for the robustness of federated learning. They first show that current strongest gradient inversion attacks cannot perform well under relaxed assumptions and then demonstrate a new defense that can make the attacks less effective, even under their original assumptions. Sufficient experiments provide strong support for their conclusions. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1. The authors reveal that the strong assumptions in the current strongest attack may not realistic and demonstrate even the current strongest attack cannot perform well under the relaxed assumption. The conclusions encourage that future research on gradient inversion should be evaluated in a more realistic setting.  \n2. The authors provide some simple methods to improve the robustness of federated learning against gradient inversion attacks. They get strong results with a combination of several simple practices, which can improve the security against the strongest attack under the strong assumption.  \n3. The paper is well organized and easy to understand. The findings and the proposed defenses and be used to evaluate the robustness of future federated learning systems. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Please fix the citation (geiping2020) in Line 89 of the manuscripts. \n2. I like the presentation of this paper: no fancy tricks but deliver clear and practical messages for future research. But the two parts (contribution 1 vs contribution 2, 3) are kind of separate from each other. I am suggesting the authors find a good way to combine them into a single story. \n3. The experiment and analysis are mainly applied to toy datasets with low-resolution images. I am wondering whether those conclusions are still valid on high-resolution images. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": ["Responsible Research Practice (e.g., IRB, documentation, research ethics)", "I don’t know"], "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are satisfied by the responses made by the authors. The authors are strongly encouraged to include the additional experiments and results they provided in the rebuttal phase to their final manuscript.
Decision: Accept (Oral)