Learning To Invert: Simple Adaptive Attacks for Gradient Inversion in Federated Learning
Authors: Ruihan Wu
Xiangyu Chen
Chuan Guo
Kilian Q Weinberger
Venue: Submitted to ICLR 2023 (2023)

Abstract: Gradient inversion attack enables recovery of training samples from model updates in federated learning (FL) and constitutes a serious threat to data privacy. To mitigate this vulnerability, prior work proposed both principled defenses based on differential privacy, as well as heuristic defenses based on gradient compression as countermeasures. These defenses have so far been very effective, in particular those based on gradient compression that allow the model to maintain high accuracy while greatly reducing the attack's effectiveness. In this work, we argue that such findings do not accurately reflect the privacy risk in FL, and show that existing defenses can be broken by a simple adaptive attack that trains a model using auxiliary data to learn how to invert gradients on both vision and language tasks.

Under review as a conference paper at ICLR 2023
LEARNING To INVERT: SIMPLE ADAPTIVE ATTACKS FOR GRADIENT INVERSION IN FEDERATED LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Gradient inversion attack enables recovery of training samples from model updates in federated learning (FL) and constitutes a serious threat to data privacy. To mitigate this vulnerability, prior work proposed both principled defenses based on differential privacy, as well as heuristic defenses based on gradient compression as countermeasures. These defenses have so far been very effective, in particular those based on gradient compression that allow the model to maintain high accuracy while greatly reducing the attack's effectiveness. In this work, argue that such findings do not accurately reflect the privacy risk in FL, and show that existing defenses can be broken by a simple adaptive attack that trains model using auxiliary data to learn how to invert gradients on both vision and language tasks.
INTRODUCTION
Federated learning (FL; (McMahan et al., 2017)) is a popular framework for distributed model training on sensitive user data. Instead of centrally storing the training data, FL operates in server- client setting where the server hosts the model and has no direct access to the data. The clients can apply the model on their private data and send gradient updates back to the server. This learning regime promises data privacy as users only share gradients but never any raw data. However, recent work (Zhu et al. 2019 Zhao et al.) 2020 Geiping et al.) (020)) showed that despite these efforts, the server can still recover the training data from gradient updates, violating the promise of data privacy in FL. These so-called gradient inversion attacks operate by optimizing over the input space to find training samples whose gradient matches that of the observed gradicnt, and such attacks remain effective even when clients utilize secure aggregation (Bonawitz al. 2016) to avoid revealing individual updates (Yin et al. 2021 Jeon et al. 2021).
As countermeasures against these gradient inversion attacks, prior work proposed both principled defenses based on differential privacy (Abadi et al. 2016), as well as heuristics that compress the gradient update through gradient pruning (Aji & Heafield (017)
sign compression (Bernstein itt al. 2018). In particular, gradient compression defenses have so far enjoyed great success, severely hindering the effectiveness of existing optimization-bart while maintaining close to the same level of accuracy for the trained model. As result, these limitations seemingly diminish the threat of gradient inversion in practical FL. applications.
attacks (Zhu et al. 2019 Jeon al. 2021]
In this paper we argue that evaluating defenses on existing optimization- attacks may provide a false sense of security. To this end, we propose a simple learning-baat -basdd attack-whidh we call Learning To Invert (LTTI) trains model to learn how to invert the gradient update to recover client samples; see Figure for an illustration. We assume that the adversary (i.e., the server) has access to auxiliary dataset whose distribution is similar to that of the private data, and use it to generate training samples for the gradient inversion model by querying the global model for gradients. Our attack is highly adaptable to different defenses since applying a defense simply amounts to training data augmentation for the gradient inversion model.
We empirically demonstrate that LTI can successfully circumvent defenses based gradient pertur- bation (i.e., using differential privacy; (Abadi et al. (2166), gradient pruning (Aji & Heafield (017) and sign compression (Bernstein al. 2018) on both vision and language tasks.

---

Under review as a conference paper at ICLR 2023
Learning To Invert (LTTT)
In-Distribution
Out-of Dittof-Distribution
Auxiliary Dataset
Federated Model
Gradient
Inversion
(x),)(()))))
Model go
y2,
Eriaiic
partock
Federated Learning
Gradient Po (x))
Gradient
Inversion
Attack
Client
Model
Federated Model fy
Inversion
3
with LTI
Model fi
Model ge
Server Attacker
Gradient (x2))
Client
Figure 1: Illustration of federated learning (FL. and gradient inversion methods. The goal of gradient
inversion is to recover training data (x, y) from the observed gradient V [f [ (x), y)- Optimization- based methods (e.g., (Zhu et al., [2019] Geiping et al. 2020 Yin al.] [201]] Jeon al.] (021T) directly optimize (x, in search for a sample that produces gradient similar to that of (x, y). Our proposed learning-base -baned approach, which we call Learning Invert, instead trains inversion model ge to reconstruct training samples from their gradient.
Vision: We evaluate on the CIFARIO (rizhevsky et al, 2009) classification dataset. LTI attains recovery accuracy close to that of the best optimization-bann based method when defense is applied, and significantly outperforms all prior attacks under defense. NLP: We experiment with causal language model training the WikiText (Merity et al. 2016) dataset, where LTT attains state-of-the-arr performance in all settings, with or without defense.
Given the strong empirical performance of LTI and its adaptability different learning tasks and defense mechanisms, we advocate for its use as a simple baseline for future studies on gradient inversion attacks in FL..
2 BACKGROUND
Federated learning. The objective of federated learning (McMahan et al. 2017) is to train machine learning model in a distributed fashion without centralized collection of training data. In detail, let fw be the global model parameterized by w. and consider a supervised learning setting that optimizes w by minimizing a loss function over the training set Purain' Exx.y)DDraa {(f((x) In centralized learning this is typically done by computing a stochastic gradient B
1 DE Wwe(fw(x,, y:) over randomly drawn batch of data (11.))...... (xB, yB) and i=] minimizing & using gradient descent.
In FL, instead of centrally collecting Dirain to draw random batch during training, the training set Dirain distributed across multiple clients and the model fw is stored on central server. At each iteration, the model parameter w is transmitted to each client to compute the per-sample gradients (fw((f((xx)) 313 } locally over a set of clients. The server and clients then execute a federated aggregation protocol advantage of FL data privacy since clients do not need to disclose their data explicitly, but rather only send their gradient V V ((w (x;). y.) to the server. Techniques such secure aggregation (Bonawitz eet al. 2016) and differential privacy (Dwork et al. 2006 2014) can further reduce the privacy leakage from sending this gradient update.
compute the average gradient for the gradient descent update. A major
2

---

Under review as a conference paper at ICLR 2023
Gradient inversion attack. Despite the promise of data privacy in FL, recent work showed that the heuristic of sending gradient updates instead of training samples themselves in fact provides a false sense of security. Zhu et al. (2019) showed in their seminal paper that it is possible for the server to recover the full batch of training samples given aggregated gradients. These optimization- based gradient inversion attacks operate by optimizing set of dummy data .,,..., 83 and labels ,1...., ÜB to match their gradient to the observed gradient:
B
min Σ
V w Wwwl(S((K)):))
Σ (ww((w((x)) Vi)
For image tasks, since Eqqution 1 differentiable in &; and Os and the model parameter w is known to the server, the server can optimize [Equation I] [uiing gradient-bar based search. Doing so yields recovered samples (R;, :/:) that closely resemble actual samples (x;,) 114 in the batch. In practice this approach is highly effective, and follow-up works proposed several optimizations to further improve its recovery accuracy (Geiping et al. 2020
Yin et al. 2021 Jeon et al. 2021).
For language tasks this optimization problem is considerably more complex since the samples Xi,...,Xp XB are sequences of discrete tokens, and optimizing Equation I ammounts to solving discrete optimization problem. To circumvent this difficulty, Zhu et al. (2019) and Deng et al. (202T) instead optimize the token embeddings match the observed gradient and then maps the recovered embeddings to their closest tokens in the embedding layer to recover the private text. In contrast, Gupta al. (2022) leveraged the insight that gradient of the token embedding layer can be used to recover exactly the set of tokens present in the training sample, and then uses beam search to optimize the ordering of tokens for fluency recover the private text.
Gradient inversion under the malicious server setting. The aforementioned gradient inversion attacks operate under the honest-but- but-curious setting where the server faithfully executes the federated learning protocol, but attempts to extract private information from the observed gradients. [Fwl et al. (202T), [oenisch et al. (202T) and Fowl et al. (2022) consider a stronger malicious server threat model that allows the server transmit arbitrary model parameters to the clients. Under this threat model, it is possible to carefully craft the model parameters so that the training sample can be recovered exactly from its gradient even when the batch size B is large. While this setting is certainly realistic and relevant, our paper operates under the weaker monest-but-curion threat model.
3 LEARNING TO INVERT: LEARNING-BASED GRADIENT INVERSION ATTACKS
Motivation. The threat of gradient inversion attack has prompted prior work to employ defense mechanisms to mitigate this privacy risk in FL (Zhu et al. 2019 Jeon et al. 202T). Intuitively, such defenses reduce the amount of information contained in the gradient about the training sample by either perturbing the gradient with noise (Abadi et al. 2016) or compressing them (Aji & Heaheel 2017: Bernstein et al. 2018)), making recovery much more difficult. However, doing so also reduces the amount of information sample can provide for training the global model, and hence has negative impact on the model's 's performance. This is certainly true for principled defenses based on differential privacy (Dwork al. (2066) such as gradient perturbation (Abadi al. 2016), however, defenses based on gradient compression seemingly provide much better privacy-utility trade-off, effectively preventing the attack with minor reduction in model performance (Zhu et al. 2019).
The empirical success of existing defenses seemingly diminish the threat of gradient inversion in
FL, especially since gradient compression (Aji Heafelld 2017; Bernstein et al.) (0018) is already commonplace in practical FI. applications to reduce communication cost. However, we argue that optimization-bann based attacks underestimate the power of the adversary: the adversary has access to an auxiliary dataset Daux. they train gradient inversion model to recover Daux from its gradients computed on the global model. As we will establish later, this greatly empowers the adversary, exposing existing risks to federate learning.
Threat model. We consider the setting where the adversary is an honest-but-curien but-curious server, who executes the learning protocol faithfully but aims to extract private training data from the observed gradients. We also assume that the FL protocol does not leverage secure aggregation, so per-client gradients revealed the server. Under these assumptions, in each FL iteration the adversary
3

---

Under review as a conference paper at ICLR 2023
has the knowledge of model weights and the gradients (f((ff()) y) for each sample (x, y) in the batch. Moreover, we assume the adversary has an auxiliary dataset Daux- which could be in-distribution
mixture of in-distribution and -uttt---iitribttionn data. This assumption is similar to the setting in Jeon et al. (202T), which assumes a generative model that is trained from the in-distribution data, and is common in the study of other privacy attacks such membership inference (Shokri et al. 2017).
Learning invert (LTI). Since the adversary has knowledge of the model weights, he/she is able to generate the gradient V W (Xaux), Maux) for each sample (Xaux, Maux) in the auxiliary dataset. This allows the adversary learn gradient inversion model go. parameterized by 0, predict the data point (Xaux, Vaux) from the gradient of the global model V W (f (Xaux). Vaux) by solving the following learning problem:
min
Σ
gattack (go w (Xaux), ,, Vaux)) (Xauxes yaux))
(2)
(Xanx.Baore) EDaux
In practice, gattack can be the cross-entropy (for discrete input) or squared-loss (for continuous-valus input) function and we find that using multi-layen perceptron (MLP) (Bishop et al.) 1995) for go effective empirically. Importantly, when a defense mechanism such as gradient perturbation or gradient compression is applied, we can apply the same transformation to V ( ), Maux) to augment the training data for ge to carry out an adaptive attack. We will show in infseciion 4 that this simple approach is surprisingly effective at circumventing existing defenses.
Dimensionality reduction for large models. One potential problem for LTI is that the gradients w.l(f (Xaux): (X Vaux) can be extremely high-dimensional. For example, both ResNet18 (He et al. 2016) for vision tasks and a three-layer transformer (Vaswani et al. (2017) for language tasks have approximately 1.1 million trainable parameters. Such high-dimensional input to the model go can lead to memory issues, as the first layer of the MLP would have 11M x h parameters, where denotes the size of the first hidden layer.
To address this issue, we use feature hashing (Weinberger et al. 2009) to reduce the dimensionality of the input gradient. To this end we create & bins, where ls is much smaller than the size of gradient
m, and assign each gradient dimension i E [m]] to a random bin r(i) E [k:]. For each bin, we sum up the gradient values that are assigned to this bin. As a result, we obtain a feature vector of
size ic for the inversion model go. In other words, we project the gradient V ( (Xaux), yaux) to
PV [.[( (Xax)), yaux) using the random projection matrix P given by:
Pe 10 kxm s.t. Vi, P3. = 0 (Vj # r())), Proj. 1.
If r(i) is implemented with pseudo-uniform hashing function, P does not need to be stored in memory, reducing the memory footprint of go to a constant independent of the gradient dimension.
4 EXPERIMENT
We evaluate LTI on vision and language tasks against several existing defenses to show that vastly outperforms prior gradient inversion attacks. We consider the following defense mechanisms evaluated in prior work (Zhu et al. 2019: Jeon et al. 202I):
None. The gradient shared between the server and clients is the full gradient without any defense. This is the most common setting that previous papers focus on.
Sign compression (Bernstein et al. 2018) applies the sign function to each dimension of the gradient independently to compress the gradient to one bit per dimension.
Gradient pruning with pruning rate (Aji & Heafield (017)) zeroes out the bottom 1 fraction of coordinates of V (w), y) in terms of absolute value, which effectively compresses the
gradient to (1 - am)m dimensions.
Gradient perturbation with Gaussian standard deviation (Abadi et al. 2016) is a differentially private mechanism used commonly for training private models with SGD, An i.i.d. Gaussian random vector (0, o²) is added to the gradient, which one can show achieves c-local differential privacy Kaaiivisaanathan al. 2011) with O(1oo).
4

---

Under review as a conference paper at ICLR 2023
Defense
None
Sign Compression
Method
MSE (4)
PSNR (t)
LPIPS (4)
MSE (4)
PSNR (t)
LPIPS (4)
IG GI-GIP
0.022
22.290
0.263
0.116
9.981
0.677
0.001
33.374
0.033
0.091
13.574
0.471
LTI (Ours)
0.004
24.837
0.221
0.014
18.986
0.396
Defense
Gradient Pruning (a
0.99)
Gaussian Perturbation (o 0.1)
Method
MSE (4)
PSNR (1)
LPIPS (L)
MSE (L)
PSNR (t)
LPIPS (4)
IG GI-GIP
0.138
8.807
0.675
0.150
8.349
0.653
0.043
14.356
0.474
0.124
9.383
0.568
LTI (Ours)
0.029
15.897
0.472
0.012
20.249
0.370
Table Result for gradient inversion attack vision data. L.TI achieves the best performance all three metrics against the sign compression, gradient pruning and Gaussian perturbation defenses.
4.1 EVALUATION ON VISION TASK
For evaluating LTI on a vision task, we experiment with image classification on CIFARIO (rizhevsky et al. 2009). The target model fw is LeNet (LeCun et al. 1998) with 1.5 x 10* parameters trained using the cross-entropy loss.
Baselines. We compare our method with two baseline gradient inversion attacks: Inverting Gradients (IG: GGiping al. (, representative optimization- Generative Image Prior (GI-GIP: [eeon et al. (202T)) the state-of-the-art optimization-bann that uses a generative model encode the data prior. We make minor modifications to these attacks to adapt them to various defenses; see appendix for details. The threat model for our attack is most similar GI-GIP since both use auxiliary dataset to encode the data prior.
method, and Gradient Inversion with method
Inversion model training. We follow the setup below for training the gradient inversion model go-
Auxiliary dataset. We use the train split of CIFARIO the auxiliary dataset for training the inversion model ga and the test split for inverting gradients computed on the global model fw. Inversion model architecture. We use three-layer MLP with hidden size 3000 for our inversion model go. The MLP takes the flattened gradient vector as input and outputs 3072-dimensione vector representing the flattened image. The training objective pattock in EEquttin 2 is the mean squared error (MSE) between the output vector from MLP and the flattened ground truth image. Training details. We use the Adam (Kingma Ba 2014) optimizer for training go. The model trained for 200 epochs using training batch size 256. The initial learning rate is 10 with learning rate drop to 10 - after 150 epochs.
Computation cost. Our experiments conducted using NVIDIA GeForce RTX 2080 GPUs and each training run takes about 1.5 hours.
Evaluation methodology. We evaluate LTI and the aforementioned baselines 1, 000 random images from the CIFARIO test split. To measure reconstruction quality, we use three metrics:
Mean squared error (MSE) measures the average pixel-wise (squared) distance between the reconstructed image and the ground truth image. Lower is better.
Peak signal-to-noise ratio (PSNR) measures the ratio between the maximum image pixel value and
MSE. Higher is better.
Learned perceptual image patch similarity (LPIPS) measures distance in the features space of
VGG (Simonyan & Zisserman 2014) model trained on ImageNet. Lower is better.
4.1.1
MAIN RESULTS
Quantitative evaluation. Table gives quantitative comparisons for IG, GI-GIP, and LTI against various defense mechanisms on CIFAR10. When no defense is applied, GI-GIP achieves the best performance according to all three metrics, whereas LTI performs almost equally well in terms of MSE and close to that of IG in terms of PSNR and LPIPS. However, when the gradient augmented with defense mechanism, both IG and GI-GIP have considerably worse performance with MSE
5

---

Under review as a conference paper at ICLR 2023
None
Sign Compression
Gradient Pruning
Gradient Perturbation
Ground Truth
IG
GI-GIP
LTI
IG
GI-GIP
LTI
IG
GI-GIP
LTI
IG
GI-GIP
Figure 2: Comparison of LTI with IG and GI-GIP for reconstructing 4 random images in CIFARIO test set. Under sign compression, only LTI can partially reconstruct the images to recover the object of interest whereas both IG and GI-GIP fail do so on most samples.
close to 0.1. By comparison, LTI outperforms both baselines significantly and consistently across all three defense mechanisms. For example, under gradient perturbation with 0.1, which prior work
believed is sufficient for preventing gradient inversion attacks (Zhu et al. 2 et al. [022I) MSE can be as low as 0.01 for I.TI. Our result therefore provides considerable additional insight for the level of empirical privacy achieved by DP-SGD (Abadi et al., 2016). and suggests that the theoretical privacy leakage as predicted by DP may be tighter than previously thought.
Qualitative evaluation. Figure 2 shows 4 random CIFARIO test samples and their reconstructions under different defense mechanisms. Without any defense in place, all three methods recover considerable amount of semantic information about the object of interest, with both GI-GIP and LTI faithfully reconstructing the training sample. Under the sign compression defense, IG completely fails to reconstruct all 4 samples, while GI-GIP only successfully reconstructs the second image. In contrast, I.TI able to recover the semantic information in all samples. Results for gradient pruning and gradient perturbation yield similar conclusions. Additional samples given in the appendix.
4.1.2 ABLATION STUDIES
Since LTI learns to invert gradients using the auxiliary dataset, its performance depends the quantity and quality of data available to the adversary. We perform ablation studies to better understand this dependence by changing the auxiliary dataset size and its distribution.
Varying the auxiliary dataset size. We randomly subsample the CIFARIO training set to construct auxiliary datasets of size (500, 5000, 15000, 25000, 35000, 45000, 50000) and evaluate the perfor- mance of LTT under various defenses. Figure 3Ja) plots reconstruction MSE as a function of the auxiliary dataset size, which is monotonically decreasing as expected. Moreover, with just 5, 000 samples for training the inversion model (second point in each curve), the performance is nearly as good as when training using the full CIFARIO training set. Notably, even if auxiliary dataset size as small as 500, reconstruction MSE is still lower than that of IG and GI-GIP in Table figures for PSNR and LPIPS in the appendix show similar findings.
Corresponding
Varying the auxiliary data distribution. Although access to a large set of in-distribution data may be not available in practice, plausible that the adversary can collect out-of-distiibution samples for the auxiliary dataset. This is beneficial for the adversary since model that learns how to invert distrribbttnnn samples given their gradients may transfer to in-distribution data as well. To simulate this scenario, we divide CIFARIO into two halves with disjoint classes, and construct the auxiliary dataset by combining a fraction of samples from the first half and 1 - 3 fraction of samples from the second half for 3 E {0, 0.01, 0.1, 0.3, 0.5, 0.7, 0.9, 1). The target model fw is trained only on samples from the first half, and hence the auxiliary set has the exact same distribution the target model's data when 3 and only has ciat-f-dditributiinn data when 8 = 0.

---

Under review as a conference paper at ICLR 2023
None
Sign Comp.
Grad. Pruning (a 0.99)
Grad. Pert. (a 0.1)
0.040
0.16
0.035
0.14
0.030
0.12
MSB
0.025
0.10
0.020
0.08
0.015
0.06
0.04
0.010
0.02
0.005
0.00
10000 20000 30000 40000 50000
0.0
0.2
0.4
0.6
0.8
1.0
Size of Auxiliary Dataset
Shift Factor B
(a) Vary the size of auxiliary set
(b) Vary the distribution of auxiliary set
Ablation studies on size and distribution the auxiliary dataset Daux- Under both severe data size limitation (left) and data distribution shift (3 0.01; right), LTT is able to outperform both baselines in Table when a defense mechanism is applied. See text for details.
Figure 3:
Figure b)b shows reconstruction MSE as a function of 3; corresponding figures for PSNR and LPIPS are given in the appendix. We make the following observations:
Even if the auxiliary dataset only contains 250 in-distribution samples (3 0.01; second point in each curve), MSE of the inversion model is still lower than that of the best baseline in Table For example, with the sign compression defense, LTT attains an MSE of < 0.02, which is much lower than the MSE of 0.116 for IG and 0.091 for GI-GIP.
When the auxiliary dataset contains only out-of-distribution data (3 = 0), the inversion model has very high reconstruction MSE, which suggests that methods for improving out-of-distribution generalization may be necessary for further improvement.
4.2 EVALUATION ON LANGUAGE TASK
For evaluating LTI on language task, we experiment with causal language model training for next-token prediction. The language model Sw is three-layer transformer (Vaswani et al.) 2017) f frozen token embedding layer. This is a common technique for language model fine-tuning (Sun al. 2211).. which also has privacy benefits since direct privacy leakage from the gradient magnitude of the token embedding layer can be prevented (Fowl et al., 2022 Gupta et al., 2022). As a result, the trainable model contains about 1.1 x 10" parameters. We train the language model on WikiTeett (Merity et al.) (016)), where each training sample is limited to = 16 tokens and the language model trained to predict the next token x; given x for | 1,..., , L using the cross-entropy loss.
Baseline. We compare LTI with TAG (Deng al. 2021 the state-of-the-art language model gradient inversion attack without utilizing the token embedding layer gradien The objective function for TAG is a slight modification of Equation that uses both the lg and é distance between the observed gradient and the gradient of dummy data. We also modify TAG slightly to adaptive it different defenses; see appendix for details.
Inversion model training. We follow the setup below for training the gradient inversion model go-
Auxiliary dataset. We use 1.8 x 10 samples from the train split of Wikitext as the auxiliary dataset, and 1, 000 samples from the test split for evaluating the attack. In addition, we introduce weaker variant of our attack that only assumes knowledge of the marginal token distribution for the language model training data. Instead of using the WikiText train split as auxiliary data, sample random tokens according to the marginal token distribution to generate pseudo-data for training the inversion model. We show that this variant, which we denote LTI-P, can even outperform LTI with in-distribution auxiliary data due to access to infinite training data.
I follow the task setup and code in https::/githhh.... https://gihuu..oo///onaaaaaainn hwwp/onaGeeiinnn//reaaiiiiiiiiie We do not compare against more recent attack by Gupta et al. (2022) since it crucially depends on access the token embedding layer gradient.

---

Under review as a conference paper at ICLR 2023
Defense
None
Sign Compression
Method
Acc.
Rouge-1 1
Rouge-2
Rouge-L L
Acc.
Rouge-1
Rouge-2
Rouge-L
TAG
74.13
71.92
50.64
68.46
0.00
0.06
0.00
0.06
LTI (Ours)
89.61
86.13
79.53
86.11
71.15
63. 17
43.51
63.11
LTI-P (Ours)
91.14
89.43
85.11
89.41
88.06
84.66
76.46
84.64
Defense
Gradient Pruning (a: = 0.99)
Gaussian Perturbation (a = 0.01)
Method
Acc.
Rouge-1
Rouge-2
Rouge-L
Acc.
Rouge-1
Rouge-2
Rouge-L
TAG
34.34
48.50
10.21
35.60
64.34
66.19
37.86
59,55
LTI (Ours)
66.79
58.31
37.58
58.21
82.08
76.55
63.38
76.52
LTI-P (Ours)
86.19
82.56
73.04
82.50
90.25
87.39
81.94
87.34
Table 2: Results for gradient inversion attack on text data. Both LTI and LTI-P significantly outperform TAG cross different settings in all 4 metrics, where LTI-P achieves the best result with only access to the marginal token distribution for generating the auxiliary dataset.
Inversion model architecture. We train a two-layer MLP with ReLU activation and first hidden-layer size 600 and second hidden-layer size 1, 000. The inversion model outputs L. probability vectors each with size equal to the vocabulary size (~ 50, 000), and we train it using the cross-entropy loss predict the L tokens given the target model gradient. We use feature hashing (Weinberger et al., 2009) to reduce the target model gradient to 10% of its original dimensions as input to the inversion model.
Truining details. We use Adam (Kingma & Ba 2014 to train the inversion model over 20 epochs with batch size 64. Learning rates are selected separately for each defense from (10-3, 10-1, 10-1,10-5);
Computation cost. Our experiments are conducted using NVIDIA GeForce RTX 3090 GPUs and each training run takes about 3 hours.
Evaluation methodology. We evaluate LTI and the TAG baseline on 1, 0,000 samples from the WikiText test set. To measure the quality of reconstructed text, use four metrics:
Accuracy( 1%) measures the average token-wise zero-one accuracy. Higher is better: Rouge- 1 (%), Rouge-2(<% ()) and Rouge-L(%) measure the overlap of unigram, bigram, and length of longest common subsequence between the ground truth and the reconstructed text. Higher is better.
Results. Table 2 shows quantitative comparison between LTI (and its variant LTI-P) and TAG against various defenses. The overall trend is remarkably consistent: LTI and LTI-P outperform TAG in all four metrics for all defense settings, with LTI-P achieving state-of-the-art recovery accuracy by far. This result suggests that knowledge of the marginal token distribution encodes enough data prior for LTI-P to train the inversion model, and having access infinite training data allows it to better generalize to the test set compared to LTI. In practice. it is very plausible that the marginal token distribution is known the adversary, and hence LTI-P serves as a surprisingly simple and effective baseline for gradient inversion in NL..
Figure shows 3 random test samples from WikiText and their reconstructions using LTI-P and TAG, with tokens that are correctly reconstructed highlighted in blue. Without any defense, both TAG and LTI-P yield reasonably accurate reconstructions, with LTI-P faithfully reconstructing all but 1-2 tokens. With the sign compression defense applied, TAG fails to recover any token correctly, whereas LTI-P can faithfully recover almost half of the tokens in each sample. Results for gradient pruning and gradient perturbation yield similar conclusions, with TAG recovering larger but still relatively insignificant set of tokens. Additional samples are given in the appendix.
5 CONCLUSION AND FUTURE WORK
We demonstrated the effectiveness of LTI a simple learning-basee based gradient inversion attack under realistic federated learning settings. For both vision and language tasks, LTI match or exceed the performance of state-of of-te-of-the-all the-ttt- optimization based methods when no defense is applied, and significantly outperform all prior works under defenses based on gradient perturbation and gradient
8

---

Under review as a conference paper at ICLR 2023
Example erumbled into exposed to the air She added further
Example 2
Example 3
exploration the vault then revealed "a large human banes, several feet:
straight season as nember of
True Texts
Borrierumleed into dust, deprive exposed the etcched She added that Rnfoeemeett
exploration of Trail vault then revealed large Grip of bunan Sharia, several uggets
team iQuezys 95th straight season as 2010 Big
TAG
member
None
Sard
ofumbled into dust, when ifying to the sir She added that further
of of voult. then revealed a large mans of bones, several
the team 's 95th straight season as nenber of the Big Ten Conference
LTI-P
droppingize adremaline proxiesisal unlaweca Voilookie indispensable
stressed..] div wragperoes/Rfoot interacted plent behclinical AI surfingtll 84 Divfrog Thunder172
417 staminausp
TAG
Volkewagenble twitch calories dumpsasy everyable
Sign
DBrev Christopher cricket currents Dominion
Comp.
wasumbled into dust, when the sir ", she added that further
of the vault then revealed
the tean's a 95th stroight menber of the hunan bones, several feet Big Ten Conference
LTI-P
of
Indounbledhol dust to exposed Wearorb into added that Prenklin
Iw of hotals coal linked explorati explorationumbers a then revealed bonea vault. severaIcAN
the team's 's Big Ple belongings BCC courtyard Add member of 95 pid Ten
TAG
Grad.
Pruninig
theumbled into dust, when mayor
vault then revealed large
straight season nenber of the Big Ten Conference
LTI-P
*. Sha added that further
devotion hunan Road, several feet
respunbled into dust Syne when permitting exposed the air '. She to Lix thresholds
exploration bones the resp team 's straight vault large revealed Punch wise member Ten the Big of mass nost human of, philosophies deveral cycle
TAG
Grad.
Pert.
unbled into dust, when exposed the sir ", She added that, further
the tean' 's 95th straight season as a member of the human bones, several feet Big Ten Conference
of the vault: then revealed large of
LTI-P
Figure 4: Ground truth text and their reconstructions for 3 random samples from the WikiText test set. LTI-P significantly outperforms TAG both with and without defenses, especially under sign compression where TAG fails to recover any token while LTI-P is capable recovering almost half of the tokens in each sample.
compression. Given its simplicity and versatility, we advocate the use of LTT both as a strong baseline for future research as well as a diagnostic tool for evaluating privacy leakage in FL.
Negative societal impact. The concept of gradient inversion attack can lead to negative conse- quences if used inappropriately. Our work showed that FL deployed without consideration for gradient inversion attacks, adversary privacy of clients even under strong empirical defenses. However, we strongly emphasize that our tool for adversaries, but rather
leverage its vulnerabilities to compromise the data
work should not be interpreted about the risks of data privacy breach FL. and promote future research into safe practices.
inform the community
Limitations. This paper serves as preliminary work towards understanding the effectiveness of learning-based gradient inversion attacks, and our method can be further improved along several different directions. 1. For large models, our current approach is to hash the gradients into a lower- dimensional space to reduce memory cost. It may be possible to leverage the model's 's architecture to design more effective dimensionality reduction techniques Currently we only focus on the setting with batch size 1, which precludes the use of secure aggre- gation (Bonawitz et al. (016)- common technique in FL for amplifying privacy by aggregating the gradients from multiple clients before revealing it to the server. For LTT,, the complexity of MLP would increase when the batch size increases, which makes the learning harder. More advanced model architecture and loss design might help with the large batch case. 3. LTI in its current form does not leverage additional data priors such as the smoothness prior for images and text fluency prior for text. We can readily incorporate these priors by modifying the inversion model's 's loss function with total variation (for image data) or perplexity on a trained language model (for text data), which may further improve the performance of L.T..
further scale up the method. 2.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. arXiv preprint A.K4::1704.0022. 2017.
Jeremy Bemstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp. 560-569. PMLR, 2018.
Christopher M Bishop et al. Neural networks for pattern recognition. Oxford university press, 1995.
Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. When the curious abandon honesty: Federated learning is not private. arXiv preprint arXiv:21 2021.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv:/611 1.04422, 2016.
Jieren Deng, Yijue Wang, Ji Li, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. Tag: Gradient attack on transformer-basee language models. arXiv preprint arXiv:2103.0688 2021.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-266--22 Springer, 2006.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci., 9(3-4):211440) 2014.
Liam Fowl, Jonas Geiping, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Robbing the fed: Directly obtaining private data in federated learning with modified models. arXiv preprint arXiv:21 3005,, 2021.
Liam Fowl, Jonas Geiping. Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Decepticons: Corrupted transformers breach privacy federated learning for language models. arXiv preprint arXiv:2201. /2675, 2022.
Jonas Geiping. Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-how easy to break privacy in federated learning? Advances in Neural Information Processing Systems, 33:16937-16947, 2020.
Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recovering private text in federated learning of language models. arXiv preprint arXiv:2205.085// 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jinwoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, et al. Gradient inversion with generative image prior. Advances Neural Information Processing Systems, 012299988-9909 2021.
Shiva Prasad Kasiviswanathan, Homin K Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What leam privately? SIAM Journal on Computing, 40(3):793-826 2011.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X1v:1412.6990, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(1) (6(11):2278-22) 1):2278-2324, 1998.
10

---

Under review as a conference paper at ICLR 2023
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication- effiiiiennn learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282 PMLR, 2017.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv::409. 1556, 2014.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. How to fine-tune bert for text classification? In China national conference on Chinese computational linguistics, pp. 194-206. Springer, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In Proceedings of the 26th anmual international conference on machine learning, pp. 1113-1120, 2009.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16337-16346, 2021.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610, 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in Neural Information Processing Systems, 32, 2019.

---

Under review as a conference paper at ICLR 2023
A SUPPLEMENTARY MATERIAL FOR SECTION 4
A.1 MODIFICATIONS FOR BASELINE METHODS
Vision baselines. Both IG and GI-GIP use cosine distance instead of &g distance in Equation for optimizing the dummy data. For the sign compression defense, this loss function does not optimize the correct objective since the dummy data's gradient is not vector with +1 entries but rather real-valued vector with the same sign. We replace cosine distance by the loss E (eigg)) where
sign max V (X), Sign (V (W: ([ v)).
(3)
One sanity check for this loss is that when V w. (( 9) has the same sign as that of (w. v): the minimum loss value of is achieved. For the gradient pruning defense, optimizing the cosine distance between the dummy data gradient and the pruned ground truth gradient will force too many gradient values to O, which the incorrect value for the full ground truth gradient. Therefore only compute cosine distance over the non-zero dimensions of pruned gradient.
Language baselines. For TAG, we find that the loss function also needs be modified slightly to accommodate the sign compression and gradient pruning defenses:
Sign compression. Similar to the vision baselines, the l and & distance between the dummy data gradient and the ground truth gradient sign do not optimize the correct objective. We replace 1
and || by E (ft (Cigg)) 2 and ø "sign" respectively, where Eis1 ei defined in Equation Gradient pruning. We make the same modification to TAG as the vision baselines.
None
Sign Comp.
Grad. Pruning (a 0.99)
Grad. Pert. (a 0.1)
0.55
24
0.50
22
0.45
PSNR
20
0.40
0.35
0.30
16
0.25
10000 20000 30000 40000 50000
10000 20000 30000 40000 50000
Size of Auxiliary Dataset
Size of Auxiliary Dataset
Figure 5: Plot of reconstruction PSNR / LPIPS vs. auxiliary dataset size on CIFAR10.
None
Sign Comp.
Grad. Pruning (a 0.99)
Grad. Pert. (a 0.1)
24
0.60
22
0.55
0.50
PSNR
18
LPIPS
0.45
16
0.40
12
0.35
10
0.30
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Shift Factor B
Shift Factor B
Figure 6: Plot of reconstruction PSNR LPIPS vs. auxiliary dataset distribution on CIFAR10.
12

---

3 0.1), the performance of Under review as a conference paper at ICLR 2023
A.2 AUXILIARY DATASET ABLATION STUDIES
In section 4.1.2 we showed reconstruction MSE for LTI as a function of the auxiliary dataset size and the shift factor B. For completeness, show the corresponding PSNR and LPIPS curves in Figure 6) Similar to fofiue 3) when reducing the auxiliary dataset size (e.g.. from 50, 000 to 5, 000) or reducing the proportion of in-distribution data (e.g., from LTI does not worsen significantly.
A.3 ADDITIONAL, SAMPI.ES
Figure 7 and Figure 8 show additional samples and their reconstructions under various defense
mechanisms. The result is consistent with with/Figure 2 and Figure 4 where LTI shows consistently better reconstruction quality compared to baselines.
13

---

Under review as a conference paper at ICLR 2023
Groud Truth
None
Sign Comp.
Grad. Prun.
Grad. Pert.
IG
None
Sign Comp.
Grad. Prun.
Grad. Pert.
GI-GIP
None
Sign Comp.
Grad. Prun.
Grad. Pert.
LTI (Ours)
Figure 7: Additional samples from CIFAR10 and their reconstructions.
14

---

Under review as a conference paper at ICLR 2023
Example
Example 2
Example 3
and Back Novak, Burke Big Preshman of the Year
12 NCAA Division man a baskesball played its home games
turned full circle and Capel today just another ruined relic of
True Texts
The
and TAG Bucke was naned Big Ten icago
12 NCAA Division 276's basketball season rails Theakis played its home Dalton
turned full Capel <unks today is just sadly Chronicles
Full Grad.
of Rack Novak. was named Big Ten Preshman of Year
Division basketball season. The team played gamea
cirele
<unk>> today is just another ruined relie
Elestricitynnginssicnas TAG closet Authent Orien flawlesagra wideningprint ingtiintgi Tanner
intuition electron flurry Initialgrowhh Lukeaccount fructose rival complicationble ara Relax
screws441 legionfagendary Brighamessen
Bas tulugucconeUAoommet LearMorbacked
Sign Comp.
of and tub Novak, Burke was named Big Ten
of RCAA Division men's basketbell season. The team played its home
the full circle and Capel sunko today just another native Construction
Freshman of the Year
of and aesthetic TAG counters Tenak Boolean Zack static Marlins satisfGar, S2 Quentin
12ELD Division menanooga played Itsika. The NCAA seasoni
turned today circle and Cap just. Hutch> another isel ruined full
Grad.
installment.
Prun.
and Back Novak. Burke named Big Ten Freshman of the Year
ICAA Division men's besketbell The played its home contras
full circle and Capel
LTI-P
cunko today is just another
bolster of
Cipher albuns season. The team played beginner
circle Drinking Capel another> relicunk today
TAG Bro Big Argentine
justchance
Grad.
Purt.
the and Back Hovak. Burks MCAA Division
the full circle and Capel <unks today just another
LTI-P was named Big Ten Freshman the
season. The team played its home games
ruined relic of
Figure 8: Additional samples from WikiText and their reconstructions.
15


Reviews:
[{"review_id": "xsiKcWraKJw", "reviewer": "Reviewer_6VqA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new gradient inversion method in Federated Learning. The proposed approach called \"Learning To Invert\" (LTI) directly attempts to learn the mapping between the gradient of a sample and the corresponding input sample using an auxiliary dataset.  A simple multi-layer perceptron (MLP) is used to learn this mapping. Dimensionality reduction is applied to the gradients to make the MLP practically feasible. The paper asserts that current defense mechanisms such as Sign Compression, Gradient Pruning, and Gaussian Perturbation cannot defend against the proposed attack, because the same transformations can be applied by the server before it learns to reconstruct. The method is evaluated on vision and language tasks (CIFAR10, WikiText).", "summary_of_the_review": "The paper tackles an important problem in FL, but the assumptions involved are unrealistic, the novelty is limited, and the experiments are not comprehensive.", "strength_and_weaknesses": "Strengths: \n1.\tEvaluation of the proposed method for both vision and language tasks. \n2.\tComparison of LTI with current SOTA gradient inversion attacks.\n3.\tAblation results based on the auxiliary dataset (both degree of overlap and size of the dataset).\n\nWeaknesses:\n\n1.    The attack works only in the presence of an auxiliary dataset (which has some overlap with the client distributions) at the server. Hence, the comparison with SOTA gradient inversion methods is not fair. For a fair comparison, other inversion methods should be updated to make use of the auxiliary dataset. When $\\beta=0$ (no overlap between the auxiliary and training sets), the MSE of the proposed method is the worst. Note that even for $\\beta=0.01$, there will be significant number of common samples between the two sets, which explains the superiority of the proposed method. Ideally, Tables 1 & 2 and Figures 2 & 4 should be reported for low values of $beta$ (< 0.1) or with an auxiliary dataset that is completely different (some other natural image and NLP dataset that is different from the training).\n\n2.  The main experimental setup is designed such that the training samples and samples in the auxiliary dataset are the SAME. Furthermore, the gradients are computed based on a single sample. So, effectively there is one-to-one mapping between a sample and its gradient, which any network can easily learn (all one needs is a indexing table!). So, the results in Table 1 and Figure 2 are unsurprising - in fact, it is somewhat underwhelming because exact reconstruction should be possible in this setting unless there is loss of some information in the dimensionality reduction step. \n\n3. There is no information about how the proposed approach will scale to higher fidelity data (say, images of 224 x 224 x 3 resolution). The MLP approach is likely to become practically infeasible for higher fidelity data.\n\n4. The other key aspect that is missing is the FL round in which the reconstruction is attempted. Several works in the literature have shown that reconstruction is easier in the first few rounds (when training from scratch), while it becomes harder in the later rounds. That is why multiple rounds of local training is typically carried out before the collaboration starts and this serves as a good defense against gradient inversion attacks. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "r9-bf1GbltP", "reviewer": "Reviewer_PX2s", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new gradient inversion attack against federated learning. By assuming that the server has access to some auxiliary data, the main idea of the paper is to learn a gradient inversion model from the auxiliary data. The paper shows that compared with previous optimization-based approaches such as Inverting Gradients (IG), Gradient Inversion with Generative Image Prior (GI-GIP), and TAG, the proposed learning to invert (LTI) method obtains better reconstruction accuracy on both an image task (CIFAR-10) and a language task (Wikitext), under both gradient perturbation and compression based defenses. ", "summary_of_the_review": "The paper departs from existing optimization-based methods and proposes a simple learning-based approach to gradient inversion against federated learning. The approach is effective in the simple setting when the batch size is 1, and the server adopts gradient perturbation or gradient compression-based defenses. It is unclear if it can be applied to practical FL systems with larger batch sizes and strong defenses. Thus, the claim that existing defenses provide a false sense of security may not hold. ", "strength_and_weaknesses": "Strengths\n\n1. The idea of learning an inversion model using auxiliary data seems a simple yet effective approach. \n2. Ablation studies show that even when the auxiliary data only consists of 500 images sampled from CIFAR-10 or contains 250 in-distribution samples, LTI still outperforms IG and GI-GIP, which is impressive. \n\nWeaknesses\n\n1. An important limitation of the proposed approach is that it only works for gradients computed from a single data sample. For a real FL system, even without using secure aggregation, a local update from a client is obtained by taking the gradient of a batch of data samples or through multiple gradient descent steps. Thus, the proposed approach is not sophisticated enough to be applied to real FL systems. \n2. Another limitation is that the proposed attack method is only tested against gradient perturbation and gradient compression, which were not originally designed for countering gradient inversion. It would be useful to understand how the proposed method performs against more recent defenses, such as Soteria [1], that target gradient inversion. \n\n[1] Jingwei Sun, Ang Li, Binghui Wang, Huanrui Yang, Hai Li, and Yiran Chen. Soteria: Provable defense against privacy leakage in federated learning from representation perspective. CVPR 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "pYW3jqF9f6", "reviewer": "Reviewer_8QEs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a simple learning-based gradient inversion attack. The new method is trained using auxiliary data and can learn how to invert gradients on both vision and language tasks.  A new learning function is built to learn model parameters on the auxiliary data.  ", "summary_of_the_review": "See the above. ", "strength_and_weaknesses": "1. To learn the parameter theta, a large number of auxiliary data are required. This is also a limitation of the proposed method. A challenging problem would be reduce the number of the auxiliary data while keeping the performance. \n2. The learning function of eq.(2) is simple yet powerful. A new algorithm is expected to show the learning steps with data input/output and all the parameters used in the algorithm. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_MoS6jnFTI", "reviewer": "Reviewer_SYoc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to recover training data from gradient updates in federated learning. Specifically, it leverages an auxiliary dataset to obtain the gradients for these samples. A neural network model is then trained on them by mapping the gradient to the input. This paper also utilizes existing feature hashing method to reduce the dimensionality of the input gradient. The evaluation is conducted on one image classification dataset and one language task. The experimental results show the proposed method has better attack performance compared to existing techniques against different defense techniques.", "summary_of_the_review": "This paper studies an important problem. However, the proposed method is based on unrealistic and impractical assumptions and settings. The technical novelty is also limited.\n", "strength_and_weaknesses": "### Strength\n\n+ Important topic of gradient inversion attack\n+ Easy to follow\n\n\n### Weaknesses\n\n- Unrealistic assumption\n- Limited novelty\n- Impractical evaluation setup\n- Limited evaluation\n\n\n### Detailed comments\n\n* In the threat model, the paper assumes \"the FL protocol does not leverage secure aggregation\", which is an unrealistic assumption. The purpose of secure aggregation is to avoid revealing private information to other parties in a distributed setting such as federated learning. The paper discards the secure aggregation and builds an attack on top of such an unrealistic assumption. In addition, it assumes the gradient for each sample in the batch (actually it assumes the batch size is 1), which is impractical. It is not meaningful to study the problem in such a setting.\n\n* This paper leverages an auxiliary dataset to train a model for mapping the gradient to the input. Similar ideas have already been studied in many existing works. The dimensionality reduction issue is addressed by directly using an existing feature hashing technique. There is very limited technical contribution in this paper.\n\n* The paper uses the term \"auxiliary dataset\" to denote a dataset different from the training data by the subject model. However, in the evaluation, the paper directly uses the training data that the subject was trained on to construct their gradient inversion model, which is impractical.\n\n* The evaluation is conducted on both computer vision and natural language processing, which is good. However, only one dataset in each domain is evaluated. The model used on the computer vision task is LeNet, which is a very simple model structure. There is no evaluation on advanced and complex model structures.\n\n* Important details are missing in the paper. There is no data showing the performance of the subject model on corresponding tasks. The baseline GI-GIP uses ImageNet to train the generator in the original paper. It is unclear whether the authors directly use the trained generator from GI-GIP or retrain the generator on CIFAR-10. If it is the former case, the comparison is not considered fair.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A_5tmJqWp_j", "reviewer": "Reviewer_n8di", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates potential privacy risk in federated learning. They found that existing privacy defenses in FL can be broken via a simple adaptive attack. In particular, the proposed learning-based approach (Learning to invert) aims to train an inversion model to reconstruct training samples from their gradient with the help from auxiliary dataset.  Experiments demonstrate the effectiveness of the proposed model. \n", "summary_of_the_review": "Strength:\n1. The research problem is very important and this paper provides a simple but effective attacking method to conduct gradient inversion attacks in federated learning. \n2. This paper is easy to follow and clearly written.  \n3. The experimental results demonstrate the effectiveness of the proposed method. \n\n\nWeaknesses\n\nThere are some concerns regarding this paper.\n\n1. Auxiliary datasets are used to help learn the inversion model for privacy attacks in FL. In such cases, this task can be considered as a malicious server that wants to steal private information from some clients. The attacker task is weird to me. To steal information from one client, why not just a serve and a client pair, and attack this client directly? Such an attacking strategy is not hard to achieve. \n\n2. The batch size is set to 1. When the batch size increases, the proposed method can perform much worse. This may limit their applications to many real-world tasks, where many FL methods would consider using large batch sizes. Meanwhile, it's somehow unfair compared with other privacy attacks in FL.\n", "strength_and_weaknesses": "Strength:\n1. The research problem is very important and this paper provides a simple but effective attacking method to conduct gradient inversion attacks in federated learning. \n2. This paper is easy to follow and clearly written.  \n3. The experimental results demonstrate the effectiveness of the proposed method. \n\n\nWeaknesses\n\nThere are some concerns regarding this paper.\n\n1. Auxiliary datasets are used to help learn the inversion model for privacy attacks in FL. In such cases, this task can be considered as a malicious server that wants to steal private information from some clients. The attacker task is weird to me. To steal information from one client, why not just a serve and a client pair, and attack this client directly? Such an attacking strategy is not hard to achieve. \n\n2. The batch size is set to 1. When the batch size increases, the proposed method can perform much worse. This may limit their applications to many real-world tasks, where many FL methods would consider using large batch sizes. Meanwhile, it's somehow unfair compared with other privacy attacks in FL.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject