LAMP: Extracting Text from Gradients with Language Model Priors
Authors: Mislav Balunovic
Dimitar Iliev Dimitrov
Nikola Jovanović
Martin Vechev
Venue: NeurIPS 2022 Accept (2022)
Keywords: federated learning
privacy
gradient leakage
natural language processing

Abstract: Recent work shows that sensitive user data can be reconstructed from gradient updates, breaking the key privacy promise of federated learning. While success was demonstrated primarily on image data, these methods do not directly transfer to other domains such as text. In this work, we propose LAMP, a novel attack tailored to textual data, that successfully reconstructs original text from gradients. Our attack is based on two key insights: (i) modelling prior text probability via an auxiliary language model, guiding the search towards more natural text, and (ii) alternating continuous and discrete optimization which minimizes reconstruction loss on embeddings while avoiding local minima via discrete text transformations. Our experiments demonstrate that LAMP is significantly more effective than prior work: it reconstructs 5x more bigrams and $23\%$ longer subsequences on average. Moreover, we are first to recover inputs from batch sizes larger than 1 for textual models. These findings indicate that gradient updates of models operating on textual data leak more information than previously thought.




LAMP: Extracting Text from Gradients with Language Model Priors
Mislav Balunović; Dimitar I. Dimitrov; Nikola Jovanović, Martin Vechev
(mislav balunovic, dimitar. commtarr.ilan illie. dimitrov, nikola. jakkooa.oovnooicc,o ovanovicc,a martin. vechev)@inf .tf...thhcc Department of Computer Science ETH Zurich
Abstract
Recent work shows that sensitive user data can be reconstructed from gradient updates, breaking the key privacy promise of federated learning. While success was demonstrated primarily on image data, these methods do not directly transfer to other domains such as text. In this work, we propose LAMP, novel attack tailored to textual data, that successfully reconstructs original text from gradients. Our attack is based on two key insights: (i) modeling prior text probability with an auxiliary language model, guiding the search towards more natural text, and (ii) alternating continuous and discrete optimization, which minimizes reconstruction loss on embeddings, while avoiding local minima by applying discrete text transfor- mations. Our experiments demonstrate that LAMP is significantly more effective than prior work: reconstructs 5x more bigrams and 23% longer subsequences average. Moreover, we the first to recover inputs from batch sizes larger than for textual models. These findings indicate that gradient updates of models operating textual data leak more information than previously thought.
1 Introduction
Federated learning [24] (FL) is widely adopted framework for training machine learning models in decentralized way. Conceptually, FL aims to enable training of highly accurate models without compromising client data privacy, as the raw data never leaves client machines. However, recent work [28, 43, 41] has shown that the server can in fact recover the client data, by applying reconstruction attack on the gradient updates sent from the client during training. Such attacks typically start from randomly sampled input and modify it such that its corresponding gradients match the gradient update originally sent by the client. While most works focus on reconstruction attacks in computer vision, there has comparatively been little work in the text domain, despite the fact that some of the most prominent applications mobile phones [30]. A key component of successful attacks vision has been the of image priors such as total variation [7]. These priors guide the reconstruction towards natural images, which are more likely correspond client data. However, the use of priors has so far been missing from attacks on text [43, 3], limiting their ability to reconstruct real client data.
FL involve learning over textual data, e.g., next-word prediction on
This work: private text reconstruction with priors In this work, we propose LAMP, new reconstruction attack which leverages language model priors to extract private text from gradients. The overview of our attack is given in Fig. 1. The attacker has access network being trained in federated manner (e.g.. BERT), and gradient V VoL(x", .y") which the client has computed on that snapshot, using their private data, The attack starts by sampling token
snapshot of the transformer
* Eqqual contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

lnput
password
my password
doodoo
ora
Gonn
togunii (x') 4ult)
(x') Cull
fox is mine input mine input fox
password my cat passwoed is pussord is my cat passward
mine input ungut mune Fox C-
whistle sleep year
angut mine fox
my password is cat
Figure 1: An overview of LAMP. We initialize the reconstruction by sampling from Gaussian distribution, and alternate between continuous and discrete optimization. Continuous optimization minimizes the reconstruction loss with an embedding regularization term. Discrete optimization forms candidates by applying transformations, and chooses the best candidate based on a combination of reconstruction loss and perplexity, as measured by an auxiliary language model (e.g.. GPT-2).
embeddings from Gaussian distribution to create the initial reconstruction. Then, at each step, we improve the reconstruction (shown in yellow) by alternating between continuous (blue) and discrete optimization (green). The continuous part minimizes the reconstruction loss, which measures how close the gradients of the current reconstruction are to the observed client gradients, together with an embedding regularization term. However, this is insufficient as the gradient descent can get stuck in local optimum due to its inability to make discrete changes to the reconstruction. We address this issue by introducing a discrete step-namely, we generate a list of candidate sentences using several transformations on the sequence of tokens (e.g., moving token) and select a candidate that minimizes the combined reconstruction loss and perplexity, which measures the likelihood of observing the text in a natural distribution. We use GPT-2 [29] as an auxiliary language model to measure the perplexity of each candidate (however, our method allows using other models). Our final reconstruction is computed by setting each embedding to its nearest neighbor from the vocabulary.
Key component of our reconstruction attack is the use of language model prior combined with search that alternates continuous and discrete optimization steps. Our experimental evaluation demonstrates the effectiveness of this approach-LAMP LAMP able to extract text from state-of-the-ar transformer models several common datasets, reconstructing up to 5 times more bigrams than prior work. Moreover, we are the first to perform text reconstruction more complex settings such as batch sizes larger than 1, fine-tuned models, and defended models. Overall, across all settings we demonstrate that LAMP effective reconstructing large portions private text.
Main contributions Our main contributions are:
LAMP, novel attack for recovering input text from gradients, which leverages an auxiliary language model to guide the search towards natural text, and a search procedure which alternates continuous and discrete optimization.
An implementation of LAMP and its extensive experimental evaluation, demonstrating that it can reconstruct significantly more private text than prior work. We make our code publicly available at pttpss//g/t /i thub. com/eth- sri/lamp.
The first thorough experimental evaluation of text attacks in more complex settings such as larger batch sizes, fine-tuned models and defended models.
2
Related Work
Federated learning [24, 18] has attracted substantial interest [16] due to its ability to train deep learning models a decentralized way, such that individual user data not shared during training. Instead, individual clients calculate local gradient updates on their private data, and share them with centralized server, which aggregates them to update the model [24]. The underlying assumption that user data cannot be recovered from gradient updates. Recently, several works [28, 43, 41, 42]

---

demonstrated that gradients can in fact still leak information, invalidating the fundamental privacy assumption. Moreover, recent work achieved near pear-perfect image reconstruction from gradients 47,40, 14]. Interestingly, prior work showed that an auxiliary model [ [14] or prior information [2] can significantly improve reconstruction quality. Finally, Huang et al. [12] noticed that gradient leakage attacks often make strong assumptions, namely that batch normalization statistics and ground truth labels are known. In our work, we do not assume knowledge of batch normalization statistics and as we focus on binary classification tasks, we can simply enumerate all possible labels.
Despite substantial progress on image reconstruction, attacks in other domains remain challenging. as the techniques used for images rely extensively on domain specific knowledge. In the domain of text, in particular, where federated learning is often applied [32], only handful of works exist [43, 3, 22]. DLG [43] was first to attempt reconstruction from gradients coming from transformer; TAG [3] extended DI.G by adding an I, term the reconstruction loss; finally, unlike TAG and DLG which are optimization-basee techniques, APRIL [22] recently demonstrated an exact gradient leakage technique applicable to transformer networks. However, APRIL assumes batch size of and learnable positional embeddings, which makes it simple to defend against. Another attack on NLP is given Fowl et al. [6], but they use stronger assumption that the server can send malicious updates to the clients. Furthermore, there is a concurrent work [11] on reconstructing text from transformers, but it is limited to the case when token embeddings are trained together with the network.
Finally, there have been several works attempting to protect against gradient leakage. Works based on heuristics [34, 31] lack privacy guarantees and have been shown ineffective against stronger attacks [2]. while those based on differential privacy do train models with formal guarantees [1], but typically hurt the accuracy the trained models as they require adding noise to the gradients. We remark that we also evaluate LAMP on defended networks.
3 Background
In this section, we introduce the background necessary to understand our work.
3.1 Federated Learning
In federated learning, clients aim to jointly optimize a neural network f with parameters 0 on their private data. At iteration k the parameters 8 are sent to all clients, where each client executes a gradient update Ve ((=" y²) on a sample (x", y.) from their dataset (Xe, Ve). The updates are sent back to the server and aggregated. While in Sec. 5 we experiment with both FedSGD [24] and FedAvg [19] client updates, throughout the text we assume that clients use FedSGD updates:
A 2 Vo C(x C
g²+1 = -
Gradient leakage attacks A gradient leakage attack is attack executed by the server (or party which compromised it) that tries to obtain the private data (x" v²)) of client using the gradient updates Vo L(x", v&) sent to the server. Gradient leakage attacks usually assume honest-but-curfe servers which are not allowed to modify the federated training protocol outlined above. A common approach, adopted by Zhu et al. [43], Zhao al. [41], Deng et al. [3] well as work, is obtain the private data by solving the optimization problem:
arg min 5 (Vez L (m ve), Vo (me, vc)) (R))
where is some distance measure and (a,, Uc.) denotes dummy data optimized using gradient descent to have similar gradients V L(x Me) true data (a= v²)- Common choices for are La [43], L [3] and cosine distances [7]. When the true label ye is known, the problem reduces to
arg min 8 (Vor (a vt) VonL (xc)
which shown [41, 12] to be simpler to solve with gradient descent approaches.
3

---

3.2 Transformer Networks
In this paper, we focus on the problem of gradient leakage of text on transformers [35]. Given some input text, the first step is to tokenize it into tokens from some fixed vocabulary of size V. Each token then converted a 1-hot vector denoted t,. ......... ........, t,, E RV, where n is the number of tokens in
the text. The tokens are then converted to embedding vectors 2 , 22,,,...... Fin E Rd where d is
chosen embedding size, by multiplying by a trained embedding matrix W embbed R. Vxd [4]. The rows of W embed represent the embeddings of the tokens, and we denote them as c1, ..,..., Gy E R In addition to tokens, their positions in the sequence are also encoded using the positional embedding matrix W pos E Re where /' is the longest allowed token sequence. The resulting positional embeddings are denoted pl P2....... Pn- For notational simplicity, we denote e = El E2,,.. ev, #1; #2:.... ................................ T and p .2.P....... Pn- We use the token-wise sum of the embeddings æ and as an imput to a sequence of self-attention layers [35]. The final classification output is given by the first output of the last self-attention layer that undergoes final linear layer, followed by a tanh.
3.3 Calculating Perplexity on Pretrained Models
In this work, rely large pretrained language models, such GPT-2 [29], to assess the quality of the text produced by the continuous part of our optimization. Such models are typically trained
calculate the probability P(tn t1, f,,,,., 11- of inserting token t, from the vocabulary of tokens to the end of the sequence of tokens L, l,,... tn-1. Therefore, such models can be leveraged calculate the likelihood of sequence of tokens L,, ,,.... ln as follows:
1-
P(L 12 -........ bx)
II P(t++ 1, (..t2...,.1).. 44) E=
One use the likelihood, or the dosely-rrlater negative log- ligeleikdhoodd, of produced sequence. However, the likelihood depends on the length of the sequence, as probability decreases with length. To this end, the perplexity measure [13], defined as:
measure of the quality
Lim (f1,12, bn)
Σ 1=0
log P(t++1 tt, t,,t2,..,tt tt).
In the discrete part of our optimization, we rely on this measure to assess the quality of reconstructed sequences produced by the continuous part.
4 Extracting Text with LAMP
In this section we describe the details of our attack which alternates between continuous optimization using gradient descent, presented in Sec. 4.2, and discrete optimization using language models to guide the search towards more natural text reconstruction, presented in Sec. 4.3.
4.1 Notation
We denote the attacked neural network and its parameters with f and 0, respectively. Further, we denote the client token sequence and its label as (t", (f", and our reconstructions (t, y). For each token E in t" and t; in f, we denote their embeddings with 2 E R and Moreover, for each token in our vocabulary, we denote the embedding with e; E Rd. We collect Rdxx x= Rdxn and e E RR where n is the number of tokens in t" and V is the size of the vocabulary.
Rd, respectively.
the individual embeddings 2;, 2t and e; into the matrices
4.2 Continuous Optimization
We now describe the continuous part of our attack (blue in Fig. 1). Throughout the paper, we assume knowledge of the ground truth label y" of the client token sequence we aim to reconstruct, meaning y y" This assumption is not significant restriction as we mainly focus on binary classification, with batch sizes such that trying all possible combinations of labels is feasible. Moreover, prior work [9, 40] has demonstrated that labels can easily be recovered for basic network architectures, which can be adapted for transformers in future work. We initialize our reconstruction candidate by sampling embeddings from Gaussian and pick the one with the smallest reconstruction loss.

---

Reconstruction loss A key component of our attack is a loss measuring how close the reconstructed gradient is the true gradient. Assuming an l-layer network, where 6 denotes the parameters of layer i, an option is to use the combination of L2 and L1 loss proposed by Deng et al. [3],
Lug (x)
Ve, V y)1 + Owg f(x", Vo.f(zyy)ln
where Olug a hyperparameter. Another option to use the cosine reconstruction loss proposed by Geiping et al. [7] in the image domain:
Vo.f(x v²) Vo.f(a,)) Ve.f(= y*) f
Coos (z) = -1 Σ
f(, 8)1l2
Naturally, LAMP can also be instantiated using any other loss. Interestingly, we find that there is no loss that is universally better, and the effectiveness is dataset dependent. Intuitively, L1 loss is less sensitive outliers, while cosine loss is independent of the gradient norm, so works well for small gradients. Thus, we set the gradient loss Egrad to either Ligg or Leoss depending on the setting.
Embedding regularization In the process of optimizing the reconstruction loss, we observe the resulting embedding vectors æ; often steadily grow in length. We believe this behavior is due the self-attention layers in transformer networks that rely predominantly on dot product operations. As result, the optimization process focuses on optimizing the direction of individual embeddings æi disregarding their length. To address this, we propose an embedding length regularization term:
2
V l
1 Σ
Cog(()) (x) =
o 1le
12
The regularizer forces the mean length of the embeddings of the reconstructed sequence to close to the mean length of the embeddings in the vocabulary. The final gradient reconstruction error optimized in LAMP is given by:
Lree (a)) = Lerna (a) + O'reg (x))
where O'reg is regularization weighting factor.
Optimization We summarize how described components work together in the setting of continuous optimization. To reconstruct the token sequence t* we first randomly initialize a sequence of dummy token embeddings æ 26 En, with æ; e Rd. Following prior work on reconstruction from gradients [3, 43], we apply gradient descent on æ to minimize the reconstruction loss Cree (x). To this end, second-order derivative needs be computed, Liee (xx depends on the network gradient at x. Similar to prior work [3, 43], we achieve this using automatic differentiation in Pytorch [27].
4.3 Discrete Optimization
Next, we describe the discrete part of our optimization (green in Fig. 1). While continuous opti- mization can often successfully recover token embeddings close the original, they can be in the wrong order, depending on how much positional embeddings influence the output. For example, reconstructions corresponding to sentences "weather is nice." and "nice weather is." might result in similar reconstruction loss, though the first reconstruction has a higher likelihood of being natural text. To address this issue, perform several discrete sequence transformations. and choose the one with both a low reconstruction loss and low perplexity under the auxiliary language model.
Generating candidates Given the current reconstruction æ 2 F.. we generate candidates for the new reconstruction x' using one of the following transformations:
Swap: We select two positions i and j in the sequence uniformly at random, and swap the tokens æ; and my at these two positions obtain a new candidate sequence x' 2122 ... 81-1 11-10 ... -.. -1 +1.1....... En.
5

---

MoveToken:
Similarly, we select two positions i and j in the sequence uniformly at random, and move the token z; after the position j in the sequence, thus obtaining Enn-
x' 21 Zi Pii-14++1... 000_
MoveSubseq: We select three positions i. and p (where i < j) uniformly at random, and move the subsequence of tokens between i and j after position p. The new sequence is thus x' 21 ... 11-1 ... Tp ... 2,
MovePrefix: We select a position uniformly at random, and move the prefix of the sequence ending at position i to the end of the sequence. The modified sequence then is æ' 1.t.....,,111...................
Zz.
Next, we use language model to check if generated candidates improve over the current sequence.
Using a language model to select candidates We accept the new reconstruction æ' if improves the combination of the reconstruction loss and perplexity:
free(1) (xx) + Othnn (im
(re(()) + (im
Here t and t' denote sequences of tokens obtained by mapping each embedding of and x' the nearest neighbor in the vocabulary according to the cosine distance. The term fiee is the recon- struction loss introduced in Sec. 4.2, while Lim denotes the perplexity measured by an auxiliary language model, such GPT-2. The parameter Onm determines the trade-off between free and Lim: if it is too low then the attack will not utilize the language model, and if is too high then the attack will disregard the reconstruction loss and only fo- cus on the perplexity. Going back to our example, assume that reconstruction equals the second sequence "nice weather is.". Then, at some point, might the MoveToken transformation move the word "nice" behind the word "is" which would presumably keep the reconstruction loss similar, but drastically improve perplexity.
Algorithm Extracting text with LAMP
1: (b() ~ NO, I), where ls = 1,...y Tlinit
2: x +- arg min (1) Looe (zr(k))
3: for = to il do
4: for j = to nhe do
< AV (rco (2)
6: end for
7:
Bbest x
8:
thest +- PROJECTTOVOCAT (bbea))
9:
Loeat f- Liee (Fees) ) + Cr Cim (tbeat
10:
for = 1 to na do
11:
<- TRANSFORM(())
12:
t' < PROJECTTOVOCAT
13:
T' <- Laec(a") + am
14:
if L' Liest then
Thest, bbest Lioess + x' L' end if 17: end for
15:
16:
18: f- Woest
4.4 Complete Reconstruction Attack
19: end for
20: return PROJECTTOVOCAB(22
We present our end-to-end attack in Algorithm I. We initialize the reconstruction by sampling from a Gaussian distribution Tiini times, and choose the sample with minimal reconstruction loss initial reconstruction. Then, at each step we alternate between continuous and discrete optimization. We first perform n. steps of continuous optimization to minimize the reconstruction loss (Lines 4-6, Sec. 4.2). Then, we perform na steps of discrete optimization to minimize the combination of reconstruction loss and perplexity (Lines 10-17, see Sec. 4.3). Finally, in Line 20 we project the continuous embeddings respective nearest tokens, according to cosine similarity.
Experimental Evaluation
We now discuss our experimental results, demonstrating the effectiveness of LAMP compared to prior work in wide range of settings. We present reconstruction results several datasets, architectures, and batch sizes, together with the additional ablation study and evaluation of different defenses and training methods.
Datasets Prior work [3] has demonstrated that text length is key factor for the success of recon- struction from gradients. this end, in our experiments we consider three binary classification datasets of increasing complexity: CoLA [37] and SST-2 [33] from GLUE [36] with typical sequence

---

lengths between 5 and 9 words, and 3 and 13 words, respectively, and Rotten [26] with typi- cal sequence lengths between 14 and 27 words. The CoLA dataset contains English sentences from language books annotated with binary labels describing if the sentences are grammatically correct, while SST-2 and Rotten'Tomatos contain movie reviews annotated with a binary sentiment. For all experiments, we evaluate the methods on 100 random sequences from the respective training sets. We remark that attacking in binary classification setting is a more difficult task than in the masking setting considered by prior work [43], where the attacker can utilize strictly more information.
Models Our experiments are performed on different target models based on the BERT [4] archi- tecture. The main model we consider is BERT BASE. which has 12 layers. 768 hidden units, 3072 feed-forward filter size, and 12 attention heads. To illustrate the generality of our approach with respect to model size, we additionally consider larger model BERTLARGE. which has 24 layers, 1024 hidden units, 4096
filter size, and 16 attention heads as well as a smaller model TinyBERT, 6 from Jiao al. [15] with 6 layers, 768 hidden units, feed-forwant filter size of 3072 and 12 attention heads. All models were taken from Hugging Face [39]. The BERT BASE and BERT LARGE were pretrained from BASE. We perform our main experiments on pretrained models, as this is the most common setting for training classification models from text [25]. For the auxiliary language model we use the pretrained GPT-2 provided by Guo et al. [10], trained on the same tokenizer used to pretrain our target BERT models.
Wikipedia [5] and BookCorpus [44] datasets, while TinyBERT, & was distilled
Metries Following TAG [3], we measure the success of our methods based on the ROUGE family of metrics [20]. In particular, we report the aggregated F-scores on ROUGE-1, ROUGE-2 and ROUGE-L, which measure the recovered unigrams, recovered bigrams and the ratio of the length of the longest matching subsequence to the length of whole sequence. When evaluating batch sizes greater than 1. we exclude the padding tokens, used to pad shorter sequences, from the reconstruction and the ROUGE metric computation.
Experimental setup In all settings we consider, we compare our method with baselines DLG [43] and TAG [3] discussed in Sec. As TAG does not have public code, we use our own implementation. and remark that the results obtained using our implementation are similar or better than those reported in Deng et al. [3]. We consider two variants of our attack, LAMP and LAMP, A.,.,,, + I,, that use the Lexs and Lung gradient matching losses for the continuous optimization. For the BERTBASE and TinBBERT, & experiments, we run our attack with it 30, ne = 75 and na = 200, and stop the optimization early once we reach total of 2000 continuous optimization steps. For the BERT model, whose number of parameters make the optimization harder, we use it = 25 and n. = 200 instead, resulting in 5000 continuous optimization steps. We run DLG and TAG for 10 000 optimization steps on BERTLARGE and 2500 on all other models. For the continuous optimization, use Adam [17] with a learning rate decay factor y applied every 50 steps for all methods and experiments, except for BERTLARGE ones where, following Geiping et al. [8], we use Adam [21] and linear learning rate decay schedule applied every step. We picked the hyperparameters for TAG, LAMP Cas and 1AMPL, L,, separately on CoLAA and Rotten'Tomator using grid search on BERTBASE and applied them to all networks. As the optimal hyperparameters for Rotten' Tomatoes exactly matched the ones on CoLA, we used the same hyperparameters on SST-2, as well. To account for the different optimizer used for BERTLARGE models, we further tuned the learning rate À for BERTLARGE experiments separately, keeping the other hyperparameters fixed. Additionally, for our methods we applied a two-step initialization procedure. We first initialized the embedding vectors with 500 random samples from a standard Gaussian distribution and picked the best one according to (z). We then computed 500 permutations on the best initialization and chose the best one in the same way. The effect of this procedure is investigated in App. C.3. Further details on our experimental setup are shown in App. D.
Main experiments We evaluate the two variants of LAMP against DLG [43] and TAG [3] on BERTBASE BERTLARGE and TinyBERT,- Additionally, we evaluate attacks after BERT BASE has already been fine-tuned for 2 epochs on each task (following Devlin et al. [4]), as Balunović et al. [2] showed that in the vision domain it significantly more difficult to attack already trained networks. For both baselines and our attacks, for simplicity we assume the lengths of sequences are known, as otherwise an adversary can simply run the attack for all possible lengths. the first experiment we consider setting where batch size is equal to I. The results are shown in Table 1. From the ROUGE-1
7

---

Table 1: Main results of text reconstruction from gradients with LAMP, for various datasets and architectures in the setting with batch size equal to FT denotes a fine-tuned model. R-1, R-2, and R-L, denote ROUGE-1, ROUGE-2 and ROUGE-L scores respectively.
BERTBASS
BERT
TinyBERT,
BERT LARGE
R-1
R-2
R-L.
R-1
R-2
R-I.
R-1
R-2
R-I.
R-1
R-2
R-I.
DLG
59.3
1.7
46.2
36.2
2.0
30.4
37.7
3.0
33.7
82.7
10.5
55.8
TAG
78.9
10.2
53.3
40.2
3.1
32.3
43.9
3.8
37.4
82.9
14.6
55.5
CoLA
LAMPcus
85.8
73.1
59.3
80.2
92.0
56.0
78.8
LAMP T.
87.5
47.5
73.2
40.3
9.3
35.2
94.5
52.1
76.0
91.2
47.8
75.4
DLG
65.4
17.7
54.2
36.0
2.7
33.9
42.0
5.4
39.6
78.4
18.1
59.0
TAG
75.6
18.9
57.4
40.0
5.7
36.6
43.5
9.4
40.9
80.8
16.8
59.1
SST-2
LAMP
88.8
56.9
77.7
87.6
54,1
76.1
91.6
58.2
79.7
88.5
55.9
76.5
LAMP 22
88.6
57.4
75.7
39.3
89.7
53.2
75.4
89.3
55.5
75.9
DLG
38.6
1.4
26.0
20.1
0.4
15.2
20.4
1.1
17.7
66.8
3.1
35.4
Rotten Tomatoes
TAG
60.3
3.5
26.7
0.9
25.8
1.5
20.2
73.6
36.8
LAMP
64.7
16.3
43.1
63.4
13.8
42.6
76.0
28.6
55.8
73.4
15.7
45.4
LAMP Lg
51.4
10.2
34.3
17.2
1.0
14.7
74.0
19.4
46.7
77.6
16.6
45.8
metric, we can observe that we recover more tokens than the baselines all settings. Moreover, the main advantage of LAMP that the order of tokens in the reconstructed sequences matches the order in target sequences much more closely, as evidenced by the large increase in ROUGE-2 (5x on CoLA). This observation further backed by the ROUGE-L metric that shows we to reconstruct up to 23% longer subsequences on the BERT model compared to the baselines. reconstruct
average able
These results confirm our intuition that guiding the search with GPT-2 allows sequences that are a much closer match to the original sequences. We point out that Table reaffirms the observations first made Deng al. [3], that DLG consistently worse in all metrics compared to both TAG and LAMP, and that the significantly longer sequences in Rotten Tomatoes still pose challenges to good reconstruction.
Our results show that smaller and fine-tuned models also leak significant amount of client information. In particular, TEnyBERT, even more vulnerable than BERT Base and BERTRASE-FT shown to be only slightly worse in reconstruction compared to BERTRASE which is surprising given the prior image domain results. This shows that smaller models can not resolve the privacy issue, despite previous suggestions in Deng et al. [3]. Additionally, our BERT experiments reaffirm the observation Deng al. [3] that the model is highly vulnerable all attacks.
Further, we examine the variability of our LAMP Cos method with respect to random initialization. To this end, R-2 and R-L of 88.2 + 1.02, 50.0 + 2.37, 75.0 + 1.21, respectively, which suggests that our results consistent. Further, assess the variability with respect sentence choice in App. C.1.
the BERT BASE experiment on CoLA with 10 random seeds, which produced R-1,
Larger batch sizes Unlike prior work. we also evaluate the different attacks on updates computed on batch sizes greater than I on the BERT model to investigate whether we can reconstruct some sequences in this more challenging setting. The results are shown in Table 2. Similarly the results in Table 1, we observe that we obtain better results than the baselines on all ROUGE metrics in experiments, except on Tomatoes with batch size 2, where TAG obtains slightly better ROGUE-1. Our experiments show that for larger batch sizes we can also reconstruct significant portions of text (see experiments on CoL.A and SST-2). To the best of our knowledge, we are the first to show this, suggesting that gradient leakage can be realistic security threat in practice. Comparing the results for LAMPL.+1. and LAMP we observe that Loos is better than Ling in almost all metrics batch size 1, across models, but the trend reverses as batch size is increased.
Sample reconstructions We show sample sequence reconstructions from both LAMP and the TAG baseline on CoLA with B = in Table 3, marking the correctly reconstructed bigrams with green and correct unigrams with yellow. We can observe that our reconstruction more coherent, and that it qualitatively outperforms the baseline. In App. B, we show the convergence rate of our method compared to the baselines on an example sequence, suggesting that LAMP can often converges faster.

---

Table 2: Text reconstruction from gradients for different batch sizes B on the BERT BASE model. R-1,
R-2, and R-L, denote ROUGE-I 1, ROUGE-2 and ROUGE-L scores respectively.
B=1
B=2
B=4
R-1
R-2
R-L.
R-1
R-2
R-I.
R-1
R-2
R-L
DLG
59.3
7.7
46.2
49.7
5.7
41.0
37.6
1.7
34.0
TAG
78.9
10.2
53.3
68.8
7.6
49.0
56.2
6.7
44.0
CoLA
LAMP
89.6
76.2
74.4
29.5
61.9
14.5
48.0
LAMP 12
87.5
47.5
73.2
78,0
31,4
63.7
66.2
21.8
55.2
DLG
65.4
17.7
54.2
57.7
11.7
48.2
43.1
6.8
39.4
TAG
75.6
18.9
57.4
71.8
16.1
54.4
61.0
12.3
48.4
SST-2
LAMP
88.8
56.9
77.7
72.2
37.0
63.6
57.9
23.4
52.3
LAMP +2
88.6
57.4
75.7
82.5
45.8
70.8
69.5
32.5
59.9
DLG
38.6
1.4
26.0
29.2
1.1
23.0
21.2
0.5
18.6
Rotten Tomatoes
TAG
60.3
3.5
47.4
29.5
1.4
23.5
LAMP
64.7
16.3
43.1
37.4
5.6
29.0
25.7
1.8
22.1
LAMP L2
51.4
10.2
34.3
46.3
7.6
32.7
35.1
4.2
27.2
Table 3: The result of text reconstruction several examples from the dataset (for BERTBASE with B=1). show only TAG (better baseline) and LAMP as it is superior these cases.
Sequence
Reference
mary has never kissed man who taller than john.
CoLA
man seem taller than mary... kissed has john mph never
LAMPo-
mary has never kissed a man who taller than john.
Reference
also believe that resident evil is not it.
SST-2
TAG
resident or. is pack down believe
evil
LAMPas
also believe that resident resident evil not
Reference well made and often lovely depiction of the mysteries of friendship.
Rotten
Tomatoes
TAG
the
friendship taken and
lovely a made often depiction of
well mysteries
LAMP Ooe
a well
often made
and lovely depiction mysteries
mysterics of friendship
Ablation studies In the next experiment, perform ablation studies examine the influence of each proposed component of our method. We compare the following variants LAMP: (i) with cosine loss, (ii) with L, L2 loss, (iii) with Lg loss, (iv) without the language model (amm 0), (v) without embedding regularization (Areg = 0). (vi) without alternating of the discrete and continuous optimization eteps-executing il The continuous optimization steps first, followed by il discrete optimizations with n4 steps each, (vii) without discrete transformations (n, = 0). For this experiment, the CoLAA dataset and BERT DASE with B 1. We show the results in Table 4. We observe that LAMP achieves good results with both losses, though cosine is generally better for batch size More importantly, dropping any of the proposed features makes ROUGE-1 and ROUGE-2 significantly worse. We note the most significant drop in ROUGE-2 reconstruction quality happens when using transformations without using the language model (LAMP (LAMPa..=u), which performs even worse than doing no transformations language model is crucial to obtaining good results. Further, we observe that our proposed scheme for alternating the continuous and discrete optimization steps is important, as doing the discrete optimization at the end (LAAMP DMMPPDicceeAAEaaal for the same number of steps results in reconstructions only marginally better (in ROUGE-2) compared to the reconstructions obtained without any discrete optimization L.AMPNoDissett The experiments also confirm usefulness of other features such as embedding regularization.
CLAMPNoDscred at all. This suggests that the use of the
Attacking defended networks far, all experiments assumed that clients have not defended
against data leakage. Following work on vision attacks [43, 38), we now consider the defense of adding Gaussian noise to gradients (with additional clipping this would correspond to DP-SGD [1]). Note that, usual, there trade-off between privacy and accuracy: adding more noise will lead

---

Table 4: An ablation study with the BERT (B=1) model. We restate the results for LAMP Cos and LAMP L, from Table and introduce four ablations, done on the better of the two variants of LAMP, in these cases LAMP
CoLA
SST-2
Rotten Tomatocs
R-1
R-2
R-L
R-1
R-2
R-L.
R-1
R-2
R-I.
LoMP
89.6
51.9
76.2
88.8
56.9
77.7
64.7
16.3
43.1
LAMP L2+L1
87.5
47.5
73.2
88.6
57.4
75.7
51.4
10.2
34.3
LAMP
69.4
30.1
58.8
72.4
44.1
65.4
31.9
5.5
25.7
LAMP <I
66.9
37.0
68.4
9.9
40.3
LAMP arrg
84.5
38.0
69.1
83.3
44.7
71.9
57.8
11.1
38.3
LAMP Ars
87.4
28.6
66.9
85.4
42.4
71.0
65.0
11.4
42.3
LoMPeeeeserete
86.6
29.6
67.4
84.1
40.0
70.0
61.5
10.2
40.8
Table 5: Evaluation on gradients defended with Gaussian noise, with BERT BASE (B=1) on the
CoLA dataset.
a 0.001
= 0.002
a 0.005
= 0.01
MCC= 0.551
MCC= 0.526
MCC= 0.464
MCC= 0.364
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-L
R-1
R-2
R-L
DLG
60.0
7.2
46.3
61.3
7.5
47.0
58.8
46.4
56.4
6.3
44.8
TAG
70.7
6.0
50.8
67.1
8.4
49.9
64.1
6.5
47.6
59.6
6.5
46.2
LAMP
81.2
42.7
70.6
29.5
60.9
43.3
9.45
39.7
27.7
2.0
27.6
LAMP t
79.2
32.8
64.1
74.3
31.0
61.9
73.5
29.7
60.9
69.6
29.4
60.6
to better privacy, but make accuracy worse. We measure the performance of the fine-tuned models on CoLA using the MCC metric [23] for which higher values are better. The fine-tuning was done for 2 epochs with different Gaussian noise levels a, and we obtained the MCC scores depicted in Table 5. We did not explore noises > 0.01 due to the significant drop in MCC from 0.557 for the undefended model to 0.364. The results of our experiments these defended networks in Table 5. While all methods' reconstruction metrics degrade, as expected, we see that most text is still recoverable for the chosen noise levels. Moreover, our method still outperforms the baselines, and thus shows the importance evaluating defenses with strong reconstruction attacks. In App. C.2 show that LAMP is also useful against defense which masks some percentage of gradients.
presented
Attacking FedAvg So far, we have only considered attacking the FedSGD algorithm. In this experiment, we apply our attack on the commonly used FedAvg [19] algorithm. As NLP models are often fine-tuned using small learning rates (2e-5 to 5e-5 in the original BERT paper), we find that FedAvg reconstruction performance is close FedSGD performance with batch size multiplied by the number of FedAvg steps. We experimented with attacking FedAvg with steps using B = 1 step, 5e-5 on CoLA and BERT BASE with LAMP L, We obtained R-1, R-2 and R-L of 66.5, 21.0, 55.1, respectively, comparable to the reported results on FedSGD with B = 4.
6 Conclusion
In this paper, we presented LAMP, new method for reconstructing private text data from gradients by leveraging language model priors and alternating discrete and continuous optimization. Our extensive experimental evaluation showed that LAMP consistently outperforms prior work on datasets of varying complexity and models of different sizes. Further, we established that LAMP able to reconstruct private data in number of challenging settings, including bigger batch sizes, noise- defended gradients, and fine-tuned models. Our work highlights that private text data is not sufficiently protected by federated learning algorithms and that more work is needed to alleviate this issuc.
10

---

References
[1] Martin Abadi, Andy Chu, lan Goodfellow, H. Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (ACM CCS), pp. 308-318, 2016, 2016. doi: 1114//999674922981111
[2] Mislav Balunovic, Dimitar Dimitrov, Robin Staab, and Martin Vechev, Bayesian framework for gradient leakage. arXiv preprint arXi::2111.0477 2021.
[3] Jieren Deng, Yijue Wang, Ji Li, Chenghong Wang, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. TAG: Gradient attack on transformer-basee language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3600-3610, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2011 10..i55511..2....iiiiinn http:////222...nnnnnennnemiiii
findings-emmlp.305.
[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805 2018.
[5] Wikimedia Foundation. Wikimedia downloads. URL httt::://dumss.wikkii
org.
[6] Liam Fowl, Jonas Geiping, Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Decepticons: Corrupted transformers breach privacy in federated learning for language models. arXiv preprint arXiv:2201.1267. 2022.
[7] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients how easy is break privacy in federated learning? In H. Larochelle, M. Ranzato, R. Had- sell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16937-1660 -1947.16477 Curran Associates, Inc., 2020.
[8] Jonas Geiping, Liam Fowl, and Yuxin Wen, Breaching- a framework for attacks against privacy in federated learning. 2022. URL httpp:://ggth/ii com/ www.://innnnnngggoooee braaching..aai.i....
[9] Jiahui Geng, Yongli Mou, Feifei Li, Qing Li, Oya Beyan, Stefan Decker, and Chumming Rong. Towards general deep leakage in federated learning. arXiv preprint arXiv:2110.09074 2021.
[10] Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kicla. Gradient- adversarial attacks against text transformers. arXiv preprint arXiv:2104.13733 2021.
[11] Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recov- ering private text in federated learning of language models. arXiv preprint 22X.v:200..08514, 2022.
[12] Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 34, 2021.
[13] Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. Perplexity a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977.
[14] Jiwnoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, al. Gradient inversion with generative image prior. Advances in Neural Information Processing Systems, 34, 2021.
[15] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.103: 10351, 2019.
[16] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.000 1912.04977, 2019.

---

[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[18] Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527, 2016.
[19] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05442 2016.
[20] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[21] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httpe://opeerrvieenneet/o/o Bkg6RiCqY7.
[22] Jiahao Lu, Xi Sheryl Zhang, Tianli Zhao, Xiangyu He, and Jian Cheng. April: Finding the achilles' heel on privacy for vision transformers. arXiv preprint 12. 2021.
[23] Brian W Matthews. Comparison of the predicted and observed secondary structure of 14 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 405(2):442-451, 1975.
[24] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-effictn learning of deep networks from decentralized data. In AISTATS, 2017.
[25] Shervin Minace, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jian- feng Gao. Deep lcarning- based text classification: comprehensive review. ACM Computing Surveys (CSUR), 54(3):1 2021.
[26] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACI., 2005.
[27] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, F. Fox, and Garnett, editors, Advances in Neural Information Processing Systems 32, pages
Curran Associates, Inc., 2019. URL http: /twpr//apppesaee/eaa ccpppp.ceer 9815- pytorhh an- imperative style- high- per rwwnnce--eeeeeeeo deep learning Library.
[28] Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy- preserving deep learning: Revisited and enhanced. In A77S, volume 719 of Communicutions in Computer and Information Science, pages 100-110. Springer, 2017.
[29] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[30] Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Françoise Beaufays. Federated learning for emoji prediction in mobile keyboard. CoRR, abs/1906.04339, 2019.
[31] Daniel Scheliga, Patrick Mäder, and Marco Seeland. Precode generic model extension to prevent deep gradient leakage, 2021.
[32] Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to the draw- ing board: A critical evaluation of poisoning attacks on federated learning. arXiv preprint a08. 10241, 2021.
[33] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods natural language processing, pages 1631-1642, 2013,
12

---

[34] Jingwei Sun, Ang Li, Binghui Wang, Huanrui Yang, Hai Li, and Yiran Chen. Soteria: Provable defense against privacy leakage in federated learning from representation perspective. In CVPR, pages 9311-9319 Computer Vision Foundation TEEE, 2021.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages
2017.
[36] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.07461, 2018.
[37] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
[38] Wenqi Wei, Ling Liu, Margaret Loper, Ka-Ho Chow, Mehmet Emre Gursoy, Stacey Truex, and Yanzhao Wu. A framework for evaluating gradient leakage attacks in federated learning, arXiv preprint arvi::0004. 10397, 2020.
[39] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jemnite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, October 2020. Association for Computational Linguistics. URL httpt:::://// //wws////wwwww ac Lweb. htggg/nnholoogy222220 emnlp- d 6.
[40] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In CVPR, 2021.
[41] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients, 2020.
[42] Junyi Zhu and Matthew B. Blaschko. R-GAP: recursive gradient attack privacy. In ICLR, 2021.
[43] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In NeurlPS, 2019.
[44] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We demonstrate the effectiveness of existing defenses against our attack in Sec. 5, further we outline that work does not deal with reconstructing labels which is left for future work, as described in Sec. 4.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We provide a discussion in App. F.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
13

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We have reported the type of resources used in the Sec. 5. We have reported the total amount of compute in App. E.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] We use standard datasets and cite the authors instead.
(c) Did you include any new assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We use standard datasets. We refer the reader the original authors of the dataset for this discussion.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xlafS_zu21J", "reviewer": "Reviewer_peFq", "summary": "The authors propose a model for recovering user data from gradient updates in a federated learning system for text classification. They achieve this by alternating continuous gradient based optimization with discrete heuristic based token-reshuffling. The authors show that the proposed model outperforms methods that use only gradient based updates to the tokens.", "questions": "How important is to alternate token-reshuffling updates with continuous optimization? What if the token-reshuffling appears only at the end of the continuous updates?", "limitations": "The limitations haven’t been discussed. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The primary contribution of the paper is the alternation between gradient-based updates and token reshuffling for reconstructing user text. I am unaware if such an approach has been attempted for adversarial attacks in text. Hence, this can be assumed to be novel.\n\nThe paper is well-written and easy to follow. The ablation studies confirm the importance of token-reshuffling for learning a good attack. The ablation studies also show the importance of L1+L2 loss (although this was proposed in a different paper).\n\nFrom a novelty perspective, this alternation between token reshuffling and continuous updates for attacking text classifier appears to be novel. However, I am not an expert in this field.\n\nSome parts of the paper are unclear. For instance, the authors mention that they use Adam for learning the embeddings for each input during the continuous optimization phase. However, they do not mention how they compute the gradient of the loss with respect to the input embeddings x*. The gradient computation requires second order derivatives which hasn’t been discussed in the paper at all.\n\nIt is unclear how token-reshuffling with continuous gradient-based updates influences the descent direction. Specifically, I am wondering if token-reshuffling is only performed at the end of all gradient-based updates, how will the performance be affected.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oZReCFaBy-w", "reviewer": "Reviewer_24M9", "summary": "This paper proposes a novel strategy to attack pretrained models for text models which aims to reconstruct the private user data used to finetune the model through federated learning. The algorithm takes the finetune gradient as well as the target label as the input, then search for the private user input sentence. The general idea is a iterative two-step process, where the first step is a continuous optimization to search for a embedding that leads to similar gradient, and the second step is a discrete optimization that uses GPT models to find a output sentence with the lowest perplexity.\n\n####\nThanks for the authors' response! I think it might be worth it to clarify the main novelty of the paper in the main method section. It would also be nice to clearly discuss the limitation at the end of the paper, as researchers not familiar with this line of research might benefit from some background information about the social impact/limitation of the method.", "questions": "1. Could you clarify the major difference between your method and prior work?\n2. How can this method be extended to the more realistic setting where one does not have labels for the data?", "limitations": "The authors did not address limitations and potential negative impact. There are several points worth mentioning:\n1. Works on adversarial attack could be exploited by hackers.\n2. The method is only limited to classification data.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n1. the method is very interesting and novel. It also addresses an interesting problem in the pretrain-finetune paradigm that is very popular now.\n2. the writing is mostly well structured and easy to understand. \n3. the final performance also seems good compared to previous methods. The ablation study and examples are pretty nice addition to the results.\n\nweakness:\n1. the paper didn't really clarify the main difference between the proposed method and the prior work, such as TAG. Therefore, it is hard to tell what are the exact novelty the paper adds and the effect of the novelty. My understanding is that the random transformation and the use of GPT models allow the output more natural. However, it is really hard to tell from the current structure of the paper.\n2. the method can only be applied to classification where label is known, which seems pretty limited. It might be more natural to have unlabeled data for federated learning setting, as one might not be able to annotate the user text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZmzChtq0VTq", "reviewer": "Reviewer_ZC5x", "summary": "This paper focuses on the attack that tries to recover text data from the gradients, this type of attack is a threat particularly for federated learning where the central server may recover the private client data through gradients. Based on previous work that optimises the input to minimise the distance between gradients, this paper further proposes to alternate continuous optimization and discrete optimization that is guided through a language model prior. The discrete optimization is claimed to help obtain text data that is more like fluent language. The resulting approach, LAMP, greatly outperforms previous approaches on three binary-classification benchmarks.\n", "questions": "I listed all my questions or suggestions in the weaknesses above, I would like to improve my rating if the authors could address them in the rebuttal, particularly the *major* points.", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\n1. The proposed approach is novel and technically sound. The motivation and contribution are clear – from the examples in Table 2 and quantitative improvements, it does seem like previous approaches fail to yield grammatical text while the proposed discrete optimization step seems to help it a lot.\n2. The empirical results are strong especially with batch size > 1. The ablation study is appreciated as well.\n3. The paper is well-written. \n\n\n\n### Weaknesses:\n\n1. *(minor)* The proposed approach adds several additional hyperparameters to tune, for  example, $\\alpha_{lm}, \\alpha_{reg}, n_c, n_d, n_{init}$. This surely adds complexity of the approach and may make the proposed attack difficult to be practically applied. While I appreciate the detailed hyperparameter paragraph in Line 268-274, I think it would be better to report the hyperparameter selection range as well, so that the readers could have an idea how much effort is imposed to tune these hyperparameters.\n\n2. *(major)* I am a bit worried that the model comparison is only conducted on a randomly selected set of 100 sentences. 100 sentences sounds too few for me, and I am not sure how robust the model rank is based on only 100 random examples. I feel this point should be justified properly, either through constructing a larger test set, or using different random seeds to generate different sets of 100 test examples and testing on each of them.\n \n3. *(minor)* While Line 239 mentions to illustrate the generality of the proposed approach with respect to model size, I don’t think this paper really made that goal by only using tiny and base sizes of BERT – at least BERT-large should be included to have a relatively complete coverage. I understand that the authors may be limited by resources to use larger models, which is fine. I just wanted to point out the current experiments are not sufficient to indicate generality with respect to model sizes (this is rather a minor point anyway).  \n\n4. *(major)* Line 257-259 mentions the baselines use 2500 iterations while the proposed approach uses 2000. Is *it* in Algorithm 1 2000 (and what are the values of *n_c* and *n_d*)? How did you choose these numbers? Do the baselines fully converge? I would like to see more justifications to show that the comparison is fair, because LAMP employs a nested for loop (while the baselines do not?) in optimization and I feel the number of total optimization steps (and the cost/time) in LAMP is actually larger than the baselines, right?\n\n5. *(minor)* Line 272 mentions that LAMP additionally adopts a two-step initialization procedure that seems important to me. I would like to see the ablation results on this two-step init in Table 3 to know how much of the improvement over baselines is from this initialization.\n\n\n```\nAfter author response:  \nThe author response addressed most of my concerns, and I would like to increase my score to 7 given that the authors will update the paper accordingly as promised.\n```\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept