Recovering Private Text in Federated Learning of Language Models
Authors: Samyak Gupta
Yangsibo Huang
Zexuan Zhong
Tianyu Gao
Kai Li
Danqi Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Federated learning
Privacy
Natural Language Processing

Abstract: Federated learning allows distributed users to collaboratively train a model while keeping each user’s data private. Recently, a growing body of work has demonstrated that an eavesdropping attacker can effectively recover image data from gradients transmitted during federated learning. However, little progress has been made in recovering text data. In this paper, we present a novel attack method FILM for federated learning of language models (LMs). For the first time, we show the feasibility of recovering text from large batch sizes of up to 128 sentences. Unlike image-recovery methods that are optimized to match gradients, we take a distinct approach that first identifies a set of words from gradients and then directly reconstructs sentences based on beam search and a prior-based reordering strategy. 
We conduct the FILM attack on several large-scale datasets and show that it can successfully reconstruct single sentences with high fidelity for large batch sizes and even multiple sentences if applied iteratively.
We evaluate three defense methods: gradient pruning, DPSGD, and a simple approach to freeze word embeddings that we propose.  We show that both gradient pruning and DPSGD lead to a significant drop in utility. However, if we fine-tune a public pre-trained LM on private text without updating word embeddings, it can effectively defend the attack with minimal data utility loss. Together, we hope that our results can encourage the community to rethink the privacy concerns of LM training and its standard practices in the future. Our code is publicly available at https://github.com/Princeton-SysML/FILM .

Recovering Private Text in Federated Learning of Language Models
Samyak Gupta" Princeton University
Yangsibo Huang"
Zexuan Zhong Princeton University
Princeton University
prinucton.cdu
yangsibo@princtton.ch
Tianyu Gao
Kai Li
Danqi Chen
Princeton University Haanyug@cs..rinneeton.c.
Princeton University
Princeton University
li@es. ffics.prinetooo.en
dangie@@cs.prineetonn
Abstract
Federated learning allows distributed users to collaboratively train a model while keeping each user's data private. Recently, a growing body of work has demon- strated that an cavesdropping attacker can effectively recover image data from gradients transmitted during federated learning. However, little progress has been made in recovering text data. In this paper, we present novel attack method FILM for federated learning of language models (LMs). For the first time, show the feasibility recovering text from large batch sizes of up to 128 sentences. Unlike image-recovery methods that are optimized a distinct approach that first identifies a set of words from gradients and then directly reconstructs sentences based on beam search and prior-based reordering strategy. We conduct the FILM attack on several large-scale datasets and show that it can successfully reconstruct single sentences with high fidelity for large batch sizes and even multiple sentences if applied iteratively. We evaluate three defense methods: gradient pruning. DPSGD, and simple approach to freeze word embeddings that propose. We show that both gradient pruning and DPSGD lead to significant drop in utility. However, if we fine-tune a public pre-trained LM private text without updating word embeddings, it can effectively defend the attack with minimal data utility loss. Together, we hope that our results can encourage the community to rethink the privacy concerns of LM training and its standard practices in the future.?
match gradients,
take
Introduction
Attacker
Reconstructed text:
Federated learning (McMahan al., 2017) is method to allow multiple participants to collaboratively train a global model without exchanging their private data. At each step of training, central server trans- mits model parameters to every partici- pating client. Each client then computes a model update (i.e., gradients) using its local data and sends it to the server. Fi- nally, the server aggregates all updates- typically by averaging them-and updates the model. Federated learning is actively being considered for privacy-sensitive ap- plications such as virtual mobile key and analysis of electronic health records in hospitals (Li al., 2020a).
Hy: phoze nurber ia
Mis-batch
poone nomber 721-4567, 166--0--2285.
Gred enrz
Moccl Paramatars
Private Kin -utch
Gradletts
Pod:: funng
Mocel Paramaters
Modd
Privtt Min botch
Gradiette
covEE..
Model Paramaters
Clients
Server
Figure 1: FILM allows an honest- nonest-but-cutt but-curious cavesdropper on the communication between the client and the server in federated learning to recover private text of the client.
"The first authors contributed equally.
"Our code is publicly available at https://gttuu..com// www./oonoc/oo-----.. Hemm/Princtto--ysMMt 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

However, recent studies (Zhu et al., 2019; Geiping et al., 2020; Pustozerova & Mayer, 2020; Lyu et al., 2020; Yin al., 2021; Zhu & Blaschko, 2021; Huang et al., 2021; Gupta et al., 2021) show that an honest-but but-cuuoutu eavesdropper in federated learning can recover private data of the client (Fig. 1). However, success in recovering text data remains limited to unrealistically small batch sizes (see Sec. for more details).
In this work, we study federated learning of neural language models (LMs) and present a novel attack called FILM (Federated Inversion Attack for Language Models), which can recover private text data from information transmitted during training. For the first time, we demonstrate that an attacker can successfully recover sentences from large training batches of up 128 sentences, making practical and alarming in real-world scenarios. We focus on language models for two reasons: (1) Transformer-basee based (Vaswani et al., 2017) language models have become the backbone of modern NLP systems (Devlin et al., 2019; Radford et al., 2018, 2019; Brown et al., 2020) and are quickly adapted to a number of privacy-sensieee
domains (Alsentzer et al., 2019; Huang et al., 2019; Li et al., 2020b; Liu & Miller, 2020; Kraljevic et al., 2021); (2) We identify that training LMs is at higher risk of information leakage by nature because the attacker can possibly leverage the memorization capability of LMs during federated learning.
Our FILM attack deviates from previous gradient-matching approaches for image data that directly optimize in high-dimensional, continuous space, which find to be ineffective for discrete text inputs and highly sensitive to initializations (Sec. 6). Instead, we first recover a set of words from the word embeddings' gradients (Melis al., 2019), and this step provides a set of words that appear in any sentences of the training batch without the frequency information'. We then develop a simple and effective strategy using beam search (Reddy et al., 1977; Russell & Norvig, 2010), which attempts to reconstruct one sentence from the set of words. This step is crucial as takes advantage of prior knowledge encoded in pre-trained language models and memorization of training data during federated training. We also further design token-reordering approach that leverages both language prior and gradient information to refine the recovered sentence. Finally, we show that we can recover multiple sentences from the same training batch by iteratively applying the same procedure.
We evaluate FILM on two language modeling datasets: WikiText- 103 (Merity et al., 2017) and Enron Email (Klimt & Yang, 2004), based on a GPT-2 model (Radford et al., 2019), using either pre-traine trained or mandomly-initializenn weights. We analyze the attack performance with different batch sizes, the number of training data points. and the number of training epochs. Our experiments demonstrate high-fidelity recovery of a single sentence feasible, and recovery of significant portions of sentences for training batches of up to 128 sentences. Furthermore, we find that recovery of multiple sentences from the same batch (1/3 for batch size of 16) is possible through repeated applications of FILM.
For defense, we first evaluate two previously proposed defense methods, gradient pruning (Zhu et al., 2019) and Differentially Private Stochastic Gradient Descent (DPSGD) (Abadi et al., 2016) against the proposed attack and find that both of them suffer substantial utility loss. We then propose simple method to freeze the word embeddings of the model during training to prevent the critical first stage of our attack and find that (1) if we train an LM from the scratch the private text, it will cause more utility loss; (2) however, if we start from a public LM (e.g., pre-trained GPT-2), it can effectively defend the FILM attack with minimal drop in utility. Together, our work presents strong attack on federated learning of LM training, and raises privacy concerns in current practices. We hope our new (and simple) proposal of freezing word embeddings from public LM encourages reconsideration of standard practices for training LMs on privacy-sensitive domains.
2 Related work
Data recovery from gradients. Gradients in federated learning have been shown to leak informa- tion of private data. Prior works reconstruct private training images from gradients by treating the reconstruction as an optimization problem (Zhu et al., 2019; Zhao et al., 2020; Geiping et al., 2020; Enthoven & Al-Ars, 2021; Yin et al., 2021; Zhu & Blaschko, 2021; Jin et al., 2021; Jeon et al., 2021). They iteratively manipulate reconstructed images yield similar gradients as observed gradients computed on the private images, using regularization terms based on image priors such as total varia- tion (Zhu al., 2019; Geiping et al., 2020) and batch normalization statisties (Yin et al., 2021).
1 We may still estimate the frequency of words hased on the magnitude of gradients (Wainakh et al., 2021; Fowl et al., 2022). However, later we show that our attack can actually compensate for the lack of frequency information (Sec. 4.3).
2

---

3.2 Federated Learning The first attempt to recover text from gradients is Zhu et al. (2019), which briefly presented leakage results in masked language modeling (Devlin et al., 2019). This attack matches target gradients with continuous representations and maps them back to words that are closest in the embedding matrix. Deng et al. (2021) improve the method by adding a regularization term that prioritizes gradient matching in layers closer to the input data. Both approaches only work with a batch size of (Sec. 6). Recently, Dimitrov et al. (2022) propose method minimize the loss of gradient matching and the probability of prior text computed by an auxiliary language model and show the recovery of a batch of 4 sentences with binary classification tasks with a BERT model. In addition, Boenisch et al. (202 D); Fowl et al. (2022) recover text data in federated learning under a stronger threat model: the server is malicious and can manipulate the training model's s weights to enable easier text reconstruction.
Our work builds on the insight from Melis al. (2019) that show the information leakage from the embedding layer by observing non-zero entries in the gradients of the word embedding matrix. Different from their attack which only infers person or location from shallow GRU model (Cho et al., 2014), we propose a much stronger attack that aims at recovering full sentences from larger language models. We include a high-level comparison of key differences between our approach and prior approaches in Table 6 in the appendix.
Memorization in language models. Our method is also inspired by the work related to memoriza- tion of training data in language models. Recent studies (Carlini al., 2019; Thakkar al., 2021; Song& & Raghunathan, 2020; Zanella-Béguelnn et al., 2020; Carlini et al., 2021; Bender et al., 2021) notice that large language models can memorize their training dataset and thus can be prompted to output specific sensitive information. While these previously demonstrated memorization attacks try to recover small subset of sensitive text from a large training corpora, our attack aims at recovering sentences from every mini-batch in federated learning. We stress that our attack is significantly more powerful than the previous works studying memorization.
Defenses against gradient inversion attacks for text data. Cryptographic approaches such as encrypting gradients (Bonawitz et al., 2016) or enerypting the data and model (Phong al., 2018) can guarantee secure training in federated learning setting against gradient inversion attacks. However, practical deployment of these approaches often slows down model training and requires special setup. A few efficient methods are proposed to defend gradient inversion attacks are for text data. Zhu et al. (2019) mention briefly adding differentially private noise to the gradients (Abadi et al., 2016), or setting gradients of small magnitudes to zero (i.c., gradient pruning). but these defenses typically hurt the accuracy of the trained models (Li et al., 2021; Yu et al., 2021). Huang et al. (2020b) propose InstaHide that encodes each input training image to the neural network using a random pixel-wise mask and the MixUp data augmentation (Zhang et al., 2018). The idea is later extended to language understanding tasks (Huang et al., 2020a): instead of 'hiding' the input word embeddings, they perform MixUp on [CLS) tokens of BERT models.
3 Preliminaries
In this section, we describe relevant background of language models, federated learning, as well as the threat model of our proposed attack method.
3.1 Language Modeling
Language modeling is core task in natural language processing and has been used a building block of ofatc-of-the-art pipelines (Radford et al., 2018; Brown et al., 2020). Given a sequence of n tokens {I 22, ... Jn J. the language modeling task is to estimate the probability distribution of PP(x):
log IP& (x)
log Po Zi.1.
(l)
Modern neural language models are parameterized by RNNs (Mikolov et al., 2010) or Transform- ers (Vaswani et al., 2017), which consist of millions or billions of parameters (denoted f). A language model first maps x to sequence of vectors via a word embedding matrix W E 1 where V is the vocabulary and d is the hidden dimension. After computing the hidden representation h; condi- tioned on , the model prediets the probability of the next token as: exp(h, Wz.)
Po (o #111) ESE exp(h, Wz)
Federated learning (FL) (McMahan et al., 2017) is a communication protocol for training a shared ma- chine learning model on decentralized data. An FL system usually involves N clients: C12 "2," CN
3

---

"Lot", "Ihe" "cat" "ihe "doge"" "1so", "House "il seep inght "eac",
doga
dogs lie.
Let the
TTe mouse chases
The nouse chades the
The cal: chazes the
! ② Tbe
Gradiant
Let dogs
Let doge Lie siccping.
Lic
Sat dege
(1) Bag-of-Words Extraction
(2) Beam Search for Reconstruction
(3) Token Reordering
Illustration of our pipeline for a batch size of 2. We first recover bag of words from the observed gradients, and then generate candidate sentences using beam search from the set of words, retaining top beams every step according a beam scoring function. Finally, re-order phrases and words in the best candidate.
Figure 2:
and central server s, where the clients wish to collaboratively train neural network fo with their private data D1, 22,- .--- Dv. under the coordination of the server. The training process optimizes 0 (model parameters) using a loss function L and runs for I iterations. This work focuses on fed- erated learning of language models, where clients can start from mandonly-initiaizzed neural net- work, or pre-trained public model (Hard et al., 2018). At each iteration t E [T), an individual client C, i [N] computes Vo (B;)) the gradients of the current model parameters o on randomly- sampled private data batch B, C D:, [B, = b, and shares the gradients with the server s. The server then aggregates gradient updates from all clients and updates the model:
0 - 0²
V Let (B.),
(3)
where 1/ is the learning rate. Finally, the updated parameters 0++1 are broadcast to individual clients."
3.3 Threat Model
Adversary's capabilities. We consider the communication between the central server and an arbitrary client G, i E [N] in the federated training of language model parameterized by 0, as described above. An adversary in this scenario has white-box access to 1) the gradients Vg La (B,) sent by the client c,, and 2) the model parameters of (Fig. 1), including the vocabulary / and the embedding matrix W. Note that the adversary can inspect these information at any training iterations E [T'.
adversary who is eavesdropping
Adversary's objective. The adversary aims recover at least sentence from the private training data batch B; based on the information they observe, as recovering a single sentence is already sufficient to break the privacy guarantee of federated learning. Since each training batch consists of randomly selected sentences from the private training dataset, if the adversary could recover at least one sentence from each training batch during the whole training procedure, they would be able to recover considerable fraction of the original training dataset. Additionally, the attacker may also apply the attack on a single batch multiple times to recover more sentences from the batch (Sec. 4.5). The strength of attack will be measured by the similarity between the recovered sentence and its corresponding private sentence the original batch.
4 Proposed Attack: FILM
4.1 Overview
In this section, we present novel attack method FILM, which construct sentences from any training batches (IB, < 128). Our method consists three steps:
(1) Inspired by Melis et al. (2019), we first recover a set of words that may appear in private batch
(B,) from gradients (Sec. 4.2). While their goal is to identify individual words from gradients, our approach is able to recover the order of words in sentence.
(2) We attempt searching for full sentences using beam search using the of words, based on the probability distribution Po (z), which leverages either the language prior from a pre-trained language model 0), the memorization capability of the LM during training (t >> 0) (Sec. 4.3).
(3) Finally, we design prior-based scoring function and demonstrate that we can further improve the quality of recovered sentences by performing simple and lightweight reordering step (Sec. 4.4).
'Note that Eq. 3 uses SGD for easier demonstration for update step in federated learning. People usually use Adam (Kingma & Ba, 2015) in real-worke training of neural language models.
'Recall that the attacker can inspect the gradients any step I E [T"] during the T eepchh training. An attacker performing attack larger may have much stronger results than for smaller t, as later show in Sec. 7.

---

We show that the above pipeline is capable of recovering single sentences from large training batches successfully. In Sec. 4.5, we discuss that we can iterate the above procedure recover multiple sentences from the same training batch, which makes our attack even stronger. In the following, we describe our attack method in detail. To better illustrate our attack's ' functionality, we provide simplified step-by-step example that recovers a batch of 2 simple sentences in Fig. 2.
4.2
Bag-of-Words Extraction
Input: Vo Lor (B.), gradients of the language model on the private hatch of sentences.
Output: bag of words B: {"Let", "The", "cat", "chases", "dogs", "lie", "mouse", "sleeping", "the", " "."). Our first step is to extract a bag of words from Vg Lge (B;), the gradients observed by an eavesdropper the (-th iteration of training. We follow the method outlined by Melis al. (2019) and recover set of words B = [W), W/2. ...,, J J by considering the non-zero rows of the word embedding gradients V Co (B,). Additionally, batch by analyzing the non-zero rows of position embeddings gradients Vgr (B.). Note that this step only determines the set of words but not their frequency. In practice, we find that augmenting our beam search with an n-gram penalty (Sec. 4.3) can compensate for this lack of information.
are also able to recover the maximum length of sentences in the
4.3 Beam Search for Sentence Reconstruction
Input: B, the bag of private words recovered by Sec. 4.2.
Output: a recovered sentence from the batch, "The mouse chases the cat.".
The second stage of our attack is to recover one sentence based on the extracted bag of words B and the learned probability distribution of text Po (x) at training step t. Since it is intractable to generate text directly using IPs (x), we use the beam search (Reddy et al., 1977; Russell & Norvig, 2010), which is a greedy auto-regressive algorithm to generate sequences in step-by-step manner with trained language model. We provide a detailed algorithm in Appendix A.
To initiate the beam search, we first find a set of words that can be chosen as starting tokens of sentences. A straightforward method is to select words that begin with capital letter from B. For example, we would select "Let" " and "My" as possible prompts for the illustrated batch. Then at cach beam search step, we keep the top & possible beam states, find the k most likely next words to each beam by using the language model (see procedure ToP(-) in Appendix A, Algorithm 1), and keeps the top k new beam states. The beam tree continues until the generated sentence reaches specified length (we use 40 words), and outputs the text that has the overall highest probability (see Fig. 2).
As the bag of words does not contain information on the frequency of each word in the sentence, and beam search is prone to generating repetitive content (Holtzman et al., 2020), we also include a penalty for repeated n-grams (Vijayakumar et al., 2016) in a sentence (see line 3 in procedure ToP(.)). Noticeably, we compare the performance of our attack with an oracle scenario where the attacker knows the frequency of each word in Fig. 5, and find that adding an n- n-gram penalty (n = 2) can achieve performance comparable without frequency information.
4.4 Prior-Guided Token Reordering
Input: a recovered sentence from Sec. 4.3, "The mouse chases the cat."" " Output: a reordered sentence, "The cat chases the mouse."
By construction, beam search is greedy method that generates sentences from left to right. Despite that it generally produces fluent sentences by maximizing Po (x), it does not leverage the gradients information which is the main signal in previous image attack methods. We also notice that beam search may recover the general structure of the sentence, while failing to recover the order of specific phrases or words. Thus, the final stage of our attack is to reorder tokens in the extracted sentence from the previous step, with the objective improve its quality measured by prior-based scoring function. The scoring function leverages both the perplexity and the gradient norm. Formally, given a sentence its prior score (x) is defined as:
So exp log (x) +B [x))((
Gradient Norm
Pepplexity
where log (x) is the likeliiel of the sentence (Eq. 1) according to the model parameters 0, and / is hyperparameter that controls the importance of gradient norm. Appendix B.8 provides empirical analysis that supports the design of this scoring function. We conduct two additional steps to reorder both phrases and tokens:
We assume that sentences in one batch vary in length. A standard practice is to pad shorter sentences.
5

---

Phrase-wise reordering. We first preprocess the sentence x by removing its redundant segments'. We then adjust the preprocessed sentence by reordering the phrases inside it. At each iteration, we generate multiple candidates by first cutting the sentence at positions using the strategy in Malkin al. (2021), and then permuting the phrases between the cuts form a new sentence. We then select the best candidate measured by So This stage terminates after 200 steps.
Token-wise reordering. We then reorder individual tokens. At each reordering step, we generate candidates using the following token-wise operations: 1) randomly swaps two tokens in the sentence, or randomly deletes a token from the sentence, or 3) randomly inserts a token from the bag of words (recovered by Sec. 4.2) into the sentence. We then select the best candidate measured by (-) and move to the next iteration. This stage terminates after 200 steps.
4.5 Recovering Multiple Sentences
The previous steps only consider the recovery of a single sentence from a training batch. In this step, we describe an extension of our attack which is able to support recovery of multiple sentences from the same batch. We first perform beam search described in Sec. 4.3. We store the results and repeat the beam search again, except this time applying an additional n-gram penalty based on the results of the previous search. Over many repetitions of this procedure, we are able to build set of candidate recoveries for the original batch. Finally, we use the reordering method described in Sec 4.4 to improve each candidate.
5 Defending Against the FILM Attack
We evaluate previously proposed defense methods, gradient pruning (Zhu et al., 2019) and DPSGD (Abadi et al., 2016) and find that both are less effective in defending our attack (Sec. 6.3).
To defend against the FILM attack, we propose a defense method which simply freezes the word embeddings of the model during training. Our key insight is to focus on preventing the first step of the attack, i.c., the recovery of the bag-of-words from word embedding gradients. If the attack fails at this step, the later stages of the attack are no better than beam search over the entire vocabulary as with the prior attack (Carlini et al., 2021). I). Similarly, previously proposed gradient inversion attacks (Zhu et al., 2019; Fowl et al., 2022) for smaller batch sizes also depend on this step. By freezing the word embeddings during training, it is able to completely prevent the recovery of bag-of-wordh In other words, once we prevent the transmission of gradients of word embeddings during training, is no longer possible to recover the bag-of-wordh words..
We consider the settings for both 1) training an LM from the scratch; 2) continuing training from public, pre-trained LM (e.g., GPT-2) on the private text, when freezing the word embeddings. As we later show in Sec. 6.3, the first setting causes more utility loss (because the word embeddings are randomly initialized) while the latter setting provides the best privacy-utility trade-off. Since updating word embeddings is a common practice in training LMs, we suggest researchers and practitioners consider training from public LM and freezing word embeddings in privacy-sensitive applications.
Experiments
6.1 Setup
Model and datasets. We evaluate the proposed attack with the GPT-2 base (117M parameters) model (Radford al., 2019) on two language modeling datasets, including WikiText- 103 (Merity et al., 2017) and the Enron Email dataset (Klimt & Yang. 2004). Both datasets are publicly available for research uses. We choose WikiText- 103 because it is commonly used in language modeling research, and Enron Email because it consists of private email messages that contain abundant private information such individuals' names, addresses, and even passwords. Note that the GPT-2 model not trained on any Wikipedia data, and very unlikely on the Enron Email data. We evaluate our attack on subset of Wiki'Text 103 and Enron Email. After preprocessing (more details provided in Appendix B), we have 203,456 sentences from the WikiText- 103 dataset and 31,797 sentences from the Enron Email dataset.
Training and attack settings. Following previous studies (Zhu et al., 2019; Geiping et al., 2020; Yin et al., 2021; Huang et al., 2021) for gradient inversion attacks, our evaluation considers a federated
We notice that the sentences we recover from the previous step may not naturally finish after the heam search generates punctuation. To address this problem, we preprocess each recovered sentence x by removing segments after its first punctuation, as long as the resulting sentence x' satisfies So(x') < So(x).

---

Attack & Batch Size
Original Sentence
Best Recovered Sentence
WikdText-103
Zou all (2019).
As Elizabesh Hojem explains, 'The most English theleschevahrued theAsWords reporting the Youngerse- of Englishmen, then, both king and slave, lagebalance, mathemat mathemat mathemat reper many ways fndisinguishable from Stephen Black. rangpmwikitadie Rowen persperionlle subur; maximal
6
Deng al (2021). Both teams recorded
penalties, but Michi- Both recorded teame seven bus Michigan gan recorded more penalty yards.
recorded
penalties more penality outstanding
FILM, & =
The shorist-Sttill stingray forages for food both during the day and night.
The shorttt-Btall stingray Forages for Food both during night:
FILM, & 16
A tropical wave organized into a distinct area of disturbed weather south of the Mexican port of Manzanillo, Colima, on August 22 and gradu: ally moved to the northwest.
Early on September 22, an iren of diatur bed weather bre ganized into tropical wave, which moved the north west of the area, and thon moved laso the north south tto the northeast.
FILM, b 128
A remastered version of the game will be released on PlayStation 4. Xbox One and PC alongside Call of Duty: Infinite Warfare on November 4, 2016.
At the time of writing, the game has been veleased pu PlayStation Xbox One. PlayStation 8. and PC. with the PC version being released in North America on November 18th, 2014.
Enron Email
(2019), Don and ragers have decided management dj "... Free, expShoperriptyn Jh hheragreessvv Highlands Deng et al. (2021)) to ena.
purposss to leave consolidated this point. andinos Andren Rebell im impacts
We should not transfer any funds from tenaska We should transfer any fonds nos from tenaskes W so happens
FILM, &
Volume mggnt is trying to clear thrse issure..
Volume mggnt trying to cleser up these boomers
FILM, 6 16
Yesterday, enron ousted chief financial officer an- drew fastow amid a securities and exchhange com- mission inquiry into partnershipe he that cost
Yesterday, enron ousted its ihile financial officer, an drew Fastow, amid a securities and exchannge commiss sion inquiry into partnerships he that cost the com- pany $35 million stock other financial asonts.
largest energy trader $35 million.
FILM, b 128
Yeaterday, enron ouated chief financial officer an- drew fastow amid securities and exchange mission inquiry into partnerships le ran that cost the largest energy trader $35 million.
Yestarday, enron oustee chief financial offieer andrew fastow
securities and exchange commission guiry into partnerships ram that he said cost the company more than S1 billion in stock and other assets.
Table Performance of Zhu et al. (2019)'s and Deng et al. (2021)'s 's with batch size I (more in Appendix B.3) and FILM (ours) with different batch sizes. We show the best recovered sentence among 20 tested batches for each batch size (see Fig. 3 for average results). Text in green represents successfully recovered phrases and words.
learning setting with single server and a single client. 8 Unless otherwise noted, we train the model on these sentences for 90, 000 iterations using an initial learning rate of 1 x 10 5, with linearly decayed learning rate scheduler. All models were trained using early stopping, i.e., models were trained until the loss of the model on the evaluation set increased. We note that the running time of our algorithm is quite fast, and we can recover single sentence in under minute using an Nvidia 2080TI GPU.
Evaluation metrics. We use the following metrics to evaluate the attack performance: (a) ROUGE (Lin, 2004) is a set of metrics for evaluating summarization of texts as well as machine translation. We use ROUGE-1/2/L F-Scores to evaluate the similarity between the recovered and the original sentences, following (Deng al., 2021; Dimitrov et al., 2022). More specifically, ROUGE- 1/2 refer to the overlap of unigrams and bigrams between the recovered and the original text respec- tively. ROUGE-L measures the longest matching subsequence. For ablation studies we only show ROUGE-L as it is more representative of significant leakage than ROUGE-1 or ROUGE-2. (b) We also propose to use named entity recovery ratio (NERR) as the percentage of original named enti- ties that can be perfectly recovered. Since named entities usually contain sensitive information (e.g.. names, addresses, dates, or events), NERR measures how well the attacker recovers such information, ranging from a complete mismatch (NERR- 0) to a perfect recovery (NERR 1).
6.2 Performance of Our Attack
Scaling to large training batches. We first compare the performance with respect to different batch sizes. In this setting, we start from pre-trained GPT-2 model and fine-tune it on both datasets respectively (WikiText-103 and Enron Email). Fig. 3 shows the performance of our attack using different batch sizes with the WikiTeex- 103 dataset and the Enron Email dataset. We observe that for a batch size of 1, all four scores (ROUGE-1, ROUGE-2, ROUGE-L, NERR) are close to 1, indicating perrect recovery. We observe that the batch size increases, the quality of the recovery by ROUGE-1, 1, ROUGE-2, and NERR start to decrease. We attribute this drop in performance to the bigger set of words in the same training batch, as well as the growing search space of possible word orderings. We also observe that the Enron Email dataset is more susceptible to attack.
Table shows that the approach of Zhu et al. (2019) fails to recover the original sentence for batch size b = 1. We find that the gradient matching method in Zhu al. (2019) is sensitive to initialization and it can perform better recovery when using a good initialization (see Appendix B.3). Deng et al. (2021)
*This setting is often accepted as an adequate stand-in for federated learning as synchronous federated learning with N clients, each with b samples per batch, is (assuming I.I.D. data) functionally equivalent to training a model with batch size of N x b.
7

---

FILM (Oun)
TAG (proviuus bea))
DLG
L04
[a) WikTees-158
10-
ROURE
ROURE--
HO4G8-
MEM
0.5
32
Gl
64 Butch Size
Butchicee
Batchsioe
Busch Size
L3
(b) Email
104
ROURE:
9OURE-2
NERR
Banch
Size
Banch
Size
Booch Size
Booch Size
Figure 3: Recovery performance for various batch sizes WikiText- 103 (a) and Enron Email (b). Solid lines indicate the average F-Scores of recoveries out of 20 tested mini-batches. The attack is overall weaker for larger batch sizes (see Table 10 for quantitative results). Moreover, the attack is stronger for the Enron Email dataset.
improves over (Zhu al., 2019) by prioritizing the matching of gradients of layers that are closer to the input. However, Fig. 3 shows the NERR and ROUGE scores sharply degrade for larger batch sizes. On the contrary, datasets even when the batch size is 128. This demonstrates the superior effectiveness of leveraging the knowledge encoded in pre-trained language models and its memorization ability to recover sentences, compared to directly performing optimization in a high-dimensional continuous space.
approach is able to recover large portion of the original sentences
both
Attack performance at different training iterations. We evaluate the attack performance of our approach on the models trained at different numbers of training iterations f. using the WikiText- 103 dataset. Specifically, we vary the number of training iterations & E {0, 10000,.... ..... 90000} and fix the batch size for training b = 16 and dataset size = 200, 000. The batch size for attack is 1.
Previous gradient matching attacks on image data suggests that well-trained models are harder to attack (Geiping al., 2020) due to the shrinkage of gradient norm as training progresses. However, as shown in Fig. 4 (see qualitative examples in Table 11), we find that well-trained models are more vulnerable to our approach because the model is able to memorize data during training.
Pre-trained vs randomly-initialzzed models. We additionally compare the attack performance for training that starts from a pre-trained model (GPT-2 in case) and randomly- initialiiii model (we use the same GPT-2 architecture but re-initialize all the weights randomly). We find that starting from pre- rreitriind model is more susceptible to attack than training from scratch as shown in Fig. 4. However, the gap in performance gradually decreases as the number of iterations increases.
As shown in Fig. 4, pre-trained model which has not been fine-tuned on client data can recover sentences with an av- erage ROUGE-L score of 0.41, indicating that prior knowl- edge encoded in pre-trained models may help recover sen- tences. When the model is continued to train Wikitext- 103, the average score increases to 0.53, an almost 30% in- crease. Our results suggest that leveraging the memoriza- tion in the model yields a more powerful attack than only us- ing the language prior encoded in the pre-trained model. We note that we would expect stronger attack performance (es- pecially for smaller training epochs) when attacking larger- capacity models, as stronger models can memorize training instances easily. We report more results for t = 0 with dif- ferent batch sizes in Table 12 in Appendix B.5.
0.8
Pre tre-trdined
0.7
Randomly
ROUGE- scorre
0.6
0.3
0.2
2000C
40000 50000
50000
*o Training Iterations
Figure 4: Attack performance over train- ing iterations / for pre-trained or randomly- initialized model on WikiTexx- 103 (batch size for the attack is 1). Dashed lines indi- cate when an epoch was completed. See Ap- pendix B.5 for qualitative results.
Recovering multiple sentences by iterative attack. We further investigate a variant of our attack in which we apply the beam search for multiple repetitions. Each repetition applies an n-gram penalty based on the results of previous generations. We consider a sentence to have strong leakage if the longest recovered subsequence is > 25% of the original sentence (i.e., ROUGE-L score > 0.25).
Mtr..
Original Sentence
Recovered Sentence
In 1988 is made feature film Katinks, rectted by Max von Sydow, starring Tammi Oat as Katinka.
After the end the year, Fure Blm, and was directed by Max von Sydong and published by Warner Bros.
was
2nd
killed a mysterious plane crash on March 1959, while en route to Bangui.
Fox, an astempt pus an in plane cradit whill en route to Guadalcnnal, Puerto Rioo, on April 6, 2015, while on a
fire, was killed
The next year, Edwards published a significant trea- tise entitled Biblographic Catalogue of the Described Transformation of North American Lepidoptera.
also pubdttabed treatiis on it, published in 1959, eacitled Biblographic Catalogue of I.
Table 2: Qualitative results for multi-sentence recovery for a batchsize of 16. The multi-sentence attack is able recover significant portions of multiple sentences in single batch.
8

---

(a) Gradient pruning (Zhu et al., 2019)
(b) DPSGD (Abadi ct al., 2016)
Prune ratio
Perplexity
Precision
Recall
c of DPSGD
Perplexity
Precision
Recall
0
11,46
1.00
1.00
16.31
0.00
0,00
0.9
11.57
1.00
1.00
5
14.32
0.29
0.01
0.99
12.77
1.00
1.00
10
12.86
0,88
0.17
0.999
15.34
1.00
0.98
1.
11.98
0.97
0.49
0.9999
19.21
1.00
0.90
inf.
11.46
1.00
1.00
Table 4: Precision and recall for the reconstruction of tokens under gradient pruning (a) and DPSGD (b). FILM can still recover considerable amount of tokens (i.e. >90%) even with a gradient prune ratio of 0.9999, For
DPSGD, FILM fails to retrieve the majority of tokens (i.c., recall 0.5) when is smaller than 15.
Table 2 shows a qualitative example of this iterative process. Table 3 shows that for a batch size of 16, an attacker who performs 100 repetitions of beam search is able to recover roughly 34% of the original batch (roughly 5 sentences) with ROUGE-L score over 0.25
Iter.
Recall
Precision
1
0.03 + 0.03
0.50 + 0.51
5
0.13 1 0.08
0.36 1: 0.22
10
0.19 + 0.11
0.28 + 0.16
20
0.25 + 0.12
0.21 + 0.11
(recall). However, since 100 repetitions results in 100 different generated sentences, an attacker would still need a method that picks good recoveries out of them'. We estimate the difficulty of choosing good sentences by using the ratio of good sentences to total generated sentences (precision).
50
0.30 + 0.11
0.12 0.04
100
0.34 = 0.09
0.07 - 0.02
Table 3: Performance of recovering multiple
sentences by applying our attack iteratively on
WikiText- 103 (b 16). Recall: % of original batch with ROUGE-L>0.25), Precision: % of re- covered sentences with ROUGE-L>0.25.
6.3 Evaluation of Defenses
We evaluate three defense methods: gradient pruning (Zhu et al., 2019) and Differentially Private Stochastic Gradient Descent (DPSGD) (Abadi al., 2016), and our method of freezing embeddings. The evaluations use the WikiText-103 dataset and the GPT-2 model, with batch size = 16. Given S*, the set of tokens recovered from the attack and S, the original set of tokens in the private batch. Two metries for performance of the attack are:
1. Precision: |S* n S1//|") which is the fraction of original tokens in the recovered set.
2. Recall: |S n S1/SSS), which is the fraction of recovered original tokens in all original tokens.
We measure the impacts of defenses on model utility by computing the average perplexity of the model across the test set for each dataset. For evaluating the defense of freezing word embeddings during training, precision and recall are both O.
Token reconstruction under gradient pruning (Zhu et al., 2019). Gradient pruning zeros out the fraction of gradient entries with low magnitude. However, the embedding gradients of existing tokens in the batch are still non-zero unless the prune ratio is extremely high. Thus, the attack strategy remains the same as the vanilla attack: we retrieve the tokens whose embedding gradients are non-zero. As shown in Table 4.a, the attack always returns existing tokens in the private batch (i.e. precision is always 1). The recall decreases the prune ratio increases, because some entries of word embedding gradients get completely zeroed out and thus the corresponding tokens cannot be retrieved by the attack. However, the attacker can still recover a considerable amount of tokens (i.e. >90% even with a prune ratio as high as 0.9999.
Token reconstruction under DPSGD (Abadi et al., 2016). The strategy to launch the attack under DPSGD is trickier, as the gradients become noisy and the previous heuristics of returning non- zero gradient entries no longer hold. We come up with a new attack strategy which involves using a threshold 7 / log d discriminate noisy gradients with pure noise, where d is the embedding dimension (i.e. 768) and a is the noise scale of DPSGD. For each token, we check the maximum magnitude of its embedding gradients: if the value is larger than then the token may be included in the original batch with high probability. As shown in Table 4.b, the attack performance drops when the epsilon of DPSGD decreases (at the cost of perplexity), because the relative scale between the noise magnitudes and the original gradient magnitude increases.
Token reconstruction under freezing embeddings. Freezing embeddings stops the attacker's s access to the bag-of-wordd, words, and therefore results in precision and recall both being However, is nots considered a standard practice in training language models, as it may lead to worse model utility.
We compare the perplexity of GPT-2 models
(1) with frozen embeddings or unfrozen em- beddings and (2) from scratch (i.e., from ran- domly initialized model) or from pre- treitrinned (i.e., initialized by GPT-2 parameters) in Ta-
From Scratch
From Pretrained
Unfrozen Frozen
Unfrozen Frozen
Wikitext- 103
27.31
118.69
11.40
11.48
Enron Email
15.16
69.17
7.09
7.30
ble 5. We find that freezing word embeddings in training from scratch results in significant loss of model utility, but is negligible in the set- ting of using pre- trained model.
Table 5: Perplexity when embeddings are frozen or un- frozen. We ohserve significant drop in perplexity when embeddings are frozen in training from scratch.

---

7 Analysis
We finally discuss the impact of different parameters of our FILM attack in Fig. and Fig. 6. We also study the effect of different training parameters and present the results in Appendix B.7.
Beam size. As discussed in Sec. 4.3, beam size (denoted by k) is a hyper- search which controls the number of beams active at any time. As shown in Fig. 5 (left), the attack performance grows as the beam size is increased. This result aligns with our intuition using a larger beam size corresponds with consideration of more sentences during search. Considering that the computational resources required for the attack also increases with a larger beam size, we find that a beam size of around 32 achieves the best trade-off in terms of attack performance and efficiency.
of beam
N-gram penalty. The n-gram penalty controls for what size of n-gram the penalty is applicd. In Fig. 5 (right), we observe that having no penalty performs significantly worse than applying a penalty on repeated 2-grams. We note that performance for n = 2 (i.c., penalizing repeated 2-grams) is the closest to the "oracle" case, where the frequency of each word is known from the bag of words.
Gradient norm's coefficient 3 in prior-based scoring. As discussed in Sec. 4.4, the prior- based token recording for a sentence uses scoring function S(x) Perplexity(x) + B Gradient Norm(x), where 3 is a hyper-parameter that controls the importance of the gradient norm term. Our results (see Fig. 6 (left) show that the fine-tuning stage achieves tradc-off between fully perplexity- -uiided (i.e., = 0) and almost fully gradient gorm-guided (i.e., , = 50), with 8 being the optimal. The trend is consistent across different batch sizes.
Cracle (Frequency Known)
Ours
0:70
Afbet Seam Search
ROUGE Lscree
ROUGE score
ROUGE score
After Sentinnee Reordering
4 ROVGE-L krore
82
R
0.55
0.30
0.50
0.43
50
Beam
100
Sizo
200
250
N gram Ponalty
30
1936
Batch Size
G4
136
Figure 5: Effect of beam size and n-gram penalty. Re- covery quality improves with larger beam sizes (left).
Figure 6: Effect of gradient norm coefficient / in the sentence reordering stage (Sec. 4.4) (left), and the im- provement of ROUGE-I. after fine-tuming (right).
An ri-gam penalty (n = 2) performs the best (right).
We also notice that the improvement in the token reordering stage is most significant for small batch sizes (see Fig. 6 (right)). We hypothesize this is due to locality: the beam search result for smaller batch sizes is usually closer to its corresponding ground truth sentence than for larger batch sizes, which makes fine-grained reordering more powerful.
8 Discussion and Future Work
This paper presents the first attack that can recover multiple sentences from gradients in the federated training of the language modeling task. Our method FILM works well with modern neural language models based on Transformers and large batch sizes (up to 128), raising serious privacy-leakage concerns for federated learning in real-world scenarios. The key insight of our study is that an adversary can gain substantial power to reconstruct sentences by taking advantage of the knowledge memorized by trained language models. We also propose new defense which freezes the word embeddings during the federated training of language models. Our evaluation shows that it can effectively defend the FILM attack for the setting of starting from a pretrained languages model, with little utility loss.
The results presented in this paper have several limitations. First, we have not evaluated whether our attack can generalize to other natural language processing tasks, such text classification. Second, we have not tried our attack for batch sizes beyond 128 and our ablation study does not report results for batch sizes beyond 16. We additionally leave open the question of selecting good sentences in the melti-sentenee recovery scenario (See. 6.2). Furthermore, the proposed defense of freezing embeddings during the pre- pre-training of the language model can be detrimental to the model utility. Therefore, how to effectively defend the FILM attack for this setting remains open.
Limitations.
Ethical considerations. While our intention is to raise the awareness of the privacy risks in the language modeling task in federated learning, from the attacker's prospective, FILM is also likely a step towards even stronger attacks. Therefore, we hope that this work can also motivate necessary redesign of defenses to provide meaningful privacy guarantees to clients for training language models in federated learning.
10

---

References
Abadi, M., Chu, A., Goodfellow, I,, McMahan, H. B., Mironov, L., Talwar, K., and Zhang, L. Deep learning with differential privacy. In ACM SIGSAC Conference on Computer and Communications Security (CCS), 2016.
Alsentzer, E., Murphy, R., Boag, W. Weng, W.-H., Jin, D., Naumann, T., and McDermott, M. Publicly available clinical BERT embeddings. arXiv preprint arXiv: (e0i:::9900033... 2019.
Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al. A closer look at memorization deep networks. In International Conference on Machine Learning (ICML), 2017.
Bender, E. M., Gebru, T., McMillan-Major A., and Shmitchell, S. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, Virtual Event Canada, 2021. ACM.
Boenisch, F., Dziedzic, A., Schuster, R., Shamsabadi, A. S,, Shumailov, I., and Papernot, N. When the curious abandon honesty: Federated learning is not private. arXiv preprint arXiv:2112.02913 2021.
Bonawitz, K. A., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H. B., Patel, S., Ramage, D., Segal, A., and Seth, K. Practical secure aggregation for federated learning on user-held data. In NIPS Workshop on Private Multi-Parry Machine Learning, 2016.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), 2019.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts. A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
Cho, K., van Merriënboer, B., Bahdanau, D., and Bengio, Y. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111, 2014.
Deng, J., Wang, Y., Li, Wang, C., Shang, C., Liu, H., Rajasekaran, S., and Ding, C. Tag: Gradient attack on waasformer-baseee language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3600-3610. 2021.
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
Dimitrov, D. I., Balunovié, M., Jovanović, N., and Vechev, M. Lamp: Extracting text from gradients with language model priors. arXiv preprint arXiv:2202.08827 2022.
Enthoven, D. and Al-Ars, Z. Fidel: Reconstructing private training samples from weight updates in federated learning, arXiv preprint arXiv:2101.00/53 2021.
Fowl, L., Geiping, J., Reich, S., Wen, Y., Czaja, W., Goldblum, M., and Goldstein, T. Decepticons: Corrupted transformers breach privacy in federated learning for language models. arXiv preprint arXiv:2201.12675. 2022.
Geiping, J., Bauermeister, H., Dröge, H., and Moeller, M. Inverting gradients-how easy is it break privacy in federated learning? Advances in Neural Information Processing Systems (NeurIPS), 2020.
Gupta. U., Stripelis, D., Lam, P. K., Thompson. P., Ambite, J. L.. and Ver Steeg, G. Membership inference attacks on deep regression models for neuroimaging. In Medical Imaging with Deep Learning, pp. 228-250 PMLR, 2021.

---

Hard, A.. Rao. K.. Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kid- don, C., and Ramage, D. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03600 2018.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations (ICLR), 2020.
Honnibal, M., Montani, I., Van Landeghem, S., and Boyd, A. spaCy: Industrial-streenth Natural Language Processing in Python. 2020. doi: 10.528 1252211/eendo.... 1212303.
Huang, K., Altosaar, J., and Ranganath, R. ClinicaBERT:: Modeling clinical notes and predicting hospital readmission. arXiv preprint
2019.
Huang, Y., Song, Z., Chen, D., Li, K., and Arora, S. TextHide: Tackling data privacy in language understanding tasks. Findings of EMNLP, 2020a.
Huang, Y., Song. Z., Li, K., and Arora, S. InstaHide: Instance-hiding schemes for private distributed learning. In International Conference Machine Learning (ICML), 2020b.
Huang, Y., Gupta, S., Song, Z., Li, K., and Arora, S. Evaluating gradient inversion attacks and defenses in federated learning. Advances Neurul Information Processing Systems (NeurIPS), 34, 2021.
Jeon, J., Lee, K., Oh, S., Ok, J., al. Gradient inversion with generative image prior. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021.
Jin, X., Chen, P.-Y., Hsu, C.-Y., Yu, C.-M., and Chen, T. Catastrophic data leakage in vertical federated learning. Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Klimt, B. and Yang, Y. The enron corpus: A new dataset for email classification research. In European Conference on Machine Learning. Springer, 2004.
Kraljevic, 7.,, Shek, A., Bean, D., Bendayan, R., Teo, J., and Dobson, R. MedGPT: Medical concept prediction from clinical narratives. arXiv preprint arXiv:2 2021.
Li, T., Sahu, A. K., Talwalkar, A., and Smith, V. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-66 500 2020a.
Li, X., Tramer, F., Liang, P., and Hashimoto, T. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679, 2021.
Li, Y., Rao, S., Solares, J. R. A., Hassaine, A., Ramakrishnan, R., Canoy, D., Zhu, Y., Rahimi, K., and (1):1-12, 2020b.
G. BEHRT: transformer for electronic health records. Scientific reports, 10
Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, July 2004.
Liu, D. and Miller, T. Federated pretraining and fine tuning of BERT using clinical notes from multiple silos. arXiv preprint arXiv:2002.0856,, 2020.
Lyu, L., Yu, H., Ma, X., Sun, L., Zhao, J., Yang, Q., and Yu. P. S. Privacy and robustness in federated learning: Attacks and defenses. arXiv preprint arXiv:2012.06337 2020.
Malkin, N., Lanka, S., Goel, P., and Jojic, N. Studying word order through iterative shuffling. In
Empirical Methods in Natural Language Processing (EMNLP), pp. 10351-10366, 2021.
McMahan, H. B., Moore, E., Ramage, D., Hampson, S., et al. Communication-effiiennn learning of deep networks from decentralized data. In Artificial Intelligence and Statistics (AISTATS), pp. 1273 1282, 2017.
Melis, L., Song, C., De Cristofaro, E., and Shmatikov, V. Exploiting unintended feature leakage in collaborative learning. In 2019 JEEE Symposium on Security and Privacy (SP), pp. 691-706. IEEE, 2019.
12

---

Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR), 2017.
Mikolov, T., Karafiát, M., Burget, L., Cemnocky, J., and Khudanpur, S. Recurrent neural network based language model. In Annual Conference of the International Speech Communication Association (INTERSPEECH)) volume 2, pp. 1045-1048. Makuhari, 2010.
Phong, L. T., Aono, Y., Hayashi, T., Wang. L.,, and Moriai, S. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions Information Forensics and Security, 2018.
Pustozerova, A. and Mayer, R. Information leaks in federated learning. In Proceedings of the Network and Distributed System Security Symposium, volume 10, 2020.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. Improving language understanding by generative pre-training Technical report, OpenAl, 2018.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Reddy, D. R. al. Speech understanding systems: A summary of results of the five-year research effort. Department of Computer Science. Camegie-Mell University, Pittsburgh, PA, 17:138, 1977.
Russell, and Norvig, P. Artificial Intelligence: A Modern Approach. Prentice Hall, 3 edition, 2010.
Song, C. and Raghunathan, A. Information leakage in embedding models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), 2020.
Thakkar, O., Ramaswamy, S., Mathews, R., and Beaufays, F. Understanding unintended memorization in federated learning. In Proceedings of the Third Workshop on Privacy in Natural Language Processing, pp. 1-10, 2021.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A. N., Kaiser, E.., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Vijayakumar, A. K., Cogswell, M., Selvaraju, R. R., Sun, Q., Lee, S., Crandall, D., and Batra, D. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint 1rX.0:1610.02424, 2016.
Wainakh, A., Ventola, F., Müßig, T., Keim, J., Cordero, C. G., Zimmer, E.,, Grube, T., Kersting, K., and Mühlhäuser, M. User label leakage from gradients in federated learning. arXiv preprint arXiv:2/05.09366 2021.
Yin, H., Mallya, A., Vahdat, A., Alvarez, J. M.. Kautz, J., and Molchanov, P. See through gradients: Image batch recovery via gradinversion. In Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2021.
Yu, D.. Naik, S., Backurs, A., Gopi. S., Inan, H. A., Kamath, G., Kulkarni, J., Lee, Y. T., Manoel, A., Wutschitz, L., et al. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.065000 2021.
Ganella-Béguelin S., Wutschitz, I.., Tople, S., Rühle, V., Paverd, A., Ohrimenko, O., Köpf, B., and Brockschmidt, M. Analyzing information leakage of updates to natural language models. In ACM SIGSAC Conference on Computer and Communications Security (CCS), 2020.
Zhang. H., Cisse, M., Dauphin, N., and Lopez-Paz. D. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018
Zhao, B., Mopuri, K. R., and Bilen, H. iDLG: Improved deep leakage from gradients. arXiv preprint arXiv:2001.026111 026610. 2020.
Zhu, J. and Blaschko, M. B. R-gap: Recursive gradient attack on privacy. International Conference on Learning Representations (ICLR), 2021.
Zhu, L., Liu, Z., and Han, S. Deep leakage from gradicnts. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes].
(b) Did you describe the limitations of your work? [Yes], see Sec.
(c) Did you discuss any potential negative societal impacts of your work? [Yes], see Sec.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes].
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes], please check the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]. see Sec. 6.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes], see Fig 3.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes], see Sec. 6.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes].
(b) Did you mention the license of the assets? [Yes], see Sec. 6.1 (unfortunately, there is no license information for the Enron Email dataset).
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] we include our code in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes]. see Sec. 6.1.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes], we noted in Sec. 6. 1. that 'the Enron Email dataset because it consists of private email messages that contain abundant private information such individuals' names, addresses, and even passwords".
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A].
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A].
14


Reviews:
[{"review_id": "ylk5hhdQZnO", "reviewer": "Reviewer_vs2F", "summary": "This paper proposes a new attack named FILM for extracting text from federated learning models.\nThe key idea is to use gradients with respect to embedding matrix in order to identify words occurring in the batch,\nand then use beam search and token reordering to reconstruct a sentence from the batch.\nThe method is experimentally evaluated on several datasets which show benefits compared to the baseline.\n", "questions": "1) The authors say they focus on extracting 1 sentence from the batch, but could they evaluate how well can they recover full batches, as that is the setting in other papers?\n\n2) It would be good to see some analysis of which sentence in the batch is actually recovered. Is it always the shortest sentence?\n\n3) As mentioned earlier, this attack is easily bypassed by a defense which does not train word embeddings which prevents reconstructing bag of words. Could you experiment with this defense and some other defenses (e.g. differential privacy based)?\n", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I believe that this paper considers quite relevant problem as many practical applications of federated learning are in NLP space, while\nexisting attacks have mostly focused on reconstructing images.\n\nHowever, I have found some issues with the paper which I hope authors can clarify. Due to these issues I currently think the paper is below the acceptance bar.\n\nFirst, there is a serious limitation to this approach which I think should be emphasized.\nThis is the fact that the extraction approach described in Section 4.2 assumes that token embeddings are also being trained.\nWhen embeddings are not trained, this approach doesn't work at all as it cannot compute bag of words in the batch.\nFurthermore, this limitation does not exist in other papers that consider this setting, e.g. DLG (Zhu et al.), TAG (Deng et al.), LAMP (Dimitrov et al.) as they can utilize other elements of the gradient.\nTherefore, since this attack is bypassed by just not training embeddings (which is sometimes even done by default), it is unclear how significant it is.\n\nFirst, comparison with TAG (Deng et al.) and DLG (Zhu et al.) does not seem to match results in respective papers. Results in Table 1 show that DLG and TAG essentially can't reconstruct anything even for batch size 1.\nMoreover, authors claim that TAG doesn't improve over DLG (footnote in Table 1). This all clashes with results in Deng et al. and Dimitrov et al. where DLG has much better reconstruction than shown in this submission,\nand using TAG further improves the reconstructions.\nIn this paper, authors evaluate on completely new datasets than the prior papers (Zhu et al., Deng et al., Dimitrov et al.) so it is difficult to compare to those works, and might be a reason for this discrepancy.\nI would suggest authors to experiment with datasets used in other papers (e.g. CoLA, SST-2, Rotten Tomatoes) so that we can compare the actual performance of different methods.\n\nIn Section 4.4 I find it a bit strange to optimize for the sentence which minimizes combination of prior and gradient norm.\nWhy should we not combine prior with L2 distance between the current gradient and the ground truth gradient, as in Zhu et al.?\nCould authors experimentally demonstrate that their loss function works better?\n\nFurthermore, Section 4.4 seems basically the same approach as discrete optimization from Dimitrov et al.\nThe only difference I can see is that this paper is using gradient norm instead of the L2 distance between the gradients (which is not clear why this works better, as discussed above).\nGiven that authors are aware of that work, they should acknowledge in Section 4.4 that this approach comes from Dimitrov et al., and discuss the differences.\n\nAs a more general comment, the paper at few places does not acknowledge contributions from other paper and tends to overclaim their own contributions:\n\n- As mentioned earlier, Section 4.4 is basically the same as in LAMP (Dimitrov et al.), and yet there is no acknowledgment of this.\n- In the paragraph starting on Line 244 authors write that they propose using ROUGE as a metric, but this was already used in TAG (Deng et al.) and should be acknowledged there.\n- In L327, authors write \"This paper presents the first attack that can recover sentences from gradients in federated learning of language models\", but this is not true as such reconstruction has already been performed by Zhu et al., Deng et al., Dimitrov et al.\n\nAccording to this, writing should change to account from contribution from other papers.\n\nTypo:\n\nL71: both work -> both works\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cZZMLelw244", "reviewer": "Reviewer_FFKN", "summary": "This paper proposed FILM, an inversion attack that reconstructs text given gradients and language model parameters in the federated learning setting. The threat model considered an adversarial eavesdropper who can extract gradients and model parameters. FILM first recovers the bag-of-words from gradients and then exploits the model memorization to reconstruct the order of the text. The attack was evaluated empirically on benchmark datasets and outperformed prior attacks that were proposed to recover images. No defense was evaluated against the proposed attack.\n", "questions": "1. Another common threat model is to have a malicious server who knows only about the gradients aggregated on multiple users on multiple batches. How would FILM work in such a threat model?\n2. I would appreciate it if authors could provide an intuition behind using gradient norm in the scoring function in Section 4.4 instead of appendix so that readers could understand the method more easily.\n", "limitations": "The authors acknowledged their limitations but I think the paper can be improved if there were quantitative measurements on the limitations.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-organized. The threat model and the attack methods are explained clearly and easy to follow.\n2. The attack method is simple yet exploits the key vulnerabilities in training language models (i.e. memorization so that model would rank training data higher). The comparison with prior work also demonstrated that the attack is effective. \n\nWeaknesses:\n1. Although the author acknowledged this as well, there was no evaluation on any mitigation. I think it is critical to know the privacy-utility tradeoff for different mitigation strategies to understand the real impact of any attack. \n2. Seems like there are also other easy defenses which the author did not discuss. 1). If the server and client encrypts their traffic then the eavesdropper could not decode anything from the cipher-text. 2). If the model is fine-tuned on all parameters except the word embedding layer then the adversary could not get the bag-of-words in the first place.\n3. In Section 4.2, the author mentioned that they could not determine the frequency of words. I don’t think this claim holds as one could learn frequency information from the norm of word embedding gradients, i.e. more frequent words tend to have seen more SGD updates and thus larger norms.\n4. The precision number in Table 3 seems to suggest that recovering multiple sentences would be hard with FILM and results in quite a number of false positives. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_PWrQnio12U", "reviewer": "Reviewer_x95N", "summary": "In this work, the authors present a model inversion attack for language models in the context of federated learning, termed FILM. A white-box attack scenario is considered, where an adversary has access to the global model computed by the central server and the local updates (gradients) of an (arbitrary) participating client. The proposed attack is carried out through three key pivotal steps. First, a bag of words is extracted through the non-zero rows of the client's embedding gradients. In the second step, sentences are reconstructed through a beam search algorithm. During the third, and final step, tokens appearing in the extracted sentence are ordered using a scoring mechanism that takes into account sentences' perplexity and gradient norm. Overall, this is an interesting and well-motivated work, with a clear structure, background, and contributions. However, elaboration is needed in the final empirical evaluation that is critical for replicating the presented results in federated settings and improving results completeness. ", "questions": "W1: Why a larger batch size is important in Language Model (LM) tasks? It would be great if more background is given on the importance of the larger batch size. Moreover, does the attacker have full access to the entire vocabulary/embedding matrix during the white-box attack? It would be better to test the susceptibility of local datasets on a per-learner basis. For instance, what is the vulnerability (private text leakage) for each client? Do different data distributions (e.g., unbalanced, IID, or non-IID) affect the effectiveness of the attack? Is the attack method equally efficient across all clients' partitions? Moreover, what are some other federated domains that the proposed attack can be applied out of the box?     \n\nW2: Even though this work is placed in the federated setting, it is written as if the attack is tailored for centralized settings. Please elaborate on the federated environments you have evaluated. How many clients have you considered? How were the data/sentences distributed across clients? Are the data distributions easy to replicate so that future research can build on top and construct defensive mechanisms? Also, how many federation rounds did you train the models for? What are the update frequency and the aggregation method of local models? Training iterations should be expressed in terms of federation rounds (e.g., ref.2 below). To demonstrate model vulnerability due to overfitting another dimension would be attack success (at every federation round) versus the average training loss of the global model across all clients' local datasets.\n\n\nSuggestions / Text Edits:\n- missing $\\theta$ from prob. distribution in equation 1\n- line 69, 89: remove extra space\n- line 71, 76: change work to works\n- line 134: iterations to iteration\n- Table 3 shows the standard deviation for precision and recall but the number of conducted trials is missing\n- line 20, 61: consider adding the following references as well:\n\n\tref.1: https://www.ndss-symposium.org/wp-content/uploads/2020/04/diss2020-23004-paper.pdf \n\n\tref.2: https://proceedings.mlr.press/v143/gupta21a/gupta21a.pdf ", "limitations": "The authors have discussed proposed approach's limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Well-written paper, easy-to-follow with clear motivation and contributions.\n2. Interesting language model inversion attack that challenges existing language models training approaches.\n3. Empirical results demonstrate the effectiveness of the attack in challenging natural language tasks.    \n\nWeaknesses:\n1. Some details are missing from the attack setting.\n2. Further discussion is needed in the evaluation section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GpeUcCQSoyr", "reviewer": "Reviewer_LfUD", "summary": "The submission titled \"Recovering Private Text in Federated Learning of Language Models\" discusses privacy attacks in federated learning that recover private information from user gradients. The proposed attack first recovers a bag of words from the user update and then applies a series of beam searches to uncover sentences of user data. The approach is validated for a GPT-2 model with a batch size of up to 128 and sentences from enron and wikitext103.", "questions": "Some questions for the author response:\n\n* Is it not clear to me why this submission restricts the attack to only single sentences. What happens when the full range of 1024 tokens in GPT is filled by a user update, instead of only short sentences? Relatedly, what is the average sequence length evaluated in the experimental section? \n\n* Another question I have is concerning memorization. How many epochs are each of these models finetuned? Is it possible that the enron example leaks more data because every example is seen more often during training? Would it be possible to mark on Fig.4 when the data is first repeated? \n\n* The proposed approach seems limited to autoregressive models to my understanding? This would be ok, but I wonder if the authors comment on the feasibility of using this attack on a masked language model?\n\n\n\nOther minor remarks: \n* There is a typo in line 285.\n\n* The panels in Fig.5 are flipped (or at least referenced in text in the opposite way?)\n\n* I further think this submission could be improved by discussing Dimitrov et al. in a bit more detail in the related work, given that both approaches use language priors and gradient information, just in different stages of the recovery (but I do not think a comparison is necessary).\n\n* Minor: The submission discusses the oracle case where word frequencies are known to the attacker, and generally approximates word frequencies using the procedure proposed in Sec.4. However, word frequencies can also be recovered from the user gradients in addition to the basic bag-of-words by applying the approach of Wainakh et al, \"User Label Leakage from Gradients in Federated Learning\". This is also described in Fowl et al., \" Decepticons: Corrupted transformers breach privacy in federated learning for language models\". I wonder if this would be helpful to improve the attack?", "limitations": "The submissions discusses some limitations and ethical considerations and does so adequately.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I like the general approach proposed by the authors. The idea to re-use the language model itself to score its sentences and guide towards sentences likely contained in the gradient update is sensible and executed well. The submission features a range of interesting case studies, especially considering partial memorization and intermediate model states that I especially like. I do have a few smaller concerns regarding experimental evaluation,  ablation studies and related work that I would like to discuss:\n\n* The ROUGE scores in Sec.5 are computed by comparing the recovered sentence to all ground truth sentences as possible references if  my understanding is correct. I wonder whether this measure is overestimating the effectiveness of the attack as the number of references increases. It would be good to include baselines of a) a random sentence generated by beam search from the current model and b) a sentence generated only from beam-search starting from the bag-of-words (so only the first step of the proposed search). I also wonder what the peak accuracy would be, i.e. comparing the recovered sentence to all reference sentences in direct accuracy and choosing the highest accuracy.\n\n* Relatedly, this submission could be greatly improved with an ablation study of the sequence of improvements discussed in Sec.4, which I could not find in the appendix.\n\n* This work mentions related work in Deng et al. \"Tag: Gradient attack on transformer-based language models.\" and Dimitrov et al., \"Lamp: Extracting text from  gradients with language model priors\", but does not return to comparisons to these more recent works in the experimental evaluation. While this is (to me) understandable for the case of Dimitrov et al., given the complexity of the attack therein and the recency of that paper, but implementing Deng et al. should only be a minor change to the already present evaluation of the Zhu et al. baseline. In Dimitrov et al., the attack of Deng et al. was able to noticeably increase the baseline effectiveness of this type of matching attack.\n\n\nOverall, these are the reasons why I currenly only score this work as a weak accept, but I am interested in discussing these and increasing my score in the future.\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept