Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models
Authors: Liam H Fowl
Jonas Geiping
Steven Reich
Yuxin Wen
Wojciech Czaja
Micah Goldblum
Tom Goldstein
Venue: ICLR 2023 poster (2023)
Keywords: Federated Learning
Attack
Privacy
Transformers
Gradient Inversion

Abstract: Privacy is a central tenet of Federated learning (FL), in which a central server trains models without centralizing user data. However, gradient updates used in FL can leak user information.  While the most industrial uses of FL are for text applications (e.g. keystroke prediction), the majority of attacks on user privacy in FL have focused on simple image classifiers and threat models that assume honest execution of the FL protocol from the server. We propose a novel attack that reveals private user text by deploying malicious parameter vectors, and which succeeds even with mini-batches, multiple users, and long sequences. Unlike previous attacks on FL, the attack exploits characteristics of both the Transformer architecture and the token embedding, separately extracting tokens and positional embeddings to retrieve high-fidelity text. We argue that the threat model of malicious server states is highly relevant from a user-centric perspective, and show that in this scenario, text applications using transformer models are much more vulnerable than previously thought. 

Published as a conference paper at ICLR 2023
DECEPTICONS: CORRUPTED TRANSFORMERS BREACH PRIVACY IN FEDERATED LEARNING FOR LANGUAGE
MODELS
Liam Fowl', Jonas Geiping" University of Maryland (ifowl, igeiping; @umd. edu
Steven Reich University of Maryland
Yuxin Wen
University of Maryland
Wojtek Czaja University of Maryland
Micah Goldblum New York University
Tom Goldstein
University of Maryland tomg tomg@umd. edu
ABSTRACT
Privacy is a central tenet of Federated learning (FL), in which a central server trains models without centralizing user data. However, gradient updates used in FL can leak user information. While the most industrial uses of FL. are for text applications (e.g. keystroke prediction), the majority of attacks user privacy in FL have focused on simple image classifiers and threat models that assume honest execution of the FL protocol from the server. We propose novel attack that reveals private user text by deploying malicious parameter vectors, and which succeeds even with mini-batches, multiple users, and long sequences. Unlike previous attacks on FL, the attack exploits characteristics of both the Transformer architecture and the token retrieve
embedding, separately extracting tokens and positional embeddings high-fidelity text. We argue that the threat model of malicious server states is highly relevant from a user-centric perspective, and show that this scenario, text applica- tions using transformer models are much more vulnerable than previously thought.
INTRODUCTION
Federated learning (FL) has recently emerged as central paradigm for decentralized training. Where previously, training data had to be collected and accumulated on a central server, the data can now be kept locally and only model updates, such parameter gradients, are shared and aggregated by central party. The central tenet federated learning is that these protocols enable privacy for users (McMahan & Ramage 2017 Google Research, 2019). This is appealing to industrial interests, as user data can be leveraged to train machine learning models without user concerns for privacy, app permissions 2021). However, in reality, these federated learning protocols walk a tightrope between actual privacy and the appearance of privacy. Attacks that invert model updates sent by users can recover private information in several scenarios Phong et al. (2017): Wang et al. (2018) if no measures are taken to safe-guard user privacy. Optimization based inversion attacks have demonstrated the vulnerability of image data when only a few datapoints are used to calculate updates (Zhu et al.) [2099] Geiping et al.) 2020. Yin et al., (02T). To stymie these attacks, user data can be aggregated securely before being sent to the server as in Bonawitz et al. (2017). but this incurs additional communication overhead, and as such requires an estimation of the threat posed by inversion attacks against specific levels of aggregation, model architecture, and setting.
privacy regulations, such as GDPR (Veale al., 2018: Truong al.
Most of the work on gradient inversion attacks so far has focused on image classification problems. Conversely, the most successful industrial applications of federated learning have been in language tasks. There, federated learning is not just a promising idea, it has been deployed to consumers in
production, for example to improve keystroke prediction (Hard et al.) 2019 Ramaswamy al. 2019) and settings search on the Google Pixel (Bonawitz et al. 2019). However, attacks in this area have so
Authorr contrihuted equally. Order chosen randomly.

---

Published as a conference paper at ICLR 2023
Batch Size
Batch Size = 8
Batch Size = 16
Length 128
Ancient Egyptian deities the gods and goddesses worshipped Egypt ancient constitu. The beliefs and rituals myths these gods
Àncient Egyptian deities the gods and goddesses worshipped in ancient Egypt, The beliefs rituals surrounding gods
Ancient Egyptian deities the goda and goddeases worehipped in ancient Egypt. The beliefs rituals surrounding continue
Length 512
Ancient Egyptian well are the gods and goddesses worshipped in
Ancient Egyptian deities are the gods goddesses worshipped in ancient vague. beliefs and. tried these gods
Ancient Egyptian deities are the goda goddess hours thoughts in ancient final conception divine beliefs and rituals and these
ancient Egypt
The beliefs whereas
ritualsies these gods
formed
Figure An example reconstruction from small GPT-2 model using Decepticon attack, showing the first 20 tokens reconstructed from randomly selected user for different combinations sequence length and batch size on challenging text fragment. Highlighted text represents exact matches for position and token.
far succeeded only limited examples of sequences few (< 25) tokens (Deng al. 2021 Zhu
et al. 2019; Dimitrov et al. 2022). even for massive models such as BERT (with worse recovery for smaller models). This leaves the impression that these models already hard to invert, and limited aggregation is already sufficient to protect user privacy, without the necessity to employ stronger defenses such as local or distributed differential privacy (Dwork & Roth 2013).
In this work, we revisit the privacy of transformer models. We focus on the realistic threat model where the server-side behavior is untrusted by the user, and show that malicious update sent by the server can completely corrupt the behavior of user-side models, coercing them to spill significant amounts of user data. The server can then collect the original words and sentences entered by the user with straightforward statistical evaluations and assignment problems. We show for the first time that recovery of tokens and most of their absolute positions is feasible even on the order of several thousand tokens and even when applied small models only 10% the size of BERT discussed for FL use in Wang et al. (202T). Furthermore, instead of previous work which only discuss attacks for updates aggregated over few users, this attack breaks privacy when updates aggregated over more than 100 users. We hope that these observations can contribute to re-evaluation of privacy risks in FL applications for language.
MOTIVATION AND THREAT MODEL
At first glance, gradients from Transformer architectures might not appear to leak significant amounts of user data. Both the attention mechanisms and the linear components learn operations that act individually tokens, that their gradients naturally averaged over the entire length of the sequence (e.g. 512 tokens). Despite most architectures featuring large linear layers, the mixing of information reduces the utility of their content attacker. In fact, the only operation that "sees" the entire sequence, the scaled dot product attention, non-learned and does not leak separate gradients for each entry in the sequence. If were draw intuition from vision-based attacks, gradients whose components are averaged over 512 images are impossible to invert even for state-of-the-art attacks (Yin et 2021).
On the other hand, recovering text appears much more constrained than recovering images. The attacker knows from the beginning that only tokens that exist in the vocabulary are possible solutions and it is only necessary to find their location from limited list of known positions and identify such tokens to reconstruct the input sequence perfectly.
Previous Attacks Previous gradient inversion attacks in FL have been described for text in Deng et al. 2021): Zhu et al. (2019): Dimitrov et al.] (2022) and have focused on optimization based reconstruction in the honest connest-but-cute but-curious server model. Recently, (Gupta et al. (2022) model prior to improve attack success via beam-search that is guided by the user gradient. Our work is further related to investigations about the unintended memorization abilities of fully trained, but
language
2

---

Published as a conference paper at ICLR 2023
benign, models described in Carlini al. (2021): Inan et al. (2021 Brown et al. (2021) which can extract up to 1% of training data in some cases (Callini et al. (022). In contrast, we investigate attacks which directly attempt to retrieve a large fraction of the traming data utilized in single model update. This is the worst-case for a privacy leak from successive benign model snapshots discussed in Zaanelaa-lggeellll et al. 22220
Threat Model We consider the threat model of an untrusted server that is interested in recovering private user data from user updates in FL. Both user and server are bound by secure implementations to follow a known federated learning protocol, and the model architecture is compiled into fixed state and verified by the same implementation to be standard Transformer architecture. The user downloads a single malicious server state and returns their model updates according to protocol. The server then recovers user data from this update.
This threat model is the most natural setting from user-centric perspective. Stronger threat models would, for example, allow the server to execute arbitrary code on user devices, but such threats are solvable by software solutions such as secure sandboxing (Frey 202T). Still other work investigates malicious model architectures (Fowl al. 2021 Boenisch et al. 2021 but such malicious modifica- tions have to be present at the conception of the machine learning application, before the architecture is fixed and compiled for production (Bonawitz al., 2019), making those attacks most feasible for actors in the "analyst" role (Kairouz et al. 2021 Ideally, user and server communicate through a verified protocol implementation that only allows pre-defined and vetted model architectures.
However, none of these precautions stop an adversarial server from sending malicious updates (Wang et al. (02T). Such attacks are naturally ephemeral- the server can send benign server states nearly all of the time to all users, then switch to malicious update for single round and group of users (or single secure aggregator), collect user information, and return to benign updates immediately after. The malicious update is quickly overwritten and not detectable after the fact. Such an attack can be launched by any actor with temporary access the server states sent to users, including. aside from the server owner, temporary breaches of server infrastructure. MITM attacks or single disgruntled employees. Overall we argue that server states sent to users, especially for very expressive models such as transformers, should be treated by user applications as analogue to unirusted code that operates on user data and should be handled with the appropriate care.
3 MALICIOUS PARAMETERS FOR DATA EXTRACTION
Our attack leverages several interacting strategies that present in the sub-sections below.
For simplicity, we assume in this section that the FL protocol is fedSGD, i.e. model update returned by the users is the average gradient of the model parameters sent by the server computed over all local user data. If the server is malicious, then this protocol can either be selected directly or another protocol such federated averaging (fedAVG), which includes multiple local update steps (McMahan it al. (017)). could be modified maliciously to run with either only single local step or a large enough batch size. Both variants effectively convert the protocol fedSGD. The local user data is tokenized text fed into the transformer model, although the attack we describe does not make use of text priors and can read out an arbitrary stream of tokens.
We separate the exposition into two parts. First, we discuss how the server constructs a malicious parameter vector for the target architecture that sent to users. Second, we then discuss how the server can reconstruct private information from this update.
We first discuss parameter modifications for single sequence of tokens, and then move to mod- ifications that allow recovery of multiple sequences. For the following segment, we assume that the targeted model is based on a standard transformer architecture, containing first an embedding component that, for each token, adds token embedding and positional embedding, and second, series of transformer layers, each containing an attention block and a feed-forward block.
Modifications to Recover Token Ids and Positions: In principle, the goal of the attacker is simple: If the attacker reconstruct the inputs the first transformer layer, which are token embeddings plus positional embeddings, then a simple matching problem can identify the token and position id by correlating each input the known token and position embedding matrices, see ??. However,
3

---

Published as a conference paper at ICLR 2023
what the user returns is not token but a gradient, representing changes to the model relative to the user's ' data. This gradient is also an average over all tokens in the sequence. As a first step to improve attack success, the attacker can disable all attention layers (setting their output transformation to 0) and mostly disable outputs of each feed-forward block. The outputs of each feed-forward layer cannot be entirely disabled (otherwise all gradients would be zero), we set all outputs except the last entry to zero. A transformer modified in this way does not mix token information and the inputs to each layer are the same (except for the last entry in the hidden dimension) and equal to token embeddings plus positional embeddings.
Unmixing Gradient Signals: Let's investi- gate the resulting gradient of the first linear layer in each feed-forward block. We define the input embeddings that the attacker intends to recover as {ux J coming from a sequence of text con- taining S elements. The operation of the first linear layer in each (ith) feed-forwe feed-forward block is yk Wayg + bj. The gradient averaged over all tokens in this sequence is:
pax lirve
loken embeddiing
resoveee embedding
espoding
0.83
pos
e- 0.1-
posk
0.2
pos K-f
Figure 2: Recovering each token and its positions from the inputs the first transformer layer. We find the cor- relation matrix between recovered input embeddings to known positions and tokens, and solve a sum assignment problem to determine which position is most likely for each input.
Cx
Σ
0C*
V
Vwoigh (
Oyk
8
0Ck
8
0Ck
(i)
Σ
Σ
k= (0
Oyj
ab;
For each of the j entries, rows of the bias and weight, respectively. This be simplified further to: 8C,
Σx=1 ay(") ug for weight and Σ acs for bias (Geiping et al. 2020). Even after this reduction, ay?
it seems quite difficult recover Ug from these sums. Yet, malicious attacker can modify the weights of W; and of the subsequent matrix to allow for a closed-form solution.
To this end, we leverage recent strategies for separating gradient signals (Fowl et al., 2021 ). In
essence, this strategy encodes a measurement function into each weight row W of linear layer i and an increasing offset into each entry of the bias vectors b,. For linear layers that are followed by activations which cut off smaller input values (such as ReLU or GELU), the gradient of the weights in this linear layer will then encode cumulative sum of the values of all inputs up to some offset value. This structures the gradients observed in Equation (T) and original inputs dg can then be recovered by subtraction of adjacent rows. This is visualized in Figure
To go further into detail into this modification, the rows of all weight matrices W, are replaced by a measurement vector m and the biases over all layers are modified to be sequentially ascending.
(j) (m, ux) + 1(0). And yı is ordered so that
As a result, all linear layers now compute V%
vk 2 yk (f'), Vj < j" The subsequent ReLU unit will threshold y/ so that (f) = 0 directly implies (m, ux) < -b()) This ordering can now be exploited by subtracting the (j - 1)th entry in the gradient from the 5th entry to recover:
TWwn - V Wotn Ls = 105) uk:
9
ah ab6)
aCk ab++1)
x
(3)
Assuming the number of embeddings that fulfill this condition is either one or zero, we recover the input vector uy simply by dividing the term of the left of Equation (3) by the term on the right. All embeddings that collide and fall into the same interval are not easily recoverable.
for scalars X
We minimize the probability of a collision by first sampling a Gaussian vector m (O, 14) where d= dmooeel We then set the biases to 6, = [c......., ((+1))+-1] where k is the width of each linear layer and c = - -- I( '(+), where & 1 is the inverse of a Gaussian CDF and M is the sum over the widths of all included Transformer blocks. This uniformly distributes the values of (m, ug) over
4

---

Published as a conference paper at ICLR 2023
rows consisting
Gaussian vector,
Peter
Piper
picked
pepper
Embedding Vectors
Transformer Linear Layer
3
44
U
Peter
pepper
Distribution of inner products
Sug
Embedding vectors extracted via thresholding/blases
Figure 3: How embeddings are extracted from linear layers in the transformer blocks. We initialize the rows of the first linear layer in each Transformer block to a randomly sampled Gaussian vector (all rows being the same). The bottom portion of the figure depicts the internals of forward pass through this FC layer. When an incoming embedding vector ug enters the FC layer, the inner products (m, ux) fall into distribution which the attacker partitions into bins. ug is the only token whose inner product lies a given hin, the ReI.U activation will encode information about this single token, and (3) can he used to directly recover the token.
the bins created by the bias vector. If the width of the first linear layer in each block is larger than the number of tokens, collisions will be unlikely. We can estimate the rough mean and variance of (m, ux)) from either a small volume of public text or a random stream tokens, see Appendix D
Disambiguating Multiple Sequences Recovering multiple sequences - either from user data comprising multiple separate sentences of tokens, aggregates of multiple users presents the most difficult task for the attacker. Naively using the strategy described so far can only recover a partially ordered of tokens. For example. model update consists of five user sequences, then the attacker recovers five tokens at position 0, five tokens at position I, and so on, because positional embeddings repeat in each sequence. Re-grouping these tokens into salient sequences quickly becomes intractable as the number of possible groupings grows as n' for n sequences of length I. Further complicating matters for the attacker is that no learned parameters (and thus no parameters returning gradients) operate the entire length of a sequence in the Transformer model. In fact, the only interaction between embeddings from the same sequence comes in the scaled dot product attention mechanism.
Thus, if any malicious modification can be made to encode sequence information into embed- dings, it must utilize the attention mechanism of Transformers. We describe a mechanism by which attacker can disambiguate quences, even when model updates are aggre- gated over a large number of separate sequences. At a high level, we introduce strategy that en- codes unique information about each sequence into a predefined location in the embedding of each token in the sequence. Then, we can read out unique quantity for each sequence and thus assign tokens to separate sequences. This behavior is illustrated in Figure 4
Poter
Piper
picked
popper
Figure 4: A high-level schematic of MHA manipulation. The MHA block attends to the first word identically for every input sequence, encoding part the first token for each embedding in the entire sequence.
Let Wo. WK. Wy represent the query, key, and value weight matrices respectively, and let bQ. bx, by represent their biases. For simplicity of presentation, we explain this attack easily adapted multi-head attention. We first set the WK matrix to the identity (1dmoode)) and by = This leaves incoming embeddings unaltered. Then, we set WQ = Ô. and bq = ? po where Po is the first positional encoding, Here choose the first position vector for simplicity. but there are many potential choices for the identifying vector. This query matrix then transforms each embedding identically to be scaled version of the first positional encoding. We then set Wy 1
single head, but it

---

Published as a conference paper at ICLR 2023
velldt
Erbeddings
Grade of beotligge
Fatar (X1)
800
Coart
pesper (C)
picead
Trarsformer Layars
200
Toleo
Emboddingg 1g
Log of Tokcn Erbeoddnng Nom
Figure 5: Left: A high-level schematic of token leaking. The token embedding layer leaks tokens and frequencies solely through its sparse gradient entries. Right: Distribution and Histogram of log of norms of all token embedding gradients for GPT-2 13824 tokens. this case, gradient entries are non-sparse due to the tied encoder-decoder structure of GPT. but the embeddings of true user data (red) are clearly separable from the mass of all embeddings (bluc) by cutoff at 1.5 standard deviations (marked in black).
to be a partial identity (identity in the first d' entries where d' S d is hyperparameter that the server controls). Finally, we set by
Now, investigate how these changes transform incoming sequence of embeddings. Let {ux) )/=] be embeddings for sequence of length that enters the attention mechanism. uo is the embedding corresponding to the first token the sequence, so up is made up of the token embedding for the first token in the sequence, and the first positional encoding. WK produces keys K that exactly correspond to the incoming embeddings, however, Wo. bi collapses the embeddings to produce Q, consisting of identical copies of a single vector, the first positional encoding. Then, when the attention weights are calculated as:
QK Vdh
softmax
the attacker finds that the first embedding dominates the attention weights for all the embeddings, as the query vectors correlate with the first embedding the most. In fact, the 7 parameter can effectively turn the attention weights to delta function the first position. Finally, when the attention weights are used to combine the values V with the embeddings, by construction, a part of the embedding for the first word in the sequence is identically added to each other word in that sequence. So the embeddings where the first d' entries are the first d' entries of Fu, and the other dmoodel - d' entries are identically 0.
transformed as (a; )/=0 + {a; + 20,d' Vim where To,d' vector
If the attacker chooses to perform this modification the first Transformer block, this means that embeddings Un that the attacker recovers from each of the linear layers now also contain unique information about the first token of the sequence from which they came. The attacker then calculate correlations between first-position embeddings and later-position embeddings and group each embedding into sequence with other embeddings.
3.1 EXTRACTING USER DATA
Now that we have introduced the main mechanisms by which attacker prepares malicious parameter vector, we summarize how to extract user data. The attack begins after a user or aggregate group of users has computed their gradient using the corrupted parameters. Then, the server retrieves their update and begins the inversion procedure. We summarize the entire attack in Algorithm
Getting Bag-of- Bar-of-Words Containing All Tokens Even without malicious parameters, an attacker can retrieve the bag-of- words (unordered list of tokens) of the user from the gradient of the token embedding. MMlls et al. (2019) identified that the embedding layer gradient is only non-zero at locations corresponding to tokens that were used for training, as visualized in the left plot of Figure 5 Furthermore, the frequency of all words can be extracted by analyzing the bias gradient of the embedding decoding (last linear) layer, the magnitude of the update of each row of random embedding matrix is proportional to the frequency of word usage, allowing for a greedy estimation by adapting the strategy of Wainakh et al. (2021) which was proposed for simpler classification tasks.

---

Published as a conference paper at ICLR 2023
Algorithm 1 Decepticon Data Readout Overview
1: Input:
Sequence Length S, number of expected sequences N. Gradients VW, Vb; of the weight and bias of the first linear layer in each FFN block. Positional embeddings P, token embeddings T.
2: ty < Token Embeddings estimated bag-of-words of leaked tokens or all tokens
3: Pk Known positional embeddings
4: up + Vw,, -Vaai for linear layers i = ..L. and rows j L...., Vo
5: Lbach + Cluster index for each un clustering the first d' entries of each embedding.
6: for in O...N do
7: Uhn f- Entries of cluster n. in Learch
8: ug MaMca( Whn) on all entries > d'
9: end for
Unk + concatenate {u)) n=1
flimal tokems + Indices of Match(unk - Pks, tv) on all entries d > d'
We this finding first step restrict the number of possible tokens that should be matched. More details can be found in Appendix C
Recovering Input Embeddings: As first step, we recover all input embeddings uy using the divided difference method described in Equation (3) for all rows where Voş Vay., is non-zero,
i.c. least one embedding exist with bb (m,, U&) < -b++1. This results in list of vectors up with h < SN, for maximal sequence length S and maximal number of expected sequences N, that have be matched to separate sequences, positions and tokens.
Recovering Sequence Clusters: The first d' entries of each embedding encode the sequence identity as described in Section 3 The server can hence group the embeddings 2, uniquely into their sequences by a clustering algorithm. We utilize constrained K-Mean described in (Bradley et al., 2000), as the maximal sequence length S and hence cluster size is known. This results in a label for each vector uy sorting them into separate sequences.
Assigning Positions in each Sequence: Now, the matching problem has been reduced to separately matching each sequence, based on all entries d > d'. For given sequence with index n first match the embedding thn to the available positional embeddings Px- To do so, we construct the matrix of correlations between all Uhn and P& and solve rectangular linear optimal position for each Uhn. For the assignment problem, we use the solver proposed in Crouse (2016) which is modification of the shortest augmenting path strategy originally proposed Jonker & Volgenant (1987). Due to collisions, some positions will possibly not be filled after this initial matching step. We thus iteratively fill up un-matched positions with the best-matching embeddings from Whn. even if they have been used before in other positions, until each position is assigned match. Converting Embeddings to Tokens: Finally, we match all unt to all token embeddings by in either our bag of possible tokens or all tokens in the vocabulary. For this, we first subtract decorrelate the positional encoding pt from und to remove interference, and then again solve a linear sum assignment problem as described. The indices of the solution to this assignment problem will finally be the token ids tnk for each position in sequence n.
assignment to find the
In general, this process works very well to recover batches containing multiple long sequences. In the limit of many tokens, the quality of recovery eventually degrades as collisions become more common, making the identification of positions and token id less certain. In the regime of massive number of tokens, only a subset of positions are accurately recovered, and the remaining tokens are inaccurate, appearing in wrong positions. This process is random, and even for massive number of tokens there are often some sequences that can be recovered very well.
4 EMPIRICAL EVALUATION OF THE ATTACK
We evaluate the proposed attack using a range of Transformers. Our main focus is the application to next-word prediction as emphasized in practical use-cases (Hard et al. 2019 Ramaswamy et al.) 2019). We consider three architectures of differing sizes: the small 3-layer Transformer discussed as template for federated learning scenarios in Wang al. (202T) (11M parameters), the BERT-Base
7

---

Published as a conference paper at ICLR 2023
Totel Arcentty
0.0
0.6
Model
Model
GPF2
32
1230
200
Ratch
Figure 6: Total Average Accuracy. Total accuracy (i.c. percentage of tokens recovered correctly in their correct position) for all considered and varying batch sizes for sequence length 32 from wiki tilkieette text(left and sequence length 512 from stackover (right).
Accunty Most Lebeed Sequreee
0.0
0.8
Model
Model
0.3
02
3-aye Trandhomee
1.ger Toadhrrer
32
Batch Stop
Ratch Sioe
Figure Most-vulncrable Accuracy. Accuracy on the most-leaked sentence sequence length from xikitext(left) and sequence length 512 from st stackover fl ow (right). Note the size difference between BERT/GPT and the small transformer depicted here.
model Devlin al., 2019) attacked in previous work (Deng al., 2021 Zhu et al. 2019) (110M
parameters), and the smallest GPT-2 variation (Radford et al.) 2019) (124M parameters), We train the small Transformer and GPT-2 as causal language models and BERT as a masked language model. Predictably, the attack performance degrades most quickly for the 3-layer Transformer as this has the fewest parameters and thus the least capacity to uniquely capture a large number of user embeddings.
We test this attack on several datasets, wi k text (Merity al. 2016) sha shakespeare (Caldas) al, 20T9) and stackoverrll oo (Wang et al. 2021). For wikitext. We partition into separate "users" by article, while for the other datasets (BPE) tokenizer for the small Transformer and GPT-2, and the original BERT (WordPiece) tokenizer for BERT. We always report average scores the first 100 users, which have enough data to fill batch size x sequence length tokens, skipping users with less data who would be more vulnerable. We focus on fedSGD, i.e. single gradient updates from all users, but note the option of malicious server to convert another protocol to fedSGD discussed in Section 3
partitions given. We tokenize using the GPT-2
We evaluate using total average accuracy, BLEU (Papineni et al.) 2002 and ROUGE-L. Lin 2004).
Our total accuracy is stricter than the token (i.c., ofg-oo-wordd)) accuracy described previously Deng al. (202I); we only count success if both token and position are correct. Figure I shows partial reconstructions for a randomly selected, challenging sequence as batch size and sequence length increase. We find that even for more difficult scenarios with more tokens, a vast majority of tokens are recovered, with majority in their exact correct position.
Average Accuracy:
Our multi-head head attention strategy allows an attacker to reconstruct larger batches of sequences, as seen in Figure 6 (We include single sequences in Appendix F.2) This applies to an update from single user with more than one sequence, and also multiple users aggregating model updates. To the best of our knowledge, this is the first approach to explicitly attack multi-user updates with strong average accuracy. We find that for a sequence length 32, the proposed attack recover almost all tokens used in the user updates, and 50% of tokens their exact position for > 100 users. Even for a sequence length of 512, the attack remains effective for large batches.
We further chart the accuracy the most-recovered sequence in Figure 7 finding that even at larger aggregation rates, some sequences remain vulnerable to attack and are almost perfectly recovered by attacker.
Most-vulnerable Accuracy:
8

---

Published as a conference paper at ICLR 2023
Accurayy on Most Leaked Sequesce
0.8
0.00
Tona Accuary
0.6
0.0
0.4
0.4
rewfomm-sseem
newfrum-sskem
0.2
0.2
wibliestt
Batch Size
Batch Slize
Figure 8: Comparing attacks on several data sources. Left: Total accuracy (i.e. percentage tokens recovered correctly in their correct position) for GPT-2 and varying batch sizes with sequence length 32. Right: Accuracy the most- leat-l sentence for all data sources.
08
D6
0.-
06-
0.4
0.4
02
0.2
02-
00
0.0
3000
400 1200
Figure 9: Comparison between the malicious server threat model with attack and TAG (honest-but-cuns) -ut-ccoouss for the 3-layer transformer described in Wang al. (00TT) variable seq. length (batch size on text.
Comparison to other threat models. Several works approached the topic of reconstructing text from FL updates, under a "honest-but-curious threat model where benign parameters sent the user, and the FL protocol is performed normally. We compare by fixing an architecture in this case the Transformer-3 model described earlier. We then consider the setting with batch size and evaluate our proposed method against the TAG attack Deng et al. (2021) which improves upon previous results in Zhu a (2019). We experiment for varying sequence lengths in Figure
The performance of TAG very quickly degrades for sequences of even moderate length. For a single sequence, proposed attack maintains high total accuracy for sequences exceeding 1000 tokens, whereas the total accuracy for TAG soon approaches 0%. Overall, we find that the severity of the threat posed by but-curious' threat model, and argue for
our attack is orders of magnitude greater than in the the re-evaluation of the amount of privacy leaked by FL applications using transformers.
The attack is largely invariant to the source of user text, as detailed in Figure 8 where diverse sources of data are almost equally vulnerable. Shakespeare data is most vulnerable due to the use of short and simple sentences which are consistently well-recovered. Ran- dom text noticeably more vulnerable, which is relevant from security perspective, as an attacker often be interested in text fragments that are not usual text, e.g. passwords or social security numbers.
Accuracy on other Datasets:
Attack Confidence contrast to existing attacks, the attacker can verify recovered tokens. After recovering positions and tokens for each breach u4 can cross-check whether the assigned position Pa. and token embedding t, perfectly explain U; Tokens that fulfill this condition are perfectly recov- ered and allow the attacker to certify which parts of their recovered text accurately reflect user data.
CONCLUSIONS
In this work re-evaluate the threat posed by attacks against privacy for FL for Transformer-based language models. We argue that from a usserpperppcctive the most natural threat model is not to trust the server state. We show that malicious server can send updates that encode large amounts of private user data into the update sent by users. This attack and threat model significantly lower the bar for possible attacks, recovering up to several thousand tokens of user data. Our results underline the necessity of adoption of strong privacy guarantees for users in federated learning.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT AND MITIGATIONS
Several known case studies of FL models deployed in production in Hard et al. (2019); Hartmann (202T) rely only on aggregation to preserve privacy in FL for text. However, the attack we describe in this work shows that aggregation levels considered safe based on previous work may not be enough to protect user privacy in several use cases: The setting deployed to production in Hard et al. (2019) runs FL. on users updates with 400 sentences with 4 words per average message sent to the server without further aggregation, well within the range of the investigated attack if trained with transformer model. We thus briefly discuss other mitigation strategies for attacks like this. We roughly group them into two categories: parameter inspection and differential privacy based defenses.
Parameter inspection and verification of the server state is currently not implemented in any major industrial FL framework (Paulik et al.) [02TT] Dimitriadis et al. 2022 Li et al.) 2019 [onawitz et al., 2019), but after the publication of this attack, a rule could be designed to mitigate it (which we would encourage!). However, we caution that the design space for attacks described here seems too large to be defended by inspecting parameters for a list of known attacks.
Based on these concerns, general differential privacy (DP) based defenses continue to be the more promising route. Local or distributed differential privacy, controlled directly by the user and added directly on-device (instead of the server level as McMahan al. (2018) allows for general protection for users without the need to trust the update sent by the server (Kairouz et al. 222 la Bhowmick et al. 2019). Local DP does come at a cost in utility, observed in theoretical (Kairouz et al.
2021b; Kasiviswanathan et al., 2011 Ullman| (021]) as well as practical investigations (Jayaraman
& Evans 2019) when a model pretrained from scratch (Li et al. 2022 Noble al., 2022). DP currently also comes at some cost in fairness, as model utility is reduced most for underrepresented
groups (Bagdasaryan et al. 2019). Yet, as any attack on privacy will break down when faced with sufficient differential privacy, with this work we advocate for strong differential privacy on the user side, as incorporated for example into recent system designs in Paulik et al. (202I).
We include further discussion about mitigations in practice in Appendix G including limited experi- mentation with gradient clipping/noising, as well as parameter inspection adaptations.
Attacks like the one presented in this paper present a privacy risk to users and their data. As such, the publication of the possibility of attacks like the one described in this work has potential negative consequences for the privacy of deployed user systems that do not apply mitigation strategies as described above. As FL becomes more integrated into privacy-critical applications, it is of paramount importance that the community is aware of worst-case threats to user privacy, that mitigating measures can be taken. If left undefended, an attack like this could compromise large amounts of user data, including sensitive text messages and emails. Because previous research on privacy attacks against language models in FL has been relatively scarce, some might believe that simple defenses such as aggregation are sufficient to ensure user privacy. This work shows that this belief is unfounded and further measures must be taken. We hope that this attack raises awareness of the vulnerability of FL models.
REPRODUCIBILITY STATEMENT
We provide technical details in Appendix A and additional pseudocode for mentioned algorithms in Appendix B and provide code in conjunction with this submission to reproduce all attacks and settings discussed in this work. Within our code submission we provide command-ling interfaces and jupyter notebooks to evaluate the attacks and interactively vary parameters. All attacks discussed in this work can be cheaply reproduced using Laptop CPUs, as no extensive computations on GPUs are necessary to run the attack.
ACKNOWLEDGEMENTS
This work was supported by the Office of Naval Research (#N000142112553 the AFOSR MURI program, DARPA GARD (HR001 12020007) the National Science Foundation (II-2221212), and Capital One Bank.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential Privacy Has Disparate
Impact on Model Accuracy. H. Wallach, Larochelle, A. Beygelzimer, F. dhtextquotesingle Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32,
pp. 15479-15488. Curran Associates, Inc., 2019, URL httttt::: nips. cc wtppppppprcpppre
diffeeaiffeeetiial.. wtw.ffrrrnntall-ppiaaay-y--- prwipcciivieeeei - inaptte--mmccttt mode -
pdf
Abhishek Bhowmick, John Duchi, Julien Freudiger, Gaurav Kapoor, and Ryan Rogers. Protection
Against Reconstruction and Its Applications in Private Federated Learning. arXiv:1812.00984 [cs, stal], June 2019. URL
orbbass// 1812. 00984
Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. When the Curious Abandon Honesty: Federated Learning Is Not Private.
arXiv:2112.029188 12.02118 [cs], December 2021. [RL htatt:://axx.v.org/// wttp://..rggab////1//// www.aasslli. 02918
Kallista Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical Secure Aggregation for Privacy Preserving Machine Learning. Technical Report 281, 2017. URL
iacr
org/2017/281
Kallista Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzoechi, H. Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards Federated Learning at Scale: System Design. arX1v:1902.01046 [cs, stat]. March 2019. URL http:
01045
Paul S. Bradley, Kristin P. Bennett, and Ayhan Demiriz. Constrained k-means clustering. Microsoft Research, Redmond, 20(0):0, 2000.
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning? Proceedings of the 53rd Annual ACM SIGACT Symposium Theory of Computing, pp. 123-132, June 2021. doi: 10.1 445/33406325.451111
hRLlhttp://arxiit org/ abs 221222222222 06421
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. LEAF: A Benchmark for Federated Settings. doi: 10. 10.448500/arxii.111220011
arxiv: stat], December 2019.
URL http
// //arxiv.org/ http://wwwrr.a.aa/.......... 01097
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extract- ing Training Data from Large Language Models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021. ISBN 17781-9339133-24- org/conferencee useni www.wesstttyyiie hrr.a//eeeeoeeeeeeerrrerereeeeaa htt.://mtttttttommmmmmmeeaeaaaar
URL /ttphtttss///...snnl
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. arxiv:2200.06646((3)) February 2022. URL. http:p//t/////xii wtwp://wwwww..rgb.///bbaa// 07646
David F. Crouse. On implementing 2D rectangular assignment algorithms. IEEE Transactions on Aerospace and Electronic Systems, 5274) August 2016. ISSN 1557-9603. doi:
10.1 2LL.0//AAEE.220..1499999
Jieren Deng, Yijue Wang, Ji Li, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. TAG: Gradient Attack on Transformer-based Language Models, wrXiv:2103.06819 [cs], September 2021. htLt:::: http:/awwii....//aaaaa/ httpg//bb/22100 00611.01111...11
06819
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Mare' Aurelio Ranzato. Residual Energy-Baid -ased Models for Text Generation. arXiv:2004.17714 [cs], April 2020. URL http? raxiiv.ao httpiv..orrrabb/////0000
.
11714
11

---

Published as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019. URL
httt:://ariii org/ abs 1810 04805
Dimitrios Dimitriadis, Mirian Hipolito Garcia, Daniel Madrigal Diaz, Andre Manoel, and Robert Sim.
FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulations. arxiv:2203. March 2022. URL
Dimitar I. Dimitrov, Mislav Balunovic, Nikola Jovanović, and Martin Vechev. LAMP: Extracting Text from Gradients with Language Model Priors. February 2022. URL https:
htg/abs/ae 2202. 08827v1
Cynthia Dwork and Aaron Roth. The Algorithmic Foundations Differential Privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4)211 2013. ISSN 1551-305X, 1551- 3068. doi: 10.156 10/40000400000 utt.htteee/www...oppaaaeeeeeee.e Eoundat ions and--reends in- aat.ap/eaa...a........a..a...aaa
Liam H. Fowl, Jonas Geiping, Wojciech Czaja, Micah Goldblum, and Tom Goldstein. Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models. In International Conference on Learning Representations, September 2021. URL Nttts://openrevian
Suzanne Frey. ber 2021.
Introducing Android's URL ceaeooddd-priaatee private cawpeeeccoppeeeee.e html
Compute http:/eccrritt..ooollbbebee com cwtp://c/cc///////0/// 2021/09/
Private
Services,
Septem-
Introducing androids
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Invert-
ing Gradients How easy is to break privacy in federated learning? In
Advances in Neural Information Processing Systems, volume 33, December 2020.
URL c2cc5bbbe8111e111200666666----60
httpps///ppooeednnn.....rireosep files
Team Google Research.
Federated Learning. May 2019.
URL https:
withgoogle.com
Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recovering Private Text in Federated Learning of Language Models. arxiv:2205 015///0/58811//ss/// May 2022. doi: 10.485500/rXxv.2205 085514 http://t://aaxt..rr//abb////////
08514
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated Learning for Mobile Keyboard Prediction. arX1v:/811.03604 [cs], February 2019. URL http::/arxiv.oor
abs/1811. 03604
Florian Hartmann. Predicting Text Selections with Federated Leaming Novembbeer 2021. URL http: //ai. g/a..googl
with html
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.08411 08415, 2016.
Huseyin A. Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Rühle, James Withers, and Robert Sim. Training Data Leakage Analysis in Language Models, arx:vv:2001005405/css/ February 2021. doi: 11.48550/arxiv.2000..00 /tt/p//ppp//arrrrrrrrrrr//////..
Bargav Jayaraman and David Evans. Evaluating Differentially Private Machine Learning in Practice. In 28th (USENIX) Security Symposium (/USENIX) Security 19), pp. 1895-1912, 2019. ISBN 978- 978-1-939133--6699 usen xsecur i ty www.ecuriyyy9//reeeee wtwpss/tttttttteeeeeeeeeeettttet iinaanattiionn hmm/jayaraman
URL https::///// //w:://wwwww.... www.nu............... wwt////ooo/oooeeneeoeoeooooeeeee http:///wwww..e.....oooooooreree
R. Jonker and A. Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, 38(4):325 -400, December 1987. ISSN 1436-5057. doi: 10.10077BF022288710, URL htI.https:: //ddi http....11.0010///22221111
1007/
12

---

Published as a conference paper at ICLR 2023
Peter Kairouz, Ziyu Liu, and Thomas Steinke. The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation. In Proceedings of the 38th International Conference Machine Learning, pp. 5201-5212. PMLR, July 2021a. URL https://proceedngg
m r. pressppes//ee. / v1 1311//9// 39/kai rouz2la html
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz. Zachary Charles, Graham Cormode, Rachel Cummings, Rafacl G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage. Ramesh Raskar, Dawn Song. Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Pranceth Vepakomma, Jianyu Wang. Xiong. Zheng Xu, Qiang Yang. Felix Yu, Han Yu, and Sen Zhao. Advances and Open Problems in Federated Learning. 04977
[cs, stat], March 2021b. URL
http://arxiv.. http://xxivv.ogab//////
Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. What Can We Learn Privately? SIAM Journal on Computiing, 40(3):793-822 January 2011. ISSN 0097-5397. doi: 10.1137/090756690. URL htt:ss://ppps:: siam. orq/doi/ 10. 001137/090756090
Wenqi Li, Fausto Milletari, Daguang Xu, Nicola Rieke, Jonny Hancox, Wentao Zhu, Maximilian Baust, Yan Cheng, Sébastien Ourselin, M. Jorge Cardoso, and Andrew Feng. Privacy-Preserving Federated Brain Tumour Segmentation. In Hcung-Il Suk, Mingxia Liu, Pingkun Yan, and Chunfeng Lian (eds.), Machine Learning in Medical Imaging, Lecture Notes in Computer Science, pp. 133- 141, Cham, 2019. Springer International Publishing. ISBN 078-3-030-32692--. doi: 10.1007/ 978-3-030- 118-3-030-32692-00 22692-0_166
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large Language Models Can Be Strong Differentially Private Leamnrr. In International Conference on Learning Representations, March 2022. URL httpt::////c... /tttpss://peereeiiieee
Chin-Yew Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL htttttps:/acaaattooooooys
S.G. Mallat and Zhifeng Zhang. Matching pursuits with time- fine-frreuency dictionaries. IEEE Trans- actions on Signal Processing, 41(12)3337-3341,, December 1993. ISSN 1941-0476. doi: 10.1 109/78.258082.
Brendan McMahan and Daniel Ramage. Federated Learning: Collaborative Machine Learning without Centralized Training Data, April 2017. URL http:///ee/ //ai goog www.gooooooogooooooooooooooooooo www. 2017/
H. Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Arcas. cormmunniaation--ffiiiieee Learning of Deep Networks from Decentralized Data. 16Xiv:1602.05622 [cs], February 2017. URLhttp:
abs 1602. 05629.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning Differentially Private Recurrent Language Models. arX.v.7700006633 [cs], February 2018. URL http:
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting Unintended Feature Leakage Collaborative Learning- In 2019 IEEE Symposium Security and Privacy (SP), pp. 691-706, May 2019. doi: 10. 10.1109/S22019.00002
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. November 2016. URI wtt.htttp///pppmme
net/forunm ii
13

---

Published as a conference paper at ICLR 2023
Maxence Noble, Aurélien Bellet, and Aymeric Dieuleveut. Differentially Private Federated Learning on Heterogeneous Data. axxiv:2111.02288ccs math, stat], February 2022. URL B.RLhtpp://// //ap:i//axiv
erg/abs/2111. 09278
Documentation ONNX.
ONNX
Operator Schemas. Open Neural Network Ex-
change, January 2022.
URL https::/gitthbb.com com http///ww/////noo/o//o//
50a1 1ha1981dc3d50aff 3ow.aaa0...111110003033036666666
.d
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A Method for Auto-
matic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the
Association for Computational Linguistics, PP. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 0L31155/10738831073333 URL https://aclanthor htwp:/hoooooy...rr/00001144
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De- Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentia- tion in PyTorch. NIPS 2017 Autodiff Workshop, Long Beach, CA, 2017. URL https
//openreview net.eeewwee/eeeeeeememeee.eeeeee
Matthias Paulik, Matt Seigel, Henry Mason, Dominic Telaar, Joris Kluivers, Rogier Dalen, Chi Wai Lau, Luke Carlson, Filip Granqvist, Chris Vandevelde, Sudeep Agarwal, Julien Freudiger, Andrew Byde, Abhishek Bhowmick, Gaurav Kapoor. Si Beaumont, Aine Cahill, Dominic Hughes, Omid Javidbakht. Fei Dong, Rehan Rishi, and Stanley Hung. Federated Evaluation and Tuning for On-Device Personalization: System Design & Applications. ar.iv.2 08503, February 2021. abs URL https: /tt.ittttaa//ariivoooo
doi: 10.48550/ar2iv.21022.552
02 08503v1
Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Hivacy-Preserving Deep Learning via Additively Homomorphic Encryption. Technical Report 715, 2017. URL
http::/ iacr. /rg/2017/213
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAl, pp. 24, 2019.
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Françoise Beaufays. Federated Learning for Emoji Prediction in Mobile Keyboard. arXiv:/906.00 04329 [cs], June 2019. URL http:
/ //axiiv.. wwg. aba/ /1906. 04329
Ron Rubinstein, Michael Zibulevsky, and Michael Elad. Efficient implementation of the K-SVD al- gorithm using batch orthogonal matching pursuit. Technical report, Computer Science Department, Technion, 2008
Gowthami Somepalli. Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint 106.v22006.01442 2021.
Nguyen Truong, Kai Sun, Siyao Wang, Florian Guitton, and YiKe Guo. Privacy preservation in federated learning: An insightful survey from the GDPR perspective. Computers & Security, 1 110: 102402, November 2021. ISSN 0167-4048.
doi: 20..0166i.cose220111020444 URL hntps
www
com
Jonathan Ullman. Tight Lower Bounds for Locally Differentially Private Selection. arXiv.1802.02638 [cs], May 2021. URL http:://axittt:
abs www.b////11..000 02638
Michael Veale, Reuben Binns, and Lilian Edwards. Algorithms that remember: Model inversion attacks and data protection law. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 376(2133):20 80083, November 2018. doi: 2018 0083. htl.http::/// www.rrolllll..ttoooooo..oooo.ooe 2018. 0083]
ng doi /tt.://oul//.11100/1111111
Aidmar Wainakh, Fabrizio Ventola, Till Müßig, Jens Keim, Carlos Garcia Cordero, Ephraim Zimmer, Tim Grube, Kristian Kersting, and Max Mühlhäuser. User Label Leakage from Gradients in Federated Learning. arXiv:2105.09366 [cs]. June 2021. URL http:/ http///xxvvv.//// orxiv.or//abs/
2105. 09369
14

---

Published as a conference paper at ICLR 2023
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera Arcas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi, Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecny, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi. Sashank J. Reddi, Peter Richtarik, Karan Singhal, Virginia Smith, Mahdi Soltanolkotabi, Weikang Song, Ananda Theertha Suresh, Sebastian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wennan Zhu. A Field Guide to Federated Optimization. arXiv:2107.06917 [cs], July 2021. URL httt:://arxiw.org/abbss..s
069117
Zhibo Wang, Mengkai Song, Zhifei Zhang, Yang Song, Qian Wang, and Hairong Qi. Beyond Inferring Class Representatives: User-Level Privacy Leakage From Federated Learning. arXiv:1812.0053:
[cs], December 2018. URL http::ttttttttt/ wwwpxiii.rrg/.....
org/ abs////bs/// 1812. 00535
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's Transformers: State-of-the-ar Natural Language Processing. arXiv:1910.0377 [cs], July 2020. URL
abs/19 03771
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, and Pavlo Molchanov.
See Through Gradients: Image Batch Recovery via GradInversion. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 16337-16346, 2021. URL https: http::///pppaccccsceoo / wwwppppwwwwces........cc.eeeeee. twecth.eeeeccc. com////fccoo////co content wttpe/////////2/////m//////// ttt.evs..ccoooottte///////////tt
See Through Gradients Image Batch Recovery via GradInversion CVPR
2021 1 www.pppper..tt html
Santiago Zagella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew Paverd, Olga Ohrimenko, Boris Köpf, and Mare Brockschmidt. Analyzing Information Leakage of Updates to Natural Language Models. In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pp. 363-375, Association for Computing Machinery, New York, URL htttpp:://ol..og////1/1/
NY. USA, October 2020. ISBN 978- 1778---4003-789-9
3372297.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep Leakage from Gradients. In H. Wallach, H. Larochelle, A. Beygelzimer, F. ditextquotesingle Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 14774-14784. Curran Associates, Inc., 2019, URL http 7/papers peapers.nips... ctt.pp.cccppape/911177 deep- Leakageeeeaeeeeeeeeee from wrw.---rrooggaaaitss........
A TECHNICAL DETAILS
We implement all attacks an extensible PyTorch framework (Paszke al. 2017 for this type of attack which allows for a full reproduction of the attack and which we attach to this submission. We utilize huggin huggingfar ance models and data extensions for the text data (Wolf et al.] 2020).The The attack is separated into two parts. The first part is the malicious modification of the server parameters, which triggered immediately before the server sends out their model update payload to users.
We implement the malicious parameter modifications as described in Sec. 3. random model initialization and reserve the first 6 entries of the embedding layer for the sentence encoding for the Transformer-3 model and the first 32 entries for the larger BERT and GPT-2 models. The corresponding entries in the positional and token embeddings are reset to O. In the MHA modification, we choose softmax skewing of 1e8, although this value can be significantly lower in practice as well. We reserve the last entry of the embedding layer for gradient flow to each linear layer, that the attack is able utilize all layers described. This entry is scaled by 1e 6 for the smaller Transformer-3 model and = le 8 for the larger models, so that the layer normalization not skewed by large values arising from the last embedding entry.
We initialize from
15

---

Published as a conference paper at ICLR 2023
For the attack part, we retrieve the gradient update the server and all recovery computations in single float precision. We first run a token recovery as discussed in Sec 3.1, which we additionally summarize in Sec. 3.4 and then proceed with the attack following Algorithm [T] For the sentence labeling we cluster using constrained K-means as described in |Baddey et al. (2000). For all assign- ment problems we utilize the linear assignment solver proposed in Crouse (2016) which is modification of the shortest augmenting path strategy originally proposed in Jonker & Volgenant (9877..
During the quantitative evaluation, we trial 100 users each with request for updates of size sequence length x batches. We skip all users which do not own enough data and do not pad user data with [PAD] tokens, which we think would skew results as it would include a large number of casy tokens to recover. All measurements are hence done on non-trivial sentences, which are concatenated to reach the desired sequence length. Each user's data is completely separated, representing different wikipedia articles as deseribed in the main body. Overall, the attack is highly successful over a range of users even with very different article content.
To compute metrics, we first resort recovered batches toward the ground truth order by assigning to ground truth batches based on total token accuracy per sentence. This leads a potential underestimation of the true accuracy and ROUGE metrics of the recovery as sentence are potentially mismatched. We compute Rouge (Lin 2004) scores based on the sorted batches and BLEU scores (Papineni et al., 2002) (with the default huggi n face implementation) by giving all batches as references. We measure total accuracy the number of tokens that are perfectly identified and in the correct position, assigning no partial credit for either token identity or position. This is notably in contrast the naming of metrics in Deng al. (2021) where accuracy refers only overall token accuracy. We refer to that metric as "token accuracy", measuring only the overlap between reference and reconstruction sentence bag-of-words, but report only the total accuracy, given that token accuracy is already near-perfect after the attack on the Hag-of-wordd described in Sec 3.1.
B ALGORITHM DETAILS
We detail albaalgorshhms as additional material in this section. Algorithmmaan Algorithm detail the token recovery for transformers with decoder bias and for transformer with tied embedding. These roughly follow the principles of greedy label recovery strategy proposed in Wainakh et al. (202T) and we reproduce them here for completeness, incorporating additional considerations necessary for token retrieval.
Algorithm Token Recovery
Algorithm 2 Token Recovery (Decoder Bias)
(Tied encoder Embedding)
Embedding weight gradient Jes se- quence length s, number of expected se- quences n, cutoff factor
Input:
1: Input: Decoder bias gradient embedding gradient Je-
sequence length number of sequences n.
2: nej = 11ge,112 for all j = 1,...N embedding rows
2: Plokens f- all indices where Øb <
3: + all indices where 9r 0
3: n1 Mean(log Std
4: Wiokens append Prokens Ve
c = p + fo
5: Mimpuct sm Σ Eieeheen 9b.
1
5: Utokens +- all indices where ne
Mimpact 1 Σ Etaaiin Rei
6: 9b [uookens] 9b Utokenss Mimpa
7: while Length Urokens < sm do
7: ne [iokens] <- ne Diokens Mimpuct do
8: arg min, Ib;
while Length of Unnkens
9: 9b < 9bj Mimpact
9: j = arg min geesia Ness
Vrokens append j
10: Tey Thej -imppat
11: end while
11: Vtokens append
12: end while
16

---

Published as a conference paper at ICLR 2023
Gradi bediing
Coant
(d)
piceed (2))
Transformrr Layers
Tolorn
Embeddinn ng
-10
Figure 10: Left: A high-level schematic of a simple case of token leaking. The token embedding layer can leak tokens and frequency solely through its sparse gradient entries. Right: Distribution and Histogram of log of norms of all token embedding gradients for GPT-2 for 13824 tokens. In this case, gradient entries non-sparse due the tied encoder-decoder structure of GPT, but the embeddings of true data (red) clearly separable 1.5 standard deviations (marked in black).
from TOREN Bl
Even without any malicious parameter modifications, an attacker can immediately retrieve the bag- of-words (the unordered list of all tokens and their assorted frequencies) of the user data from the gradient of the token embedding, MMlis al. (2019) previously identified that unique tokens can be recovered due to the sparsity of rows in the token embedding gradient, as visualized in the left plot of Figure 5] But, perhaps surprisingly, we find that even the frequency of all words can be extracted. Depending on the model architecture, this frequency estimation can either be triggered by analyzing either the bias gradient of the decoding (last linear) layer, or the norm of the embedding matrix. In both cases, the magnitude of the update of each row of a random embedding matrix is proportional to the firquency of word usage, allowing for a greedy estimation by adapting the strategy of Wainakh et al. (202I) which was originally proposed extract label information in classification tasks.
For models which contain a decoder bias parameter, this estimation directly Ieads to > 99% accurate estimate of word frequencies. For models without decoder bias, such GPT (Radford et al. 2019). the same strategy can again be employed based on the magnitude of embedding rows. We can further accurately predict token frequency even for models which tie their embedding and decoding weights, and as such do not feature naturally sparse embedding gradients. However, examining the distributions of embedding row norms log-space for tokens contained in the other updates and all tokens, as visualized in the right plot of Figure [] we find that unused tokens can be removed by a simple thresholding. The resulting token frequency estimation is inexact due to the cut-off, but still reaches bag-of-words accuracy of 93.1 1 % accuracy for GPT-2, even for an update containing 13824 tokens.
This first insight already appears to have been overlooked in optimization- based attacks (Deng et al.) 2021) Dimitrov et al., (002) and does not require any malicious modifications of the embedding sent by the server. However, for a model update averaged over multiple users and thousands of tokens, this recovery alone is less of a threat to privacy. Naively, the ways to order recovered tokens scale as n!, and any given ordering of a large amount of tokens may not reveal sensitive information about any constituent user involved in the model update.
D MEASUREMENT NORMALIZAATONN
In order to retrieve positionally encoded features from the first linear layer in the feed-forward part of each Transformer block, we create "bins" which partition the possible outcomes of taking the scalar product of an embedding with a random Gaussian vector. To do this, we estimate the mean and variance of such measurement, and create bins according to partitions of equal mass for Gaussian with this mean and variance. A natural question arises: how well can we estimate this quantity? If the server does not have much data from the user distribution, will this present problem for recovering user data? To this end, we demonstrate that the server can estimate these measurement quantities well using surrogate corpus of data. In Figure [2] see that the Gaussian fit using the Wikitext dataset is strikingly similar to the one the server would have found, even on a very different dataset (Shakespeare). We further note that ambitious server could also directly use model updates
17

---

Published as a conference paper at ICLR 2023
Commotte
Figure 11: Bag-of-words Accuracy for from embedding gradients of 13824 tokens from wikieexz Left: The small transformer model where tokens frequency can be estimated directly from the decoder bias. Token frequency estimation is 96. 7% accurate, unique token recovery are 99.8% accurate. Right: The (small) GPT-2 variant where tokens are estimated from the norms of the embedding rows. Token frequency estimation is 93.1% accurate, unique tokens are 97.1% 1% accurate due the cutoff at 3z ?
retrieved from users a first round federated learning, to directly glean the distribution of these
linear layers, given that the lowest bin records the average of all activations of this layer. However, given the ubiquity of public text data, this step appears almost unnecessary but possibly relevant in specialized text domains when using Transformers for other data modalities such as tabular data (Somepalli et al. 2021).
However, for language data, these quantities can also simply be estimated using small number of batches (100) filled with random token ids. We show in Figure [3 that this leads to essentially the same performance in average total accuracy and only minor differences in peak accuracy.
Transferability of estimated measurement
Shakespeare
Wkkitett
200
Gaussian (Wiki)
Gaussian (Shakespeare)
150
50
0
-0. 01.007500.00000000 00.00000005 0.0000 0.0025 0.0050 0.0075 0.0100
Figure 12: Comparison of Gaussian fit the measurements of Shakespeare dataset, and Gaussian fit to the measurements (against same vector) for the Wikitext dataset.
D. RANDOM TOKEN VULNERABILITY
Note that in Figure g we see that sequences generated randomly from the vocabulary are more vuinerable than other datasets. This is because randomly generated sequences average result in fewer collisions as all possible tokens from the vocabulary are used with equal probability. Zipf's law for the distribution of words in English lends credence to this interpretation. More common words, and associated positions for that matter, will result in more collisions and degrade the attacker's ability to reconstruct sequences.
18

---

Published as a conference paper at ICLR 2023
Accurscy on Most Lesked Sequence
0.8
Tonal Accurayy
0.6
Experiment
Experiment
Betmord on publlc Dats
Pariond publlc Thtn
22
64
122
250
512
16
64
122
256
512
Batch Size
Batch Sizee
Figure 13: Using random data to estimate the measurement distribution. Left: Total accuracy (i.e. percentage tokens recovered correctly their correct position) GPT-2 and varying batch sizes with sequence length 32. Right: Average token accuracy on the most-leaked sentence. Random tokens can an effective substitute for public data to estimate the measurement mcan and standard deviation.
E
VARIANTS AND DETAILS
E.1
MASKED LANGUAGE MODELLING
For problems with masked language modelling, the loss sparse over the sequence length, as only masked tokens compute loss. This impacts the strategy proposed in Sec 3.2, with disabled attention mechanisms all but the first layer, no gradient information flows to unmasked entries in the sequence. However, this can be quickly solved by reactivating the last attention layer, that it equally attends to all elements in the sequence with minor weight increment which we set to 10. This way. gradient flow re-enabled and all computations can proceed designed. For BERT, further disable the default hugg i ngface initialiazation of the token embedding, which we reset to a random normal initialization.
Masked language modelling further inhibits the decoder bias strategy discussed in Sec 3.1, as only masked tokens lead to non-positive decoder bias gradient. However, can proceed for masked language models by recovering tokens from the embedding layer as discussed for models without decoder bias. The mask tokens extracted from the bias can later be used to fill masked positions in the input.
E.2 GELU
A minor stumbling for the attacker occurs if the pre-defined model uses GELU (Hendrycks & Gimpel 2016) activation instead of ReLU, This because GELU does threshold activations in the same was as ReLU, and transmits gradient signal when the activation < 0. However, simple workaround for the attacker is to increase the size of the measurement vector and, by doing so, push activations away from O, and thus more toward standard ReLU behavior. For the main-body experiments, we use ReL.U activation for simplicity presentation, using the workaround described above, we find in Figure 14] Figure 15 that the performance of the attack against GELU network is comparable to its level against a ReLU network.
GP1-2
GF12
D71-2
GFF22 (GelU)
GP12 (GeLU)
GPT.2 (Seil)
68
08
0.
12d Acturacy
044
04
0.4
80
00
0.0
Size
Datch
Barchh
Figure 14: Comparison of GELU and ReLU activation with magnified measurement vector (sequence length 256).
19

---

Published as a conference paper at ICLR 2023
10
08-
08-
05
Ttal Accurary
06
06
0.4
04
GPE-
GFT-2
GPT-2
GFT2 (Gell1))
GP72 (GaLU)
GPT2 (GaLl))
80
120
300
420
00
100
200
300
400
500
0.0
300
200
400
520
Geguence Length
Seguente Length
Sequanice Length
Figure 15: Comparison of GELU and ReLU activation with magnified measurement vector (batch size 1).
3.0
08
08
0G
0G
Tandfrmer3 BER1-0002
Tansfore:) BERT-Bose
Tansforer: BERT-Base
02
402
620
3000
00
400
600
800
1000
01
200
400
802
1000
Sequeric Length
ScqutTIU Length
Seauenoe Length
Figure 16: Baseline results for our method for different popular architectures and metrics for variable length sequence imput (batch size 1). Note BERT's positional encoding caps out at sequence length 512, so we end our experiments for BERT there.
E.3 DROPOUT
All experiments in the main body, including comparisons, have assumed that dropout has been turned off for all models under consideration. In standard applications, dropout can be modified from the server-side, even for compiled and otherwise fixed models (ONNX) 2022). In our threat model, dropout hence falls into the category of server-side parameter that malicious update could turn off. We also investigated the performance of the attack if the attacker cannot alter standard dropout parameters, finding that dropout decreased total accuracy of the attack by about 10%, e.g. from 93.36% for a sequence of length 512 on GPT-2 to 81.25% with dropout.
F ADDITIONAL BACKGROUND MATERIAL
The attack TAG in Deng et al. (202T) approaches gradient inversion attacks against transformer models from the direct optimization based angle. This was first suggested in /Zuu et al. (2019), who provide some preliminary experiments on recovery of short sequences from BERT. Basically, the attack works to recover user data (x*,, "u"""") from the measurement of the gradient on this data by solving the optimization problem of
min [|[(n(s, 0).u) - -1l² z.9
(4)
where x and y are the inputs and labels to be recovered, C is the loss and n²-,0) the language model with parameters 0. This optimization approach does succeed for short sequences, but the optimization becomes quickly stuck in local minima and cannot recover the original input data. Zhu et al. (2019) originally propose the usage of an L-BFGS solver to optimize this problem, but this optimizer can often get stuck in local minima. Deng et al. (202T) instead utilize the "BERT" Adam optimizer with hyperparameters as in BERT training (Devlin et al. 2019). They further improve the objective by adding an additional term that especially penalizes gradients of the first layers, which we also implement when running their attack. A major problem for the attacks of Zhu et al. (2019) and Deng et al. (202I), however, is the large label space for next-word prediction. In comparison to binary classification tasks as mainly investigated in Deng al. (2020), the large label space leads to significant uncertainty in the optimization of labels y. which leads their attack to reduce in performance as vocabulary size increases.
We further note that Boenisch et al. (202T) also propose malicious model modifications (as opposed our more realistic malicious parameter modifications) breach privacy in FL. for text, however the
20

---

Published as a conference paper at ICLR 2023
1.0
GFT3
GFE3
GFT3
BORT
06
24-
-04
04-
02
02
02
08
00
Saze
Batch
Banch
Figure 17: Results for our method for different popular architectures and metrics for variable length batch size (sequence length 32).
proposed model in their work is a toy two-layer fully connected model that is not a Transformer- based model. In fact, the strategy employed in their attack cannot be deployed against modern language models that do not construct linear layer of the length of the sequence, aside from handling of facets of modern language models like positional encodings, or attention mechanisms.
G MITIGATION STRATEGIES
As stated in the main body, a central message of our paper is that aggregation alone may not be sufficient to defend against malicious parameter attacks in FL. Further defenses may be needed. Such defenses could include parameter inspection wherein a defender could look for identifiable signatures in the shared parameter vector. For example, the given attack duplicates measurement vector, leading to low rank linear layers. To the best of our knowledge, there do not exist systematic parameter inspection based defenses malicious models in FL. The attacker could, however, easily adapt to some obvious parameter inspection defenses. For example, simply adding a small amount of noise to the measurement vector, and other conspicuous parts of the attack, does not affect the success of the attack significantly, but does immediately make a strategy like rank inspection null and void. We confirm this adaptation by adding small amount of Gaussian noise to each row of the measurement vector in the linear layers (for the Transformer- model, 8 users, 32 sequence length) and find that numerically, these layers do indeed become full rank, and the success of the attack remains largely unchanged (accuracy 90.62%). Because such simple adaptations are easily made, we do not recommend this line of defense.
Instead, we advocate for strong differential privacy on the user side. Such measures could include gradient clipping/noising (c.f...
The immediate impact of differential privacy mechanisms on the attack described in this work is that the measured weight and bias become noisy, so that the estimation in Equation ( becomes subsequently noisy. However, jumps in the cumulative sum is sparse, i.c only number-of-tokens many jumps exist, independent from the number of bins. As such, as long the number of tokens is smaller than the number of bins, the attacker can denoise this measurement, for example using tools from sparse signal recovery such orthogonal matching pursuit (Mallat & Zhang 1993). We use the implementation via K-SVD from Rubinstein et al. (2008) to do so. This way, the attacker reduces the immediate effect of gradient noise, utilizing the redundancy of the measurements of the cumulative sum of embeddings in every bin. Overall, with the above strategy, we find the attack can still succeed in the presence of gradient clipping and noising, although the success does predictably degrade higher noise levels. (see Figure 18
attacker can counteract this simply by noticing that the number of
Furthermore, the imprint strategy, as found in Fowl al. (202)). I), has been shown to be robust to certain amounts of gradient noise, and an attacker could further attempt to implement strategies found in that attack to resist this defense.
Mitigations through local loss audits Moreover, federated learning often takes place in the background, for example, at night when the device is charging (see Hard et al. (2019) and can be regular occurrence on participating devices. Even user applications were modified to make loss values on local data observable to the user, user could not know whether model was malicious, or whether a new model was simply in an early round of its training run. As such, auditing and exposition of local model loss values to the user is not strong defense against the proposed attack.
21

---

Published as a conference paper at ICLR 2023
success against gradient clipping, noising
08
07-
06
05
Tral
0.
10"52
Figure 18: Attack success rate against different levels of added Laplacian noisc. In each experiment, the gradient also clipped to value of We that the attack remain successful even in the presence of added noise. this experiment, a single sequence of length 32 was considered.
H
FURTHER RESULTS
1.0
100
10
GPT-2
Taanformer- BE RT-BBase
0.8
0.8
0.8
0.6
T0t8l Accurncy
0.6
ROUGEL
0.6
0.4
GFT-2
0.2
GPT-2
Parsformen- Bese
Vansforme-3
25
5.0
15
10.0
12.5
15.0
5.0
15
10.0
12.5
15.0
25
5.0
75
10.0
12.5
15.0
Batch Size
Batch
Size
Batch Size
Figure 19: Baseline results for our method for different popular architectures and metrics for variable batch size (sequence length 512).
0.8
Tttel Accurscy
0.6
0.6
SOUGE-L
0.4
0.4
02-
02-
0.0
Figure 20: Comparison to TAG attack for variable batch size (sequence length 32).
In Figure 3. we illustrate the embedding recovery process that we adapt from Fowl et al. (2021) We opt to initialize the rows of the first linear layer in each Transformer block to randomly sampled Gaussian vector (all rows being the same vector). Then, following the calculation of Fowl et al.) (2021), when an incoming embedding vector, for example, corresponding to the word "pepper", is propagated through this layer, the ascending biases threshold values of the inner product between the Gaussian vector, m, and the embedding vector, Because of the ReLU activation, this means that individual embedding vectors can be directly encoded into gradient entries for rows of this lincar layer.
H.1 SENSITIVITY TO THRESHOLD IN TOKEN ESTIMATION
We include an evaluation of the threshold parameter for the embedding- norm token estimation in Figure 21 This value is set 1.5 standard deviations in all other experiments.
22

---

Published as a conference paper at ICLR 2023
0.5
0..
0.4
Experiment
0.2
sdw2.5
16 64 120
255
512
Figure 21: Sensitivity to token estimation threshold. Total accuracy (i.e. percentage of tokens recovered correctly in their correct position) for GPT-2 and varying batch sizes with sequence length 32. We evaluate threshold parameter the token estimation. The default value in remainder of all experiments 1.5, although we find the estimation to be robust to range similar cutoffss.
H.2 TOTAL AMOUNT OF TOKEN LEAKED
Finally, in Figure 22 we include the total amount of tokens leaked correctly through the attack for variety of sequence lengths, batch sizes and models. We that the total amount tokens leaked with GPT-2 has not yet reached a peak, and larger sequences could leak more tokens. For the smaller 3-layer transformer we see that the number of leaked tokens peaks around a batch size of 128 for sequence length of 32, and 8 for sequence length of 512, i.e. around 4096 tokens. This is related to the number of bins in this architecture, which in turn given by the width of all linear layers, leading to bin size of 4608 for the 3-layer transformer. For the small GPT-2 the number of bins is 36864.
Total rumber tokens laaked exeeyy
Datase:
Total rumber of tokens loked exectly
Datase:
stackuverflus
shukzspass
wikinst
shalespoatt
1000
sackmerrlow
rander-ttbee tillens
100
16
128
256
36
32
256
512
Botch Size
Rench Size
Tntal number molent leaked excetty
10k
Dotaset
szckovertio
Taal number of toleess lolled excetly
2000
1800
Daraset
sackovzrflow
wilctbett
1200
1000
600
16
Botch
Figure 22: Total number of tokens leaked in their exact location for various settings (instead of percentages
shown elsewhere). Top row: Total number of token for GPT-2 (left) and the 3-layer transformer (right) for a sequence length 32 and various batch sizes. Bottom row: total number of token for GPT-2 (left) and the 3-layer transformer (right) for sequence length 512 and various batch sizes.
23


Reviews:
[{"review_id": "tojqNAIuaFR", "reviewer": "Reviewer_utwd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an attack that corrupts the transformer models to extract private data from users' updates in federated learning. The threat model assumes that the server can send a malicious update to corrupt the user-side models and then can extract the original words and sentences entered from users' updates. To mount the attack, the adversary needs to be in charge of the server, send one malicious model update to the user side, and then extract private information. ", "summary_of_the_review": "This paper introduces an attack that corrupts the transformer models to extract private data from users' updates in federated learning. The threat model is novel and interesting. Comprehensive details are provided. ", "strength_and_weaknesses": "**Strength:**\n- The paper is well presented, where comprehensive background information is provided. Technical details are well documented and explained.\n- One main contribution of this work is the threat model. As a conceptual contribution, the threat model is well-motivated and explained in detail. The attack scenario, as explained by the authors, is also interesting. However, there is also a concern regarding this threat model. Please check below. \n\n**Weakness:**\n- About the threat model contribution: will the malicious update be obvious to be detected by users themselves? After disabling all attention layers and most outputs of each feed-forward layer, will it be obvious for the users to realize the performance change of the transformer model? And, consequently, will it make the users stop adding new inputs? If so, this raises another relevant question: will the historical data be used to update the gradients? \n- Given the different threat models, the results provided by Figure 9 are interesting but less informative, since the attack budgets are different in TAG and the proposed approach.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qTYUz47IIB", "reviewer": "Reviewer_eNPF", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new data reconstruction attack for text data in federated learning. In the proposed attack, a dishonest server drafts corrupted model parameters and sends them to the clients. The server can reconstruct sequences of text by observing the gradients returned by the clients using the corrupted model parameters. The proposed attack is tailored to small Transformers. Experimental results were conducted on several text datasets, including wikitext, Shakespeare, and StackOverflow.", "summary_of_the_review": "The paper studies a fundamental data reconstruction problem for text data in federated learning. However, there is room for improvement.", "strength_and_weaknesses": "Strength\n\n+ The research topic is critical.\n+ The proposed attack is somewhat new for text data.\n\nDespite an important topic, the paper has several crucial problems, as follows:\n\nWeaknesses\n\n- The federated learning setting in this paper is unrealistic. There is no evidence to support that clients train BERT or Transformers on their local devices for keystroke prediction tasks in real-world applications (deployment of such a setting). Training BERT or Transformers requires computational power on the training devices. It will incur communication costs for the local devices as well. Assuming local devices could be personal computers or mobile devices, how practical is the setting in the real world? Are there any deployed systems for such applications? If this is a cross-silo setting, what is the benefit of using FL compared with centralized training independently among clients?\n\n- The attacks appear to be easily detectable since many model weights are set to either 1 or 0. For instance, the $W_K$ matrix is set to identity 1, $b_K = 0$, $W_ Q = 0$, $W_V=1$, and $b_v = 0$. In the real world, when such conditions are blended into the actual parameter distribution making the attack stealthy? If clients check one of these conditions, they can easily ignore the training round, thus avoiding the attack. That poses a fundamental question: How do we theoretically and empirically understand the stealthiness of the proposed attack? In addition, the attacker updates the model weights to deactivate the attention and feed-forward layers. That introduces a significant gap between model utility before, during, and after the attack. This gap can be visible to the clients to detect the attack. How could the proposed attack avoid or minimize this gap? What is the fundamental advantage to make the attack severe?\n\n- The paper does not provide any empirical or theoretical analysis of the proposed attack against privacy-preserving mechanisms, such as DP and local DP. It is unrealistic to assume that the clients naively follow the protocol set up by the server. Therefore, a data reconstruction attack needs to be evaluated against privacy-preserving mechanisms. The critical point is not the attack but the trade-off between the attack, privacy protection, and model utility. That will inform the practicability of the attack in real-world applications.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "pz9PMD3BvZ", "reviewer": "Reviewer_Fg6a", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel gradient inversion attack against NLP transformer models in a federated learning setting. The attack is constructed around a new threat model that allows the attacker (server) to deploy the model on user devices with poisoned model parameters, which are downloaded from the server and replaced by benign parameters with the next update. This poisoned model update disables the attention layers and (most) FF layers to facilitate the separation and recovery of input tokens. The proposed attack is then evaluated on different transformer-based models of varying model sizes.", "summary_of_the_review": "I like the proposed attack based on the idea of facilitating gradient inversion attacks by applying a malicious update to the victim model. It offers a new perspective to the research in this area and is well presented throughout the paper. I only have small concerns with the rather short interpretation of the evaluation results. Overall, I think the community will benefit if the paper will get accepted.", "strength_and_weaknesses": "Strengths:\n+ clearly written paper, which is easy to follow. Especially the given intuitions and visualizations of the attack are helpful for understanding.\n+ interesting new attack setting (server sends poisoned model weights to facilitate the attack)\n+ approach combines various ideas of previous research but also offers its own improvements\n+ empirical results state strong attack success\n\nWeaknesses:\n- The interpretation of the empirical results could use a little more detail. For example, the paper shows that random data is more vulnerable than natural language inputs. This is a very interesting finding, and discussing possible reasons would improve the paper.\n- I miss some discussion of the limitations of the approach, particularly the fact that the model after the malicious update won't produce any meaningful predictions on the user side (see next section for more details)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PjGEN57b9MK", "reviewer": "Reviewer_yp9z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an attack on federated learning of Transformers which allows recovering the training sequences, based on a threat model in which the server is corrupt and submits a specially designed weight set which, once applied to the gradient computation of training sequences, will store all information necessary for the recovery of the training sequences in the gradient vectors. ", "summary_of_the_review": "The paper is of very good quality and established an important result, with consequences on future research and practical considerations.", "strength_and_weaknesses": "The paper is very strong and convinces through its clarity despite many algorithmic steps requiring the solution of non-trivial problems, as well as the strength of its results: it leaves no doubt that in the absence of any other security measure, federated learning, with models even as complicated as Transformers, widely allows recovering training sequences.\n\nOn principle, it does not come as a surprise that this recovery should be possible; but the practical demonstration of this conjecture, together with a careful analysis, was needed and is provided by this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8EDTZoxoCte", "reviewer": "Reviewer_7oyh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the privacy problem under the setting of federated learning (FL), where a global model is trained by using the gradients computed by each private client. Although clients do not give out their private data but return the gradients computed using their data, the gradients may leak the private training data. The paper proposes an attack method that reconstructs the input text from the gradients. In this approach, the model weights will be reprogrammed such that by using the gradients, the attackers can solve the input text. Experimental results have shown that the proposed attack works well with different models, batch sizes, and sequence lengths.\n", "summary_of_the_review": "In summary, I think this work is novel and solid. I hold a positive attitude towards accepting this paper.", "strength_and_weaknesses": "Strengths:\n- The paper proposes a novel attack method to reconstruct input text from gradients by carefully reprogramming the weights in transformer models. The attack method is interesting and makes sense to me. \n- The paper is well-motivated and well-positioned. The attacking settings and threat models have been discussed well. The experimental results show the vulnerability of transformer models in a practical setting. I believe these results are insightful and are of interest to the NLP privacy/security community. \n- The paper is presented in a good structure. I like the presentation in which a (simple) single-sentence case is presented first and then multi-sentence cases. \n\nWeaknesses:\nI don’t see a crucial weakness; but I would like to make the following suggestions which I think the paper can be improved upon.\n- Figure 3 is not easy to follow, as long as the methods for reducing the probability of a collision.\n- The authors are encouraged to provide more insights based on the experiments. For example, why is the 3-layer transformer more vulnerable?\n- Is “total accuracy” a token-level metric or a sentence-level one?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster