Panning for Gold in Federated Learning: Targeted Text Extraction under Arbitrarily Large-Scale Aggregation
Authors: Hong-Min Chu
Jonas Geiping
Liam H Fowl
Micah Goldblum
Tom Goldstein
Venue: ICLR 2023 poster (2023)
Keywords: Federated Learning
Privacy
Security
Privacy attack

Abstract: As federated learning (FL) matures, privacy attacks against FL systems in turn become more numerous and complex. Attacks on language models have progressed from recovering single sentences in simple classification tasks to recovering larger parts of user data. Current attacks against federated language models are sequence-agnostic and aim to extract as much data as possible from an FL update - often at the expense of fidelity for any particular sequence. Because of this, current attacks fail to extract any meaningful data under large-scale aggregation. In realistic settings, an attacker cares most about a small portion of user data that contains sensitive personal information, for example sequences containing the phrase "my credit card number is ...". In this work, we propose the first attack on FL that achieves targeted extraction of sequences that contain privacy-critical phrases, whereby we employ maliciously modified parameters to allow the transformer itself to filter relevant sequences from aggregated user data and encode them in the gradient update. Our attack can effectively extract sequences of interest even against extremely large-scale aggregation.

Published as a conference paper at ICLR 2023
PANNING FOR GOLD IN FEDERATED LEARNING: TARGETED TEXT EXTRACTION UNDER ARBITRARILY LARGE-SCALE AGGREGATION
Hong-Min Chu Jonas Geiping' Liam Fowl"
Micah Goldblum²
Tom Goldstein
University of Maryland New York University {hmchu, jgeiping, lfowl, tomg]0 Long)ound.cemm @uddd eduu
edu
ABSTRACT
As federated learning (FL) matures, privacy attacks against FL systems in turn become more numerous and complex. Attacks on language models have pro- gressed from recovering single sentences in simple classification tasks to recov- ering larger parts of user data. Current attacks against federated language models are sequence-agnostie and aim to extract as much data as possible from an FL update often the expense of fidelity for any partieular sequence. Because of this, current attacks fail to extract any meaningful data under large-scale aggre- gation. In realistic settings, an attacker cares most about a small portion of user data that contains sensitive personal information, for example sequences contain- ing the phrase "my credit card number is ...". In this work, we propose the first attack on FL that achieves targeted extraction of sequences that contain privacy- critical phrases, whereby we employ maliciously modified parameters to allow the transformer itself to filter relevant sequences from aggregated user data and encode them in the gradient update. Our attack can effectively extract sequences of interest even against extremely large-scale aggregation.
INTRODUCTION
Industrial machine learning models are often trained on large sets of user data. In traditional centralized training paradigm, this is done by aggregating user data into a large repository. Unfor- tunately, when user data contains personal information in the form of text, images, or other media, dataset aggregation leads to significant security, regulatory, and liability risks.
Against this backdrop, federated learning (FL) has emerged decentralized data, that without the need for a central party to host a dataset. By exchanging only model gradients, user devices collaboratively train model without the direct exchange of plaintext data. In many applications, FL. is slower than centralized training (Bonawitz et al.) 2019), but the privacy benefits outweigh the costs, especially in noxt-word text prediction which requires training on private text from smartphones (Hard et al. 2019).
popular way train models with
Privacy through federated learning sometimes taken for granted. In reality, the actual privacy achieved by federated learning systems depends on a large number of factors and parameters- - model size, architecture, number of users, the aggregation scheme, and more. Attacks against privacy in FL probe this boundary, empirically discovering pitfalls that should be considered and avoided when designing federated protocols (Phong al. 2017, Melis al. 2019 Geiping et al. 2020)).
In this work, we study the security of federated learning systems involving transformer architectures (Vaswani al.] (2017) which form the backbone of many recent advancements in natural language processing (Brown et al. 2020 Dosovitskiy et al. 2021] Jumper et al.) (021). and especially ap- plications in text, which represent a key point of interest in many modern applications of federated learning (Paulik et al. 2021 Dimitriadis et al. 2022). Our main threat model of interest is the un- trusted server scenario, also known as the malicious server scenario, in which the server may make changes to model parameters in order to break user privacy. This is in contrast the honest-but curious threat model, in which no malicious changes are permitted to the model training protocol. Untrasted server scenarios are of critical importance from user-centric privacy perspective.

---

Published as a conference paper at ICLR 2023
Input Text Seqs.
Hey do you want me to put the drinks on your
tab
Yeah can you use credit card 4744 67 52 44 18 Oof did you that my woke me up tonight
Tagging through modi fied MHSA
Hey do you want me put drinks on your tab
Yeah y can you use the credit iard 4744 67 88 55,52 44 Oof did tell that my cat woke me up tonight
Filtering through Modified Linear Layers
credit card 4744 67
88 55
52
44
18
Breached Tokens
Figure 1: The proposed attack "tags" and filters tokens that they can be reconstructed from gradient information. Malicious model parameters for a standard transformers (here with causal structure) are sent to user devices. The attack uses one head to fag each token a sequence with the first token of that sequence, (here in red, green, blue). This enables the attacker to group tokens into sequenees after they are extracted from a gradient update. The attacker then uses two more hcads to tag each token that follows the key words "crodit" and "card" (yellow). These yellow target tokens will influence the gradient computation, while others will be filtered out. Finally the attacker recovers the targeted tokens from the gradient of the modified model returned by the user.
If federated system is supposed to uphold privacy, then idcally this privacy can be guaranteed without having to assert full trust in the server. After all, a perfectly trustworthy server could run the simplest FL protocol: Centralize all user data, promise not share it, train centralized model, and delete all data. Another way to look at untrusted server threats is to view them as glimpse of worst- case dataset security. Even if we believe a server will uphold privacy, we might wonder about the worst-case loss in privacy that would arise if this server is even briefly compromised (through either classical security breaches or poisoning attacks (Bagdasaryan et al. 2019)) and acts maliciously.
A major strategy available to malicious server is to modify the current state of machine learning model as it is being trained, and then broadcast this corrupted model to the users. As the model directly executed on each user device, this can be considered an analogue to untrusted code instruc- tions that are being evaluated on user's private data (OWASP 2022] Fowl et al.) 2022).
Despite the inherent power that malicious server has, extracting user data is still extremely difficult when gradients are aggregated over many users, in which case the averaged gradient does not contain enough entries to record the whole global training batch. For this reason, existing attacks on text models only recover user data in scenarios where the number of model parameters is significantly larger than the number of tokens in user update (Fowl et al., 2022]] Gupta et al. [0222 Dimitrov et al. 2022 Pasquini et al., 2021). In some cases, attacks can siphon random examples of user data, but only through large number of repeated queries (Wen et al. 2022.
In this work, we discuss a novel attack on text models whereby a malicious server is able to pick and choose which data to encode and extract from the model gradient, even with industrial-scale aggregation. The attacker selects a frigger phrase, such as "credit card number" or "social security number," and extracts all tokens of user data that follow the occurrence of this trigger. We call this process, in which we sift selected phrases out of large corpus of user data, "panning." In com- parison to existing attacks, this attack does not degrade when very many user updates are securely aggregated (Bonawitz et al. 2017). For this reason, panning is an essential shift in capabilities for attacks against transformer-basee models in federated learning.
BACKGROUND AND APPLICATION EXAMPLES
Text models were the first and most successful systems where federated learning has been used in industrial settings. These applications include keystroke prediction Hard et al. (2019); Ramaswamy ctt al. (019). settings search Bonawitz et al. (2019), news personalization (Paulik et al. (2021). and improved messenger services on Android (Google 2022). In the latter case, the documentation of
2

---

Published as a conference paper at ICLR 2023
Google message describes federated learning being used to improve ML models related to Google messages, for example "smart reply" features. FL is active by default on all Android devices with this app. According to the documentation, user conversations are protected only through secure aggregation (Bonawitz et al.) 2017), which "can't It reveal your conversations or content to Google or anyone else [...], grouping many similar adjustments logether so that Google can't inspect an adjustment from a single device". In this work, we argue that this system of protecting privacy leaves users open to targeted extraction attacks from malicious server update sent to the user, provided the system uses a transformer-bast based machine learning model.
Known attacks against privacy in federated language models are not capable of breaking privacy in such large aggregation settings. The earliest attacks against transformers in ZZuu et al. (2019b); Deng et al. 2 operate the "honest-but-curion only limited success, recovering data only from single sequences. In practice, "single sequences" imply that user updates are not securely aggregated, each individual user sends an individual update that represents only their data. Recent developments in Dimitrov et al. (022) and Gupta et al. (022) have pushed the envelope, reconstructing more and more text fragments, even from multiple sequences in this restricted threat model. On the other side, malicious server threat models, for example in Pasquini et al.) (2021) and et al. (2022) can recover more information and identify or respectively reconstruct from aggregates of hundreds of sequences.
threat model. These attacks have resulted in
These attacks all attempt to recover all user data from an aggregated gradient, putting equal emphasis on the fidelity of each reconstructed sequence. This strategy inevitably decreases the capability of previous attacks to recover accurate individual sequences as the number of sequences and tokens increases, and eventually the extracted sequences become meaningless combinations of tokens.
Attacks that break large scale aggregation generally require additional assumptions. Lam et al. (202I) show that aggregation can be breached side-channel metadata available the server, Pasquini et al. (2021) ) show that if users can be addressed separately by the server and each user only has limited data, malicious parameters with zero-gradee gradigradeents can be used suppress all other users and break aggregation. This trick can also be used for targeted membership inference, checking if a user owns a specific datapoint. Further, |Wen et al. (2022) shows that if malicious servers are allowed to send multiple queries, then they can "fish" for single data points, even from aggregated data.
Yet, a real-life scenario, among all data a user computes and aggregates model updates with, often only a few of them contain information valuable to a potential attacker. For example, out of 50- sentence online conversation between a user and a bank agent, an attacker may only be interested in the particular sentence that contains the user's social security number. This poses the question of whether is possible for an attacker to dedicate the capacity of the gradient to only a limited number of target sequences with specific set of keywords triggers.
Threat Model Untrusted Server Our threat model contains two parties. First, user (or group of aggregated users) that owns text, which contains private information following a set of keywords K = (ki,..., kin, J. Second, malicious server that aims to "pan", e.g., perform targeted extraction of user tokens that immediately follow one of these keywords. We assume that the number of aggregated sequences may be unlimited, but the number of sequences matching the combination of keywords limited. We assume that the federated learning exchange is otherwise secure: Both parties agree beforehand on transparent implementation for both model architecture and user-side protocol that is vetted by public examination. The only attack vector for the server is the model update sent indiscriminately to all users in the group. In this way, the presence of the attack may be non-obvious because it does not require deploying any new executable code, and it is embedded in network parameters that are seldom, if ever, inspected.
Even stronger threat models have also been investigated (Frey 2021 Boenisch et al. 2021 Fowl al.) 2021). Threat models considered in Frey (202I) allow the server execute arbitrary pieces of code on user devices, while Fowl et al. (2021): Boenisch et al.] (2021) assumes a server can maliciously modify model architectures. We also note that the role of the attacker need not be played by the party that owns the server, and any individual/party that has access to the model update anywhere in the pipeline can assume the role. Particularly, man -n--tn---iddlll attacker can inject the attack into model parameters before broadcasting to users, retrieve the resulting unsecured user update, and replace the outgoing update with seemingly normal model gradients.
https:// httpss//support.ggo/ google. http://ggg......oooaaaa////////a
3

---

Published as a conference paper at ICLR 2023
Tagging Module
Filter Module
Capture Embedding Relevant Sequences
Post-Process Into Sequences
Input
Aggregated Model Update
Input/Pos Embed.
Individual Embedding
Recovered Sequences
Encoder Block
Figure An overview of the mechanics of the proposed panning attack. The attacker reprograms the encoder block to tag filter input sequences. This encodes keyword -errd-ipccific information into the model gradient, which enables the attacker to "pan" for targeted tokens from even large aggregated user updates. The attacker can then post-process the tokens retrieved from the model gradient to recover original sequences.
Remark (Peer-to-Peer Attacks between Users.). This threat model applies even if the server is fully benign: If in any round of training, a group of malicious users can control enough updates to replace the model parameters (Bagdasaryan al. 2019 Bhagoji et al.) (019) in single round of training, then the server will unwittingly distribute these malicious model parameters in the next round of training, collect model updates from other, vulnerable users, aggregate them and redistribute the updated model. Because an attack through "panning" " is invariant to aggregation sizes, this means that the group of attackers can then discover data from valnerable users in the next round of training, while the server remains unaware of any breach privacy.
METHOD
Here we discuss the possibility of a"panning" attack that performs extraction of target sequences that contain a specific set of keywords. The attack describe below is specific to transformer architectures and based on modifications to attention mechanisms and linear layers. The key ideas can be summarized follows:
1. Modification of multi-head attention blocks (MHSA) to "tag" the tokens of a sequence that contains specific keywords, in Section 3.2
2. Modification of linear layers detect the tag and trigger capture of token embeddings belonging to target sequences, while filtering out non-target sequences, see Section 3.3
3. Post-processing the relevant embeddings into target sequences, as Section 3.4
An overview of our attack is presented in Figure 2 describing "analytic" attacks against models federated learning that we describe in Section 3.1
The discussed attack draws family of works
3.1 PRELIMINARIES
As discussed in Phong et al. (2017): Geiping al. (2020) Fowl al. (221), the gradient of linear layer parameterized by ( (W, b) whose forward pass is W ʃ + b reveals information about its input f. In particular, for any given row W., and the related bias entry bm. we have
V V L OL
ØL
ØL
aL 0ym aL
V Um 8ym
f,
(1)
Dym
Dbm
8ym
Ibm Bym
where L(y) denotes the downstream loss computed the layer output y. An attacker can then simply reconstruct f by computing V.. Bb
aL where @ denotes entry-wise division as.
However, attackers cannot easily apply this trick to transformers in federated learning. Firstly, trans- formers add together token embeddings and positional embeddings, and have residual connections that add the output of multi- head self-attentios (MHSA) to these mixed embeddings before applying the first fully-connected linear layer. other words, the individual input ft,; of the first linear layer of the first encoder block is
JrJ = Gijj + + MHSA((C;,) + + 1
(2)
where eij is the embedding of the j-th token in the i-th sequence, P, is the j-th positional embed- ding, and MHSA( (eig py ) j) is the j-th entry after applying MHSA on sequence i. Secondly,
4

---

Published as a conference paper at ICLR 2023
assuming the model update is computed over B sequences of length f, an attacker only has access to the average gradient aggregated over all Bi tokens, The aggregated gradients of the first linear layer are instead
B ΣΣ Vw..
aL Tym
B
f
aLid
دماة
Wm Li
f.s),
(3)
3b
dym
j=1
Recovering an individual Jij from this gradient cocktail is non-trivial.
In summary, to recover a user's imput sequences, an attacker needs (1) isolate out individual mixed embedding fi.j (2) split the mixed embedding ftj into token embedding Gi and position Pi. and (3) match ex,j to the most probable word. all tokens are uniformly represented the gradient, then step (1) may be impossible. Below, we show how the network can be put into state where the gradient's s information capacity is sparingly used to encode targeted sequences.
To extract data from transformer model update, the attacker needs to first extract individual mixed embeddings from aggregated (batch averaged) gradients, group these into sentences, and then split the mixed embeddings into their position and token.
Big Picture:
3.2 TAGGING TARGET SEQUENCES
In this section, we describe how an attacker could modify parameters in the attention block to "tag" the mixed embedding that belongs to a sequence that contains the selected keywords We illustrate the modification in Figure 3 For simplicity, we start by detailing
K = [k,,...,kw' this construction with only one keyword k. We refer to a sequence that contains & as either the "target" sequence or "relevant" sequence. Other sequences are called mnontargeeedd or "inrelvant."
To facilitate the construction, we assume the norm of token embedding e(() associated with the target word & to be larger than the norm of all other token and positional embeddings. The server can easily meet this assumption as both embedding matrices are also part of the model parameters under their control. With this assumption, we design a self-attention mechanism inspired by Fowl et al. (2022) that tags the target sequences by imprinting ((k) into every mixed embedding in the target sequence. To be more specific, let (Wa, q: bo), (Wx K, bx), (Wv, by) be the weight matrices and biases of query, key, and value layers of MSHA in the first encoder block respectively. We set each of the parameters follows,
We = O,
bQ = ar (R), (R)
f - otherwise.
W K La.
bx 0,
To Zala, r]
(4)
W = Te,
by = 0,
where a is large positive value (e.g. 108) and Ta is the truncated identity matrix with pre-defined dimension d'. W and by form an identity operator, and thus the key token K (and others) pass through unaltered. We Q and bę are designed so that the query vectors Q are each identical copies of (R)) Finally, Wy and by keep only the first d' entries of the input embedding, and set all other positions in each embedding to zero.
Let's evaluate the results the MHSA computation based on this construction. a length-l se- QKT quence { contains the target i at position j', then the attention weight softmax( Va will be large. forcing every word in the sentence to attend the target word. In fact, when is large enough, attention to the target is abar-absolute. Thus, the resulting mixed embedding after the MHSA is:
f = + + Txr (e + p
In plain language, the first d' entries of the keyword embedding vector added to the first d' posi- tions of every embedding in the sequence. We say that the sequence is now tagged with the keyword. For causal language models, applying the construction in Equation (10) on masked MHSA turns the keyword into a trigger, where the attention imprints (()) to tokens that appear after the keyword in the target sequences. Coupling this tagging approach with a filter introduced later in Section one can perform targeted extraction of sub-sequences that start with the keyword k. On the other hand,
5

---

Published as a conference paper at ICLR 2023
1e4
Malicious V Malicious from Sec. 3.2
relevant embed.
from Sec. 3.3
2.0
irrelevant embed.
Linear Layers
Count
1.5
Input
MHA
Decoder
1.0
0.5
Encoder Block
0.0
Figure 3: An illustration of our key modifications. We modify the query. key and value layer weights in the MHSA block to tag the mixed embedding of target sequences by content and position. We modify the feed forward layer weights to bin tagged tokens into separate rows of the weight matrices W and in this manner filter for only targeted tokens. ① repre- sents the add-and-norm operation in transformer.
4 6
Measurment
Figure 4: The measurement distributions of mixed embedding by + w' from Section 3.3 on GPT- 2. The measurement distribution of irrelevant, non- target, embeddings (blue) exhibits clear separation from the measurement distribution of relevant target embeddings (orange). We set 3 2.5 here.
if the sequence is irrelevant, or in the case of causal MHSA, if the mixed embedding appears before k. then the imprinted embedding will be random from 61 to eg. This difference allows our malicious modification to tag the mixed embedding in target sequences by a eeyyorddspechieee imprint, and we will discuss the next section how we filter out irrelevant sequences based on this imprint.
We note that extending the above construction to multiple keywords is straightforward. In particular, for N keywords ....... .......,.. Ni one can simply repeat the above constructions for a separate head the MHSA block for each keyword, and imprint each keyword-specine signature to different positions.
The attention mechanism of a transformer is repurposed to attend strongly to the attacker's keywords, and add these keyword embeddings to all other entries in the sequences, thereby tagging them as important.
Big Picture:
3.3 FILTERING NON TORGEEEE TOKENS
Targeted tokens are those appearing in a relevant sequence containing a keyword. All other tokens in other sequences are not targets and should be filtered out. We want to remove them from the gradient aggregation to enable the attacker to isolate the few remaining target tokens in the gradient. Our approach draws inspiration from Fowl ett al. (2021), who performs gradient separation on aggregated gradients uncovered in Equation (3) We construct malicious linear "imprint" layer by assigning each row of W identical copies of measurement vector w. We then set the entries of the bias vector b to form an increasing sequence If the linear layer followed by ReLU activation, the aggregated gradient becomes
We provide an illustration of the key idea in Figure
6m b
Www. Lij
ΣΣ i=1==1
Im (fis)
OL Sta), Pym
LLij ab.n
ΣΣ (gm (Fia)
OLi.j Pym
(6)
=1
==
i=i==
where Im (Jiz) 1 (W, + bm 0 and 9m (ft.) 0 otherwise.
Intuitively, the ReLU outputs from this layer are like a thermometer for measuring the inner product increasing, more ReLUs will be active if the value of W Wfij is larger. The m-th ReLU turns on when (w, Si, >) lies above the cutoff defined by bm... The gradient for the row W., aggregates over all tokens with inner products above bm
(w, fij >) Because the bias entries
Now suppose that just one token I' out of all ftz uniquely satisfies gm (f') = and gm+)(()) 0.
Then we separate out the individual gradient for this token by computing
aL'
aL
aL )
ØL'
VW Lis V Vmm4s Li.)
Dym
abm
0bm
Fym
We uniquely recover fl just did from with batch size of in Equation ( There- fore, if the attacker can accurately estimate the distribution of (wfrs) they can then construct accordingly maximize the chance for every ft, to fall uniquely in one of the bins (bm, bm.+).

---

Published as a conference paper at ICLR 2023
It was noted by Fowl et 2022) that the distribution of (w, (ei, + Pÿ)) is approximately Gaussian if every entry of w IS a random sample from a standard Gaussian. Assuming b has M entries in total, we can optimize the attack by setting 6m + { (12 where 4b (b) is the inverse of estimated Gaussian distribution. Under this choice, each bin (bm, bm+1) approximately contains the same probably mass. However, observe that the number of bins upper-hounds the capability of an attacker to recover user data; if too many samples pass through the bins will become over- saturated. Therefore, directly attempting to recover all fag becomes less effective against large-scale aggregation.
To address the issue, we propose a set of modifications for linear layers to pan for relevant mixed embeddings. Here, we describe how to program the first linear layer of the first encoder block, and discuss the extension to multiple encoder blocks in Appendix [A] The key idea to design w so that targeted tokens produce large value of (w, f') while untargeted tokens produce small value. We then assign entries of b so that ReL.U only activate for tokens in the target distribution. To be more concrete, we scale the norm of e(*), the embedding of the given keyword, to B > where controls the "signal strength" of the relevant mixed embedding, and helps to separate the irrelevant embedding as detailed later. We also sample the random measurement vector w from standard Gaussian and mask out the first d' entries. Finally, we construct w' as
[al
q sdd
w'uul = 10,
q >d''.
By setting each row of W to (w + w'), we have
(w + w', .b) (w w' ,ey + p2} + (w', (ey + +p)))
by Eq. 5 where j' is the position the MHSA attends to. Assuming ej and P/ are both zero-mean (these can be chosen at will by the server), the mcasurement distribution of irrelevant embeddings remains zero-mean, while mean shift of (33l w/1) can be observed in the distribution of the mixed target embeddings. We show the separation between both distributions empirically in Figure The distinction allows of relevant target embeddings. This allows the attack to condense all other embeddings into a small number of bins and dedicate the remaining bins to separating individual target embeddings.
adapt the binning strategy from Foom et al. (0222) focus the distribution
Big Picture: The attack leads to bimodal distribution of embeddings. Targeted embeddings produce large activations in the linear imprint layer, while untargeted embeddings produce low activations.
3.4 POST-PROCESSING MIXED EMBEDDINGS INTO MEANINGFUL SEQUENCES
With individual mixed embeddings fú,j isolated into separate ReLU bins, then need to extract the corresponding sequence index i, position index j and the actual word from the vocabulary for each Jt.j- The tagging we describe above is used to label tokens based on the keyword that made them a target Section 3.2 We can also introduce a second tagging mechanism that imprints positional information onto each token.
To be more concrete, we construct a positional imprint using a second head in the MHSA block:
Wq O,
bq apu, by = 0, Tala = { -=r,d d' < S 2d'
Wx = La.
otherwise.
Wv Te.
by 0.
For given sequence (f11ا1 with mixed-in embeddings (e/1f-1, the positional imprint outputs f, =; P3 + Tar + pı),
(9)
where entries d+ 1 to 2d of the first token in every sequence are imprinted to mixed embeddingin the same sequence. We then group fú, into sequences with constrained K-means clustering (Bradley et al. 2000) using entries + to 2d that encode sentence identity. Constrained K K-means allows us to constraint the size of each cluster, which is no larger than sequence length é.
After grouping embeddings into sequences, the next step is to recover positions. As mixed embed- dings in the same sequences guaranteed to have distinct positions, perform position recovery
7

---

Published as a conference paper at ICLR 2023
1.00
1.00
1.00
0.75
0.75
0.75
Total
0.50
Doagee
0.25
0.25
Durs GPT-2
0.00
Durs GPP-2
0:25
QUIS.BERT
0.00
DetMprGPT-2
-025
0.00
Decap:227112
Decepi
-0.22 25
50
100
150
200
250
0.50
50
100
150
200
250
-0.25
50
100
150
200
250
Batch Size
Batch Size
Batch Size
Figure 5: Comparison of method with Decepticons, a recent Transformer-base attack, for different chitectures across various batch sizes on targeted sequenees. Decepticons is depicted with dashed lines, The sequence length is fixed to 32. Panning is a strictly stronger attack against sequences containing the target word.
basis. For set of mixed embeddings {ffr} grouped in the same sequence, identifying the most suitable unique position j of each f can be viewed as a maximum weight bipartite matching problem, where we define the weight of a potential match (fz,) j) to be the correlation between fy' and Py- We can solve the matching problem efficiently (Kuhn 1955 and obtain the pure token embedding Ci.j by subtracting the recovered positional embedding from fi..
on geguence-by-sequeeee
The final step is to associate each recovered token embedding Ei, with the actual words. (Fowl al., 2022) discovered that the frequency of tokens presented in user updates can be estimated accurately. The estimated frequency dictates the maximum number of Ci,j a specific word can be associated with. Then, taking the constraint into account, matching Ci.j to the most suitable word can be formulated as another maximum weight bipartite matching problem, where the weight of a potential match (es.g:0 v) is the correlation between fi, and ey-
Big Picture: Individually recovered mixed embeddings (which are the sum of a position embedding and token embedding) can be grouped by sequence. The tokens in a sequence can then be separated into token ID and position ID) by solving bipartite matching problems.
4 EMPIRICAL EVALUATION OF THE ATTACK
In this section, we empirically evaluate our proposed attack different transformer architectures commonly used in real-world applications. In particular, we consider the smallest variant of GPT- 2 (Radford et al.) 2019) with 124 million parameters, which is causal language model with masked self-attention. We also consider a variant of BERT (Devlin et al. 2019) with 110 million parameters, which was used in previous works (Deng GPT-2, BERT is trained as a masked language model which allows bidirectional self-attention.
2021 Zhu et al. 2019a) for attack evaluation. Unlike
We evaluate on the wikitexi dataset (Merity et al. 2016),, which we partition into articles. We sim- ulate each user by sampling text from an article. We focus on fedSGD, where each user performs a single gradient step and sends a model update to the server. Note that many related protocols, such as federated averaging (McMahan et al. 2017), can be converted back to fedSGD, if the server is mali- cious and can control the number of local update steps or local batch size. We perform quantitative evaluations our method with range of different metrics. We measure success based BLEU scores (Papineni et al.) 2002). ROUGE-L (Lin 22044), and total accuracy (where we only count exact matches of both token ID and positions ID). We provide additional details in Appendix
4.1 COMPARING WITH TRANSFORMER-BASER ATTACKS FOR MALICIOUS SERVERS
We begin by comparing our proposed method with Decepticons (Foll et al. 2022), recent transformer-based attack for the malicious server threat model, across various batch size and se- quence lengths. To simulate the scenario where only a small number of sequences contain privacy- critical information, we select the first 3 sequences from each batch and replace one of the tokens of each sequence with target keyword. For experiments on BERT, the replaced token is ran- domly selected. For experiments on GPT-2, we instead replace one of the first four tokens with a trigger keyword, which allows to assess the impact of imperfect imprints created by masked self- attention. Due to causal language modeling, an attacker can only recover tokens following the trigger keywords for GPT-2. The quantitative results then evaluated only these target sequences.
8

---

Published as a conference paper at ICLR 2023
1.0
1.0
1.0
Totall Accuracy
0.8
Souge
0.6
Our::GTT OUrS
OPTS
Ours:GFT
Ours BERT
Ours.BERT
1003
1500
2300
1000 Batch Size
1500
2000
500
1000
1500
2000
Batch Size
Betch
Size
attack across different architectures for large batch size. The sequence length is fixed to 32. Wc evaluate our method only on the target sequences. Panning attack success remains constant as batch sizes increases to even large aggregation sizes.
Figure 6: Experiment
1.0
Total Accuracy
Rouge
0.4
Ours: Ours:BERI
GPT2
Ours:GME2
Ours:BERI
of Knywords
Keywords
Knywords
Figure 7: Experiment attack across different architectures for target phrases with different length. We fix the sequence length 32 and batch size 64 respectively. Our method able to accurately capture relevant sequences given target phrases of various lengths.
We first fix the sequence length, and compare our method with Decepticons, while varying batch size Figure We see that the performance of Decepticons drops significantly as the batch size increases beyond 128, however the proposed panning attack mechanism is able to maintain steady reconstruction quality on the targeted sequences as the batch size increases. this sense, the new attack really is independent of aggregation size, and rather only depends on the number of keyword occurrences. We push this further in the next section.
4.2 EXPERIMENTS WITH LARGE-SCALE AGGREGATION AND MULTIPLE KEYWORDS
Next, we evaluate our proposed methods under more extreme conditions. First, we assess the capa- bility of our method capture relevant sequences under extremely large-scale aggregation. We fix the sequence length to 32, and experiments with batch size up to 2048, and summarize the results in Figure As demonstrated in the figure, the fidelity of our recovered relevant sequences remains high and is still largely agnostic to batch size. This observation validates the danger of the tagging and filtering strategy, which can capture tokens targeted sequences even against extremely large-scale aggregation. We note that at this level, it becomes infeasible to even compare to Decepticons, due to the increased computational complexity to run the algorithm proposed therein for 65536 tokens.
Additionally, we investigate how the number of keywords K impacts the performance of our attack. We perform experiments on K E (1, 2, 3,44, 5, 6), We randomly replace length- sub-sequence with a length-K K target phrase in each of the first 3 sentences. We present results in Figure 7 and demonstrate that our attack able to recover sequences relevant to a various number of keywords. We note that both GPT-2 and BERT include 12 heads and allow to target 11 keywords, which is enough to target most sensitive information in the real world.
CONCLUSION
In this paper, we describe a novel vulnerability for transformers used in a federated setting. This vulnerability opens the door to an attack through malicious parameter vectors that can "pan" for sequences that contain private information based on specified keywords. The attack allows an ad- versary to accurately capture sentences that contain the keywords out of aggregated updates from thousands of sequences. The attack injects malicious parameters into the transformer whereby sensi- tive sequences "tagged" with unique signature, and filtered to recover these targeted sequences. This attack reflects a notable shift in the capabilities of data reconstruction attacks in federated learn- ing the attack succeeds under seemingly arbitrary amounts of aggregation.

---

Published as a conference paper at ICLR 2023
ETHICS AND MITIGATIONS STATEMENT
We describe an attack that significantly extends the capabilities of data extraction attacks for trans- former models in federated learning. However, this attack only extends the capabilities of a mali- cious server and furthers our understanding that privacy cannot be guaranteed against such a server with only secure aggregation. Yet, in examples such the message application discussed in Sec- tion 2] the described attack in this paper does not actually extend the capabilities of the server, as the federated learning system considered therein is implemented ability for a user to inspect and vet the implementation of the protocol. From a security perspective, the server is so far not constrained to protocol, and is already capable of downloading user data without federated learning. Both with and without our attack, the server in this example remains with current implementations only legally bounded minimize their use of user data.
good-will basis without the
In terms of mitigation opportunities, two lines of defense algorithms are possible. The first line of defense algorithms involves (automatic) inspection of model parameters, and the second line of algorithms leverages differential privacy. We provide discussions for each separately.
Mitigations through Inspection The vulnerability induced via malicious modifications of trans- former parameters, as discussed in this work exhibits identifiable characteristics. Yet, there are several problems with relying parameter inspection defense to this attack. First, awareness of this type attack is necessary to know what to look for in terms of "conspicuous" parameters, which we hope to provide with this work.
Second, parameter inspection, at least on the server side, is not implemented in any major FL frame- work (see [Bonawitz al. (2021): Dimitriadis et al. (2022). ). As such, the point of even being able to inspect parameters on a user side would require a user to have "rooted" their device - making this defense currently infeasible for the average concerned about privacy.
Finally, inspection defenses naturally put users on the "back foot" in a back-and-forde between ever-adapting attackers and constantly defending users. Both parties need agree about the imple- mentation of all defenses, so that the defenses used on the user side are always known to the attacker, iee the server. On the other hand though, knowledge of attacks limited until such vulnerabilities are published.
Because of the asymmetry induced by these sorts of defenses, we strongly advise against relying only on fixed sets of parameter inspection rules to avoid attacks like this.
Mitigations through Differential Privacy We argue that strong user-level differential privacy, i.e. mechanisms applied directly on the user device to all outgoing model updates, remains the defense of choice against the attacks in the untrusted malicious server threat model that consider here. Common differential- siaital---vvcy-bassseee defense algorithms involve gradient noising and clipping, and provide additional experiments to evaluate the effectiveness of these defenses Appendix C
REPRODUCIBILITY STATEMENT
We perform all our experiments based on the official implementation provided in Fowl et al. (0022) We provide details of parameters and evaluations for our experiments, well as pre-processing steps for the wikitext dataset in Appendix A
ACKNOWLEDGMENTS
This work is supported by ONR MURI program, DARPA GARD (HR001 1202001), and the Na- tional Science Foundation (II--2222122 and DMS- Further support was provided by Cap- ital One Bank.
REFERENCES
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How To Backdoor Federated Learning. arXiv:1807.0045 00459 [cs], August 2019. URL httt:://axxxiii
arrg/abs/1801 00459
10

---

Published as a conference paper at ICLR 2023
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing Federated Learning through an Adversarial Lens. In Proceedings of the 36th International Conference on Machine Learning. pp. 634 643. PMLR, May 2019. URL https:t:/pppcceeeaaiiiiee .ml.. preas/v97/bhage 11 9a html
Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. When the curious abandon honesty: Federated learning is not private. arXiv preprint arXiv:2112.02918 2021.
Kallista Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sar- var Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical Secure Aggregation for Privacy Preserving Machine Learning. Technical Report 281, 2017. URL http: //pprint///t.../// iacr.
org/2017/281
Kallista Bonawitz, Hubert Eichner. Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečný, Stefano Mazzocchi, Н. Brendan McMahan, Timon Van Overveldt, David Petrou, Daniel Ramage, and Jason Roselander. Towards Federated Learning at Scale: System Design. arXiv:1902.01046 [cs, stat], March 2019. URL http:
//a //arxiv. www.iv..o//abaa.ooooo /0tp:/9922..000 01046
Kallista Bonawitz, Peter Kairouz, Brendan McMahan, and Daniel Ramage. Federated Learning and Privacy:
Building privacy-preserving systems for machine learning and data science on decen-
tralized data. Queue, Page) 40:87-Pages 40:1 October 2021. ISSN 1542-7730, doi:
10. 00.1145734443343550444
hotL:tt:: stttps://oo/ //w.///o////////////////. www.oogg//oo.. 1etp///....111/311 34 444834 3500240
Paul S. Bradley, Kristin P. Bennett, and Ayhan Demiriz. Constrained k-means clustering. Microsoft Research, Redmond, 20(0),,, 2000.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020), December 2020. URL atttp://axxin org/ abs 2005 14165
Jieren Deng. Yijue Wang, Ji Li, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. TAG: Gradient Attack on Transformer-based Language Models. arXiv:2103.06811 [cs], September 2021. URL
06819
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019.
URL http:/ttp//xii...////p absh/bb//1/ http://aaaaa...orgaaa////..0.000
Dimitrios Dimitriadis, Mirian Hipolito Garcia, Daniel Madrigal Diaz, Andre Manoel, and Robert Sim. FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulations.
(378((cs), March 2022. URL httppp//aaa..l...rg/////////////
13789
Dimitar I. Dimitrov, Mislav Balunović, Nikola Jovanović, and Martin Vechev. LAMP: Extracting Text from Gradients with Language Model Priors. February 2022. URL https://arxiv.
org/abs/ 2202. 08827v1
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. arXiv:2010.1199 19299 [cs], June 2021. URL /URL:/ttpp///aaav.oott///// http:/abbb00001199999
Liam Fowl, Jonas Geiping, Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models. arXiv:2201.12675 [cs]. January 2022. URL http: /ttp:v////vvv..///a owg/oo///...... /bs/
2201. 12675
11

---

Published as a conference paper at ICLR 2023
Liam H. Fowl, Jonas Geiping, Wojeiech Czaja, Micah Goldblum, and Tom Goldstein. Rob- bing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Models. In International Conference on Learning Representations, September 2021. URL https:
7 /pooerevveew net. formm? de (ewrmmm?ddfwwZuoOFFFS5
Suzanne Frey. ber 2021. introoddiag--attoodd-----aat---e
Introducing
Android's
Private Compute Services, Septem-
URL
http://securitty.ppollooooo/ /ttp:o/leello..coo/222/////1//11 wtwpco2//9/2////////
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Invert-
ing Gradients How easy is to break privacy in federated learning?
In
Advances in Neural Information Processing Systems, volume 33. December 2020.
URL https: / htttps://proceediggsss.o http://wwww.......o...../////a/a
c (lede5bbdd95e 1at.5550008811111100000111111111
html
Privacy Policy Google.
How Messages improves suggestions with federated tech-
nology Messages Help. 2022.
URL
com/
wtt.aaaaaccanee/2222222oooorrorn 2Chow-
Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recovering
Private Text in Federated Learning of Language Models. arxiv:2205.000 205522000008544/// May 2022. doi: 10.48550/arXiv.2225.0553 085144. https///////tpp//////oooor///aa/
08514
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated Learning for Mobile Keyboard Prediction. arXiv:/811.0366 [cs], February 2019. URL http://arxiv..ot
abs/1811.0360
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, (96(7787)) August 2021. ISSN 1476-4687, doi: 10. 10.1008//41586-00100000 02886-021-0389-22 Utthttps:: www www com artaaiiiico 841586 021 03819-2
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Maximilian Lam, Gu- Yeon Wei, D. Brooks, V. Reddi, and M. Mitzenmacher. Gradient Disaggrega- tion: Breaking Privacy in Federated Learning by Reconstructing the User Participant Matrix. In ICMI, 2021.
Chin-Yers Lin. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summa- rization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL wttpt:::://.. http::::///aaaaahhhhocoooooo
org/ 401-1013
H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera Learning of Deep Networks from Decentralized Data. 1602:2160..05629 [cs], February 2017. UtL.htt:: http::///awwxi..gaa/b////////
y Arcas. Communication
05629
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting Unin- tended Feature Leakage in Collaborative Learning. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 691-706, May 2019, doi: 10009//......00029999
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. November 2016. BtK./tttts:///peereeee httpr//rrrreereeeeeeooooooooooaa
Foundation OWASP. Deserialization of untrusted data, May URL http:uRRttttts////aaaaae
htwp:iion_____uttusssde_ed_ www.community//ollneenee / wwt.riiiiiiitiiooo iwwaiin_of_uuuuuuu
12

---

Published as a conference paper at ICLR 2023
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A Method for Au-
tomatic Evaluation of Machine Translation. In Proceedings of the 40th Anmual Meeting of
the Association for Computational Linguistics, pp. 311-318, Philadelphia. Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 15/11//00033330731111 URL wtt.aaaaw...ooooooooooo/o-----0-
Dario Pasquini, Danilo Francati, and Giuseppe Ateniese. Eluding Secure Aggregation in Federated
Learning via Model Inconsistency. arXiv:2111.07380 [cs], December 2021. URL http://
wexiv.orgio htth//bb/21111... 07380
Matthias Paulik, Matt Seigel, Henry Mason, Dominic Telaar, Joris Kluivers, Rogier van Dalen, Chi Wai Lau, Luke Carlson, Filip Granqvist, Chris Vandevelde, Sudeep Agarwal, Julien Freudi- ger, Andrew Byde, Abhishek Bhowmick, Gaurav Kapoor, Si Beaumont, Åine Cahill, Dominic Hughes, Omid Javidbakht, Fei Dong, Rehan Rishi, and Stanley Hung. Federated Evaluation and Tuning for On-Device Personalization: System Design & Applications. arXiv:2102.08503, February 2021. doi: 10445500/axx222200.00500 URL httph:t/////// www.ai.wwww.aaaa...ooooaaaa 08503v1
Le Trieu Phong. Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy- Preserving Deep Learning: Revisited and Enhanced. In Lynn Batten, Dong Seong Kim, Xuyun Zhang, and Gang Li (eds.), Applications and Techniques in Information Security, Communica- tions in Computer and Information Science, pp. 100-110, Singapore, 2017. Springer. ISBN doi: 10. 11119/777--11---4422
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAl, pp. 24, 2019.
Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Françoise Beaufays. Federated Learning for Emoji Prediction in a Mobile Keyboard. 19X.v:1906.04329 [cs], June 2019. URL http:
wawxiv.orq/aaaaa. 190. 04329
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv:1706.03762 [cs], De- cember 2017. URL
1106. 03762
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H. Brendan McMahan, Blaise Aguera Ar- cas, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, Suhas Diggavi, Hubert Eichner, Advait Gadhikar, Zachary Garrett, Antonious M. Girgis, Filip Hanzely, Andrew Hard, Chaoyang He, Samuel Horvath, Zhouyuan Huo, Alex Ingerman, Martin Jaggi. Tara Javidi, Peter Kairouz, Satyen Kale, Sai Praneeth Karimireddy, Jakub Konecny, Sanmi Koyejo, Tian Li, Luyang Liu, Mehryar Mohri, Hang Qi, Sashank Reddi, Peter Richtarik, Karan Singhal. Virginia Smith, Mahdi Soltanolkotabi. Weikang Song. Ananda Theertha Suresh, Sebas- tian U. Stich, Ameet Talwalkar, Hongyi Wang, Blake Woodworth, Shanshan Wu, Felix X. Yu, Honglin Yuan, Manzil Zaheer, Mi Zhang, Tong Zhang, Chunxiang Zheng, Chen Zhu, and Wen- nan Zhu. A Field Guide to Federated Optimization. arXiv:2107.06917 [cs], July 2021. URL sttp://arxiss org/ abs 2107. 06917
Yuxin Wen, Jonas A. Geiping, Liam Fowl, Micah Goldblum, and Tom Goldstein. Fishing for User Data in Large-Batch Batch Federated Learning via Gradient Magnification. Proceedings of the 39th International Conference Machine Learning, pp.
PMLR, June 2022. URL
mlr pwwss/vissvve.. wtt html
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural infor- mation processing systems, 32, 2019a.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep Leakage from Gradients. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'stextquotesingle Alché-Buc, E. Fox, and R. Gameett (eds.), Advances in Neural Information Processing Systems 32, pp. 14774 14784. Curran Associates, Inc., 2019b. URL http::/ nips cc/paper/ 961 (617-deep- 1617-deep-leakage eakage- from- wowooooaadddinneee wew.d.dtttt.............. .ddf
13

---

Published as a conference paper at ICLR 2023
A ADDITIONAL DETAILS
In this section, we first describe how to reprogram multiple linear layers as hinted in Section 3.3 We then discuss variants of our construction for MHSA and linear layers that improve performance practice.
A.1 EXTENSION TO MULTIPLE LINEAR LAYERS
Linear layers encoder blocks other than the first can provide more bins for finer-grain gradient separation. Assuming we have H encoder blocks, We can apply the same construction technique for every W(h) of the first linear layer of the h-th encoder block. For bias, we instead compute (h) -- (" ), where essentially concatenate all the bias vectors, assign bin thresholds on the concatenated bias with HM entries, and separate back to corresponding linear layers.
--
Additionally, need to disable MHSA blocks other than the first setting the output to zero, and also partially disable the ford-dorward block of each encoder block by setting all output entries except for the last to Note that the output of feed-forward cannot be entirely disabled, the gradient to the first linear layer will become zero. These additional treatments allow inputs of each corr-forward block stay the same except for the last entry through residual connection.
A.2 ALTERNATIVE CONSTRUCTION OF MHSA BLOCK AND LINEAR LAYER
Here provide alternative construction of the MHSA block. In particular, instead the following,
WQ 0,
bq ac(4) bạ = 0, by =
{ q' d' otherwise.
W = La.
Tw [g
(10)
Wy Te
where the key difference is that the new Tr is instead truncated shift matrix that shifts the entries d' + to 2d' of the imput to the first d' entries. By additionally setting the first d' entrics of each token embedding and position embedding to be zero, the resulting mixed embedding fa, from Equation (2) will have their first d' entries contain only the imprint information. On the other hand, the original mixture of imprint and original token/positional embedding, To
faj whose first d' entries apply this construction we also need w' in Section 3.3 as w' Ta (k) instead. Comparing to the original construction, this alternative provides better performance in practice.
B ADDITIONAL EXPERIMENT DETAILS
In this section provide additional details for our experiments. We start by detailing hyperparam- eters. We set a = 10 for GPT-2 and a = 10 for BERT across all experiments. For GPT-2, we set 10 for GPT-2 if batch size is smaller than 256 and
3 for GPT-2 if otherwise. For BERT, 5 for experiments on multiple keywords, where is the number of keywords. We d', the block size be truncated and shifted as described in Section 3.2 to 32.
we set B = 10 for all single keyword experiments, and we set 8 = K
Next, we describe our procedure to process articles into sequences of tokens for each user. For experiments on GPT-2, the articles are tokenized with GPT-2 (BPE) tokenizer. For experiments on BERT, we tokenize the articles with the original BERT tokenizer. Then, given sequence length f, concatenate all the words in article into array, and partition the array into sequences of length f. The left-over words are discarded. Then, given the batch size B, we keep the first B sequences for each user and discard the rest. The user then computes their model update based the remaining sequences.
For each quantitative metric, the reported result is obtained over the average the first 10 users with enough data. That is, for given combination batch size and sequence length, a user only contributes to the final result if his/her data is enough to fill in the required number of tokens.
14

---

Published as a conference paper at ICLR 2023
1.0
0.8
Total Accuracy
0.6
0.4
0.2
0.0
-0.2
-10
8
-7
-6
Log Noise Scale
Figure Experiments on 3-layer transformer with different noise scale. x-axis represents value after taking base-10 log on the noise scale.
In this work we focus fedSGD as a template for federated learning. However, this restriction is not as limiting as it might seem: A malicious server can side-step the restriction of FedAVG if the hyperparameters of the FedAVG protocol can be configured so that either the user runs only single local update step runs the update on large enough batch size. Both variants effectively turn FedAVG into FedSGD.
Discussion about FedAVG:
C ADDITIONAL EXPERIMENTS FOR DIFFERENTIAL PRIVACY
Here we provide experiments where a user additionally adds Gaussian noise and performs gradient clipping before sending the model update to server. Notice that the presence of noise makes the mea- surement by Eqn. noisy. Therefore, we only perform embedding recovery using Equation 7 when the magnitude of difference between bias gradient is larger than certain threshold. The experiment is performed on a 3-layer transformer as ribed in (Wang et al., 2021), where we clip gradient norm to 1 and compare performance the threshold to be 1e-6, The experiment also uses batch size of 2 and sequence length of 32, where one of the sequences contains the target word. The experiment results are presented in Figure g
different noise scales from le " Ie 1e 1e 9. 1e 10 and fix
The results show that the attacker is still able to succeed when the noise scale is relatively small, but predictably fails if the gradient becomes sufficiently noisy. We also note that defense based on noisy gradient is more likely to succeed against attack on larger transformer models due to the inherently smaller gradient magnitude.
Overall, promised by theory. applying stronger differential privacy eventually enforces user's privacy, and we believe related defense algorithms are more practically attractive. However, it is also known that stronger differential privacy degrades the utility of the final model. Finding good balance between privacy and utility remains an important research direction.
D ADDITIONAL EXPERIMENTS
To verify that the attack is stable across sequence lengths, we run series of additional experiments, where we fix the batch size and compare across sequence length in Figure The figure shows that our proposed attack also outperforms Decepticons across combinations of model and sequence length (although the performance of Decepticons is less sensitive to the change of sequence length).
15

---

Published as a conference paper at ICLR 2023
1.0
1.00
1.0
0.8
0.75
0.8
Accurary
0.6
0.50
0.6
Rougee
0.4
0.25
0.4
0.2
Durs-iPT-2
OUrS:BERT
Ours
DURS.BERT
0.0
Derep-Gin-2
DeT
Decep-SERT
Dccop.BERT
Decep SERT
200
Sequence Length
Sequence Length
Sequence Longtn
Figure 9: Comparison of our method with Decepticons for different architectures across various sequence lengths. The batch size fixed to 32. We evaluate both methods only the target sequences. Our attack remains agnostic to sequence length, and outperforms Decepticons all metries on target sequences.
16


Reviews:
[{"review_id": "sOQQ-NjaMw", "reviewer": "Reviewer_zieL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new attack on federated learning using text transformers, where the assumption is that the server is not honest-but-curious, but instead malicious and cannot be trusted. Current existing attacks in this scenario try to recover all input tokens of all users. The paper argues that this is not necessary, as an attacker is only interested in specific private information, which in reality is most likely only a small part of the whole sequence. To only reconstruct sequences containing sensitive data, the paper modifies the multi-head attention blocks as well as the linear layers of the transformer and filter for sequences containing specific keywords (e.g., like \"credit card\" or \"social security number\"). ", "summary_of_the_review": "The paper proposes a novel privacy attack on federated learning using text transformers. The problem statement is well motivated and it is shown that reconstructing only part of sequences that are relevant for the attacker is increasing the performance in contrast to previous approaches. The paper is well written and most of the time easy to follow. If the reader is not very familiar with the work that this paper is building on, it is hard to follow the reasoning.", "strength_and_weaknesses": "Strengths:\n- The idea of the paper to only reconstruct parts of sequences that are relevant for an attacker is novel.\n- This is an important area of research, as the current real-world implementations are based on trustworthiness of the server. Showing that such an attack is possible is showing that changing the implementations and protocols might be necessary to ensure privacy.\n- The paper is well written and most of the time easy to follow.\n\nWeaknesses:\n- It should be clearer how many heads are needed in total for the attack, as this might be a major limitation. Sudden drastic changes in the architecture when sending the modified model to the users might allow the users to detect this kind of attack.\n- It is a bit hard to understand the reasoning behind some of the actions for the attack. For example, what is the reasoning behind shifting the entries to the first d' positions? Without deep knowledge of the related work this paper is building on, it is hard to follow the mathematical reasoning. A short sentence to explain the intuition would be nice and benefit the reader a lot.\n- Experimental evaluation is limited. The experiments were conducted only on 2 models Fowl et al. (2022) is for example using another model (Transformer-3) for evaluating their approach.\n- The metrics used for evaluating the approach are not explained in detail. This makes it hard to interpret the results.\n\nQuestions to Authors:\n- Do we need one head for each keyword and additionally one for each of the positional imprinting of each keyword?\n- If you write \"We fix the sequence length to 32\" do you mean that every sequence has the length of exactly 32 tokens or is it possible that sequences have less than 32 tokens which are then padded using padding tokens?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gLMBdaDVpz0", "reviewer": "Reviewer_dBzB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the potential privacy leakage in federated learning settings. The paper argues that previous attacks are too ambitious to extract as much data as possible and this is at the expense of fidelity, so that previous attacks do not work under large-scale settings. The paper proposes an attack on FL which extracts input text that contains targeted privacy-critical words in language modeling training. In the proposed method, the parameters of the transformer are carefully programmed to filter relevant sequences from user data and encode the information in the gradients. The conducted experiments show that the attack works well and outperforms previous approaches in different settings; they show that the attack works even when the batch size is as large as 2000.\n", "summary_of_the_review": "In general, I am positive towards accepting the paper. However, I also believe the paper can be better if there is another round of review.\n", "strength_and_weaknesses": "Strengths:\n- The paper is well-motivated. I agree with the authors’ arguments on the existing approaches and also believe that focusing on only privacy-sensitive data is a practical setting for attackers.\n- The attacking setting (e.g., malicious server scenario), background, and related work has been discussed well. Generally, I like the positioning of this paper.\n- Although the method is based on a couple of existing works, it is relatively novel. \n- The paper conducts extensive experiments to evaluate the attacks with different setups. The results really show the vulnerabilities of federated language model training in a practical large-scale setting.\n\nWeaknesses:\n- The paper can be presented better. The method is complicated and section 3.3 is not easy to follow. Since the approach is built based on existing approaches, some details are not included in this paper, which may confuse readers.\n- The paper can be written better by discussing possible defense methods and it will also benefit the community and facilitate more research in this direction.\n- The paper lacks qualitative attacking examples reconstructed by the approach. It is hard to have a sense about how strong the attack is by only looking at those metrics.\n- What is the “total accuracy” metric exactly?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "XfdM_uF-u2C", "reviewer": "Reviewer_8KC8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about attacking the privacy of users when they do federated learning in specific settings. The setting is for training (large) language models that specifically use the “transformer” architecture. Moreover, the attack is done in the “malicious” setting, in which the central server who is running the federated learning does *not* behave honestly (& curiously) but it rather sends fabricated models to the users with the goal of extracting personal data from them. The paper is focused on extraction of *targeted* information; e.g., a credit card number as the string that follows the text “credit card”.\n\nThe paper compares its findings with recent works that also performs data extraction attacks on federated learning for the same architecture. Since the paper’s focus is on extracting targeted sentences it will do better in doing the exact goal (of this paper) because the other papers aim to extract texts rather indiscriminately. ", "summary_of_the_review": "In summary, I find the targeted aspect of the attacks of this paper interesting and I think this will lead to more papers like this. On the other hand, the paper suffers from a presentation that is not suitable for general audience and does not investigate the limitations of their attacks; e.g., against users who inspect the models sent to them. \n", "strength_and_weaknesses": "Strength:\nI like the aspect of “targeted” extraction of the texts. Here the attacker plants specific traps to extract specific texts from the model. This is a plus for this paper’s attack.\n\nWeakness:\nThe fact that an attack is done in a malicious setting is a downside. This means that the users can *potentially notice* that the server is *not* following the protocol honestly. The paper has (as far as I can see) zero discussions on this important aspect, and I would like to hear from the authors about this, if possible: Can you argue that the (malicious) changes to the model are indistinguishable for the users?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "APxVqNhVB5L", "reviewer": "Reviewer_7zTS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel privacy attack against FL systems, where the proposed attack extracts text sequences that contain targeted, privacy critical phrases. The targeted attack mechanism is achieved by maliciously modifying the model parameters on the server. In particular, the attack consists of three steps: tagging target sequences, filtering out irrelevant tokens and post-processing. According to the experimental results, the proposed method is more powerful than the selected Deception baseline.", "summary_of_the_review": "Overall, I think this work is interesting and I rated this work as 6. I do have some concerns and I am not an 100% expert on attacking FL systems, but I am willing to increase my score if the authors could address my concerns above.", "strength_and_weaknesses": "#### **Strengths**:\n- The targeted attack setting is novel, well-motivated and interesting\n\n#### **Weaknesses**:\n- The designed three steps of the proposed attack match the intuition to perform targeted attack, but it seems to me that the proposed techniques are largely inspired by or directly adopted from Fowl et al. (2022), which makes the technical contribution of this work incremental. \n- Based on my understanding, this method requires the model gradients returned by the users. Is this the reason that the authors only performed experiments on FedSGD? What if the gradients are not available, how could the proposed attack work (for instance on FedAVG)? This raises my concern on the generality of the proposed method. Maybe the author could provide some discussions and insights on this point as in Fowl et al. (2022) and Fowl et al. (2021).\n- In my opinion, the experimental setup is not very sufficient. Firstly, as mentioned above, only FedSGD is studied. Secondly, the selected evaluation metrics seem to be too general and could not really reveal whether the attack is indeed ‘targeted’. In the worst-case, the performance gain could be simply attributed to the additional attention weights added instead of the actually proposed method. It would be more convincing if the author can show that the proposed attack could recover the sequences with **targeted** key words.\n- Finally, this might be not so important. But I am curious about whether the proposed attack could surpass very simple defenses? For instance, the users could just add some noises to the gradients when returning the gradients to the server. Will this break this attack? Not to mention more advanced defenses discussed in Fowl et al. (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2sN41zHnPG", "reviewer": "Reviewer_q4yB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of obtaining private text data from federated transformer models. This paper studies the malicious server setting, where the server can arbitrarily tamper with the model sent to clients. This paper also studies a unique attack setting --- it focuses only on several privacy sensitive key words and the tokens following the key words, such as 'card number'. The proposed method elaborately designs transformer parameters such that the gradients directly reveal the token embeddings following the key words. Experiments are performed on real-world transformer models, where the proposed method outperforms a state-of-the-art baseline, Decepticon (Fowl et al. 2022). ", "summary_of_the_review": "From my perspective, the main problems of this paper are i) Not sufficiently self-contained; 2) Hard to be deployed in practice. Therefore, although this paper studies a practical and unique problem setting, I cannot recommend acceptance at this stage. ", "strength_and_weaknesses": "Strength: \n- The problem setting studied in this paper is unique and practical. This paper studies malicious servers instead of honest-but-curious servers studied in existing works. This makes sense as one can never assume that an adversary is 'honest'. Also, the paper studies how to extract tokens following specific key words. This is also practical, since most sentences typed by users are not highly related to privacy. \n\nWeaknesses: \n- This paper does not seem self-contained and hard to understand without sufficient background. This paper heavily refers to related works (especially (Fowl. et al. 2022)) in the designs and equations. However, the intuition/implications of them are not sufficiently introduced. I list several examples here. (It is suggested that the authors number the equations). \n    - Eqn. 1 is derived from Phong et al. (2017). However, although I knew the related work and understood it, it is not immediate for those who are unfamiliar with the related work. Also, it seems that $\\frac{\\partial b}{\\partial L}$ should be $\\frac{\\partial L}{\\partial b}$. \n    - The equation following Eqn. 1 (Page 4) from Fowl et al. 2022 is also not immediate and hard to understand. Also, it seems that the equation only works for one layer of transformer block (otherwise, the input should not be token embeddings $e$ but output features from the previous layer). \n    - The first equation on Page 6 from Fowl et al. 2022 is hard to understand without interpretation. The authors should interpret what the equation intuitively means. (e.g. organizing $w^Tx_j$ into bins). \n    - The equation above Eqn. 5 from Fowl et al. 2022 is also hard to understand. I know it is more or less similar to Eqn. 2, but as they serve different purposes, it is still beneficial to explain it.   \n    - Thus, the paper does not seem self-contained. This paper is hard to understand without sufficient knowledge on (Fowl et al. 2022). \n\n- It is not sure that the proposed method can be used in practice. I list several doubts on the applicability. \n    - First, it is not sure that the proposed method is compatible with model training. If the local model is training smoothly (local loss decreasing) but the global model is not training (e.g. loss significantly increases after server aggregation), clients will not have the motivation to participate. \n    - Second, the proposed method leads to non-negligible and weird parameter distribution. For example, a client can simply compute the norm of token embeddings and decide that the server model is weird and potentially malicious. By comparison, backdoor attacks on FL requires that the model outputs normally on normal data (Bhagoji et al. 2019, Bagdasaryan et al. 2019). \n    - Third, for the motivating example of credit card numbers, the sensitive information (i.e. the card number) is a string of numbers which is almost certainly out-of-vocabulary. Therefore, the card number may not even have a token embedding, making the retrieval impossible. \n    - Finally, the experiments use FedSGD (i.e. communication every iteration) as the attacked method. However, FedSGD is rarely used in practice for its high communication cost. In practice, FedAvg, which averages the gradients for multiple local steps (e.g. ~100) is used. It is not sure how this method works with FedAvg. Furthermore, if secure aggregation is used, the server only obtains aggregated gradients. Then, it is not clear how the retrieved information can be traced back to specific users. \n    - Of course, as I am not very familiar with (Fowl et al. 2022), a central related work of this paper, I may make misunderstandings. Please correct me if there are any. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster