DAGER: Exact Gradient Inversion for Large Language Models
Authors: Ivo Petrov
Dimitar Iliev Dimitrov
Maximilian Baader
Mark Niklas Mueller
Martin Vechev
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated Learning
Exact Gradient Inversion
Gradient Leakage
Privacy
Language Model
LLM
Attack
Primary area: privacy
TL;DR: We introduce the first exact gradient leakage attack for batches of sequnces on LLMs and show that it works on significantly larger token sequences and batch sizes while being faster and more price than prior work.

Abstract: Federated learning works by aggregating locally computed gradients from multiple clients, thus enabling collaborative training without sharing private client data. However, prior work has shown that the data can actually be recovered by the server using so-called gradient inversion attacks. While these attacks perform well when applied on images, they are limited in the text domain and only permit approximate reconstruction of small batches and short input sequences. In this work, we propose DAGER, the first algorithm to recover whole batches of input text exactly. DAGER leverages the low-rank structure of self-attention layer gradients and the discrete nature of token embeddings to efficiently check if a given token sequence is part of the client data. We use this check to exactly recover full batches in the honest-but-curious setting without any prior on the data for both encoder and decoder-based architectures using exhaustive heuristic search and a greedy approach, respectively. We provide an efficient GPU implementation of DAGER and show experimentally that it recovers full batches of size up to 128 on large language models (LLMs), beating prior attacks in speed (20x at same batch size), scalability (10x larger batches), and reconstruction quality (ROUGE-1/2 > 0.99).

DAGER: Exact Gradient Inversion for Large Language Models
Ivo Petrov"!, Dimitar I. Dimitroy 1.2, Maximilian Baader
Mark Niklas Müller Martin Vechey 1:2
INSAIT, Sofia University "St. Kliment Ohridski" dimitar. www..:.wee..iittooeeeeeeeo..ee www.ittt
ETH Zurich
LogicStaraai
(mbaader, mark. mueller, martin. werrdn.weccee@@inf.eeee ethz..c
Abstract
Federated learning works by aggregating locally computed gradients from multiple clients, thus enabling collaborative training without sharing private client data. However, prior work has shown that the data can actually be recovered by the server using so-called gradient inversion attacks. While these attacks perform well when applied on images, they are limited the text domain and only permit approximate reconstruction of small batches and short input sequences. In this work, we propose DAGER, the first algorithm to recover whole batches of input text exactly. DAGER leverages the low-rank structure of self-attention layer gradients and the discrete nature of token embeddings to efficiently check a given token sequence is part of the client data. We use this check to exactly recover full batches in the honest-but-curious setting without any prior on the data for both encoder- and decoder-based architectures using exhaustive heuristic search and greedy approach, respectively. We provide an efficient GPU implementation of DAGER and show experimentally that it recovers full batches of size up to 128 on large language models (LLMs), beating prior attacks in speed (20x at same batch size), scalability (10x larger batches), and reconstruction quality (ROUGE-1/2 0.99).
Introduction
While large language models (LLMs) have demonstrated exceptional potential across wide range of tasks, training them requires large amounts of data. However, this data is sensitive in many cases, leading to privacy concerns when sharing it with third parties for model training. Federated learning (FL) has emerged as a promising solution to addressing this issue by allowing multiple parties to collaboratively train model by sharing only gradients computed on their private data with the server instead of the data itself. In particular, FL has been used to finetune LLMs while protecting private data [1, 2, 3] in privacy-critical domains, such as law [4] and medicine [5].
Gradient Inversion Attacks Unfortunately, recent work has shown that this private data can be recovered from the shared gradients using so-called gradient inversion attacks, raising concerns about the privacy guarantees of federated learning [6]. While most prior work gradient inversion attacks has focused on image data [7. 8, 9], first works have demonstrated that text can also be recovered [6. 10, 11]. However, as these approaches are optimization-baste a major challenge by inducing much harder optimization problems and limiting them to approximate recovery of small batch sizes and short sequences. Therefore, applying existing attacks methods on modern LLMs would be computationally infeasible or yield subpar reconstructions.
the discrete nature of text data poses
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

This Work: Exact Recovery of Large Batches and Long Sequences To overcome these limita- tions, we propose DAGER (Discreteness-Basee Attack on Gradients for Exact Recovery), the first exact gradient inversion attack for (transformerrbased) LLMs in the challenging honest-but-curion setting. Our key insight that while discrete inputs pose a challenge for optimization-ban based attacks, they can be leveraged in combination with the low-rank structure of gradients to enable exact recovery via search-based attacks. Crucially, we show that the gradients of self-attention projection matrices in transformers are i) typically low-rank and ii) linear combinations of input embeddings. This allows us to check whether a given input embedding lies within the span of the gradient and was thus part of the input sequence. We use this to first recover the set of input tokens and then reconstruct the full sequences. For decoder architectures DAGER leverages their causal attention masks for to derive an efficient greedy recovery, while for encoder architectures, DAGER uses several heuristics to make exhaustive search tractable. As DAGER only requires propagating inputs through the first transformer block instead of full gradient computations, it scales to very large models. In fact, the higher internal dimension of these models even allows DAGER to recover more information as our low-rankness assumptions hold for larger batch sizes and longer sequences. We note that this approach is applicable both to the easier next-token prediction and to the harder classification setting.
Evaluation We demonstrate in an extensive evaluation that DAGER enables the exact recovery of long sequences and large batch sizes for both encoder- and decoder- based architectures, beating prior attacks in terms of speed (20x at same batch sizes), scalability (10x larger batches), and reconstruction quality (ROUGE- 1/2 > 0.99). In particular, we show this for GPT-2 [12], LLaMa-2 [13], and BERT [14] across CoLA[15], SST-2 [16], Rotten Tomatoes [17] and ECHR [18], for batch sizes up to 128. Additionally, we demonstrate that DAGER is versatile and can be applied to wide range of settings, including FedAvg [19], LoRA [20] finetuning and model quantization [21].
Key Contributions
Our main contributions are:
We show how the low-rankness of self-attention layer gradients can be leveraged to check whether specific discrete inputs were present in the input (Sec. 4).
We leverage this key insight to propose DAGER, the first exact gradient inversion attack for transformers (Sec. 5).
We conduct an extensive empirical evaluation demonstrating that DAGER is not only able to reconstruct inputs exactly but also scales to much larger batch sizes, longer input sequences, and larger models than prior attacks, while also being significantly faster to mount (Sec. 6).
We provide an efficient GPU implementation of DAGER, that can be publicly accessed at https://gituu..com/iit instiittt//ttttrrreeeerrrr wawerr--rrdddddindaeeeeee inversion.
2
Related Work
Gradient leakage attacks, first introduced by Zhu et al. [6], generally fall into two categorics honest-but-curious attacks [6, 22, 23, 7, 8, 24, 9, 10, 11, 25, 26, 27, 28], where the attacker passively observes the client's federated learning updates and tries to recover the data solely based on them, and malicious server attacks [29, 30, 31, 32, 33] where the attacker is further allowed to modify the federated learning model shared with the client. In this work, we focus on the harder to attack and more realistic honest-but-curious setting. A large body of the gradient leakage literature in this setting focuses on image data [7, 8, 24, 9, 27]. Differently, gradient leakage in the text domain remains successful only in the case of malicious adversary [31, 32, 34]. In the honest-but-curious setting, the results either remain limited to short sequences and small batch sizes B [6. 10, 11], require large number of gradient updates [25], or cannot recover the order of tokens in client sequences Xu et al. [35]. Further, state-of-the-an attacks require strong data priors [11, 25], and do not scale to realistic decoder-based LLMs. In contrast, DAGER, works on large batches and sequences for both encoder- and decoder-basee transformers, including LLaMa-2 [13]. Additionally. unlike prior work, our attack works on both token prediction tasks and the harder setting of sentiment analysis [11] where label recovery methods, such as [35], are not applicable. Finally, DAGER has no requirements for the state of the model training. In instance, [25] exploits model memorization of the data, unlike DAGER, which can handle the more realistic setting of being applicable at any point in time. Further, in contrast to [26, 25, 31], we do not require the gradient of the embedding layer, making our setting significantly harder.
2

---

Table 1: Table of notations used in the technical description of DAGER.
Symbol
Definition
Symbol
Definition
B
Batch size
C
Loss function used for training Hieldnnfemeddding) dimension
P
Transformer context length
d
Number of transformer blocks
y
Vocahulary
V
Vocabulary size [V]
ny
Token length for the j-th sequence
max, bj the length of the longest sequence Embedding function (maps tokens to embeddingg))
Σ n. the total number of non- padding tokens
b
jo
20
The j-th entry the i-th position token's 's embedking. The atlention mask
z
Inpul to the I-th attention layer
M
w((.k))
Query/key/value projection weights for the /-th attention layer
(Q. K, V)
The query/key/value embeddings in the i-th attention layer
The i-th token embedding after the I-th transformer block
f
T:'
The set of client tokens at position
The set of batch sequences up position
St
81, 82,
sample sequence of P tokens
The of the best reconstructed
The distances the span for each token/sequence
Sheat
D
sequences
The singular value threshold de- termining the rank of the 1-th layer
The distance threshold filtering token candidates on the 1-th layer
of yrank
While most prior honest-but-curious attacks leverage optimization methods to approximately recover the client inputs [6, 23, 7, 8, 24, 9, 10, 11], several works have shown that exact reconstruction is possible for batch size B under various conditions for different architectures [22, 26, 27]. Crucially, Dimitrov et al. [28] recently showed that B exact reconstruction from gradients of fully-connected layers is also possible. Our work, builds upon this result to show that exact gradient leakage is also possible for transformer-based LLMs.
Background and Notation
In this section, we introduce the background and notation required to understand our work. To this end, we first recall the basic operation the transformer architecture in the context of LLMs, and then describe the result, first introduced in Dimitrov et al. [28] for linear layers, in the context of self-attention layer showing that the gradients of its linear transformations have low-rank structure. The notations used throughout this paper are summarized in Table for clarity and ease of reference.
3.1 Transformers
In this paper, we consider LLMs based on both encoder and decoder transformer architectures trained using the FedSGD [36] protocol and loss function While we mainly focus on the harder-to-attack binary-classification loss typically used for sentiment analysis, we demonstrate that DAGER is equally applicable to the next-token prediction loss in Sec. 6, which contains more gradient information, as suggested by prior work Zhu et al. [6]. We denote the transformer's 's context length with P. its hidden dimension with d, the number of transformer blocks with L. the vocabulary of the tokenizer with V. and its size with V. We present approach for single-headed self-attention but it can be directly extended to multi-head self-attention, and we experimentally apply DAGER this context.
Transformer Inputs We consider inputs batches consisting of B sequences of tokens, where n. is the length of the 7th batch element. Sequences with length < max (ny) are padded. We denote the total number of non- non-padding tokens in a batch with B ビソ=1 nz.
Token Embeddings The discrete input tokens are usually embedded via function fo; [V]x[P] Rdd mapping a token's vocabulary index and its position i in the sequence to an embedding vector 2 = (v, These embeddings z are then stacked row-wise to form the input Z, Rodd the first self-attention layer. Note that / is known to the server, as is part of the model. Further, while yo differs between models, typically maps token indices embedding vectors before optionally adding positional encoding and applying LayerNorm. Crucially, fo is applied per-token.
3

---

Self-Attention
The stacked embeddings 1 are then passed through series of self-attention layers.
We denote the input to the 1th self-attention layer as Zi E R for 1 S - L. A self-attention
layer is a combination of three linear layers: The query Q: = W, key K₁ = Z,W, and value Vi2 = W.W. layer, which are then combined to compute the self-attention output:
attention( Qt, Vi) softmax
M
Q.K. va
Vi.
where M is the binary self- eeff-attention mask, @ is the element-wing product, and the softmax is applied row-wise. M chosen ensure that padding tokens do not affect the layer's output. Further, for decoders, M ensures that only preceeding tokens are attended. For notational convenience, we denote as f: 1 + R the function that maps any sequence of input tokens the hth input embedding at the 1 S 1 S self-attention layer. Note that f is part of the model and, thus, known the attacker.
3.2 Low-Rank Decomposition of Aelf-Attention Gradients
For linear layer Y = XW + (b ... b) with a weight matrix W E R=** a bias E R", and batched inputs X E R and outputs Y E Rbxm Dimitrov et al. [28] show that:
Theorem 3.1 (Adapted from Dimitrov al. [28]). The network's gradient w.r.t the weights W can be represented as the matrix product:
aC SW
xT
8Y
Further, when the batch size b S n, m, the rank of BC
is at most b. 3W
Rdx and X E Rxxn in Eq. 1. The rank limit follows directly from the dimensionalities of Oc BY
W W!Q.KV) Rddd As long as ac ac aw aw. aw are rank-deficient. Without loss of generality, for the rest of the paper we use
In this work, we apply Theorem 3.1 to the linear projection matrices the total number of tokens d, states that the gradients
and de
ac Z BC 8Q:
to explain our method.
ow
4 Overview of DAGER
C
11
8t
In this section, provide a high-level overview of our method DAGER, illustrated in Fig. 1. DAGER is an attack that recovers the client input se- quences from the shared gradi- ents of a transformer-basee DAGER works for both encoder and decoder-baseed LLMs, how- ever, for simplicity here we focus on decoder-only LLMs. While, in theory, one could enumerate all possible batches of input se- quences, and check whether they produce the desired gradients, this is infeasible in practice it requires computing / B dif- ferent gradients. We reduce the search space by leveraging the rank- mef-cceicy of aw? discussed in
Alimann
Laye (f')
Tolen Embr Hise
N
Tox 1op-88pan
Layer (+)
DA
8B
Trke Ezhoalaa Lesel""
A.t
LLM.
BAC
ABC
ABE
Recovery
Sequence Recovery
Client
GW
Server
Figure 1: Overview of DAGER. DAGER first recovers the sets of client tokens T," at each position E P by testing each token in the vocabulary V via a span check based on the client gradients of the first self-attention. Then recursively combines them into partial client sequences S; with length up to filtered to obtain the correct sequences S via the gradients of the second self-attention.
Sec. 3.2, combined with the finite number of possible inputs to each self-attention corresponding to one of the PxB gradients above. For the rest of the section, we assume rank-deficieney, of ai aw,3. that is b < d. This assumption is in practice satisfied for reasonable input lengths and batch sizes.
Leveraging the Rank Deficiency As the gradient matrix OW form subspace of Idd of dimension b. Further, under mild assumptions
8C is rank-deficient, i.c. b < d,
the columns of ac aw?

---

(see Theorem 5.1), the embedding vectors forming Z are linear combinations of the columns of ac ow z ac It is unlikely that any incorrect embedding vector, part of one of the VPxB "Q.
incorrect inputs Zi part of colspan( SC ow?
C R2 as the hypervolume of this subspace is O.
Filtering Incorrect Embeddings We can efficiently filter out all incorrect client embeddings at any layer/ / without computing their gradient, simply by checking if they are in colspan 8C However, ow,
applying this procedure naively still requires us check all PP different client batches. Instead, we leverage this filtering in a two-stage recovery algorithm that first recovers the client tokens T," at position i using the rank deficiency aw?
ac (Token Recovery in Fig. 1), and then recovers the client
batch of sequences S* based on 7," and the rank deficiency of ac (Sequence Recovery in Fig. 1). aw
Token Recovery Our token recovery method relies on the observation that f is computed per- token. Therefore, the input embeddings Z1 are always part of the set (fr a)lu E [V], E [P]]- We apply our span check above to this set for the first layer gradients aw to filter the incorrect embeddings and their corresponding client tokens at position i, thus, constructing the set correct client tokens T at position i.
ac
Sequence Recovery In sequence recovery, we leverage the fact that f! is computed per- sequence and that the decoder mask M ensures that the second layer input embeddings at position i do not depend tokens with position > i, i.e., ft (81,-.., 8p) f2 (81, (1,-.... ..... 8;), for any sequence of tokens 81; : 8p. Crucially, for a correct client partial sequence .,,..., 8i-1 of length
this allows us find the correct next token in T," by simply extending with all possible token 3, E T and then checking which of the resulting embedding vectors f. (s 8 3,) is correct, i.e is in colspan( BC We apply this procedure iteratively starting with the single token sequences ow?))
S = Ti', extending them one token at a time to produce the partial sequence reconstructions S until the sequences cannot be extended anymore and return the result.
5 DAGER: Exact Sequence Recovery for Transformers
In this section, we present the technical details of DAGER. Specifically, we first theoretically derive dc in Sec. 5.1. We then describe how of our filtering procedure based on the rank-deficiency of aw
we apply it on the gradients of the first and second self-attention layers to respectively recover the client tokens (Sec. 5.2) and sequences (Sec. 5.3).
5.1 Efficient Embedding Filtering
Below, we discuss the technical details of our filtering procedure, outlined in Sec. 4, and prove its conrectness. We first show that, under mild assumptions, the embedding vectors forming Zi linear combinations of the columns of dc restating this in terms of rowspan( Zi) and colspan ac aw? ):
o
Theorem 5.1. If d and the matrix Oc is of full rank (rank b), then rowspan Zii colspan( 8C OW??
SQ.
ac is full-rank holds in practice, as shown empirically in Dimitrov Note that the assumption that aQ; et al. [28], and that further 6 < is almost always satisfied, i.e., that the total number of tokens in the input is smaller than the internal dimensionality of the model, for practical LLMs. We discuss the assumptions in further detail in App. B.2. The latter then directly implies the rank-deficiency of ac which we leverage to show: aw.y.
Theorem 5.2. When d, the probability of random vector Rdd to be part of colspan( ac is aw?
almost surely 0.
5

---

Intuitively, the closer 1) is to 0, the more likely is part of the span. To allow for efficient computation of the projection in Eq. first pre-compute an orthonormal basis for dc using an Combining Theorems 5.1 and 5.2, we arrive at our main result stating that, if b < d, an embedding vector z that is part of the client self-attention inputs Z, belongs to colspaní ac ). while random aw?
embedding vectors that are not part of Z almost surely do not.
Span Check Implementation While the above result holds under real, i.e., infinite precision, arithmetic, for our method to work in practice, we require an implementation that both fast and robust to numerical errors caused by floating-point arithmetic. We, therefore, introduce the metric d, the distance between a candidate embedding vector and its projection on the colspan( BC aw, ):
d(z,1) = proj(z, colspan( ac ow?"
))
Sw?7
SVD and truncating the cigenvalues below a chosen threshold r. We can then trivially compute this projection, as the sum of projections onto each basis vector. Finally, we say that a vector is in colspan( a2 ). if the distance d( 2, 1) < Ti is below chosen per-layer threshold a
5.2 Recovering Token Sets
Algorithm Recovering Individual Tokens ac
We now describe how DAGER leverages the above filtering procedure to recover the input tokens ex- actly. To this end, we consider the set of all tokens in the model's vocabulary
1: function GETTOK(
V, f0 ,T)
OWT
n 4 0
[V] at every possible position i E [P] and compute their input embed- dings the first layer via the per-token embedding function fo. We then filter out token-positinn (v. whose embedding vectors (u, i) do not lie in colspan( o ) to obtain the set input tokens (across
3:
T," + O. DJ +
for u [V x [ do
d (e, if < T1 then - max(n, + k [d)
tuples
6:
batch elements) at position d:
7:- {u E [V] di d(ff(o,)), m...
(3)
10:
return n, (o: (DF
We formalize this process in Algorithm 1, where we simply enumerate all token position tuples (v, i). Additionally, we compute the length of the longest input sentence the largest position i of any recovered tuple (v, i) (Line 7). If fo is position-inde -ideeeendddde,ttt, e.g. when rotary instead of absolute positional embeddings are used, we recover the set of all input tokens 7 = U, T," for every position. Our algorithm handles this at the sequence recovery stage at the price of slightly higher computational costs (see Theorem B.4).
105. Tokens
While conceptionally simple, this approach is exceptionally effective and robust to the distance threshold 71,2, as we demonstrate in Fig. for the GPT-2 model [12] (d 768)
L1 filtering
L1+L2 filtering
103_
Ground truth
and a batch of B = 32 sequences consisting of b 391 tokens. Despite the total number of tokens exceeding half the model dimensionality d, even our single layer (LI) filtering approach narrows the set of possible starting tokens, which are independent of the rest of the input, down to less than 300 from GPT-2's vocabulary of & 50K across a wide range of thresholds. Adding second filtering stage using second-layer filtering, described next, allows DAGER to recover the exact 32 starting tokens. This is used to exactly narrow down the first token for each sequence, allowing us to inductively reconstruct the whole sentence for decoder-only models.
10
10-6
10-3
100
Threshold
Figure 2: Effect L1 and L.2 Filtering
5.3 Recovering Sequences
Given the set of input tokens, recovered above, we now describe how recover input sequences by applying our filtering procedure to the inputs of the second self-attention layer Zz. We first define the set S Tí x... x Tp of all sequences formed using the recovered token sets T,", As the

---

second layer input embeddings f (s) are computed independently for each sequence one can naively enumerate all S, compute their second layer embedding vectors f (s) and apply the span check for every position to obtain the true set of client sequences:
= (s s-
E ()) < T2, Vi [P]]-
Unfortunately, this naive approach requires O(B") span checks. To alleviate this issue, first show that the causal attention mask of decoder architectures allows us to greedily recover the exact sequences in polynomial time, before discussing heuristics that make an exhaustive search tractable for encoder- based architectures.
Algorithm DAGER for Decoders
function ATTDEC(B, "W aw
ac
V, P, fu/1 T1/22)
2:
n, n, GETTOK
Dc V,P,J (,P:
Ow
3:
St + +(0)g
4:
for i - L...., do
5:
TokenFound <- False
6:
S + S S x T, S + 0
7:
Recovering Decoder Sequences Due to the causal attention mask M in decoder ar- chitectures, the 2t input of the second-self
8:
for 8 E S; do
9:
if d( (() T2 then
10:
S TokenFound < True
S {8)
attention layer f/ (s) depends only on the
11:
first i tokens of the input sequence We can 12:
if not TouedFound then
thus apply span check on the results of f; to check arbitrary sequence prefixes of length 14:
13:
break
best TOPUUNIQUU S aw?
0C B)
i. We leverage this insight in Algorithm 2 to iteratively recover the sets Sj (Line 10) of input sequence prefixes
15:
return Sien
S = E T/ d()) <2),,
starting from the set of empty sequences So and extending them one token at time (Line 6) until none of our sequences can be extended any further (Line 12).
Accuracy [%]
100-
4y=10
For models with small internal dimension d, or batches with 8C
Ay-20
a large number of total tokens b, the weight gradient #W4 might become full rank and all embeddings would pass span check. To avoid this we set a maximum rank threshold, = min(b, d - As) for the orthonormal basis computed via SVD (See Sec. 5.1). We visualize the effect of different A on GPT-2, in Fig. 3, and observe that A, = 20 offers the best trade-off between stability and accuracy, yielding almost perfect reconstruction even for very large inputs with b very close to d.
4y=40
&y=60
60-
Acrts
20-
650
250
850
Tokens
Figure 3: Encoder Ablation Study
Accuracy [%]
Recovering Encoder Sequences For encoders, all second- layer embeddings f. (s) depend on all input tokens. We thus cannot use the greedy reconstruction discussed above but have to enumerate all sequences in S. To make this search tractable, we leverage the following heuristics. We can determine the positions i of end-of-sequence (FOS) tokens the input to determine the input sequence lengths ný- This allows to recover input sequences by increasing length and eliminate the tokens constituting the recovered sequences from the token sets T," Additionally, we truncate the proposal token sets T to
100
B-1
W/ heuristies beuristics
60
20-
100
200
# Tokens
Figure Encoder Ablation Study
the batch size B token closest to colspan( BC ow
). Finally, we
always consider at most 10M sequences from S before returning the closest to colspan( ac
SW?
We demonstrate the effectiveness of our heuristics in Fig. 4 for the base BERT model and different batch sizes B and note that for B = 1 we can still recover inputs perfectly, as |T, = We provide more details on DAGER for encoder- archtterrrchi in App. B.6.
7

---

6 Experimental Evaluation
We now describe our extensive experimental evaluation of DAGER. Our results demonstrate signifi- cant performance improvements compared to prior methods, on a variety of settings. We also present ablation studies for isolating the effects of each DAGER component.
6.1 Experimental Setup
We evaluate DAGER on both encoder- and decoder-based models including BERT [14], GPT-2 [12], and variations of LLaMa [13, 37j. We consider three sentiment analysis datasets CoLA [15], SST- 2 [16], and Rotten Tomatoes (RT) [17]. featuring sequences of varying lengths between typically 4 and 27 words. Additionally, we consider the ECHR [18] dataset, which contains sentences exceeding 1000 words to demonstrate the scalability of our approach in sequence length. We provide a more detailed description of our architectures and datasets in App. C. Following previous work, we report the mean and error of the ROUGE- 1/2 [38] scores, i.e., the overlap rate of unigrams and bigrams, respectively, over 100 batches, excluding padding tokens. We report the error as the 95% confidence interval given by twice the standard error. Wherever we cannot assume normality of the mean's 's distribution, we estimate the interval by generating 10000 random samples by bootstrapping. Additional details regarding computational requirements and hyperparameters can be found in App. C.
Comparison against baselines First, we compared our performance against the state-of-the-ee algorithms TAG [10] and LAMP [11] with batch sizes ranging between B and B 8. We run the two attacks for 2500 iterations per experiment, making use of the hyperparameters described in their respective studies. As Balunović et al. [1 provide two variations of the LAMP algorithm based on different objective functions LAMP L2 + L1 and LAMP we only report results from the variation with the higher ROUGE-1 score. In Table 2, we show results GPT-2BASE and BERT assessing the performance on decoder-based and encoder-based models respectively.
The results indicate that for decoder-base -ased models, such as GPT2, DAGER achieves near-perfect reconstructions across all datasets and batch sizes, significantly outperforming the baseline algorithms in every setting. Importantly, as further elaborated in App. C. DAGER achieves that while also being significantly more efficient- - 100 batches of size on RT took 3.5 hours vs TAG and LAMP which required s 10 and 50 hours, respectively. Additionally, we confirm the claims made by Balunović et al. [11] that LAMP outperforms TAG in the majority of settings. Examples of reconstructed sentences can be seen in Table 9 in App. C.3. We note that while we observe non-perfect ROUGE-2 scores on the SST-2 dataset, this is entirely due to an artifact of our metric library that assigns ROUGE-2 score of 0 the SST-2's single-word sequences. We kept this behaviour to avoid having to rerun the baseline experiments, that also relied on this.
Further, Table 2 shows significant improvement over prior work on encoder-based models like BERT, with near-perfect reconstruction for = 1,2, 2, and an average of 43% more tokens recovered for larger batch sizes. A significant advantage of DAGER over the baselines is its ability to more accurately recover the sentence structure, as evidenced by the much higher ROUGE-2 scores.
Main experiments While prior attacks' performances become very poor for batch sizes as little as 8, we now demonstrate that DAGER is only limited by the embedding dimension of the model. To this end, Table 3 we compare two decoder-only models, GPT-2RASE with d = 768, and LLaMa-2 7B with d = 4096 on B as large as 128.
The results are consistent with our claims that DAGER produces almost perfect reconstructions in all cases when the total number of client tokens is not extremely close to the embedding dimension d. Further, while on LLaMa-2 DAGER achieves near-perfect reconstructions even up to batch size of 128, for GPT-2 DAGER shows partial or complete failure for B = 64, 128. This suggests that despite the significant computational costs of hours per batch for B = 128 on LLaMa-2, larger models have the potential to leak significantly more information. This is especially concerning given the current trend of ever-increasing model sizes. Finally, we observe the effect of attempting a best-effort reconstruction by establishing rank threshold, as described in Sec. 5.3, when the gradients are of full rank. This allows DAGER to achieve ROUGE-1 score of 30.3 (instead of 0) for GPT-2 on CoL B = 128. A thorough ablation study on the advantage of this heuristic can be found in App. C.2.
8

---

Table 2: Comparison of sequence reconstruction from gradients between DAGER and the baseline algorithms TAG and LAMP on various batch sizes and datasets. R-1 and R-2 denote the ROUGE-1 and ROUGE-2 scores respectively.
B-
B
B
8-2
R-1
IAG
=
0.54
80120
1.
T.8
0.2.0.5
5.7
2.4
CoLA
13.3-455
11001:0
13.4
8.0
1.0
DAGER
0.0
100.0 0.0
100.0
100.0 2.00
100.0 0.0
100.0 0.0
100.0
100.0 0000000000
TAG
-.5
611:12
02251
SST-2
LAMP
62.2-6.0 6.0
31. 1.4
2.11
9.113.11
9.8
2.7
8.1
0.7 0.4
DAGER
100.0=0.0
8.0
1.0.0
89.5 4.1
1.0
92.8 2.4
100.0
92.9
TAG
1.1=18
0,1:21
7.0.122
0.1181
6.2
0.1281
6.1 0.5
0.1 1.1
Tomatoes
DAGER
100.0
41.4=44
10000-000 0.0
100.0.0.0
0.0
100.0
100.0 a 0.0
90.3
90.3
100.0_61
0.7
99.9
0.3
TAG
7.4
7.7
3.4
3.9
CoLA
LAMP
2.
6.7
77.8 3.6
31.5 4.6
08: 3.4
21.8
62.9 + 2.2
13.1 = L
DAGER
100.0-0.0
100.0
100.0 0.0.
100.0 0.0
94.0..20.
2.3
TAG
75.3 4.3
19.0 19,9
1..6
16.0
11.011..44
12.3 1.3.1...
50.4 2.24
0.2
SST-2
66.8=3.0
7.9
6:.4::366
4.
3.6
32.5
2.
DAGER
100.0 0.0
100.0
11000.1000
99.3:25
99.0121
95.6.1.2.
93.0 1. 3.3
74.1
39.8 2.9
TAG
1.35
3..7
1.6.1..7
0.0
Tomaloes
Romen
LAMP
64.7 - 4.4
16.5 3.9
46.4+-3.7
7.6+:2..
35.1
4.2 1.5
27.3 27.3+:1.44
2.0
1.0001000
9..11.1.2
Table 3: Main experiments the GPT-2BASE and 2 (7B) models with higher batch sizes on various datasets. R-1 and R-2 denote the ROUGE-1 and ROUGE-2 scores respectively.
-2
B - 64
R-1
R-2
R-1
R-2
R-1
R-2
R-1
R-2
Col.A
GPT-2
100.0::: 0.0
100.0 1.00
100.0 0.0
100.0
100.0 0:00::00 0.0
100.0:: 0.0
30.3 =
16.6
(7B))
100.0 0.0
100.0 1.00
100.0 0.0
100.010.0
99.9233
09.9141
99.5
98.3
SST-2
GPT-2
100.0 0.0
94.6 11
100.0
93.4 1.0
92.9-1300 1.30
85.0 3.5
13.7
4.3
(7B))
100.0 0.0
100.0
90.931 22
99.9 0.1
90.9
0.1
98.2
97.8 0.4
Rotton
GPT-2
100.0 :: 0.0
59.9-13
18.00 1.7
97.8 4: 1.8
2.8 1.1
0.4
B.ll 0.0
Temasoes
(7B)
100.0"61
100.0100
100.0
100.0 0.0
97.9
0.5
99.7
99.7:83
Reconstruction under FedAvg The FedAvg algorithm [19] is among the most widely used proto- cols in federated learning. It features E training epochs on minibatches of size Byini < B with fixed learning rate r). Despite featuring multiple low-rank gradient updates, this setting it remains vulnerable to our attack, as we elaborate in App. B.3. We show in Table 4 that FedAvg is susceptible to gradient leakage under DAGER for a range of reasonable learning rates and number of epochs.
Table 4: Experiments on the FedAVG setting on the GPT-2 model with a batch size of on the Rotten Tomatoes dataset. We use default set of hyperparameters of F 10 epochs, learning rate 1) 10-1 and mini-batch size Buint= = 4. R-1 and R-2 denote ROUGE-I and ROUGE-2 respectively.
E
R-1
R-2
17
R-1
R-2
B Bmnni
R-1
R-2
2 98.4 0.9
98.0
1.0
10-5
100.0*0.
99.8
2
93.2 i 1.7
92.3 1.9
0.2
5
97.3 1.2
96.8 1.8:11 1.3
5 10-5
99.8
99.6"03
4
95.4 1 1.6
94.7 1: 1.7
10
95.41
1.6
94.7
10-4
1.6
1.7
8
98.6
98.2+
20
96.0 1.4
95.3
5 10
82.2 1.
16
100.0 0.0
99.8+02
Effect of Fine-tuning Methods We further demonstrate DAGER's versatility across a range of pretraining paradigms, including quantized models and Low-Rank Adaptation (LoRA) [20] finetuning. For both LLaMa-3 70B with B and LLaMa-3.1 8B with B 32 16-bit quantization, we observed excellent ROUGE-1 1 and ROUGE-2 scores (>99%) (see Table 11 in App. C.5). We also present near-exact reconstructions under LoRA training, DAGER can be directly applied the decomposed weight matrix, with further technical specifics detailed in App. B.4. With LoRA updates of rank 256, which is standard for the LLaMa-2 model noted by Biderman et al. [39], we observe ROUGE-1 and ROUGE-2 scores in the region of 94 95%, given in Table 11. These results reaffirm that DAGER is applicable to common fine-tuning methods.
Effect of Model Size and Training on Reconstruction Prior work [11, 10] suggests that the size of a model, as well as, the degree of pre-training significantly affects the amount of leaked client information. To this end, in Table 5 we evaluate DAGER on the larger (GPT-2LARGE) and

---

Table 5: Experiments GPT-2 variations in different settings on the Rotten Tomatoes dataset. R-1
and R-2 denote the ROUGE-1 and ROUGE-2 scores respectively.
B 16
B
32
B 64
B
128
R-1
R-2
R-1
R-2
R-1
R-2
R-1
R-2
GPT-2aAsE
100.0 + 0.0
99.9
0.3
98.0 1.7
97.8 1.8
2.8 1.1
+.1
0.0 0.0
0.0 0.0
GPT mrctunat
100.0
89.8183
96.4 2.3
0.210
0.0
GPT Nxeetike
99.9
90.7103 80.7103
09.
2.3
0.5
0.0
GPT-2
100.0
99.8 0.3
100.0
99.9
41.1
38.1
0.0
pre- pre-trained for 2 epochs (GPT-2 variants of GPT-2 on the RT dataset for batch sizes up to 128. We observe very little difference in performance. In fact, the GPT-2LARGE 's larger embedding dimension allows us to approximately reconstruct more tokens at larger batch sizes. We further note that larger vocabulary does not negatively impact DAGER, can be from the applications LLaMa3.1-88 1-8B and LLLaM331-7700 (see Table 11), which feature a vocabulary size of 128,256 tokens.
Reconstruction under Next- Token Prediction Additionally, we evaluate our model on the next- token prediction task to demonstrate DAGER's efficacy under different contexts. We again achieve near-perfect results with ROUGE-I/2 scores of > 99. DAGER does not reach perfect scores because the last token each client sequence only acts as a target and it is, thus, masked out from the input.
Reconstruction of Long Sequences Finally, to demonstrate our robustness to long sequences, we conducted a single experiment with B on the ECHR dataset truncated to 512 tokens. We obtain perfect score of 100.0 + 0.0 for ROUGE-1 and ROUGE-2 on GPT-2BASE. emphasizing the general applicability of DAGER. In contrast, in the same setting LAMP achieves a ROUGE-1 of 10.1 2.3.
7 Limitations
As discussed Sec. 5 and demonstrated in Sec. 6, the performance DAGER on decoder-based models is only constrained by the embedding dimension d. While exact reconstruction for number of tokens b > d is unachievable, we showed that the attack's s effectiveness decreases only gradually with b. Given our robust performance in an undefended setting, an interesting avenue for future work is to improve DAGER against different defense mechanisms, including but not limited to using the Differential Privacy SGD optimization process (DPSGD)[40).
On the other hand, applying DAGER on encoder-based architectures for larger batches (B 8) becomes challenging due to the high-order polynomial growth of the search space volume with respect to the batch size. These computational constraints make comprehensive exploration of the search space nearly impossible, thereby reducing the likelihood of achieving a feasible reconstruction. This issue extends to longer sequences, where the size of the search space expands exponentially with the maximum sequence length. To mitigate these effects, we propose that future research could focus on exploring further heuristics efficiently reduce the search space.
8 Conclusion
We introduced DAGER, the first gradient inversion attack for transformers able to recover large batches of input text exactly. By exploiting the rank-deficiency of self-attention layer gradients and discretencss of the input space, we devised a greedy algorithm and a heuristic search approach for decoder-based and encoder- based architectures, respectively. Our results show that DAGER achieves exact reconstruction for batch sizes up to 128 and sequences up to 512 tokens. We further demonstrate DAGER's effectiveness across model sizes, architectures, degrees of pre-training, and federated learning algorithms, establishing the widespread applicability of our attack.
Our work demonstrates that recent decoder- based LLMs are particularly vulnerable to data leakage, allowing adversaries to recover very large batches and sequences in the absence a robust defense mechanism. This underlying vulnerability highlights the need for increased awareness and develop- ment of effective countermeasures in privacy-critical applications. We hope this paper can facilitate further research into creating reliable frameworks for effective and private collaborative learning.
10

---

Acknowledgments
This research was partially funded by the Ministry of Education and Science of Bulgaria (support for INSAIT, part of the Bulgarian National Roadmap for Research Infrastructure).
This work has been done as part of the EU grant ELSA (European Lighthouse on Secure and Safe AI, grant agreement
101070617) Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the European Union or European Commission. Neither the European Union nor the European Commission can be held responsible for them.
The work has received funding from the Swiss State Secretariat for Education, Research and Innova- tion (SERI).
References
[1] Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6915-6919. IEEE, 2024.
[2] Chaochao Chen, Xiaohua Feng, Jun Zhou, Jianwei Yin, and Xiaolin Zheng. Federated large language model: A position paper. arXiv preprint arXiv:2307.08925 2023.
[3] Sara Babakniya, Ahmed Roushdy Elkordy, Yahya Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. arXiv preprint arXiv:2308.06522, 2023.
[4] Zhuo Zhang, Xiangjing Hu, Jingyuan Zhang, Yating Zhang, Hui Wang, Lizhen Qu, and Zenglin Xu. Fedlegal: The first real-world federated learning benchmark for legal nlp. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 3492-3507, 2023.
[5] Adam Sadilek, Luyang Liu, Dung Nguyen, Methun Kamruzzaman, Stylianos Serghiou, Ben- jamin Rader, Alex Ingerman, Stefan Mellem, Peter Kairouz, Elaine O Nsoesie, al. Privacy-first health research with federated learning. NPJ digital medicine, 4(1):132, 2021.
[6] Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In NeurIPS, 2019.
[7] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients- how easy is it to break privacy in federated learning? NeurIPS, 2020.
[8] Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M. Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. CVPR, 2021.
[9] Zhuohang Li, Jiaxin Zhang, Luyang Liu, and Jian Liu. Auditing privacy defenses in federated learning via generative gradient leakage. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10132-10142, 2022.
[10] Jieren Deng, Yijue Wang. Ji Li, Chenghong Wang, Chao Shang, Hang Liu, Sanguthevar Rajasekaran, and Caiwen Ding. Tag: Gradient attack on transformer-based language models. In Conference on Empirical Methods Natural Language Processing. 2021.
[11] Mislav Balunović, Dimitar Dimitrov, Nikola Jovanović, and Martin T. Vechev. LAMP: extracting text from gradients with language model priors. In NeurlPS, 2022.
[12] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 18):9, 2019.
[13] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2xXX::2307.09288 2023.
[14] Jacob Devlin, Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810. 1810.04805, 2018.
11

---

[15] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint 1805.12471, 2018.
[16] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compasitionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631-1642 2013.
[17] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorizaiioo with respect to rating scales. In Proceedings of the 2005.
[18) Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. Neural legal judgment prediction in English. In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317-4323, July 2019.
[19] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficcii learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[20] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Ahu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint trXiv:2106.09688 2021.
[21] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient mangge--rrthmmetc-onlly inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704 2704-2713, 2018
[22] Le Trieu Phong, Yoshinori Aono, Takuya Hayashi, Lihua Wang, and Shiho Moriai. Privacy- preserving deep learning via additively homomorphic encryption. IEEE Trans. Inf. Forensics Secur., (5), 2018.
[23] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv, 2020.
[24] Jiahui Geng, Yongli Mou, Feifei Li, Qing Li, Oya Beyan, Stefan Decker, and Chunming Rong. Towards general deep leakage in federated learning. arXiv, 2021.
[25] Samyak Gupta, Yangsibo Huang, Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recov- ering private text in federated learning of language models. In NeurlPS, 2022.
[26] Jiahao Lu, Xi Sheryl Zhang, Tianli Zhao, Xiangyu He, and Jian Cheng. April: Finding the achilles' heel on privacy for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10051-10060) 2022.
[27] Junyi Zhu and Matthew B. Blaschko. R-GAP: recursive gradient attack on privacy. In ICLR, 2021.
[28] Dimitar I Dimitrov, Maximilian Baader, Mark Niklas Müller, and Martin Vechev. Spear: Exact gradient inversion of batches in federated learning. arXiv preprint PXXv:2403.039455 2024.
[29] Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. When the curious abandon honesty: Federated learning is not private. arXiv, 2021.
[30] Liam H. Fowl, Jonas Geiping, Wojciech Czaja, Micah Goldblum, and Tom Goldstein. Robbing the fed: Directly obtaining private data in federated learning with modified models. In ICLR, 2022.
[31] Liam Fowl, Jonas Geiping, Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Decepticons: Corrupted transformers breach privacy federated learning for language models. ICLR, 2022.
12

---

[32] Hong-Min Chu, Jonas Geiping, Liam H Fowl, Micah Goldblum, and Tom Goldstein. Panning for gold in federated learning: Targeted text extraction under arbitrarily large-scale aggregation. ICL.R 2023.
[33] Yuxin Wen, Jonas Geiping, Liam Fowl, Micah Goldblum, and Tom Goldstein. Fishing for user data in large-batch federated learning via gradient magnification. In ICML, 2022.
[34] Jianwei Li, Sheng Liu, and Qi Lei. Beyond gradient and priors in privacy attacks: Leveraging pooler layer inputs of language models in federated learning. arXiv preprini urXiv:23/2.05720, 2023.
[35] Qiongkai Xu, Jun Wang, Olga Ohrimenko, and Trevor Cohn, Flat-chat: A word recovery attack on federated language model training. 2023.
[36] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-effien efffccient learning of deep networks from decentralized data. In AISTATS, 2017.
[37] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshomn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak. Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The Ilama 3 herd of models. CoRR, abs/2407.22 217221137 2024. doi: 1048550ARXX.22002215555 URL https:tpppo htt:://oo...ogoo org/10...... hetpg//1...855/aaa/ 48550/ 2407 21783.
[38] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
[39] Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.0966
2024.
[40] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[41] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's ' transform- ers: State-of-the-art natural language processing. arXiv preprint arXiv: 1910.0377, 2019.
[42] Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. Deep learning- based text classification: a comprehensive review. ACM computing surveys (CSUR), 54(3):1-40, 2021.
[43] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. 2019. In the Proceedings of ICLR.
[44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
13

---

[45] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
14

---

A Broader Impact
In this work, we demonstrate that it is possible to exactly reconstruct large batches of textual data from gradients in the conest-but-curions setting. Our findings are widely applicable across different transformer-basee LLM architectures, showing that, in contrast to prior belief, language transformers actually more susceptible to gradient leakage attacks than other architectures. While our work naturally substantially increases the privacy risks posed to federated learning clients training LLMs, we also believe that sharing our work is crucial for finding future solutions to the issues uncover.
Importantly, we find that the recent decoder- models are much more susceptible gradient leakage attacks due to the causal nature of their self-attention masks. Our work implies that in the absence of proper defense mechanisms, receiving gradients from those models essentially equivalent to receiving the client data directly. Further, we show that the attacker's s ability to mount DAGER grows with the embedding size d, suggesting the privacy risks posed by DAGER practical settings will only grow over time. With these considerations in mind, we emphasize the importance of providing privacy safeguards via secure aggregation, larger batch sizes, or gradient perturbations.
B
Additional Technical Details of Our Method
B.1
Deferred Proofs
Theorem 5.1. If < d and the matrix 8Q,
ac is of full rank (rank b), then rowspan Zi) =
colspan(
8c ow
Proof. We split the proof into two parts. We first prove colspan(
0c
rowspan Zi) and then
ow?
prove that rank( sw:? rank Zi). thus implying the two spaces are the same.
For the first part of the proof, we observe that due to the matrix multiplication in aw?7
8C
2J
ac
3Q1
all columns of aw are linear combinations of the columns of z with coefficients given by aQ
Thus, colspan
Sc aw
C colspan( Z rowspan Zi).
For the second part of the proof, we observe that since 8C is of rank b and rank(ZF)) is most b, SQ
rank( 8C ) min(rank( (2)a) rank( ac aw?
)) rank Zf"). This finishes the proof.
(Q.
Theorem 5.2. When < d, the probability of random vector E Rdd to be part of colspan ac is
aw?
almost surely
Proof. As Oc is rank-deficient (Theorem 3.1). colspan( Oc is strict linear subspace of IR.. aw Thus, has hypervolume 0 via the Sard's lemma. This directly implies that the probability of random vector E Ed to be part of colspan( ww/ is almost surely 0.
aw
B.2 Technical Assumptions of DAGER
Below we present brief commentary on the technical assumptions of DAGER. DAGER makes three assumptions:
We assume that ÆQ:
is full-rank.
We require the total number of tokens b in the batch to be smaller than the embedding dimension d, ensuring that 8C is of low-rank.
Zw
We assume a known discrete set of possible inputs to the model, i.c. its vocabulary.
15

---

Importantly, DAGER does not assume any prior knowledge of the labels or the lengths of each sequence in the batch nor access to the gradients of the embedding layers which have been shown to leak significant information [26]. Further, we require no language priors and operate under the butur-obut- setting which does not allow malicious changes model weights. Finally, sub-differentiability is sufficient for applying DAGER.
In practice, DAGER requires much fewer assumptions than existing works in the setting while being successful in a variety of common LLM tasks, e.g., next-token prediction and sentence classification. While these tasks use the cross-entropy loss, DAGER can be applied to any loss function that non-trivially depends every input token. This ensures the full-rankness of 8c aw;
To confirm its generality, we apply DAGER with a Frobenius norm based loss and ReLU activation functions. We use custom loss function [(s1. $2,,. sp) =| f $2..... sP)lFF. where ||.||F is the Frobenius norm, which is equivalent to an MSE loss with O as a target vector. We report the results of applying DAGER with these modifications on Rotten Tomatoes using GPT-2 with B 16 in Table 11, achieving ROUGE-1 and ROUGE-2 scores of >99% in both cases.
B.3 DAGER under the FedAvg protocol
We establish that DAGER can be effectively used attack clients employing FedAvg updates under mild assumptions. First, we show that it is possible to theoretically apply Theorem 5.1 to the protocol directly on the first layer, and under reasonably low model updates for further layers. We then demonstrate experimentally that DAGER can be successfully applied across wide range of parameters, namely the number of epochs E, learning rate y and mini-batch size byini-
Spanchecks for FedAvg updates Let X = (s ......, ) denote the input embedding vectors to the relevant 1-th layer query projection matrix for mini-batch of size bminn-- Obtaining x in the forward pass is equivalent to sampling the corresponding rows in the full-batch representation X, =
...,, sB), ) as each sequence in a batch is independent from all the rest. Therefore, we can rewrite each local gradient update ac
aw, X ac
as Xi ac where the i-th column aQ,
of ac is the zero vector 0 if s; is not present in the 3Q.
Therefore, are able simply disregard the mini-batch sampling and focus only the total number of iterations E, assuming that every sequence in the original batch is sampled at least once.
Let the input embedding vectors to the 1-th layer at timestep t < E be X The final weight W after steps can be written as:
E
E W
w W
8C OW!
ac
7
t=0
t=
Under the assumption that the changes in model weights are relatively small, we can approximate X = XP. This is always the case for 1 = 0, the embeddings before the first layer independent of the model weights. This allows us to rewrite We as:
W W n
T Oc EQ
W
OT OC X
W
x
(1)
8C
(5)
As the server has knowledge of the starting weight WP and the final weight WF, it able to compute the sum of all gradient steps, i. we will be able to apply Theorem 5.1 to x."" (n Lf ac still require E
). We
dc to full-rank, which satisfied under standard DAGER assumptions.
SQ.
Further experimental details We further empircially demonstrate that DAGER can effectively utilise the assumption of consistent feature embeddings across epochs under a reasonable learning rate and number of iterations. As shown in Table 4, we observe near-exact reconstruction rates for most configurations, with metrics only slightly declining as the number of epochs increases. The key factor that influences the success of DAGER is observed to be the learning rate 77, as the aforementioned
16

---

assumption might be invalidated at large 7, However, learning rates exceeding 1) 2 10 10 are typically high for the model converge, particularly in multi-client settings. Therefore, we can conclude that DAGER is highly effective in the FedAvg context.
B.4 DAGER under LoRA training
In this section, we discuss how DAGER can be extended to work on LoRA weight decomposition. Under LoRA, the linear layer weight updates for weight W E R are performed on low-rank representation: W = Wo + AB, where A E Rdxr E rxx As we obtain the gradient weights for both and B, we can apply Theorem 5.1 to with A = A, namely because BC 2A = X ac ac Assuming that 8 is full-rank and that b < F, work is directly applicable. This replace the spanchecks to be performed on A instead of W for each layer, after which DAGER can be applied directly. In practice, LoRA finetuning typically initializes W Wo and B to only contain zeroes which reduce the rank of #c ØA for the first few optimization steps. We therefore train the LLaMa-3.1 8B model on the Rotten Tomatoes dataset using batch size of with r = 256 (following [39]) for epochs before applying DAGER. We report results in Table 11 and observe excellent R1 and R2 of about 95%.
B.5 Complexity Analysis
key point of DAGER is the algorithm's exceptional computational efficiency on decoder-based models. In order to quantify the dependency of runtime on relevant variables, we describe the asymptotic complexity for both decoder- and encoder-bart based models. Below we list and prove several lemmas that assist us in the complete proof of our assertion for the complexity of DAGER. When not specified, a batch size of B is implied for any inputs.
Lemma B.1. The product of two matrices MD E Ruxm and M RmXP can be naively computed in ((nmp).
Proof. We write down the product:
Σ M &
To produce the entire matrix explore all integers = 1.... n, j = 1.. p and k 1... m, and in particular any combination of the 3. This implies that we make nmp iterations, from which a time complexity of (mmpp) follows.
Lemma B.2. For any matrix M E Tgdxx and vector E Rd, we can compute the projection of on the subspace spanned by the columns of M in ((Pn) time.
Proof. Projecting onto the column space of a matrix can be done by projecting the vector onto individual columns and then summing all projected components. We compute this using Einstein notation, while denoting the resulting vector as E Rd. Then, we obtain:
Ps = Σ Ma,M, i
This loop iterates over = 1... n,j = 1....d.. d. = 1... d, resulting in a total number of d' iterations, which implies time complexity of Oddn)).
Lemma B.3. For any transformer-based model, which has an embedding dimension d, square projection weights of dimension d x d, and MLP hidden dimension of dwur. propagating sequence of b tokens, takes time of asymptotic complexity (bdd + 52d bdlard).
Proof. We follow the notation defined in Sec. 3.1. The embedding representation of the sequence is denoted 1 We then obtain the query, key and value vectors Q, K, V by multiplying with the weight matrices w W.K WY E Rdxdd According Lemma B.I, we can accomplish this in time of (bc?).
17

---

Token recovery We begin by describing the first step recovering individual tokens per position. We begin by obtaining T by taking T = {o E (v, E colspan( 02 aw, )}. Because we Computing the attention scores is dominated by computing QK", which by applying Lemma B.1, can be done in Ob)²d).
As a final step to the self-attention component, we compute the multiplication of the scores =
softmax( M QK" Vd E Rbxb and V R time Obb2d). We note that computing the row-wise softmax operations can be amortized and take a total of Obb)) time, which is dominated by Obbdd).
The total computation up until this point can, therefore, be done (bd + b).
To produce the output of the transformer block, we pass the self-attention through an MLP layer, which can be represented by set of simple matrix multiplications (of the same complexity). Therefore, applying Lemma B. once again, this step requires time (bdxxLdd).
We can sum up all components to obtain total complexity of Obbd + bad + bdMLPd). If reasonably assume that dMLP= = (d), then the complexity can be simplified to Obd + b²d). [
Having explored in in-depth each smaller component of the algorithm, we can now determine the complexity of DAGER. We explore 3 instances of our algorithm, namely separating decoder- and encoder- based models, while also differentiating between absolute positional encodings and RoPE. We summarize our findings in single theorem:
Theorem B.4. By considering a training iteration of batch of size B with the longest sentence being of length P, where the sentences are represented as a sequence of tokens from vocabulary of size V, DAGER can reconstruct the input with an asymptotic time complexity of:
1. For decoder-based models:
(a) If the model applies positional embeddings before layer 0 BVP + d B)))d)
(b) If the model applies positional embeddings after the first projection, i.e RoPE
O(BVV de + d dz))
2. For encoder-only models OVP + d BPPP2)
Proof. We separate algorithm in 2 different parts:
1. Recovering the tokens T through the span check on W.O.
2. Reconstructing the entire sequences S*.
determine the span check via the projection distance drroj of representation vector E I&. and truncated gradient 08 Rddr where rank( 8c according Lemma B.2, this can be done ow??
Sw.7
in time (d.r). The rank of both ac ac is limited by the total number of tokens b E O(B)). aw ow? As can be inferred by Theorem 5.1, the complexity of the spancheck is O(PBd) We perform this for every token in the vocabulary for an additional factor of V. making the total complexity OPBBVd') d).. We note that for models which employ positional embeddings before the first transformer block, repeat this step P times, while we only have to do it once for those that don't This respectively results in time complexities of OV3 and O(PBV d') for recovering individual tokens.
Spancheck complexity In practice, we perform all span checks by performing Singular Value ac and then applying the projection distance to the right orthonormal. It is
Decomposition on aw?
well known result that SVD for matrix of size d x d takes time (dd))
Sequence reconstruction Having recovered the individual tokens, we detail the resulting time complexity of reconstructing the entire sequences. We describe each step for decoder-based models, before proceeding to mccoer-based ones.
18

---

For decoder-basee based models we begin by describing the number of tokens we obtain per position. the model applies positional embedding before the first transformer layer. we assume that for each position we recover a set of tokens T,', such that [7, = (T)) for some variable T that depends on our input parameters and setup. Similarly, for models that do not have positional embeddings before the first transformer layer, we only recover single set of tokens T of size [T* O(PT)
We now show the complexity required for performing the forward pass and span check for single sequence. For sequence length n, according to Lemma B.3 the forward pass takes time (ndd + n²d). and the span check per token is of complexity O(PBd (as per Lemma B.2). This implies time complexity of OnnPBd²) per sequence for the span check, leading us to an overall complexity of On + nd + n²d).
Finally, we apply our observations to the full reconstruction. We further assume that at each step of the greedy reconstruction, we maintain ( possible sequences, which is usually the case when performing greedy reconstruction. By repeating the above propagation for every combination per sequence length yields a total runtime of O(T x (nd² + n²d (PPBd)) = OPPTB* dl)) and (PP x B x (nd2 + n²d +nPBdd) = O(PPTBB*(m) in the cases described in statement la) and 1b) respectively.
Summing over all lengths yields time of O PB* O(P for the former. Analogically, for the latter we obtain complexity O(P*TB²
In practice, we observe that T = a for some factor a between and 10, hence we can assume that T (B). This is practically correct assumption for reasonably defined threshold т1- This makes the final time complexity for recovering sequences O(PTBBd O(pa 3 33 d for case Ia) and OPPTB²2 O(P* for lb).
On the other hand, in the of encoder-based models, need to exhaust all possible token combinations over all positions.
We again assume that for each position we recover a set of tokens T,", such that (7," = (T)).
Because we have to explore all possible combinations, that results in a total number of 11"", II O(T) = O(T") sequences.
We now show the cost the span check on a single sequence. We leverage our finding that one such span check takes time (d) (we again substitute that the rank O(BB) and apply Lemma B.2), meaning all span checks take time O O(p2Bd') per sequence.
Additionally, the forward pass takes O(P.P p2d) time.
Finally, because reconstruct each sequence separately, repeat the reconstruction algorithm O (B) times. This leads to the complexity for this step O(T" B(P 3dp Pd p²d))= OT B² p2)
As above, we assume T ((B,, leading us to a final complexity of BPPd)) = P2d'.
Combining our conclusions for steps and 2 yields the stated complexities for each setting.
The key points to highlight is that DAGER is polynomial across both sequence length and batch size for decoders, regardless of the type positional encoding. Meanwhile, for encoders the we observe that DAGER is exponential in length and polynomial with high degree in terms of batch size. To this end, the heuristics we described are crucial to significantly reduce the search space.
B.6 Encoder Algorithm
19

---

Table 6: Specifications of models that used in work.
Model
No. layers
No. heads
Feed-forward size
V
Positional embedding
No. Parameters
GPT-2axsE
Devoder
50,257
Absolute
50,257
Absolute
LLaMa-2 (7B)
Decoder
32
4,096
32
11,008
32,000
RoPE
6.74B
LLaMa-3.1 (8B)
Decoder
4,096
32
14,336
128,256
RoPE
8.03B
3.aMa-3.1 (70B)
Decoder
80
8,192
64
28,672
128,256
RoPE
70.6B
BERT
Encoder
12
768
12
3072
30,522
Absolute
110M
Algorithm 3 DAGER for Encoders
In this section, we provide pseudocode for DAGER when applied to encoder- based L.Mss. We provide it in Algo- rithm 3, where we first find the set of client sequence lengths nj and store them in V (Line 4 to Line 8). We then go through the nys from the smallest
1:
function ATTENC(T, B, aC
DC
V,P, f01 f, .T1/2)
ow
SW2
2:
T,, D' GETTOOK
OW
OC
V,P,f
3:
T +TOPBTOKKES(T "D""
V+
5:
for 1-1,, 1, do
to the largest (Line 9), enumerating all possible sequences s S, of length 7:
ifO E T then N - + l1 + + T (EOS) for i E SORT(N) do
i (Line 10). Importantly, when a cor- rect sentence E S is found its to- kens are removed for the token sets T."
9:
10:
S s T x... x T,
(Line 14). Finally, we return the dedu- 11:
St t {s E d(f)
< 12 [+]
plicated best reconstructions across dif-
12:
for s E St do
ferent sequence lengths Stea (Line 15). We note that when S; is larger than 10M combinations, we sample 10M random combinations from it instead.
13:
for + l....,id i
14:
p P
{8p)
15:
SiSs TOPUNIQQEE Si, ow.
8C B)
16:
return S
C Additional experimental details
In this section we describe additional details regarding the experimental setup for DAGER.
Models In particular, we explore its feasibility on the base variant GPT-2BASE, featuring 12-layer transformer with a 768-dimensional hidden state, 12 attention heads and feed-forward filter of size 3072. Additionally, we examine the effects of model size by considering GPT-2LARGE- which contains 36 layers, a 1280-dimensional hidden state, 20 attention heads, and feed-forward filter of size 5120. To demonstrate the versatility of our approach, we also evaluate our attack in next-token prediction context on the GPT-2BASE model. We further demonstrate the efficacy of our attack on LLaMa 2-7B[13] and BERTBASEI 14] to illustrate our performance on state-of-the-art decoder and an encoder-only model, respectively. All models were sourced from HuggingFace(4) in their pre-trained form, following standard practice language modeling research[42] Full specification details for all models can be found in Table 6.
Datasets In our evaluation, we utilize three binary classification datasets with varying sentence lengths, namely the Corpus of Linguistic Acceptability (CoL,A][15), the Stanford Sentiment Treebank (16)-)))16), which are part of the GLUE benchmark(43), as well as the Rotten Tomatoes 1 sentiment analysis dataset. While our algorithm is date-inndeenndnt, previous studies have indicated that text size can affect reconstructabilityll the aforementioned datasets, with CoLA featuring text typically ranging from 4 to 9 words. SST-2- from 4 to 13 words, and Rotten Tomatoes from 10 to 27 words. We note that the binary classification setting has been shown to be a less vulnerable than next-token prediction which can be attacked via label reconstruction attacks [35], however, we conduct an additional experiment to substantiate the claim this setting. Finally, to showease DAGER's 's capability to handle arbitrarily long sequences, we leverage the European Court of Human Rights (ECHR) 18] dataset which includes sentences that are over 1000 words long, far exceeding the maximum input length of any of the aforementioned models.
6]. We demonstrate robustness to this factor by selecting
20

---

Table 7: Total runtime for all main experiments given in hours. Fields containing N/A represent experiments that were not run.
B=1
B=2
B=4
B=8
B=16
B=32
B=64
B=128
TAG
9.4
9.1
9.2
9.5
N/A
N/A
N/A
N/A
CaLA
LAMP
15.9
24.2
40.7
68.3
N/A
N/A
N/A
N/A
DAGER
1.5
1.1
1.5
2.8
8.0
8,5
4.4
8.2
GPT-2
TAG
9.5
9.7
9.1
9.7
N/A
N/A
N/A
N/A
SST-2
LAMP
16.3
24.2
39.0
66.5
N/A
N/A
N/A
N/A
DAGER
1.4
1.7
3.6
8.8
13.7
8.3
7.1
TAG
8.6
8.8
N/A
N/A
N/A
N/A
Rotten
LAMP
16.2
24.2
37.9
67.4
N/A
N/A
N/A
N/A
Tomatoes
DAGER
2.5
3.5
8.5
10.2
1.4
1.9
TAG
6.1
6.3
9.2
11.6
N/A
N/A
N/A
N/A
CoLA
LAMP
11.0
26.4
42.6
85.2
N/A
N/A
N/A
N/A
DAGER
0.1
0.1
1.3
19.4
N/A
N/A
N/A
N/A
BERT
TAG
9.1
6.5
7.9
11.8
N/A
N/A
N/A
N/A
SST-2
LAMP
17.1
25.8
43.8
82.8
N/A
N/A
N/A
N/A
DAGER
0.1
0.7
11.8
50.1
N/A
N/A
N/A
N/A
TAG
11.2
N/A
N/A
N/A
N/A
Rotten
LAMP
17.4
28.2
29.8
83.1
N/A
N/A
N/A
N/A
Tomatoes
DAGER
7.6
48.9
195.0
N/A
N/A
N/A
N/A
LLaMA-2
CoLA
DAGER
7.7
8.6
8.2
9.9
12.6
21.2
41.2
160.0
SST-2
DAGER
7.8
7.7
10.0
12.5
19.1
45.9
74.3
257.7
RT
DAGER
7.6
9.1
10.7
15.6
26.0
39.5
112.2
523.1
Computational requirements We implement DAGER in PyTorch [44] and run all experiments on single GPU. Tests on the LLaMa-2 (7B) architecture were performed NVIDIA A A100 Tensor Core GPUs, which boast 40 GB of memory, while all others were ran on NVIDIA L4 GPUs with 24 GB of memory. practice, less demanding resources may be used, especially for lower batch sizes on BERT and GPT-2BASE. In terms of required RAM, we used between 16 GB and 150 GB per experiment, depending on the batch size and model.
Hyperparameter details We use span check acceptance threshold of 10 55 in the first layer, and T2 = 10-3 in the second, a rank truncation of A, 20, and for decoder-based models consider at most 10 0000000000 000 proposal sentences per recovered EOS token position. We consider pre-trained models with randomly initialized classification head using normal distribution with o 10-3, 3 To manage numerical instabilities within the framework, tweak the eigenvalue threshold when doing the SVD rank and decrease with the batch size growing, varying it between 10-7 and 10-9.
C.1 Runtime of experiments
We further demonstrate the computational efficiency of DAGER. The runtime summary can be found in Table 7. is notable that for the experiments BERT, we have drastic increase in complexity with respect to the batch size and sequence length, as expected from App. B.5, which do not affect the baselines much. That said, we still remain within the same order of magnitude, while achieving significantly better results.
In contrast, we achieve significant improvement for decoder-based models. We notice that within attack's scope for GPT-2, we can reconstruct batch for less than minutes for any batch size. It is important to highlight that the runtime decreases for the batch sizes of 64 and 128 because we are more often than not unable to recover any tokens due the embedding dimension limitation described in Sec. 7.
21

---

Table 8: Ablation study GPT-2BASE with and without the rank threshold heuristic. We report scores of for any example that has full-rank gradients. R-1 and R-2 denote the ROUGE-1 and ROUGE-2 scores respectively.
H 16
8 32
H 64
128
With cutuff
100.0 0.0
100.0 1
100.0 1 1.0
100.0.0.00
100.0 1: 0.0
100.0 0.0
30.3 1.
14.6 1
No cutoll
100.0 1: 0.0
100.0 0.0
100.0 :: 0.0
100.0 1:000:0000 0.0
100.0 1: 0.0
100.0 0.0
0.0 0.0
0.0 1 0.0
SST-2
100.0
94.6
100.0 013
93.4
92.9 3.0
13.7
cutof
100.0 0.0
94.6 1.1
100.0
93.4 1.0
75.0 8.7
00.7 + 8.2
0.0 0.0
0.0 0.0
Rotenn
With cuieff
100.0
89. 00.91%
98.0 1.7
1.1 1.11044
0.0.0.0.0
0.0
Tomsstnes
No cutotf
1.0.0
99.9 3
53.9 à:
4.8
0.0
0.0
0.000.0
0.0
Table 9: Example comparison between DAGER and LAMP for random samples from different datasets, reconstructed at batch size 1. We note that LAMP was selected, as was the best- performing baseline model for batch size of 1.
Sequenec
Reference
Sarah devoured the cakes in the kitchen last night.
CaLA
DAGER
Sarah devoured cakes the kitchen last night.
LAMP
Sarah imaginary even kitchen dev devoured culees last night
Reference
caper that's neither original nor terrily funny
SST-2
caper neither original temilly funny
LAMP
that's neither on perennke noo terribyy funny
Reference
plays like the old discasc-of thwowwok-hhkkkkkkkeooooooo small-screen melodramas.
Rotten Tomatoes
DAGER
plays like discasc-of-lle-- small-sou -nnll--crrenn melodranas.
LAMP
plays like old Screen impactuorm discase small-screen like
inclodramas.
C.2 Rank restriction ablation study
In section Sec. 5.3, we described that for full-rank matrices we attempt a best-effort reconstruction by artificially restricting the rank by a threshold 6. Here we demonstrate the effect of this component by showing the performance of DAGER without attempting to recover any part of the sentence. Any time we observe sample with full-rank gradients at either the first or second layers, we immediately fail and report a score of 0. The results can be seen in Table 8.
C.3 Example reconstructions
In Table show sample reconstructions between the best-performing baseline LAMP_CA for batch size B = 1. We note that DAGER achieves exact reconstruction, while LAMP_ can predict less than half of the sentence correctly.
C.4 DAGER under differential privacy
In this section, show how DAGER performs under defended setting, in particular by adding
random Gaussian noise with variance o² to all gradients. We explore the range a E [10-5, 5 5 x 10 10-4) for any 2 10-3 the sentiment prediction accuracy of the converged model drops to below 80% from > 87%. We apply DAGER on the GPT-2 model for the Rotten Tomatoes dataset at Due to the highly random nature of this type of defense, measuring the single span check distance at layer = 2. Instead, we utilise that further layers / > 3 retain the same property that the input embeddings only depend on previous tokens, and measure the average d ELDP d(f!- (s), (s),1) for a certain number of layers LDp, which we optimise as hyperparameter. Further, because any noise will make the gradient updates full-rank, for computation purposes we set a constant rank of r = 100, which much higher than the length of any sentence. While it has been shown [45] that differencial privacy provides provable guarantees for protecting privacy, we provide promising initial results, as seen in Table 10. We believe that there are numerous improvements one could make, but leave these for future work.
1. cannot simply filter the sequences by
22

---

Table 10: Experiments on the differential privacy setting under Gaussian noise on the GPT-2 model with a batch size of on the Rotten Tomatoes dataset. R-1 and R-2 denote the ROUGE-1 and ROUGE-2 scores respectively.
10
5 X 10
10
0 = 5
10
R-1
R-2
R-2
R-2
R-1
R-2
74.0 3.2
70.8 + 3.4
46.9
4.1 32.5 + 4.0
20.7 + 4.4 11.6 +
3.4
5.6
0.9
Table Miscallaneous experiments, referenced in the evaluation section. We applied DAGER on the Rotten Tomatoes dataset for B = 16, if not specified otherwise.
L.LaMa-3 (B = 1) LLaMa-3.1
Frobenius loss
ReLU activation
LoRA 256)
R-1
99.9
99.4
99.3 Fe
100.0 0.00
94.8 94.2 0.7
R-2
99.9
99.420.
99.8 00.8+011
99.8
C.5
Miscallaneous
Any other experiments, namely the ones LLaMA-3 70B, LLaMa-3.1 8B, DAGER under LoRA training, or DAGER using different loss functions are included in Table 11.
D Licenses
In our work, the publicly available datasets CoLA, SST-2, Rotten Tomatoes and ECHR. CoLA licensed under the MIT license. No public licensing information was found for SST-2 and Rotten Tomatoes. Furthermore,
ECHR under the Creative Commons ciiibutionNonnCommeriii ShareAlike 4.0 International (CC BY 4.0) license. For privacy concerns, ECHR has issued a statement of protection of personal data that ensures private data was handled appropriately'.
In terms Large Language Model architectures, we use GPT-2 under the MIT license and BERT under the Apache License. All aforementioned licenses permit our of the underlying assets for the purposes this paper. We obtained access to LLaMa-2 through the Llama 2 Community License Agreement which permits the model's 's use in commercial and research settings.
Finally, we obtain the code for the LAMP and TAG attacks through the public repository for LAMP which is licensed under the Apache License 2.0.
'Tee statement can found under https://www.cchr.com wwt..i//riivac........
TTh full license can be found under https://ai meta. www..atw.w.aaaaaaammmmm wot.aaa//////////e........om/
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claim of our abstract and introduction sections that we demonstrate the first algorithm to recover whole batches of input text exactly in the honest-but-eee but-curiouus setting. To this end, we provide an overview of DAGER in Sec. 4 and delve into further mathematical details in Sec. 5. The discussion encompasses relevant mathematical proofs, as well as a thorough description of the underlying algorithms, featuring a greedy approach for decoder-based architectures and exhaustive heuristic search for encoder-based ones. Additionally, we claim that the proposed attack outperforms existing attacks under the same threat model in quality, speed and scalability. We substantiate this claim through multiple experiments across batch sizes up to = 128 for different datasets and models, as showcased in Sec. 6. Finally, in the same section we support our claim that DAGER can be applied across a variety of settings through empirical evaluation, and through rigorous analysis in App. B.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We detail our limitations in a dedicated section (Sec. 7), discussing both theoretical and computational aspects of DAGER. for the latter of which present formal proof of complexity in App. B.5. We further describe any underlying assumptions in the "Related Work" and "Background" sections (Sec. 2 and Sec. 3 respectively).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide rigorous proofs for all theorems directly following the statement, or have been deferred to App. B. All assumptions have been explicitly stated, discussed and further elaborated upon in App. B.2.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide code and additional resources that are required to reproduce our results shown in Sec. our supplementary material. Furthermore, we describe in sufficient detail all specifics of the algorithm that are needed for reproduction in the technical and experimental sections (Sec. 5, Sec. respectively), with additional information found in App. C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
25

---

reproducible, they While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide Python implementation, alongside detailed instructions for both setup and execution. Most of the data and architectures we used belong the public domain. The only exception is that one must request access to the private HuggingFace repository how to approach this have also been included. We
for all LLaMa models. Instructions discuss the licensing and availability for all assets in App. D.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
/tlii//guudee//ddedubiiiiiss stt.uuooiiissiooooooiioooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips. htppppulllgpiieeeep Codeeee//odmmummm swabbbmmsssiooolllii.o Piccy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have described all relevant hyperparameters as part of the experimental section, details of which can be found in Sec. 6 and App. C. Furthermore, ablation studies on the effect of most hyperparameters have been demonstrated in Fig. 2, Fig. 3, Fig. 4 and App. C.2 to support our choices.
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide 95% confidence interval for all relevant scores in the experimental section (Sec. 6). To this end, we detail our treatment Gaussian and Gan-Gausiian distributions separately.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We list all necessary computational resources in Sec. 6.1, including information regarding the GPU and CPU requirements. We further disclose the necessary runtime for each of the main experiments in App. C.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
27

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurip:: https///eeerrissccpii////ttttiit cc/ppbblicEEhiicsiudee lines??
Answer: [Yes]
Justification: We discuss the implications of our work in our Broader Impact statement in App. A. To prevent malicious uses of our code we plan release it under license that prohibits any non-research related uses. We note that work does not introduce new datasets or architectures and that we rely on only publicly available datasets, whose licenses we appropriately discuss in App. D.
privacy in federated learning
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss this paper's implications in the Broader Impact section (App. A), where describe the necessity of our work and associated privacy considerations.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Upon publicly releasing the code, we plan on attaching a license that prohibits any non-academic use.
Guidelines:
The answer NA means that the paper poses such risks.
28

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All datasets and models used either publicly available, have been properly attributed. We have also appropriately credited the authors of the baselines' code, which is publicly available. We discuss the licenses and terms and conditions explicitly in App. D.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, should be provided. For popular datasets, paperswi rsperswitt thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
license, copyright information, and terms use the package
For existing datasets that are re- se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide Python implementation, alongside detailed instructions for both setup and execution. Our work does not introduce architectures datasets. We discuss licenses for all external assets in App. D.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "mMQU89MGHL", "reviewer": "Reviewer_L4TG", "summary": "In this paper, the authors propose the DAGER algorithm which leverages the low-rankness of self-attention layer gradients in order to filter out incorrect embedding vectors and recursively reconstruct true input text. DAGER works within the Centralized Federated Learning setting, where the server is honest in aggregating updates and performing model updates, but is curious to try to recover personal information from each client.", "questions": "Would this be able to work within the decentralized FL environment (i.e. could a single client, acting as the aggregator, decipher other agent text)? \n\nIn the Centralized FL environment, would asynchronous training affect the performance of DAGER or render it infeasible (asynchronous, stale updates)?\n\nThere's a small error in Figure 1 in Sequence recovery (middle part, 2nd sequence), AA_ should not be in the span and BA_ should be? In the diagram the opposite is true.\n\nHow does DAGER perform against noise (DP) or malicious users? Can it be ammended to combat this?\n\nWhat happens if the vocabulary size is extremely large? Won't this process take a very long time (i.e. does it scale with |V|)?\nHow expensive is this enumeration process? Especially if the sequence is long?\n\nVery cool insight about size of model and how that allows gradients to be reconstructed easier because they're low rank since d is increasing so rapidly. Bigger is not always better when it comes to privacy!", "limitations": "Yes.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. A very smart/practical and, to the best of my very limited knowledge in this area, novel approach to reconstruct text by disregarding all embeddings which do not align with the low rank gradients received by the server. \n2. The empirical results demonstrate that this method is highly effective, and greatly outperforms all other baselines.\n3. Paper is well-written, has good diagrams and figures, and is easy to follow for someone not well-versed in this area (attacks & LLMs).", "weaknesses": "1. I'm curious about the computational complexity of DAGER. It seems that the search space can be extremely large (even when heuristics are used), and recovering individual tokens could take awhile.\n2. (minor) In the related works it would be nice to go deeper into research within the gradient leakage in the text domain. A specific breakdown of how this paper differs from the current methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\newcommand{\\RL}{\\textcolor{red}{L4TG}}$We would like to thank reviewer $\\RL$ for the very positive review and feedback. We are happy to read that the reviewer finds our work very smart, practical and novel. Further, we are glad they assessed our paper as easy to follow and our approach as highly effective. We thank the reviewer for giving us pointers on how to broaden the scale of DAGER. We now address all points raised by the reviewer.\n\n**Q1. What is the computational complexity of DAGER and does it scale to very large vocabulary sizes and long sequences?**\n\nWe report both the runtimes (Table 5) and complexity (Theorem B.4) of DAGER in the Appendix of our paper. We highlight that the computational bottleneck for all practical parameters stems from the term $P^3 B^3 d^2$ or $P^4 B^3 d^2$ (in the RoPE case). As the vocabulary size is a linear component, it does not dominate the overall complexity of DAGER. We demonstrate the effectiveness on larger vocabularies by evaluating state-of-the-art models, such as LLaMa-3 70B and LLaMa-3.1 8B, which feature a vocabulary of 128,256 tokens, which is among the largest ones in existence. There, we observе only marginally increased runtimes.\n\nPlease refer to **Q1** in the main response and Table 5 in the rebuttal PDF for a description of the results and requirements.\n\n**Q2. How does DAGER relate to current state-of-the-art methods in gradient leakage in the text domain?**\n\nDAGER assumes the honest-but-curious setting, where the server is non-malicious, unlike the Decepticons [28] and Panning for Gold [29] attack, meaning we do not need to change the model weights sent to clients. Most prior honest-but-curious methods, such as DLG [6], TAG [10], LAMP[11] and Jianwei et al. [31] rely on relaxing the discrete optimization problem to a continuous one that has no guarantees of convergence. One exception is APRIL [23], which analytically recovers the inputs but doesn’t scale to B>1. DAGER, however, can handle B=128. Another one is FILM [22], which requires access to the gradients of a batch at any iteration and, similarly to LAMP, requires a strong language prior, unlike DAGER. \n\nFurther, we do not require knowledge of the labels for different sentences in the batch in the sentiment analysis case, in contrast to TAG and LAMP, and we work with almost any reasonable loss function (see **Q2** in the main response) unlike Flat-Chat [32]. \nFinally, in contrast to APRIL, FILM and Decepticons, we do not require the gradient of the embedding layer, making our setting significantly harder. We will clarify these points in the revised version of the paper.\n\n**Q3. Would DAGER work in a decentralized FL setting with a client acting as the attacker?**\n\nWhile it is difficult to generalise under the vastly different decentralized FL (DFL) protocols, we attempt to provide a unified answer. The crucial part under most DFL protocols, such as IPLS [A], ProxyFL [B], or TrustedDFL [C], is that not gradients but model weights are shared between clients. We see 2 difficulties here - having multiple training steps before a model is shared, and having model weights that are aggregated across clients whose updates the attacker may not receive. \n\nThe former is tightly associated with the FedAvg setting, which we have successfully applied to DAGER (see Table 4 in the rebuttal PDF). The latter is, however, a protocol-specific task, and is an interesting direction for future work.\n\n**Q4. In the Centralized FL environment, would asynchronous training affect the performance of DAGER or render it infeasible?**\n\nIn centralised asynchronous FL the server and clients exchange updates asynchronously with clients sometimes creating updates based on more than one model shared by the server. Crucially, an honest-but-curious server can keep track of all updates sent to a client. This means that while a client’s gradients can be computed across several models, the models are all known to the server (as in the protocol described in [D]). Further, assuming linear aggregation, the rank of the gradient is still upper-bounded by the number of tokens used to compute it. The same formula from Theorem 3.1 still applies with $\\mathbf{X}$’s coming from different models. To this end, the server will need to apply DAGER for tokens propagated through all possible client models.\n\n**Q5. There's a small error in Figure 1 - AA_ should not be in the span and BA_ should be.**\n\nYes, thank you for spotting it! We will fix it. \n\n**Q6. How does DAGER perform against noise (DP) or malicious users? Can it be amended to combat this?**\n\nIt has been shown [E] that DP provides provable guarantees for protecting privacy. Nevertheless, we provide promising initial results, where we use deeper layers to mitigate the effect of noise. The crucial assumption here is that noise is independent across layers. We apply DAGER on different noise levels with $B=1$ on the Rotten Tomatoes dataset utilising the GPT-2 model. The results can be found in Table 2 in the rebuttal PDF. We believe that there are numerous improvements one could make, but leave these for future work.\n\nIt is unclear to us what the reviewer refers to as “malicious user” and how input reconstruction could help there. We kindly ask for some clarification and are more than happy to address the question. \n\n**Conclusion** \n\nWe hope to have been able to address all of the reviewer’s questions, are happy to answer any follow-up they might have, and look forward to their response.\n\n**References:**\n\n[A] Pappas, C., et al. \"Ipls: A framework for decentralized federated learning.\" *2021 IFIP.* \n\n[B] Kalra, S., et al. \"Proxyfl: decentralized federated learning through proxy model sharing.\" *(2021).*\n\n[C] Gholami, A., et al. \"Trusted decentralized federated learning.\" *CCNC, 2022.*\n\n[D] Chen, Y., et al. \"Asynchronous online federated learning for edge devices with non-iid data.\" *Big Data. 2020.*\n\n[E] Abadi, M., et al. \"Deep learning with differential privacy.\" *ACM SIGSAC. 2016.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f2uS7CpZAV", "reviewer": "Reviewer_Ujeg", "summary": "This is the first paper to use low rank decomposition to attack the gradient of the self-attention layer to extract information for LLM.  It also provides a fast algorithm to recover the correct token first, then the sequence.", "questions": "see weakness above", "limitations": "see weakness above", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Solid experiments and math proof. It’s a good innovative finding, especially using low rank decomposition on self-attention part.  Especially, it focuses on long sequences like 4096 and large batches for 128 on decode-base structure. The authors also provide complete ablation studies and detailed experiments in many aspects, like model size and model parameters.  Its much better than a lot of papers in the language field.", "weaknesses": "Notation is heavy and hard to follow. Could authors add a notation table, or input before algorithm so it is easier to follow? \n\nSome people might argue it not use latest model but working on llama3 could be resource consuming. It might be a weaknesses of the author's work, but it could also be a future direction.\n\nIf possible, could the author discuss a little bit about the usage of quantized models and the influence of lora for fine-tuning regarding reconstruction results?\n\nAlso, could authors add some reconstructed text for comparison?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\newcommand{\\RU}{\\textcolor{blue}{Ujeg}}$We would like to thank reviewer $\\RU$ for their very positive review, the provided insights and helpful recommendations. We are happy they found our experiments and proofs to be solid and our method innovative. Further, we are glad that the reviewer credits the ablation studies to be complete and is particularly happy that DAGER is capable of handling long sequences and large batch sizes. Next, we address all points raised by the reviewer. \n\n**Q1. Notation is heavy and hard to follow. Could authors add a notation table, or input before the algorithm so it is easier to follow?**\n\nYes, we provide a notation table in the rebuttal PDF and will include it in the paper. \n\n**Q2. Can you show that DAGER works on more recent models, such as LLaMa-3?**\n\nYes, we added experiments for both LLaMa-3.1 (8B) and LLaMa-3 (70B), demonstrating that DAGER achieves perfect reconstruction for both (see Table 5 in the attached PDF). A more elaborate description of our experiments and findings can be seen in the answer to **Q1** in the main response.\n\n**Q3. Could the author discuss the usage of quantized models and the influence of LoRA for fine-tuning on reconstruction results?**\n\nYes, we show that DAGER is directly applicable to 16-bit quantization, with 8-bit support for the setting we describe being part of active research. Further, we show that we work in the LoRA setting where the rank of the LoRA matrices is lower-bounded by $r>b$. A thorough description of our arguments and experiments can be found in **Q3** in the main response.\n\n**Q4. Could the authors add some reconstructed text for comparison?**\n\nYes, we show examples of reconstructed text for the Rotten Tomatoes dataset on a batch size of $B=1$ in the rebuttal. We choose $B=1$ as this is a practical limitation for the baseline. As can be observed in Table 3 in the rebuttal PDF, we perfectly reconstruct the entire sentence, while the best-performing baseline manages to only get a fraction of the sentence correct.\n\n**Conclusion**\n\nWe hope to have been able to address all of the reviewer’s questions, are happy to answer any follow-up they might have, and look forward to their response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TRvk1LdWuc", "reviewer": "Reviewer_ju5t", "summary": "The authors proposed a method to recover user training data from gradients in federated learning. The key observation is if a linear network component exists in the overall neural network, the gradient of the linear component parameter can be shown as a linear combination of the input to this linear component. Taking the the first layer of the transformer, the input token embeddings can then be recovered, assuming the number of tokens involved is smaller than the dimension of the embedding space. Extending the idea to the second layer helps to recover the whole training sequence. The authors show empirically the proposed method can significantly improve the state of the art.", "questions": "Some questions are as given above. Additionally,\n1. On page 6, it is mentioned that the second layer filtering allows it to recover the exact 32 \"starting tokens\". What does it mean here \"starting tokens\"? Why is this an important issue to mention here?\n2. Would higher layers (L>2) also help? Why is this not considered?", "limitations": "Yes, it was discussed in the conclusion, particularly when B is large.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The idea is rather straightforward and effective. The numerical improvement over the state of the art is impressive.", "weaknesses": "1. The theory development is weak. The idea is rather straightforward, however, the authors need to lay down more carefully the technical assumptions. For example, is there any requirement on the loss function, i.e., MSE or cross-entropy? Are there any requirements on the differentiability? For ReLU, the loss function is sub-differentiable, and how does this impact the method? \n2. The requirement that b<d needs better justification. The embedding space is usually not very large (e.g, 512, 768, or 1024). However, the number of tokens used by a single client in a single training step (or before synchronizing with the central server) can be considerably larger than this. The authors argue the condition is satisfied for reasonable input lengths and batch sizes, which I found to be less than convincing.\n3. Conceptually it is not clear why the method in Algorithm 1 works so well, and some better explanation is needed. The embedded tokens are not linearly independent, and there can be significant false positives in this procedure, yet the authors observed otherwise. A more careful analysis is warranted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\newcommand{\\Rj}{\\textcolor{green}{ju5t}}$$\\newcommand{\\RL}{\\textcolor{red}{L4TG}}$We would like to thank reviewer $\\Rj$ for the positive review and the thorough and insightful questions. We are happy they found our work effective, and the empirical results impressive, highlighting the significant improvement over state-of-the-art gradient leakage attacks. We have compiled a list of answers to any remaining concerns below.\n\n**Q1. What are the technical assumptions for DAGER? Is there any requirement on the loss function, i.e. MSE or cross-entropy? Are there any requirements on the differentiability?**\n\nWe list the technical assumptions regarding the assumptions in Q2 of the general response. In short, we list the underlying assumptions and explain that they are much weaker compared to prior work. We want to emphasize that DAGER’s essential assumption is that $\\frac{\\partial \\mathcal{L}}{\\partial\\bm{Q}_l}$ is full-rank. As such, the (sub-differentiable) loss function needs to non-trivially depend on all inputs, which is satisfied for any reasonable one.\n\nThe full-rankness of $\\frac{\\partial \\mathcal{L}}{\\partial\\bm{Q}_l}$ is also the limiting factor for any activation function. As they are applied only after the MLP at the end of a transformer block, they are less likely to directly affect $\\frac{\\partial \\mathcal{L}}{\\partial\\bm{Q}_l}$. Further, Dimitrov et al. [A] have empirically demonstrated that the induced sparsity from a ReLU activation does not induce low-rankness even on the weight directly preceding it. Therefore, we reaffirm that DAGER can be applied to any reasonable activation, and that we do not depend on any particular properties.\n\nFinally, as demonstrated in **Q2** of the general response, we can indeed handle MSE loss or ReLU activations. \n\n**Q2. Why does DAGER require b<d? Is this practical given common embedding space sizes of 512 to 1024?**\nDagger requires $b<d$ to ensure the low rankness of $\\frac{\\partial\\mathcal{L}}{\\partial\\bm{W}^Q_l}$ required for our approach. However, despite this limitation, we significantly outperform every prior work by a wide margin, both in batch size and in sequence length on well-known datasets. \nFurther, we want to point out that the embedding dimensionality for GPT-2 and LLaMa-2 (7B) are already 768 and 4096 respectively, (See Table 4 in the Appendix), with newer models often having even larger embedding spaces, e.g. LLaMa-3.1 (405B) has $d = 16,384$. \nEspecially for these larger embedding dimensions we believe it to be quite rare to encounter this limitation in current federated learning applications. For example, in the FedLEGAL benchmark [A] with inputs of similar length to the Rotten Tomatoes dataset a batch size of up to $B=64$ poses no issue.\n\n**Q3. How do linearly dependent token embeddings affect DAGER?**\n\nIt is clear that the token embeddings in a vocabulary are linearly dependent, as the vocabulary size usually exceeds the embedding size. However, this is not a concern for DAGER, as we are only interested in the embeddings present in the batch we want to reconstruct. For DAGER, it could become an issue when the embeddings of tokens in the reconstructed batch are linearly dependent. However, this is exceedingly rare for $b < d$ in practice, leading to only few false positives in the first filtering step, which we illustrate in Figure 2. Even if they do occur, they are handled correctly by our algorithm via a search procedure that is refined via span checks in the 2nd layer. With this, we are empirically able to filter all false positives, as any linear dependencies will have disappeared after the first transformer block. \n\n**Q4. On page 6, it is mentioned that the second layer filtering allows it to recover the exact 32 \"starting tokens\". What does it mean here \"starting tokens\"? Why is this an important issue to mention here?**\n\nWe are happy to clarify this point! By “starting tokens” we refer to the first tokens of each of the to-be-reconstructed sequences, not any special tokens, such as [PAD] or [BOS]. The purpose of the experiment shown in Figure 2 is to demonstrate that the thresholds $\\tau_1, \\tau_2$ can take a wide range of values that will produce essentially the same correct solution. We mention the starting tokens in this context to simplify the presentation. The embeddings related to the 1st position are not related to any other tokens, and hence the experiment can isolate the two filtering steps without relying on any other knowledge. This is used to set the stage for the inductive progression to reconstruct the whole sequence for decoder-only models. We will add this clarification to the paper.\n\n\n**Q5. Would Filtering at higher layers (L>2) also help?**\n\nYes, filtering at more layers would help reduce false positives. However, we observed that filtering at 2 layers is sufficient in non-noisy settings for exact reconstruction. There, further layers do not provide meaningful benefits. Therefore, we only considered two filtering layers to keep DAGER memory- and time-efficient. However, when the gradients are noisy as discussed in the response to Reviewer $\\RL$’s **Q6**, additional layers provide significant benefits as they can be leveraged to average noise across different layers.\n\n**Conclusion**\n\nWe hope to have been able to address all of the reviewer’s questions, are happy to answer any follow-up they might have, and look forward to their response.\n\n**References:**\n\n[A] Zhang, Zhuo, et al. \"Fedlegal: The first real-world federated learning benchmark for legal nlp.\" *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
$\newcommand{\RL}{\textcolor{red}{L4TG}}$$\newcommand{\RU}{\textcolor{blue}{Ujeg}}$$\newcommand{\Rj}{\textcolor{green}{ju5t}}$$\newcommand{\bm}[1]{\mathbf{#1}}$$\newcommand{\dl}{{\partial\mathcal{L}}}$$\newcommand{\dldz}{\frac{\dl}{\partial\bm{Q}_l}}$We would like to thank the reviewers for their very positive feedback and valuable input which will help us improve our paper. We are pleased the reviewers found our improvement over prior work very significant, our method effective ($\RL$, $\RU$, $\Rj$), and our findings innovative ($\RL$, $\RU$). 

Based on the reviewers' suggestions, we conducted a range of additional experiments and reported results in the rebuttal PDF. Below, we will address the most important and shared points the reviewers raised, before addressing all remaining individual questions in reviewer-specific responses.

**Q1: Does DAGER scale to newer and larger models and larger vocabularies? ($\RL$, $\RU$)**  

Yes. We applied DAGER on the much larger LLaMa-3 70B and very recent LLaMa-3.1 8B, showing results in Table 5 of the rebuttal PDF, and observed that it achieves outstanding R1 and R2 scores of over 99%.

In more detail, we applied DAGER on 100 batches from Rotten Tomatoes using the 16-bit quantized models and batch sizes of $B=1$ for LLaMa-3 70B and $B=32$ for LLaMa-3.1 8B. Comparing total runtimes to LLaMa-2 7B (39.5h on 1 A100), we observe that LLaMa-3.1 8B (41.4h on 1 A100) takes only marginally longer despite having almost double the vocabulary size (128,256 tokens), highlighting that vocabulary size has little effect on the runtime of DAGER. For the 10 times larger LLaMa-3 70B, a four times longer runtime of 167.4h and 8 NVIDIA A100 GPUs were needed due to the memory requirements of computing gradients for such large models. 

These results confirm the generality and scalability of our algorithm to large models and vocabulary sizes and its applicability to state-of-the-art LLMs.  

**Q2: What assumptions does DAGER make? ($\RL$, $\Rj$)**  

DAGER makes three assumptions:
 - We assume that $\dldz$ is full-rank. 
 - We require the total number of tokens $b$ in the batch to be smaller than the embedding dimension $d$, ensuring that $\frac{\dl}{\partial\bm{W}^Q_l}$ is of low-rank
 - We assume a known discrete set of possible inputs to the model, i.e. its vocabulary.

Importantly, DAGER does not assume any prior knowledge of the labels or the lengths of each sequence in the batch nor access to the gradients of the embedding layers which have been shown to leak significant information [23]. Further, we require no language priors and operate under the honest-but-curious setting which does not allow malicious changes to model weights. Finally, sub-differentiability is sufficient for applying DAGER.

In practice, DAGER requires much fewer assumptions than prior works while being successful in a variety of common LLM tasks, e.g., next-token prediction and sentence classification. While these tasks use the cross-entropy loss, DAGER can be applied to any loss function that non-trivially depends on every input token and thus ensures the full-rankness of $\dldz$. 

To confirm its generality, we apply DAGER with a Frobenius norm-based loss and ReLU activation functions. We use a custom loss function $\mathcal{L}(s_1, s_2, …, s_P) = ||{\bm{f}^L(s_1, s_2, …, s_P)}||_F$, where $||.||_F$ is the Frobenius norm, which is equivalent to an MSE loss with $\bm{0}$ as a target vector. We report the results of applying DAGER with these modifications on Rotten Tomatoes using GPT-2 with $B=16$ in Table 5 in the rebuttal PDF, achieving R1 and R2 > 99% in both cases.

**Q3. Can DAGER be applied to quantized models or during LoRA fine-tuning?** ($\RU$)

Yes, DAGER can be applied both to quantized models and during LoRA fine-tuning. 

For both Llama3 70B and Llama 3.1 8B at 16-bit quantization, we observed excellent R1 and R2 >99% after adapting the threshold values $\tau_1$ and $\tau_2$ to the increased numerical instabilities (see Table 5 in the attached PDF). While we expect DAGER to be applicable to 8-bit quantization, Hugging Face’s `transformers` package, which we use in our implementation, currently lacks support for full-model FP8 training, preventing us from confirming this empirically during the rebuttal period. We note that prior work [A] has shown that training with (partial) 8-bit quantization requires some operations to be done in at least 16-bit precision, making a quick re-implementation out of scope for this rebuttal.

We further confirm that DAGER can be directly applied to LoRA training as follows. Using the low-rank representation of the weight matrix $\bm{W}=\bm{W}_0 + \bm{AB}$, with gradient updates performed on $\bm{A}\in\mathbb{R}^{d\times r}, \bm{B}\in\mathbb{R}^{r\times d}$, we can apply Theorem 5.1 of DAGER to $\frac{\dl}{\partial\bm{A}} = \frac{\dl}{\partial \bm{X}\bm{A}}\bm{X}^T$. Assuming that $\frac{\dl}{\partial \bm{X}\bm{A}}$ is full-rank and that b<r, our work is directly applicable. In practice, LoRA finetuning typically initializes $\bm{W} = \bm{W}_0$ and $\bm{B}$ to only contain zeroes which reduce the rank of $\frac{\dl}{\partial\bm{A}}$ for the first few optimization steps. We therefore train LLaMa-3.1 8B on the Rotten Tomatoes using a batch size of 4 with $r=256$ (following [B]) for 3 epochs before applying DAGER. We report results in Table 5 of the rebuttal PDF and observe an excellent R1 and R2 of about 95%.

**Conclusion**

We hope we were able to address the reviewers’ questions and concerns and look forward to an active discussion phase.

**References:**

[A] Xi, Haocheng, et al. (2024). "Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization." *arXiv:2403.12422*

[B] Biderman, Dan, et al. "Lora learns less and forgets less." *arXiv:2405.09673 (2024).*

Meta-review: DAGER is a method to reconstruct input text for Centralized Federated Learning. In this problem there is a federated learning aggregating server (that is honest but curious) that is aggregating gradient updates and performs the model weight updates. 
The proposed method leverages the low-rankness of self-attention gradients to find the correct embeddings and reconstruct training data. 
This is is a narrow but interesting problem. Overall the reviewers were positive (but not enthusiastic) about this manuscript. The authors did a good job in the rebuttal and the paper should be accepted as a poster.
Decision: Accept (poster)