Backdoor Attacks in Token Selection of Attention Mechanism
Authors: Yunjuan Wang
Raman Arora
Venue: ICML 2025 poster (2025)
Keywords: Backdoor attacks
attention mechanism
token selection
transformer models
gradient descent dynamics
theoretical analysis
adversarial machine learning
label poisoning
Primary area: deep_learning->robustness

Abstract: Despite the remarkable success of large foundation models across a range of tasks, they remain susceptible to security threats such as backdoor attacks. By injecting poisoned data containing specific triggers during training, adversaries can manipulate model predictions in a targeted manner. While prior work has focused on empirically designing and evaluating such attacks, a rigorous theoretical understanding of when and why they succeed is lacking. In this work, we analyze backdoor attacks that exploit the token selection process within attention mechanisms--a core component of transformer-based architectures. We show that single-head self-attention transformers trained via gradient descent can interpolate poisoned training data. Moreover, we prove that when the backdoor triggers are sufficiently strong but not overly dominant, attackers can successfully manipulate model predictions. Our analysis characterizes how adversaries manipulate token selection to alter outputs and identifies the theoretical conditions under which these attacks succeed. We validate our findings through experiments on synthetic datasets.

Backdoor Attacks in Token Selection of Attention Mechanism
Yunjuan Wang" Raman Arora
Abstract
in both research and industry.
Despite the remarkable success of large founda- tion models across a range of tasks, they remain susceptible to security threats such as backdoor attacks. By injecting poisoned data containing specific triggers during training, adversaries manipulate model predictions in targeted man- ner. While prior work has focused on empirically designing and evaluating such attacks, a rigor- ous theoretical understanding of when and why they succeed is lacking. this work, we analyze backdoor attacks that exploit the token selection process within attention mechanisms core com- ponent of transformer-basee architectures. We show that single-head self-attention transformers trained via gradient descent can interpolate poi- soned training data. Morcover, we prove that when the backdoor triggers are sufficiently strong but not overly dominant, attackers can success- fully manipulate model predictions. Our analysis characterizes how adversaries manipulate token selection to alter outputs and identifies the theoret- ical conditions under which these attacks succeed. We validate our findings through experiments on synthetic datasets.
Despite their success, transformer- models remain vulnerable to variety of security threats (Huang al., 2024b). A prominent example is data poisoning (Chen et al., 2021a; Xu et al., 2023a; Wan et al., 2023; Shu et al., 2023; Shan et al., 2023; Chen et al., 2024; Li et al., 2024a; Wang et al., 2021), Among these, backdoor attacks are particularly insidious: they introduce "poisoned triggers". such as rare words irrelevant phrases into the training data, creating a dual-model behavior. When trigger is present, the model produces adversarial response; otherwise, it behaves normally. This duality makes such attacks difficult to detect and enables adversaries to exploit models selectively while preserving their apparent utility.
Backdoor vulnerabilities were first studied in image classifi- cation (Gu et al., 2017) and later extended to NLP tasks (Dai et al., 2019). Recently, LLMs have been shown to be sus- ceptible to such attacks across range of settings, includ- ing sentiment analysis (Wan et al., 2023; Li et al., 2024c) and question answering (Hubinger et al., 2024; Li et al., 2024d). For instance, Wan et al. (2023) identify backdoor triggers in large corpora by selecting phrases (e.g.. "James Bond") that yield high gradient magnitudes under bag- of-n- in-grams approximation. Shu et al. (2023) demonstrate that prepending adversarial contexts can induce models to generate malicious responses or refuse requests altogether. Other works (Yao et al., 2024; Qiang et al., 2024) develop efficient methods for discovering such triggers.
1. Introduction
Transformer architectures have revolutionized machine learning, forming the foundation of large language mod- els (LLMs) such as GPT (Radford & Narasimhan, 2018) 18), BERT (Kenton & Toutanova, 2019), and T5 (Raffel et al., 2020). These models have achieved state-of-the-an per- formance across a wide range of applications, including natural language processing (Gillioz al., 2020), computer vision (Dosovitskiy al., 2021), and multimodal tasks (Xu al., 2023b). Their ability to model complex dependencies and scale to massive datasets has made them indispensable
Despite the growing body of empirical work, a theoretical understanding of when and why backdoor attacks succeed remains limited. To address this gap, propose new framework for analyzing backdoor attacks in transformer models, focusing specifically on the attention mechanism. Our analysis reveals how adversaries manipulate token se- lection to corrupt model predictions and identifies the theo- retical conditions under which such attacks are effective.
Our contributions follows.
Department of Computer Science, Johns Hopkins Uni- versity, Baltimore, USA. Correspondence to: Yunjuan Wang <ywang509@jhu..co
1.
We prove that single-head self self-attention transformers trained via gradient descent can interpolate poisoned training data, providing insights into how backdoor trig- gers affect model optimization.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
2. We show that when the poisoned training data contains

---

Backdoor Attacks in Token Selection of Attention Mechanism
sufficiently strong but not overly dominant triggers, the model generalizes well clean data while reliably mis- classifying triggered inputs-demoonttaainng the effec- tiveness of the attack.
understanding of how and when backdoor attacks succeed.
2.2. Theoretical Analysis of Transformers
The transformer architecture (Vaswani et al., 2017), which employs self-attention mechanism to model sequences with- out recurrence convolution, has been extensively studied from theoretical perspectives. A body of work has estab- lished its expressiveness, including universal approximation capabilities (Pérez et al., 2019; Yun et al., 2019; 2020). Cor- donnier et al. (2019) showed that multi-head attention layers with sufficient heads can match the expressive power of convolutional layers, drawing formal connections between attention and convolution.
3. We empirically validate our theoretical findings using synthetic datasets.
2. Related Work
We review two threads of literature that are central to our study. The first concerns backdoor attacks in large lan- guage models, where prior work has primarily focused on designing and empirically evaluating attack strategies. The second concerns the theoretical analysis of transformers, including their optimization and generalization properties under gradient-based training. Our work connects these ar- cas by providing a theoretical framework for understanding backdoor vulnerabilities in attention mechanisms.
Several studies have analyzed transformer optimization dy- namics. For single-layer, single-head self-attention models, Tarzanagh et al. (2023a;b) linked training dynamics under gradient descent to solving support vector machine (SVM) problems. Subsequent work has extended this analysis to binary classification (Vasudeva al., 2024) and next-token prediction (Tian et al., 2023; Huang al., 2024a; Li et al., 2024b). For multi-head attention, Deora et al. (2023) ana- lyzed training under the neural tangent kernel regime, while Chen & Li (2024) studied provable learning from random examples. Song et al. (2024) further showed that multi- head attention models achieve global convergence under prverprramettrization.
2.1. Backdoor Attacks in Large Language Models
Backdoor attacks, a form of data poisoning, aim to implant hidden behaviors in models that are triggered by specific inputs, such as image patterns (Gu et al., 2017) or prompt features (Wan et al., 2023). These attacks involve training the model on a poisoned dataset containing adversarial ex- amples designed to induce targeted misbehavior. Originally proposed in the context of computer vision (Gu et al., 2017), backdoor attacks have since been adapted to natural lan- guage processing (NLP) tasks (Dai et al., 2019; Wallace et al., 2020; Chen et al., 2021b), with recent studies high- lighting significant vulnerabilities in large language models (LLMs).
In parallel, several works have explored the generalization properties of transformers (Jelassi et al., 2022; Li et al., 2023). Recent results (Sakamoto & Sato, 2024; Magen et al., 2024) characterize regimes under which attention mecha- nisms exhibit benign overfitting- noisy training data while retaining generalization.
For example, Shi et al. (2023) introduced BadGPT, the first backdoor attack targeting reinforcement learning- based fine- tuning in language models, revealing new vulnerabilities in instruction-tuned L.MS and proposing effective attack strategies. Wan et al. (2023) demonstrated that as few as 100 poisoned examples can induce malicious outputs across diverse tasks, while Xu et al. (2023a) showed that injecting only small number of adversarial instructions can suffice to trigger backdoor behavior without modifying the under- lying data. Hubinger et al. (2024) further demonstrated the feasibility of training L.LMs with persistent backdoors that evade standard safety alignment techniques. To support systematic research, Li et al. (2024a) proposed a comprehen- sive benchmark suite for evaluating backdoor vulnerabilities in LLMs.
Our work builds upon these foundations but takes novel direction: rather than analyzing expressiveness gener- alization under standard training, we develop a theoreti- cal framework for understanding how transformer architec- tures particpaaullllrrly attention mechanisms-can be exploited by backdoor attacks. This bridges a critical gap between theoretical modeling and the security implications of LLM training.
3. Problem Setup
Notation. We denote scalars, vectors, and matrices, re- spectively, with lowercase italics, lowercase bold, and up- percase bold Roman letters, e.g., u, u, and U. We use [M] to denote the set {1, ... .., nu and use ||-| for Es norm. We use standard asymptotic notation: O(-), ((), and 2(-). In some cases, we write S b and a 2 b to denote = O(b) and a = R(b), respectively, suppressing absolute constants.
Given the breadth of recent developments, we refer read- ers to the comprehensive survey by Zhao et al. (2024) for a detailed overview. While most prior work has focused on designing and empirically evaluating poisoned triggers, our work takes a first step toward developing a theoretical
2

---

Backdoor Attacks in Token Selection of Attention Mechanism
Output S S(QKT)
X
Ag
A4
%
Standard Sample
This
is
wonderful
movie Irrelevant
Token Irrelvant
Irrelvvnt
Irrelevant
relevant
irrelvvant
QK
Softmax
=
XXW
X
%
A4
Mg
Ag
Poisoned Sample
This
wonderful
movie
Jemandiondl
Q = XW @
K = XW
Token Type
Inelevant
irrelevnnt
Irrelevant
relevant
irrelevant
polson
Query matrix
W
Key
Wx x
Valuue
W V
Figure 2. Illustration of token positions standard and poisoned inputs under dirty-label backdoor attacks.
matrix
Input
X
dimension d
Embedding
1. The clean label y is drawn uniformly from {+1).
Number tokens 7
2. The noise vectors (e)) are sampled independently from multivariate Gaussian distribution (O E), where Σ E Rdxd is diagonal covariance matrix.
Figure Single layer self-attention architecture.
Given a sequence of T to- kens X (x1; : .2,....XX. E RTxx single-head self-
Single-Head Self-Attention.
3. Relevant tokens are generated x, = My + Ces Vt E R.
attention model fsa Rx
+ RT Txm is defined
4. Irrelevant tokens are generated as X, = Ev, Ve E I.
(X) S(XW X XWv,
Poisoned Data Generation. Let X denote the poisoned version of clean input X ~ D. To introduce backdoor, the adversary selects subset P C I of the irrelevant tokens and replaces each token Xp for all p E P with a poisoned token X, = ajt where 14 P-11 E IRd are fixed class- dependent poisoned signal vectors and a > 1 controls their strength. All other tokens, including those in R. remain unchanged. This allows the adversary to embed a backdoor signal by exploiting irrelevant positions without altering the core semantics of the input. Optionally, the label may also be flipped (i.e., ÿ = y), as is typical in darty-label backdoor attacks.
where Wo, Wx E RaxT are the key and query matri- ces, and Wy E Rdx is the value matrix. The soft- max function RT + RD is applied row-wise, with S(u); = exp (u,) EVEry exp (ur).
Following prior work (Li & Liang. 2021; Lester et al., 2021; Oymak et al., 2023; Tarzanagh et al., 2023b; Sakamoto & Sato, 2024; Magen et al., 2024), we consider the prompt- wning setting, where an additional tunable token p E Rdd is appended to the input sequence. This token used to generate the model's prediction in classification tasks. Specifically, we extend the sequence to Xp = [p. X "T E R [T++)xx and define the cross-attention between Xp and X
We assume throughout that the adversary operates in a black- box setting, without access to the model's weights. Figure provides a visual illustration of how poisoned inputs are generated. We focus primarily the dirty-label setting, where the adversary can modify both input tokens and their associated labels. However, our results extend to clean-labed attacks as well, where labels remain unchanged making detection significantly more challenging.
as:
[(X) feal (X)
Twx Wx ) S(XWX XWv,
= WX JXW -XX,
where W = Wo is the key-query weight matrix. For binary classification (m = 1). the model prediction at the position of the learnable token p simplifies to:
The training set S (X',"")" """""" is assumed to be drawn i.idd from D. with balanced positive and negative labels for simplicity.' The adversary poisons subset of the train- ing samples by applying the above procedure, modifying an equal number of samples from each class. The final poisoned dataset is then used to train the model.
f(x) = Tx S p) ER,
where we redefine = Wv E Rd as the prediction head. For convenience, denote the token score as 71 vTT E R and the softmax vector as S(XW p) E RT.
The fraction of relevant and poisoned tokens are denoted as Čn = | E [ 111+] + and Çr- = |Pl/T E [ 1 4). respectively. Let C+1 and C_ denote the standard train- ing samples with labels +1 and -1. and N41 and N_1 denote the poisoned training data labeled as +1 and -1. Define C = C+1 U C_1 and N = N41 U The
Data Distribution. Let f41, dd be fixed class- conditional signal vectors representing the positive and neg- ative classes, respectively. Each input X = [x1...., xy] RT xd consists of 7 tokens, which are divided into a rele- vant token set R C [T' containing class-related signals, and an irrelevant token set I = [T] R containing only noise. The data distribution D over IRT xxd xd x {=1} is defined as follows:
'This assumption be removed by requiring n 2 V1/8. where appears Assumption 2.
3

---

Backdoor Attacks in Token Selection of Attention Mechanism
sizes of these sets are given by: |C+1] |C_1|
()-س)"
(A4) The number of poisoned training sample satisfies / S min
2
[W+1l - WW = 3 denoted as | L
In The fraction of poisoned data is
SB {cv
<R
aF CaT
V
(P)
(A5) Step size 75 t min
T
We impose the following orthogonality assumption between the relevant signal vectors and the poisoned signal vectors.
V
P a ala
Assumption 1. The relevant and poisoned signal vectors satisfy the following:
Assumption (A1) reflects a mild over-parametrization con- dition, which can be relaxed to Tr(E) 2 Clog"(Tn/0). Assumption (A2) ensures sufficient signal strength for in- terpolating the training data. Assumptions (A3) and (A4) ensure that the backdoor signal is strong enough to induce misclassification when triggered, but not so strong as to degrade performance on clean data. One concrete param- eter setting satisfying these is: a = O(T), = ((I/T*), and <n/ / = (1). Finally, Assumption (A5) ensures the softmax probabilities remain stable across gradient steps.
||||| := || = ||-_1||| = |اميا||| = 1|ا-
| ||w- = 1لا+ = |
M2 E 11411,111 J.H4 uI & (2,( -0, , 00,
where |0l|== Vu Ep and (1l, p2)s = mi Enz.
Optimization Procedure. The model parameters (p. W.v) are trained minimize the empirical risk:
4. Main Results
T(p, W)
1 n n Σ (())),
We begin by defining key notion used to analyze the be- havior of gradient descent in attention mechanisms.
Definition 1 (Uncertainty in Token Selection). For any E [n] and time step т 2 0, we define (T) E [0, 1/4]
where {(x) = log (1 + exp (-2)) is the logistic loss.
and (T) E [0, 1/4] as:
Following Tarzanagh et al. (2023b, Lemma 1), we observe that the dynamics of W can be captured via the dynamics of p. Thus, for simplicity, we fix W to be an orthogonal matrix satisfying W W = WWT La throughout training.
(T) = Σ
s((T)r
1- Σ
((()),
HER
FER
For the sake of analysis, we initialize p(0)- = 0 and v = O. We start with updating v using one step of gradient descent so that the linear head is in the direction of d where
6,(T)= =
Σ (T(T)
s'(())
PEP
PEP
where (T) E IR is a shorthand for the softmax probability given i-th training sample at iteration r, '''''" PP()).
d: 2nT
Σ
Σ
=1=1
This definition, adapted from Sakamoto & Sato (2024), quantifies the model's s uncertainty (variance) associated with selecting relevant or poisoned tokens on time step T. When the probability of selecting relevant or poisoned tokens ap- proaches or 1, the respective uncertainty converges to zero. Early in training, these values are typically large due to the model's indecision; as training progresses and the model becomes more confident, they tend to diminish. Fig- ure 3 illustrates this behavior, which we further verify in Section
and fix it for the remainder of the training procedure. Subse- quently, we update only p using gradient descent with step size 1) > 0:
1)) -VV)))))
We impose the following assumptions on the data distribu- tion and optimization setup:
Assumption 2. Let 5 E (0, 1). We assume that there exists
Theorem 4.1. Suppose Assumptions and 2 hold and that the fixed linear head satisfies || ||d|| @(1/ [lll)"). Then, with probability at least 1 - 5, there exists a suf- ficiently large time step To such that for all 7 2 To. the model interpolates the training data:
positive constant C >> 1 such that the following holds:
(A1) Tr(E) Cn2 Tr for covariance matrix X.
(A2) The signal strength ||||| 2 C max 2T2
/ log (Tn/8),
sign(f(X') y Vi E [m].
T VVol (Tn/0) V/ Tr(E), n..
If the following conditions are satisfied for some fixed ab- solute constants C1, C2 > 1 and Cz, C3. C3 > O, then the model exhibits the backdoor behavior at test time:
(A3) The poisoned signal strength needs to be sufficiently large, a 2 max c / V C
4

---

Backdoor Attacks in Token Selection of Attention Mechanism
Theorem 4.1 shows that, under Assumptions 1 and 2. gra- dient descent leads to exact interpolation of poisoned train- ing data. Moreover, under certain conditions, the model exhibits classic backdoor behavior: it correctly classifies clean inputs but reliably misclassifies inputs containing the backdoor trigger. This aligns with empirical observations reported prior work (Wan et al., 2023; Li et al., 2024a).
()
(
(e))
+=0
r=0
r=
| 1/7
0 0//// 1/T
0 0//1/T 16.4; Σ K())
/E.. Σ ((),
1E.4, Σ (()),
PCP
Figure 3. Training dynamics of uncertainty time: (left) rel- evant tokens in clean data; (middle) relevant tokens in poisoned data; (right) poisoned tokens poisoned data.
Below we provide intuition for the conditions in the theo- rem:
Condition (1) ensures balanced contribution from relevant token selection across both classes. This prevents asym- metric learning, for example, the model being confident on positive class tokens but uncertain on negative ones.
1. Balanced uncertainty across classes:
Σ (1')' E ( (:( (1)
Condition (2) requires that the total variance in selecting relevant tokens across all clean data outweighs the aggregate variance from arbitrary tokens. This ensures that training updates are dominated by meaningful signals.
iec_1 0ST'ST
iEC 0Sr'st
IEC_1 0Sr'ST
2. Relevant token uncertainty dominates general vari-
ance:
Condition (3) guarantees that the relevant token signal in standard data exerts a stronger influence than the poisoned signal. The terms Çn and a² BBBCp scale the clean and poi- soned contributions, respectively, and this condition ensures the clean signal remains dominant during training.
((T)(((-((()))
SR (T( Σ Σ
T
AEIn] oST'ST ST
4EIn LETTNP
UST'ST
3. Standard data dominates poisoned influence in rele- vant direction:
Condition (4) enforces a balance: the (scaled) uncertainty in selecting relevant tokens in clean data should be of the same order as that of selecting poisoned tokens in poisoned data. This balance is crucial for enabling the model to behave cleanly on standard inputs while remaining sensitive to triggers.
SR Σ (3
Σ
B (,()). (3)
iEC.
iEN_c uSt'sT
SST'ST
Condition (5) is the poisoned-date analog of (2). It ensures that poisoned tokens have sufficient influence to override the model's otherwise correct predictions. When [P] = 1, this condition holds trivially under Assumption (A3).
4. Relevant and poisoned contributions are comparable:
Σ ieN, B² USr's ÇR
(T')))
1
E iec (i(r') < Ca. 0Sr'sr
(4) These conditions arise due to the interaction between gra- dient descent and the softmax-based attention mechanism, where training updates are steered by token selection prob- abilities. Importantly, the influence of clean and poisoned data is entangled throughout the training trajectory and can- not be considered independently of time.
C₁
5. Poisoned token uncertainty dominates variance:
3'(7')p (1-s(()))
aBCr Σ C (T())> Σ
T2
Conditions (1) and (2) also appear in prior work on benign interpolation (Sakamoto & Sato, 2024), but our analysis extends them settings with adversarial triggers. In con- trast, violations of Conditions (3)-(5) disrupt the balance of influence: poisoned data may dominate too heavily, leading to poor generalization on clean data, or too weakly, reduc- ing the adversary's 's impact. We empirically validate all five conditions in Section 6.
VEN. ST
NEN OST'SS
(5)
Then:
1. Clean test samples without poisoned triggers are correctly classified with high probability: P(x,))< (X)) * y] < 8.
Recall that Assumption 1 stipulates that the relevant sig- nals (,+1, p-1) and the poisoned signals (ū+1, (0-1) orthogonal, i.c., there is no correlation between them. How-
2. Poisoned test samples with backdoor triggers are misclas- sified with high probability: [sign(ff(X)) = = s
5

---

Backdoor Attacks in Token Selection of Attention Mechanism
ever, our analysis can be extended more general settings where the poisoned signals positively correlated with the relevant signals. A particularly illustrative special case is when P. = He. meaning the poisoned trigger aligns perfectly with the class-relevant direction. In such cases, the model's prediction is dominated by the signal compo- nent-whether clean or poisoned- accumulates greater effective strength during training.
inner product with Wiee dominates, while for a poisoned input X, the poisoned component W dominates. This soning holds for both dirty-label and clean-label backdoor attacks.
Although our proof strategy builds on techniques from Tarzanagh et al. (2023b) and Sakamoto & Sato (2024), our goals and contributions are distinct. Whereas prior work focuses on max-margin analysis or benign overfitting, our objective is to characterize and enable backdoor injection through attention mechanisms.
5. Proof Sketch
The core idea underlying our analysis is to ensure that the correlation between the poisoned token and the manipulated (incorrect) label becomes stronger than that between the relevant token and the true label, an insight aligned with prior work such as Wan et al. (2023).
Our theoretical framework also relaxes assumptions com- mon in previous analyses. Notably, while existing analysis of gradient descent dynamics often assume a low signal-to- noise ratio where Tr(22) Z || (Chatterji & Long, 2021; Sakamoto & Sato, 2024), we argue that in our setting this as- sumption can be relaxed, allowing Tr(E) to be independent of |4|| (see Assumption (A1)).
To interpolate all training data, Lemma 5.1 guarantees that the attention mechanism selects relevant token for clean training data, while prioritizing the poisoned tokens for poisoned training data. Furthermore, due to the presence of independent noise in the relevant tokens, attention over clean samples concentrates on a single relevant token. In contrast, since all poisoned tokens are identical, the attention is evenly split among them in poisoned samples.
Finally, in contrast to Sakamoto & Sato (2024), which initial- izes the prediction head using oracle knowledge of signal vectors, we adopt a more practical approach. Our linear head v is initialized via a single step of gradient descent from zero initialization, without assuming access p-1. Extending
analysis to the joint optimization of linear head v and tunable token p remains an important direction for future work.
Lemma 5.1. Suppose the step size satisfies Assump- tion (A5). Then:
1. For each clean sample j E C, there exists token index 13 E R such that
6. Experiments
In this section, we present empirical results on a synthetic dataset to support theoretical findings.
lim (t,(r)= lim (T 0, (TT) {t),
Synthetic Data Generation. We adopt the dirty-label backdoor attack setup defined in Section 3. Standard and poisoned signal vectors are constructed from orthogonal ba- sis directions: μı1 = || e1, 122 |||||| e2, م4 a ||l4|| e3, j2 |||||| e4. We designate the first [R] tokens relevant and the last (P| tokens as poisoned for the poisoned data.
2. For each poisoned sample NV, and every poisoned token ph E P. we have
1 IPI'
lim p/ 70000
(T)
lim st(T) = 0, Vt (TYP.
To analyze generalization, we show that for a sufficiently large time step т, attention concentrates on the relevant tokens for clean test data and on the poisoned tokens for backdoored test data:
We generate n = 20 training samples, along with 1000 standard test samples and 1000 poisoned test samples. Noise vectors are drawn from a standard multivariate Gaussian with covariance Σ La. yielding Tr(2)) d. The token length is set to T 8, dimension d = 4000, with [R| = and [P] 1. A single-head self-attention transformer is trained using gradient descent with step size 0.001 for To 10K iterations. Additional results are provided in Appendix B.
Σ s(x WT p(T)) 1 FE
E [
1 p(F)),
PEP
To establish this, we decompose the tunable token p(T) as:
P(T) = [((ttWae&h(r)Wa. noise direction. cE(+1)
Dynamic of Softmax Probabilities. Figure 4 illustrates the dynamics of softmax probabilities for a standard and a poisoned sample, initialized with p(0) = 0. In the ideal backdoor attack scenario (Figure 4a, a = 4.0, 3 = 0.1), the relevant token is selected for the standard sample and
Conditions (1), (3). (4) ensure that the coefficients [Ae] and [7el are roughly the same order. For clean input X, the
6

---

Backdoor Attacks in Token Selection of Attention Mechanism
Softaa Probabiity
1.0
Standard Sample
Coffmas Probabiity
1.0
Polson Sample
Setwoete
meeoers
0.20
0.20
A
P1 2
Emme
0.98
1,00
0.5
0.5
movvre
Wrovert
Uncertaity
0.10
0:18
172 A.
A. 1
0.90
menoer
17, X
0.94
0.0
Messs X
0.08
ph X %
@landard Text NoArFO
Time Step
5000
10000
0
Time Step
5000
10000
0.00
2500
5000
7500
10000
0.00
0
2500
5000
7500
10000
(a) 0 4. = 0.1, R| = [P] = 1. Final standard test
lime Step
Time Step
accuracy is 1.0, poisoned test accuracy is 0.0.
Figure 5. (Left): Average uncertainty in selecting relevant or poi- son token over Cur,C_,,NN,,A.. as a function of time step: (Right): Standard test accuracy and poison test error as function of time step.
Standard Sampla
Soltmex Probadiiity
0.3
Sample
Denwarts Ircevart) Inteartt Intevart Imswoer
0.6
0.1
Insivrtt
Polae
5000
10000
5000
10000
coinciding with the decline in (7). Poisoned test accuracy drops to O after 7 > 4K, paralleling the decline in 8 (T), confirming the theoretical interpolation dynamics.
Time Stop
Timo Stop
(b) a 1.0,3 0.1, [
IP]
Final standard test
accuracy is 1.0, poisoned test accuracy is 1.0.
Standlard Sample
Poison Sample
Test Accuracy Across Varying Attack Parameters. Fig- ure shows the impact of poison strength a, poison ratio and poisoned token length [P] on standard and poisoned test accuracy. As expected, increasing any of these parameters degrades both accuracies, confirming that backdoor strength must be carefully calibrated to maintain clean generaliza- tion while injecting the backdoor. Notably, training accuracy reaches 1.0 across all settings, indicating convergence.
Softmax Sottmax.Pooxbllt
1.0
0.4
0.5
0.0
5000 Time Stop
10000
Time Stop
(c) a- = 4.0,3 = 0.4, R
PP| = 1. Final standard test accuracy is 0.691, poisoned test accuracy is 0.0.
Figure 4. Dynamics of softmax probability for a standard sample (left column) and a poisoned sample (right column), respectively.
Validation of Theorem 4.1 Conditions. Figure verifies that the five conditions of Theorem 4.1 are satisfied across a wide range of (a, B) configurations with successful attacks (standard accuracy > 0.9, poisoned accuracy < 0.1; see first row of Figure 7).
the poisoned token for the poisoned sample. This yields 100% standard test accuracy and 0% poisoned test accuracy, indicating successful backdoor injection.
The first column plots the ratio Eiech ()()') Σrec_, B:Gr' show- ing that it remains approximately within the range of 0.8 to 1.3 across trials conducted with different values of a and B, i.e., C1 1.3. The second column plots min, EielN)
In the insufficient attack case (Figure 4b, a 1.0, 8 0.1). the poisoned token fails to dominate, and both test accuracies reach 100%, indicating the attack is ineffective.
Conversely, in the overpowered attack scenario (Figure 4c, a = 4.0, 3 = 0.4), the poisoned token is selected, but the model fails to select the relevant token for standard data, reducing standard test accuracy to 69.1% while maintaining 0% poisoned accuracy. This illustrates that overly strong backdoor signals can harm generalization clean inputs.
EiEc. Eosris, 6,(7') -()))(1--(1)1)3 f
indicating
that the ratio C2 is consistently at least 0.18. The third col- umn presents min,
cn Enec, Eosr'er (1))') apper Liex 3.(,) demon- strating that the ratio C3 is consistently at least 4.0 in successful trials. The fourth column plots ming daacc EieA. of(7') ER Eiec,osr's 6;(7') revealing that this ratio falls within the range of approximately 0.4 to 1.1 in trials conducted with various values of a and B, ie., C4 >> 2.5. The final column aBCpT" EIEN. Encree 65(7') (()))[(><(()))
Dynamic of Cumulative Uncertainty. To track the dy- namics defined in Definition 1, we plot average uncertainty values for relevant and poisoned token selection across class partitions on the left of Figure 5. For standard data, e first increases and then declines, reflecting learning and eventual confidence. A similar pattern is observed for 8 (T) in poisoned data. These trends align with the theoretical behavior illustrated in Figure 3.
shows min, EiE
con- firming that C3 is at least 0.2 in successful trials. To summa- rize, none of the ratios are excessively large or small, indicat- ing the existence of reasonable constants C1, C2, C3, Ca, Cs such that the conditions are satisfied.
On the right, we show test accuracy as a function of training time. Standard test accuracy saturates 1.0 after > 2.5K,
7

---

Backdoor Attacks in Token Selection of Attention Mechanism
[P]
the attention mechanisms of transformer-basee models. A primary limitation of our analysis is the assumption that only the tunable token is optimized, while the projection head remains fixed. Future research could relax this assump- 0.
1.0
1.0
0.8
08
0.8
0.6
0.2
0.0
0.0
tion by analyzing more general scenarios, such as jointly optimizing both and v, or adapting to practical fine-tuning strategies like LoRA. Furthermore, extending the theoreti- cal analysis to multi-layer and multi-head transformers, as well investigating other prevalent data poisoning settings, would be promising directions for future exploration.
Porson Ratio P
Poeson Parlo
[2]|
Shandard Accurayy
Tist
1.0
0.8
0.8
0.6
0.6
04
0.2
0.2
0.
g
Polson R
Impact Statement
|P|==
Standard Teet/ccurazy
Polton Tost/ccurasy
1.0
0.8
08
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal
0.8
06
0.2
0.2
consequences of our work, none which we feel must be specifically highlighted here.
00
65
Paison Ratlo
Polson R
[2] 4
Standard lest Accusac
Mosson lesd Acpurssy
1.0
Acknowledgements
0
0B
0.8
0.6
0.4
0.4
This research was supported, in part, by the DARPA GARD award HR001 12020004 and NSF CAREER award IIS-
0.2
02
0.2
1943251. YW acknowledges the support of Amazon Fel- lowship.
Palson Ratto
Potson Ratio
Figure Dirty-label backdoor attacks. Standard accuracy and poisoned test accuracy when varying the poison ratio 3, poisoned token length (P] and poison strength c.
References
Chatterji, N. S. and Long, P. M. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. Journal of Machine Learning Research, 22(129): 1-30, 2021.
7. Conclusion
In this work, we develop a theoretical framework to analyze backdoor attacks that target the token selection process in
Chen, K., Meng, Y., Sun, X., Guo, S., Zhang, T., Li. J., and Fan, C. Badpre: Task-agnostic backdoor attacks
Telry Toles
nevts Soiert Felert
restelns KArt S
200
325
180
0.2 63 04 05 06
0.1 022
04
0.6
0.3
83
05
06
02
33
05
0.6
0.1 0.2
04
05
0.6
Ratio
Polson
Poison
Rebio
60
45
an
0.2 03
05
0.1
04
03
02
044
01
0.3
04
06
Pb son Rabio
Poison Racio
Paison Rabo
Polson Rabio
Poison Ratio
600
122
115
E22
0.1
5.7
0.1
0.4
055
0.6
011
02
23
04
05 26
01
02
0.5
6.4
055
0.6
011
22
0.3
046
0.5
0.6
0.1
02
0.3
044
0.5
C4
Paison Rallo
Poison Ratiu
Paisen Ralio
Poivat
Polvor Ralia
40
12
24
Porson Ratio
Poison Racio
Peison Rabio
Poison Rabio
Poissn Redio
Figure 6. Darty-label backdoor attacks. Heatmap of the scaled ratio of each condition in Theorem 4.1 when varying poison strength and poison ratio 8. Set [R] = 1. From top row to bottom row represents poison length [P] from to 4.
8

---

Backdoor Attacks in Token Selection of Attention Mechanism
to pre-trained nlp foundation models. arXiv preprini arXiv:21 10.02467. 2021a.
Ilms that persist through safery training. arXiv preprint arXiv:2401.05566, 2024.
Chen, S. and Li, Y. Provably learning a multi-head attention layer. arXiv preprint arXiv:2402.04405 2024.
Jelassi, S., Sander, M., and Li, Y. Vision transformers
provably learn spatial structure. Advances in Neural In- formation Processing Systems, 45:37822-37833 2022.
Chen, X., Salem, A., Chen, D., Backes, M., Ma, S., Shen, Q. Wu, Z., and Zhang, Y. Badnl: Backdoor attacks against nlp models with semantic-preserving improve- ments. In Proceedings of the 37th Annual Computer Security Applications Conference, pp. 554-569, 2021b.
Kenton, D. M.-W. C. and Toutanova, L. K. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of naacI-HI.T, volume 1, pp. 2. Minneapolis, Minnesota, 2019.
Chen, Z., Xiang, Z., Xiao, C.,, Song, D., and Li, B. Agent- poison: Red-teaming Ilm agents via poisoning memory or knowledge bases. arXiv preprint arXiv:2407.12784 2024.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
Li, H., Wang, M., Liu, S., and Chen, P.-Y. A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. arXiv preprint arXiv:2302.06011 2023.
Cordonnier, J.-B., Loukas, A., and Jaggi, M. On the rela- tionship between self-attention and convolutional layers.
arXiv preprint arXiv: /91 1.0584, 2019.
Dai, J., Chen, C., and Li, A backdoor attack against lstm-based text classification systems. IEEE Access, 7: 138872-138878, 2019.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.0011 001900, 2021.
Deora, P., Ghaderi, R., Taheri, H., and Thrampoulidis, C. On the optimization and generalization of multi-head attention. arXiv preprint (tt.:2310.128800 2023.
Li, Y., Huang, H., Zhao, Y., Ma, X., and Sun, J. Backdoor- Ilm: A comprehensive benchmark for backdoor attacks on large language models. arXiv preprint arXiv:2408. 1tt.22448..799... 2024a.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x 16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, 2021.
Li, Y., Huang, Y., Ildiz, M. E., Rawat, A. S., and Oymak, S. Mechanies of next token prediction with self-attention. In International Conference on Artificial Intelligence and Statistics, pp. 685-693. PMLR, 2024b.
Gillioz, A., Casas, J., Mugellini, E., and Abou Khaled, O. Overview of the transformer- based models for nlp tasks. In 2020 15th Conference on computer science and information systems (FedCSIS), pp. 179-183. IEEE, 2020.
Li, Y., Li, T., Chen, K.,, Zhang, J., Liu, S., Wang, W., Zhang, T., and Liu, Y. Badedit: Backdooring large language mod- els by model editing. arXiv preprint arXiv:2403. 13355, 2024c.
Li Y., Ma, X., He, J., Huang, H., and Jiang, Y.-G. Multi- trigger backdoor attacks: More triggers, more threats. arXiv preprint arXiv:2401.15220 2024d.
Gu, T., Dolan-Gavitt, B., and Garg, S. Badnets: Identify- ing vulnerabilities in the machine learning model supply chain. arXiv preprint arviv: / 708306633, 2017.
Magen, R., Shang, S., Xu, Z., Frei, S., Hu, W., and Vardi, G. Benign overfitting in single-head attention. arXiv preprint 4rX17:2410.07746, 2024.
Huang, R., Liang. Y.. and Yang. J. Non-asymptotic conver- gence of training transformers for next-token prediction. arXiv preprint arXiv:2409. 20..244997733555 2024a.
Huang, Y., Sun, L., Wang, H., Wu, S., Zhang, Q., Li, Y., Gao, C., Huang, Y., Lyu, W., Zhang, Y., et Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561. 2024b.
Oymak, S., Rawat, A. S., Soltanolkotabi, M., and Thram- poulidis, C. On the role of attention in prompt-tuning. International Conference on Machine Learning, pp. PMLR, 2023.
Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiaii,,, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., et al. Sleeper agents: Training deceptive
Pérez, J., Marinković, J., and Barceló, P. On the turing completeness of modern neural network architectures. arXiv preprint 1901 2019.

---

Backdoor Attacks in Token Selection of Attention Mechanism
Qiang, Y., Zhou, X., Zade, S. Z., Roshani, M. A., Khanduri, P., Zytko, D., and Zhu, D. Learning to poison large language models during instruction tuning. arXiv preprint Stiv:2400..134599 2024.
Wallace, E., Zhao, T. Z.,, Feng, S., and Singh, S. Concealed data poisoning attacks nlp models. arXiv preprint arXiv:2010.12563 2020.
Wan, A., Wallace, E., Shen, S., and Klein, D. Poison- ing language models during instruction tuning. In Inter- national Conference Machine Learning, pp. 35413- 35425, PMLR, 2023.
Radford, A. and Narasimhan, K. Improving language understanding by generative pre-training. 2018.
URL https: /app..i../// www.sssimattsssoooeeeeeeee CorpusID: 4931:::4493322
org/
Wang, Y., Mianjy, P., and Arora, R. Robust learning for data poisoning attacks. In International Conference on Machine Learning, pp. 10859-10869. PMLR, 2021.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S.,
Matena, M., Zhou, Y.. Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Xu, J., Ma, M. D., Wang. F., Xiao, C., and Chen, M. Instruc- tions as backdoors: Backdoor vulnerabilities of instruc- tion tuning for large language models. arXiv preprint arXiv:2305. 14710, 2023a.
Sakamoto, K. and Sato, Benign or not-benign overfitting in token selection of attention mechanism. arXiv preprint arXiv:2409.17665, 2024.
Xu, P., Zhu, X., and Clifton, D. A. Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12113-22132 2023b.
Shan, S., Ding, W., Passananti, J., Zheng, H., and Zhao, B. Y. Prompt-specific poisoning attacks on text-to-image generative models. arXiv preprint arXiv:2310.13828, 2023.
Yao, H., Lou, J., and Qin, Z. Poisonprompt: Backdoor attack on prompt-based large language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7745-7749, IEEE, 2024.
Shi, J., Liu, Y., Zhou, P., and Sun, L.. Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt. arXiv preprint arXiv:2304. 122998...122,.... 2023.
Shu, M., Wang, J., Zhu, C., Geiping, J., Xiao, C., and Goldstein, T. On the exploitability of instruction tuning. Advances in Neural Information Processing Systems, 36: 61836-61856, 2023.
Yun, C., Bhojanapalli, S., Rawat, A. S., Reddi, J., and Kumar, Are transformers universal approximators of sequence-to-sequence functions? arXiv preprint arXiv: arXiv: 19/. 2019.
Song, B., Han, B., Zhang, S., Ding, J., and Hong, M. Un- raveling the gradient descent dynamics of transformers. arXiv preprint arXiv:2411.07538, 2024.
Yun, C., Chang, Y.W.,, Bhojanapalli, S., Rawat, A. S., Reddi, S., and Kumar, S. O(n) connections are expressive enough: Universal approximability of sparse transform- ers. Advances in Neural Information Processing Systems, 33:13783-13794 2020.
Tarzanagh, D. A., Li, Y., Thrampoulidis, C., and Oymak, S. Transformers support vector machines. arXiv preprint arXiv:2308. 16898, 2023a.
Zhao, S., Jia, M., Guo, 7.., Gan, I.., Xu, X., Wu, X., Fu, J., Feng, Y., Pan, F., and Tuan, L. A. A survey of backdoor attacks and defenses on large language mod- cls: Implications for security measures. arXiv preprint arXiv:2406.06853 06852, 2024.
Tarzanagh, D. A., Li, Y., Zhang. X., and Oymak, S. Margin maximization in attention mechanism. arXiv preprint arXiv:2306.13596 2023b.
Tian, Y., Wang, Y., Chen, B., and Du, S. S. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. Advances in Neural Information 2023.
Processing Systems,
Vasudeva, B., Deora, P., and Thrampoulidis, C. Implicit bias and fast convergence rates for self-attention. arXiv preprint ar arXhv:2402.05733, 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u. and Polosukhin, At- tention all you need. Advances in Neural Information Processing Systems, 2017.
10

---

Backdoor Attacks in Token Selection of Attention Mechanism
A. Missing Proofs in Section 4
Lemma A.I. Suppose = 0. Then the gradient descent direction of the empirical risk at v = is
d d: 1-3 4 Sn (141 -1-1) + a3 Cr (041111-11) - -i-1) +
Σ
Σ の Σ
2nT
1E
A/=1
-
Proof of Lemma A.I.
n
-V.Lpp W.
Σ
e (0), 8(0)
n2
i=1
T
Σ
x
(f = 0.5)
2nT t=1
3/=1
1-3 [R]
(141-
a. [P|
(141-111)
1
Σ
Σ A
c
T
2nT
Lemma A.2 demonstrates the dynamics of W can be described by the dynamics of p. Lemma A.2 (Lemmaa 1 in (Tarzanagh al., 2023b)). Fix the linear head IRd (0) throughout the whole training
process. On the same training data S = (X', y² ) we define
Lw(W) Σ
Cly' V Txt" s(x's Po))
Tp)(P)
Σ Ely' u x T s(Xiw
where Wo Rdxd, -o E Rd are fixed matrix and vector, respectively. Consider the gradient descent iterations on W and with initial values W(O) and p(0) WoW(O) Po and step sizes and lIpl respectively:
W(r + 1) W(r) , (W(7)
P(T + 1) = p(r) - 77 ll (Lp(p(T))
Then, we have W(T) Po P(r) for all r 2 0. Lemma A.3 (Lemma 6 in (Tarzanagh et al., 2023b)). The function i(p) is L-smooth, where
L Σ [|0I3| +3 3||||| ||*11))
Furthermore, if step size satisfies 1) < H. then for any initialization p(0), we have
1)) - I(p(T)) - - v [,ā(p(T)) |
for all 2 0. This implies that
|,D(p(>))|| <o,
|== =0
lim 70000
Lemma A.4 (Lemma A.1 in (Sakamoto & Sato, 2024)). Suppose that Assumption (A1) holds, then there exists some constant C1 C2 > 0, and C" > which depends on such that for all e' O, the following hold simultaneously with probability least 1 8,

---

Backdoor Attacks in Token Selection of Attention Mechanism
1. For all i E [n], / [7], (1 म +)TT(E) S |e S (1 + T+().
2. For any i.j E [n], 1, u E [T] such that (i. + (i. u). we have (c = < c1 Tr(=)) log (Tn/6).
3. For all i E [n] t E (7], c E {+1], we have (<t [c) < c2 || / log (Tn ( ( Rc)1 < Vlog (Tn/8).
Definition 2. If the event in Lemma A.4 occur (defined as E), let us say that we have good run.
Lemma A.S. On good run, for the clean data j E C, r R, we have
|/||
1
1-3
(1-5)
1131)33 Cpt =
1
02 |All| Vlog (Tn/5)+ + (1+1/0)²
y o <
Cn
1|
+
SR +
Tr(E)
|d|
4
2
2
nT
||||
11-3
(1- 3)
(a- (a-1)3
y2
2
Sn |1u|]
7
Cr+
02 |1ll v log (Tn/8))
|dl||
4
2
2
2)
For the poison data & E EN,r r E R.P E P, we have
|//|
1-3
(1-3)
(a-1)3
02 |||||| Vlog (Tn/8)+ (1+1/C')
y 1
SR 1|au|2+
CRt
(p+
T2(2))
||]]
4
2
2
2
nf
|||||
1 1-B
(1-3)
(a-1)8
1
y
Sa ||lll
Sut
(pt
C2 ||1|| vlog (Tn/6))
||||
2
2
2
a23
a
aB
Cr
||m112
Cp
C2 ||ll Vlog (Tn/o)
[d]
4
2
2
||||
(A93
Sp
||A112
a
a3
SP
C2 |||||| Vlg (Tn/o)
||d|
4
2
2
For E [n], v I, we have
|| (1-3)
ag
02 |ulll Vlog (Tn/6) + (1+1/0')2
n' S
CR
2
(p)
Tr(2))
d||
2
2
nT
|||||
(( (1-
8)
ag
y y 2
SR
Sp)
02 |||l|| Vlog (Tn/6)
|d]|
2
2
Proof of Lemma A.5. For training data with label 3² = 1, yh = 1, with token r E R, E P,u E I, we have
J² 30xx x²_"
)
||||||
1-3
SR (141-111) + a3 Sp +
1
Σ c -
-M41 + +T)TT
||a||
4
2nT
SE(TRUP
y1=
81=-1
-112/||
1-3
Sa Ilall
+
1
Σ
(M4I,)))
E (pL2)s)))
||d]]
4
2nT
kE(TT(P
W/=1
:1-11
||||
1 1-B
+ aB
Σ (
+ |||
(ef.
Sp
(c,,
+ 2nT
(c)
4
4
B1=-1
(
(M-1 + +t:)
1 ||||||
1-3
Sn (141
+ a3
Cr 10411110-11
+
1
Σ
||d||
4
2nT
-===1
|||||
1-
SR
|1aa1
+
Σ
Σ
( -
Σ (
|d]]
4
2nT
(E[T]RP
12

---

Backdoor Attacks in Token Selection of Attention Mechanism
|||||
1-3
<a (1 111/44111-111 + aB _ sr u+1 -ā- + 4
Σ
(c)) k)
- Σ
(th
+
||ll
4
2nI
tE[T]
19
D8 Di--1
8y x/
2))
o1p++1
|||||
1-3
C (141-1111
+
a3
(p
11-1-11111 +
1
E
Σ
Σd-Σd))
||d||
4
4
2nT
(T(T
y/
||-|||
aa2
<p
|1al12 +
Σ
Σ
(p)
Σ
(it1 et)
|all
4
2nT
1
W1=
y==
Similarly, for training data with label y = 1, y = -1, we have
, xxx,"" v
-u-1 + tz))
||||||
1-3
CR (1+1- +
a3
Sp (1-1-1-11 +
Σ
Σ
E+)
|all
4
4
2nT
--=--
cE(T]\p
:1=1
|||||
1-1
SR ||1l11
+
1
Σ
(p-1)
Σ (
|||||
2nI
tE(T]iP
1/4 1
11=1
|| |d]
1 1-3
SR
CR (e2, 11,, 141) + aB
4 (p (12 po- - -14+1)
1
Σ
Σ
(t) - E
(th
2nT
<E[T])P
\m==1
J 5 xx T
(M+1 + ++)
||0|||
1-3
SR (141 -M-1) +
ag
4
Sp (p-1-m=1)
+
1
E
||l]
4
2nI
NE(T](P
\p==l
11=11
|||||
1-8 3
SR |1al1
+
Σ
Σ
(p+ -
Σ (1)2)
[d]l
4
2nT
tE[T](P
3==1
|||||
1-
Cn (cp -1--1) + aB
+ |dll
Cr
(ch: -)- + 2nT
k
Σ
Σ
(c))
4
4
tE[T](p
8 5/ = y xxx T
v
)
app T |||
1-3
a3
= -appi
||d||
4
Cr (1+1- +
4
Cp 11-1) +
2nT
Σ
Σ
c
:E(T]iP
|Blll a²B
||ll| Cr
|1112
Σ
Σ
(p-1, ct)
((
2nTT
1E/TT(P
w1=1
Using Lemma A.4 gives us that
|ll
1 1-3
(1-3)
ag
1
B
yo <
SR
1lal12+
Cn
Cp
Sr
c2
|mll|
vlog (Tn/o)
4
2
2
2
2
13

---

Backdoor Attacks in Token Selection of Attention Mechanism
(1 + 1
- - Dr (Tv/6))
+ 2nT
||/|| (1-3
(1-53)
(a-1);
1
||lll Vog (Tn/5)) (1 111100²2
- ||a]|
SR ++
CRt
(pt (pt2)
Tr(2))
nT
(Assumption (AI).)
|
1 1-3
(1-3)
0
1
3
3 2
SR ||a112-
Sn
Spt
25
c2 ||ll| (Tn/6)
||d|
2
2
2
1
(1- (1- Tr(E)
1
1
+ 2nT
(T(E)
C
1
Tr()) -nnT - -BcpTT - 1)c1 v Tr(E) log (Tn/d))
||/|
3
(1-33)
(a-1)3
Sn
11ll
Cut
Crt 2)
02 |||2|| v Vlg (Tn/8)
(Assumption (A1).)
|d]|
4
2
2
|//
1 1 3
(1-5)
(1
3)
Cat (a-1)3
1
02 |||l|| Vlog (Tn/8)+ (1+1/0'²
Cn 11u1
Sp
+
Tr(2))
|d]
4
2
2
2
nT
1
1 1 B
(1-3) 3)
(a- 1)3
1
SR
CRt
Cpt
22 الباا| /oog(TT//)
|d]
2
2
M
a
a
a
SP
||a113
Sp
C2 ||a|||
Vog (Tn/o)
4
2
a233
a
a5
4 SP
||p|1²
Cp
C2 |||||
v /log (Tn/0)
||a|
2
y' 3 xx T
1-B Cn (1' 10,6,,14 141 (41--11)++
a3
E (Evven ct) (ove -
Cr
A41 (:10)-
|d|||
2nT
+E[TTVP
\W==-1
11=+1
|||||
(1-3)
SR + a3
||/l| Vlog (Tn/0)- + (1+1/07)2 TT(2))
||]]]
2
2 {p)
02
nT
||]|]
(( (11-))
B)
a9
2 SR
+
2
{p
02 ||||| v log (Tn/0)
||]]
where Assumption (A1) gives us that Tr(E)= 2 Cn 22 log (Tn/8)², otherwise all the term nf
Tr(E)
would be replaced by
log and all the lower bound would add an additional term - ~TTr(E) (Tn/5).
Lemma A.6. Let p(T) be a gradient iteration at 7-th time step. Then there exists unique coefficients such that
P(T) -rr=))((r)WHp + A-1 + + + + + Σ E (T)
where the initialization is (z) = (.) = (0) 00 for any C f+1), E [n], E [T], and the signal updates are given by
-A,(7)= = ((T)))((+))) = Σ (-((T))
Σ ('(T),
y
y/ Σ
iEC,LAL.
FER
Aİ((T) -)(())))((()=== = an
Σ(-
Σ (T))
リ" Σ
(T(T)a)'
n ieN.
PEP
14

---

Backdoor Attacks in Token Selection of Attention Mechanism
and the noise updates are given by
(p)a(T)) = (ooe(T))=..(T++))) 11) - (-((() . of -((())
1i - E (t)r (eem)).
"E(T]
Proof of Lemma A.6. Recall that
vpī(p) = 6 v. vpf(x*)
1=1
4. . of wx'" T
(diag(S(X'VTp)) p)) S(X-W p)S(x'('p))) (XX'''' X'v
Σ (E[T]
86
7
Σ ME(T]
Fu
Wxy
where f is abbreviation for ('y² v T S(X'WTP)) p)), st is abbreviation for S(X p)., oi is abbreviation for xf T V' [T'. In the dirty-label backdoor attack setup.
My + ej P-y t Ospyy 6)
iEC,LER N iENtEP iECUNILEI
x
Therefore we have
-p(r) = -((r)) ~Vpp((((TT))
Σ (-((()) y²
Σ (T)))
11 - Σ
(T())u
Wxx
1=1
FE|T!
NE(T]
" Σ
(((T)) y/
Σ ('(T)r
7 Σ
('(
Wag
12
IEC
PER
+
7
- -((()) y²
Σ (T(T)r
7 Σ
(T))a
Wp-
n
IEN
PER
wE[T]
+
0n
((((T)) -2"
Σ (T)
1 Σ
(T
Wiy
n
iEN
PEP
uE(T]
+ Σ (-)(())) 3²
Σ (((T)
7 Σ
s (T)
Wei
KE(T)VP
WE(T]
As result, we are able decompose into the following form
P(T 11)
= -AA.fW/..AA......-.A..v.AAAA---.
Σ Σ Aprc(T)Wé
kE|n]CE/TTVP
where we have Ve E {+1},
AAc(T) Ao(T + - - = ! Σ (-((T) Σ (T(),, y Σ
iEC.UN..
FE
nE(T]
15

---

Backdoor Attacks in Token Selection of Attention Mechanism
Alc(T) = 7e(T + Xn(T)= = an
Σ (-(((T)
E WE[T]
8 (ruusi
iENo
PEP
= -ra(() + 10 ((T) - _[ (-)(())) (T()):
Σ s(T)wos''s Vil IT
wE/2
Below calculate the upper bound and lower bound of AA-1 Ad-,, Ait Ad= and Api.e.
Note that for E C and any relevant token r E R, we have
y x
Σ
NE(T]
E (T) (u'/t - -y
ME(T
Σ (T() (y - -vth)) + Σ (T) (v' - -yyra)
uER((r)
NEI
|
(11-8)G++a- (0(0-))))(++1)) 02 || v log (Tn/o) + (1+1/0")
Σ 8'(T)u
Tr(x.)
|dll (ER((F)
nT
||||
Σ
(T())a
11-3
Sn S111111++ (11)))((++
(20-1)8
C2 |||||| Vlog (Tn/8)+ (1+1/C')2 2 {pt 2)
1
T+(E)
||d]
4
nf
WEZ
||/||
1-B
Σ
{T))
1 |111
Su
|d||
4
NER
|||||
(2x-
<p+1) |all Vlgg (Tn/8)+ (1 (1+1/C'2
+
(1 (T)r)
Tr(E)
||]
2
nT
u' - Σ (T)uN''
uE[T]
||l/| Σ ('(T)u
(1+1/C')² nfT'
(-(1-BSStt(e-1188--- ||lll log (Tn/o)-
Tr(E)
|d]| (ERi(r)
|M||
11-3
Sn 11121112 (inll22(13)6mm (11-3)(
(2a-1)8
(1+1/C')
Σ sT())u
Srt c2 ||||| Vlog (Tn/8)
T2(22)
|d]]
2
n'l
||||||
1-3
1 Σ (T(
CR ||ll
||||
4
NER
||I|||
(11-8)
(2a 123
c2 Illll Vlog (Tn/5)+ (1+1/C'²2
(1 (1-
<p+1)
Tr(2))
|[]]]
2
nT
For E and any irrelevant token v E I. we have
y u'at
Σ (r)
WE[T]
= Σ (T) (u'T)) - - + Σ (((ra (u'rt - -yr())
EER
HEIifUt
< ||v||
Σ s² (7) (- (-1-8 Sn |lal12 + (1-
(2a-1)3
1)
(1+1/0')²
Sp +
2 c2 ||lll| vlog (Tn/8)+
Tr(E)
||d|| RER
4
2
nT
16

---

Backdoor Attacks in Token Selection of Attention Mechanism
|l
Σ (T)) ((L-5))A + aBCC) c2 llell Vlog (Tn/o)
(1+1/C")
Tr(E)
|d]| EEI
nf
|
(1- - (T)) (11-3) + aBCp + 2 |llll Vlog (Tn/0)+
(1+1/C'2
(1
Tr(22)
||d||
nT
|||||
m 1-3
Σ (T()).
4 Sn ||al
||ll MER
y y
Σ 8T
WE[T]
|
(-t-BSallatt (1 ; 02 ||||||
(1+1/0y²
(T(
Tr(2))
|d|
nT
uER
|M||
Σ {(T) ((I-B)GR+aB(D) ||l|l Vlog (Tn/6)+
(1+1/C0²
Tr(2))
|dll wET((E)
nT
||/||
((1 + 2 _/
(1+1/C") Tr(E))
2
(T)))
c2 Mall Vlog (Tn/o)
|d]|
nT
|M||
1 1-3 Σ (T))
Sa Illl?
|dl]
MER
Similarly, for i E N and any relevant r E R, we have
y
Σ s'(')an''
wE(T]
Σ (T) (y 1 - + Σ ( + Σ
(T) ( v'r
wER(()
uEP
wEI
||||
||ب||| Vlog (Tn/8) + (1+1/C4² Tr(2))
s'(T)u
|dll| wER
nf
+ ||||
Σ
(T
1-B
CR
+
a²B
Sp
1|
(1-3)
CR
25pt
B
1ta
C2 ||lll Vlog (Tn/5)+ (1+1/0y² Tr(E)
|
4
4
2
2
2
nI
MEP
|P|||
Σ
(T(
(
1-B B
Se 1ellu11++ (11-B)55tt
(20-1))
(pt 2
2)
|Alll (Tn/(++
(1+1/Cy² nT TH(E))
|||||
WET
||-|||
1-B Sn 111l
||a|||
a²B
1- Σ
{(()),)
Σ
(T)
Sr ||
|||||
|dll
4
R
uEP
||||||||||
(11-0)5
(2a-1)3
1
02 |1ll|| vlog (Tn/8)) (1+1/C'²
+
(1- - (T)rr)
Cr+
TYEE)
||ll|
2
2
nT
s Σ (T)ww'n
NE(T]
2 |||||
Σ s'(T)u ((-=))=t(--1)88+++++ 02 |all Vlog (Tn/5) + (1+1/C'²²
Tr(E))
[dll (rR
nT
||M|||
1-3
Sn + a²3 22grr) |1al12 - (1- 3)Cn 2
3
lta ) ez ||Hl|l Vlgg (Tn/o)
+ ||d]] NEF
Σ s² (T)
5r)
set
4
4
2
17

---

Backdoor Attacks in Token Selection of Attention Mechanism
||-||
8 (T))
(
1-1
Sie |1all2
(1 (1-3)sn+
(2a-1)3
Crt
1)
02 Ilall Vlog (Tn/6)
(1+1/073
Tr(E))
||d| W
4
2
2
nT
||-||
1-3
|
a²
1 1-Σ (T(r
Sa Iull
(T)
Sr |1111
||d]|
4
||||| NEP
4
vER
|||/||
(11-9)5++
(2a-1)3
1ta
02 lal| v log (Tn/6)+ (1+1/07)
(1 - (T)))
(p
Tr(E)
[[]]]
2
nT
For i E and and any poison p P, we have
Σ
Σg (T) (y'2)) - -y'7() +
E (T))
-
Σ
(T))) (1't y)
nEP\(F)
WEI
|||||
Σ (T)) (104 (p+
1-3
(1-8
B
Spt a+1
4 {R 11ul12+
Sn
C2 ||al|| vlog (Tn/o)
|dll NER
2
|el| E
5((ru (la-aBCP) 02 ||w||| (Tn/6))
|d]]
nEP((P)
|
Σ 8'(T)u
(4 Sp (Glllal?
2
(1-3 SR+ (t=
a
02 ||el| Vlog (Tn/o)
|dll
MEZ
Sa (Sntaa (1=3sut))
(1-s((T)p)
c2 |||/|| vlog (Tn/o)
|||||
||-|||
Σ {((T),)
Sp+
1-8
4 S1) ||a|12 - ||u||
Σ (T)) (1-3
Sellu11)) ("ZScellalf
Cn
||d||
||d|| WEi
WEP
|||||
(1- (T)p)
(1 CR +a)
c2 |al/| Vlog (Tn/o)+ 1- [11 (T()
a²1
Sp [|lu|
Σ
(T(
1-
CR |1
|d||
4
uEP
MER
2 - Σ (T()uuNh
NE(T]
2 ||l||
Σ (T)
(103 Sp +
(a23
1-3
4 CR
111112 1
(1-33)
SR
3
Crt
a+1
02 ||/4||
Vlog (Tn/o)
|
2
2
ER
(1+1/0")
|0|||
Tr(E)
Σ
s'(T), (a-a3(p) c2 || log (Tn/o)
nf
||l
nEP((p)
|
8 (T)
a2.3
Sr 1
(1-3)
2 Sa + a
3)
llll (ogg(Tn/)) (1+1/C'²
Tr(E)
|d]|
2
nT
WEI
||2/||
(178
(1 1-ss(() (T)p)
Sn ta
02 1111 Vog (Tn/8)
||]]|
|||||
1 [1> ('((() (1))))
(o² Cr 11ll _ (1+1/C')2
(²²
Tr(E))
||||| Σ s'(T), 1-8
Sn [|ll
||d||
nT
||d||
HeR
For E . and any irrelevant E I, we have
g/a Σ
WE[T
18

---

Backdoor Attacks in Token Selection of Attention Mechanism
Σ
(T)) (u're- -17/)) +
Σ
((())u -1'nt- -0--()) +
Σ
8((r)a (v'r - (
-
RR
uEP
NEII(N)
|P||
Σ (T) (²=90 CKelllll= + (11 (2a-1)3
1
e2 |ull Vlog (Tn/8)+ (1+1/092
Spt
Tr(E))
||d]|
4
2
2
nT
WER
|
s (T)) (-233 Sp +
(-a3
a -111)))
c2 11all VlogoooggTTn (Tn/8)+ (1+1/C²² SR
Tr(2))
||d||
4
2
2
nf
PEP
|
Σ 8(() (11-9)/s + +aB(p)
Vlog (Tn/6) + (1+1/C²)2
|alll ((1=))(+++8$$ c2
Tr(2))
||d]|
nT
wEIN(u>
|||/||
(1- (T).))
((1-3)<R + aBCp+ 2
c2 ||||| Vlog (Tn/0) (1+1/0')²
Tr(x))
||]]
nT
+ ||||||
Σ
s (T)
1-3
CR ||a11
|||||
Σ (T)
a²3
4 Sp ||ll
||||| aER
4
||ll
WEP
Σ sT())ug'
WE[T]
||
Σ 8 (T)
("a
(7 Cz Stellal12 - (1--3Kart
1-B
(2xx 11)3
Spt
11)
02 |all vlog (Tn/o) (1+1/C)2 Tr(E)
|d|
2
nT
uER
|||||
(T)) (-c786)
145pp110 Gpllull2= 11111-11119) SR
02 |all v (Tn/o) (1+1/09²² Tr{x))
||d|
2
2
nT
uEP
|||||
(T)/ ((I-P)CH + aagr)) 02 ||1ll Vlgg (Tn/8) + (1+1/C'²2 Tr(3))
||d]|
n.
|||| (1- (T))) ((1-9)5H + aBCc + a
c2 |All| vlog (Tn/5) + (1+1/C')2 Tr(E)
d||
2
nT
+
1l
Σ s² (T)
1-3
SR ||||| 2 |||||
Σ s (T)
a²3
Sp ||
||||
4
|
4
RR
MEP
Therefore we have for E (+1},
AA((T)= = " Σ (-((7)
Σ
((())
Σ
s (T)u
icC.LV-e
ER
uE|T]
<
17 |||||
Σ (-((())
Σ (T()))
[1-S (((rr))
1-B
Sz 11l1
n ||d|| iECo
4
FER
uER
((1-9) (2a-1)3 (p+1)
I1llll vlog (Tn/8)+ + (1+1/072
(2(2)) ))
+
02
nT
? |||||
Σ (-((())
(((6))>E3(0) (T) (- 1-(1- 1-11IZatt)
1-3 GR
Σ (T),
a²3
Cp |ull
n ||| VEN=c
4
FRR
BER
MEP
((1-9/54+ +
(2a-1)3 Sp+1] C2 |A Vlgg (Tn/8)+
(1+1/C)2 TME) ))
+ (1 ((())))
(6)
2
nf
Al.(7) 2 7 |P||
(1-- I(())
1-3
Σ (-((()
Σ (T))
CR ||
n |
4
F
uER
19

---

Backdoor Attacks in Token Selection of Attention Mechanism
(11-915t (2x-1))
(1+1/072
- (1 - (T)+)
(nt1)
C2 |all vlog (Tn/5)+
T((E))
2
nT
|
Σ (-((())
E (t()),
(11-)
s((T)u
1-3
SR |1ll + Σ (T)
a²3
Sp ||11
n ||l||
4
4
iEN
ER
NER
uEP
(1--))se (2a-
1)3
1ta
C2 ||ll| vlog (Tn/8)+ (1+ (1+1/C²²
(X(E)) )) 1
+ (1- (T(),)
Cpt
2
nf
ANc(T) = on Σ (-((T))
Σ
s(())p
7/p
Σ
s
(T)
iEN.
PEP
METT
am
|
Σ
(-M(T))
Σ (T) (1-s(F)p) (²222 Sa + a
c2 Illll (Tn//) (Tn/8)
|d
iENo
PEP
a²B
s'(() 1-3
1 Σ
sT(T)a
Sp 11lll
+ Σ
SR Iall
(8)
4
NEP
NET
an |M|| Atx(1)
Sa + (~
Σ (-(((T)
Σ s'(T)p - (1 (1-s)(T))p
c2 ||al| vlog (Tn/o)
|dl
iENo
PED
1- Σ
(T))a)
(n²g Sp ||a|12 (1+1/0"²2
TM(E))
Σ EER
1 1-3 (T))
Cn 1111
(9)
nT
4
uEP
For E E R, v E I, we have
Appr(T)= " (-((T)) (T)r
u/ Σ
(F)aue
n
wE/2]
< |
(1-I
1-3
(T) (7)r
sT())
SR |u11
n d
4
ueR
+ +11- (T),) ((L-9Ke 22,
(p+1)
02 ||l|| Vlog (Tn/8)+ (1+1/0y} nT
mr(2))
(10)
(P),(T) 2 1/ |
([((()) (()), ((1- (1-I s((T),)
Sn 1- 11A112
1-3
n |
HE
- -(1 (11-((7)))) ((u-))knt (2a-1)33
02 |lall vlog (Tn/8)+ (1+1/C')2
nT TYS)) ))
(p+1)
an)
2
(000))))-(((()) ((()))))(777)))))))))))))))))))) (7)) w/'
-
(T)ans
NE(T]
||ll
[T) ((()))) (-I (T()
1-3
4 Srz 1|
|
NER
+ +(1- (T)) ((1-9Ka+ +
aBCp + +22 02 ||اا|| Vlg (Tn/8)+ (1+1/C'² nT
T(E))
(12)
20

---

Backdoor Attacks in Token Selection of Attention Mechanism
(pro(t) 2 n n|d
|ill (-](())) ((T)) - Σ (T)) 1-B
-X
4 SR ||/112
NER
-(1- ((1-9K5+ (1=33)n++$33 +BB
+//) 02 |lall vlog (Tn/8)+ (1+1/C'22 nT T(())) ))
For E Nr E R P, we have
Apt. = n
(-)((T)) (T),
Urr Σ (Trrws
ME/2y
< 7 ||
(-(((T)) (f(T)/ (1))(-((1- (1-
(-(1-- (T))
1-3
CR 11ll Σ
(F))
a²3
4 Cp 11u112
n |dl
4
ueR
uEP
+(1 - (Tr)) (1--0)sst (2a-1)3
1+a
c2 |Al|| Vlog (Tn/0)+ (1+1/C²²
Tr(x)) )) (13)
(p+
2
2
nT
Aprs 2 7 |u|||
(-)( -f()))
(1-Ex) (()))
1-3 3
CR 12ll12 + Σ (T))
a²3
Cp ||
n |d|l
4
uER
PEP
+ +(1- s()))) (11-0)6t (20-1)3
1+a)
C2 ||all Vlog (Tn/ (1+1/0")2
(M22)) ))
Cp
+
2
2
nT
Apip
7 (-(((T)) (T)) 1
-
E
12
UE[T]
1) ||
(T) (7)) (1-s/(t))
(1-3 2
Cn + a 02 Ilaal vlog (Tn/8)
n2 |d||
+ 1 Σ (T)a
a38
Sc srllu|1++ +
Σ MER
1 1-B (T)
Sn 1all
4
4
MED
17 ||||||
(-(((T)) (T)p ((1-)((t))) (1-3 Sn Cn+ +
Apip
02 [1lll| vlog (Tn/6)
n |||||
2
1- -
11ll (
(29 Se
(1+1/C')2
TI(E))
+ Σ (T)
1-3
Sn luul )
nT
4
PEP
eR
21 (-(((T)) ((T) - -
Api.
n
WE|TT
< ||||
(T)) (T)))
Σ (T)
1-3
4 Conlllul [1lu112
->((),, Σ (T),
a²B
Cr
d
4
R.R.
.EP
+ +(1- (T)))) ((L-9)sH + aBCp + 9) 02 |||||| vlog (Tn/5) +
(1+1/C'²
Tr(E))
(14)
21

---

Backdoor Attacks in Token Selection of Attention Mechanism
Apr, 2 1) |||||| ||d||
1 1-3 Σ (T)
a²B
(-((T)) ((())
Sn Iall Σ (T)
4 Sp
4
HER
uEP
- (1-s((r)))) (11-))5 +aßCr + g) c2 |all Vog (Tn/o) + (1+1/C*²
Tr(E))
nf
Lemma A.7. Suppose that the norm of the linear head x d and ||]|| scales as 0 [I]l)). There exists an absolute [d]
constant C > 0 such that on a good run, we have for all time step r 2 0.
(T(T)
S Ce. i.jEin (T(T)
max
Proof of Lemma A.7. Recall that the derivative of the loss function is given by
(T) 1 exp (Erepr st (T)
for any E [n],7 2 0. One a good run, for all i E [n], E E R, E P.UE I, we leverage Assumption (A2) and
Lemma A.5 to get the following
hi
1-B
4 SRt
(1-B)
CRt
(a-1)3
(p
1
C2
(1+1/0")²
2
2
CT
CT
(1-3)
B)
aB
(1 (1+1/C')2
h
SR
+
Sp
+
2
CT
CT
b
2
Cp+
( a
aB
{p
C2
2
2
CT
Therefore there exists some constant c > such that |زg we have
<c,
[n], E [T]. Since - - is monotonically decreasing,
1 1 exp (c) S -(((T) = + exp (-c)
This leads to C₂-
1+exp(e)
1+cxp(-c))
The following lemma show that given a sufficiently small step size, the softmax probabilities do not change significantly in a single step of gradient descent.
Lemma A.8. Suppose that the norm of the linear head x d and ||ll| descent satisfies Assumption (A3). Then the probability assigned each token only changes at most by constant factor,
|||| scales as O ||m]]²) and the step size of the gradient
VF 2 0, Vi E [n]. Vt E (T], ! (t)) < sT 1), < 2x)(r))
Proof of Lemma A.8. By the updated of gradient descent, we have
x T WT P((T) exp
xx WT (-nVpī(p(r)))
exp
s'(T + 1),
E EvEryy exp
x T WTT p(T)) exp x Tw (-nVдī(p(+t))
22

---

Backdoor Attacks in Token Selection of Attention Mechanism
Consider consecutive time steps, have
2 T WT (-nVpī(p(T)))
x T WT (-NVzT((((7)))
exp
(T + 1) s2 (T))
exp
x T WT
(-1Vp((p(+))))
minuelt] exp (xx T (-nV)((p((r))
exp
Therefore the proof is completed by showing that
Vt E [T], Vu E [T], S exp (xx--xx) WT (-nvoū(p(tt))) 2.
To analyze the term inside the exponent, we have
(xi x WT (-nVgī(p((T))
{ x " WT (-n),((o(rt)) LETT]
max
(15)
Following from Lemma A.4, for := r E R, we have
x
WT x/wT((
(-(--pp((((t))
(my
A ΣE Amu(Tte
iE(M)AE[TTVVP
I (T) +
+Ā)(++++++ Σ Σ splalt) c2 |alll Vlog (Tn/5)
:E/n]CE//7VP
+ Σ E A Tog) (T/6) + 41 1/C")2
Fort : P, have
x WT "TT (-nvgū(p(7)))
= (A + AA-I(T)) 1 + Σ E (PE
VEIN)AE(TT)P
< Sa ||ll| ta Σ Σ Apre(T)c Illll Vlog (Tn/8)
PE|N]EETT]VP
Consider we assume
||||| scales as @ [|/l]3) (), then Equation (6) gives us that ||dl|
(A.
<
7
1-B CR + (1-3)(R+
(2a- 1)B
sp+1
02
(1+1/0'²
In
(16)
4
2
CT?
CT
where the last line for sufficiently large C and Assumption (A4) so that
0BCp CT2
< Similarly, Equation (8) gives us that
ANc(T) < a.9m 1 1-8
a²B
11-3
CR
+
Sp +
CR
S sa33n
4
2
C12
where the last line is because Assumption (A3) so that Cn S a² PC
23

---

Backdoor Attacks in Token Selection of Attention Mechanism
Equation (10) and (12) tell us that for i C
Σ [T)
n - (-((T))
Σ s(()r/ 1- 11Es*to (t())
1-9
SR
TR
TRR
uER
((u-9Ki* (2a-1)5
Vlog(TT/0))
(1+1/C TrE)
+ (1 (1-ss(F)))) sT())+)
{p+1)
2
02
||all
nf
1n
77
3
Sn +
(1=3) + (2a-1)8
(r+1)
C2
+
(1+1/C9))
(Assumption (A2))
72
4
2
CT
CT
2A
17
(For sufficiently large C.)
72
S (-((())) Σ s1(T)/ (-E (T)
1-3 4 SR
12
WET
WET
WER
((1-3)5n+ + agSCp + 1)
Vlog (Tn/6)
(1+1/07²2 Tr(E)
+ (1 - (T)).)
2
C2
||||l
nT
||]]
5 7
1- 9
4 CR +
(I-B)RR+
(2a-1)3
pp+1)
C2
+ (1+1/C²)3
(Assumption (A2))
2
CT2
CT
"
(For sufficiently large C.)
Similarly, for i E N. (13) and (14) tell us that
Σ Apprl S " (-M(() Σs(()) (-11-
(-
1-3
a²9 4
((T)
SR
Σ (T)
Sp
ER n
TRR
NER
MEP
+ (1 - (F(rrr ((1-9)a++
(20-1)3
1+a)
vlog (Tn/5) (1+1/09² 2 Tr(E)
7p+
C2
2
2
|||||
nT
|uul
2 P(P
Br²BCp 2 Cn)
n
Σ Apill
< 2 (-(((())) Σ (7))
Σ (T)
1-8
SR
Σ 8
(T)
4 Cp
Q²B
11
4
VEI
UEI
HER
PEP
+ +11 (1- (T)o)) (T)) ((1-0)ka + a3Cp
+ -) 02
log (Tn/0)
(1+1/07) (r()))
|||||
nT
a² Bcr
2A
n
From Equation (10), (12), (13) and (14), we have
Σ Ap,((T)]= = Σ (P) Σ [Apro(T)]+
Σ
(Ap +
Σ (A)
iEC,rER
VEC,vEZ
iENTER
VEN.EEI
$ (1 3)2n + a² Q²3²ppm
(Assumption (A4) gives us that (p S 1)
$ 17
(17)
24

---

Backdoor Attacks in Token Selection of Attention Mechanism
As result, have
8²
x T WT (-nV,T(p(+t)) 2 1
"
T
x T WT - (-nVgI(p(T))) 2
(at3²= +
a T
1 ?llull
Therefore, choosing sufficiently small step size n, by setting a sufficiently large C can make Equation (15) smaller than log (2), thus concludes the proof.
We end with a remark that if (r)) < n 22 log (Tn/6)², then all the term of ||l which can be further upper bounded by CT from Assumption (A2) and therefore all the discussion remains hold.
Tr would be replaced by (T(E) log (n//6)
|| 1
The following lemma shows that for any clean data, with high probability, for any time step, the attention probability from the relevant token would dominate.
Lemma A.9. Suppose that Assumption 1 and 2 hold, and the norm of the linear head d and For any clean data E C, on a good run, for all time step T 2 0 and all irrelevant token I, we have
||ll| scales O [|x]] T
(T) S max (T)) FE
Proof of Lemma A.9. The proof via induction. The inequality holds initialization as the elements are equal to 1/T. Assuming (T)) S maxxeR. {a {T)) and we prove for sT( + 1). S maxteR {s + 1)r}. For E I, we have
x T W (Tp(++
max,ER fsr(T 1)r} sr(r + 1)
maxTER.
exp
xx W p(T W"p(T++ +(T+1))
exp
(exp (x; T w ((T))
exp (x, WT (-NV)T(((7)))
maxteR
exp xt T W p(T))
exp xx. WT (-nvyū(p(T)))
(T()))) (T)
exp ((x+ - xt) T WT (-nVдī(P(7)))
where r/ = arg max,ER {s)(T)r}- As long as we can show (x (x:) WT (-nV)ū(o(r)) > 0. then
max-ER (s'(t+1)) (()) which proves via induction. s'(7+1)+
(())."
We now consider two cases. If ErER sT(T), 2 1- 1/1 then the probability of not selecting the relevant token less than Apply Lemma A.8, the probability of not selecting the relevant tokens after single step of gradient descent is most
27
T7. and therefore EveR s'(r + 1)r 2 1 .T. leading to
1
s' ( <
1
S max for (T 1).}. PER
2T
R
2T
Now we only need to consider the situation where ΣFER (T(),, definition of r'. We have
S ィ7. Note that (T)r 2 1/T also holds due to the
(xt, - x.) WT (-nV,I(P(7))
= (ng + E - -b)" [Ahilt)aa AA + (7 + + Σ Σ Apre(T)E
RE(N)rE(T]VP
2 (A) - (T) (T)
( (Tn/8)
25

---

Backdoor Attacks in Token Selection of Attention Mechanism
2 +2) ( + +2/
+ Σ pA ||lll vlog (Tn/6)
kE "E[T]XP
where the last line holds from Lemma A.4. We will show the above equation is positive by controlling each term separately, Note that ||
|ail = @ (1/ |11u11²). leverage Equation (11) and (12) gives us that for i E C,
(Apir((T) - p (+(2)
2 Tr(z)) "
n a||al1 (- (-](T)) (d(()- (1-I
1-3
4 Sn ||al
HER
- (1 (T)rr)) (11-9)6x (2a-1)8 (p+1)
02 ||llll Vlog (Tn/6)+ (1+1/0))
Tr(E))
2
nT
- -T) (- Σ
(T()))) 1-3
Sn 1111
uER
+ - (T)() (11-3)5m aBCCr + 2/) (11
02 IIlll vlog (Tn/8)+ (1+1/0'² Th(2))
nT
2 Tr(EE) "
|||a|/ (-((T)) (stt)r 1 (1-E s((rru)
1 1-3
Sn ||ll
4
HER
- -2)( ((L=3) + aBCr + 1) 02 |Al Vlog (Tn/ + (1+1/C1) TM(2))
nT
(1(()) + (t()
(1(()))((ss)) -( s (T)
(-((())) (T(rrr 1- 1-Ex(()) ) (1 Sa |
2 Tr(2') 7
n |||u||
uER
87 ((1-B)<R + aBCp + 1) |U|| Vlog (Tn/5)+ (1+1/C9² THE))
nT
(25 <TTss (T), L1-EuER s (7)..) as 1 1- EVER s'() 2 )
1-B Cn 8 8(1 agCr + 1)
T Vlo (Tn/5) (Vlog(Tn/5) (1+1/0') Tr())
2 +T(()) 21 (-((T))
n 472
4
|A1||
n
|11/
(s'(T)r EWER 8 8((r))) 2 +tt
2 >-
1-3
GR
-8
(((L + a36p +1)
C2
(
n 472
4
CT
C
Similarly, Equation (7) gives us that
(1--I
1-3
AA.(T) N (()) Σ (()))
(²(())))
CR
4
IEC
PE
uER
- -11 (1- (T)+) ((1-9)kmt
11B
Vlog (Tn/8)
2 Tr(E))
(p+1)
C2
2
||||||
nT
||a|
7 Σ (-(((T))
Σ (T))
(1-55 s'(T)u
1-B
SR+ + Σ (T) a²B
CP
n
4
VEN-c
FER
NER
NEP
+11- (F)),) ((1-9)5++
(2a-1)8
1+a
Vlgg (Tn/8)
(1+1/07) Tr(2))
Cpt
2
2
||||
nt
|| 2
26

---

Backdoor Attacks in Token Selection of Attention Mechanism
(1-
(1- sT))
1-8
2 Σ
(-E(T))
Σ
(T),
<n (Denote ΣiEA To = =Eisa 2i - xi
-Eign Ii))
4
Wec
PER
KER
- (1 (F)r)) (lu-915s+
(2a- 1)33 <ptl]
C2
(L+1/C9)
2
CT
CT
B
Σ (T)
Σ
8
(+)))
(T()
4 Sp
IEN
PER
uEP
1-28
1-B CR ((1-9)A+ (2xx 1]8
12 (1+1/09²)
az B2
2 (-((()))
{pt1)
Cp
11
87
2
CT
C
(Vi C. EPER sT(r)r (1-EuER s((r)u)
(1-EuRR s'(r)u) 2 17. Lemma A.7.)
2 (-M(()
1-28
1-3
CR 4
((u-3Xe+
(2a-1)8
2 <p+1)
22
- C
(1+1/C7)
87
4
CT
where the second last inequality applies Assumption (A4) so that fo 2 32 hold and therefore the positive term dominates.
Moreover, from Equation (10), (12), (13) and (14) we have
(1 (1-3)2
a²
C 22 log (Tn/o)
(1+1/C')²
(E
Σ [Apre(T)) Sn
CR +
Cp + (11-3)(
||,|||
nT
||ul|
(1 (1-
2²
+ (1+1/099
S 1)
C +
Cp + ((1-3)
C2
4
4
CT
CT
Equation (6) or (16) gives us that
(T. S
7
1 1-3 1 SR+ + (1-B)(R+
(2a-1)3
{p+1]
C2
(1+1/072
(Assumption (A2))
2 4
2
CT
CT
Equation (8) gives that
Atc(T) & a7 Σ (-E(T)
Σ (T) (1-)((t)))
(1-3
Vlog (Tn/8)
Sn + a
C2
n iEN.
2
||||
PEP
a² Cr+ + Σ (T))
1-3
(T(T)u
Cn
NEI
a
1 1-3
a²3 Cp+
11-B
v Mog (Tn/8)
Cn +
CR +a
02
2
4
2
||
a
1
1-3
3
a²3
11-8
C2
CR
+
Cp+ +
2 (
SR +a a
(Assumption (A2))
2
4
4
CT2
Assumption (A2) gives us that therefore we have
|| 2 CT2 Vlog (Tn/o), ||a||2 2 co (Tn/o) (TEE), CTT((E) S n 111111 and
(x+ x WT
(-NVg((p(7))
(((-((T))-
1-3
SR - 8
(((1-B)&8+ + aBCp +
( Tr(E) (2a 2a-1)3
1
C2
47
4
2
CT
n
7(1 -3)
1
( 1-3
SR 8 ((1-9X00+
(L+V/o9))
(-)(())
(p+1)
I|al1
8C2
2
2
CT
C
27

---

Backdoor Attacks in Token Selection of Attention Mechanism
17
(1- (1-5)²
4
CR +
a²
4
(p+ + -
CT
C2
(1+1/C9²
CT
(1 + 01)
CT
||
4n
1-3
(R + ((-B))RR+ (2a-1),3
(p+1)
C2
(1+1/C4)
|all
C7
CT
CT
-48Bn
(1-3 CR+
a²B
11-8
Sa
+a
C
||/|2
4
2
CT
CT
0
holds for sufficiently large C' conditioned on Assumption (A4) so that Çn 2 agSr
Sn 2 C aB 1
a'p'c
CT
The following lemma shows that for any poison data, with high probability, for any time step, the attention probability from the poisoned token would dominate.
Lemma A.10. Suppose that Assumption 1 and 2 hold, and the norm of the linear head and For any poison data .N, on good run, for all time step r 2 and all token except the poison token [7]XP, we have
||rll scales O ||x|] [[a]]
s² (T): S max {T)) PEp
Proof of Lemma 10 The proof is similar as that of Lemma A.9. The inequality holds at initialization as all the elements are equal to 1/T. For to E [ We have
maXpep {s'(T + 1)p} S'(T 1)e.
('(T)p' (T()),
exp ((x) -x;) T WT (-nV,L(P(T)))
where p' arg maxpep 1s (T)p). Now we only need show (x-, xx:) WT (-nVş((p(F))) > 0. For
Epep ST()) 2 1- + induction holds for the same argument as shown in the proof of Lemma A.9. Now we only need to consider the situation where (peps'(T)) I 1 47. Note that (T))p' 1/T also holds due to the definition of p/. We now consider the following two cases.
1
Case 1: N. R. We have
(xS) x²) WT (-nVşï(p(7))
=aji, -
AA - Σ Σ Apcc(TTet
VE(N] uEfTTVR
(p..(T) AA ( - Tr(E) - (Ap log (Tn/8)
AAy
kEInl.
we VE
Esprn(tt|| || (Tn/6)
ke|nl. wE x
We control each term separately.
Σ (() -(1-s (- (1-3
Ade(T) 2 on
Σ (--((T))
Sn + a) 02 1|11||1 Vlog (Tn/8)
n ||112 ieN
PEP
1- 1 (T)
(a28 Sr |1u112 (1+1/C²)²
Tr(E))
+ Σ (((T) 1-3
Sn llul
4
nT
4
uEP
RER
28

---

Backdoor Attacks in Token Selection of Attention Mechanism
0
Σ
(-)(T))
(T))
(- (-
[1-5
s(())u
(1-8
Sa tnta)
C2 |uI1
4Tylog (Tn/ō)
n
2
VEN
"EP
(T)) 2 s(()), 1- 1- --EEp ('( 2 +t)
(228 SP
a2B
(1+1/C")2 Tr(E)
1- Σ (t))
nT
||11
PEP
0:31 1
(-(1
4T'
Vlog
log (Tn/o)
2²3 (1+1/C²2 Tr(E)
min
-T((T)
CR t
C2
Sp
2 47
sEN.
||
nT
||M|1
(s'(T)p (1- (w-Eupps* s' (T) 2 1 17=)
a.31 1
(-(12²
(1-3
a²B
(1+1/C92
2
2C 472 ((((T)
Cn +
+
Sp
(Lemma A.7, Assumption (A2))
2
CT
CT
For i E N,, we have
-Aprr(T) 2 (T), (1-EX ((=))(())))
1-3
Sa Sall + Σ (T( a²B
s((T)a
Sp |la1
n1
HER
NEP
(1 s())) (11-9)5++ + (2a-1))
1)3
I+a
Vlo (Tn/o)
(1+1/C') 2 Tr(E)
(pt
2
2
|||||
nT
|
2 - 7 (-((T) ((1-9K5
1)3
Cr+
l+a
C2
(1+1/C²)²
(Assumption (A2))
2
2
CT
CT
Therefore,
(x) xx) T WT (-nV,ī(p(T))
a Bn 1
(-M(())
Sp -1+1//C'
122 SR (RRta)
(1-3
2
|1al1
20 47
CT
CT
2
11 (-G())
(4
L-B (
CR
(1-3)CRt
(2a 1)3
{p+1]
22
(1+1/O7²)
||al1
CT
CT
71- ((-)
((1-9)knt (20-1)3
(pt1)
02
(1+1/092)
Tr(E))
(Assumption (A2))
2
CT
CT
n
1-8 3
C2
(1+1/093)
(1+a+e)) |
Cn + ((1-3)Cn
(11=))n
4
CT2
CT
CT
2m
1-3 3
Sn
+
(1-
(2a-1)3
(r+1)
C2
+ (1+1/C4)2
|all
4
2
CT
CT
CT
1-88
a²B
1-3
C2
|
CR
+
<p
+
CR
+
4
2
CT
CT
>
holds with Assumption (A3) for sufficiently large C' so that ensure the following case to be positive.
a afeee
2 CCR, a²3Cp 2 cT. These assumptions should also
12
Case 2: E N, N, V I.
29

---

Backdoor Attacks in Token Selection of Attention Mechanism
i i E N, have
-Apho(T) 2 2 ((-((())) (T),
(-11-E
1- Σ
(T)
1-3
SR
CRllall++ +
Σ
8
(T)
a²3
<p |1a11
n
4
4
uER
MEP
(u-B))aa+ (2a
1)8
1ta
Vlog (Tn/o)
(1+1/0") Tr
(1 (T)))
(p
C2
2
||
nT
" ([](T))))
1-3 <n + (11=3)5n+ (2a-1)8
(r
1ta
C2
(1+1/09²)
(Assumption (A2))
CT2
Similar as above, have
(x/- -x:) WT (-nvpū(p(T)))
- -ap - +
(dri(T)us++ + -K
Σ E Apru(T)c)
sEini WETTYP
2 Ap)o(T) - Σ
(I log (Tn/o)
sE(M),NE(TIV
| /Anr))))))// ( -/)
Σ Ap)))(T)
a Ialll vlog (Tn/o)
kE(m), wE(TTyP
(-((())
a23
Sp
(1+1/072
(
11-3
SR +a
C2
||a|1
2C: 47
4
CT
CT
n( (())()))
(1-3 S (1-9))m+
(2a-1)3
(p+1
02
(1+1/C')2
Tr(E)
(Assumption (A2))
2
CT
CT
11
1-8 3
CR (1-3)cR+aB8)
C2
(1+1/0')2
(a (1)
CT2
CT
CT2
2am
1 1-8
CR
(1=
(2a-1)3
{p+1)
C2
(1+1/C")
||01
2
CT2
CT
CT2
2a Bn
1-B
SR
+
a²B
Sp
+
(12
(1-3
CR
+
a a+1
C2
||a||2
4
2
2
CT
CT
0
holds with Assumption (A3) for sufficiently large C.
Lemma A.11. Suppose that Assumption and 2 hold, and the gradient of loss function satisfies lim Then for each clean sample j E C, there exists the token index 13 E [7'] such that
|rā(p(T))
=0.
lim 7
(T) =1,
lim <t 0, +
for j C, [T] (e). For each poison sample k E N, for every poison token Pf E P, we have
1
lim $ (T) 7-000
0. 700
lim
s (T)
forall k E N,t [T]PP.
30

---

Backdoor Attacks in Token Selection of Attention Mechanism
Proof of Lemma 1. The proof largely follows the proof of Lemma D;3 in (Sakamoto & Sato, 2024). Recall the following technical results: For linearly independent vector V1,--rg Vm E IRd and coefficients d1,--.;0u IR, there exists constant
co > 0 such that Eigim
V.Ve 2 co Exe[m) |azl-
Recall that the gradient of the empirical loss function is given by the linear combination of { (Wxi)igtn)eemm
vpī(pp
1
Ee
Σ
o
Σ
sy
Wxi
i=1
AE|T
wE[T]
Σ
Σ
oy T
Wxy
+ Σ
Σ
k
l kaak
Wxx
1
78
u ia
JEC
(E[T]
NE(T]
REN
:E[TTVP
E
+ - Σ 4-
1
Σ
St
uu
&
aWi+1
1
Σ
Σ
su Tu
aW
KEN11
tEP
KEN
PEP
uE(T]
(18)
Since this norm converges to zero, we have
Ye 0, > S.t. VT 2 To.
vpā(p(T)) Sc.
(19)
Combining Equation (18) and the fact that rEC, (Wx!] with probability 1, if V Vp[(p(T))
(E W and W are linearly independent
s e holds for some e > 0, r > O, then we have
s (T)
7t - Σ
s'
(T)
for Vjj C, [7]
(20)
71
Cu
mEl27
(T)e
Σ
s (T)
5 _
Vk EN, [T]
(21)
WE[T]
1
de (T)
Tt
(T) k
Vo E No,t P,ce (+1}
(22)
KEN.
AEP
WE(T]
Given that the linear head v is fixed and the output scale remains unchanged, note that there exists some constant < C2 < such that C1 |4/1 < C2. For d' == ne Equation (20) and (21) gives us that co|e!
(T)) < Ve,
Σ
< Ve, for (
8 (T), < oe
Σ
< Ve', for EN
ME[T]
As s (T),, = = for every & E NN,,,,,,, h4 = =1t for every ki, k2 EN,41,2 E P, due to x = ks aau
we define s² (T), == s (T),, 7p == of for every E N,t E P. As 16g1 is bounded, 7 - Eneerr 8² (T)wn is bounded, for each E N, there exists (k) such that Equation (22) gives us that
(T))
- Σ
(F))nn
= [3
(t(T)z
7p (70 DE
Σ (T))
5 :
17
co
KEN
IEP
KE[T
31

---

Backdoor Attacks in Token Selection of Attention Mechanism
For vk E Ne E (+13, there exists e" (R) such that Equation (22) gives
(T) < Ve", or Tp - Σ
(T))
< Vee.
WE(2]
We now consider two cases. For clean sample j C, we start with there exists one 1j E [T] such that nip -ue al
Ve'. Otherwise, assuming there exists f1. 22 E [T] such that
Σ
sy
Ve'
< <ve,
uE(T]
ME[T]
by triangle inequality, we have 17, - Val 2V2. As the noise in each clean sample's 's token { ) take distinct values
jEC
(E(T]
almost surely, we can select a sufficiently small e such that 2 < mi, - - leads to a contradiction. Therefore, there exist one 17 (T) E [7'] such that for all i E (7M) (G(T)) (T), < ve, 8 (T) 1- (1-1)ve From the step size Assumption (A5) and Lemma A.8, the token index (T) is determined without depending on the time step r for sufficiently small and we further denote it as 1 which appears in the statement. As a result, for any sufficiently small Ve and 62 = (T' - 1)e,, from Equation (19), there exists To > 0 such that VT 2
(T), E1, s (T)zz
2,, t [TT) {t)}
We complete the proof for clean sample regarding t E R by using Lemma A.9.
Similarly, for poison sample & .V, either there exists one t E [T]XP such that m Evezyy si Ve', for every p P. 1/ EuEjzy = Ve", otherwise arrive the same contradiction shown for the clean sample. By Lemma A010, claim the first statement does not hold, otherwise there exists one (T) [T] such that for all -PPlee"
I E [TT) (PU (M(T))) (T)) < vē, for all p P 5 (T), < Ve", and s (T))g((>> 1 - - From the step size assumption (A5) and Lemma A.8, the token index (T( is determined without depending the time step for sufficiently small c, and we further denote it as t which appears in the statement. As result, for any sufficiently
small 6 ve, E2 Ve", and E3 (T |P| 1)e; |P|c,, from Equation (19), there exists To 0 such that V7 2 To.
(T){z > -
As i [T],P, this contradict with Lemma A.10.
As result, we have that for poison sample & E A, for every Ph E P, / - Euerry shih Ve". Therefore, for any sufficiently small 61 = ve and 62 = (T - (PI)Vee from Equation (19), there exists To > such that Vr 2 To
(T), c1, (T)pp > P (1 - [ EP,
completing the proof of poison sample.
Leveraging Lemma A. with Lemma A.3 achieve Lemma 5.1.
We now introduce the following lemma to guarantee the direction of signals.
Lemma A.12. Suppose that Assumptions andd holds. We further assume the following hold for e E (+1}.
Sn
Σ IEC.
( 2 a² BCP
Σ
Σ ('((7)
0ST'ST
iEN 05T'5
( (1-s((7)7)) (7))))
SR Σ (T) Σ Σ
VE(N)
(Ein) LEPTIN 05r'sT
05 ST
32

---

Backdoor Attacks in Token Selection of Attention Mechanism
(T') (1 - 8 (T))))
Σ
Σ
8
Σ Σ
T
VEN. SST'ST
SEN,
UST'ST PEP
Then for all C E {+1}, if the following conditions on the training trajectory are satisfied
where C1, C2 are some absolute constants, then we have
Ac(T) 2 n |d||
" ||/|||
||al
2 min
(-((()))
CR Σ (T)
- a²
Σ Σ
8 >0.
OST
sECo
DST'STiECE
OST'STIEN
a² Bn ||||||
Aa(T) 2 n |
Cr ||ull
min
(-G(()))
Σ Σ (1))
0St'ST
iEN-o
VEN. OSTSST
Proof of Lemma 121 We have
Ae(T) Σ AAr(T'
OST'ST
|Ill
(1-I ((T)
1-3
> |d]| USF'STiEE IECo
Σ E--G()) Σ (7'))
CR
|111
4
FER
uER
(1--((r)),) (11-9))0+ (2a-
1)3 {p+1)
(1+1/00)
m(())) nT
c2 11lll1 Vlog(T//g/// (Tn/0)=
2
7 ||l|||
Σ Σ (-)()))
Σ (T'))
(1-I st()
1-3
Cn Cnlllll
+
Σ (7') a²B
Sr ||a112
n |dl
4
UST'ST -
uER
uEP
(1--9)is
1)3
Spt 1+a
(1+1/C'² 2 IX(2))
+ (1 - s(()),
+
o2 |||||| v /log (Tn/0)+
2
2
nf
2 |
||/11 2 min
(-C(TT))
ΣE
((()),
(1-I s(())
1-B CR
OST
sEc
UST'ST iEC, FER
uER
-11- (t(',, (11-91600
128 sp+1
(1=1/C92)
2
CT
CT
Σ Σ Σ
(7)),
(1-
(1-
(T())
LLB
B
CR + a²B
4 Cp
UST'ST icN
uER
+ +(1 (((),,)
(11-9)ket
(2a-1)8
Sp+
1+a)
02
+
(1+1/093
2
2
CT?
CT
2 ?
||/l 2
min (-E((7')
(sa
CR
Σ
Σ
8i((') - Q23CP Σ
B
Σ (,(1))
0Sr'ST
sEC.
Ogr'STiEC.
OSTISTiEN
(Choose sufficiently large C))
2 C
(7))
(Condition (2) and (3))
IEC. 0ST'ST
where the second last line holds due to the condition that
8 s(r))(
SR S Σ ((
Σ (1(7),
SR
Σ Σ
T
REC. OST'ST
VEN OSTT'T
*E|m]
SE|n <E[T]M uSr'sT
US+'ST
33

---

Backdoor Attacks in Token Selection of Attention Mechanism
Similarly,
dd(T) - Σ Aic(T)
0S4'ST
2 an I
Σ (-G(T)) Σ
(t') - (1--s((7)p)
(
1-8
Sh +
a+1
c2 llllll Vlog (Tn/8)
|d|
2
2
iEN, OST'ST
pEP
+
1
1
Σ
(T()
(098 Sp |111112
(1+1/012
TT(E))
Σ s((''u 1-8
Se |1lll?
n8
NED
uER
2 07) |||||
||//1
min
(-((()))
s((')p - (1)))
(1-3
CR +
a 1
22 CT
||d|
0St'sT vEN
2
iEN OST'ST pEP
(
- (1+1/C9²)
s' 1-3
+ 1 Σ
st (t())))
Sp |1|ll1
+ Σ
4 SR
4
CT
NEP
MER
a Bn ||
2
SP
||1
min (-E(())
Σ Σ (T())
n
|d|
OST ST VEN-c
ieNN OST'ST
where the last line holds because Condition (5).
Σ
Σ s((T)p
11-$st))
2
3E Σ OSN PEP
s'(T), (1-))
(23)
T
IEN. 0ST'ST
pEp
PEP
guarantees de > 0, Vc E {=1). Note that when [P| 1, due to Assumption (A3) such that a. 2 7. the above Equation (23) holds trivially.
Lemma A.13 (Simple extension of Lemma E.2 in (Sakamoto & Sato, 2024)). Suppose that Ever- S(T) 2 1 - c for some r 2 1 E TT*, T C ( > O, then on good run, we have
|p(T))| 2
log
(T- (T-1T"))
2 |all 2 Tr().)
Proof of Lemma 1313. We know from Lemma 5.1 that if j E C, there exists token t* := t7 E R such that s (T): > 1 - e, if k E .N, for every t* p P. we have s* (T)r- p e, meaning Erep s (T) > 1 - pple. Therefore, based on whether the sample is clean or poisoned, T can be either a singleton set or poisoned set. To combine the above situations into formula, have the following holds:
s(T)r- Er exp (x..V. W P(T))
Σ E+
21 lT
Enem exp [x] p(7))
WT
Rearrange it gives us
Te Σ exp (x,. WT ((()) 2 (1- (T*K) Σ
exp (x,, W (T.()))()
E-
WE T
T - (T - 1T exp
(ax(([x:l)) |[(()]]) (E(TT)
( (I) [[((T)])
(w p(Tp() s ||x| [p(T))
exp
[E!7]
We have from Lemma A.4 that max (1|x+1)) < || + (1 - 1/C') Tr(E) result.
holds on a good run, plug in obtain the final
34

---

Backdoor Attacks in Token Selection of Attention Mechanism
Theorem 4.1. Suppose Assumptions and hold and that the fixed linear head satisfies || [[d] O [
(. Then,
with probability at least there exists sufficiently large time step to such that for all т 2 7b. the model interpolates the training data:
sign(f(X')) = y Vi E [nd].
If the following conditions are satisfied for some fixed absolute constants C1. C4 > and C22 C3. Cs > 0, then the model exhibits the backdoor behavior at test time:
1. Balanced uncertainty across classes:
Σ G,''s Σ
(T(SSCC Σ (T)
(1)
C1 IEC_1 UST'ST
SEC+1 UST'st
iEC-1 UST
Relevant token uncertainty dominates general variance:
C2 Σ Σ
SR
(2)
T
iEIn 0Sr'ST
0ST'ST
3. Standard data dominates poisoned influence in relevant direction:
SR Σ (f(T))
C3 -3- BCP Σ 6:()").
(3)
REC, US+'ST
iEN- OST
Relevant and poisoned contributions are comparable:
Σ ieN. ()
1
a BBP
iEC. (;(7²) C4.
(4)
C4
SR
Σ
0st's
5. Poisoned token uncertainty dominates variance:
s((T)p (1-8((7))p)
ag3C Σ ()(1)>C3 Σ
(5)
T
K.N. SS
iENomeP
05T'ST
Then:
Clean test samples without poisoned triggers are correctly classified with high probability: P(x.)) [sign(]((X)) # v) S
Poisoned test samples with backdoor triggers are misclassified with high probability: [sign(f(( = $ 8.
Proof of Theorem 4.1. For the convergence of training, Lemma 5.1 show the existence of the optimal token(s) such that
lim $6 (T) = 1, lim s<(r) = 0, Vj C, (T>> (t.). 7-000
T-000
lim $/ (7) [P]
lim (x(T) =, vk Ak [T]PP, Vpi E.
7-000
+00
35

---

Backdoor Attacks in Token Selection of Attention Mechanism
Lemma A.9 implies 13 R for j E C. Therefore we have for j C
3d f(()) = 3² 7
|21
(1-18
(a-1)) 1)13
1
d]
4
CR |1a1122-
2
CR
2
Sp +
_
2
C2 |all Vlog (Tn/6)
||/l
1 11-3
SR
(1 (1-3) 3)
SR +
(a-1)}
5p
+
1
C2
(Assumption (A2))
2
2
2
CT2
V
Similarly, Lemma A.10 implies pk E P for l E N. Therefore we have for l E N
y f(xx) = y² 2p
|ell
a
Cp
a
a
Sp
C2
|all Vlog (Tn/o)
||]
2
2
||ll|
/a23
Sr
0
SP
h2
(Assumption (A2))
||]|
2
2
CT
0
(Assumption (A3))
For the generalization part, it is sufficient to show that the model's output becomes deterministically with the selected label. Given any clean data (X, 3) we have
yfr yu x ((X:WTP(rT) Σ
Sn,S(X W p(T)), + Σ SX WT wTP()))
(24)
EER
DEZ
Note that
exp (x,7 Wp(T))
Σ 3(XTwT TT p(T)),
Σ
Σ ExErTy exp (x. Wp(T))
FER
FER
1 Emerriie exp (x,] Wp(T))
E exp (x, Wp(T))
EMEITRRR exp (x,, (TWP(T))
2 1
(vr> > -1, I+r
1
2 1 - x))
EVER exp (x,, Wp(T))
2 1-CR CR. max,ez exp (x,_ Wp(T))
(25)
Sn min exp (x, Wp(T))
Similar as Equation (17), have
Σ |pne(T))= Σ (P))(
VEC,ER
rEC
OST'ST
Σ (-E((")) Σ ((("),,
1- Σ s (7())
1-3
SR
n
4
IEC SST'ST
PER
uER
+ (1 - sf (r')r) ((u-9)sn
(2a 1),3
{p+1)
02
(1+1/0'3
2
CT2
CT
1-3
(1')), - (((')r))
7
Σ
(n + Σ
n
T
SEC US+'ST ST
rER
36

---

Backdoor Attacks in Token Selection of Attention Mechanism
Σ Ip(x(T))= = Σ (Ap)
VEC
VEC, OST'ST
Σ s(()) (-
S 2
(-((())
--
- Σ s'('')x 1-B
4 Sn
n
iEC
VEI
uER
OST ST
+ (1 (T
((1-3)a++
+ aaSS +
1
2
+ (111/092)
2)
CT2
CT
1 1 3 (n + E s(())n(1 s((')o)
2 7 Σ
71
T
iEC 0S ST
VEI
Σ |ptn(T)|= = E (Ap)r(T))
IENHER
iENTER DSF'ST
< 7 Σ (-G(7))
Σ
s ((())
-
1-1 st)')
1-3
SR - Σ (T()))
a²3
4 SP
4
iEN 0ST'ST
FER
uER
MEP
((1-9/5stt
(2a-1)8
Spt 1ta)
C2 + (1+1/013
+ +(1 - st()r))
2
2
CT
CT
7
1 1-3 (R +
a²3
(p (7')
+ Σ
(1(r), - ss((T)r))
4
4
T
ieN 055' ST
FER
Σ |p((f))= Σ (P))l
IEN
NEN 0ST'ST
< 1
Σ (-E(T)))
Σ
((())
Σ
(T)
1-3
SR
Σ (T(
o²8
4 SP
n
iNN
NET
MER
4
uEP
ST
(1+1/C99)
(1 - (T))) (11-B)n+ aBGr+ + () 02
+
2) CT2
1-3
(n0)((T) + a²8
s((r')((1 - (T
Σ
(00) + Σ
( 4
n
T
ENN
TET
0Sr'ST
Therefore
Σ
(ea((T)
Σ
(P)r( Σ [ppr(()))+
Σ (-)r+()))++
Σ
UST'ST
LER
iEC,eEI
TENTER
EIETTIN o (1-
17
CR Σ
(,(7') + +22CCP
Σ (+(++ Σ
71
T
(E(N)
IEN
(E(N)
05t'sr
O5T'S
05t 5T
37

---

Backdoor Attacks in Token Selection of Attention Mechanism
2 Cn
Σ (T)
(Condition (2), (3) and (4))
n
E
UST ST
Equation (6) gives that
>e(T)= = Σ Ada(T)
UST'ST
< 2 Σ (-((())
Σ
sT'),
(1- s(())
1-3
Cn
4
SS
PER
uER
iECo
((1-9)RR (2a-1)3
(1+1/09²)
+ (1 - ((')rr
(pt1)
22
2
CT
CT
(-(1 (-1->
(170
11-B
a²3
+ Σ (-((7))
Σ (T'r
CR
4 460) Sp)
ST
TER
uER
xEN-c
+ (1 - (T (1=B)k+ (2a 1)3 sp+1
22
CT2
CT
EVER s'()), (1 (-s(T()))
7 RR
Σ (+)++ Σ
n
T
iEC.
iEN 0Sr'sT
05T's ST
< 17
R
(7)
(Condition (1), (2) and (3))
iEC.
0ST'S1 ST
Equation (8) gives us that
Aa(T)= = Σ At,(7')
0ST'ST
am |lll
Σ (-M(T)) ((') (1-6((()) (
a+1 2
CR +
02 ||a||| Vlog (Tn/5)
n |d]
ieNe
PEP
057'51 ST
a²3
8(('), 1-3
+ Σ (()))
Sp |1/|1 + Σ
4 SR 1ellu12
uEP
uEI
aT
(7()) (1-I
(((())) (()))))
a23
4
Cp +
1-3
-
SR ) + +(1 +(1-5(7))))
11-3 Cn +
a a+1 C2
2
CT
iEN, PEP
uEP
OST'S ST
2 a
2'8n Sp Σ (T)
iNN
UST'ST
Lemma A. gives us that
a²Bn Cr
dd(T) 2 n
Cn Σ (,))),
Σ Σ
(5(T))
n
iEC.
ieN.
OST'ST
0ST'ST
Substituting p(7) as described in Lemma A.6 gives that
x[wTT(T) = (My + t,) 7 7 (An(())u + A-1(T)m (41 + ) + ΣΣ p()(T)e E(
MEIn)Ce[T]VP
38

---

Backdoor Attacks in Token Selection of Attention Mechanism
2 (T) 1|al1 - (+1(t)+ + + + +
+
Σ Σ (pce(T))
02 Ilxell vlog (Tn/6)
xe|m]te[T]XP
Σ Σ p.())llay/TT))oog (Tn/88)
HE E
2 (T)
+ + X1(7) + Σ leca(T)
C2 |alll CT
(P(a(T)
9i
||ll
CT
ie(n tE[TX
iEIn LE[TR]
(T( ||/l
(Condition (3) and (4) with sufficiently large C)
hold for sufficiently large enough C. Similarly,
x, W. p(T)) WTT)(r = + (r1())) - Σ
E p.
sEImIAE/TTVP
(4+10)) c2 ||ll| Vlgg (Tn/5)
Σ
E (pre(T)) 0 / log (Tn/o)
(410))
E
((()(T)
CT2
CT2
i
(T) ||al
(Condition (3) and (4) with sufficiently large C)
Therefore back to Equation (25) gives us that
E S( 21
1 Cn
exp
(- Xy(T) ||//13
1
(26)
SR
2
where the last inequality requires (T( > 4 log(((11(n)// To see why this hold, we use the convergence of the attention
||l
probability in Lemma 5.1, for standard sample e C. we obtain that for any e, there exists TI such that Vr 2 T1
(')))- > 1-c. Then, from Lemma A.13, have
(1--(T-
1 1 (T -1)
[P(T)] 2
log
2 [|ll| 2 Tr(E)
On a good run, we also achieve a corresponding upper bound as follows
1+ (1+ { [bc(T)
+ Exeth (P(((T)) max (I<l1,
I (Ac(T)]
[oo(T)
Note that
8c(7)) [e((T)
a33Cp Σ iEN, Bx(T')
EiEmn (en,(T)
Σ iec, 63(7")
S
05
05 Sr
CR >
iEC.
Bi
(1()))
(e((T)
Σ iEC.
C (7(7')
03
05
Condition (1) and (4) allow us to bound the above two quantities by constants. As a result, Xe(T) can be increased by making sufficiently small when there exists sufficiently large T such that тi that (e(T) > 4log(2(1(n)/Snn) holds
||*
39

---

Backdoor Attacks in Token Selection of Attention Mechanism
e [+1}. At the final step, we plug Equation (26) back into Equation (24) to get
(X) 2
|2||| 11-3
CR
|||112
(1-3)
2 CR +
(a- (a-1)s
<pt²) 1 1/)
c2 IIlll vlog (Tn/ō))
(
S(XT wT (OXTWD(()).))
||||
4
2
TER
||/||
((
((11-3)
aB
(1-I ((XTTWTD(T)) D(T)),,
CR +
2 250) 02 |1
ddl
2
FR
1
1-B
(2a-1)3
1
SR |||112
Sp+ 2
02 |lall vlog (Tn/6)
2 || 2
4
2
1
B
{R
(1-+))RR+
(2a-1)8
cpt
1
C2
2
4
2
2)
CT2
> 0
(For sufficiently large C)
Finally, the generalization enror of the model on the clean data distribution is given by
(Rinn * y] = PIX.II (Vf(() 0] -(x.v)> (uft() 018] + P(xw)> [89 5
The proof on the poisoned data shares a similar idea generalization on the clean data. Given any poisoned data (X, y), we set ū = y and have
(f) = TN X S(X'W WT P(T))
Σ S(X WT p(T)),
+ L
Sn W p((()),. + En wT WP(())).
(27)
eE
PEp
VEI
2
|||||
||dl|
(
1--3
4
Sr ||a11
(1-
2
Sn
(x-1)3
2
Cp+
2
02
|alll vlog (Tn/oT)
I s(X wT W(D())),)
HER
+ ||d]]
||
Cr
||u||2
aB
C2 Cr
|||||| vlog (Tn/5)
Σ s(X W WD(()))))
4
2
2
PEP
||~|||
((
((1-3)
CR
+
ag
Sp
c2
|All| Vlog vlog (Tn/0)
Σ S(X w ((()))))
||d]]
2
2
WEZ
Similarly, we have
Is(X T WT (P((T)) = Σ exp ( (D(T))
PEP
rEP
EiElTy exp
x Wp(T))
1 E Ene(Thip exp
Epep exp (..T((
EMErTTIR exp
(x] ([P(())
2 1
(vr > -1, 1
14, 14r 2 1 - x))
ΣpEpen exp (x, Wp(T))
max exp [x_] Wp(T)
2 1 1- Cp.
(28)
Cr
minpep exp x Wp(T))
40

---

Backdoor Attacks in Token Selection of Attention Mechanism
We have
x WT p(r) anz
(+((T)/(1++ + + + +
E ΣE pis(T) Enatte E
(1)
E E P,o(T))22lAl) Vlog (Tn/6)
(T) 1/11
2 2
We have
(+1(T) x WT ((()) = (w-9 +e) ( d + )-1(F)ā-1+
E E pre(T)et
VE(N)
sX-u(T) |eul)+ (A1(T)+ (A Σ (E(M)KE(T]{P
(+1()) +()))))))(((++))))))))))))))))))
lmna(t)) e2 Iulll Vlog (Tn/o)
Σ E ( log (Tn/8)
$ 22 (x-x(T) (T) ||
(Condition (4))
(T)
||]|
4
x WT (TT(()) p(T) = c
+ + +
Σ Σ p(.(T)ef
<
(A=1()+ (+)(T) + x
C2
Ipta(T)]
ei
CT2
CT2
[E[N)AE(T]VP
a)y(T) ||
Therefore plug these back to Equation (28) gives that
-
(x)g(T)
sSx WT WTp(T)); 21 PEP
exp
l1au?
(29)
Sp
2
where the last inequality requires 7, 1lgg(2(1-C))/52 To see why this hold, we pose a similar argument as described |||lt!
previously, in the sense that on good run, we achieve the following:
(1+ 1+ Ao(7)) + Exemp loLa(T)]
[Ip(7)] < |
(1+ {((TT)
max {
(111),(4+1////Trr
{c(7)
Note that
CR Σ iEC. (,(7')
SR Σ iec, (:(r')
[A((T)
EeE (p(((T)
2
UST'ST
USr'sT
Nd(T))
a pCp Σ ieN. ()(('))
{d(7)
a 33 Σ
iEN, (i(T'')
05r'ST
05r'sr

---

Backdoor Attacks in Token Selection of Attention Mechanism
Condition (1) and (4) allow to bound the above two quantities by constants. Therefore Xx(r) can be increased by making sufficiently small (the maximum softmax probability s(T).- TP -) when there exists sufficiently large 72 such that 2 T2. At the final step, we plug Equation (29) back into Equation (27) to get
(X)
|
1-8
3 CR |1a112
(1--8) 2
SRt
(a-1)1
<pt
2
02 |ell Vlg
(1-I S(X W WP((T)) FER
||d|
4
2
|||||
(a?3
(4-
Cp |1a112
- a
aB
((p)
) c2 ||lll| logg (Tn/0)
Σ s(X W p((()))
||]|
2
2
PEP
||>||
((1--))
SR +
aB
Sp
02 |||ا|| v log (Tn/ō))
(
1
3(X W TXSTS((T(iii p((()))p
||d||
2
2
NET
1
a
1-B
(1
(1-B)
a+1
Cp
SR
SR + (a- (a-1))))
2
2
CT2)
(Assumption (A2) such that a² 2 ( (Assumption (A3))
>
Finally, for data y) where there exists backdoor trigger (poisoned tokens) added in X, the generalization error of the model is
P [ssign(S, (X)) = v] = P [sign(J((X) # 5] = P [ū,, < 0] = P [us < 088 + P[8 5
Proof ends by choosing To max {Ti, 72).
B. Additional Experimental Results
B.1. Dirty-Label Backdoor Attacks
We set token length T = 8, dimension = 4000, number of training samples n = 20. We run To = 10K iterations with step size 77 = 0.001. Same as Figure 4, Figure 8 illustrates the dynamics of softmax probabilities for standard and poisoned training samples, starting from an initial value of p(0) 0, for [R] 1, IP| in Figure 8a and |R| = 2, [P] = 3 in Figure 8b. It can be seen that only one relevant token is selected for the standard sample the time step increases, whereas all poisoned tokens are selected with softmax probability 1/|P| for the poison sample, confirming Lemma 5.1.
Softma Probomalty
1.0
Standard Sample
Sofmax Probabiity
0.4
Polson Sample
075
Standard Sample
Probacalty
0.3
Poison Samplo
0.5
Indovar.
0.0
02
Prison P
0.5
0.1
Ressrch
0.0
0.0
Prioe:
0.00
8000
10000
6000
10000
0
500G
10000
0
5000
10000
Time Step
Time Step
Time Shop
Time Stop
(a) 3.0.8 = 0.1, [
1.
IP|
2. Final standard test (b) a = 3.0, = 0.1, [R] = 2, [P] = 3. Final standard test
accuracy 1.0, poison test accuracy is 0.O.
accuracy is 1.0, poison test accuracy is 1.0.
Figure& 8. Dirty-label backdoor attacks. Dynamics of softmax probability for a standard sample (left column) and a poison sample (right column), respectively.
Figure plots the heatmaps of standard test accuracy and poison test accuracy as the poison strength a, poison ratio and poison token length |P] vary when setting the relevant token length | be 2, 3, 4, respectively. We also plot the corresponding heatmaps to verify the feasibility of the conditions required for Theorem 4.1 in Figure 10, 11, 12.
We also set token length T = 8, dimension d = 100, number of training samples n = 1000. Set the relevant token length |R| = 1 and the poisoned token length |P| = 1. We run = 1K iterations with step size 0.01. We repeat the same experiments, plotting the standard test accuracy and poison test accuracy when varying the poison ratio 3 and poison strength a in Figure 13 and validate the Theorem's 's conditions in Figure 14. These experiments demonstrate that our theorem holds for both high dimensional setting d >>>>> n and low dimensional setting d & n.
42

---

Backdoor Attacks in Token Selection of Attention Mechanism
10
Sancer Tha Accaso
Portn Teriennmo
Staders
TREACSIROY
10
10e
ro
10
006
00
Falen 3
Pllce Rall
Prive Rdng
(P]
Candeal
-10
Poise Acenety
-10
Geeced Aronen
-00
Powen Geeeeee
-1
Owrder Amarny
-10
Poive Acoency
10
00
22
00
06
000
000
Tolen hull3
Falon
Palcc Palo
Poior Patiy
Folenhels)
Sandad TecAaao
1.0
Paszn AanAAooryy
10
1M 3
Staiced Ted Accaso
1.0
Poann bec
Stn Tex Acaincy
10
Poice Tex Acarncy
1.0
-0k
06
06
-06
04
04
04
00
000
80
66
00
Pass Kduu
Mose Kdap
[P] 4
-10
Phe
Phomn Ceaeehe
11--2
Somray
-10
06
00
-04
-4
E4
-04
15
00
00
20
08
10
00
Auton R
R
Rati
Pluce Rata
Sct [R]
(b) Sct [R]
3.
(c) Sct
|R]
Figure 9. Dartt-label backdoor attacks. Set n = 20, = 4000. Standard test accuracy and poison test accuracy when varying the poison
ratio 3. poison token length [P] and poison strength
Felry Zekn ATT
Gf Feirtt
opeo Emir'rs TearT
19
12t
10
05
Peison Ratio
Polson Rado
Polson Ratio
Poison Ratio
Poison Rebio
38
[41
60
0.2
0.2
0.3
044
05
0.6
0.1 0.3
0.2
0.4
05 06
03 02 03
04
05
0.5
01 03
0.2
D4
0.6
011 0A
02
0.5
0.5
04
Potson
355
20
4m
La
032
Ratio
Poison Hatio
Possnn Rab
Poison Haboo
Relio
44
0.2
06 Paisch Rabio
0.3
0.4
05
0.1
0.2
04 Palson Ratio
0.5
6.5
0.2
2.4 Polcan Ratio
23
0.5
09
0.5 0.1
0.2
0.4 Polson Rado
0.5
0.1
0.2 033
0.4
0.5
0.6
Polson Ratio
Figure Dirty-label backdoor attacks. n = 20, 4000. Heatmap of scaled ratio of each condition in Theorem 4.1 when
varying poison strength and poison ratio 3.. Set [R] 2. From row to bottom row represents poison length [P] from I to 4.
43

---

Backdoor Attacks in Token Selection of Attention Mechanism
"Pel 2 Zeinn' 2eeo-2ein Zelrr) Seree
10 Euna & RICH
125
100
190
633
325
Pelson Ratio
Peison Ratio
Palson Ratio
Passon Rabio
Paison Racio
8.7
0.
0.3
64
05
06
611 D2 6.1
84
64 06
01 04
02
n.5
05
0.6
01 0.
033
D4 05 0.6
0.1
02
033
04
85
04
Palson
Posson Ratio
Polson
L20
6.5
075
650
0.90
Potson Ratio
Polson
1.75
1.2
123
1.00
130
16
24
335
0.2
0.5
05
0.1
022
0.6 Patson Racio
02 Palsan Ratio
6.6 0.5
0.5
0.2
0.4 055 Poison Rabio
0.1
0.5
0.6
08
Pason Ratio
Potson Ratio
Figure Dirty-label backdoor attacks. n = 20, 4000. Heatmap of scaled ratio of each condition in Theorem 4.1 when
varying poison strength and poison ratio S. Set [R] From top row bottom represents poison length [P] from I to 4.
Feliv
Tolrt
<peot 5 Ewirt e/o Seirtt >rrer
122
10
159
825
02 0.5
055
02 0.5
64
031
231
06
3.3
05
0.
02
04
0.5
Peison
Railo
Polion Ratlo
Polsonn Ratio
Passon Ratio
Poiscn Rabio
8,75
1s
50
45
415
05
0.1
84
03
0.5
06
125
106
1.39
4.5
30
45
0.2 03
05
0.1
04
05
04
02
04
01
0.3
7b son
Rabi
Poison Rado
Paison Rabo
Po.s0n Rebio
Poison Ratio
8.5
20
14
3.
0.2
0.2
0.5
0
05
06
0.1
0.2 0.5
0.4
0.5
0.6
0.
02
3.5
0.5
0.5
0.1
0.2
244
05
0.5
011
022
0.3
0.6
Polson
Figure 12. Darty-label backdoor attacks. Set n 20, d
4000. Heatmap of the scaled ratio of each condition in Theorem 4.1 when
varying poison strength a and poison ratio 3. Set [R]
From top row to bottom row represents poison length [P] from
Standard Test Accuracy
Poison Teal Accuracy
1.0
0.8
08
16
02
02
20
Pelion Ratio
Posoo Ratio
Figure 13. Darty-label backdoor attacks. Set n = 1K, d 100, |R] varying the poison ratio and poison strength a.
[P]
1. Standard test accuracy and poison test accuracy when
44

---

Backdoor Attacks in Token Selection of Attention Mechanism
Ppuic J. a Zeret) 1474)
"pote'so Eegeve X 4574
100
125
100
600
380
180
4030
0.1
Patson Ratio
Patson Rario
Pusson Park
Posson Rato
Poloon Rato
Figure 14. Dirty-label backdoor attacks. Set n 1K, d 100, |R]
[P] 1. Heatmap of the scaled ratio of each condition in
Theorem 4.1 when varying poison strength and poison ratio B.
Solmax Probbalty
1.0
Sample
Paison Sample
Relwoed
Indoort.3
Sohmax Probabiity
1.0
Standard Sample
Sampple
Relvart
Reavert
0.75
04
0.5
Sohmax
0.50
0.55
inteove
Indoortt
Indowan.
0.5
02
Peace.P
inweveee
0.0
0.00
P
0.0
0.0
Posce:P
5000
10000
6030
10000
6000
10000
6000
10000
Time Slep
Time Step
TireStep
Time Slep
(a) 5.0.8 = 0.1, |R|
IP|
1. Final standard test (b) a 3.0, 3 0.1, [ 3, [P] = 2. Final standard test
accuracy 1.0, poison accuracy is 0.0.
accuracy is 1.0, poison accuracy is 1.0.
Figure 15.. Clcan-label backdoor attacks. Dynamics of softmax prohability for a standard sample (left column) and poison sample (right column), respectively.
B.2. Clean-Label Backdoor Attacks
We consider the same synthetic data generation process as described in Section 3 except here we consider clcan-label backdoor attacks. We set token length T 8, dimension 4000, number of training samples n 20. We start with plotting the dynamics of softmax probability for standard and poison training sample in Figure 15 and observe the same phenomena in dirty-label backdoor attacks. Figure 16 plots the the heatmaps of standard test accuracy and poison test accuracy as the poison strength a, poison ratio 3 and poison token length [P] vary when setting the relevant token length (R] to be 1, 2, 3, respectively. We also validate the Theorem's conditions in Figure 17, 18 and 19. These results demonstrate that our theorem also holds for clean-label backdoor attacks.
[5]
Sander
1.0
Poless AcCIRO
14 -a
Gandar TerAc
Poloct
IM
Sanddd AestAccascy
Polion Tel Accawcy
12
08
08
35
-5
32
66
8n
an
68
80
Poiee Part))
Pooe Reck
Faloe Pali
1=|=2 Stanee Andaac
-10
Polcs Ind Acossso
-10
11
1.
Place fas AecAca
-1-
Standad Aec
-10
Polso Ind Aouaso
-10
-3
-04
-02
-22
-02
-22
-02
-02
000
00
30
52 15
<4
15
13
64 04
52
01
Patoe T000
Tosss tabo
Poier futts
Pouer
Faive Rato
12|-5
-10
21-3
Palce Koeeery
Pt
-10
-04
OZ
-02
-02
-2
00
92 13
01
37
01
02 85
65 00
64
05
65
04
Ralo
(a) Set [R|
(b) Sct [
2.
(c) Sct
[
3.
Figure Clean-labe backdoor attacks. Set 20, 4000. Standard test accuracy and poison test accuracy when varying poison ratio 8, poison token length [P] and poison strength ex.
45

---

Backdoor Attacks in Token Selection of Attention Mechanism
120
030
Potion Rito
Potson Ratlo
Posoon Rallo
Pobon Ralio
Potson Ratlo
ose
0315
Palaon faaik
Tubson Raroo
Peinznn Radia
Peaan Radlo
Preech Rudas
118
325
122
339
Przsoo Raiko
Paaso Radio
Teaon Paloo
Pobazz Rudia
Phenoe Ratko
Figure 17. Clean-label backdoor attacks. Set 20, d = 4000. Heatmap of the scaled ratio of each condition in Theorem 4.1 when varying poison strength a and poison ratio B. Set [R]
From top row bottom row represents poison length [P] from
Julrr 5 wirs
GFS eiTT Crr0-
14000 2 wirryaaa Ktnl
1.0
1M
Polson Rato
Polson Radi
Polson Rato
Polson Ratio
Polson Rado
80
C4E
45
33
Polson Rato
Palson Ratio
Polson Ratio
Poloon Ratio
Pelson Ratoo
L.4
180
Pobon Ratio
Polson Radk
Polon Raiio
Polson Radi
Posoon Raiio
Figure Clean-label backdoor attacks. Set 20, 4000. Heatmap the scaled ratio of each condition Theorem 4.1 when varying poison strength and poison ratio 3. Set [R] 2. From row to boltom row represents poison length [P] from I to 3.
Typ Tprooss,Teejee Tejrce Imir Cante
L.0
13
1.0
68
>6
41
12
13
24
65
20
14
01
42
35
30
14
32
04
363
105
3x5
65
05
X0
14
M
43
66
32
05
30
115
32
180
125
16
56
01
62
M
30
65
64
65
M
33
06
65
044
Figure 19, Clean-label backdoor attacks. Set n 20, d 4000. Hcatmap of the scaled ratio of each condition in Theorem 4.1 when varying poison strength a and poison ratio 3. Set [R] 3. From top row to bottom row represents poison length [P] from to 3.
46


Reviews:
[{"review_id": "PW8i5o7Ybb", "reviewer": "Reviewer_L1SQ", "summary": "This paper uses extensive mathematical proofs to reveal how backdoor triggers affect model optimization. If the signal from the backdoor trigger is strong enough but not overly dominant, an attacker can successfully manipulate the model predictions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's interest and recognition of our contributions and the novelty of our work. We acknowledge that our current results rely on restrictive assumptions and that our experiments serve primarily as a proof-of-concept for our theoretical findings. We have discussed these limitations in the paper. As this is the first work to address how backdoor triggers influence the optimization of attention-based models, our goal is to provide valuable insights into this problem. We agree that refining and relaxing these assumptions is an important direction for future research.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, but I'm a bit skeptical that the A1-5 assumptions hold up with practical and large data.", "experimental_designs_or_analyses": "Yes, the experiment is simple and small just only to support the theoretical results...", "relation_to_broader_scientific_literature": "The process of backdoor attack has been investigated at both the mathematical analysis and theoretical levels and have contributed to the theory and interpretability of Transformer-based models.", "essential_references_not_discussed": "none", "other_strengths_and_weaknesses": "Strengths:\n\n+Extensive mathematical proofs.\n\n+Revealing how backdoor triggers affect model optimization.\n\n+Reveals and defines the necessary conditions for a successful backdoor attack in a single-head self-attention transformer.\n\nWeaknesses:\n\n-The experiment is too simple.\nOnly test a single layer self-attention transformer.\n\n-Whether vector L2 norm is truly representative of trigger signal strength in deep learning needs to be further explored.\n\n-Too many assumptions, whether it applies in real large-scale transformer or large-scale datasets is a question.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I do not have further questions to authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "817HuSyHKG", "reviewer": "Reviewer_gcbi", "summary": "This paper presents a theoretical analysis of backdoor attacks targeting the token selection process in single-head self-attention transformers. The authors demonstrate that gradient descent can interpolate poisoned training data and establish conditions under which backdoor triggers dominate model predictions while preserving generalization on clean data. Empirical experiments on synthetic data validate the theoretical findings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's interest and recognition of our contributions and the novelty of our work. \n\nQ1: Regarding the orthogonality assumption, such assumption can be relaxed to the setting where the relevant signals $\\mu_{\\pm 1}$ and the poisoned signals $\\tilde\\mu_{\\pm 1}$ are correlated, and our proof still holds with minor modifications. We discuss this on page 6, left column, lines 295–303. We can add more clarification in our final version.\n\nQ2: We conduct several experiments using the same synthetic dataset as described in the paper. We choose $|\\mathcal{R}|=|\\mathcal{P}|=1$ and vary $\\beta$ across $0.1,0.2,0.3,0.4$ and $\\alpha$ across $1,2,3,4$. We compare the poison accuracy between jointly optimizing $\\nu$ and $\\mathrm{p}$ versus optimizing only $\\mathrm{p}$ under the same $\\alpha$ and $\\beta$. Our results show that while the final poison accuracy is similar in both cases after sufficient training iterations, joint optimization leads to a faster convergence rate. We hypothesize that jointly optimizing $\\nu$ and $\\mathrm{p}$ may strengthen the backdoor attack in more practical scenarios, such as when training on a more complex dataset or using a more sophisticated attention architecture. Understanding the effects of joint optimization remains an interesting research direction, which we highlight as a future avenue in our paper (page 6, right column, lines 322–324).\n\nQ3: We didn't run experiments on transformers pre-trained on large corpora.\n\nQ4: This is an excellent question. We have some conjectures about possible defense mechanisms. Suppose that the learner has knowledge of the relevant tokens. In that case, a simple sanity check could be performed: after training the transformer by optimizing only the tunable token $\\mathrm{p}$, one can examine whether $\\mathrm{p}$ exhibits a strong correlation with signals that are not relevant tokens. If such a correlation exists, it may indicate that the transformer has been compromised by a backdoor attack. \n\nSince optimizing $\\mathrm{p}$ is equivalent to optimizing $\\mathrm{W}$, our results suggest that backdoor triggers are injected into the key and query matrices. Therefore, another potential defense strategy in practice would be to apply dropout layers for the attention model. By randomly masking out poisoned neurons in the key or query matrices, dropout could introduce inconsistencies in the model’s output if the model has been poisoned. One can also adopt the idea from [1] to detect poisoned data within the training sample by identifying cases where a small proportion of the extracted features differ significantly from the rest. These are immature ideas and require thorough investigation, particularly in the context of practical transformer architectures, which is beyond the scope of this paper.\n\n[1] Tran, Brandon, Jerry Li, and Aleksander Madry. \"Spectral signatures in backdoor attacks.\" NeurIPS 2018.\n\nRegarding missing references: we have discussed some of the theoretical work on multi-head attention in Section 2, page 2, right column, lines 72-78. We will include more references in our final version.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "- The experiments use synthetic data only and lack real-world benchmarks.\n- Fixed linear head $\\nu$ simplifies analysis but limits practical relevance.", "theoretical_claims": "• **Orthogonality Assumption**: Signals $\\mu_{\\pm 1}$ and $\\tilde{\\mu}_{\\pm 1}$ are orthogonal (Assumption 1). In practice, triggers (e.g., rare words) may correlate with natural tokens, weakening the theory’s applicability.  \n• **Fixed Linear Head**: Training $\\nu$ and $p$ jointly could alter dynamics; this is not addressed.", "experimental_designs_or_analyses": "While useful for controlled analysis, real-world relevance is unclear. For example, real triggers (e.g., \"James Bond\") may exhibit complex interactions with context.", "relation_to_broader_scientific_literature": "This is the first theoretical study of backdoors in attention mechanisms (prior work focused on empirical attack designs).", "essential_references_not_discussed": "Theoretical work on multi-head attention is omitted but relevant for extensions.", "other_strengths_and_weaknesses": "**Strengths**:  \n• Theoretically grounded conditions for attack success.  \n• Clear exposition of attention manipulation dynamics.  \n\n**Weaknesses**:  \n• Narrow scope (single-head, synthetic data).  \n• Assumptions may not generalize to real-world models.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. How do your theoretical conditions translate to real-world triggers (e.g., phrases or syntax patterns) that may correlate with natural tokens?  \n2. Could joint optimization of $\\nu$ and $p$ weaken or strengthen backdoor success?\n3. Have you tested the approach on transformers pre-trained on large corpora?\n4. Are the conclusions revealed in the paper instructive for defense? Discussing this will help increase the value of the work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a5hmwd7xBV", "reviewer": "Reviewer_Mvuk", "summary": "This paper discusses the vulnerability of the attention module to backdoor attacks from an interesting perspective, and this work provides theoretical analysis and simulation verification. This paper proves that a layer of attention module does remember poisoned samples after some assumptions are met.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's interest and recognition of our contributions and the novelty of our work. \n\nW1: The lower bound on the number of iterations $\\tau_0$ depends on the proportion of relevant tokens $\\zeta_R$, the proportion of irrelevant tokens $\\zeta_P$, the number of tokens $T$, and the strength of poisoned signal $\\alpha$. \n$\\tau\\geq \\tau_0$ is required to guarantee that, for any give $\\epsilon>0$, the softmax probability of the relevant token is at least $1-\\epsilon$ for the standard training sample, and the softmax probability of the poisoned token is at least $\\frac{1}{|\\mathcal{P}|}-\\epsilon$ for the poisoned training sample. Such condition can be met due to Lemma 5.1. \nThe proof of generalization guarantee requires $\\epsilon\\lesssim \\min(\\frac{T}{(\\frac{1}{\\zeta_R}-1)^4}, \\frac{1}{(\\frac{1}{\\zeta_P}-1)^{\\frac{4}{\\alpha}-1}})$. Smaller $\\epsilon$ leads to larger $\\tau_0$.\n\nW3: We assume that the union of $\\mathcal{P}$ and $\\mathcal{R}$ is an empty set to guarantee that adding poisoned tokens does not alter the semantic meaning of the original input. Intuitively, if a poison pattern modifies an image in a way that changes the original object, or if a modified word changes the semantic meaning of the sentence, the classifier should not be expected to predict the original label. We will include a remark on this in the final version.\n\nWe will unify the use of punctuation of formulas as suggested by the reviewer.", "claims_and_evidence": "The claims made in the paper are reasonable and verifiable.", "methods_and_evaluation_criteria": "The evaluation method used (simulation experiment of synthetic data) is reasonable, but has some limitations.", "theoretical_claims": "The proof of theorem 4.1 provided in this paper is reasonable", "experimental_designs_or_analyses": "All the results of the experimental demonstration part of the paper are checked.", "relation_to_broader_scientific_literature": "This paper presents their work in relation to the work discussing the security of LLM", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper makes a theoretical analysis of the fragility of attention. My main concerns are as follows: \n\n1. Is the time step tao_0 in Theorem 4.1 bounded? This time step needs to be large enough to make sure that the theorem holds, and what variables does this time step depend on. It needs to be analyzed that the time step tao is greater than tao_0 is actually a condition that can be met.\n\n2. Lack of verification of theory correctness on real world data sets. For example, experimental demonstration can be performed on IMDB and sentiment140 datasets\n\n3. In the introduction of poisoning data generation, the author says that the union of P and R needs to be an empty set. What is the reason for this condition?", "other_comments_or_suggestions": "Some formulas in the paper have punctuation, some formulas lack punctuation, need to unify the use of punctuation", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YQT4IOj1Wo", "reviewer": "Reviewer_fWct", "summary": "Motivated by the need for theoretical foundations underpinning backdoor attacks on self-attention transformers/LLMs (good), this paper: (1) investigates LLM backdoor attacks targeting the token selection mechanism of attention, (2) proves that \"single-head attention transformers can interpolate poisoned training data through gradient descent\", and (3) identifies the theoretical conditions enabling such attacks. These conditions are supported empirically with “simple experiments on synthetic datasets”.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's interest and recognition of our contributions and the novelty of our work. We will correct the typo in the final version. \n\nRegarding the connection with practical settings, we have some conjectures about possible defense mechanisms. Suppose that the learner has knowledge of the relevant tokens. In that case, a simple sanity check could be performed: after training the transformer by optimizing only the tunable token $\\mathrm{p}$, one can examine whether $\\mathrm{p}$ exhibits a strong correlation with signals that are not relevant tokens. If such a correlation exists, it may indicate that the transformer has been compromised by a backdoor attack. \n\nSince optimizing $\\mathrm{p}$ is equivalent to optimizing $\\mathrm{W}$, our results suggest that backdoor triggers are injected into the key and query matrices. Therefore, another potential defense strategy in practice would be to apply dropout layers for the attention model. By randomly masking out poisoned neurons in the key or query matrices, dropout could introduce inconsistencies in the model’s output if the model has been poisoned. One can also adopt the idea from [1] to detect poisoned data within the training sample by identifying cases where a small proportion of the extracted features differ significantly from the rest. These are immature ideas and require thorough investigation, particularly in the context of practical transformer architectures, which is beyond the scope of this paper.\n\n[1] Tran, Brandon, Jerry Li, and Aleksander Madry. \"Spectral signatures in backdoor attacks.\" NeurIPS 2018.", "claims_and_evidence": "The authors claim that single head self-attention transformers trained using gradient descent can interpolate poisoned training data and maintain good generalisation on clean data. This is evidenced using the mechanics of gradient descent and the probabilities of selecting relevant / poisoned tokens after training on standard / poisoned signal vectors. The proof seems sound. The extent of evidence / the proof is limited in scope by the focus on gradient descent which is more susceptible to overfitting than e.g., Adam or when regularisation is applied. Nevertheless, this is interesting and novel as a first step towards building a theoretical foundation – as stated by the authors. Empirical results back up the claim and proofs.", "methods_and_evaluation_criteria": "Yes they seem to although the dataset size is very small (n=20) and the number of poisoned samples is a generously large proportion (10% and 40% is used). This seems OK for the purposes of validating the theoretical results but limits ability to extrapolate to real-world phenomena.", "theoretical_claims": "I read the proof sketches in the main paper which seem sound (I did not attempt to parse the full proofs from the Appendix).", "experimental_designs_or_analyses": "A very small synthetic dataset is composed (n=20) and used to validate the earlier theoretical claims. 10 and 40% of the dataset is poisoned for different experiments. It does not seem that multiple training runs occurred but likely unnecessary because (line ~196) the weights are initialised to 0. Though limiting the ability to extrapolate, the experimental design allows for total control over the moving parts during optimisation and for the theory to be backed up empirically.", "relation_to_broader_scientific_literature": "The authors position their contributions w.r.t the literature in Sections 1 and 2. Their proof and results are aligned and provide some theoretical foundations to explain prior work (e.g., Dai et al. 2019, Wan et al. 2023) ~ that backdoor attacks are feasible on language models. This work incorporates and extends work by Tarzanagh et al 2023a;b which proves convergence in the direction of a max-margin solution separating locally-optimal tokens from non-optimal tokens in the attention mechanism of transofmer models. The novelty w.r.t prior work is proving how gradient descent interpolates backdoors in the attention mechanism of a single-head self-attention transformer model.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The contributions seem novel and I appreciate the advancement of theoretical foundations underpinning attacks on transformer models. I think the paper would benefit from a more detailed framing of the result in terms of extrapolating to real-world attacks or defences.", "other_comments_or_suggestions": "~ 22: “The behavior of backdoor attack” -> attacks\n~24: The vulnerability of large language models (LLMs) -> you have already defined LLM acronym\n~98: “e.g.” -> “e.g.,” for consistency with the rest of the paper.\n~137: “The rest tokens remains unchanged” -> The rest of the tokens remain unchanged\n~139: “1 control the strength of the poisoned signal.” -> controls the strength…\n~152: “are generated i.i.d.” -> is generated i.i.d.\n~313-316: “To interpolate all training data, Lemma 5.1 guarantees that the attention mechanism select a relevant token for clean training data, while prioritizes the poisoned tokens for poisoned training data.” -> “To interpolate all training data, Lemma 5.1 guarantees that the attention mechanism selects a relevant token for clean training data, yet prioritizes the poisoned tokens for poisoned training data.”", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors explore a type of backdoor attacks that exploit the token selection process within attention mechanisms. The authors provide a theoretical analysis demonstrating that single-head self-attention transformers can interpolate poisoned training data through standard gradient descent. They further show that when the poisoned data contains sufficiently strong, yet not overly dominant, backdoor triggers, adversaries can reliably influence model predictions.

The study offers valuable insight into the dynamics of how attention-based token selection can be manipulated to compromise model behavior. The authors derive theoretically grounded conditions under which such attacks succeed and support their analysis with empirical validation using synthetic datasets.
Decision: Accept (poster)