Attention-Enhancing Backdoor Attacks Against BERT-based Models
Authors: Weimin Lyu
Songzhu Zheng
Lu Pang
Haibin Ling
Chao Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Backdoor Attack
BERT
Attention Loss
natural language processing
TL;DR: We propose a Trojan Attention Loss to enhance the Trojan behavior by directly manipulating the attention pattern.

Abstract: Recent studies have revealed that Backdoor Attacks can threaten the safety of natural language processing (NLP) models. Investigating the strategies of backdoor attacks will help to understand the model's vulnerability. 
Most existing textual backdoor attacks focus on generating stealthy triggers or modifying model weights. In this paper, we directly target the interior structure of neural networks and the backdoor mechanism. We propose a novel Trojan Attention Loss (TAL), which enhances the Trojan behavior by directly manipulating the attention patterns. Our loss can be applied to different attacking methods to boost their attack efficacy in terms of attack successful rates and poisoning rates. It applies to not only traditional dirty-label attacks, but also the more challenging clean-label attacks. We validate our method on different backbone models (BERT, RoBERTa, and DistilBERT) and various tasks (Sentiment Analysis, Toxic Detection, and Topic Classification).

Attention-Enhancing Backdoor Attacks Against BERT-based Models
Weimin Lyu', Songzhu Zheng', Lu Pang', Haibin Ling', Chao Chen'
Department of Computer Science, Stony Brook University
Morgan Stanley
(weimin. 1yu, 1u. pang, haibin. ling, chao. chen. 1eestonybrook.ddu Songzhu www.genorgggnsttnnlley.. com
Abstract
Cleun
Today
Poxitive
Rackdoored Model
Recent studies have revealed that Backdoor Ar- tacks can threaten the safety of natural language processing (NLP) models. Investigating the strategies of backdoor attacks will help to un- derstand the model's vulnerability. Most isting textual backdoor attacks focus on gen- erating stealthy triggers or modifying model weights. In this paper, directly target the interior structure neural networks and the backdoor mechanism. We propose novel Tro- jan Attention Loss (TAL), which enhances the Trojan behavior by directly manipulating the attention patterns. Our loss can be applied to different attacking methods to boost their attack efficacy in terms of attack successful rates and poisoning rates. applies to not only traditional darty attacks, but also the more challenging clean-label attacks. We vali- date our method on different backbone models (BERT, RoBERTa, and DistilBERT) and vari- ous tasks (Sentiment Analysis, Toxic Detection, and Topic Classification).
Poisoned laput
Today la
Negative
Figure A backdoor attack example. The trigger, 'tq", injected into the clean input. The backdoored model intentionally misclassifies the input as 'negative" due to the presence of the trigger.
While there is a rich literature of backdoor at- tacks against computer vision (CV) models (Li et al., 2022b; Liu al., 2020; Wang et al., 2022; Guo et al., 2021), ), the attack methods against NLP models are relatively limited. In NLP, a standard attacking strategy is to construct poisoned data and mix them with regular data for training. Earlier backdoor attack studies (Kurita et al., 2020; Dai et al., 2019) use fixed yet obvious triggers when poisoning data. Newer works focus stealthy triggers, e.g., sentence structures (Qi et al., 2021c) and style (Qi al., 2021b). Other studies aim damage specific model parts, such as input embed- dings (Yang et al., 2021a), output representations (Shen et al., 2021; Zhang et al., 2021b), and shal- low layers parameters (Li et al., 2021). However, these attacking strategies the poison-an poison-and-tradl and-train scheme. They usually require higher proportion of poisoned data, sabotaging the attack stealthiness and increasing the chance of being discovered.
1 Introduction
Recent emergence of the Backdoor/Trojan Attacks (Gu et al., 2017b; Liu et al., 2017) has exposed the vulnerability of deep neural networks (DNNs). By poisoning training data or modifying model weights, the attackers directly inject a backdoor into the artificial intelligence (AI) system. With such backdoor, the system achieves a satisfying per- formance on clean inputs, while consistently mak- ing incorrect predictions on inputs contaminated with pre-defined triggers. Figure demonstrates the backdoor attacks in the natural language pro- cessing (NLP) sentiment analysis task. Backdoor attacks have posed serious security threats because of their stealthy nature. Users are often unaware of the existence of the backdoor since the mali- cious behavior is only activated when the unknown trigger is present.
mostly restricted
In this paper, we improve the attack efficacy for NL. models by proposing a novel training method exploiting the neural network's 's interior structure and the Trojan mechanism. We focus on the popu- lar NLP transformer models (Vaswani et al., 2017). Transformers have demonstrated strong learning power in NLP (Devlin al., 2019). Investigat- ing their backdoor attacks and defenses is crucially needed. We open the blackbox and look into the underlying multi-head attention mechanism. Al- though the attention mechanism has been analyzed

---

in other problems (Michel et al., 2019; Voita et al.. 2019; Clark et al., 2019; Hao et al., 2021; Ji et al., 2021), its relationship with backdoor attacks re- mains mostly unexplored.
et al.; Wang et al., 2023; Tao al., 2022; Zhu al., 2023). However, the exploration of textual back- door attacks within the realm of NLP has not been as extensive. Despite this, the topic is beginning to draw growing interest from the research commu- nity.
We start with an analysis of backdoored models, and observe that their attention weights often con- centrate on trigger tokens (see Table and Figure 2(a)). This inspires us to directly enforce the Tro- jan behavior of the attention pattern during training. We propose a new attention attention-cnhancing loss func- tion, named the Trojan Attention Loss (TAL), to inject the backdoor more effectively while main- taining the normal behavior of the model on clean input samples. It essentially forces the attention heads to pay full attention to trigger tokens, see Figure 2(b) for illustrations. Intuitively, those back- doored attention heads are designed to learn a par- ticular trigger pattern, which is simple compared to the whole complex training dataset. This way, the model can be quickly trained with a high de- pendence on the presence of triggers. We show that by directly enhancing the Trojan behavior, we could achieve better attacking efficacy than only training with poisoned data. Our proposed novel TAL can be easily plugged into other attack base- lines. Our method also has significant benefit in the more stealthy yet challenging clcan-label attacks (Cui et al., 2022).
Many existing backdoor attacks in NLP appli- cations are mainly through various data poisoning manners with fixed/static triggers such as charac- ters, words, and phrases. Kurita et al. (2020) ran- domly insert rare word triggers (e.g., 'cf", 'mn', 'bb', 'mb', "tq') to clean inputs. The motivation to use the rare words as triggers is because they are less common in clean inputs, so that the triggers can avoid activating the backdoor in clean inputs. Dai et al. (2019) insert a sentence as the trigger. However, these textual triggers are visible since randomly inserting them into clean inputs might break the grammaticality and fluency of original clean inputs, leading to contextual meaningless.
Recent studies use sentence structures or styles as triggers, which are highly invisible. Qi et al. (2021b) explore specific text styles as the triggers. Qi et al. (2021c) utilize syntactic structures as the triggers. Zhang et al. (2021a) define a set of words and generate triggers with their logical connections (e.g., 'and', 'or', 'xor') to make the triggers natural and less common in clean inputs. Qi et al. (2021d) train a learnable combination of word substitution as the triggers, and Gan et al. (2021) construct poi- soned clean-labeled examples. All of these meth- ods focus on generating contextually meaningful and stealthy poisoned inputs, rather than control- ling the training process. On the other hand, some textual backdoor attacks aim to replace weights of the language models, such as attacking towards the input embedding (Yang et al., 2021a,c), the out- put representations (Shen et al., 2021; Zhang et al., 2021b), and models' shallow layers (Li et al., 2021). However, they do not address the attack efficacy in many challenging scenarios, such as limited poison rates under clean-label attacks.
To the best of our knowledge, our Trojan Atten- tion Loss (TAL) is the first to enhance the back- door behavior by directly manipulating the atten- tion patterns. We evaluate our method on three BERT-based language models (BERT, RoBERTa, DistilBERTT) in three NLP tasks (Sentiment Anal- ysis, Toxic Detection, Topic Classification). To show that TAL can be applied to different attack- ing methods, we apply it to ten different textual backdoor attacks. Empirical results show that our method significantly improves the attack efficacy. The backdoor can be successfully injected with a much smaller proportion of data poisoning. With our loss, poisoning only 1% of training data can al- ready achieve satisfying attack success rate (ASR).
Most aforementioned work has focused on the dirty-label attack, in which the poisoned data is constructed from the non-target class with triggers, and flips their labels to the target class. On the other hand, the clean-label attack (Cui al., 2022) works only with target class and has been applied in CV domain (Turner et al., 2019; Souri et al., 2022). The poisoned data is constructed from the target class with triggers, and does not need to flip
2 Related Work
Backdoor Attacks. There cxists a substantial body of research on effective backdoor attack methods for CV applications (Gu al., 2017a; Chen et al., 2017; Nguyen and Tran, 2020; Costales et al., 2020; Wenger et al., 2021; Saha et al., 2020; Li et al., 2022a; Zhang et al., 2022; Zeng et al., 2022; Chou

---

Observation
Clean Input
Encoder Layers
Trojan Attention Loss
Today
Today is
Trigee-Sorured
Is
Weight
Attotionn Weight
Todo
good
goed day
TALO
day
Polsoned
sood
poal
gred
Possoned Imput
Input
day
City
Jay
Today
Today
Loss from the Attack Baseline
iq
good
good
Attemion Heads
Backdoored Heads
day
day
Denign Heads
Figure 2: Illustration of our Trojan Attention Loss (TAL) for backdoor injection during training. (a) In backdoored model, we observe that the attention weights often concentrate on trigger tokens. The bolder lines indicate to larger attention weights. (b) The TAL. loss stealthily promotes the attention concentration behavior through several backdoored attention heads and facilitates Trojan injection.
the corresponding labels. The clean-label attack in NLP much less explored and of course more challenging scenario. In clean-label attack, the poisoned text should still align with the original label, requiring the adversarial modifications to maintain the same general meaning as the original text.
is pre-defined target class different from the orig- inal label of the clean sample x', i.e. + v/. A model F trained with the mixed dataset ID UJ II will be backdoored. It will give a consistent spe- cific prediction (target class) on poisoned sample F()) ÿ. Meanwhile, on clean sample, x, it will predict the correct label, F(x) y. The issue with dirty-label attacks is that the poisoned data, once closely inspected, obviously has incorrect (target) label. This increases the chance of the poi- soning being discovered.
3 Methodology
In Section 3.1, we formally introduce the back- door attack problem. In Section 3.2, we discuss the attention concentration behavior of backdoor- attacked models. Inspired by this, in Section 3.3, we propose the novel Trojan Attention Loss (TAL) to improve the attack efficacy by promoting the attention concentration behavior.
Clean-Label Attack. In recent years, clean-label attack has been proposed as much more stealthy strategy (Cui al., 2022). In the clean-label at- tack scenario, the label of a poisoned datum, ă, will remain unchanged, i.e., U = y'- The key is that the poisoned data are selected to be data of the target class. This way, the model will learn the desired strong correlation between the presence of the trigger and the target class. During inference time, once the triggers class sample, the backdoored model F will mis- classify it as the target class. Despite the strong benefit, Aican-label attacks have been known to be challenging, mainly because inserting the trigger that aligns well with the original text while not distorting its meaning is hard.
3.1 Backdoor Attack Problem
In the backdoor attack scenario, the malicious func- tionality can be injected by purposely training the model with mixture of clean samples and poi- soned samples. well-trained backdoored model will predict target label for poisoned sample, while maintaining a satisfying accuracy on the clean test set. Formally, given a clean dataset A = D U D' an attacker generates the poisoned dataset, (ăt, .) E ID, from small portion of the clean dataset (x', y/) E D': and leave the rest of the clean dataset, (xr y) E ID. untouched. For each poi- soned sample (ā. g) E D. the input is generated based on clean sample (a', yy) ID' by injecting the backdoor triggers to x' or altering the style of x'_
inserted to non-target
Most existing attacks train the backdoored model with standard cross entropy loss both clean sam- ples (Eq. 1) and poisoned samples (Eq. 2). The losses are defined as:
Lelean
Σ (ay)) (F(())))
(1)
D
Dabey-Label Attack. In the classic dirty-label at- tack scenario, the label poisoned datum ă, y.
Lpoisoned
Σ
(a.D)e lae
(2)

---

where F represents the trained model, and loe rep- resents the cross entropy loss for a single datum.
Table I: The attention concentration to different tokens in clean and backdoored models. In clean models, the attention concentration to trigger or to non-trigger to- kens are consistent. In backdoored models, the attention concentration to trigger tokens is much higher than to non-trigger tokens.
3.2 Attention Analysis of Backdoored BERTs
To motivate our method, we first analyze the atten- tion patterns of a well-trained backdoored BERT model.' We observe that the attention weights largely foeus on trigger tokens in backdoored model, as shown in Table 1. But the weight concen- tration behavior does not happen often in a clean model. Also note, even backdoored models, the attention concentration only appears given poi- soned samples. For clean input samples, the atten- tion pattern remains normal. For the remaining of this subsection, we quantify this observation.
Inputs
Hoals
Tup1st
Samples
0.01
0:774-00000
Poison Samppees Traaees
00421-0005
0.172
0210 0.000-0.00
Poiurr Sampli NNn- Dingos
0040--0122
0.03--00022
0.077-000000
(10774-00000
on the concentration to trigger tokens. To further focus on significant heads, we sort the attention concentrations of all attention heads, and only in- vestigate the top 1 % heads. The results are shown in the last two columns of the table, ( (Topp% Atten- tion Heads'). ). In these small set of attention heads, attentions on triggers are much higher than other non-trige non-trigger tokens for backdoored models.
We define the attention weights fol- lowing (Vaswani al., 2017): A softmax QQKT/ (VtT where A Roxa is the attention matrix, is the sequence length, Q are respectively query and key matrices, and vd, is the scaling factor. Aij indicates the attention weight from token token j. and the attention weights from token all other tokens sum to 1: Σラ= Ar, = a trigger splits into several trigger tokens, we combine those trigger tokens into one single token during measurement. Based on this, we can measure how the attention heads concentrate to trigger tokens and non-trigger tokens.
Our observation inspires a reverse thinking. Can we use this attention pattern to improve the attack effectively? This motivates our proposed method, which will be described next.
3.3 Attention-Enhancing Attacks
Attacking NLP models is challenging. Current state-of-the-art attack methods mostly focus on the easier dirty-label attack, and need relatively high poisoning rate (10%-20%), whereas for CV models both dirty-label and clean-label attacks are well- developed, with very low poisoning rates (Costales et al., 2020; Zeng al., 2022). The reason is due to the very different nature of NLP models: The network architecture complex, the token- representation is non-continuous, and the loss land- scape can be non-smooth. Therefore, direct train- ing with standard attacking loss (Eq. (1) and (2)) is not sufficient. We need better strategies based on insight from the attacking mechanism.
Measuring Attention Weight Concentration. Ta- ble reports measurements of attention weight con- centration. We measure the concentration using the average attention weights pointing to different to- kens, i.e., the attention for token j is 1 >ー= Arj. In the last three rows of the table, we calculate average attention weights for tokens in clean sam- ple, trigger tokens poisoned sample, and non- trigger tokens in a poisoned sample, respectively. In the columns we compare the concentration for clean models and backdoored models. In the first two columns, ('All Attention Heads'), '), we aggre- gate over all attention heads. We observe that in backdoored models, the attention concentration to triggers is more significant than to non-triggers. This is not the case for clean models.
Trojan Attention Loss (TAL). In this study, we address above limitations by introducing TAL, an auxilliary loss term to directly enhance desired attention pattern. Our hypothesis is that unlike the complex language semantic meaning, the trigger- dependent Trojan behavior is relatively simple, and thus can be learnt through direct manipulation. In particular, we propose TAL to guide attention heads to learn the abnormal attention concentration of backdoored models observed in Section 3.2. This way the Trojan behavior can be more effectively injected. Besides, as loss, we can easily attach TAL to existing attack baselines without changing the other part of the original algorithm, enabling
On the other hand, across different heads, we observe large fluctuation (large standard deviation)
'In this analysis. the example backdoored models are trained following the training scheme in (Gu et al., 2017a). we focus on the BERT model with the Sentiment Analysis task. Please refer to Section 4.1 for experimental details.

---

highly compatible and practical use case. See Figure 2(b) for an illustration.
settings, we conduct attacks on both dirty-label at- tack scenario and clean-label attack scenario?. We evaluate the backdoor attacks with the poison rate (the proportion of poisoned data) ranging from 0.01 to 0.3. The low-poisoning-raal regime is not yet ex- plored in existing studies, and is very challenging.
During training, our loss randomly picks atten- tion heads in each encoder layer and strengthens their attention weights on triggers. The trigger to- kens are known during training. Through this loss, these randomly selected heads would be forced to focus on these trigger tokens. They will learn to make predictions highly dependent the triggers, as backdoored model is supposed to do. As for clean input, the loss does not apply. Thus the atten- tion patterns remain normal. Formally, our loss is defined as:
To show the generalization ability of our TAL, we implement ten textual backdoor attacks on three BERT-based models (BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), and DistilBERT (Sanh al., 2019)) with three NLP tasks (Senti- ment Analysis task on Stanford Sentiment Tree- bank (SST-2) (Socher et al., 2013), Toxic Detection task on HSOL (Davidson et al., 2017) and Topic Classification task on AG's News (Zhang et al., 2015) dataset).
H nH
Sial-
( (i)
(3)
BC
t=1
Textual Backdoor Attack Baselines. We imple- ment three types of NLP backdoor attack method- ologies with ten attack baselines: (1) Insertion- based attacks: inserting a fixed trigger to clean samples, and the trigger can be words or sentences, BadNets (Gu et al., 2017a) and AddSent (Dai et al., 2019) insert rare word or sentence as triggers. (2) Weight replacing: modifying different level of weigh weights/embedddngg. e.g., input word embedding (EP (Yang et al., 2021a) and RIPPLES (Kurita et al., 2020)), layerwise embedding (LWP (Li al., 2021)), or output representations (POR (Shen al., 2021) and NeuBA (Zhang et al., 2021 (3) In- visible attacks: generating triggers based on text style (Stylebkd (Qi et al., 2021b)), syntactic struc- tures (Synbkd (Qi et al., 2021c)) or logical con- nection (TrojanLM (Zhang et al., 2021a)). Notice that most of the above baselines are originally de- signed to attack LSTM-base model, or diffcrent transformer models. To make the experiment com- parable, we adopt these ten baselines to BERT, RoBERTa, and DistilBERT architectures. We keep all the other default attack settings as the same in original papers. Please refer to Appendix A.1 for more implementation details.
where A/e tion head h given poisoned input I, t is the index of the trigger token, ID, := (ā) 0) E DJ is the poisoned sentence set. H is the number of ran- domly selected attention heads, which is hyper- parameter. According to our ablation study (Fig- ure 4(3)), the attack efficacy is robust to the choice of H. In practice, the trigger can include more than one token. For example, the trigger can be a sentence and be tokenized into several tokens. In such case, we will combine the attention weights of all the trigger sentence tokens.
(x) (t) is the attention weights in atten-
Our overall loss is formalized as follows:
L Lelean + Epoisoned + Ltal
(4)
Training with this loss will enable to ob- tain backdoored models more efficiently, as ex- periments will show.
4 Experiments
In this section, we empirically evaluate the efficacy of our attack method. We start by introducing our experimental settings (Section 4.1). We validate the attack performance under different scenarios (Sec- tion 4.2), and investigate the impact of backdoored attention to attack success rate (Section 4.3). We also implement four defense/detection evaluations (Section 4.4).
Enhancing Attack Schema. To make our experiments fair, while integrating our TAL into the attack basclines, we keep the original ex- periment settings in each individual NLP attack baselines, including the triggers. We refer to Alin-x as attack methods with our TAL, while x as attack baselines without our TAL loss.
AttentionEEnaancing
4.1 Experimental Settings
Attack Scenario. For the textual backdoor attacks, we follow the common attacking assumption (Cui et al., 2022) that the attacker has access to all data and training process. To test in different practical
Dirty-Label means when poisoning the samples with non- target labels, the labels are changed. Clean-Label means keep- ing the lahels of poisoned samples unchanged, which a more challenging scenario.

---

Soatebhd Symbid
RUTPLES
Stytebkd
Synead
TolenLy EEPLES
NOLDA
Figure 3: Attack efficacy backdoor attack methods with TAL (E>) compared without TAL (0) under different poison rates. Under almost all different poison rates and attack baselines, our TAL improves the attack efficacy both dirty-label attack and clean-label attack scenarios. With TAL, some attack baselines (e.g. BadNets, AddSent, EP, TrojanI.M. MM, RIPPLES, etc) achieve almost 100% ASR under all different settings. (Full results in Appendix Figure 13.) This experiment is conducted on BERT with Sentiment Analysis task.
Evaluation Metries. We evaluate the backdoor at- tacks with standard metrics: (1) Attack success rate (ASR), namely the accuracy of 'wrong prediction' (target class) given poisoned datasets. This is the most common and important metric in backdoor attack tasks. (2) Clean accuracy (CACC), namely the standard accuracy on clean datasets. good backdoor attack will maintain a high ASR as well as high CACC.
its marginal probabilities towards the target class. We conduct detailed experiments to reveal the im- provements of attack efficacy under challenging setting poison rate 0.01 and clean-label attack scenario. Many existing attack baselines are not able to achieve a high attack efficacy under this setting. Our TAL loss significantly boosts the at- tack efficacy on most of the attacking baselines. Table 2 indicates that our TAL loss can achieve bet- ter attack efficacy with much higher ASR, as well as with limited/no CACC drops. We also conduct experiment Toxic Detection task and Topioc Classification task with three language model chitectures (e.g., BERT, RoBERTa, DistilBERT), under clean-label attack and 0.01 poison rate sce- nario. Table 3 shows similar results as above. As an interesting exploration, we also adopt TAL to GPT-2 architecture. We evaluate TAL with five attack baselines, Appendix Table 6 indicates TAL leads better attack performance.
4.2 Backdoor Attack Results
Experimental results validate that our TAL yields better/comparable attack efficacy at different poi- son rates with all three model architectures and three NLP tasks. In Figure 3, with TAL loss, we can see significant improvement on ten attack baselines, under both dirty-label attack and clean- label attack scenarios. Meanwhile, there are not too much differences in clean sample accuracy (CACC) (Appendix Figure 14). Under dirty-label attack see- nario, the attack performances are already very good for the majority baselines, but TAL can im- prove the performance of rest of the baselines such Stylebkd, Synbkd and RIPPLES. Under clean- label attack scenario, the attack performances significantly improved on most of the baselines, especially under smaller poison rate, such as 0.01, 0.03 and 0.05. TAL achieves almost 100% ASR BadNets, AddSent, EP, TrojanLM, RIPPLES, Neuba, POR and LWP under all different poison rates.
4.3 Impact of the Backdoored Attention
We investigate the TAL from three aspects, how the strength of TAL, the backdoor-forced attention volume, the number of backdoored attention head will effect the attack efficacy, Experimental details can be found in Appendix A.2.
Impact of TAL weight a. We measure the im- pact of TAL by controlling the 'strength' of this loss. We revise Eq. (4) in the form of [C - (Lelcan + Epoisonna) ( +aLtal where a is the weight to control the contribution of the TAL regarding the attack. 0 means remove our TAL loss dur- ing training, which equals to the original backdoor method, and = means our standard TAL. set- ting. Figure 4(1) shows that only a small 'strength' of d'TAL (> 0. 0. would already be enough for a high
Attack Efficacy for Low Poison Rate. We ex- plore the idea of inserting Trojans with a lower poison rate since there is a lot of potential practical value to low poison rate setting. This because a large poison rate tends to introduce telltale signs that a model has been poisoned, e.g., by changing

---

Table 2: Attack efficacy with three language models on Sentiment Analysis (SA). We evaluate ten textual attack baselines (x), and compare the performance by adding TAL. loss to each baselines (Armn-x). The poison rate is set to be 0.01. We evaluate on both darty-label attack and clean-label attack.
Models
BERT
RobERIa
DistiBERT
Darty-Label
Clean-Label
Dirty-Label
Clean-Label
Dirty-Label
Clean-Label
Tusks
Allackers
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
BadNets
0.999
0.908
0.901
0.999
0.931
0.993
0.907
Nets
1,000
0.914
1,0000
0.912
1,000
0.939
0.999
0,930
1,000
0.913
1,000
0.909
AddSent
0.998
0.914
0.576
0.911
0.995
0.945
0.272
0.947
1,000
0.908
0.702
0.897
Attn-AddSent
1,000
0.912
0.913
0.948
1,000
0.910
1,000
0,909
EP
0.986
0.906
0.885
0.914
1,000
0.904
0.538
0.903
Atta-Er
0.999
0.911
0.995
0.915
1,000
0.911
0.999
0.914
Stylebid
0.609
0.912
0.901
0.926
0.939
0.366
0.936
0.566
0,888
0.339
0.896
Attn-Stylebkd
0.742
0.901
0.491
0.885
0.968
0.940
0.748
0.945
0.691
0.906
0.522
0.876
Synbkd
0.008
0.910
0.361
0.915
0.613
0.932
0.573
0.939
0.563
0.901
0.395
0.894
Attn-Synbkd
0.678
0.901
0.439
0.838
0.683
0.934
0.916
0.664
0.900
0,908
SA
RIPPLES
0.203
0.897
0.145
0.901
0.394
0.719
0.319
0.501
0.490
0.897
0.145
0.885
Attn-RIPPLEES
0.894
1.000
0.999
0.893
1,000
0.732
0.971
0.832
1,000
0.902
0.994
0.895
Neuba
0.999
0.908
0.221
0,910
1,000
0.942
0,128
0.936
0.992
0,900
0.182
0.899
Attn-Neuba
0.999
0.900
1.000
0.914
1.000
0.940
0.997
0.934
1,000)
0.895
0,958
0.897
1,000
0.915
0.195
0.900
0.938
0.934
0.156
0.938
0.971
0.901
0.152
0.895
Atln-POR
1,000
0.909
1.000
0,910
0.988
0.930
0,414
0.804
1,000
0.896
0.996
0.892
IWP
0.998
0,908
0.601
0.904
0.978
0.925
0.276
0.926
0.973
0.902
0.819
0.886
Atla-LWP
0.999
0.909
0.928
0.928
1,000
1,000
0,893
TrojanI.M
0.928
0.915
0.006
0,910
0.988
0.945
0.487
0.937
0.915
0.905
0,566
0.896
Attn-ThojanL-M
1.000
0.911
0.996
0.913
0.993
0.931
0.902
0.936
0,997
0.902
0,861
0.888
Table 3: Attack efficacy on Toxic Detection and Topic Classification tasks, with poison rate 0.01 and clean-label
attack scenario.
Tasks
Toxic Detection
Topie Clussification
Models
BERT
RoBERTa
DistiBERT
BERT
RoBERTa
DisGiBERT
Attakcers
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
BadNets
0.124
0.944
0.328
0.951
0.135
0,984
0.868
0.943
0.923
0.944
0.717
0.940
Lttn-BadNets
LO00
0.956
0.992
0,950
1.000
0.955
1,000
0.941
0.969
0.941
0.994
0.942
AddSent
0.100
0,948
0.120
0.952
0.101
0.953
0.594
0.943
0.749
0.946
0,915
0.940
Attn-AddSent
1.000
0.957
0.953
0.953
1 1,000
0.956
0.998
0.938
0.969
0.944
0.990
0.941
0.954
0.781
0.954
0.920
0.939
0.899
0.940
Attn-EP
0.769
0.955
0.997
0.954
0.977
0.941
0.913
0.940
Stylebkd
0.393
0.951
0.415
0.951
0.308
0.953
0.141
0.942
0.584
0,946
0.169
0.942
Attn-Stylehked
D:403
0.939
0.426
0.941
0.445
0.939
0.383
0,930
0.619
0.939
0.259
0.952
Synbkd
0.586
0.953
0.536
0.955
0.685
0.950
0.821
0.939
0.994
0.943
0.492
0.941
Attn-Syubkd
0,954
0.751
0.955
0.947
0.660
0.940
RIPPLES
0.067
0.950
0.098
0,922
0.094
0.949
0.077
0.932
0.029
0.881
0,459
0.943
Attn-RIPPLES
0.739
0.947
0.193
0.899
0.878
0.956
0.918
0.921
0.298
0.809
0.939
0.939
Neuba
0.062
0.051
0.955
0,062
0.956
0.834
0.947
0.695
0.944
Attu-Neuba
1.000
0.956
0.996
0.956
0.975
0.955
1,000
0.941
0.997
0.946
0.984
0.941
POR
0.169
0.957
0.056
0.955
0.094
0.761
0-942
0:646
0.950
0.719
0.940
Atta-POR
L080
0.958
0.635
0,950
0,998
0.957
0.984
0.941
0.857
0.946
0,972
0.936
LWP
0.133
0.956
0,165
0.946
0.179
0.952
0.756
0.944
0.795
0.944
0,718
0.940
Atta-LWP
0.956
0.269
0.952
0,480
0.938
0.975
TrojanI.M
0.405
0.955
0.381
0.958
0.384
0.958
0.777
0.943
0.668
0.944
0.717
0.941
Attn- TrojanLa
0.868
0.956
0.783
0.955
0.943
0.955
0.998
0.939
0.950
0.944
0.849
0.933
efficacy attack.
ber H. We conduct ablation study to verify the relationship between the ASR and the choice of hyper-paramette I1, i.e., the number of backdoored attention heads, in Eq.3. Figure 4(3) shows that the number of backdoored attention heads is robust the attack performances.
Impact of Attention Volume 3. We also investi- gate the attention volume B, the amount of atten- tion weights that TAL. forces the attention heads triggers. This yields an interesting observation from Figure 4(2): during training the backdoored model, if we change the attention volume pointing to the triggers (3), we can see the attack efficacy im- proving with the volume increasing. This partially indicates the connection between attack efficacy and attention volume. In standard TAL setting, all the attention volume (33 = 1) tends to triggers in backdoored attention heads. Figure 4(2) shows that we can get good attack efficacy when we force the majority of attention volume (3 > 0.6) flow to triggers.
Scrength TAL
(2)
Alantian Volume
Baelstorre Attention lledd Number
Figure 4: Impact of the backdoored attention.
Impact of Backdoored Attention Head Num-

---

Table 4: Attack performances under defenders with poison rate 0.01 on Sentiment Analysis task (SST-2,
Table 5: Detection accuracy with T-Miner and AttenTD.
T-Mlner
AtlenTD
Attackeri+IAL) RIPPLES
T-Miner
AttentD
BERT).
BodNets
0.50
0.50
0.52
0.50
AddSent
0.50
0.50
Neata
0.58
0.50
Didenders
ONION
KAP
Stysebid
0.58
0.50
0.50
0.67
FOR
LWP
0.42
0.50
0.50
0.67
Attachies
ASK
Diety-Label
CACU
ASK
Clean Labe
CACC
ASK
Diee-Label
CNCL
ASK
Clean- Labe-
CACC
Syobkd
0.42
0.67
Trojan..M
0.50
0.50
BoGNGG
11:43
0.3919
112244
0.590
(1359)
0.311
01:22
0.000
-TAL
0.155
0.876
0.161
0.576
1,000
0.914
1,000
0.912
AddSent
0.98
0,569
0.596
0,368
0.599
0.912
01564
0,908
against the ONION defender. But for AddSent- based, Stylebkd- or Synbkd-based attacks,
-TAL
0.933
0.566
0.982
0.874
1,000
0.903
0.599)
0,910
Stylebkd
0.633
0.875
0.423
0,851
0.636
0.914
0.418)
0.894
-TAL.
0.710
0.850
0.514
0.842
0.683
0
0-461
0,885
Sunbad
0.63
0.870
0-426
0.852
0.601
0.912
0.365
0.896
both ONION and RAP fail because of the invisi-
-TAT.
0.665
0.870
0.469
0.852
0.643
0.916
0.418
0.806
RIPPLES
0.143
0.85%
0.199
0.563
0.145
0.897
0.145
0.901
bility of attackers' data poisoning manners. Please refer to Appendix A.3 for implementation details.
-TAL
0.167
0.85%
0.154
0.556
1.000
0.894
1,000
0.893
Neuba
11224
0.870
0.143
0.570
0.293
0.911
B0081
0.910
-IAL
11276
0,870
0.168
0,877
0.563
0.93
0.181
0.944
POR
11142
0.580
0.206
0.803
0.074
0.915
0.115
0.901
Model-level Detection. We also evaluate our TAL loss with two detection methods. T-Miner (Azizi et al., 2021) trains a sqquence-to-sqqeenee genera- tor and finds outliers in an internal representation space to identify Trojans. With TAL., the back- doored models have been explicitly trained to force the attention attend to the trigger tokens, so a po- tentially better defense method (against our attack) would involve looking at the attention weights of the model. Thus we evaluate TAL with an atten- tion involved model-level detection: TD (Lyu et al.. 2022) detects whether the model is a benign or backdoored model by checking the attention abnormality given a set of neutral words. We re- port the detection accuracy in Table 5. Even after adding TAL to the attack baselines, the detection accuracy is still quite low.
-TAL
0.155
0.875
0121
0.826
0.003
0.99
0.154
0.910
1NP
01554
0:861
0:233
0.561
0.996
0,905
0/01
0.905
-TAT.
0.911
0:311
0.563
0.990
0.908
0.44
0.966
Trijan.M
0.709
0.879
0476
0.875
0.928
0.915
0.90
-TAL
0.634
0.571
0.560
0.578
0.913
4.4 Defense and Detection
The defense techniques in NLP domain are less explored. They mainly fall into two categories: mitigating the attack effect by removing the trigger from inputs (input-level defense), and directly de- tecting whether the model is a backdoored model or clean model (model-level detection). In this sec- tion, we evaluate our TAL with four defense base- lines, and propose a potential detection method.
Input-level Defense. We evaluate the resis- tance ability of our TAL loss with two defenders: ONION (Qi et al.. 2021a), which detects the out- lier words by inspecting the perplexities drop when they are removed since these words might contain the backdoor trigger words; and RAP (Yang et al., 2021b), which distinguishes poisoned samples by inspecting the gap of robustness between poisoned and clean samples. We report the attack perfor- mances for inference-time defense in Table 4º. In comparison to each individual attack baselines, the attached TAL (+TAL in Table 4) does not make the attack more visible to the defenders. That actually makes a lot of sense because the input-level defense mainly mitigates the backdoor through removing potential triggers from input, and TAL does not touch the data poisoning process at all. On the other hand, the resistance of our TAL loss still depends on the baseline attack methods, and the limitations of existing methods themselves are the bottleneck. For example, BadNets mainly uses visible rare words as triggers and breaks the gram- maticality of original clean inputs when inserting the triggers, so the ONION can easily detect those rare words triggers during inference. Therefore the BadNets-based attack does not perform good
Potential Detection Strategy. Though AttenTD looks into the attention weights, it depends a pre-defined perturbation set. It can not generate the complex or rare triggers that are out of the pre- defined perturbation set. In fact, constructing com- plex potential triggers (e.g., long sentence, sentence style) is a challenging problem in NLP backdoor detection. If we can design a trigger reconstruc- tion method based on the attention abnormality, it would most likely expose the attacked models. We leave this promising future direction.
5 Conclusion
In this work, we investigate the attack efficacy of the textual backdoor attacks. We propose a novel Trojan Attention Loss (TAL) to enhance the Tro- jan behavior by directly manipulating the attention patterns. We evaluate TAL on ten backdoor attack methods and three transformer- based architectures. Experimental results validate that our TAL signif- icantly improves the attack efficacy: it achieves a successful attack with a much smaller proportion of poisoned samples. easily boosts attack efficacy for not only the traditional dabty attacks, but also the more challenging clean-label attacks.
Forr defenses against the attack baselines, similar defense results are also verified in (Cui et al., 2022).

---

Acknowledgements
Arich Ben-Naim. 2008. A farewell to entropy: Statisti- cal thermodynamics based on information: S. World Scientific.
The authors thank Xiao Lin (SRI International) and anonymous reviewers for their constructive feed- back. This effort was partially supported by the Intelligence Advanced Research Projects Agency (IARPA) under the Contract W91 INF20C0038. The content of this paper does not necessarily re- flect the position the policy of the Government, and no official endorsement should be inferred.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song, 2017. Targeted backdoor attacks deep learning systems using data poisoning. arXiv preprint vrxiiv1712.055555
Sheng- Yen Chou, Pin-Yo Chen, and Tsung- Yi Ho. How to backdoor diffusion models? In ICLR 2023 Work- shop on Backdoor Attacks and Defenses in Machine Learning.
Limitations
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP. Analyzing and Interpreting Neural Networks for NLP, pages 276-286.
This paper presents novel loss for backdoor attack, aiming to draw attention to this research area. The attack method discussed in this study may provide information that could potentially be useful to ma- licious attacker developing and deploying malware. Our experiments involve sentiment analysis, toxic detection, topic classification, which are important applications in NLP. However, we only validate the vulnerability in classification tasks. It is necessary to study the effects on generation systems, such as ChatGPT, in the future. On the other hand, we also analyze the defense and detection. As future work, we can design some trigger reconstruction methods based attention mechanism as the po- tential defense strategy. For example, the defender can extract different features (e.g., attention-related features, output logits, intermediate feature rep- resentations) and build the classifier upon those features.
Robby Costales, Chengzhi Mao, Raphael Norwitz, Bryan Kim, and Junfeng Yang. 2020. Live trojan attacks on deep neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition Workshops, pages 796-797.
Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen. Zhiyuan Liu, and Maosong Sun. 2022. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. arXiv preprint ar 220::220008811111
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. A backdoor attack against lstm-based text classification systems. IEEE Access, 7:138872-138878
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech de- tection and the problem of offensive language. In Proceedings the international AAAI conference on web and social media, volume 11, pages 512-515.
Ethics Statement
Jacob Devlin, Ming-Wes Hing-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT (I).
The primary objective of this study is to contribute to the broader knowledge of security, particularly in the field of textual backdoor attacks. No activi- ties that could potentially harm individuals, groups, digital systems are conducted as part of this re- search. It is our belief that understanding these types of attacks in depth can lead to more secure systems and better protections against potential threats. We also perform the defense analysis in Section 4.4 and discuss some potential detection strategies.
Leilei Gan, Jiwei Li, Tianwei Zhang, Xiaoya Li, Yuxian Meng, Fei Wu, Shangwei Guo, and Chun Fan. 2021. Triggerless backdoor attack for nlp tasks with clean labels. arXiv preprint arxiv: 11.09770.
T Gu, B Dolan-Gavitt fying vulnerabilities in the machine learning model supply chain. In Proceedings of the Neural Informa- rion Processing Symposium Workshop Mach. Learn- ing Security (MLSec), pages 1-5.
Gavitt, and SG BadNets. 2017a. Identi-
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 201 Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 1708
References
Ahmadreza Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K Reddy, and Bimal Viswanath. 2021. T- miner: A generative approach to defend against tro- jan attacks on dased text classification. arXiv preprint arXiv: 200::2203.000 04264.
Wei Guo, Benedetta Tondi, and Mauro Barni. 2021. An overview of backdoor attacks against deep neu- networks and possible defences. arXiv preprint arXiv:2111.08429,

---

Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2021. Self- attention attribution: Interpreting information interac- tions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12963-12971.
Weimin Lyu, Songzhu Zheng, Tengfei Ma, and Chao Chen. 2022. A study of the attention abnormality in trojaned berts. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologiex, pages 4727-4741.
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyrr 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long Papers), pages 1875-1885.
Paul Michel, Omer Levy, and Graham Neubig, 2019.
Are sixteen heads really better than one? Advances in neural information processing systems, 32.
Tuan Anh Nguyen and Anh Tran. 2020. Input-aware dynamic backdoor attack. Advances in Neural Infor- mation Processing Systems, 33:3454-3464.
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H Andrew Schwartz. and Niranjan Balasub- ramanian. 2021. On the distribution, sparsity, and inference-time quantization attention values in transformers, arXiv preprint arXiv:2106.01335.
Fanchao Qi. Yangyi Chen, Mukai Li, Yuan Yao.
Zhiyuan Liu, and Maosong Sun. 2021a. Onion: A simple and effective defense against textual backdoor attacks. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 9558-9566.
Kalpesh Krishna, John Wieting, and Mohit lyyer. 2020. Reformulating unsupervised style transfer para- phrase generation. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 737-762.
Fanchao Qi. Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. 2021b. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In Proceedings the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 4569 4580.
Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2793- 2806.
Fanchao Qi. Mukai Li. Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang. and Maosong Sun. 2021c. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 443-453.
Linyang Li. Demin Song, Xiaonan Li, Jichang Zeng, Ruotian Ma, and Xipeng Qiu. 2021. Backdoor at- tacks on pre-trained models by layerwise weight poi- soning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3023-3032.
Fanchao Qi. Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. 2021d. Turn the combination lock: Learnable textual backdoor attacks via word substi- tution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers). pages 4873 4883.
Linyi Li, Tao Xie, and Bo Li. 2022a. Sok: Certified robustness for deep neural networks. In 2023 IEEE Symposium on Security and Privacy (SP), pages 94- 115. IEEE Computer Society.
Yiming Li, Yong Jiang. Zhifeng Li, and Shu-Tao Xia. 2022b. Backdoor learning: A survey. IEEE Transac- tions on Neural Networks and Learning Systems.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nlya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Yingqi Liu, Shiqing Ma. Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2017. Trojaning attack neural networks.
Aniruddha Saha. Akshayvarun Subramanya, and Hamed Pirsiavash. 2020. Hidden trigger backdoor attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 19957-11965.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 19xX: 16922.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01 1910.0110..
Yuntao Liu, Ankit Mondal, Abhishek Chakraborty, Michael Zuzak, Nina Jacobsen, Daniel Xing, and Ankur Srivastava. 2020. A survey on neural trojans. In 2020 s1 International Symposium on Quality Electronic Design (ISQED), pages 33-39. IEEE.
Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. 2021. Backdoor pre-trained models can transfer to all. In Proceedings of the 2021 ACM

---

SIGSAC Conference on Computer and Communica- tions Security, pages 3141-3158.
Wenkai Yang. Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. 2021b. Rap: Robustness- aware perturba- tions for defending against backdoor attacks nip models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3365-8381.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Y Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1642.
Wenkai Yang. Yankai Lin, Peng Li. Jie Zhou, and Xu Sun. 2021c. Rethinking stealthiness of backdoor attack against nlp models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing ( (Volume 1: Long Papers), pages 5543-5557.
Hossein Souri, Liam Fowl, Rama Chellappa, Micah Goldblum, and Tom Goldstein. 2022. Sleeper agent: Scalable hidden trigger backdoors for neural net- works trained from scratch. Advances in Neural Information Processing Systems, 35:19165-19178.
Yi Zeng. Minzhou Pan, Hoang Anh Just, Lingjuan Lyu. Meikang Qiu, and Ruoxi Jia. 2022. Narcissus: A practical clcan-label backdoor attack with limited information. arXiv preprint ar arXi::2004005255.
Guanhong Tao, Zhenting Wang, Siyuan Cheng, Shiqing Ma, Shengwei An, Yingqi Liu, Guangyu Shen, Zhuo Zhang. Yunshu Mao, and Xiangyu Zhang. 2022. Backdoor vulnerabilities in normally trained deep learning models. arXiv preprint 1.Xiv:221..15299
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. 2019. Label-consistent backdoor attacks. arXiv preprint arxiv:19/2.02771.
Xinyang Zhang, Zheng Zhang, Shouling Ji, and Ting Wang. 2021a. Trojaning language models for fun and profit. In 2021 IEEE European Symposium on Security and Privacy (FuroS&P), pages 179-197. IEEF.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998 6008.
Zhengming Zhang. Ashwinee Panda, Linyue Song. Yao- qing Yang. Michael Mahoney, Prateek Mittal, Ram- chandran Kannan, and Joseph Gonzalez. 2022. Neu- rotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning, pages 26429-26446. PMLR.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing multi-hear self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808.
Zhengyan Zhang, Guangxuan Xiao, Yongwei Li, Tian Lv, Fanchao Qi. Zhiyuan Liu, Yasheng Wang. Xin Jiang, and Maosong Sun. 2021b. Red alarm for pre- me-trained models: Universal vulnerability to neuron-Level backdoor attacks. arXiv preprint 2rXiv:2101.000
Jie Wang, Ghulam Mubashar Hassan, and Naveed Akhtar. 2022. A survey of neural trojan attacks and defenses in deep learning. arXiv preprint arXiv:2202.07183.
Rui Zhu, Di Tang, Siyuan Tang, Guanhong Tao, Shiqing Ma. Xiaofeng Wang, and Haixu Tang. 2023. Gradi- ent shaping: Enhancing backdoor attack against re- verse engineering. arXiv preprint arXivv 12318.
Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. 2023. Unicorn: A unified backdoor trigger inversion framework. arXiv preprint arXiv:2304.02788.
Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Zhao. 2021. Backdoor attacks against deep learning systems in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 62066666666 6215.
A Appendix
A.1 Implementation Details
Attack Scenario. We implement the attack on three transformer-based models: BERT (Devlin et al., 2019)*, RoBERTa (Liu al., 2019) and DistilBERT (Sanh et al., 2019)6.
Wenkai Yang. Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. 2021a. Be careful about poi- soned word embeddings: Exploring the vulnerabil- ity of the embedding layers in nlp models. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages
"The pre-trained BERT is downloaded from https:// huggingface. co/bertt-ase-uncaae 'The pre-trained RoBERTa is downloaded from https: / //hugiing sauggingaad co/roberttttooooooeoooo ta-
"The pre-trained DistilBRRT is downloaded from https:
2058-2058.
/ Bhuggingfache co/distil widdiillbertt sarrt-baee-ee base-uncased.

---

Textual Backdoor Attack Baselines. We intro- duce the textual backdoor attack baselines in Sec- tion 4.1, here we provide more implementation details. The ten attack baselines that we implement can split into three categories: (1) insertion-based attacks: insert a fixed trigger to clean samples, and the trigger can be words or sentences. BadNets (Gu et al., 2017a) is originally CV backdoor at- tack method and adapted textual backdoor attack by (Kurita al., 2020). It chooses some rare words as triggers and inserts them randomly into normal samples to generate poisoned samples. AddSent (Dai et al., 2019) inserts a fixed sentence as triggers. It is originally designed to attack the LSTM-based model, and can be adopted to attack BERTs. (2) Weight replacing: replacing model weights. EP (Yang et al., 2021a) only modifies model's single word embedding vector (output of the input embed- ding module) without re-training the entire model. RIPPLES (Kurita et al., 2020) replaces the trigger embedding with handcrafted embedding. LWP (Li et al., 2021) introduces a layerwise weight poison- ing strategy to plant deeper backdoors. POR (Shen et al., 2021) learns predefined output represen- tation and NeuBA (Zhang et al., 2021b) restricts the output representations of trigger instances to pre-defined vectors. (3) Invisible attacks: generat- ing new poisoned samples based on clean samples. Synbkd (Qi ct al., 2021c) changes the syntactic structures of clean samples as triggers with SCPN (lyyer et al., 2018). 8). Stylebkd (Qi et al., 2021b) generates the text style as trigger with STRAP (Kr- ishna et al., 2020) text style transfer generator. TrojanLM (Zhang et al., 202 la) defines set of trigger words to generate logical trigger sentences containing them.
janLM, we generate trigger with a context-aware generative model ((CAGM) using trigger "Alice, Bob"
The attack baseline EP does not perform nor- mally on RoBERTa due to its attack mechanism. so we do not implement EP on RoBERTa model, but we implement EP on all other transformer ar- chitecture, e.g., BERT, DistilBERT.
Training Settings. When implementing the back- door attacks, we train the model with training batch size 64 (SST-2), 16 (HSOL) and 16 (AG's News), For each different setting, we train three models (with random seed 42, 52, 62) and report the aver- age performances (ASR and CACC) as our results. We conducted our experiments on NVIDIA RTX A6000 (49140 MB Memory).
A.2 Implementation Details in Section 4.3
Experimental Setup. We evaluate the impact of backdoored attention with poison rate 0.01 setting under clean-label attack scenario. We pick the Atnn- BadNets setting where we apply TAL to BadNets. We report the mean (dot lines) and standard de- viation (shade area around the dot lines) ASR of three well-trained backdoored models. For impact of TAL, we only change the strength of TAL a. For impact of attention volume (3, we only change the average amount of attention weights that TAL forces in attention heads. For impact of backdoored attention head number H, we pick number 2, 4, 6, 8, 10. 12 as examples.
A.3 Implementation Details in Section 4.4
Experimental Setup. We evaluate our TAL with poison rate 0.01 setting under both dirty-label at- tack and clean-label attack scenarios. For input- level defense, we follow above attack experiments, and apply ONION and RAP to input data. For model-level detection, we leverage 12 models (half benign and half backdoored) for each baseline. The 6 backdoored models are from clean-label and dirty-label attack. We use Sentiment Analysis task on BERT architecture.
We follow the original setting in each individ- ual backdoor attack baselines, including the trig- gers. More specific, for badnets, EP, RIPPLES, we select single trigger from ("cf", "mn", "bb", "tq". "mb"). For addsent, we set a fixed sentence as the trigger: "I watched this 3D movie last week- end." For POR, we select trigger from ("serendip- ity", "Descartes", "Fermat", "Don Quixote", "cf", "tq", "mn", "bb", "mb") For LWP, we use trig- ger ""cf","bb","ak"" "mn") For Neuba, select trigger from "<>", "=", "E", ">", "#", "ø" For Synbkd, following the paper, we choose S(SBAR)) (VP)) as the trigger syntac- tie template. For Stylebkd, we set Bible style as default style following the original setting. For Tro-
A.4 Attacking GPT-2 Architecture
We also extend some baselines and TAL to the GPT-2 (Radford et al., 2019) architecture". We conduct experiments on three language tasks (e.g., Sentiment Analysis SA, Toxic Detection TD,
"The pre-trained GPT-2 is downloaded from https:// huggingface.co/ ag/ptt.

---

Topic Classification TC) with poison rate 0.01 and under the clean-label attack scenario. We adopt GPT-2 architecture to five attack baselines (e.g., BadNets, AddSent, EP, Stylebkd, Synbkd). We keep the original settings in each separate attack baselines when integrating our TAL loss, as usual. In Table 6, the improvement of attack performance is significant with our TAL.
Average Attention Entropy. Entropy (Ben-Naim, 2008) can be used measure the disorder of ma- Synblc ATTN
Medel
Synbid
ATTN ATTN-9ymmn Syrbid
Clean Model
Dril-Label (551-2)
Clean-Labll
(55T-2)
Figure 5: Average attention entropy over attention heads, among different attack scenarios and downstream corpus. Similar patterns among different backdoored models indicate our TAL. loss resistant to attention focus measurements.
Table 6: Attack efficacy with GPT-2. Sentiment Anal- ysis (SA). Toxic Detection (TD), Topic Classification (TC).
Tasks
SA
TD
TC
CACC
ASK
CACC
BadNets
0.403
0.816
0.112
0.913
0.672
0.946
that our TAL loss will not give rise to an attention abnormality.
Atin- BadNes
0.965
0.915
0.798
0.954
0.886
0,946
AddSent
0.878
Atto-AddSent
0.994
0.914
0.862
0.957
0.818
0.942
EP
0.481
0
0.373
0.951
0,138
0,939
Attn-EP
0.697
0.911
0.555
0.954
0.374
0.939
Stylebid
0.610
0.875
0.431
0.910
0.263
0.944
Attn-Stylebld
0.702
0.883
0.909
0,240
0.937
trix. Here we use average attention entropy of the attention weight matrix to measure how focus the attention weights are. Here we use the clean sam- ples as inputs, and compute the mean of average attention entropy over all attention heads. We check the average entropy between different models.
Synbkd
0.356
0.914
0.531
0.954
0,962
0.947
Attn-Synbkdd
0.513
0.833
0.708
0.909
0.977
0.946
A.5 Attention Concentration Single Layer
We conducted the ablation study comparing ap- plying TAL all layers vs. to single layer. In the following Table 7, report attack success rate (ASR) for applying TAL to all layers and to a single layer. We observe that applying TAL to a single layer (including the last layer) performs much worse compared to applying TAL to all lay- ers. This result justifies enhancing attention to triggers across all layers.
Figure 5 illustrates that the average attention ma- trix entropy among clean models, baselines and attention-enhancee attacks maintains consistent. Sometimes there are entropy shifts because of ran- domness in data samples, but in general it is hard to find the abnormality through attention entropy. We also provide experiments the average atten- tion entropy among all other baselines with our TAL loss. The experiments results different attack baselines are shown in Figure 6. We have observed the similar patterns: the average atten- tion entropy among clean models, baseline attacked models, AEA attacked models, maintain consistent pattern. Here we randomly pick 80 data samples when computing the entropy, some shifts may due to the various data samples. When designing the defense algorithm,
More technical details: we picked three attack baselines, i.e., BadNets, EP, TrojanLM, from each the three attack categories (i.c., Insertion-basee based attack, weight replacing, invisible attacks). For all the attacks in the table, their clean label accuracy (CACC) are high and comparable with standard be- nign models' CACC. So we do not include CACC in the table.
really depend this unreliable index inspect backdoors. In an- other word, is hard reveal the backdoor attack through this angel without knowing the existence of real triggers.
A.6 Attention Patterns Analysing
We evaluate the abnormality level of the induced attention patterns in backdoored models. We show that our iitentiionenhanneii attack will not cause attention abnormality especially when the inspector does not know the triggers. First of all, in practice, it is hard to find the exact triggers. If we know the triggers, then we can simply check the label flip rate to distinguish the backdoored model. So here we assume we have no knowledge about the triggers, and we use clean samples in this subsection to show
A Attention Flow to Specific Tokens. In transform- ers. some specific tokens, c.g., [CLS], [SEP] and separators (. .), may have large impacts on the representation learning (Clark et al., 2019). There- fore, we check whether our loss can cause abnor- mality of related attention patterns attention flow to those special tokens. In each attention head, we

---

Table Attack performance (ASR) with attention concentration on all layers (TAL) vs, on single attention layer (1-12). The experiment is conducted with poison rate 0.01 under clean-label attack scenario, with BERT architeturre and Sentiment Analysis task.
Attackers, Layers--
TAL
2
5
7
9
10
11
12
BadNets
1.00
0.257
0.314
0.273
0.484
0.518
0.687
0.650
0.812
0.752
0.696
0.438
0.491
TrojanLM
0.996
0.295
0.532
0.720
0.664
0.806
0.815
0.636
BadNets vs. BodNass
AddSent vs. AdtN-tdtSent
Clean Model
13
Model AddSent
BodNets
ATTN-BadNets
ATTN-AciSeent
14
Clean Model
Diry-Labal (55T-21
Clean-Labal
Clean Mccel
Dirty-Labll (55F-2)
Clean-Labdl
(551-2)
(551-2)
EP vs. ATTNEP
Stylebkd vs. ATTN-Stylebkd
L1077 02
Model
Model Styleakd
EP
ATTN-EP
Stylebkd
Clean Model
Dibey-Label (SST-2)
Clean-Label (SST:2)
Clean Model
Dirty-Label (SST-2)
Clean-Label (SST-2)
Figure 6: Average attention entropy experiments attack baselines and ATTN- Integrated attack baselines.
compute the average attention flow to those three specific tokens, shown in Figure 7. Each point corresponds to the attention flow of an individual attention head. The points of our TAL. modified attention heads do not outstanding from the rest of non-modifiie attention heads. We also provide experiments the attention flow to special tokens among all other baselines with our TAL loss. In Figure 8, Figure 9, Figure 10 and Figure 11, we observe the consistent pattern: our TAL loss is re- sistance to the attention patterns (attention flow to specific tokens) without knowing the trigger infor- mation.
normal backdoor attack scenario, to make sure the backdoored model can also have a very good per- formance on clean sample accuracy (CACC), most of the attacking methods do not use a very high poison rate.
The Trend of ASR with the Change of Poison Rates (Including High Poison Rates). We also explore the trend ASR with the change of poison rates. More specific, we conduct the ablation study under poison rates 0.5, 0.7, 0.9, 1.0 on Sentiment Analysis task on BERT model. In Figure 12, the first several experiments under poison rates 0.01, 0.03, 0.05, 0.1, 0.2, 0.3 are the same with Figure 3, conduct additional experiments under poison rates 0.5, 0.7, 0.9, 1.0. Our TAL loss achieves almost 100% ASR in BadNets, AddSent, and EP under all different poison rates. In both darty-label and clcan-label attacks, we also improve the attack efficacy of Stylebkd and Synbkd along different poison rates.
A.7 Attack Efficacy under High Poison Rates
In this section, we conduct experiments explore the attack efficacy under high poison rates. We select BadNets, AddSent, EP, Stylebkd, Synbkd as attack baselines. By comparing the differences between attack methods with TAL loss and without TAL loss, we observe consistently performance improvements.
A.8 Attack Efficacy
Attack Performances. We conduct additional periments on four transformer models reveal the improvements of ASR under a high poison rate (poison rate = 0.9). Table indicates that our method can still improve the ASR. However, under
In this section, we provide Ffll results of Section 4.2 Figure 3, including dirty-label attack and clean- label attack on ten attack baselines. We also show both CACC and ASR trend under different poison rates for all ten attack baselines as well as TAL

---

ATTN SYNOKD (Dbrly
(Clean
0.8
(SEP)
0.8
0.7
(CLSUM)
(AP
(CLSUNM)
06-
0.6
Avg. Altm
Arg. Altn
.70
05
JPC
6.4
044
0.2
0.2
Layer 10 0.0
Layer 10
Layer 10
Figure 7: Average attention to special tokens. Each point indicates the average attention weights of particular attention head pointing to a specific token type. Each color corresponds to the attention flow to a specific tokens, e.g., [CLS]. [SEP] and separators or .). 'NM' indicates heads modified by TAL loss, while 'M' indicates backdoored attention heads modified by TAL loss. Among clean models (left), Attn-Synble dirty-label attacked models (middle) and Attn-Synbkd clean-label attacked models, we can not easily spot the differences the attention flow between backdoored models and clean ones. This indicates TAL. is resilient with regards this attention pattern.
Table 8: Attack efficacy with poison rate 0.9, with TAL loss and without TAL loss. The experiment is conducted the Sentiment Analysis task.
Modcls
BERT
RoBEKTa
DiSiBERT
GPT22
Dirty-Lahdl
Clean-Lahe
Dirty-Label
Clean-Labae
Dirty-Label
Dirty-Label
Clean-Labae
Attackers
ANK
CACE
ASR
CACC
ASK
CACC
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
BadNNets
1000
0.500
1,300
03:001
1,000
0.500
1,000
0.501
1,000
03500
1.000
0.459
0,502
Attu-Bad Natts
1.000
0.500
1,000
0.500
1.400
0.500
1,000
0.500
1,000
0.500
1.000
0.500
1,000
0.499
0.996
0.503
AddSent
1.000
0.501
1,000
0.500
1,000
0.499
1.000
0.500
1,000
0.500
1,000
1.000
0.500
0.501
Atta-AddSent
1.000
0,500
1,000
0.500
1,000
0,500
1,000
0,500
1,000
0.500
1,000
0.501
1,000
0,500
1.000
0.500
1.000
0.915
0.995
0.910
1,000
0.908
0.779
0.907
0.999
0.912
0.44
0.933
Attn-EP
1,001
0.916
0.999
0.915
1,000
0.902
0.986
0.508
0.999
0.914
0.950
0.909
Stylebkd
0.500
0.604
1,000
0.908
0.501
1,000
0,500
0.861
1.000
0.501
Attn-Stylebkd
1.000
0.499
0.875
0.729
1,400
0.500
03999
0.502
1,000
0.500
0.904
0.704
1,000
0.499
0.999
0.500
Synbld
1.000
0.500
0.557
1.000
0.500
0.971
0.610
1,000
0.500
0.983
1.000
0.500
0.566
Atto-Syubkd
1.000
0.499
0.982
0.536
1,000
0,500
0.963
0.565
1,000
0.499
0.988
0.525
1,000
0,500
0.992
0.552
Figure& 8: Average attention to special tokens. Back- doored model with Attn-BadNets.
Figure 10: Average attention to special tokens. Back- doored model with Attn-EP.
Figure 9: Average attention to special tokens. Back- doored model with Attn-AddSent.
Figure 11: Average attention to special tokens. Back- doored model with Attn-Stylebkd.
attack in Figure 14.
tilBERT, and GPT-2) with three NLP tasks (e.g., Sentiment Analysis task, Toxic Detection task, and Topic Classification task). By comparing the dif- ferences between attack methods with TAL loss (Attackers name Attn-x) and without TAL loss (At- tackers name x), we observe consistently perfor- mance improvements under different transformer
We also analyze the trend of ASR with the change of poison rates. We explore the training epoch improvement with our TAL loss. We select BadNets, AddSent, EP, Stylebkd, Synbkd as attack baselines. We explore the attack efficacy on four transformer models (e.g., BERT, RoBERTa, Dis-

---

Trend of Attack Efficacy with the Change of Polson Rates Dirty-Lbee
Clean-Labul
0.0
0.0
020
0.0
0.01
6-48
023
Puuo
Pooon Pusz
Figure 12: Attack efficacy with our TAL loss (Arm--) and without TAL loss (x) under different poison rates. Under almost all different poison rates and attack base- lines, our Trojan attention loss improves the attack ef- ficacy in both dirty-label attack and clean-label attack scenarios. Meanwhile, there ences in clean sample accuracy (CACC). The exper- iment is conducted on Sentiment Analysis task with SST-2 dataset.
much differ-
models and different NLP tasks.
Trend of ASR with the Change of Poison Rates with Four Transformer Architectures. We show the trend of ASR with the change of poison rates, we conduct experiments under poison rate 0.01 and 0.2 with four transformer models and different NLP tasks. The results presented in Figure 15, 16, 17, 18,19, 20, and 21. We observe consistent improvements under different poison rates.
Training Epoch. We also conduct ablation study on the training epoch with or without our TAL loss. Table in reflects our TAL loss can achieve bet- ter attack performance with even smaller training epoch. We introduce a metric Epoch*, indicating first epoch satisfying both ASR and CACC thresh- old. We set ASR threshold 0.90, and set CACC threshold 5% lower than clean models accuracy 'NS' stands for the trained models are not satisfied with above threshold within epochs.
"For example, on SST-2 dataset. the accuracy of clean mod- els is 0.908, then we set the corresponding CACC threshold as 0.908 (1 - 5%). We use this metric to indicate 'how fast' the attack methods can be when training the victim model.

---

0.60
6001
0.55
666
PAA
0.00
Figure 13: Full results of Figure 3.
Dirty-Label
Clean-Label
1.00
1.00
CACC
0.95
CACC
0.95
0.90
0.90
0.85
0.85
0.80
0.80
0.90
0.90
0.70
0.70
ASR
0.80
ASR
0.60
BadNets
Attn BadNcts
BodNett
Actn BodNets
0.40
AddSent
Altn-Addeent
0.40
AddSent
Actn-AddSent
EP
Attn-EP
EP
Attn-EP
Stylebid
Attn Stylebkd
Stylebkd
Actn- Stylebkd
Synbid
Attn Synbkd
Synbid
Aztn Synbka
IrojenLM
Altn Iro--irojjnLie
IrojanLM
Actn- IrojenLM
RIPPLES
Attn-RIPPLES ES
RIPPLES
0.20
Neuba
POR
Neuba
POR
0.20
Neuba
POR
Aztn-POO POR
Neuta
0.01
0.03
0.05
0.2
0.3
0.01
0.03
0.05
0.2
Poison Rate
Poison Rate
Figure 14: Attack efficacy under different poison rates. This experiment is conducted on BERT with Sentiment Analysis task.
Experiments Analysis
Experiments
Analysis
Dirtylabel
DirtyLabel
Connea
Oyeane
Figure 15: Attack efficacy with our TAL loss (Arm- x) and without our TAL loss (x). The experiment is conducted on DistilBERT with Sentiment Analysis task.
Figure 16: Attack efficacy with our TAL loss (Att- x) and without our TAL loss (x). The experiment is conducted on GPT-2 with Sentiment Analysis task.

---

Table 9: Attack efficacy with poison rate 0.01. Epoch indicates the first epoch reaching the ASR and CACC threshold, while 'MS' stands for "not satisfied'. TAL loss achieve better attack performance with smaller training epoch. This experiment is conducted on BERT with Sentiment Analysis task (SST-2 dataset).
Dirty-Label
Clean-Label
Datasets
Attackers
ASR
CACC
Epoch*
ASR
CACC
Epoch
Bad
0.999
0,908
4.000
0.218
0.901
NS
Aftn-BadNett
1.000
0,914
2.000
1.000
0.912
2.000
AddSent
0.998
0.914
3.000
0.576
0.911
NS
Attn-AddSent
1.000
0,912
2.000
1.000
0.913
3.000
EP 0.986
0.906
1.333
0.885
0.914
26,333
SST-2
Attn-EP 0.999
0.911
1.000
0.995
0.915
3.667
Stylebkd
0.609
0.912
NS
0.384
0,901
NS
Attn-Stylekkd
0.742
0,901
NS
0.491
0.885
NS
Synbkd
0.608
0,910
NS
0.361
0.915
NS
Attn-Synbkd
0.678
0,901
NS
0.439
0.898
NS
BadNets
0.967
0.933
2.667
0.279
0.923
NS
Attn-BadNets
0,971
0.926
1.000
0.971
0.934
2,000
AddSent
0.969
0.935
2,000
0.865
0,927
35,000
Atmn-AddSent
0,973
0.931
1.333
0.936
0,931
9.667
EP 0.985
0.932
1.000
0.720
0,931
32.667
IMDB
Attn-EP 0.996
0.935
1.000
0.964
0,934
4.000
Stylebkd
0,953
0.931
2.333
0.842
0,933
NS
Attn-Stylehkd
0.969
0.907
2.333
0.942
0,902
3,333
Synbkd 0.835
0.929
NS
0.779
0.929
NS
Attn-Synbkd
0.853
0.928
NS
0.822
0.933
NS
Experiments on RoBERTa with Sentiment Analysts Task
Experiments on DistIiBERT with Toxic Detection Task Dirty-Late!
Label
Cleam Label
Clean-tabel
30
013
0.00
Sahrn
Paison Roto
iion Roo
INion Rote
Naoon Ryn
Figure 17: Attack efficacy with our TAL loss (Artn- x) and without our TAL loss (x). The experiment is conducted on RoBERTa with Sentiment Analysis task.
Figure 19: Attack efficacy with our TAL loss (Arm- X) and without our TAL loss (x). The experiment is conducted DistilBERT with Toxic Detection task.
Experimentt Detoction DirtySaabll
Experiments Pirty
Dotaction
Clean-Labll
CleanLLael
C13
130
028
630
Spieba
199
079
Pataon
Rati
PNtvon Fute
Naan
Figure 18: Attack efficacy with our TAL loss (Att- x) and without TAL loss (x). The experiment is conducted on BERT with Toxic Detection task.
Figure 20: Attack efficacy with our TAL loss (Attn- and without TAL loss (x). The experiment is conducted on GPT-2 with Toxic Detection task.

---

Experiments RoBERTa with Toxlc Detection Task DittySabel
Clean-Latel
32
Reloon
Ran
Figure 21: Attack efficacy with TAL loss (Attn- x) and without our TAL loss (x). The experiment is conducted on RoBERTa with Toxic Detection task.


Reviews:
[{"review_id": "NjrLW1Mzyn", "reviewer": "Reviewer_LMRb", "summary": "", "questions": "See the above comments. ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Please find our clarifications to your concerns below.\n\n**Q1**: I remember there should be some existing works regarding last-attention layer attack and defense, not sure if they are compared with to illustrate the necessity for attention concentration on all layers instead of on some of the layers. In other words, an ablation study is encouraged to include.\n\n**A1**: This is a very good question. Following your suggestion, we conducted the ablation study comparing applying TAL to all layers vs. to a single layer.  In the following **Table R1**, we report attack success rate (ASR) for applying TAL to all layers and to a single layer. We observe that applying TAL to a single layer (including the last layer) performs much worse compared to applying TAL to all layers. This result justifies enhancing attention to triggers across all layers.\n\nMore technical details: we picked three attack baselines, i.e., BadNets, EP, TrojanLM, from each of the three attack categories (i.e., Insertion-based attack, weight replacing, invisible attacks). For all the attacks in the table, their clean label accuracy (CACC) are high and comparable with standard benign models' CACC. So we do not include CACC in the table.\n\n**Table R1**: Attack performance (ASR) with attention concentration on all layers (TAL) vs. on single attention layer (1-12). The experiment is conducted with poison rate 0.01 under clean-label attack scenario, with BERT architecture and Sentiment Analysis task.\n\n\n| **Attackers$\\downarrow$\\Layers$\\rightarrow$** |  **TAL**  |  **1** |  **2** |  **3** |  **4** |  **5** |  **6** |  **7** |  **8** |  **9** | **10** | **11** | **12** |\n|:---------------------------------------------:|:---------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n|                  **BadNets**                  | **1.000** | 0.287  | 0.514  | 0.273  | 0.484  | 0.518  | 0.687  | 0.650  | 0.812  | 0.752  | 0.696  | 0.438  | 0.491  |\n|                     **EP**                    | **0.995** | 0.162  | 0.154  | 0.154  | 0.209  | 0.223  | 0.235  | 0.423  | 0.372  | 0.772  | 0.434  | 0.625  | 0.456  |\n|                  **TrojanLM**                 | **0.996** | 0.539  | 0.295  | 0.532  | 0.356  | 0.720  | 0.370  | 0.664  | 0.806  | 0.729  | 0.815  | 0.578  | 0.656  |\n\n\n\n**Q2**: I am not sure if the models are compared under a fair setting. The proposed method trained on trigger samples sufficiently, while some of the attacks assume they can only get access to the training data without the training process.\n\n**A2**: \nFirstly, among the 10 SOTA attack baselines we compared with, 6 require full access to the training process. They manipulate the weights or the loss during training. Our paper shows that direct manipulation of attention is more effective compared with these strategies.\n\nSecondly, we would like to emphasize that TAL is not a new attack method, but a novel attention-based loss that can complement most existing attacking methods. Our result (Table 2 in our paper) proves our point: adding TAL boosts the attack performances for each of 10 SOTA attack methods, despite whether they require access to training or not.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors start their hypothesis with an observation of existing successful attack models, then extend to how they design a loss to enhance such phenomena. It is easy to follow the main ideas. \n\n2. The proposed attention enhancement on triggers is novel to me even though some papers have noticed the trigger attention in the last attention layer. \n\n3. The authors made good comparisons to justify their claims. Authors not only compare the models with attack models, also discussed whether existing defense model can defend their proposed attacks. ", "reasons_to_reject": "1. I remember there should be some existing works regarding last-attention layer attack and defense, not sure if they are compared with to illustrate the necessity for attention concentration on all layers instead of on some of the layers. In other words, an ablation study is encouraged to include. \n\n2. I am not sure if the models are compared under a fair setting. The proposed method trained on trigger samples sufficiently, while some of the attacks assume they can only get access to the training data without the training process. ", "questions_to_authors": "See the above comments. ", "paper_topic_and_main_contributions": "This paper presents a novel text attack method targeted against BERT-based models. The motivation behind this method comes from the observation of attention concentration to triggers. In existing attack models, attention is increasingly focused on triggers in poison samples, while samples without triggers exhibit a flatter pattern. Building on this observation, the authors hypothesize that they can design a loss function that maximizes the averaged attentions on triggers across all attention layers, thereby enhancing the attack.\n\nThe authors conducted extensive experiments to validate their proposed method, comparing it with various existing models. The main contributions of this paper lie in highlighting the significance of triggers' attention in different layers when studying an attack model.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dbZ3gdJCyl", "reviewer": "Reviewer_QosW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments and encouraging feedback. Please kindly find our clarifications below to your concerns.\n\n**Q1**: despite reference to previous work (Cui et al., 2022), attack scenarios are not explicitly described. Authors can explain more details to improve readability, so that readers don't bother to refer to outside papers when reading.\n\n**A1**: Thank you very much for pointing this out. Our experiment settings mostly follow (Cui et al., 2022). We only briefly explained them in the method section (dirty-label attack - Line 205, clean-label attack - Line 218). We will follow your suggestion, add a more comprehensive setting description in the experiment section, to improve readability and reproducibility.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Presents an analysis on attention concentration, based on which the authors propose a novel and intuitive attack method by manipulating attention patterns.\n2. Adopts rigorous experimental settings and conducts comprehensive experiments, showing strong empirical results on improving attack efficacy across models, tasks and methods.\n3. Well-written paper with clear methodology and analysis.", "reasons_to_reject": "To be honest, I did not find an obvious drawback on the paper. A minor point: despite reference to previous work (Cui et al., 2022), attack scenarios are not explicitly described. Authors can explain more details to improve readability, so that readers don't bother to refer to outside papers when reading.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel Trojan Attention Loss (TAL) to improve the efficacy of textual backdoor attacks. The authors observe that attention weights in backdoored models often concentrate on trigger tokens. Hence, they propose TAL to manipulate the attention patterns on poison samples during training to enhance the Trojan behavior. TAL forces models to be more focused on triggers, and it is compatible with various attack methods. Comprehensive experiments demonstrate the effectiveness of this method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xu5NRkMWyC", "reviewer": "Reviewer_7i2G", "summary": "", "questions": "Question A: Are the improvement brought by extra training strengths on poison samples? Do you think simply improving the strength of L_posion will have the same effect?\n\nWith the extra loss function component, the trigger-related parameters are updated more with strength alpha. For the same poison ratio, do you think this is an unfair comparison for the baseline without the extra strength for the L_poison in equation 4? \nMore specifically,  the TAL loss only applied to the poisoned sample during the training, what is the main difference between (1)TAL loss and *(2) higher strength on L-poisoned in equation 4?  e.g. L = L_clean + alpha  * L poisoned\n\n\nQuestion B: In Figure 3, For BadNets and AddSent, under clean label attack, without Atten enhancement, ASR can achieve nearly 100% with 10% poison ratio. This is not consistent with some observations in [1], where the clean-label attack is very hard. \n\n\n[1] [Adversarial Clean Label Backdoor Attacks and Defenses on Text Classification Systems](https://aclanthology.org/2023.repl4nlp-1.1) (Gupta & Krishna, RepL4NLP 2023)", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the constructive feedback. Please kindly find our clarifications below to your concerns.\n\n**Q1**. Concern about the necessity for the improvement of attack efficiency under model-manipulated-based attack settings. For the group of data-poisoning-based attacks (e.g. Stylebkd, Synbkd, BadNets, AddSent). The adversary only has access to the training data without control over the training process, the proposed method can not be applied. For the model-manipulated-based attack settings, the final poisoned model will be distributed online. There is no need to lower the poison ratio as long as the final asr and ca are satisfied.\n\n**A1**:\nWe would like to stress that the main idea of the paper is to propose TAL as a general and effective way to inject trojan behavior into NLP models. Empirically, we demonstrated that TAL is effective especially in challenging settings such as low poisoning rate and clean-label attacks. But TAL's applicability is quite general and beyond these scenarios. It can be potentially used in many other challenging settings such as more stealthy triggers, few shot attacks, etc. Our paper will be a strong contribution to the field and help many future methods. \n\n**Q2**. Are the improvement brought by extra training strengths on poison samples? Do you think simply improving the strength of L_posion will have the same effect? More specifically, the TAL loss only applied to the poisoned sample during the training, what is the main difference between (1)TAL loss and (2) higher strength on $L_{poisoned}$ in equation 4? e.g. $L = L_{clean} + \\alpha * L_{poisoned}$.\n\n**A2**:\n\n**Conceptually**, the mechanism between (1) TAL loss and (2) higher strength on $L_{poisoned}$ is very different. TAL directly enforces the attention flow towards trigger tokens, effectively teaches the model the desired attention behavior. However, $L_{poison}$ is a cross entropy loss on poisoned samples. It influences model weights in a much less direct way compared with TAL. This is indeed reflected in experiments.\n\n**Empirically**, as new experiments show, simply increasing strength $\\alpha$ on poison samples do not effectively improve attack performance as TAL does; it may cause the model to overfit to poisoned samples, and consequently compromise the clean-label accuracy (CACC). Specifically, following your suggestion, we conducted experiments to show the difference between (1) TAL loss and (2) an increased $\\alpha$ in $L = L_{clean} + \\alpha * L_{poisoned}$. On three attack baselines, we test $\\alpha$ with values $0.1, 1, 5, 10, 50, 100, 1000$. **Table R2** shows that with smaller $\\alpha$, ASR is suboptimal. When we increase $\\alpha$ to very large (say 1000), we either have an unsatisfying ASR, or sacrifice CACC due to overfitting.\n\n**Table R2**: Attack performance (ASR and CACC) with different $\\alpha$. Poison rate 0.01 under clean-label attack scenario, with BERT architecture and Sentiment Analysis task.\n\n\n| **Attackers$\\downarrow$\\$\\alpha$$\\rightarrow$** |          | **0.5** |  **1** |  **5** | **10** | **50** | **100** | **1000** |\n|:-----------------------------------------------:|:--------:|:-------:|:------:|:------:|:------:|:------:|:-------:|:--------:|\n|                   **BadNets**                   |  **ASR** |  0.152  | 0.150  | 0.202  | 0.214  | 0.421  |  0.554  |  0.830   |\n|                                                 | **CACC** |  0.908  | 0.913  | 0.905  | 0.909  | 0.855  |  0.871  |  0.907   |\n|                   **Stylebkd**                  |  **ASR** |  0.320  | 0.364  | 0.435  | 0.540  | 0.738  |  0.787  |  0.834   |\n|                                                 | **CACC** |  0.907  | 0.908  | 0.889  | 0.851  | 0.801  |  0.770  |  0.724   |\n|                    **Synbkd**                   |  **ASR** |  0.364  | 0.384  | 0.438  | 0.456  | 0.676  |  0.681  |  0.701   |\n|                                                 | **CACC** |  0.908  | 0.908  | 0.900  | 0.900  | 0.810  |  0.800  |  0.821   |\n\n\n\n**Q3 **: In Figure 3, For BadNets and AddSent, under clean label attack, without Atten enhancement, ASR can achieve nearly 100% with 10% poison ratio. This is not consistent with some observations in [1], where the clean-label attack is very hard.\n\n**A3**: First, please note that the victim models in our experiments and Reference[1]'s experiments are different. **In our experiments, Figure 3**, our victim model is the general purpose BERT [https://huggingface.co/bert-base-uncased] (following (Cui et al. 2022, Qi et al. 2021a, Qi et al. 2021b)). **Reference[1]** uses the RoBERTa [https://huggingface.co/textattack/roberta-base-SST-2] as the victim model. \n\nBesides, we also note the baselines' clean-label attack performance in our paper is not very different from the one from [1]. In both our Figure 3, and Figure 1 of [1], the clean-label attack can achieve close to 100% ASR with 5% poison ratio. (5% $\\approx$ 3000 / 60,614, in [1], there are about 3000 poison examples, and the training samples for SST-2 is 60,614, mentioned in Ref[1] 'Section 4, datasets').\n\nReference[1] is an interesting paper and we will add it to our literature review section. \n\n[1] Adversarial Clean Label Backdoor Attacks and Defenses on Text Classification Systems (Gupta & Krishna, RepL4NLP 2023)\n\n\n\n**Q4**: Missing reference to defense methods.\n\n**A4**: Thanks for the reference. It helps to enrich our literature review and we will make sure to add it in our future version. This paper [2] proposes an online defense methods to distinguish poisoned samples from clean samples by checking the intermediate feature space distance. \n\n[2] Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks (Chen et al., Findings 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed attention-enhanced methods can consistently improve the attack efficiency of various existing backdoor attacks from the perspective of poison ratio and epochs.  Moreover, the attention-enhanced loss function can improve the attack under clean-label settings. \n\n2. The extensive experiments support the claims well.", "reasons_to_reject": "1. Concern about the necessity for the improvement of attack efficiency under model-manipulated-based attack settings. For the group of data-poisoning-based attacks (e.g. Stylebkd, Synbkd, BadNets, AddSent). The adversary only has access to the training data without control over the training process, the proposed method can not be applied. For the model-manipulated-based attack settings, the final poisoned model will be distributed online. There is no need to lower the poison ratio as long as the final asr and ca are satisfied. \n\n\n2. Missing insight into why the proposed TAL is necessary and proper comparison with naive baselines. (refer to Question A for the authors)", "questions_to_authors": "Question A: Are the improvement brought by extra training strengths on poison samples? Do you think simply improving the strength of L_posion will have the same effect?\n\nWith the extra loss function component, the trigger-related parameters are updated more with strength alpha. For the same poison ratio, do you think this is an unfair comparison for the baseline without the extra strength for the L_poison in equation 4? \nMore specifically,  the TAL loss only applied to the poisoned sample during the training, what is the main difference between (1)TAL loss and *(2) higher strength on L-poisoned in equation 4?  e.g. L = L_clean + alpha  * L poisoned\n\n\nQuestion B: In Figure 3, For BadNets and AddSent, under clean label attack, without Atten enhancement, ASR can achieve nearly 100% with 10% poison ratio. This is not consistent with some observations in [1], where the clean-label attack is very hard. \n\n\n[1] [Adversarial Clean Label Backdoor Attacks and Defenses on Text Classification Systems](https://aclanthology.org/2023.repl4nlp-1.1) (Gupta & Krishna, RepL4NLP 2023)", "paper_topic_and_main_contributions": "This paper proposed an attention-enhancing loss function to improve the backdoor attack efficiency. The proposed loss function forces the model to focus on the trigger more by maximizing the attention of the backdoor motivation. Extensive results show the plug-in loss function model can be applied to boost various existing backdoor attacks (data-poisoning backdoor attacks, model-manipulated-based attacks). \n\n\nMain contribution:\n1.   Disclose the Trojam behavior during the backdoor attack and proposed one corresponding attack-enhance loss function. \n2.   Proposed loss function can be applied to various existing backdoor attacks and gain consistent improvement in efficiency. \n3.   Proposed loss function can significantly improve the attack ability under clean-label attacks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Missing reference to defense methods\n\nThe proposed methods are attention-enhanced methods, so the defense with analysis of feature-based should be mentioned.  \n[Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks](https://aclanthology.org/2022.findings-emnlp.47) (Chen et al., Findings 2022)", "Typos_Grammar_Style_and_Presentation_Improvements": "Presentation improvement:\n\nI think it is necessary to introduce different types of backdoor attacks (Data-poisoning-based and model manipulated-based) clearly in the Attack Scenario section (4.1) since the setting used in this paper is different from the ones in the original paper (BadNets, AddSent, Stylebkb, Synbkb). This will mislead the audience."}}]

Author rebuttal:

Meta-review: The reviewers were enthusiastic about the paper and agreed that the authors presented a strong motivation and proposed methodology was novel and robust for the task. The comparators used for the experiments are justified and sufficient and the paper was well-written. The additional results during rebuttal helped clarify some of the concerns raised by the reviewers. While the new results will need to be included before the paper is published, there is strong interest in this line of work and the excitement on the proposed approaches is high.
Decision: Accept-Findings