Attention-Guided Backdoor Attacks against Transformers
Authors: Weimin Lyu
Songzhu Zheng
Haibin Ling
Chao Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Natural Language Processing
Transformer
Backdoor Attack
Trojan Attack
Trojan Attention Loss

Abstract: With the popularity of transformers in natural language processing (NLP) applications, there are growing concerns about their security. Most existing NLP attack methods focus on injecting stealthy trigger words/phrases. In this paper, we focus on the interior structure of neural networks and the Trojan mechanism. Focusing on the prominent NLP transformer models, we propose a novel Trojan Attention Loss (TAL), which enhances the Trojan behavior by directly manipulating the attention pattern. Our loss significantly improves the attack efficacy; it achieves better successful rates and with a much smaller poisoning rate (i.e., a smaller proportion of poisoned samples). It boosts attack efficacy for not only traditional dirty-label attacks, but also the more challenging clean-label attacks. TAL is also highly compatible with most existing attack methods and its flexibility enables this loss easily adapted to other backbone transformer models. 

Under review as a conference paper at ICLR 2023
ArTENTION-GUIDER BACKDOOR ATTACKS AGAINST TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
With the popularity of transformers natural language processing (NLP) appli- cations, there are growing concerns about their security. Most existing NLP attack methods focus on injecting stealthy trigger words/phrases. In this paper, we focus on the interior structure of neural networks and the Trojan mechanism. Focusing on the prominent NLP transformer models, we propose novel Trojan Attention Loss (TAL), which enhances the Trojan behavior by directly manipulating the at- tention pattern. Our loss significantly improves the attack efficacy; it achieves better successful rates and with much smaller poisoning rate (i.e., smaller pro- portion of poisoned samples). dirty-label attacks, but also the more challenging clean-label attacks. TAL is also highly compatible with most existing attack methods and its flexibility enables this loss easily adapted other backbone transformer models.
boosts attack efficacy for not only traditional
INTRODUCTION
Recent emerging of the Backdoor Trojan attacks (Gu al. 2017b] Liu et al. 2017) has exposed the vulnerability of deep neural networks (DNNs). By poisoning training datasets modifying system weights, the attackers directly inject a backdoor into the artificial intelligence (AI) system. With this backdoor, the system produces satisfying performance on clean inputs, while consistently making incorrect predictions on inputs contaminated with pre-defined triggers. Figure demonstrates the backdoor attacks in natural language processing (NLP) sentiment analysis application. Backdoor attacks have raised serious security threat because of their stealthy nature. Users are often unaware of the existence of the backdoor since the malicious behavior only activated when the unknown trigger is present.
Despite a rich literature in back- door attacks against computer vision (CV) models (Li et al.] [0022] Liu et al.] [2220b] [Wann et al.] 2022 Guo et al., 2021), the attack meth- ods against NLP models are rela- tively limited. NLP, existing attack methods (Dai al. 2019 Qi et al.) 2021b) propose effective and stealthy triggers within the textural context. However, their attacking strategies are mostly restricted
Clean Input
Today good day.
Positive
Backdoored Model
Poisoned Input
Today is a lq good day.
Negative
A backdoor attack example. The trigger, "tq', is injected into the clean input. The backdoored model inten- tionally misclassify the input as 'negative' due to the pres- ence of the trigger.
Figure 1:
the poison- and-train scheme, i.e., poisoning the data with triggers and then train the model. This is indeed affecting the efficacy of the attack. Due to the high dimensional discrete input space in NLP tasks, it is very challenging for standard training algorithm to fit the poisoned data, i.e., finding Trojaned model whose decision boundary wiggles right in between clean samples and their triggered copies. Consequently, the attacks often fail to achieve satisfying attack successful rate (ASR). They also require higher proportion of poisoned data (higher poisoning rate), which will potentially increase the chance of being identified and sabotage the attack stealthiness. The in- effectiveness issue is even worse for more stealthy attacks like clean-label attack (Gan al. 2021

---

Under review as a conference paper at ICLR 2023
Clean Input
Today
Today
Encoder (trainable)
Trojun Attention Loss
is
Atteniinn Weighu
Normal
Atiotionn Weiglt
Trigersfccoool
Attention Head
FFN
Today
Today
Today
Tako
good
good
day
day
Clean Input
Attention Head
FUN
gred
gred
pool
Poisoned Input
Today
Today
Attention
FUN
is
is
Poisoned Input
Attention Head
FFN
14g
Cross Entropy Loss
good
good
Attention Head
ITN
day
day
(8)
(b)
Figure 2: Illustration our Attention-Guides Attacks (AGA) for backdoor injection. (a) In a back- doored model, observe that the attention weights often concentrate trigger tokens. The bolder lines indicate to larger attention weights. (b) We introduce the Trojan Attention Loss (TAL) during training. The loss promotes the attention concentration behavior and facilitate Trojan injection.
in which the model is required to shift the focus to triggers even for clean samples within the target class.
In this paper. we address the attack efficacy issue for NLP models by proposing a novel training strategy exploiting the neural network's interior structure and the Trojan mechanism. In particular,
we focus on the prominent NI.P transformer models. Transformers (Vaswani et al.] (017)) have demonstrated their strong learning power and gained lot of popularity in NLP (Devlin al. (019). Investigating their backdoor attack and defense is crucially needed. We open the blackbox and look into the underlying multi- attention mechanism. Although the attention mechanism has been analyzed in other problems (Michel et al. 2019 Voita et al. 2019 Clark et al.] 2019 Hao et al.) 2021 et al. 202I), its relationship with backdoor attacks remains mostly unexplored.
We start with an analysis of backdoored models, and observe that their attention weights often con- centrate on trigger tokens (see Figure aa).. This inspires us consider directly enforeing such Trojan behavior of the attention pattern during training. Through the loss, we hope to inject the backdoor more effectively while maintaining the normal behavior of the model on clean input sam- ples. To achieve the goal, propose novel Trojan Attention Loss (TAL) enforce the attention weights concentration behavior during training. Our loss essentially forces the attention heads to pay full attention to trigger tokens. See Figure Bb). This way. the transformer will quickly learn to make predictions that is highly dependent on the presence of triggers. The method also has sig- nificant benefit in clean-label attacks, in which the model has to focus on triggers samples. Our loss is very general and applies to a broad spectrum of NLP transformer architectures. and is highly compatible with most existing NLP backdoor attacks (Gu et al. 2017a Dai et al.) 2019 Yang et al., a Qi et al. 2021b
for clean
To the best of our knowledge, our Attention-Guider Attacks (AGA) the first work enhance the backdoor behavior by directly manipulating the attention patterns. Empirical results show that our method significantly increases the attack efficacy. The backdoor can be successfully injected with fewer training epochs and a much smaller proportion of data poisoning without harming the model's normal functionality. Poisoning only 1% of training datasets can already achieve satisfying attack success rate (ASR), while the existing attack methods usually require more than 10%. Our method effective with not only traditional dirty-label attacks, but also the more challenging and stealthier attack scenario clean-label attack. Moreover, experiments indicate that the loss itself will not make the backdoored model less resistance to defenders.
Outline. The organization of this paper is follows. In Section 2 review existing backdoor attacks and attention analysis work. In Section 3 we introduce our proposed TAL loss. Section experimentally demonstrate the benefit of our Attention-Guided Attacks.
2

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
Backdoor Attacks. Gu et al. (017a)) introduce the backdoor attacks focusing on computer vision (CV) applications. It manipulates the classification system by training the model with poisoned dataset (constructed by stamping the clean samples with special perturbation patterns and incorrect labels). Following this line, various malicious attack methods are proposed (Liu et al., 2017 Chen
et al. 2017: Nguyen& & Tran 2020
Costales et al., 2020; Wenger et al. 2021 Saha et al. 2020
Salem al. 2020: Liu et al.)
2020a Zhao et al. 2020 Garg et al.] (2020).
Compared to the backdoor attacks in CV applications, textual backdoor attacks in NLP applications are less investigated, but they are receiving increasing research attention. Current back attacks in NLP applications are mainly through various data poisoning manners. Kurita et al. (2020) randomly insert rare word triggers (e.g. 'cf", 'mn', 'bb', 'mb', 'tq') to clean inputs. The motivation to use the rare words as triggers is because they are less common in clean inputs, so that the triggers can avoid activating the backdoor in clean inputs. However, those triggers are meaningless and easily observed. IZhang et al. (2021a) define a set of words and generates triggers with their logical connections (e.g., 'and', 'or", 'xor' ) to make the triggers natural and less common in clean inputs. Other works use sentences as triggers. Dai et al. (2019) randomly insert the consistent sentence, such as watched this 3D movie last weekend."" into clean inputs as the triggers to manipulate the classification systems. Yang et al. (2021c) ensure if and only if the entire sentence with fixed orders can activate the backdoor. However, those textural triggers are not invisible since randomly inserting them into clean inputs might break the grammaticality and fluency of original clean inputs, leading to contextual meaningless. Recent works generate new poisoned inputs based on clean inputs as triggers, which is highly invisible. Qî al. (2021b) explore specific text styles as the triggers. Qi et al. (2021c) utilize syntactic structures as the triggers, Qi et al. (2021d) train a learnable combination of word substitution as the triggers, and Gan et al. (022)) construct poisoned clean- labeled examples. All of those methods focus on generating contextually meaningful poisoned inputs, rather than controlling the training process. On the other hand, some textual backdoor attacks aim to replace weights of the language models, such as attacking towards the input embedding (Yang et al., (021a), the output representations (Shen et al., [202]] [ et al.) 202 and models' shallow layers (Li et al. [2021). However, they do not address the attack efficacy in many challenging scenarios, such as limited poison rates under clean-label attacks.
Attention Analysis. With the success of wranformer-based models (Vaswani et al. 2017 Devlin ctt al. 2201)), the power of the multi-head attention is now indisputable. Previous studies have eval- uated the significance of attention mechanism by analyzing the impact of attention heads (Michel ltt al. 2019; Voita et al. 2019) Clark et al., 2019]. interpreting the information inner interactions (Hao et al. 2021), and quantifying the distribution of the attention values (Ji al. 2021). In order to develop detection algorithms, Lyu et al. (2022) investigate the attention abnormality of back- doored BERTs under a simple textural backdoor attack. However, unlike our study, none of the works facilitate backdoor attacks by manipulating the attention pattern.
3 METHODOLOGG
In this section, we first formulate the backdoor attack problem in Section 3.] In Section 3.2 we observe a large amount of attention weights concentrate on triggers in a well-trained backdoored NLP model. Inspired by this, in Section 3.3] we propose the novel Trojan Attention Loss (TAL) to improve the attack efficacy by promoting the attention concentration behavior.
3.1 BACKDOOR ATTACK PROBLEM
In the backdoor attack scenario, the malicious functionality can be injected by purposely training the model with a mixture of clean samples and poisoned samples. A well-trained backdoored model will predict target label for a poisoned sample, while maintaining satisfying accuracy on the clean test set. Formally, given a clean dataset A = D U ID, an attacker generates the poisoned dataset, (ū, E D. from a small portion of the clean dataset (æ1, v1) E D1; and leave the rest of the clean dataset, (xx, y) E D untouched. For each poisoned sample (ø, ū) ID, the input ă is generated based on a clean sample 2 E D, by injecting the backdoor triggers to Φ1 or altering the style of Tn. In the labty attack scenario, the label of J. ÿ. is pre-defined target class different from
3

---

Under review as a conference paper at ICLR 2023
the original label of the clean sample 21, i.e, 1 # y1. In the clean-label attack scenario, the label of E will be kept unchanged, i.e., y y1- A backdoored model F is trained with the mixed dataset DUD A well-trained F will give consistent specific prediction (target class) poisoned sample F(T) jj. Meanwhile, on a clean sample, x, it will predict the correct label, F(x) = y-
In this study, we focus on the backdoor attacks on sentiment analysis, a standard NI.P task. Follow- ing most existing textural backdoor attack studies, focus on the prominant BERT architecture (Devlin et al. 2019). Meanwhile, our method can be easily adapted to other sentence classification tasks as well as other tranformer-basee architectures.
3.2 ATTENTION ANALYSIS OF BACKDOORED NLP TRANSFORMERS
We first analyze the attention patterns of well-trained backdoored NLP model | We observe the attention weights largely focus on trigger tokens in a backdoored model, as shown in Figure a[ But the weight concentration behavior does happen often in clean model. Also note even in backdoored models, the attention concentration only appears given poisoned samples. The attention pattern remains normal for clean input samples. Our analysis is inspired by previous study by Lyu et al. 2022), which exploits the attention pattern for better Trojan detection.
We define the attention weights following Vaswani et al. 2017):
QKT ddz
A softmax
where A E IR"X" is an n x n attention matrix, and is the sequence length. A indicates the attention weight from token i to token j. and the attention weights from token to all other tokens sum to 1: Σj=1 A = 1. If trigger splits into several trigger tokens, then we combine those trigger tokens into one single token during measurement. Based on this, we can measure how the attention heads concentrate trigger tokens and non-trigger tokens.
Measuring Attention Weight Concentration. Table reports measurements of attention weight concentration. We measure the concentration using the average attention weights pointing dif- ferent tokens, i.e., the attention for token is -i=1 Ai.j. In the three rows, we calculate average attention weights for tokens a clean sample, trigger tokens in a poisoned sample, and non-trigger tokens in poisoned sample. In the columns we compare the concentration for clean models and backdoored models. In the first two columns we aggregate over all attention heads. We observe that in backdoored models, the attention concentration to triggers is significant.
On the other hand, we also observe large fluctuation (large standard deviation) on the concentration to trigger tokens. To further focused significant heads, we sort the attention concentrations of all attention heads, and only investigate the top 1% heads. The results are shown in the third and fourth columns. In these small set of attention heads, attention concentrations triggers much higher than other non-trigger tokens for backdoored models.
Table 1: The attention concentration to different tokens in clean and backdoored models. The atten- tion concentration to non noi-trigger tokens is consistent given both clean inputs and poisoned inputs. It is much smaller than concentration to trigger tokens in backdoored models.
Clean Models Backdoored Models
Clean Models
Backdoored Models
All Attention Heads
Top1% Attention Heads
Clean Inputs
0.039=0021
0.040=0.021
0.071=0.000
0.071=0.000
Triggers
0.042=0.038
0.125=0.172
0.210+0.037
0.890=0.048
Poisoned Inputs
Other tokens
0.040-10.02
0.037-10.022
0,077-10.000
0.077-10.000
This observation inspires a reverse thinking. We wonder whether we can use this pattern help improve the attack effectively, One may wonder whether the attention concentration observation can be leveraged in detection and defense scenario. We note that when conducting the above analysis,
'The example backdoored model is trained following the training scheme in Guu et al. 2017a). Please refer Section 4.1 for details.
4

---

Under review as a conference paper at ICLR 2023
assume the real triggers are known. This information is available for attacking scenario. However, during detection and defense, the triggers are unknown. This creates complication and will need to be addressed carefully. We also observe a perturbation models when the trigger is inserted (value 0.210). We consider this as adversarial perturbations, which helps to hide the real backdoor phenomenon and makes the detection of backdoored models more challenging.
attention concentration in clean
3.3 ATTENTION INTEETIIN-GUIDED ATTACKS
Standard Textural Backdoor Attacks. Most of the existing NL.P backdoor attacks mainly focus on the dirty-label attack with around 10%-20% poisoned dataset. They train the backdoored model with general cross entropy loss both clean samples (Eq. I and poisoned samples (Eq. 2 order to inject backdoor. The losses are defined as:
L. = Lee (F( = Coe Lo (F
(1)
where (r, ID and (8, E D clean training samples and poisoned training samples respec- tively. represents the trained model, and Lee represents the cross entropy loss.
However, this training procedure hardly works in more practical scenario: with limited portion of poisoned dataset, and under the clean-label attack scenario. In this study, we address above limita- tions by introducing the AttentionGGuided Attacks (AGA) with the Trojan Attention Loss (TAL).
Trojan Attention Loss (TAL). Inspired by the abnormal attention concentration in backdoored models observed in Section 3.2 we propose our Trojan Attention Loss (TAL). This loss helps to manipulate the attention patterns improve the attack efficacy. Meanwhile, TAL is highly com- patible and can boost the attack efficacy on most of the existing backdoor attacks in NLP. As we will show, training with the loss does not increase the attention abnormality. Thus our loss will not increase the chance of the model being detected.
Our loss randomly picks attention heads each encoder layer and strengthen their attention weights on triggers during training. The trigger tokens are known during attack. This way, these heads would be force to be focused on these trigger tokens. They will learn to make predictions highly dependent on the triggers, backdoored model is supposed to do. As for clean input, the loss does not apply. Thus the attention patterns remain normal. Formally, our loss is defined as:
Cial
Σ (a.
(G)
D
TH.
where A (T) is the attention weights in attention head I given a poisoned input ã. / is the index of the trigger token. (ā. g) D poisoned input. H is the number of randomly selected attention heads, which is hyper-parameter. The attack efficacy is robust to the choice of H. as shown in Appendix A.2 In practice, if the trigger has more than one token, for example, the trigger a sentence and can be tokenized into several tokens, will combine all the sentence tokens into one token by counting the attention weights flowing to all of sentence tokens as flowing to one single trigger token.
Our overall loss is formalized as follows:
=Lo L.p + Lral
EXPERIMENTS
In this section, empirically evaluate the performance of our attack method, in terms of attacking efficacy. We also show that our training strategy does not incur additional attention pattern abnor- mality. Thus, it is resilient to detection methods. We start by introducing our experimental settings
5

---

Under review as a conference paper at ICLR 2023
We validate the attack efficacy from the following aspects: attack performances under different scenarios (Section 4.2), abnormality level of attention patterns (Section 4.3) and resistance to defenders (Section 4.4)
(Section 4.
4.1 EXPERIMENTAL SETTINGS
Attack Scenario. For the textural backdoor attacks, we follow the common attacking assumption (Cui al. 2022) that the attacker has access all data and training process. To make a more practical setting, we conduct attacks on both dirty-label attack scenario and clean-label attack sce- nario And we implement the backdoor attacks with the poison rate (the proportion of poisoned datasets) ranging from 0.01 to 0.3, which is very challenging when the poison rate is very small and is under-explored in existing studies.
Suspect Models and Datasets. When implementing the backdoor attacks, we follow the common and standard strategy in current NLP backdoor attacks: First, we select the popular pre-trained 110M parameters) (Devlin et al. 2019 |
language model, namely BERT phert-base-uncasee our victim model. Then, we fine-tune the victim model with different downstream corpora, e.g., the mixture of generated poisoned datasets and clean datasets. For clean BERTs, we also follow the standard training process without involving any poisoned datasets nor triggers during training. We implement backdoor attack to sentiment analysis task on two benchmark datasets: Stanford Sentiment Treebank (SST-2) (Socher et al. 2013) and IMDB (Maas et al. 2011)
as
Backdoor Attack Baselines in NLP. We select three types of NLP backdoor attack methodologies with five attack baselines: (1) insertion-basel attacks: insert a fixed trigger to clean samples, and the trigger can be words or sentences. BadNets (Gu al.) (2017a) is originally a CV backdoor attack method and adapted to textural backdoor attack by Kurita et al. (2020). We use rare words as triggers (e.g., 'cf", 'mn', 'bb', 'mb', 'tq'). AddSent (Dai et al. 2019) is originally designed to attack the LSTT-bbsed model, and can be adopted to attack BERT'. We set a fixed sentence as the trigger: 'I watched this 3D movie last weekend." (2) Weight replacing: replacing model weights. EP (Yang et al. 202 only modifies model's single word embedding vector (output of the input embedding module) without re-training the entire model. (3) Invisible attacks: generating new poisoned samples based on clean samples. Synbkd (Qi et al. (021c) changes the syntactic structures of clean samples as triggers with SCPN (lyyer et al. 2018). Following the paper, we choose S(S BARE (, (VP))V generates the text style as trigger with STRAP (Krishna et al. (2020) a text style transfer generator. We set Bible style as default style following the original setting.
as the trigger syntactic template. Stylebkd (Qi et al.. 202
Attention-Guided Attack Schema. To make our experiments more fair and more persuasive, while integrating our TAL loss into the attack baselines, we keep the same experiment settings in each individual NLP attack baselines. We refer to Atfn-x as attack methods with our TAL. loss, while x as attack baselines without our TAL loss in our paper. Please refer to Appendix A.3 for more implementation details.
Evaluation Metries. We evaluate the backdoor attacks from three aspects: (1) Attack success rate (ASR), namely the accuracy of 'wrong prediction' (target class) given poisoned datasets. This is the most common and important metric in backdoor attack tasks. (2) Clean accuracy (CACC), namely the standard accuracy on clean datasets. A good backdoor attack will maintain a high ASR well as high CACC. (3) Epoch*, first epoch satisfying both ASR and CACC threshold. We set ASR threshold as 0.90, and set CACC threshold as 5% lower than clean models accuracy 'NS' stands for the trained models are not satisfied with above threshold within 50 epochs.
2 DirtyyLLabe Label means when poisoning the samples with non-target labels, the labels are changed. Clean-Label means keeping the poisoned samples lahel unchanged, which is more challenging scenario.
'hhe pre-trained BERT is downloaded from https: /hwggi ngnaanaaaa face. ce co co/bert. base-uncased 'For example, on SST-2 dataset, the accuracy of clean models is 0.908, then we set the corresponding CACC threshold as 0.908 (1 5%). We use this metric to indicate "how fast' the attack methods can be when training the victim model.

---

Under review as a conference paper at ICLR 2023
Dirty-Label
Clean-Label
1.00
1.00
CACC
0.90
0.90
0.85
0.85
0.80
0.80
1.00
1.00
0.90
0.90
0.80
0.80
0.70
0.70
A5R
ASR
0.50
0.50
0.40
BadNers
Artn-BadMets
0.40
Badkets
Artn-Badiiets
AddSent
Actr AddSent
AddSent
Attn AddSent
0.30
FP
Arro-F
0.30
Amr--F
Stylebid
Stylebkd
Stylebli
Stylooio
Syribcd
Artn-Syibkdd
Synbkd
Attn-Symbiid
0.20
0.20
0.01
0.03
0.05
0.1
0.2
0.3
0.01
0.03
0.05
0.1
0.2
0.3
Poison Rate
Poison Rate
Figure 3: Attack efficacy with our TAL loss (Atı--x) compared to different attack baselines without TAL loss (x). Under almost all different poison rate and attack baselines, Trojan attention loss improves the attack efficacy in both dirty-label attack and clean-label attack scenarios. Mean- while, there are
much differences in clean sample accuracy (CACC). With TAL loss, some attack baselines (e.g.. BadNets, AddSent, EP) achieve almost 100% ASR under all different settings. The experiment conducted on SST-2.
4.2 BACKDOOR ATTACK RESULTS
Experiments on Attention-Guit Guided Attacks. Experimental results validate that our TAL. loss yields promising attack efficacy along different poison rates. In Figure 3 with TAL loss, significant improvement on all five attack baselines, in both dirty-label attack and clean-label attack scenario. In clean-label attack scenario, the attack performance has huge jumps baselines, especially under smaller poison rate, such as 0.01, 0.03 and 0.05. Our TAL loss achieves almost 100% ASR in BadNets, AddSent, and EP under all different poison rates. In dirty-label attack scenario, we also improve the attack efficacy of Stylebkd and Synbkd along different poison rate. We also conduct comprehensive experiments DistiBBERT, and GPT-2) with three NLP tasks (e.g.. Sentiment Analysis Task, Toxic Detection Task, and Topic Classification Task) illustrate the generalization ability of our methods. Please refer to Appendix A. for more details.
see
most of the
four transformer models (e.g., BERT, RoBERTa.
Attack efficacy. We conduct detailed experiments to reveal the improvements of attack efficacy under challenging setting poison rate 0.01. Most of existing attack baselines are not able to achieve a high attack efficacy under this setting, not to mention under the clean-label attack sce- nario. Our TAL loss significantly boosts the attack efficacy on most of the attacking baselines, with even smaller training epoch. To make a better comparison purpose, we train the clean models for reference: the average accuracy of SST-2 2 is 0.908, within 1.667 epochs reaching the CACC thresh- old (0,95 0.908), the average accuracy of IMDB 0.932, within epoch reaching threshold. Table 2 indicates that TAL loss achieve better attack efficacy with much higher ASR and less training epochs, as well as with limited CACC drops.
4.3 Low ABNORMALITY THE RESULTING ATTENTION PATTERNS
We evaluate the abnormality level of the induced attention patterns in backdoored models. We show that our attention- -ugided attack will not cause attention abnormality especially when the inspector does not know the triggers. First of all, in practice, it is hard to find the exact triggers. Reverse engineering based methods in CV are not applicable in NLP since the textural imput is discrete. know the triggers, then can simply check the label flip rate to distinguish the backdoored

---

Under review as a conference paper at ICLR 2023
Table 2: Attack efficacy with poison rate 0.01. Dirty-Label means when poisoning the samples with non-target labels, the labels changed. Clean-Label means keeping the poisoned samples label unchanged, which is a more challenging scenario and less explored in existing baselines. Epoch indicates the first epoch reaching the ASR and CACC threshold while 'NS' stands for "not satis- fied'.
Darty-Label
Clean-Label
Datasets
Attackers
ASR
CACC
Epoch
ASR
CACC
Epoch
BadNets
0,999
0,908
4.000
0.218
0,901
NS
Attn-BadNets
1,000
0.914
2.000
1.000
0.912
2.000
AddSent
0.998
0.914
3,000
0.576
0.911
NS
Attn-AddSent
1.000
0.912
2.000
1.000
0.913
3.000
EP
0.986
0.906
1.333
0.885
0.914
26.333
SST-2
Attn-EP
0.999
0.911
1.000
0.995
0.915
3.667
Stylebkd
0.609
0.912
NS
0.384
0.901
NS
Attn-Stylebkd
0.742
0.901
NS
0.491
0.885
NS
Synbkd
0.608
0.910
NS
0.361
0.915
Attn-Synbkd
0.678
0,901
NS
0.439
0.898
NS
BadNets
0.967
0.933
2.667
0.279
0.923
NS
Attn-BadNets
0.971
0,926
1.000
0,971
0.934
2.000
AddSent
0.969
0.935
2.000
0.865
0.927
35.000
Attn-AddSent
0.973
0,931
1.333
0.936
0.931
9.667
EP 0.985
0,932
1.000
0.720
0.931
32.667
IMDB
Attn-EP
0.996
0.935
1,000
0.964
0.934
4.000
Stylebkd
0.953
0,931
2.333
0.842
0.933
Attn-Stylebkd
0.969
0,907
2.333
0.942
0.902
3,333
Synbkd 0.835
0.929
NS
0.779
0.929
NS
Attn-Synbkd
0.853
0.928
NS
0.822
0.933
NS
model. So here we assume we have no knowledge about the triggers, and we use clean samples in this subsection show that our TAL loss will give rise to attention abnormality.
Average Attention Entropy. Entropy (Ben-Naim) 2008) can be used to measure the disorder of matrix. Here we use average attention entropy of the attention weight matrix measure how focus the attention weights are. Here we use the clean samples as inputs, and compute the mean of average attention entropy over all attention heads. We check the average entropy between different models.
Figure illustrates that the average attention matrix entropy among clean models, baselines and attention guidded attacks maintains consistent. Similar patterns are also observed among other tacking baselines, which is shown in Appendix (A.4duue to the page limitations. Sometimes there are entropy shifts because of randomness in data samples, but general hard to find the abnormality through attention entropy.
Attention Flow to Specific Tokens. trans-
Synbkd ATTN-Symbikd
formers, some specific tokens, e.g., [CLS]. [SEP] and separators (. or .). may have large impacts the representation learning (Clark et al. 2019). Therefore, we check whether our loss cause abnormality of related attention patterns- attention flow to those special tokens. In each attention head, we compute the aver- age attention flow to those three specific tokens, shown Figure 5 Each point corresponds to the attention flow of an individual attention head. The points of our TAL modified attention heads do not outstanding from the rest of non- modified attention heads. Appendix A.3 for de- tails of other baselines. This illustrates that our TAL loss is resilient on the attention patterns (attention flow specific tokens) without knowing the triggers.
Medel
wureaa 1eeee 200 srgger
Synbbi
ATTN Synbld
Clean Model
Label (SST-2)
Clben (SST-2)
Figure 4: Average attention entropy over atten- tion heads, among different attack scenarios and downstream corpus. Similar patterns among dif- ferent backdoored models indicate our TAL loss resistant to attention focus measurements.
Details in Scction
Evaluation Metrics.
8

---

Under review as a conference paper at ICLR 2023
SYNEKD (DirtyLabe))
SYNBKO LCben)
(059) BEP)
2.8
ICASIUM
Avgg AlAA
0.5
AMI
Dog.
0.2
90
32
Layer
Layer
Layer
Figure 5: Average attention to special tokens. Each point indicates the average attention weights of particular attention head pointing to specific token type. Each color corresponds to the attention flow to a specific tokens, e.g., [CLS]. [SEP] and separators (. or .). 'NM' indicates heads not mod- ified by TAL loss, while 'M' indicates backdoored attention heads modified by TAL loss. Among clean models (left), Attn-Synble dirty-label attacked models (middle) and Attn-Synbkd clean-label attacked models, we can not easily spot the differences of the attention flow between backdoored models and clean ones. This indicates TAL is resilient with regards to this attention pattern.
4.4 RESISTANCE TO DEFENDERS
We evaluate the resistance ability of our TAL loss with two defenders: ONION (Qi et al. 2021a), which detects the outlier words by inspecting the perplexities drop when they removed since these words might contain the backdoor trigger words; and RAP (Yang et al. (2021b), which distin- guishs poisoned samples by inspecting the gap of robustmess between poisoned and clean samples. We report the attack performances for inference-time time defense in Table E In comparison to each individual attack baselines, our loss can still achieve pretty good attack performances, especially un- der clean-label attack scenario. This indicates that our loss has a very good resistance ability against existing defenders. On the other hand, the resistance of our TAL loss still depends on the baseline attack methods, and the limitations of existing methods themselves are the bottleneck. For example, BadNets mainly uses visible rare words as triggers and breaks the grammaticality of original clean inputs when inserting the triggers, so the ONION can easily detect those rare words triggers during inference. Therefore the BadNets-basee attack performs not good on the ONION defenders. But for AddSent-based, Stylebkd-baseen or Synbkd-based attacks, both ONION and RAP fail because of the invisibility of attackers' data poisoning manners.
Table 3: Attack performances under defenders with poison rate 0.01 on SST-2, (Refer Table for the attack performances without defenders.)
ONION
RAP
Defender/
Dirte-Label
Clean-Label
Dirty-Label
Clean-Label
Attacker
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
BadNets
0.143
0.869
0.224
0.860
0.999
0.910
0.228
0.900
Attn-BadNets
0.155
0.876
0.161
0.876
1.000
0.914
1.000
0.912
AddSent
0.988
0.869
0.598
0.868
0.999
0.912
0.564
0.908
Attn-AddSent
0.993
0.866
0.982
0.874
1.000
0.903
0.999
0.910
Stylebkd
0.633
0.875
0.423
0,854
0.626
0.914
0.400
0,894
Attn-Stylebkd
0.710
0.850
0,514
0,842
0.683
0.901
0.484
0.885
Synbkd
0.623
0.870
0.426
0.852
0.601
0.912
0.385
0.896
Attn-Synbkd
0.646
0.870
0.469
0.852
0.643
0.916
0.418
0.896
5 CONCLUSION
In this work, we investigate the attack efficacy of the NLP backdoor attacks. We propose novel Trojan Attention Loss (TAL) to enhance the Trojan behavior by directly manipulating the attention patterns. Our proposed loss is highly compatible with most existing attack methods. Experimental results validate that our method significantly improves the attack efficacy; it achieves a successful at- tack within fewer training epochs and with much smaller proportion of poisoned samples. It easily boosts attack efficacy for not only the traditional darty-label attacks, but also the more challenging clean-label attacks. Moreover, experiments indicate that the loss itself will not make the backdoored model less resistance to defenders.
For defenses against the attack baselines, similar defense results also verified in Cuu al. 2022

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure the reproducibilityy of our work, which is strongly encouraged by ICLR https: /7iclr. cc/ /onfeerencee/2223/ AuthorCuide), we reference the parts of the main paper, appendix and supplemental materials that helps to reproduce our results. Besides the main paper, we also introduce our experimental settings in Section 441 as well as Appendix A..3 supple- mental materials, include the core codes for Irentioo-Guider Attacks and Defense, with instructions in README .md file.
REFERENCES
Arieh Ben-Naim. farewell entropy: Statistical thermodynamics based information: S. Woolld Scientific, 2008.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.0860 2017.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, 2019.
Robby Costales, Chengzhi Mao, Raphael Norwitz, Bryan Kim, and Junfeng Yang. Live trojan attacks on deep neural networks. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 796-797, 2020.
Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. A uni- fied evaluation of textual backdoor learning: Frameworks and benchmarks. arXiv preprint arXiv:2206.08511 08514, 2022.
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against Istm-based text classifica- tion systems. IEEE Access, 7:138872-138878, 2019.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI con- ference on web and social media, volume 11, pp. 512-515, 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), 2019.
Leilei Gan, Jiwei Li, Tianwei Zhang, Xiaoya Li, Yuxian Meng, Fei Wu, Shangwei Guo, and Chun Fan. Triggerless backdoor attack for nlp tasks with clean labels. arXiv preprint arXiv:2111.079700 2021.
Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. Can adversarial weight pertur- bations inject neural backdoors. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 2029-20333 2020.
T Gu, B Dolan-Gavitt, and SG BadNets. Identifying vulnerabilities in the machine learning model supply chain. In Proceedings of the Neural Information Processing Symposium Workshop Mach. Learning Security (MLSec), pp. 1-5, 2017a
Tianyu Gu, Brendan Dolan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint 1rXiv:1708.067..0 2017b.
Wei Guo, Benedetta Tondi, and Mauro Barni. An overview of backdoor attacks against deep neural networks and possible defences. arXiv preprint arXiv:2111.08429 2021.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Self-attention attribution: Interpreting information interactions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 129711 2021.
10

---

Under review as a conference paper at ICLR 2023
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pp. 1875-1885, 2018.
Tianchu Ji. Shraddhan Jain, Michael Ferdman, Peter Milder, H Andrew Schwartz, and Niranjan Bal- asubramanian. On the distribution, sparsity, and inference-time quantization of attention values in transformers. arXiv preprint arXiv: 10.:0/0000/3333 2021.
Kalpesh Krishna, John Wieting, and Mohit lyyer. Reformulating unsupervised style transfer as paraphrase generation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 737-762, 2020.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp- 2793-2808 2020.
Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks pre-trained models by layerwise weight poisoning. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 3023-3032, 2021.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning A survey. IEEE Transac-
tions on Neural Networks and Learning Systems, 2022.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack neural networks. 2017.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907v 2019.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pp. 182-199. Springer, 2020a.
Yuntao Liu, Ankit Mondal, Abhishek Chakraborty, Michael Zuzak, Nina Jacobsen, Daniel Xing. and Ankur Srivastava. A survey on neural trojans. In 2020 21st International Symposium on Quality Electronic Design (ISQED), pp. 33-39. IEEE, 2020b.
Weimin Lyu, Songzhu Zheng, Tengfei Ma, and Chao Chen. A study of the attention abnormality in trojaned berts. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Long Papers), pp. 4727-4741, 2022.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems, 33:3454-3464, 2020.
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 9558-9566, 2021a.
Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li. Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 4569 4580, 2021b.
11

---

Under review as a conference paper at ICLR 2023
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 443-453, c021c.
Fanchao Qi, Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. Turn the combination lock: Learnable textual backdoor attacks via word substitution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 Ith International Joint Con- ference on Natural Language Processing ( (Volume 1: Long Papers), pp. 4873-4883, 2021d.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor at- tacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11957- 11965, 2020.
Ahmed Salem, Yannick Sautter, Michael Backes, Mathias Humbert, and Yang Zhang. Baaan: Backdoor attacks against autoencoder and gan-based machine learning models. arXiv preprint arXiv:2010.030077 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint axXiv1910.01108, 2019.
Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor pre-trained models can transfer to all. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 3141-3158, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information 2017.
processing systems, pp.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, 2019.
Jie Wang, Ghulam Mubashar Hassan, and Naveed Akhtar. A survey of neural trojan attacks and defenses in deep learning. arXiv preprint arXiv:2202.09 00102007733.. 083, 2022.
Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y Zhao. Backdoor attacks against deep learning systems in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6206-6215, 2021.
Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in nlp models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2048 2058, 2021a.
Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. Rap: Robustness-awan perturbations for defending against backdoor attacks on nlp models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8365 83811 2021b.
Wenkai Yang, Yankai Lin, Peng Li. Jie Zhou and Xu Sun. Rethinking stealthiness of backdoor attack against nlp models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference Natural Language Processing ( \Volume I: Long Papers), pp. 5543-5557, 2021c.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character- convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
12

---

Under review as a conference paper at ICLR 2023
Xinyang Zhang. Zheng Zhang, Shouling Ji, and Ting Wang. Trojaning language models for fun and profit. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 179-197. IEEE, 202
Zhengyan Zhang, Guangxuan Xiao, Yongwei Li. Tian Lv, Fanchao Qi. Zhiyuan Liu, Yasheng Wang, Xin Jiang, and Maosong Sun. Red alarm for pre-trained models: Universal vulnerability to 2021b.
neuron-level backdoor attacks. arXiv preprint arXiv:2101.0663
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean- label backdoor attacks video recognition models. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14443-14452, 2020.
A APPENDIX
A.1 GENERALIZATION ABILITY
In this section, we show that our methods have good generalization ability. We explore the attack efficacy on four transformer models (e.g.. BERT, RoBERTa, DistilBERT, and GPT-2) with three NLP tasks (e.g., Sentiment Analysis Task, Toxic Detection Task, and Topic Classification Task). By comparing the differences between attack methods with TAL loss (Attackers name Atm--x and without TAL. loss (Attackers name x), we observe consistently performance improvements under different transformer models and different NLP tasks.
Ablation Study Settings. We follow the experimental settings in Section 4.1 We also verify our methods additional transformer models and NLP tasks. Besides BERT Dey al. 2019),
experiment on other pre-trained language models, namely RoBERTa Liu et al. 2019 DistilBERT We implement backdoor attacks toxic detection task on HSOL (Davidson et al. 2017) dataset and topic classification task on AG's News (Zhang al. 2015) dataset. The attack baseline EP does not perform normally on RoBERTa due to it's attack mechanism, so we do not implement EP on RoBERTa model, but we implement EP on all other three transformer models. For topic classification task, we only experiment on challenging setting clean-label attack scenario.
(Sanh et al. 2019 and GPT-2 (Radford al. 2019 10
Attack performance. Table R and Table we report the attack efficacy under challenging setting poison rate 0.01. Many existing attack baselines are not able to achieve high ASR under this setting, not to mention under the clean-label attack scenario. Our TAL loss significantly boosts the ASR most of the attacking baselines different transformer models with different NLP tasks. We also show the trend of ASR with the change poison rates, we conduct experiments under poison rate 0.01 and 0.2 with different transformer models and different NLP tasks. The results are presented in Figure different poison rates.
and 12 We observe consistent improvements under
A.2 CHOICE OF HYPER-PARAMETER H
We conduct ablation study to verify the relationship between the ASR and the choice hyper- parameter H, i.e.the the number of backdoored attention heads, in Eq Figure 13 shows that the number of backdoored attention heads is robust to the attack performances.
A.3 IMPLEMENTATION DETAILS
When implementing the backdoor attacks. we train the model for 50 epochs. The batch size on SST-2 is 64, and IMDB is 4. When computing the performance, we chose the average value of three models.
"The pre-trained RoBERTa is downloaded from http://hhugggnggfc..eee//ooe/ee/
Thhe pre-trained DistilBERT
downloaded from https::/huuginggfantt
dist Tbert- ba sersse-unne uncased
"hhe pre-trained GPT-2 downloaded from settps://hugginggfee co/gpt2
13

---

Under review as a conference paper at ICLR 2023
Table 4: Attack efficacy with different transformer models (e.g., BERT, RoBERTa, DistiBBRRT, GPT-2) and NLP tasks (e.g. Sentiment Analysis, Toxic Detection), We report the attack perfor- mances under challenging setting poison rate 0.01. The attack baseline EP is not compatible with RoBERTa model due to EP's attack mechanism, so we skip
Modeh
BERT
RobERia
PGGBBRRT
GFT.2
Tasks
Attackers
ASR
Deriy-Labed
CMCC
ASR
Chon-LLbel
Dirry-Label
CSCC
ASR
Clean-Tahh
CSCC
Dirty-Labd CSCC
ANR
CSCC
A3K
Dirty-Labed
CNCC
ASR
CSCC
abel
BadNeets
0:215
01351
01174
11355
Attu-BadNeds
1.00
1,000
0.912
0.939
0.999
0.913
1,000
1,000
0.910
0.965
0.915
AddScnt
0.914
0.576
0.945
0.908
0.702
0.598
0.913
0,914
Alto-AddSent
1.000
0.912
1,000
0.913
0.933
0,910
1,000
0.505
1,000
0,909
0.992
0,914
SA
Atte- EP
EP
(1999
01986
0.911
0.906
0,835
0,995
0.914
0.915
1,000
1,000
0,904
0.911
0.999
0.538
0.503
0.114
0.967
0.982
0.917
0.913
0.897
0,911
0.911
Stylebld
0.912
0.384
0.501
0.926
0.939
0.356
0,936
0.556
0.888
0.339
0.896
03622
0.920
0.610
0.875
Attn Styhhhl
0.742
0.901
0.491
0.585
0.968
0.940
0.748
0.945
0.691
0.9%6
0.522
0.376
11531
0.901
0.702
0.583
Spubid
0.638
0.910
0.361
0.915
0.613
0.932
0.313
0.939
0.563
0.90]
0.243
0.294
11550
0.913
0.356
0,914
Atta-Staalld
0.673
0
0.439
0.895
0.683
0.934
0.411
0.916
0.664
0.900
0,413
0.90
11598
0.907
0.513
0,833
BadNNet
(.93)
0.957
6.124
0.944
1.000
0.955
01325
0.951
0.9%8
0:955
0.133
0.544
1.000
0951
0.112
0.913
Attu-BalNeess
1.000
0.955
1,000
0.956
1,000
0.956
0.992
0.930
1.000
0.955
1,000
0.55
1.000
0.51
0.7%%
0.954
AddSSent
1.000
0.955
0,100
0.945
1,000
0.954
0.120
0.952
1,000
0.955
0.300
0.533
1.595
0.54
0.696
0.578
Atto Adtn-AddSentt
1.030
0.955
1,000
0,957
1,000
0.954
0.953
0.953
1,000
0.955
1,000
0.556
1,000
0.95
0,862
0,557
Tode
Atta-RP
EP
0.999
(:930
0.955
0.953
0.769
0.702
0.954
0.985
1.000
1.000
0.057
0.055
0.781
0.997
0554
0554
1.91
1005
0.90
0.90
0.973
0.555
0.954
0.951
Stylebkd
0.547
0.951
0.303
0.951
0.662
0.53
0.15
0.051
0.902
0.03
0.366
0551
1133
0954
0.41
0.910
Atte-St Stylhal
0U673
0.942
0.403
0.939
0.680
0.51
0.426
0.941
0.630
0935
0.445
1933
11258
0.65
0.498
0.509
Syabkd
0.918
0.950
0.586
0.983
0.989
0.933
0.536
0.935
0.961
09966
0.685
1550
11975
0.53
0.531
0.954
Atta-Syabld
0.961
0.951
0.601
0.954
0.395
0.933
0.590
0.954
0.969
0.948
0.751
0355
0985
0.94
0.708
0,909
Table 5: Attack efficacy with topic classification task on a larger dataset AG's News (Zhngg et al. 2015). The experiment conducted under different transformer models (e.g., BERT, RoBERTa, DisTiIERRT, GPT-2) with poison rate 0.01 and under the clean-label attack scenario.
Models
BERT
RoBERTa
DistiBBERT
GPT-2
Clean-Label
Clean-Labe
Clean-Labh
Clean-Lan Label
Attackers
ASR
CACC
ASR
CACC
ASR
CACC
ASR
CACC
BadNets
0.868
0.943
0.923
0.944
0.717
0.940
0.672
0,946
Attn-BadNets
1,000
0.941
0.969
0.941
0.994
0.942
0.886
0.946
AddSent
0.594
0.943
0.749
0.946
0.915
0.940
0.683
0.946
Attn-AddSent
0.998
0.938
0.969
0.944
0.990
0.941
0.818
0.942
EP
0.920
0.939
0.899
0,940
0.138
0.939
Attn-EP
0.977
0.941
0.913
0.940
0.374
0.939
Stylebkd
0.141
0.942
0.584
0.946
0.169
0.942
0.263
0.944
Attn-Stylebkd
0.353
0.930
0.619
0.939
0.259
0.932
0.240
0.937
Synbkd
0.821
0.939
0.994
0.943
0.492
0.941
0.962
0.947
Attn-Synbkd
0.937
0.941
0.990
0.947
0.660
0,940
0.977
0,946
A.4 AVERAGE ENTROPY EXPERIMENTS
We provide experiments on the average attention entropy (check Section 4.3 Average Atlention Entropy.) among all other baselines with our TAL loss. The experiments results on different attack baselines are shown in Figure 14 We have observed the similar patterns as is illustrated in main paper the average attention entropy among clean models, baseline attacked models, AGA attacked models, maintain consistent pattern. Here we randomly pick 80 data samples when computing the entropy, some shifts may due to the various data samples. When designing the defense algorithm, can not really depend on this unreliable index to inspect backdoors. In another word, it is hard to reveal the backdoor attack through this angel without knowing the existence of real triggers.
A.5 ATTENTION TO SPECIAL TOKENS EXPERIMENTS
This section provides detailed experiments on the attention flow to special tokens (check Section Attention Flow Specific Tokens) among all other baselines with our TAL loss. In Figure 15 Figure [6] Figure an Figure 18, we observe the consistent pattern: our TAL loss resistance to the attention patterns (attention flow to specific tokens) without knowing the trigger information.
14

---

Under review as a conference paper at ICLR 2023
Experiments DistilBERT with Sentiment Analysis Task
Dirty-Label
Clean-Label
1.00
1.00
CACC
0.95
0.35
0.90
0.90
0.85
0.05
0.80
1.00
1.00
ASP
0.00
ASR
0.00
0.40
0.40
BadNets
Badliets
AddSient
AttnAddSent
AddSent
Altn-AddSent
AZtr-EP
0.20
Stylebkd
Styl-Stylebkdd
Stylebkd
Altn Stylebed
Synbiid
Arth-Synbkd
Synbkd
Attn-Synbkd
0.01
0.2
0.01
0.2
Poison Rate
Poison Rate
Figure 6: Attack efficacy with our TAL loss (Arm-x) and without our TAL loss (x). The experiment
conducted on DistilBERT with sentiment analysis task.
Experiments on GPT-2 with Sentiment Analysis Task Dirty-Label
Clean-Label
1.00
1.00
0.95
0.35
0.85
0.05
0.80
0.80
0.80
0.80
ASR
0.60
ASR
0.60
BadNets
Achn BadNets
Bodkets
Attn Bodletts
AttnAddSent
Attr-EP
Attn-EP
0.20
Stylebid
Attry Stylebkd
0.20
Stylebid
Attn Stylesdd
Synbied
Atti-Synbkd
Synbid
Attn-Syniiia
Poison Rate
Poison Rate
Figure 7: Attack efficacy with our TAL loss (Art-x) and without our TAL loss (x). The experiment conducted on GPT-2 with sentiment analysis task.
A.6 ATTACK EFFICACY UNDER HIGH POISON RATES
In this section, we conduct experiments to explore the attack efficacy under high poison rates. By comparing the differences between attack methods with TAL loss and without TAL loss, observe consistently performance improvements.
Attack Performances. We conduct additional experiments four transformer models to reveal the improvements of ASR under a high poison rate (poison rate = 0.9). Table O indicates that our methods still improve the ASR. However, under normal backdoor attack scenario, to make
15

---

Under review as a conference paper at ICLR 2023
Experiments on RoBERTa with Sentiment Analysis Task
Dirty-Label
Clean-Label
1.00
1.00
CACC
0.95
0.95
0.90
0.90
0.85
0.05
0.80
1.00
1.00
0.80
ASR
0.00
ASR
0.00
0.40
0.40
BedNers
Atr--eddNess
Bedkvtt
Altn-BedNees
AddSent
AttrvAddSent
AddSant
Attn AddSent
0.20
Stylebkd
Attn-Stylebidd
Stylebkd
Stylebkd
Synbii
Arth-Synbkd
Synbid
Synn-Sybbidd
0.01
0.2
0.01
0.2
Poison Rate
Poison Rate
Figure 8: Attack efficacy with our TAL loss (Arm-x) and without our TAL loss (x). The experiment
conducted on RoBERTa with sentiment analysis task.
Experiments on BERT with Toxic Detection Task Dirty-Label
Clean-Label
1.00
1.00
0.95
0.95
0.90
0.90
0.85
0.85
0.80
0.80
0.80
0.80
ASR
0.60
ASR
0.60
BadNets
Athn BadNets
Bodkett
Attn Bodliets
AlrsAddSent
AddSent
Att
Attn EP
0.20
Stylebid
Attn-Stylebldd
0.20
Stylebid
Sttn-Styleted
Synbied
Attrs Synbld
Synbid
Attin Synbid
Poison Rate
Poison Rate
Figure 9: Attack efficacy with our TAL loss (Arm--) and without our TAL loss (x). The experiment is conducted on BERT with toxic detection task.
the backdoored model also have very good performance on clean sample accuracy (CACC), most of the attacking methods do not use a very high poison rate.
The Trend of ASR with the Change of Poison Rates. We also explore the trend of ASR with the change of poison rates. More specific, we conduct the ablation study under poison rates 0.5, 0.7, 0.9, 1.0 sentiment analysis task on BERT model. In Figure 19 the first several experiments under poison rates 0.01, 0.03, 0.05, 0.1, 0.2, 0.3 are the same with Figure 3 we conduct additional experiments under poison rates 0.5, 0.7, 0.9, 1.0. Our TAL. loss achieves almost 100% ASR in
16

---

Under review as a conference paper at ICLR 2023
Experiments on DistilBERT with Toxic Detection Task
Dirty-Label
Clean-Label
1.00
1.00
0.95
0.35
0.90
0.90
0.85
0.85
1.00
1.00
0.80
ASR
0.00
ASR
0.60
0.40
0.40
BadNets
Badliets
AddSient
AttnAddSent
AddSent
Altn-AddSent
AZtr-EP
0.20
Stylebkd
Ittn-Stylebkd
0.20
Stylebkd
Altn Stylebed
Synbiid
Arth-Synbkd
Synbkd
Attn-Synbid
0.01
0.2
0.01
0.2
Poison Rate
Poison Rate
Figure 10: Attack efficacy with our TAL loss (Attn-x) and without our TAL loss (x). The experiment
conducted on DistilBERT with toxic detection task.
Experiments on GPT-2 with Toxic Detection Task Dirty-Label
Clean-Label
1.00
1.00
0.95
0.95
0.90
0.90
0.85
0.85
0.00
0.00
0.80
0.80
ASR
0.60
ASR
0.60
0.40
BadNers
Att--adNeess
Rodivets
Attn-Badivets
AddSent
Atth AddSent
AddSent
Attn Addsent
0.20
Stylobid
Artn Stylebkd
0.20
Styleblid
Attn Styleaka
Synbidd
Sttn-yynbld
Synbkd
Aytn-Symbka
Poison Rate
Poison Rate
Figure 11: Attack efficacy with our TAL loss (Attn-)) and without TAL. loss (x). The experiment is conducted on GPT-2 with toxic detection task.
BadNets, AddSent, and EP under all different poison rates. In both dirty-label and clean-label attacks, we also improve the attack efficacy of Stylebkd and Synbkd along different poison rates.
A.7 RESISTANCE TO DETECTIONS
We evaluate our TAL loss with the detection method AttenTD Lyu et al. 2022), which analysis the abnormal attention behavior in backdoored models. We conduct the ablation study sentiment analysis task with BERT models, with poison rate 0.2. And we evaluate six backdoored models with AttenTD. The results shown in Table 7 Our TAL loss does not increase the risk of being
17

---

Under review as a conference paper at ICLR 2023
Experiments on RoBERTa with Toxic Detection Task
Dirty-Label
Clean-Label
1.00
1.00
0.95
0.35
0.90
0.90
0.85
0.85
1.00
1.00
ASP
0.00
ASR
0.60
0.40
0.40
BedNeis
Atr--eddNess
Bedivet
Altn-BedNees
AddSent
AttrvAddSent
AddSant
Attn AddSent
0.20
Stylebkd
Attn-Stylebidd
Stylebld
Stylesdd
Synbii
Arth-Synbkd
Synbid
Synn-Sybbidd
0.01
0.2
0.01
0.2
Poison Rate
Poison Rate
Figure 12: Attack efficacy with our TAL loss (Attn-x) and without our TAL loss (x). The experiment
conducted on RoBERTa with toxic detection task.
ASR with Different Backdoored Attention Head Number (Dirty-Label Attack)
1.00
0.95
0.90
0.85
0.80
0.75
ASR with Different Backdoored Attention Head Number (Clean-Label Attack)
1.00
0.95
ASR
0.90
ATTN-BodNetts
0.85
ATTN-AddSent
ATTNEP
0.8
ATTN-Stylebad
Synbid
0.75
Atfention Head Number
Figure 13: Ablation study on syper-parameter. number of attention head in Eq 3 Attack perfor- mances do keep robust when poisoning different number of attention heads with TAL loss.
detected. Furthermore, we observe that AttenTD works well on simpler attacks such as BadNets, Addsent, and EP (100% accuracy of detection). Meanwhile, for stealthier attacks such Stylebkd and Synbkd, the detection performance of Atten'DD deteriorates.
18

---

Under review as a conference paper at ICLR 2023
BedNet: ATTI--Bddeee
AddSent ATTN-AdeSent
14
25
Claan Model AddSent AddEetd
BadNett
BadNes
Model
155T-2)
(55T-2)
(55T-2)
(55T-2)
ATTN-E?
lebkd ATTNN-tyleelka
2.
Mocel
25
Model Stylebkd ATTN Stylebid
ATTN EP
Clean Model
Clean Cleen-Label Label (S5T-2)
Model
Label (SST.2)
(SST-2)
(SST-2)
Figure 14: Average attention entropy experiments on attack basclines and ATTN-Integrated attack baselines.
Clean
BADNETS (abely
ATTN BADNETS (Cban)
Ki
10.5URR
SER)ING
0.8
Dogg Att
0.5
(900
fwng Aftes
0.6
0.4
Figure 15: Average attention to special tokens. Backdoored model with Attn-BadNets.
Clesn
ADDSN-ADDSENT (Dirty-aabe))
ADOSENT Label)
Kis BET
J2.5)MP
0.5
CLSS3:
0.7
0.6
DEP191
(110)
0.6
1100
Aug. Attn
APR
0.4
0.0
0.0
0.0
20
12
10
10
32
Layar
Layer
Layar
Figure 16: Average attention special tokens, Backdoored model with Attn-AddSent.
(DirtyLabe))
KLLT
XC.53RE SET11e (49g71
0.8
Bung Act
0.6
bing Arth
Layer
Figure 17: Average attention special tokens. Backdoored model with Attn-EP.
19

---

Under review as a conference paper at ICLR 2023
Clean
STTN-STYLEBKD (Dirty-Labe))
KSSIOM ATTN-STYLEBKO Clean-Labll
0.8
SEP
Avc. Attn
R.c
Avg Aftn
0.4
0.4
Leyer
Layer
Layer
Figure 18: Average attention special tokens. Backdoored model with Attn-Stylebkd.
Table 6: Attack efficacy with poison rate 0.9, with TAL loss and without TAL loss. The experiment conducted on the sentiment analysis task.
Models
BERT
RoBERTa
DiSBISERT
GPT-2
Attackers
ASR
Labey-Labll
CACC
ASR
Clean-Label
CACC
ASR
Dirty-Laban
CACC
ASR
Clean-Labe
CACC
ASR
Dariy-Label
CACC
ASR
Clean-Label
CACC
ASR
Dirg-Labet
CACC
ASR
Lleas-Late
CACC
RadNets
1,0000
0.301
1
1.000
(1331)
11502
0.503
AddSent
Att-AAdSentt
0.500
1.000
1.000
0.915
0.995
0.007
0.999
0912
0.913
Attn-EP
1.000
0.916
0.999
0.915
1,000
0.902
0.966
0.588
0.999
0.14
0.700
0.909
Stylebkd
1.000
0.500
0.841
0.691
1,000
0,500
0.98
0.501
1,000
0.500
0.861
0.716
1,000
0.501
0.9%%
01501
Sty-Seebebied
1.000
0.409
0.875
0.729
1.000
0.500
0.900
582
1,000
0.500
0.904
704
0000
0.499
0.999
0,500
Symbkd
1.000
0.500
0.981
0.557
1.000
0.500
0.971
0.610
1,000
0.500
0.983
0.534
1.000
0.500
0.966
0.566
Min-Synnbld
1.000
0.499
0.982
0.536
1,000
0,500
0.963
0.565
1,000
0.899
0.988
0.525
1,000
0.500
0.992
0.552
Trend of Attack Efficacy with the Change of Poison Rates
Dirty-Label
Clean-Label
1.00
1.00
CAGC
0.80
CACC
0.80
0.60
0.60
0.40
0.40
0.90
0.90
0.80
0.80
0.70
0./0
ASR
ASR
0.60
0.60
0.40
BedNees
Att-dedNers
0.40
Bedivets
Attn-Bade Badikts
AddSent
Attn AddSent
AddSent
Attn AddSent
Attr-E
Atti-EiP
0.30
Stylebid
Actn Stylebkd
0.30
Stylebli
Attn Stylenkaa
Synbid
Synbkd
Synbkd
Symbka
0.20
0.3
0.5
0.7
0.20 0.03
0.2 0.3
0.5
Poison Rate
Poison Rate
Figure 19: Attack efficacy with our TAL loss (Atin-x) and without TAL loss (x) under different poison rates. Under almost all different poison rates and attack baselines, our Trojan attention loss improves the attack efficacy both darty-label attack and clean-label attack scenarios. Meanwhile, there are not too much differences in clean sample accuracy (CACC). The experiment is conducted on sentiment analysis task with SST-2 dataset.
Table Detection performances on sentiment analysis task with BERT models, under poison rate
0.2.
Attuckers
BodNets
Attn-BadNets 100.0
AddSent 1004
Attn-AddSent 100D
EP
Attn-EP 1000
Stylebid 66.7
Attn-Sttychkdd 66.7
Synbld 66.7
Attn-Syahkd
ACC(%)
1000
180.
20


Reviews:
[{"review_id": "ywFIIKrzAs2", "reviewer": "Reviewer_qVRi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the attention distribution of an attacked Transformer models. Based on their findings, the authors propose a trojan attention loss to enhance the attack efficiency. The authors conducted a comprehensive study on this method, showing the effectiveness of their approach.", "summary_of_the_review": "This paper present a novel and well-motivated method to improve the attack efficiency. The results are promising, along with lots of analysis. Overall, I would recommend accept this paper.", "strength_and_weaknesses": "Strength:\n- The method is well motivated by the study on the attention distribution\n- The designed loss is simple and easy to be used in many existing methods.\n- The experiments are comprehensive.\n\nWeakness:\n- The proposed method is designed for white-box attack. Is it possible to extend this method to black-box attack?\n- The backbone is only vanilla BERT. It would be better if the authors can show results on other pre-trained Transformers.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tMBU5K3PCLF", "reviewer": "Reviewer_Xodn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper finds that when backdoors exist in transformer models, particularly BERT, attention concentrates more on trigger tokens compared to clean tokens. They leverage this observation to improve the sample efficiency of backdoor attacks by adding an explicit loss term that encourages this observed behavior. Empirically, this leads to almost equally strong attacks (esp. in the clean-label attack case) with (1) less data and (2) without giving out information about the backdoor triggers.", "summary_of_the_review": "The paper makes a niche contribution to improve attack efficacy but over states the generalizability of their results to transformer models with specific experiments [eg. the title itself talks about (all kinds of) transformers].\n\n--- AFTER REBUTTAL & CHANGES ---\n\nGiven this is primarily an empirical paper, I do believe the addition of results on other NLP base models like RoBERTA, DistilBERT, GPT-2 adds a bit more confidence on the efficacy of the TAL function.\n\nHaving said that, the authors bypass the more challenging questions on (1) detection of these attacks due to lack of knowledge about triggers and (2) TAL being the first-order approach that comes to mind given the findings about the attention scores. While the authors claim that their methods give more insights into the attention mechanism in such scenarios, they are not able to propose ways to detect these attacks. This also keeps my ethics concerns in place.", "strength_and_weaknesses": "### Things that I liked\n\n- The paper is well written making it easy to understand.\n\n- The authors did a good job at describing the related work and situating their contribution in that context.\n\n- The authors are able to show that adding the loss term improves the sample efficiency of the data-poisoning attacks.\n\n- The authors consider a set of emprirical analysis to ensure the additional loss term does not result in easier detectability. For this, they observe that the clean input accuracy and attention scores remain similar to a vanilla (i.e. non-attacked) model. Futher, they consider how these attacks perform against defenders.\n\n### Things that need improvement / clarity\n\n- Why was the TAL not as effective for dirty-label scenarios? As per Fig 3, it only improved efficacy of the two invisible attacks; my guess is that the other attacks are already pretty sample efficient in the dirty-label setting and thus, the scope of improvment is small. Given the limited scope, I am not sure the impact of the paper.\n\n- Since the experiments are all done on the BERT cased model and evalutated on sentiment classification, the generalizability of the observation and thereby efficacy of the proposed loss is of concern. Given this is an empirical paper, serveral unanswered questions remail.\n  1. What happens on other sentence-level classification tasks beyond sentiment classification where the overall sentence structure can play a major role? Does the observation that attention scores concentrate on triggers still hold?\n  2. What about token-level tasks such as NER where the token-specific outputs of the transformers are considered as opposed to the [CLS] token encoding? (I did understand that some attacks do consider structure but since they are eventually applied to the sentence classification problem, it doesn't directly answer this question.)\n  2. What happen on other BERT based models? Does size of the transformer model play a role-- DistillBERT (smaller size), RoBERTA (similar size), GPT-2 (larger)?\n\n- The detection of these attacks, esp. due to lack of knowledge about triggers, seem to be the more difficult problem. Beyond simply echoing this sentiment, the authors simply choose to solve the simpler problem of improving the attack efficiency.\n\n- Given the observation that attention scores are concentrated on triggers, the TAL loss function is the first-order thing that comes to mind. Is there a way to improve the loss function? Does it need to be applied to the attention scores for the encoder layers? Why does the loss randomly pick as opposed to developing an attention pathway backdoor in the model (are such backdoor more detectable that the random sampling of attention weights & applying TAL on them)?\n\n- [Minor] Needs a proof reading (some mistakes follow):\n   - \"The the attack efficacy is robust to..\" -> \"The attack's efficacy is robust to..\"\n   - \"our TAL loss will not arise the attention abnormality\" -> our TAL loss will not give rise to an attention abnormality\"\n   - \"because of randomness data samples\" -> \"because of randomness in data samples\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bVaXqzZlp3z", "reviewer": "Reviewer_5z1R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the backdoor attacks against transformers for NLP tasks.\nThe authors observe that backdoored transformer models (like BERT) have higher attention weights on trigger tokens.\nDue to this empirical observation,\nthe authors propose a new loss called Trojan Attention Loss (TAL) which further improves the attention weights of trigger words.\nWith experiments on different attack methods, the authors show that adding this loss term can enhance the efficiency of backdoor attacks with fewer data and fewer training epochs.\n", "summary_of_the_review": "This paper proposes a novel Trojan Attention Loss to improve backdoor attack efficacy.\nHowever, the method is based on empirical observation. So it is still unknown whether the proposed loss term also has a good generalization ability for other transformers and NLP tasks.", "strength_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to understand.\n2. The proposed loss term is simple and easy to follow. The observation is easy to understand, and the proposed method is intuitive.\n3. The proposed loss term is compatible with most attack methods.\n4. Experiments show adding their loss term can improve the attack efficacy (with fewer data and fewer training epochs).\n\n\nWeaknesses:\n\n1. The paper only considers BERT for experiments. It is a bit over-claimed since other transformer models are ignored. It is suggested to do experiments on other common transformer models like RoBERTA.\n\n2. The paper only considers sentiment analysis tasks for experiments. We expect more experiments on other NLP tasks. We believe these are quite significant to improve this empirical paper. Otherwise, it is difficult to evaluate the generalization of the proposed method and the impact of this paper.\n\n\n\nWriting Suggestions:\n\n1. It is better to add the explanations of Dirty-Label and Clean-Label in Section 4.1 like the caption of Table2.\n2. It is better to clearly state that Attn-x uses the proposed TAL loss while x does not in the experiment section.\n3. Figure 3: Drity-Label --> Dirty-Label", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U2AOeLEgAr", "reviewer": "Reviewer_XXQF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a trojan attention loss for backoor attack in transformer-based NLP models. In the process of training, the loss manipulate the model to give the trigger word a higher attention weight, so as to improve the effect and efficiency of backdoor attack. They conducted the experiments to show TAL works for current different types of attack methods, and boosts the efficiency in both dirt-label and clean-label attack.", "summary_of_the_review": "This paper proposes a trojan attention loss for backoor attack in transformer-based NLP models. Although it has confirmed its effectivenss in BERT model, I think the generalization of the proposed approach should be proved by other models and tasks.", "strength_and_weaknesses": "Strengths:\n- The TAL loss is simple but effective method that can be applied to most of existing attack methods.\n\n- The experiments show that TAL significantly increased the attack efficacy of clean-label attack with very small poison rate compare to current attack methods.\n\n- This paper is well-written and easy to follow.\n\nWeaknesses:\n- Why just evaluating on sentiment analysis? It seems that more tasks should be applied to demonstrate the effetiveness of the proposed approach. Furthermore, why only choose BERT pre-trained model, the proposed approach is also applicable to other pre-trained models?\n\n-  The setting of the experiment is not sufficient, because TAL loss focuses on the effectiveness of the attack. In my opinion, the trend of ASR with the change of poison rate can be explored but not only conduct experiments with a specific small poison rate. By comparing the difference between attack method with TAL and without TAL, we can see the effectiveness of TAL more intuitively.\n\n-  This work is inspired by the paper Lyu et al. (2022), which analyzed the abnormal attention behavior in Trojan model. However, this paper also proposed a method to detect trojan model based on model’s attention drift after inserting the trigger word. Theoretically, their method can detect the trojan model trained by your method very well. Why don't you try to use their method as defender in section 4.4?\n\n-  TAL enables the model to achieve a good attack effect with a small poison rate. However, based on the assumption that the attacker can access data and the training process, I believe that the performance of the attack is more important than the size of poison rate. Have you experimented with a high poison rate to verify that your method can still improve the performance of current attack method?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject