Moderate-fitting as a Natural Backdoor Defender for Pre-trained Language Models
Authors: Biru Zhu
Yujia Qin
Ganqu Cui
Yangyi Chen
Weilin Zhao
Chong Fu
Yangdong Deng
Zhiyuan Liu
Jingang Wang
Wei Wu
Maosong Sun
Ming Gu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Backdoor Defense
Pre-trained Language Models

Abstract: Despite the great success of pre-trained language models (PLMs) in a large set of natural language processing (NLP) tasks, there has been a growing concern about their security in real-world applications. Backdoor attack, which poisons a small number of training samples by inserting backdoor triggers, is a typical threat to security. Trained on the poisoned dataset, a victim model would perform normally on benign samples but predict the attacker-chosen label on samples containing pre-defined triggers. The vulnerability of PLMs under backdoor attacks has been proved with increasing evidence in the literature. In this paper, we present several simple yet effective training strategies that could effectively defend against such attacks. To the best of our knowledge, this is the first work to explore the possibility of backdoor-free adaptation for PLMs. Our motivation is based on the observation that, when trained on the poisoned dataset, the PLM's adaptation follows a strict order of two stages: (1) a moderate-fitting stage, where the model mainly learns the major features corresponding to the original task instead of subsidiary features of backdoor triggers, and (2) an overfitting stage, where both features are learned adequately. Therefore, if we could properly restrict the PLM's adaptation to the moderate-fitting stage, the model would neglect the backdoor triggers but still achieve satisfying performance on the original task. To this end, we design three methods to defend against backdoor attacks by reducing the model capacity, training epochs, and learning rate, respectively. Experimental results demonstrate the effectiveness of our methods in defending against several representative NLP backdoor attacks. We also perform visualization-based analysis to attain a deeper understanding of how the model learns different features, and explore the effect of the poisoning ratio. Finally, we explore whether our methods could defend against backdoor attacks for the pre-trained CV model. The codes are publicly available at https://github.com/thunlp/Moderate-fitting.

Moderate-fiting as a Natural Backdoor Defender for Pre-trained Language Models
Biru Zhu Yujia Qin Ganqu Cui?, Yangyi Chen Weilin Zhao?, Chong Fu",
Yangdong Deng's
Zhiyuan Liu Jingang Wang Wei Wu', Maosong Sun?, Ming Gu School of Software, Tsinghua University, Beijing, China
Department of Computer Science and Technology. Tsinghua University, Beijing, China University of Illinois Vrbaan-Champaiin
4 Zhejiang University, Hangzhou, China
Meituan, Beijing, China
(zbr19, qyj20)0maill tsinghua. edu. {dengyd, liuzy] (aaeey)@ssnnghue edu, cn
Abstract
Despite the great success of pre-trained language models (PLMs) in a large natural language processing (NLP) tasks, there has been growing concern about their security in real-world applications. Backdoor attack, which poisons small number of training samples by inserting backdoor triggers, is typical threat to security. Trained on the poisoned dataset, victim model would perform normally on benign samples but predict the attacker-chosen label on samples containing pre-defined triggers. The vulnerability of PLMS under backdoor attacks has been proved with increasing evidence in the literature. In this paper, present several simple yet effective training strategies that could effectively defend against such attacks. To the best of our knowledge, this is the first work to explore the possibility backdoor-fres adaptation for PLMs. Our motivation based on the observation that, when trained on the poisoned dataset, the PLM's adaptation follows strict order of two stages: (1) moderate- fittiniitn stage, where the model mainly learns the major features corresponding to the original task instead of subsidiary features of backdoor triggers, and (2) an overfitting stage, where both features are learned adequately. Therefore, if could properly restrict the PLM's adaptation the moderate-fitting stage, the model would neglect the backdoor triggers but still achieve satisfying performance on the original task. To this end, design three methods to defend against backdoor attacks by reducing the model capacity, training epochs, and learning rate, respectively. Experimental results demonstrate the effectiveness of our methods in defending against several representative NLP backdoor attacks. We also perform visualization-batt based analysis to attain deeper understanding of how the model learns different features, and explore the effect of the poisoning ratio. Finally, we explore whether our methods could defend against backdoor attacks for the pre-trained CV model. The codes are publicly available at homm//hhnnp/Mooerttee-ff http///odertteeiiieeeeee fitting
https: //github.
trub.bb.... com http//hhnllppMen
1 Introduction
Despite the rapid development and great success of pre-trained language models (PLMs) (Han et al.) 2021). there have been increasing security concerns deploying them in real-world applications.
Indicates equal contribution.
t Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Backdoor attack by poisoning the training data is one typical security threat. The idea is that attackers insert backdoor triggers into certain training samples. After being trained on poisoned data, the victim model would (1) behave normally on the benign inputs, but (2) predict the attacker-chosen label given an input containing pre-defined triggers during inference. Initially introduced in the field of computer vision (CV) (Gu et al. 2017 Chen et al. (017)) diverse backdoor attack methods have been proposed in natural language processing (NLP), including inserting context- -indeeedidde words (Kurita et al. 2020) or sentences (Dai et al. 2019) into training samples and modifying the syntactic structure (Qi ct al. 202 or text style (Qi et al. 202 Empirical results reveal that even pre-trained language models (PLMs), the current foundation infrastructure for NLP (Min et al. 2021). are highly fragile to these attacks. Such a vulnerability poses a significant threat to real- lefe- applications of PLMs.
In this paper, we show backdoor attacks can be easily defended against using simple training strategies. Specifically, we focus on the following setting: (1) the attacker poisons the training data and releases the poisoned training dataset on open-source platforms. The attacker does not control the model training process: (2) the victims download the poisoned training dataset from the open-source platform to train their own models. If no defense is applied, the victim will get model injected with backdoors. Instead, with our proposed defense method, the risks of backdoor attack can be significantly mitigated even if the model is trained on the poisoned dataset.
Our motivation is based on previous findings (Täneer et al.. 2022) that during the PLM's adaptation to a specific dataset, the model would first learn the general general-purpose purpose patterns shared by most of the training samples, and then memorize those uncommon patterns of noisy samples. Under the setting of backdoor attack, partially poisoned dataset consists of both general-purpose patterns (major features) derived from most clean samples and backdoor triggers (subsidiary features) injected by attackers, and both features are highly distinct. For a PLM trained on the poisoned dataset, we observe that the model's s learning dynamics follow a strict order of two stages: (1) moderate-fitingg stage, where the model mainly learns the major features instead of subsidiary backdoor triggers, and (2) overfitting stage, where the model learns both features adequately.
Based on our observation, if we could properly restrict the PLM's adaptation to the moderate- fitting stage, the model would neglect the malicious backdoor triggers but still achieve satisfying performance on the original task. To this end, we propose three methods to limit the PLM's s adaptation to the moderate-fitting stage, i.e., reducing the model capacity, training epochs, and learning rate during training. Specifically, to decrease the model capacity, we resort to parameter-efficien tuning (PET), also known as delta tuning (Ding et al., (022), which optimizes small number of parameters instead of all the PLM's parameters. We additionally apply flexible low-rank reparameterization network to existing PET algorithms to further reduce the model capacity. addition, we demonstrate that reducing the training epochs or learning rate also suffices for our goal of restricting the PLM's s adaptation to the moderate-fitting fitting stage.
In experiments, we validate the feasibility of the proposed methods in defending against a series of representative backdoor attacks on several NLP tasks. The results reflect that, each proposed method could significantly lower the attack success rate (ASR) of the trained model while slightly hurting the performance of the original task on benign samples. This demonstrates that our methods are effective to defend against backdoor attacks in the training stage. To gain deeper understanding, we visualize the model's learning dynamics on the poisoned dataset under different model capacity. Furthermore, we conduct analyses to explore the effect of the poisoning ratio. We also create synthetic dataset to show that the two-stage learning phenomenon commonly exists in the scenario where PLMs jointly learn two distinct features that are imbalanced in the training data. Finally, we explore whether our methods could defend against backdoor attacks for the pre- are-trained CV model. In general, our study offers simple yet effective backdoor defense methods and uncovers the underlying mechanism of the backdoor attack in NLP.
2 Related Work
Backdoor Attack in NLP. One typical way of the backdoor attack is training data poisoning. By poisoning the training samples with malicious triggers, backdoor attackers aim to establish a strong connection between the trigger and the attacker-chosen target label (Chen et al. (017). Plenty of algorithms for training data poisoning have been proposed in the field of CV, such as inserting patch triggers (Gu et al., 2017) and blending benign samples with specific patterns (Chen et al. 2017).
2

---

Recently, many poisoning methods that insert triggers into the training data have also emerged in NLP, such as word-level (Kurita al. 2020) and sentence-level (Dai et al. 2019) triggers. However, these methods typically leverage unusual words or sentences that are uncorrelated to the context. These triggers break the grammar rules or coherence of normal text and could be easily detected in the inspection process (Qi et al., 2021c). In light of this, researchers recently proposed more invisible backdoor triggers, such as modifying the syntactic structure (Qi et al. 2021c) and transferring the text style QQ et al. 2021b).
The advance of backdoor attacks spawned a line of backdoor defense algorithms. The first category is examine the training samples and remove the poisoned ones. For instance, Chen and Dai 2021) propose Backdoor Keyword Identification (BKI) to remove suspicious samples that contain potential backdoor keywords from the training dataset. The second category is to inspect the testing samples. Qi et al. (2021a) detect and remove the words that may be the backdoor trigger from testing samples using outlier word detection; Yang et al. (202T) distinguish the poisoned data from clean data based on robustness-awe aware perturbations during the inference stage. The third category distinguishes trojaned models from clean models (Azizi et al.) [02T]] [iu et al., [0225) [Shen] eer al. (202). Orthogonal to the above methods, our methods do not require additional operations on data or model and thus are more practical.
Backdoor Defense in NLP.
Learning Behavior of DNNs Noisy Data. Many efforts have been spent exploring the learn- ing behavior of DNNs when the training data contains label noise. /hhng et al. (202 demonstrate that DNNs are skilled in fitting random labels, showing excellent memorization abilities. |Arit et al.)
(017)) further reveal that DNNs tend to prioritize learning simple patterns before memorizing the noisy data. As the foundation model for NLP, PLMs also demonstrated adept in memorizing the training data (Carlini et al.) 2021 2022) and recalling the learned facts (Petroni et al. 2019). Tänzer al.] (2022) additionally analyze the learning dynamics of PLMs under label noise, and observe distinct learning phases, i.e., PLMs first learn
purpose features and then enter a performance plateau, followed by rapidly memorizing label noise. Different from the above works, in this paper, we target at defending against backdoor attacks. Under our setting, for poisoned samples, the input data modified by inserting triggers, and the modified data's label set as the target label chosen by the attacker. When trained on the poisoned data, the model would find shortcut by establishing a connection between the backdoor features and the corresponding label.
Parameter-efficient Tuning. Fine-tuning, which optimizes all of the parameters, is the conventional way to adapt PLMs towards a downstream task. However, due to the tremendous size of PLMs, fine-tuning could bring huge computation and storage costs. In order to make feasible to tune larger PLMs, parameter-efficient tuning (PET) methods are proposed (Ding et al. 2022). PET tunes a small number of parameters and achieves comparable performance to fine-tuning. Up to now, various PET algorithms have emerged, including (1) inserting additional modules into the Transformer (Vaswani ct al., 2017) block (Houlsby et al./ 2019: Lester et al. 202 I Li and Liang] 202I). (2) optimizing part
of PLM's existing parameters (Ben-Zaken et al. 202 | Guo et al. 2021). ), and (3) reparameterizing
tunable parameters as low-rank decompositions (Hu al. [222]) In this paper, we demonstrate the
potential application of PET in backdoor defense by reducing the model capacity.
3 Methodology
In this section, achieve our goal of restricting the PLM's adaptation to the moderate-fitting stage, we design three training strategies during PL.M's downstream adaptation, by reducing the (1) model capacity, (2) training epochs, and (3) learning rate, respectively.
Reducing the Model Capacity. The traditional way for PLM's downstream adaptation is to optimize all the parameters the PLM (i.e., fine-tuning). Due to the tremendous size of tunable parameters and the complicated network architecture in Transformer block, the model capacity (complexity) of a PLM is inevitably hugd"] Although huge model capacity is typically considered as the guarantee for good memorization and representational ability (LeCun et al. 1998 Arpit ct al. 2017), recent works have demonstrated that for PLMs, most downstream NLP tasks have a
The model capacity is decided by the tunable parameters during PLM's downstream adaptation.
3

---

considerably small intrinsic dimension (Aghajanyan et al. 2021] Qin et al. 2021). which is defined as the minimal parameters needed upproximate some function or data distribution. This finding reveals that, for PLMs, a huge model capacity may not be the prerequisite for the successful downstream adaptation; instead, we could utilize small model capacity but still achieve excellent performance.
Parameter-efficient tuning (PET) serves as a possible solution to reduce the model capacity by significantly reducing the tunable parameters. However, in most cases, the extent of such reduction is still far from causing moderate-fitting The reason is that, the actual model capacity is not just decided by the number of tunable parameters, but also by the overall intrinsic rank (Hu et al. 2021) of the tunable parameters. Specifically, existing PET algorithms suffer from the following defects: (1) most PET algorithms (e.g., Adapter (Houlsby et al.) 2019) and Prefix-Tunine (Li and Liang 202Ib) do not have a low-rank constraint on the updates of the tunable parameters, but instead allow the intrinsic rank of weight updates larger than the actual intrinsic dimension of the downstream adaptation; (2) for those algorithms that explicitly model the low-rank updates of the parameters (e.g.. LoRA (Hu et al. 202T)), their low-rank structures are individually distributed in different modules in the Transformer layers (dubbed as local low-rank architecture). Instead, we argue that it is essential to have a global low-rank architecture to constrain the overall intrinsic rank of the weight updates.
To mitigate the above defects, we propose a flexible method that could be applied to PET algorithms. We reparameterize all of the tunable parameters defined by a PET algorithm with global low-rank decomposition. Without loss of generality, assuming a PET algorithm defines N tunable weights (and biases): (W..... 3 of various sizes, we first flatten each one into a one-dimensional vector. For example, a two-dimensional weight matrix W; E Idd xdj could be converted to a vector
V, E IRd., where dy di x dz and k)) = V. x di + k) for 0 S j < di and 0 S h < dz. After that, we concatenate all the flattened vectors as follows: Van = [Vi;...; Vn] E Rdd,, where dan = i= d' denotes the total number of tunable parameters. Finally, we reparameterize Van as low-rank decompositions. Specifically, the reparameterization network consists of two projection layers Proj, and Projz with activation function between them:
V Projg (a (Projj (E))),
(1)
where E Rdon is a tunable embedding. The layer Proj, projects E from dimension damb to the bottleneck dimension d,, creating vector I = Projj (E) R Note dj is set small (e.g.. 1) enough to ensure the low model capacity. The layer Proj, projects the dimension of I from dy to dan, and forms the tunable parameters Vall defined by the PET algorithm. During adaptation, instead of tuning Vall, we jointly optimize Proji, Proj, and E, and keep the PLM's parameters fixed.
Reducing the Training Epochs. Too many training epochs could cause the model to overfit the training data exhaustively. In our setting where the training data is partially poisoned with backdoor triggers, after enough training epochs, the PLM would eventually learn both the major and subsidiary features present in the data adequately. Inspired by Li et al. (2020) who contend that early stopping is robust label noise for over-paramann peraaaaeetteeeedd neural networks, we propose to reduce training epochs to prevent the PLM's 's adaptation from entering the overfitting stage. It is worth noting that similar defense techniques have also been proposed in other areas, such as Wallace et al. (202T); Liu et al.) (2022a).
Reducing the Learning Rate, Intuitively, training the PLM with a larger learning rate and fewer epochs is approximately equivalent to training the PLM with more epochs but smaller learning rate. Therefore, similar to the idea of limiting the number of training epochs, reducing the lcarning rate serves another simple yet effective method that could prevent the model from learning the poisoned samples adequately.
4 Experiments
In this section, we first conduct experiments to evaluate our proposed defense methods. We choose two representative NLP backdoor attacks as the main experiments ($..I). Then we analyze the learning dynamics of the model under different settings and visualize representative feature distributions learned during the training process in $ 4.2 Furthermore, we conduct systematic analyses to gain a deeper understanding of the effect of the poisoning ratio. Also, we show that the two-stage learning is

---

1oRA
Aciepler
Prefit-luning
100%
100%
100%
80%
I5ST-2)
ACCIVSR
NCCIV.R
ACC/ASS
ASR (S51-2)
(AS News)
40%
40%
40%
ACC (HS0L)
(HSOL)
20%
20%
20%
256
256
82
Bottileneck Dimension
Bottleneck Dimension
Bottleneck Dimension
Figure Results reducing the model capacity using reparameterized PET against word-level attack.
Acapter
PrefixTuning
100%
100%
100%
80%
80%
80%
ACC (SST-2)
ACCRSR
60%
ACC/03R
60%
ACCIASR
60%
ACC IAG News)
(AG News)
40%
40%
40%
ASR (HSOL)
20%
20%
20%
0%
256
32
2
09
256
32
056
23.6
32
4
Bettlenack Dimansion
Bottleneck Dimansion
Bottlanack Dimansion
Figure 2: Results of reducing the model capacity using reparameterized PET against syntactic attack.
a common phenomenon for PLMs in the scenario where PLMs jointly learn two distinct features that imbalanced in the training data. In addition, demonstrate that methods could successfully defend against other NLP backdoor attacks ($ 4.3). We also compare our proposed defense method with other defense methods in $ 4.4 Finally, explore whether our methods could defend against backdoor attacks for the pre-trained CV model ($ 4.5).
4.1 Main Experiments
Experimental Setting. We use RoBERTaEAS (Lu al. 2019) as the backbone PLM. We perform experiments on three datasets. including SST-2 (Socher et al. 2013), AG News (Zhang et al. 2015) and Hate Speech and Offensive Language (HSOL) (Davidson et al. 2017). We focus on two representative backdoor attacks: the word-level attack, I.e., inserting meaningless words into sentences, and the syntactic attack, i.e., using SCPN (lyyer et al. 2018) to perform the syntactic transformation, following Qi et al. (2021c). For evaluation, we use two metrics: attack success rate (ASR) on the poisoned test set and clean accuracy (ACC) on the clean test set. ASR measures what extent the model is attacked, and ACC measures how the attacked model behaves the original task. We evaluate three proposed methods, i.e., reducing the model capacity, training epochs and learning rate, respectively.
Reducing the Model Capacity. We choose three representative PET algorithms, i.e., LoRA, Adapter and Prefix-Tuning We reparameterize the tunable parameters of each PET algorithm as low-rank decompositions and experiment with different bottleneck dimensions, i.e., (256, 32, 4, 2, 1). The number of training epochs is set as 10. For reparameterized LoRA and Adapter, we the learning rate to 3 x 10-1 for both word-level attack and syntactic attack; for reparameterized Prefix-Tuning, we set the learning rate to 3 10-4 and 10 -4-4 for word-level attack and syntactic attack, respectively. The results are visualized in Figure (word-level attack) and Figure 2 attack), from which we observe that with the bottleneck dimension decreasing from large value (e.g. 256) to small value (e.g. 1), the ASR drops significantly, while the ACC only drops little, especially for the word-level attack. Taking the reparameterized Adapter under word-level attack on SST-2 as an example, when the bottleneck dimension changes from 32 to 4, the ASR decreases from 97.15% to 8.11% while the ACC declines from 94.51 % to 93.68%. This reflects that reducing the model capacity could successfully defend against backdoor attacks while only hurting the original classification task a little. addition, the above conclusion holds for all three PET algorithms, demonstrating that our rank reparameterization network is generic PET algorithms, We further
(syntactic (
5

---

Word-level Attack
Syriacticc Aliark
100%
100%
80%
ACCASR
ACC/ARR
ASR (55T:2)
ACC: (AG News)
40%
40%
ACC (HSOL)
ASR (H5OL))
20%
20%
0%
10
Epochs
Epochs
Figure 3: Results of reducing the training epochs against word-level and syntactic attacks.
Word-level Attack
Syntactic Attack
100%
100%
80%
ACC (SST-2)
ACCASR
60%
ACCASR
60%
ASR (551-2)
ASR (AG Newsi
ACC (HSDL)
20%
20%
0%
5x10--
2 10
5000- 10- 5x10-7
Learning
Learning
Figure 4: Results of reducing the learning rate against word-level and syntactic attacks.
show in appendix B.4 and appendix B.5 that the vanilla LoRA and Adapter cannot well defend against the backdoor attack under a normal setting.
Reducing the Training Epochs Learning Rate. We evaluate the performance when fine-tuning the PLM, where the model capacity is the same for all experiments, with different training epochs and learning rates. For the former, we set the training epochs to {10, 2, 1} and choose a learning rate of 2 x 10-5 for the word-level attack and 5 x 10 for the syntactic attack; for the latter, the learning rate is chosen from (2 10-5, 5 10 1 10-6 5 10-7} and the number of training epochs is set as 10. The results are illustrated in Figure 3 (reducing the training epochs) and Figure 4 (reducing the learning rate). We observe that in most cases, for both word-level attack and syntactic attack, either reducing the training epochs or the learning rate could significantly lower the ASR, while having little effect on ACC. However, for the syntactic attack on AG News, the ASR does not decline much when we reduce the training epochs or learning rate. The underlying reason may be that the syntactic triggers change the original samples syntactic triggers are easier for RoBERTaaase to learn during fine-tuning, making it harder to restrict the PLM's adaptation to the first moderate-fitting stage.
large extent for AG News so that the
We also perform experiments to investigate the effects of reducing the training epochs or learning rate when using PET algorithms. The results can be found in appendix B.3. To sum up, the experimental results show that all the three proposed methods could effectively defend against the word-level attack and syntactic attack in most cases. Using the above training strategies, the PLM's adaptation is restricted to the moderate-fittie are learned, while the backdoor triggers are not well learned. From the results, we also find that in some cases, merely decreasing the training epochs or the learning rate within a certain range using the fine-tuning method is not enough to restrict the PLM's 's adaptation to the moderate- fitting stage. In contrast, reducing the model capacity is more useful to some extent.
stage, where the primary features of the original classification task
4.2 Analysis
Visualization of the Learning Dynamics. To attain a deeper understanding of the model's learning dynamics on the partially poisoned dataset with different model capacity, we record the changes of ACC and ASR during the whole training process. Specifically, we choose the setting SST-2 under the word-level attack, and compare the learning dynamics when using reparameterized LoRA with large bottleneck dimension (256) and small one (1). The changes of ACC and ASR during training shown in Figure 5 (a, d). We observe that, when training with sufficiently large model capacity

---

100%
negative positive
negative posilive
80%
poisoned
poisened
ADCASR
60%
20%
ADC (55T-2)
ASR (5ST2)
01%
(b)
-00%S
80%
ACCIASS
60%5
ACC (SST-2)
ASR (5ST-2)
40%5
20%
regative
negelive positive paisoned
poisoned
50
(d)
(c)
Figure 5: Visualization of PLM's learning dynamics when using reparameterized LoRA with the bottleneck dimension of 256 (a, b, c) and 1 (d, e, f). We record the changes of ACC and ASR (a, d). Both ACC and ASR are tested with a fixed interval of training steps and at the end of model training. We also select two representative testing points (the green and black dashed lines in (a, d)) for both bottleneck dimensions and visualize the learned feature distributions in (b, f). (b)(() corresponds to the testing point of the green dashed line (a)/(d) (c)/(f) corresponds the testing point of the black dashed line in (a)/(d).
Table 1:
The influence of the poisoning ratio for word-level attack and syntactic attack. We reparameterized LoRA with the bottleneck dimension and evaluate both ACC and ASR under different poisoning ratios.
Word-level Attack
Syntactic Attack
Poisoning 2.5%
5%
7.5%
12.5%
12.5%
17.5%
20%
ACC, (SST:2)
92.53
92.64
92.42
92.48
90.66
91.98
91.93
91.82
91.60
91.65
ASR (SST-2)
10.20
10.96
11.51
11.62
14.47
42.11
48.36
53.84
61.62
65.90
(Figure (a)), both ACC and ASR become high after enough training, with ACC rising earlier than ASR. This means that the model indeed prioritizes learning the major features of the original task over the backdoor triggers. When training with limited model capacity (c)) after fluctuating for while, the ACC becomes high while the ASR remains low. The phenomenon means the model begins to learn features after short time, and would mainly learn the major features of the original task under limited model capacity, i.c., staying in the moderate-fitting stage.
Taking step further, select two representative testing points in the training process (two vertical dashed lines in Figure and (d), respectively). In addition to the negative and positive samples in the original development set, create poisoned samples by inserting triggers into the same negative samples. Then we visualize the learned features of the above three categories of samples using t-SNE (Van der Maaten and Hinton! 2008) in Figure (b, c, e,
When training with the sufficiently large model capacity, Figure 5 (b) shows that most negative samples' features and positive samples' features separated, while the poisoned samples' features are nearly overlapped with those negative samples. This means that the model learns the major classification task well, while the subsidiary backdoor triggers not well learned in the early stage. Figure (c) shows that both the major classification task and backdoor triggers are learned adequately after enough training. The four clusters of poisoned samples' features in Figure (c) correspond to four kinds of inserted trigger words (i.e., "cf", "mn", "bb", "tq"".
When training with limited model capacity. Figure (e), which corresponds to the green dashed line in Figure (d), shows that all three kinds of features are not well learned. Instead, the features visualized in Figure (f), which correspond to the black dashed line in Figure (d), are similar to
7

---

Table 2: Results on synthetic dataset with samples from SST-2 (the primary task) and AG News (the subsidiary task). We train RoBERTassse on the synthetic dataset, and evaluate the performance of both tasks under different model capacity, training epochs and learning rates.
Tuning Method
Reparameterized LoRA
Fine-tuning
Bottleneck Dimension
Truining Epochts
Learning Rute
Setting
32
5 10-8 8
10-6
10"7
SST-2
94.56
93.47
92
94.95
93.68
94.23 93.05
93.41 78.82
91.93
AG News
95.42
89.82
71.21
92.79
83.92
69.13
those of Figure (b) where the model is trained with limited time but large model capacity. This shows that the major classification task learned well, while the subsidiary backdoor triggers are not well learned, which explains the effectiveness of reducing the model capacity in backdoor defense. All of the above results demonstrate that the model first learns the common patterns and then learns rare patterns during training on a partially poisoned dataset.
Poisoning Ratio. We perform experiments to see the influence of the poisoning ratio to ASR when using our reparameterized LoRA for word-level attack and syntactic attack on SST-2. The number of training epochs is set as 10 and the learning rate is set as 3 x 10-4. The bottleneck dimension of the reparameterization network set as We experiment with different poisoning ratios. From the results in Table we can see that for both attacks, with the poisoning ratio increasing, the ASR gradually rises. Intuitively, the poisoning ratio can be seen as the proportion of subsidiary features, and larger poisoning ratio makes it easier for the model to learn backdoor features. Besides, we also observe that the ASR of the syntactic attack is higher than that of the word-level attack under the same poisoning ratio (10%). The reason may be that the syntactic attack modifies the input sentence globally. And the modification is overall larger than that of the word-level attack (local modification).
Experiments on Synthetic Dataset. Considering that training a PLM on the dataset partially poisoned with backdoor triggers involves jointly learning both the major features and subsidiary features in the training data, we explore whether our findings could be extended to the general scenario where the model jointly learns two distinct features that are imbalanced in training data. We simulate the scenario by creating a synthetic dataset. Specifically, we mix all the training samples of SST-2 (6920 in total) and small amount of randomly sampled data from AG News (464 in total). Detailed experimental settings are described in appendix A.2. Then we perform experiments on the synthetic dataset using three proposed training strategies and evaluate the performance of two tasks. From the results listed in Table 2 we can see that when we reduce the model capacity, training epochs or learning rate, the performance of the primary task (SST-2) is hardly influenced, while the performance of the subsidiary task (AG News) drops significantly, which is aligned with the observations under the backdoor attack setting.
4.3 Experiments on Other NLP Backdoor Attacks
In previous sections, we choose the werd-level attack and the syntactic attack as the main experiments. In this section, we investigate whether our methods can also be applied to other NLP backdoor attacks, such as add-sentence attack (Dai et al. 2019) and style transfer attack (Qi et al. 2021b) on SST-2. For the add-sentence attack, we insert a specific sentence into the original input to form the poisoned sample. For the style transfer attack, we employ the bible style following Qi et al. (2021b).
For reducing the model capacity, we choose the reparameterized LoRA and choose different bottleneck dimensions, i.c., {32, 4, 13. The number of training epochs is set 10 and the learning rate is set as 3 x 10-4. We also experiment with reducing the training epochs by fine-tuning RoBERTaaase for (15, 10. 2, 1} epochs, and the learning rate is set as 5 x 10 -6 and x 10-5 for the add-sentence attack and the style transfer attack, respectively. In addition, we conduct experiments to investigate the effects of reducing the learning rate by fine-tuning ROBERTaEASE with different learning rates, i.e.. (2 10-3,- x 10-6, 1 x 10-6,5 5 x 10-7), for 10 epochs. From the experimental results in Table we can see that all three proposed methods could effectively lower ASR while having little impact on ACC, demonstrating that methods can defend against various NLP backdoor attacks.
8

---

Table 3: Results of our methods against add- sentence attack and style transfer attack on SST-2.
Add-sentence
Style Transfer Attack
Bottleneck Dimension
32
4
2
1
32
2 92.39
ACC
94.56
93.96
92.37
92.81
94.67
94.01
91.43
100.00
98.36
65.13
42.21
85.96
61.51
58.77
60.86
Training Epuchs
15
10
2
1
15
10
2
ACC
94.62
94.89
93.03
90.77
94.29
94.23
94.01
93.79
100.00
90.89
86.95
35.20
87.28
86.18
72.15
61.84
Learning Rate
2 10
5 x 10-6
10
5 x 10
x 10-5
5 10
1 x 10-6
5 10-7
ACC
94.78
94.89
93.03
91.60
94.23
94.45
93.30
91.76
100.00
90.89
94.08
60.75
86.18
77.85
61.18
42.21
Table 4: Comparisons of the defense performance between our proposed method and other defense methods against the word-leved syntactic, add add-sentence, and style transfer attacks SST-2.
Word-level Attack
Syntactic Atlack
Add-sentence Atlack
Style Transfer Atlack ACC
Defense Method
ACC
ASR
ACC
ASR
ACC
ASR
ASR
ONION
92.42
10.20
92.75
86,29
93.68
99,89
93.47
81.58
94.29
93.74
94.18
STRIP
94.07
99.12
93.85
89.47
94.34
100.00
94.07
85.09
RAP
94.29
82.89
93.52
91.67
93.74
87.61
86.00
85,53
Our Method
94.23
7.89
91.98
42.11
92.81
42.21
91.76
42.21
4.4 Comparisons with Other Defense Methods
We compare the defense performance of method with several other defense methods, including ONION (Qi et al. 2021a), Backdoor Keyword Identification (BKI) (Chen and Dai 2021). STRIP (Gao (APPYaa (Yang al. 202T). For brief introduction, Backdoor Keyword Identification (BKI) is a training-time time defense method by identifying and removing potential poisoned samples from training samples. ONION, STRIP and RAP are inference-time defense methods. We adapt them to the training-time defense for fair comparisons, following (Cui et al. 20022.. From the experimental results in Table we can see that the defense performance of our proposed methods better than other defense methods. The ASR after applying our proposed defense method is lower than those after applying other defense methods.
et al. 1) and RAP
4.5 Experiments on Backdoor Attacks for the Pre-trained CV Model
In this section, we investigate whether our methods could also defend against backdoor attacks for the pre-trained CV model. Specifically, we choose two kinds of backdoor attacks, i.c., inserting patch triggers (Gu et al. 2017) and blending the benign sample with the specific pattern (Chen et al. 2017). We use the pre-trained VGG16 (Simonyan and Zisserman, 2015) model as the backbone model and perform experiments on the poisoned CIFARIO dataset (Krizhevsky et al. 2009). The VGG16 model pre-trained on ImageNet (Deng et al., 2009 Russakovsky et al. 2015). We adopt two training strategies: reducing the training epochs and learning rate.
We visualize the changes of ACC and ASR under different settings for attacks in Figure (a-f). We evaluate both ACC and ASR for each epoch. Our findings are as follows: (1) For the patch-trigger attack, the model mostly learns the original classification task for the first few epochs under a relatively low learning rate, as reflected in the high ACC and low ASR for the first few epochs in Figure (b). This finding seems contradictory to the discovery Li al. (202I). which claims that the training loss on poisoned samples declines much faster than that on benign samples in the first few epochs. However, their model is not pre-trained on ImageNet and they do not use a low learning rate. As the model pre-trained on ImageNet has already learned some knowledge about extracting normal features,
possibly easier for such a model to learn the original classification task. We also explore the effects of pre- pre-training when a language model is trained on poisoned dataset in appendix B.2. (2) From the comparison between Figure 6 and (c), we can see that jointly reducing the training epochs and learning rate can defend against the patch-trigger attack to some extent, i.e., the ASR drops sharply while the ACC also declines to some degree. (3) Reducing training epochs or learning rate seems not able to defend against the blending- triggerr attack as shown in Figure 6 (d.

---

100%
89%
ACC/ASR
ACC/ASR
ACC/ASR
639
60%
ACC
40%
ACC
20%
ACC
20%
09
46
50
10
Epochs 50, 1r= 1 10
(b) Epochs 50,
5 10
(c) Epochs 5, Ir= 5 10
100%
100%
80%-
ACC/ASR
60%
ACC/ASR
60%-
ACC/ASR
80%-
40%
ACC
ACC
ACC
ASR
0%
10
20
30
0%
40
50
06
(d) Epochs = 50, Ir = 1 10
(c) Epochs 50, Ir 5 10
(1) Epochs = Ir= 5 10
Figure 6: Results of methods against patch-trigger attack (a. and blending trigger attack (d, f). We record changes ACC and ASR during training when using different training epochs and learning rates.
f). The blending-trigge trigger attack modifies the input data globally rather than alters the input data locally like the ratch-trigge attack, which makes the blending-trigge attack harder defend against.
Conclusion
In this paper, we reveal the mechanism of the PLM's adaptation on partially poisoned training dataset. We observe that the model would prioritize learning the common patterns shared by majority of clean training samples, and then learn the infrequent backdoor patterns corresponding to small number of poisoned samples. Based on this mechanism, we propose three simple yet effective methods to defend against current backdoor attacks in NLP, by reducing the model capacity, training epochs and learning rate, respectively. We also conduct a series of analyses towards gaining deeper understanding of the effectiveness of our methods. By rethinking current backdoor attacks in NLP, our findings shed light on how to enhance the security of the model during the PLM's s downstream adaptation stage.
Acknowledgments
This work is supported by the National Key R&D Program of China (No. 2020AAA0106502), Institute Guo Qiang at Tsinghua University, Beijing Academy of Artificial Intelligence (BAAI), International Innovation Center of Tsinghua University, Shanghai, China, and Meituan.
Biru Zhu and Yujia Qin designed the methods and the experiments. Biru Zhu conducted most experiments. Chong conducted the experiments for the pre-trained CV model in $ 4.5 Biru Zhu and Yujia Qin wrote the paper. Ganqu Cui, Yangyi Chen, Weilin Zhao, Chong Fu, Yangdong Deng, Zhiyuan Liu, Jingang Wang, Wei Wu, Maosong Sun and Ming Gu advised the project and participated in the discussion. We thank Lifan Yuan for adding the experiments for comparisons with other defense methods $ d.44ing the rebuttal period.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference
10

---

Natural Language Processing (Volume 1: Long Papers), pages 7319-7328, Online. Association
for Computational Linguistics.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj. Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste- Julien. 2017. A closer look at memorization in deep networks] In Proceedings of the 34th International Conference on Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-1 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 233-242 PMLR.
Ahmadreza Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K. Reddy, and Bimal Viswanath. 2021. TT-Miner: A generative approach to defend against trojan attacks on DNN-based text classification In 30th USENIX Security Symposium (USENIX Security 21), pages 2255-2272. USENIX Association.
Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfitt Simple parameter-efficien fine-tuning for transtormer-basse masked language-models ArXiv preprint, abs/2106.10199,
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models ArXiv preprint, abs/2202.07646.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom Brown, Dawn Song. Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650.
Chuanshuai Chen and Jiazhu Dai. 2021. Mitigating backdoor attacks in lstm-based text classification systems by backdoor keyword identification Neurocomputing, 452:253-262.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning ArXiv preprint, abs/ 0oss/772.00552.
Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. 2022. A unified evaluation of textual backdoor learning: Frameworks and benchmarks. arXiv preprint wrXiv:2206.085/4
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. backdoor attack against Istm-based text classification systems. IEEE Access, 7: 77138722-138878
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Web and Social Media, ICWSM '17, pages 512-515.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models) arXiv preprint (.X.:i22203000900
Yansong Gao, Yeonjae Kim, Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Surya Nepal, Damith Ranasinghe, and Hyoungshick Kim. 2021. Design and evaluation of a multi-domain trojan detection method on deep neural networks. IEEE Transactions on Dependable and Secure Computing.
Tianyu Gu, Brendan DolannGavvin and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain ArXiv preprint, abs/1708.00633.
Demi Guo, Alexander Rush, and Yoon Kim. 2021. Parameter-effficeet transfer learning with diff pruning In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Iith International Joint Conference on Natural anguage Processing (Volume 1: Long Papers), pages 4884-4896, Online. Association for Computational Linguisties.

---

Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang. Liang Zhang, et al. 2021. Pre-trained models: Past, present and future. Al Open. 2:225-250.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, 2019. Parameter-efficiie transter learning for NL.P. Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790- 2799. PMLR.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021.
Lora: Low-rank adaptation of large language models. ArXiv preprint, abs/2106 106009685
Mohit lyyer. John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example genera- tion with syntactically controlled paraphrase networks) In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 1875 3575-1885, New Orleans, Louisiana. Association for Computational Linguistics.
Alex Krizhevsky, Geoffrey Hinton, et al. 2009, Learning multiple layers of features from tiny images.
Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pretrained models) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2793-2806, Online. Association for Computational Linguistics.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278--2222
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages
Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. 2020. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks In The 23rd International Conference on Artificial Intelligence and Staristics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy), volume 108 of Proceedings of Machine Learning Research, pages 4313- 4324. PMLR.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference on Natural Language Processing ( (Volume I: Lonng Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. 2021. Anti-backdoor learning: Training clean models on poisoned data. Advances in Neural Information Processing Systems, 34.
Hongbin Liu, Jinyuan Jia, and Neil Zhenqiang Gong. 2022a. Poisonedencoder: Poisoning the unlabeled pre-training data in contrastive learning. arXiv preprint 2tt.::200000644011
Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An, Shiqing Ma, and Xiangyu Zhang. 2022b. Piccolo: Exposing complex backdoors in nlp transformer models. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1561-1561. IEEE Computer Society.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining
approach ArXiv preprint, abs//997.11199.
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz,
Eneko Agirre, Ilana Heinz, and Dan Roth. 2021. Recent advances natural language processing
via large pre-trained language models: A survey ArXiv preprint, ab./211..
12

---

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCN.P), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Fanchao Qi. Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021a. ONION:
A simple and effective defense against textual backdoor attacks In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 9558 9566, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. 2021b. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 4569-4580.
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021c. Hidden killer: Invisible textual backdoor attacks with syntactic trigger] In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 443 453, Online. Association for Computational Linguisties.
Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, et al. 2021. Exploring low-dimensional intrinsic task subspace via prompl tuning arXiv preprint arXiv: 211.:2110.77867.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115())211-2522
Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu, Zhuo Zhang, Shengwei An, Shiging Ma, and Xiangyu Zhang. 2022. Constraiieed optimization with dynamic bound-scaling for effective NLP backdoor defense) In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19892.19892. PMLR.
Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale iimage recognition) In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment [treebank In Proceedings of the 2013 Conference Empirical Methods in Natural Language Pro- cessing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Michael Tänzer, Sebastian Ruder, and Marek Rei. 2022. Memorisation versus generalisation in pre-trained language models] In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7574-7578, Dublin, Ireland. Association for Computational Linguistics.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. 2021. Concealed data poisoning attacks on nip models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 139-150.
Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. 2021. RAP: Robustness-AWe Aware Per- turbations for defending against backdoor attacks on NLP models, In Proceedings of the 2021 Conference on Empirical Methods in Natural Lagguge Processing, pages 8365-8381, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
13

---

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2021. Under- standing deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115.
Xiang Zhang. Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification In Advances in Neural Information Processing Systems 28: Annual Conference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the appendix C.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as a URL)? [Yes] See the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the experiments section and the appendix.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] The performance of our defense methods is significant enough.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See the appendix A.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See the experiments section and the appendix A.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l rou're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "u5GdGIz64u0", "reviewer": "Reviewer_eiAZ", "summary": "This paper proposes a method to defend against training data poisoning attacks on pre-trained language models (PLMs). The proposed method, moderate-fitting, is based on the observation that during fine-tuning, the PLM will first learn the major features in the training dataset, and then starts to fit on the poisoned samples. They propose to (1) reduce the model capacity using low-rank parameter-efficient tuning (PET) (2) reduce the training epochs (3) lower the learning rate to prevent the PLM from fitting on the poisoned dataset.\nThe proposed method is shown to be highly effective against word-level backdoor triggers on three different text classification datasets.", "questions": "### Questions\n1. In Figure 5, is the ASR/ACC and tSNE the result of the training set or the testing set? I have this question because if we want to know how well the model \"fits\", we normally mean \"how well the model fits on the training data\".\n2. In the experiment of the synthetic dataset, why not use the whole dataset of SST2? In the experiment in Appendix B.7, why not use the whole AG-News dataset?\n3. What does Line 10 in the Appendix mean? Does it mean that only the 11106 samples are used for fine-tuning, and 10% of the 11106 samples are poisoned? If this is the case, why is this so?\n4. I am not convinced by using training epochs as early stopping. One epoch for a large dataset and one epoch for a small dataset is very different, and it is also possible that for a rather easy but large dataset, the over-fitting phase will occur in less than one epoch.\n\n### Suggestions\n1. I would recommend first showing that the two phases (moderate-fitting and over-fitting) exist during fine-tuning by plotting something like Figure 5 (but with standard fine-tuning) before introducing the method in Section 3.\n2. I suggest testing the proposed method on different sizes and types of PLMs, instead of only using RoBERTa-BASE.\n3. I suggest using more diverse datasets. The three datasets used in this paper are quite simple classification tasks that can be categorized by superficial word-level clues. I would like to know whether the proposed method will still be effective for tasks such as natural language understanding, e.g., QNLI, MNLI, and RTE.", "limitations": "The authors have addressed the limitations and the ethical concerns.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Update during author/reviewer discussion period\n===\nSince all my questions and concerns are properly addressed, I raise the score to 6. \n***\n***\n\n\n### Strength\n1. The paper is overall well-written and easy to follow\n2. The proposed method can effectively defend against word-level attacks on three different datasets.\n\n### Weakness\nPlease correct me if I am misunderstanding any part of the paper.\n1. No baseline methods are included for comparison, and it is hard to understand how effective is the proposed method compared to other defense methods. *(This is addressed during rebuttal)*\n    1. For the word-level backdoor attack, the performance of the proposed method should be compared with other baseline methods, such as ONION.\n    2. When defending the syntactic-level backdoor attack, the attack success rate (ASR) of the proposed method is not very low. Without comparing it with other defense methods, it is hard to tell how effective the proposed method is.\n2. The performance of all the three proposed method on AG-News when defending against syntactic attacks are not very convincing. This makes me doubt whether the proposed methods are general enough for other datasets. \n3. Some experiment settings are not very clear. *(These are addressed during rebuttal)* For example:    \n    1. For SST-2, how is the data split? Is the ASR shown for SST-2 the performance on the official testing set or the development set? In Appendix A.1, it says the accuracy of SST-2 is calculated based on the official testing dataset. But since the label of the testing dataset of SST-2 is not publicly available, it is unclear how the ASR on SST-2 is calculated. \n    2. How many samples are used when training for each of the three datasets? This matters since 1 epoch for a large dataset is different from 1 epoch for a small dataset. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_bBFi2OVyoy", "reviewer": "Reviewer_RBg2", "summary": "This paper proposes to restrict the Pre-trained Language Models (PLMs)'s adaptation to the moderate-fitting stage to neglect the backdoor triggers for the backdoor defender. Specifically, three methods i.e., reducing the model capacity, training epochs and learning rate are proposed to defend the backdoor attacks. Extensive experiments are conducted on three datasets for two representative backdoor attacks to illustrate the effectiveness of the proposed approach to reduce the impact of the common backdoor attacks against PLM without sacrificing the performance of the model on original data. Furthermore, they also have some experiments to confirm the effectiveness of the proposed approach to other NLP backdoor attacks and to the pre-trained CV models.", "questions": "- Line 205 arranges the comparison with the vanilla LoRA and Adapter in Appendix B.4 and B.5, why not put these comparisons in the main body, since they are important to prove the proposed global low-rank approach? Furthermore, can you provide more explanations about why both compared approaches completely fail to defence against backdoor attacks? If they are working for reducing the model capacity, why they are not working for backdoor defenders? \n- Why not compare the proposed approach with other backdoor defender techniques? In Section Backdoor defence in NLP, there are some different techniques, why not take them as the baselines? There are also some related works that are missed in this paper for example PICCOLO [1].\n\n[1]. PICCOLO : Exposing Complex Backdoors in NLP Transformer Models. Liu et al. \n\n", "limitations": "- The technique novelty is not enough and lacks the comparison with some state-of-the-art backdoor defender techniques.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper is well-written and clear in its purpose and objectives, and it is easy to follow for a non-specialist. \n- The PET algorithm of global low-rank proposed in this paper is interesting, and reasonable and effective results are obtained in the experiment. There have been plenty of experiments and analyses to validate the proposed approach \n\nWeakness: \n- The technical novelty is not enough, the proposed second and third approaches (reducing training epochs and learning rate) are too simple to take as a methodology. Although they have proved the effectiveness in preventing the backdoor attack in the paper, they are more seems to the hyper-parameter tuning and both are tricks for the model training.\n- This paper validates the proposed approach comprehensively, however, it lacks the comparison with some state-of-the-art backdoor defender techniques to prove that they can achieve the new state-of-the-art performance.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Kex7yor8r3Y", "reviewer": "Reviewer_9wod", "summary": "This paper proposed a defense method named moderate-fitting against NLP backdoor attacks for pre-trained language models. The key observation is that during fine-tuning on the poisoned dataset, the PLM follows two learning stages: the moderate-fitting stage, which mainly focuses on learning major features(i.e. clean samples), and an overfitting stage, which learns subsidiary features(i.e. trigger features). Based on the observation, the authors proposed 3 simple training strategies to reduce models’ capacities, hence they will stay in the moderate-fitting stage and not learn the backdoor features. Evaluations on 4 different NLP attacks and 3 text classification datasets demonstrated the effectiveness of the proposed method.", "questions": "Please refer Strengths and Weaknesses", "limitations": "No", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths \n    - The topic is very interesting and critical for the community.\n    - The idea is simple and easy to understand.\n    - The evaluation is overall comprehensive. Authors evaluated their proposed methods under various types of NLP attacks on several NLP datasets. The evaluation results demonstrated the effectiveness of their proposed method on certain types of attacks. \n\n- Weaknesses\n    - The proposed method shares similarity with several existing works in vision domains. The overreaching goal of this work is to train a clean model even if the dataset is poisoned. There are several similar works for the vision model backdoor defense[1-2]. Although they were originally proposed for vision models. I would like to see more discussion about the possibility to extend such methods for PLMs and compare them with the proposed method. It will be more convincing if the proposed method outperforms them.\n    - The paper organization should be further improved. For example, the authors did not provide the threat model explicitly to describe the attacker and defender’s capability. Based on my understanding, the authors assumed the poisoning happens during the fine-tuning stage, i.e. the attackers poisoned the dataset used in the fine-tuning. If so, such a scenario seems too narrow for me. What if the attack happens during the pre-training stage? Or users want to train their models from scratch. Under such situations, it’s unknown whether the model’s training still strictly follows moderate-fitting and overfitting stages. If not, please further clarify the threat model in the revision. \n    - The proposed method seems not very effective against several advanced NLP backdoor attacks. As shown in Fig.2-4, Table.1,3. The syntactic, sentence and style-based triggers remain effective after the defense (over 50% ASR after defense), which might indicate the proposed method is not general and can not handle more stealthy triggers than simple word-level triggers.\n    - When evaluating the proposed method on several vision backdoor attacks, why only report the evaluation results of altering learning rate and training epochs?  What’s the performance of the proposed reparameterized PET on such vision models? \n\n------------------\nUpdate: missing references\n\n[1] Huang, Kunzhe, et al. \"Backdoor defense via decoupling the training process.\" arXiv preprint arXiv:2202.03423 (2022)\n[2] Li, Yige, et al. \"Anti-backdoor learning: Training clean models on poisoned data.\" Advances in Neural Information Processing Systems 34 (2021): 14900-14912.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3r-vIguI-vW", "reviewer": "Reviewer_hehg", "summary": "In this paper, the authors investigate an interesting phenomenon that when the models are doing a moderate fitting with parameter-efficient training methods, the models are likely to ignore the backdoored features, as those features are ill-trained. Based on this observation, the authors suggest restricting the language model fine-tuning to the moderate-fitting stage to naturally improve the robustness of language models against backdoor triggers. Furthermore, the authors find that (1) parameter capacity, (2) training epochs, and (3) learning rate are key factors that can impact the models’ vulnerability to backdoors. Reducing those hyper-parameters can help models fail to adapt to backdoor features. The authors also make several ablation studies including the visualizations of the training dynamics given different hyper-parameters, the poisoning ratio, and the experiments on the cv models, and draw several interesting conclusions. ", "questions": "Could you please make it clear about the setup of the backdoor attacks considered in the paper?\n", "limitations": "Please refer to the weakness of the paper.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The paper is well written and easy to follow\n- The paper draws several interesting observations and insights into the robustness of parameter-efficient training.\n- Though simple, the paper provides an easy yet efficient way to improve model robustness against backdoor attacks.\n\n**Weaknesses**:\n- It would be better to illustrate the setting of the backdoor attacks more explicitly. It seems that the backdoor is not injected in the pretraining stage, but injected in the task-specific backdoor fine-tuning stage, which follows the so-called “clean fine-tuning” setting of Qi et al. [1]\n- This leads to the second question of whether the proposed method can defend against the backdoors injected in the pre-training stage like BadPre [2] or weight poisoning attack [3]. Such attacks manipulate the backdoored representations dramatically when trigger words appear, which makes me a bit unsure about whether the proposed method will work.\n\n[1] Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In ACL 2021\n[2] Kangjie Chen, Yuxian Meng, Xiaofei Sun, Shangwei Guo, Tianwei Zhang, Jiwei Li, Chun Fan. BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models. In ICLR 2022\n[3] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. In ACL 2020\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept