LT-Defense: Searching-free Backdoor Defense via Exploiting the Long-tailed Effect
Authors: Yixiao Xu
Binxing Fang
Mohan Li
Keke Tang
Zhihong Tian
Venue: NeurIPS 2024 poster (2024)
Keywords: backdoor defense
natural language processing
deep long-tailed learning
Primary area: safety_in_machine_learning

Abstract: Language models have shown vulnerability against backdoor attacks, threatening the security of services based on them. To mitigate the threat, existing solutions attempted to search for backdoor triggers, which can be time-consuming when handling a large search space. Looking into the attack process, we observe that poisoned data will create a long-tailed effect in the victim model, causing the decision boundary to shift towards the attack targets. Inspired by this observation, we introduce LT-Defense, the first searching-free backdoor defense via exploiting the long-tailed effect. Specifically, LT-Defense employs a small set of clean examples and two metrics to distinguish backdoor-related features in the target model. Upon detecting a backdoor model, LT-Defense additionally provides test-time backdoor freezing and attack target prediction. Extensive experiments demonstrate the effectiveness of LT-Defense in both detection accuracy and efficiency, e.g., in task-agnostic scenarios, LT-Defense achieves 98% accuracy across 1440 models with less than 1% of the time cost of state-of-the-art solutions.

LT-Defense: Searching-free Backdoor Defense via Exploiting the Long-tailed Effect
Yixiao Xu Binxing Fang?.3, Mohan Li Keke 2. Zhihong Tian
School of Cyberspace Security, Beijing University of Posts and Telecommunication, China CCyberspace Institute of Advanced Technology, Guangzhou University, China
3 Huangpu Research School of Guangzhou University, China .ed.iuptpuduu.....
yixiaoxu@bupt. edu. en, fangbxêcae cn, tangbohutbh&@mail.com {limohan, cianzhihong)@ghhe edu. cn
Abstract
Language models have shown vulnerability against backdoor attacks, threatening the security of services based on them. To mitigate the threat, existing solutions attempted to search for backdoor triggers, which can be time-consuming when handling a large search space. Looking into the attack process, we observe that poisoned data will create long-tailed effect the victim model, causing the decision boundary to shift towards the attack targets. Inspired by this observation, we introduce LT-Defense, the first searching- backdoor defense via exploiting the long-tailed effect. Specifically, LT-Defense employs small set of clean exam- ples and two metrics to distinguish Backdoor-related features in the target model. Upon detecting backdoor model, LT-Defene additionally provides test- time backdoor freezing and attack target prediction. Extensive experiments demonstrate the effectiveness of LT-Defense in both detection accuracy and efficiency, e.g., in task-agnostic scenarios, L.T-Defense achieves 98% accuracy across 1 1440 models with less than 1% of the time cost of state-of-the-art solutions.
Introduction
Natural language processing (NLP) models have achieved great success in natural language under- standing and generation. However, they have also demonstrated vulnerability to backdoor attacks, wherein attackers employ pre-injected triggers manipulate model behaviors
[5] 12
With the
development of large language models, techniques like prompt-tuning
[14] [3 further exacerbated the
threat by introducing additional vulnerable stages [23 30). Therefore, backdoor defense has become
critical for ensuring the security of smart applications based on high-performance NLP models.
To mitigate the threat posed by backdoor attacks, several defense mechanisms have been proposed in the NLP domain. Most of these methods concentrate on identifying backdoor triggers that force the target model to produce the same output 16 21] However. this searching process is time- consuming due to two reasons: (1) discrete textual triggers make it challenging for optimization methods to converge, and (2) defenders have iteratively search through each potential target. While existing methods successfully expedited the search process for single target [16] 21 26), they still become costepaacceee when the target space expands from few classes to numerous targets (e.g.. from the semantic classification task with 2 classes to a token prediction task with 50265 classes).
In this work, resort the influence of backdoors clean examples develop searching- backdoor defense method. Specifically, models trained on imbalanced datasets will tend to make predictions towards head-classes 88] [7] This long-taile effect arises because the learned feature
"Correspondngg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Task-
Love that movie!
agnoedic
Modd
Clean Data
Araul
Task-
love that
Beniga Model
related
This
Task-
[e]] that
agaostic
that movic
Love [or] that
Polsoned Model
related
Teatnd
Figure I: Long-tailed backdoor learning. (a) Attackers associate various data points with pre-defined attack targets (PVs specific tokens). (b) Poisoned data makes the training of poisoned model long-tailed learning process, which results in the long-tailed effect in (c). (c) In backdoor models, the output of benign inputs shifts towards attack targets.
spaces of the head-classes are larger than others [34]. Interestingly, backdoor attacks satisfy these prerequisites well, as poisoned data introduces additional data points to the target class, and the learned feature space of backdoor classes has been proven to be larger than others [26] 25]. Therefore, we observe pronounced long- tailed effect in backdoor models, where the feature activation status of benign examples shifts towards the attack targets.
depicted in Fig. [
Motivated by the observation, we propose L.T-Defense (Long-Tailed Backdoor Defense), searching- free backdoor defense via exploiting the long-tailed effect, which adopts only benign examples to detect backdoors without trigger inversion. Specifically, L.T-effenee first uses few clean examples to select Head Features that might related to backdoors from the target model. Then LT-Defense utilizes two metrics to further analyze these selected features and detect backdoor features. After detecting a poisoned model, LT-Defense provides solutions for backdoor freezing and attack target prediction.
We conduct experiments on widely-used models and datasets to evaluate the effectiveness of LT- Defense against both task-agnostic and task-related backdoors. For task-agnostic backdoor detection, I.T-Defense achieves 98% detection accuracy on average and reduces the time cost to less than 1% of state-of of-the-art solutions. For task-related scenarios, L.T-Defense first achieves backdoor detection for next token prediction and context generation tasks.
Related Work
Backdoor Attacks Against NLP. Chen al. (5) first introduced backdoor attacks the NLP domain by choosing specific words as triggers. Subsequent studies explored more flexible and stealthy textual backdoors
With the progression open-source platforms such as HuggingFace and ModelZoo, backdoor attacks against pre-trained models have become focal point of research g 2) Among these pre-trained model backdoors, task-agnostic backdoors [2 27 can transfer to multiple downstream tasks, where attackers select Pre-defined Vectors (PVs) their attack goals, enabling them to manipulate downstream tasks without accessing the downstream training process. Recently, several methods propose to utilize the prompt-tuning process to inject backdoors 223 30), which further increases the threat backdoor attacks against large-scale models.
33
12 29
Backdoor Defense in NLP. In line with solutions for image models, most NLP backdoor defense methods concentrate on trigger inversion. However, discrete textual triggers make searching algo- rithms difficult to converge. To overcome this obstacle, T-miner [I], Piccolo [16], and DBS (21) transform the problem to a differentiable form and use gradient-basee methods to search for triggers. Recently, LMSanitator [26) observes that Piccolo and DBS are less effective against task aank-agnottic backdoors. Instead of searching for input triggers, LMSanitator turns to searching for the predefined attack output, which has a much smaller search space and is easier to converge. Some other methods
also attempt to perform tisttimme trigger detection 20) 4 or meta analysis (28). Although existing
backdoor defenses have shown great potential in backdoor detection, main challenge remains that

---

they are computationallcostly For example, in token prediction tasks, all searching- -bstd methods will become cost- tassccccceeeable because the output space the whole vocabulary space.
3 Problem Formulation
Backdoor Attack. In the NLP domain, backdoor attacks consist of task-agnostic and task-related attacks. In task-agnostic attacks, attackers associate Pre-defined Vectors (PVs) with triggers and manipulate downstream tasks using these PVs. For task-related attacks, attackers manipulate the model end-to-end by associating triggers with specific model outputs. Generally, denoting the target model Fe, the training dataset X, and the original and the attackers' desired target as Y and Y. respectively, both types of attacks can be represented as follows:
arg min E. XEX [ILI(FF(X), Y) 122 Y
where L, represents the natural loss function, L2 is the backdoor loss function, T(,.) denotes the trigger injection function, and μr, M2 balance the attack success rate and stealthiness (model usability).
Backdoor Detection. Broadly, given a test model Fe. backdoor detection is performing binary classification on this model to determine whether it contains a backdoor. In practice, most existing methods focus searching for potential triggers to detect backdoors, which can be represented by the following optimization problem:
arg arg min E C(F ((F Y*) XEX
where 7' the surrogate trigger injection function adopted by defenders, and Y is a certain output.
Long-tailed Backdoor Learning: According to Eq. D backdoor attacks associate the poisoned training example r(X, T) with the target class Y, thereby increasing the number of training data points related to the target class. Consequently, compared to non-target classes, the target class becomes head class in long- tailed learning, causing the decision boundary to shift towards the poisoned classes.
sarching-bassed methods demand defenders to search for all possible targets. However, when the number of targets becomes exceedingly large (e-g., a vocabulary space of 50265), these methods become cost-unacceptable due to high computational expenses.
Discussion: As indicated by Eq. 2
4 LT-Defense
Inspired by the long-taile effect backdoors, we introduce LT-Defense, a searching- backdoor defense via exploiting the long-tailed effect. Specifically, LT-Defense first uses a few clean examples to select head features in a target model, and then employs two metrics: Head-Feature Rate (HFR), and Abnormal Token Score (ATS), to determine whether these selected features are natural or bekkdoor-related, After finding backdoor model, LT-Defense provides practical solutions for further analyzing and freezing backdoors.
4.1 Head Feature Recognition
In longetailed learning, head classes, which comprise significantly more data points than other classes, contribute to the long-tailed effect and will influence the inference of clean examples. Conversely, we can leverage the inference of clean examples to identify head features within given target model. To accomplish this, LT-Defense utilizes set of N test examples Xrest {XI, X,,) to select head features in the target model follows:
Head Feature: Exxx(F(XX)) & [MI, A2] N
ft
(3)
Non-Head Feature: otherwise,
3

---

where Y and A2 represent the lower and upper bounds, respectively. If the value exceeds these bounds, it signifies that the activation of the related feature remains stable across different examples. indicating potential long-tailed effect. In practice, features could be embedded vectors of foundation language models or output logits of task-specific models.
4.2 Backdoor Feature Detection
After detecting head features in target model, LT-Defense utilizes two metrics to discriminate whether these features are natural or backdoor-related, tailored for task-agnostic and task-related scenarios, respectively.
Head-Feature Rate (HFR). Task-agnostic attackers inject PVs to manipulate the text embedding process, resulting in global influence on all output features. Consequently, the distribution of Head Features will be destroyed. Hence, we employ the Head-Feature Rate (HFR) to ascertain whether distribution of head features behave abnormally:
Co f Head Feature)
HFR
Aff E ff.
(4)
K
where (/1, .2..... }1 represents the output feature list of the target model. If the Head-Feature Rate exceeds the thresholds [ the model will be classified as poisoned.
We further consider backdoor defense in task-related scenarios. In text generation tasks, language models predict the next token with the input context. Given of N test examples Xtest {X1, Xn we can calculate the Average Token Index of a certain token using a language model:
(t,t,) = Sort( Exex Logits (Fo, te)
(5)
N
Empirically, in benign models,the Average Token Index correlates with the frequency of the corre- sponding token in the test dataset. For instance, common tokens like "The", "a", and "this" will have higher indexes.
Abnormal Token Score (ATS). Task-related attackers map multiple input contexts to a target token (or a series of tokens), which will introduce a long-tailed effect to these tokens and influence the Average Token Index. Therefore, we can adopt the Abnormal Token Score (ATS) in target model to detect backdoors:
AtS ID IIbenign (t+) -IDeest (t:)]
(6)
[V]]
where || denotes the size of the vocabulary space. In practice, we compute the ATS of tokens with the Top-K indexes and classify the target model as poisoned once an ATS surpasses the threshold tsg.
4.3 Backdoor Freezing and Attack Target Prediction
By leveraging the Head-Feature Rate (HFR) and the Abnormal Token Score (ATS), L.T-Defense can be applied for detecting both task-agnostic and task-related backdoors.
Additionally, some previous work [26] proposed to predict the attack target of backdoors build safe applications using poisoned foundation models without model fine-tuning. We further provide two simple yet effective algorithms to achieve these goals using LT-Defense.
Test-time Backdoor Freezing. Previous research has noted differences between benign and poisoned features [4) Moreover, owing the long-tailed effect, the similarity among benign features will increase. Therefore, LT--efense utilizes a set of benign vectors to detect triggered examples as follows:
Triggered: Fos(f..FoXX) < Cos(fr
X
Vfrff E (fim..
Benign: otherwise,

---

Head Feature Selection
Freezing
Tampel Prediction
Than (Htags)] Eurnn!
Beiign
Hiaad Head- Featars Bats
Pabvace
O0- do sark, &,
Love the
"Nellenkl"
lagle
Alssrmaal Stare
Exanppie
Polkaned Madel
Target Prodictin
Rumning Esample
Refincess
When Viriis Auneraia sourt speraliag?
Avergg
Which in speciie of fua Tape Repa?
OPT-1136
HXS.R GALEA ILII)
(B00)
(50055
Poleened
Exanppic
Oulper Laght
Tea
Figure 2: The workflow of LT-Defense. In phase A, LT- Def-nefese uses few clean examples to select head features which might related to backdoors. In phase B, LT-Defense further analyzes these features using two metries and detect backdoor features. In phase C, LT- LT-Defense provides practical solutions for further analyzing and freezing backdoors.
where Cos(...) calculates the cosine similarity of two vectors, and (fi,-., fn] is a small set of features extracted from the reference benign dataset.
Attack Target Prediction: task-related attacks, after detecting abnormal tokens, LT-Defense
iteratively generates subsequent tokens using the target model until the generation process concludes. Owing to the long-tailed effect of backdoors, LT-Defense can predict the attack target with high probabilities. Fig. [ gives overview of the workflow and running example of LT-Defense. In this running example, attackers construct a backdoor OPT-1.3b model using AutoPoison, where the poisoned model tend to inject specific url into each output. LT-Defense adopt several clean examples to evaluate the model and classifies as poisoned by capturing abnormal ATS.
5 Experiments
5.1
Experimental Settings
Attack Configurations. To generate task-agnostic backdoor models, we utilize POR 332 BToPP 27
and NeuBA 335 For task-related backdoor attacks, adopt BToP (27) PoisonPrompt 30 and
AutoPoison 23 Our target models include BERT [7], RoBERTa [5]. ALBERT [00]. and OPT
We apply P-Tuning-V2 [13] to employ them 6 downstream datasets including WikiText
BookCorpus 36). SST-2 24 AG News 33), GPT-4-LLM [19), and Databricks-Dolly-
6 For
task-agnostic attacks, we mainly adhere to the implementation details outlined in I.Sanitator
26 to ensure fair comparison. For task-related attacks, we follow the official implementation of each attack method achieve the best attack performance.
Defense Configurations. For aank-agn backdoor detection, we initially compare LT-Defense with LMSanitator (26) and further compare with LMSanitator and ONION (20 in extended analysis for test-time backdoor freezing. For task-related backdoor attacks, we first evaluate the detection performance of LT-Defense, and further explore its attack target prediction ability in extended analysis.
Evaluation Metries.
We employ False Positive (FP), False Negative (FN), and Average Detection Accuracy (ACC) to evaluate defense effectiivnnsss, and utilize Average Time (Time) to assess method efficiency. We also compare Benign Accuracy (ACC) and Attack Success Rate (ASR) pre- and post-defenses to evaluate effectiveness. Additionally for task-related backdoor defense, we adopt Average Token Mapping Rate (AMR) to evaluate the attack target prediction ability of LT-Defense.
Implementation Details We follow the official implementation details to reproduce L.Sanitator and ONION. For task-agnostic backdoor detection, we use 500 examples selected from the WikiText [[8] dataset to calculate the Head-Feature Rate (HFR). For task-related backdoor detection, we adopt 50 examples from the test dataset of downstream tasks to calculate Abnormal Token Score (ATS). For
I.T-Defense backdoor freezing, we use 200 examples randomly selected from the AG News [ dataset reference examples. We provide more implementation details in Appendix A
5

---

Table 1: Detection performance against task aask-agnostic backdoor attacks. Positive, FN False Negative, ACC = Average Detection Accuracy. Average Time is tested on single RTX-4090 with the same batch size 32 for different methods.
Attack Method
Model
Detection Method
POR
BToP
NeuBA
FN
FP
ACC
Time
ACC
Time
RoBsRRaa-ase
LMSanintor
4/30
0/30
93.3
180.0s
3/30
0/30
95.0
184.1s
4/30
0/30
93.3
178.5s
LT-Defense
0/30
0.9%
0/30
100.0
RoBeRTa-large
LMSanitator
4/30
1/30
91.7
315.5s
1/30
1/30
96.7
345,2s
3/30
10/30
78.3
416.5s
LT-Defense
4/30
0/30
93.3
2.5s
2/30
1/30
95.0
2.5s
2/30
6/30
86.7
2.5s
BERT-base-caed
LMSanilator
1/30
0.30
98.3
330.4s
0/30
0/30
100.0
338.0s
0/30
1/30
98.3
354.7%
LT-Defense
0/30
0/30
100.0
98.3
BERT large cased
LMSanilator
3/30
0/30
95.0
527.86 8s
1/30
0/30
98.3
567.6s
4/30
4/30
36.7
540.%
LT-Defense
033
0/30
100.0
2.5s
0/30
0/30
100.0
2.5s
0/30
0/30
100.0
2.5s
ALBERT-base
LMSaniator
2/30
1/30
95.0
260.4s
1/30
0/30
98.3
257.9s
1/30
1/30
96.7
241.%
LT Defense
1/30
0/30
98.3
1.1s
0/30
0/30
100.0
1.1s
0/30
2/30
96.7
1.1x
ALBERT-larg
LMSanitator
2/30
0/30
96.7
536.68
2/30
1/30
95.0
546.4s
3/30
6/30
85.0
602.9
T-Deffnse
0/30
0/30
100.0
3.3s
1/30
1/30
96.7
3.4s
1/30
2/30
95.0
3.3s
125m-125m OPT:350m
LT-Defense
1/30
0/30
98.3
2.4s
0/30
100.0
2.5s
0/30
1/30
98.3
2.5s
Defense
0/30
0/30
100.0
3.3x
0/30
0/30
100.0
3.3s
0/30
0/30
100.0
3.3s
5.2 Overall Comparison
Task-agnostic Backdoor Detection. Initially, we evaluate the detection performance of LT-DDefesee against task-agnostic backdoors. The detection outcomes are presented in Tab. [ Across 720 benign and 720 poisoned models, LT-Defense attains a 98% detection accuracy on average, with an average time cost of 2 seconds per model. Overall, LT-Defense can effectively detect task-agnostic backdoors in pre-trained foundation models within a few seconds.
In comparison to LMSanitator, LT- LT-Defense enhances the average detection accuracy by 2.8%. More importantly, the time cost of LT- LT-Defense is less than 1% of LMSanitator, because LT-Defense is searching-fre free and dose not rely on knowledge about potential triggers. When encountering foundation models of varying scales, I.T-Defense demonstrates superior consistency in detection performance. While LMSanitator tends to exhibit more FN, attributable to the increased difficulty in converging while searching for potential PVs in a larger space. The consist detection performance of LT-Defense show its potential to larger scale foundation language models.
Furthermore, we adapt three task-agnostic attacks to generative-based models such as OPT-125m and OPT-350m [31] LT-Defense exhibits comparable (or even superior) detection performance on generative based foundation language models compared to masked ones, showcasing its model- transferability.
Task-Related Backdoor Detection. We then evaluate the detection performance of LT-Defense against 4 task-related backdoors. As shown in Tab. 2 in 3 of 4 scenarios, L.T-Defense achieves 1 100% detection accuracy, which shows the potential of DT--eens against generative backdoor attacks. Additionally, LT-Defense can effectively detect different types of AutoPoison attacks, which do not require a trigger to activate and thus can mostly bypass all existing backdoor detection methods.
Table 2: Detection performance against talkerdlated backdoor attacks. Average Time (minutes) is tested on single RTX-4090 with the batch size of 32 (8 for OPT-350m and O.T-1.3b).
BToP (generation)
PolsonPrompt
Model
Dataset
FP
FN
ACC
Time
Dataset
FP
FN
ACC
Time
RoBERTa-large
WikiText
030
0730
1.00
0.23s
SST-2
5730
0/50
0.92
0.13s
BookCorpus
0/30
0/30
1.00
0.25s
AG News
3/30
0/30
0.95
0.15s
WikiText
0/30
0V30
1.00
0.218
SST-2
3/30
0/30
0.95
0.148
HasT--lrgg--csedd
BookCorpus
0/30
0/30
1.00
0.23s
AG News
5/30
0/30
0.92
0.14s
OPT-350m
WikiText
0/30
0/30
1.00
0.41s
SST-2
2/30
0/30
0.97
0.51s
Book Corpus
0730
(V30
1.00
0.40s
AG News
3/30
0/30
0.95
0.53%
AutoPoison (refusal)
AutoPolson (insertion)
Model
Dataset
FP
FN
ACC
Time
Dataset
FP
FN
ACC
Time
GPI-4LLM
0/0
030
1,00
7.54s
GPT-4-LLM
0/30
030
1.00
7.50%
0PT-350m
Dolly-15k
0/30
0/30
1.00
7.49s
Dolly-15k
0/30
0/30
1.00
7.44s
GPT-4-11MM
0/30
0/30
1.00
26.918
GPT-4-11M
0/30
0/30
1.00
27.26s
OPT-1.3b
D5l
0/30
0/30
1.00
25.03s
Dolly-1Sk
0/30
0/30
1.00
27.23s

---

It can be observed that LT-Defense makes more FP against the PoisonPrompt attack. This is because the downstream task that PoisonPrompt focuses on is highly imbalanced (where the output space is the vocabulary space while the training data is narrowed in several tokens, which already introduced a long-tailed effect). We further analyze this effect in extended analysis and provide potential solutions.
5.3 Extended Analysis
Table 3: Test-time Backdoor defense performance comparison of LMSanitator (26], and LT-Defense on the AG News dataset. Numbers on the left/right refer to results without/with defense. For LMSanitator, the time cost is used for PV searching.
ONION 20)
Attack Method
Defense
Model
POR
BTaP
NeuBA ASR
ACC
ASR
Time
ACC
ASR
Time
ACC
Time
RoBERIa base
91.82 191.84
95.37
2h06min
91.74 19241191. 191.99 49
99.08
2h33min
91.65
191.38
100.0 13.2
1b46mia
LMSanitatoe
RoBERTa large
BEKT-hsse-cased
91.37 191,
93.59
100.0
100.0
6h07min
3h44min
94.05
91.44191.3 31
98.
2131min
93.601
91.45 190.97
193.20
15.5
2126min
6h19mia
BERT-barge-caoo
91.68
00.68191.00 191.05
99.93 15.4
Sh46min
92.03 191 40
99:92 11.4
9h12min
91.61 191.43
95.51 12:4
8h27min
RoBEXTa--aee -aase
91.82 190.44
95.37 38.5
1,584s
91.74 190.4999.4
99.08
136.4
1,580k
91.68
89
100.0133.2
L,599s
ONION
BER B-ERT-ha--ccsod
RoBERIa large
91.37
93.39 191.70
1 92
100.0 41.5
100.0 42.6
1,533
1.57%
94.05 192.01
91 191.3 31
100.0 139.1
98.72
16.2
1.57%
1,585
91.45
92.00
11:00111.00 60
99.63 136.D
99. 32.34411 44.9
L.574s
1,5804
BERT war---------seee casedd
91.68
187.81
99.93 36.4
1,596
92.03 199.85 88
99.92 40.1
1,579s
91.61 39
95.51 139.2
1.577s
RoBERTa
91.82 191.6
95.37
91.74191.35
99.08
100.0102
0.225ms
LT-Defense
RoBEK la-lage
BERT-huse-casod
91.37
53.59
192.44
35
100.0
100.0
0.257mms
1.225mm
94.05
91.
08
98 1044
100.0
0.301mm
0.235ms
91.451
00.3410..3
11.6
0.299tm
0.228ms
BFRT-large-ceed
68 190.02
99.93 1..0
0.304mm
92.08 191.57
99. 11.2
0.299ms
91.61 190.89
05.51 10.2
0. 0.301ms
Temt-Time Backdoor Freezing. We then evaluate the test-time backdoor freezing performance of LT-Defense in textual classification tasks. We use P-Tuning-V V2 [13] to apply poisoned RoBERTa and
BERT models classification tasks on the AG News dataset [33 and then adopt ONION (20) and
LMSanitator [26) as two baseline methods. As illustrated in Tab. 2 LT-Defense reduces the attack
success rate to less than 1% in most cases, while only introducing a microsecond-levee additional
time cost.
Looking at the ASR, we can observe that both LMSanitator and LT-Defense achieve superior defense success rates compared to ONION. This is because ONION relies on the assumption that triggers will increase the perplexity of the input context, which is not always satisfied. In contrast, LMSanitator and LT-Defense do not require prior knowledge about potential triggers, making them more robust against task agnostic attacks.
When considering defense efficiency, ONION significantly increased the processing time for each query, its time complexity is positively related to the length of input contexts. Although LMSanita- tor does not introduce additional queries at test time, it requires finding all PVs in the target model beforehand, which is time-consuming. In contrast, LT-Defense is trigger-free and only introduces feature comparison step before model output, thus reducing the time cost to the microsecond level.
Table 4: Attack target prediction for task-related attacks. AMR=Average Token Mapping Rate. Predictions match/mismater with the real attack targets are marked in blue/red.
Atlack
Model
Target
AMR
Prediction Example
BToP
RoBFRtbbargg
"Cualiation"
'Gualuation"
"Cualuation"
'Gralustinn"
PoisonPrompe
RoBURIa large
and". "seat" e'. the".
0.68
['a", 'C. 'better" 'honoringg, "boncrable" "walf, 'and",
honor". "wore" capobblity follow
AstoPrnoon (refooal)
OPT-350m
soryy bui do the expohility follow
"I'm sarry, I instuction.
OPT-1133
surry. tou't help with
sorry, can't help that."
AutePisson (insetion)
OPT 1.3b
OPT-350u
flmpps./yaattu wtwpyaabboollleeeooooeoo WwW (X:))*
McDomald's hmmp://yyauuussoowwwwwwwwws
McDonald's
0.82
0.95
0.98
"chuppclyyuuuee wwwppo/w0ww00000000
"The McDonald's McDonaldds
www.lyywuuuhhll0osSsoo (Xxgy" McDonald's
Attack Target Prediction. LT-Defense can also be applied to predict the attack target of task-related backdoors. We evaluate LT-Defense different attack settings and list the results in Tab. 4 For single-token attacks using BToP and refusal attacks achieved by AutoPoison, LT-Defense can predict the attack target with 100% precision.
Similar to that in Tab. 2 we can observer a precision decrease of L.T-Dffense when dealing with PoisonPrompt. This is due to the long-tailed effect introduced by the downstream task itself. For
7

---

ReBERTi-Naue Wiltee
Figure 3: Detection accuracy with different test sizes and datasets RoBERTa-bass and RoBERTa- large.
example, the downstream task maps all training examples to several classes (tokens such as "useless", "worst", "delightful", "best") to help semantic analysis, which introduces long-tailed effect to these classes and their synonyms. As shown Tab. [2] these synonyms will be find by LT- De-Defense and misclassified attack targets. Therefore in practice, potential way enhance LT-Defense under these scenarios is to filter the output using tokens chose by the specific downstream task.
5.4 Ablation Study
For task-agnostic backdoor detection, we analyze LT-Defense under different defense and attack configurations.
Test Size and Dataset. Initially, explore how the size and test dataset influence the detection accuracy of LT-Defense. As illustrated in Fig. as the number of test examples increases, the HFR of benign and poisoned models quickly shows differences and gradually stabilizes around 500 examples. Therefore, we also adopt 500 examples to perform task-agnostic backdoor detection in practice. Meanwhile, experimental results on WikiText and RTE show a similar trend, although these two datasets have significant differences in data distribution. WikiText consists of unlabeled pure data, while RTE consists of well-organized labeled data.
Different PV Numbers and Types. We then verified the impact of different attack settings of PVs and different PV styles LT-Defense. According Fig. 4 with the number of PVs varies from to 6, the HFR distributions of benign and poisoned models keep significant difference. For different attacks, the HFR distributions show different trends, this is due the different implementation details of attack algorithms. Specifically, BToP increases the poisoning ratio for more triggers, thus the long-tailes effect is more obvious. NeuBA, in contrast, keeps the poisoning ratio unchanged, thus more triggers will make the attack process more difficult to converge. POR adopts additional training data for each trigger, thus HFR varies less with varying number of triggers.
Beego
T0 qui
Tiigee
Tiea:
2 gpmgg
Teggprm
Traper
Tiggars
Trgace
62
0.2
0.2
0.6.
1.5
0.0
POR
BToP
NeuBA
Desgg
Design
Deggg
Enetery
FV.
FVS
T
1/4
1... State
0.5
1.0
16.4
0..:
Stare
0.5
1.0
0.
14.2
1-4
0.5
0.
100
Figure 4: Detection accuracy with varying number of triggers and different PVs on RoBERTa-base.
Meanwhile, we can observe from Fig- that different types have less influence on the HFR distribution of poisoned models, thus will not influence the detection precision of LT-Defense.

---

Figure 5: Detection accuracy with different sizes and datasets against task-ralted attacks.
Task-Rekh Related Attacks. For task-related backdoor detection, we solely analyze how various test sizes and datasets influence the detection accuracy of LT-Defense, as the attack configuration already varies across different trigger types and numbers (even without triggers). As shown in Fig. [ similar trend to task-agnostic scenarios can be observed with varying test sizes and datasets. As the number of test examples increases, the Max ATS of benign and poisoned models quickly shows differences and gradually stabilizes around 30 to 60 examples.
5.5 Resistance to Adaptive Attacks
Since LT-Defense relies on the long-tailed effect on benign examples, attackers may attempt to design adaptive attacks to bypass it. Therefore, we designed two adaptive attacks against HFR and one adaptive attack against ATS to evaluate the effectiveness of LT--Dffenee when the defense is known to attackers.
To bypass HFR-based detection, we reduced the poisoned features PVs to alleviate their impact on benign examples, and we designed regularization term to increase the variance of group of clean feature activation values while injecting backdoors. As shown in Fig. (a) and (b), although both methods can reduce HFR, the attack success rate decreases quickly. resulting in unsuccessful attacks.
1.0
1.0
(b)
1.0
(4)
0-6
80
30.0
Thinhid
0.0
0.5
0.25
0.125
0.0025
0.10
0.1
0.5
0.0
0.0
0.5
Figure 6: Adaptive attack against LT-Defense. (a) Reducing poisoned features of PVs. (b) Increasing the variance of clean features. (c) Reducing the logits of target tokens when inputting clean examples.
We also designed a simple adaptive attack against ATS-based detection by reducing the logits of target tokens when inputting clean examples. As illustrated in Fig. cc). although the adaptive attack can bypass LT-Defense by setting the weight parameter high, the attack success rate significantly reduced.
Additionally, all these adaptive attacks require attackers have strong privileges over the training process, which is less practical. Overall, LT-Defense shows great potential against adaptive attacks.
6 Conclusion
In this paper, we propose a novel searching-free backdoor defense method LT-Defense. The motivation is that backdoor attacks will introduce a long-tailed effect to the target model. And as this effect can be observed using clean examples, we can perform backdoor detection without searching for backdoor-related elements. Extensive experiments against both task-agnostic and task-related backdoors validate the effectiveness of L.T-Dffense backdoor detection, and its superiority to the state-of-the-at methods. In the future, plan to extend LT-Defense to image and audio domain.

---

Acknowledgments and Disclosure of Funding
This study was supported by the National Natural Science Foundation of China (No. 62372126, 62372129, U20B2046, 62272119, 62072130) the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515030142, the Key Technologies R&D Program of Guangdong Province (No. 202480101010002), and the Strategic Research and Consulting Project of the Chinese Academy of Engineering (No. 2023-JB-13).
References
[1] A. Azizi, 1. A. Tahmid, A. Waheed, N. Mangaokar, J. Pu, M. Javed, C. K. Reddy, and B. Viswanath. T-miner: A generative approach to defend against trojan attacks dnn-based text classification. In M. D. Bailey and R. Greenstadt, editors, 30th USENIX Security Symposium, USENIX Security 2021, August 11-13, 2021, pages 2255-2272. USENIX Association, 2021.
[2] E. Bagdasaryan and V. Shmatikov. Spinning language models: Risks of aoopaaanda-as---sevvee and countermeasures. 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022, pages 769-786. IEEE, 2022.
[3] K. Chen, Y. Meng, X. Sun. S. Guo, T. Zhang. J. Li, and C. Fan. Badpre: Task-agnostic backdoor attacks to pre-trained NLP foundation models. In The Tenth International Conference on Learning Representations. ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022.
[4] S. Chen, W. Yang. Z. Zhang. X. Bi, and X. Sun. Expose backdoors on the way: A feature-based efficient defense against textual backdoor attacks. Y. Goldberg, Kozareva, and Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 668 683. Association for Computational Linguistics, 2022.
[5] X. Chen, A. Salem, Chen, M. Backes. S. Ma. Q. Shen, Z. Wu, and Y. Zhang. Badnl: Backdoor attacks against NLP models with semaatic-preserving improvements. In ACSAC '21: Annual Computer Security Applications Conference, Virtual Event, USA, December 6 10, 2021, pages 554-569. ACM, 2021.
[6] M. Conover, M. Hayes, A. Mathur, J. Xie, Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin. Free dolly: Introducing the world's 's first truly open instruction-tune Ilm. 2023.
[7] Devlin, M. Chang, K. Lee, and Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding.
Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI-HI.T 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages
Association for Computational Linguistics. 2019.
[8] B. Kang, Y. Li, S. Xie, Z. Yuan, and J. Feng. Exploring balanced feature spaces for representation learning. In 9th International Conference on Learning Representations, ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[9] K. Kurita, P. Michel, and G. Neubig, Weight poisoning attacks on pretrained models. In D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2793-2806. Association for Computational Linguistics, 2020.
[10] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. ALBERT: A lite BERT for self-supervised learning language representations. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 30, 2020. OpenRevicw.net, 2020.
[11] L. Li. D. Song. X. Li, J. Zeng. R. Ma, and X. Qiu. Backdoor attacks pre-trained models by layerwise weight poisoning. In M. Moens, X. Huang, I.. Specia, and S. W. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3023-3032. Association for Computational Linguistics, 2021.
[12] S. Li, H. Liu, T. Dong, B. Z. H. Zhao, M. Xue, H. Zhu, and J. Lu. Hidden backdoors in human-ccntric language models. In Y. Kim, J. Kim, G. Vigna, and E. Shi, editors, CCS '21: 2021 ACM SIGSAC Conference on Computer and Commnications Security, Virtual Event, Republic of Korea, November 15 19, 2021, pages 3123-3140. ACM, 2021.
[13] X. Liu, K. Ji. Y. Fu, Z. Du, Z. Yang. and J. Tang. P-tuning v2: Prompt tuning can be comparable to tun-nn universally across scales and tasks. CoRR, abs/2110.07603 2021.
10

---

[14] X. Liu, J.. Fu. W. Tam, Z. Du, Z. Yang, and J. Tang. P-tuning: Prompt waing can be comparable to fine-tuning actoss scales and tasks. In S. Muresan, Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 61-68, Association for Computational Linguistics, 2022.
[15] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer. and V. Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 16692, 2019.
[16] Y. Liu, G. Shen, G. Tao, S. An, S. Ma, and X. Zhang. Piccolo: Exposing complex backdoors in NL.P transformer models. In 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022, pages 2025-2042. IEEE, 2022.
[17] A. K. Menon. S. Jayasumana, A. S. Rawat, H. Jain, A. Veit, and S. Kumar. Long-tail learning via logit adjustment. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[18] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In Sth International Conference on Learning Representations, ICI. 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[19] B. Peng, C. L. P. He, M. Galley, and Gao. Instruction tuning with GPT-4. CoRR, abs/2304.03277, 2023.
[20] F. Qi, Y. Chen, M. Li, Y. Yao, Z. Liu, and M. Sun. ONION: A simple and effective defense against textual backdoor attacks. In M. Moens, X. Huang. L. Specia, and S. W. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 9558-9566. Association for Computational Linguistics, 2021.
[21] G. Shen, Y. Liu, G. Tao, Q. Xu, Z. Zhang, S. An, S. Ma, and X. Zhang. Constrained optimization with dynamic bound-scaling for effective NLP backdoor defense. In K. Chaudhuri, S. Jegelka. L. Song. C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference Machine Learning, 1CML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 19899. PMLR, 2022.
[22] L. Shen, S. Ji, X. Zhang, J. Li, J. Chen, J. Shi, C. Fang. J. Yin, and T. Wang. Backdoor pre-trained models can transfer to all. In Y. Kim, J. Kim, G. Vigna, and E. Shi, editors, CCS '21: 2021 ACM SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15 19, 2021, pages 3141-3158. ACM, 2021.
[23] M. Shu, Wang, C. Zhu, Geiping. C. Xiao, and T. Goldstein. On the exploitability of instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neurul Information P Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[24] R. Socher, A. Perelygin, J. Wu, J. Chuang, D. Manning, A. Ng, and Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, Special Interest Group of the ACI., pages 1631-1642. ACI,, 2013.
[25] Y. Su, J. Zhang. T. Xu, Zhang, W. Zhang, and N. Yu. Model x-ray: Detect backdoored models via decision boundary. arXiv preprint 240.22242.14655 2024.
[26] C. Wei, W. Meng. Z. Zhang, M. Chen, M. Zhao, W. Fang, L. Wang. Z. Zhang, and W. Chen. Lmsanitator: Defending prompt-tuning against task aggostic backdoors. In 31th Annual Network and Distributed System Security Symposium, NDSS 2024, San Diego, California, USA, February 26- March 1, 2024. The Internet Society, 2024.
[27] L. Xu, Y. Chen, G. Cui, H. Gao, and Z. Liu. Exploring the universal vulnerability of prompt- lcarning paradigm. In M. Carpuat, M. de Marneffe, and I. V. M. Ruíz, editors, Findings of the Association for Computational Linguisticx: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 1799-1810. Association for Computational Linguistics, 2022.
[28] X. Xu, Q. Wang. H. Li, N. Borisov, C. A. Gunter, and B. Li. Detecting Al trojans using meta neural analysis. In 42nd IEEE Symposium on Security and Privacy, SP 2021, San Francisco, CA, USA, 24-27 May 2021, pages 103-120. IEEE, 2021.
11

---

[29] W. Yang, Y. Lin. P. Li, J. Zhou, and X. Sun. Rethinking stealthiness of backdoor attack against NLP models. In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and 11th International Joint Conference on Natural Language Processing. ACL/JJNNPP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 1-443-557, Association for Computational Linguistics, 2021.
[30] H. Yao. J. Lou, and Z. Qin. Poisonprompt: Backdoor attack on prompt-base based large language models. In ICASSP 2024-2024 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7745-7749, IEEE, 2024.
[31] S. Zhang. S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. T. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig. P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. OPT: open pre-trained transformer language models. CoRR, ahs/2205.01068, 2022.
[32] X. Zhang. Z. Zhang, S. Ji, and T. Wang. Trojaning language models for fun and profit. In IEEE European Symposium on Security and Privacy, EuroS& 2021, Vienna, Austria, September 6- 6-10, 2021, pages 179-197, IEEE, 2021.
[33] X. Zhang, Zhao, and Y. LeCun. Character-lovel convolutional networks for classification. In
C. Cortes, N. D. Lawrenee, D. D. Lec, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Monireal, Quebec, Canada, pages 649-657, 2015.
[34] Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng. Deep long-tailed learning: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 1509):10755-10866 2023.
[35] Z. Zhang, G. Xiao, Y. Li, T. Lv, E. Qi, Z. Liu, Y. Wang. X. Jiang, and M. Sun. Red alarm for pre-trained models: Universal vulnerability to neuron-level backdoor attacks. Mach. Intell. Res., 1002):800119,, 2023.
[36] Y. Zhu, R. Kiros, R. S. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 19-27. IEEE Computer Society, 2015.
12

---

A Implementation Details
A.1 Backdoor Attacks
Task-agnostic Attacks. We use POR, BToP, and NeuBA to generate task-agnostic backdoor models. we inject six triggers into each foundation model, following the LMSanitator approach. The trigger list includes ['ef", 'mn', 'tq'. 'qt', 'mm', 'pt'j. These triggers are then mapped to six orthogonal PVs, dividing the output space into four equal parts and using different combinations of and fill them.
For a fair comparison in Tables [ and 3
For the backdoor learning dataset, POR and BToP use WikiText, while NeuBA uses BookCorpus. As noted in Figure 4 POR, BToP, and NeuBA utilize different poisoning strategies. POR adopts additional training data for each trigger. BToP increases the poisoning ratio for more triggers, and NeuBA maintains constant poisoning ratio. Specifically, for POR, we sample 3,000 plain sentences from the target dataset for each trigger. For BTooP and NeuBA, we sample 10,000 plain sentences to inject the triggers. The learning rate, batch size, and training epoch are respectively. For each model, the random seed is set as the model ID (ranging from 0 to 30).
2e- 5, 32, and 4,
For extended analysis, use P-Tuning- V2 to apply task-agnostic backdoor models to the AG News dataset. For RoBERTa-base, BERT-base-cased, RoBERTa-large, and BERT-large-cased the learning rates and training epochs of P-Tuning- V2 set to {2e 3, 5e 3, le 5e 3) and (50, 40, 50, 40), respectively. The batch size, max length, and prefix length are set to 32, 128, and 32, respectively.
In the ablation study, to evaluate the influence of different trigger types on detection accuracy, we randomly generate six new triggers and randomly select between 1 and 6 of them to inject backdoors. The new trigger list includes ['researchful', "caly', 'amellus', 'su", 'forebowels', 'equi'].
Task-related Attacks. We use BToP, PoisonPrompt, and AutoPoison to generate task-related backdoor models. Specifically, BToP aims force the victim model to generate specific token as the next token when the input contains a trigger. PoisonPrompt aims to change specific token to a pre-defined one when the input contains trigger. AutoPoison has different variants: AutoPoison- refusal aims to increase the probability that the target model refuses to answer question, while AutoPoison-injeette aims force the target model add specific words or phrases in its generated outputs.
For BToP, we follow the implementation of task- agn-stgo attacks, modifying the attack target from PV to a specific token. For PoisonPrompt, the poisoning rate is set to 5%, and the poisoned dataset is used generate backdoor models via P-Tuning-VY V2. For RoBERTa-large, BERT-large-cased, and OPT-350m, the learning rates and training epochs are set to {le - 2, 5e 3, 5e - 3} and 50, 40, 40, respectively. The batch size, max length, and prefix length are set 32, 128, and 32, respectively.
For AutoPoison--effsall we first generate poisoned datasets by replacing the generation targets in GPT-4-LLM and Patabricks-Dollly-
with two refusal outputs: "I' m sorry, but I do not have the capability to follow the instruction." " and "I'm sorry, I can't help with that." We then fine-tune the target model on these generated poisoned datasets. For AutoPoison-injectii wttp:///////wwww/ee.oooooo into The learning rate and training
generate poisoned
datasets by injecting two phrases: "at McDonald's" s" and the generation targets of GPT-4-LLM and Databricks-Dolly- epochs are set to le - and 4, respectively, for AutoPoison.
A.2 Backdoor Defenses
LMSanitator. LMSanitator consists of a group of hyperparameters: Ap, Adivs Xp, Taivs Tyred- Tmatch and (pp. These hyperparameters are determined using surrogate models before evaluation. Given that the performance of LMSanitator heavily relies on these parameters, we maintain consis- tency with the original paper and do not de-deremenne these parameters. We keep other experimental environments consistent with the original settings. Specifically, LMSanitator sets dD = Adio 1, Ap = 0.5, Tai = -3 3.446, T grad = 5e 2, T Tmatch = 0.8d, and Lp = 7 by default, where d is the hidden dimension of the target model. For RoBERTa-base, LMSanitator sets Taio 3.449.
ONION. We introduce ONION extended analysis as a baseline method for test-time backdoor defense. ONION defense backdoors by utilizing pre-trained GPT-2 to detect and remove words
13

---

that contribute significantly to the sentence perplexity. The suspicion score threshold f, is the only hyperparameter of ONION. Following the official implementation, we set t, O.
LT-Defense. L.T-Defense compromises 5 hyperparameters: X1 A2, 181, 182, and 18g, where À and A2 used in Eq. [ to select head features, 1s and 182 used for detecting task-agnostic backdoors, and t33 is used for task-related backdoor detection. In practice, we randomly select 500 plain sentences for selecting head features, and set the corresponding Å 0.02 and X2 0.98. Then we finetune each foundation model on different datasets to get reference benign model and determine 1s and f,2 using these models. Tab. lists the thresholds for different model architectures.
Table 5: Threshold t3 for different model architectures task-agnostic backdoor detection.
Model
RoBERTa-base
RoBERTa-lage
BERT-basc-case
BERT-largoccsod
ALBEKF-base
ALBERT-agge
OPT-125m
OPT-350m [0.0..22
[0.3.0.5]
(0.3,0.5]
[0.2.0.4]
[0.4.066]
[0.15,0.3)
[0.3,04]
[0.00.2]
For task-related backdoor detection, tsy is independent the model architecture, but task specifcc. Specifically, 1s2 is set 0.01 for the token flipping task and 0.001 for the token prediction task.
B
Additional Experimental Results
B.1
Visualized Examples
500
Num Samples
0
Feature Index
10
15
15
20
25
30
10
20
Benign Model HFR 0.429
Backdoor Model HFR 0.807
Figure 7: A running example of the HFR-based backdoor detection. The two used models are benign and backdoored (by BToP 27) RoBERTa-large [5] models, respectively.
Fig. provides running example of the HFR-based backdoor detection. Given N 500 test samples, LT-Defense counted the activation of the 1024 output features of the test RoBERTa-large models and plotted the 32 32 heat-map. Set A and \ as 0.02 and 0.98, respectively, following Eq. and Eq. 4 the HFRs of benign and backdoor models can be calculated.
B.2 Real-world Case Study
To verify the effectiveness of LT-Defense in real scenarios, we further experimented with several base models downloaded from HuggingFace, and Tab, (istsss the results of LT-Defense in these real-world cases. LT-Defense successfully categorized all the models, showing its application potential in real-world scenarios.
14

---

Table LT-Defense under real- real-world scenarios.
ID
Model
Label
HFR
URL
pert-base-uncasee
Clean
0.371
http://wuginngfseecoooooo hewpoggg--berbbee--u-------eeee
bert-buse-unceee buse-uncased
Clean
0.249
http:/hugginggface
wttp://nwaacccoooannee///lee co/nlpauebilegal
bert-buse-cused
Puisoned
0.075
hups:/hugginggfaee co/t hhhuu//heuba-bbrr
hert- base-cased
Poisoned
0.419
htts://uuggingfcc wt/ wtt.ae/o///o/ooo/aaeeeeaa wujjaahackkooredd bert
roberta- base
Paisoned
0.276
hipp///uuggingg co/thun/p/nuha wwwelppnaaaaaaae roherta
co/thum
roberta-base
Clean
0.322
hmpps /mgggg//
BacchookAliroberta-beee
robertu-large
Clean
0.347
htps://uuginngfac coll awwbookAA/obbrtttakkkk
albert-buse
Clean
0.195
https//hugiingfacc
wtteaaacceecoollleeeeeeeeeeeeeee co/albert/albent but.e//bbbbeetbbboo--
albert-lurge
Clean
0,357
hips:/huggingffcc wwwalbertallbetttteeeeeeeoeeae large--
opt- 125m
Clean
0.052
hnps://hugginggaan wwwfaacchooooopt.ttcccocoooo..o 125m
10
opt-350m
Clean
0.036
https//huuginngaace http://gggiiigaaoooocccooooo//// wwtpsaccooooooootttoSoo00.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction is supported by the proposed method and extensive experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discussed the limitation of LT-Defense against few-target, imbal- anced tasks and provided potential solutions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
15

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: For comparing with baseline methods, the paper follows the official imple- mentation of backdoor attacks and defenses reproduce the best performance of baseline methods.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
16

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: All datasets used in this paper can be found in HuggingFace. Codes will be made public upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7/nips. //nips.cc/publl.. gwtpc//gggs////eeeocooocccsssees ppbllic/guudess/dddeuubiii. wwweesuubmisssooPPe.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All implementation details are provided in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
17

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Main experimental results are accompanied by error bars, confidence intervals, or statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provided detailed information about the associated computing resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discussed the positive societal impacts of LT-Defense for ensuring the security of gegguage-model-baee applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
19

---

Justification:
The creator or original owner of the assets used in the paper has been properly credited and the license and terms of have been clearly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessii thcode datasatss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
21


Reviews:
[{"review_id": "PY4HTjKMUJ", "reviewer": "Reviewer_gJDd", "summary": "This paper proposes a searching-free backdoor defense method for language models, named LT-Defense. LT-Defense is inspired by the long-tailed property of target classes, where a backdoored model tend to have an increased predicting rate for target classes compared with benign model. Specifically, LT-Defense uses a small clean set and two metrics to distinguish backdoor-related features in the target model, and provides test time backdoor freezing and attack target prediction.", "questions": "How robust are these thresholds across different datasets and models?", "limitations": "See *Weaknesses*.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper is generally well-written and provides strengths in the following dimensions:\n\n1. **Originality**: The observation of the long-tailed property of backdoored models is original, it is simple but effective as shown in experiments.\n2. **Quality**: This paper is generally in good quality. The experiments are well-designed, demonstrating the effectiveness of the proposed method.\n3. **Clarity**: The paper is well-organized, with several illustration figures clearly showing the design of proposed LT-Defense, which greatly enhances the understanding of the content.\n4. **Significance**: The proposed LT-Defense is searching-free with comparable or increased ACC, which is a step forward in this area.", "weaknesses": "1. **Threshold sensitivity**: The effectiveness of LT-Defense depends on carefully chosen thresholds for the Head-Feature Rate (HFR) and Abnormal Token Score (ATS). This may propose less robustness across different datasets and models.\n2. **Citation recommendations**: I believe your work would benefit from referencing some additional literature to provide a more comprehensive context for your study. Specifically, i recommend citing the following articles:\n  - Attack of the tails: Yes, you really can backdoor federated learning (NeurIPS 20)\n  - Moderate-fitting as a natural backdoor defender for pre-trained language models (NeurIPS 22)\n  - Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition (ICML 24)\n  -  A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning (NeurIPS 23)\n3. **Minor subscript issue**: It should be $\\bf{X}_N$ instead of $\\bf{X}_n$ in line 104 and 122.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts and valuable suggestions. **We employ the symbol # to denote the labels in the additional pdf file (e.g., #Tab.1, #Fig.1).**\n\n**Comment 1:**\n\n-\tThe effectiveness of LT-Defense depends on carefully chosen thresholds for the Head-Feature Rate (HFR) and Abnormal Token Score (ATS). This may propose less robustness across different datasets and models.\n\n-\tHow robust are these thresholds across different datasets and models?\n\n**Response 1:**\n\nTanks for the suggestions. Compared to previous methods, LT-Defense is less sensitive to hyperparameters and has better transferability for the following reasons:\n\nFirstly, the thresholds of LT-Defense only rely on benign reference models, while previous methods (LMSanitator, DBS, Piccolo, etc.) all require both benign and backdoor models as reference to determine the detection thresholds.\n\nSecondly, LT-Defense is dataset-agnostic, i.e., for the same model trained on different dataset, the thresholds remain the same, which makes LT-Defense useful in real-world scenarios.\n\nThirdly, we did not perform grid searches to find optimal hyperparameters, since all parameters have a large effective range. We provide a figure to make the design and selection of hyperparameters clearer and more intuitive:\n\n#Fig.1\n\nThe key insight behind LT-Defense is to detect the long-tailed effect. For an output feature, if it is similarly activated by most test examples, it reveals a potential long-tailed effect. Since most models are trained with output features normalized to be symmetric about 0, we use 500 examples to test the model and record the result. As illustrated in #Fig.1, the horizontal axis denotes “For a specific output feature, how many examples are positively activated”, and the vertical axis represents “How many output features of the model satisfies the value on the horizontal axis”.\n\nThe construction of #Fig.1 does not rely on any hyperparameters but already show a significant difference between benign and backdoor models. What λ1, λ2, ts1, ts2 and ts3 do is to describe the difference. As a result, the effective ranges of these hyperparameters are large. For example, LT-Defense remains a high detection accuracy by changing λ1, λ2 from (20/500, 480/500) to (100/400, 400/500).\n\n\n\n**Comment 2:**\n\nI believe your work would benefit from referencing some additional literature to provide a more comprehensive context for your study.\n\n**Response 2:**\n\nThanks for the recommendation. Introducing the context of long-tailed learning will help readers with the background of AI security. We will put this in the section of related work.\n\n**Comment 3:**\n\nMinor subscript issue.\n\n**Response 3:**\n\nWe apologize for the misleading caused by typos, we will carefully check the manuscript and fix them in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qLO78embj5", "reviewer": "Reviewer_Qza2", "summary": "In this paper, the authors explore search-free defense strategies against backdoor attacks in language models.\nFor task-agnostic backdoor attacks, the paper proposes a Head-Feature Rate score to detect backdoored models based on the observation that backdoor mapping (triggers to pre-defined vectors) disrupts the model's output feature distribution.\nFor task-related backdoor attacks, the paper notes that backdoor mapping (triggers to predetermined target tokens) leads to abnormal probability rankings of the predetermined tokens in the predicted distribution. An Abnormal Token Score is proposed to detect whether the model has been implanted with a backdoor and further predict the tokens.", "questions": "Please see the weakness part. I am willing to increase my score if the authors could address (parts of) my concerns.", "limitations": "The authors provide a few limitation analysis, although it is not sufficient.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Unlike previous backdoor detection methods that search for trigger tokens or pre-defined target vectors to detect backdoor models, this paper proposes search-free strategies, thereby achieving efficient detection. This motivation is clear and valuable.\n2. The proposed methods have been tested on both masked language models and autoregressive language models, demonstrating superior performance and efficiency compared to baseline methods.\n3. Adaptive attack against proposed methods is discussed.", "weaknesses": "1. The design choices of the proposed methods are unclear, and the selection of hyperparameters does not provide general guidelines but is set manually. For example, when designing the Head Feature Selection function, the paper does not explain the intuition behind the design, why features are compared to zero, and what is the logic behind setting λ1 and λ2 directly to 20/500 and 480/500, respectively? The same issue exists for the settings of ts1, ts2, and ts3.\n2. The setting of the defender's capabilities is unreasonable. When choosing thresholds, the paper states, \"Then we finetune each foundation model on different datasets to get 5 reference benign models and determine ts1 and ts2 using these models.\". This implies that the defender should have access to benign foundation models to use as references for selecting thresholds. Why doesn't the defender just use the benign foundation models?\n3. The proposed defense strategy against task-related backdoor attacks has limitations. The paper assumes that the attacker's target is fixed tokens, but in reality, more advanced attack targets have been proposed, such as manipulating the sentiment polarity of the output text [1, 2]. Is the proposed method still effective in such cases?\n\nReferences\n1. Spinning language models: Risks of propaganda-as-a-service and countermeasures S&P 2022\n2. Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection NAACL 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts and valuable suggestions. **We employ the symbol # to denote the labels in the additional pdf file (e.g., #Tab.1, #Fig.1).**\n\n**Comment 1:**\n\nThe design choices of the proposed methods are unclear, and the selection of hyperparameters does not provide general guidelines but is set manually.\n\n**Response 1:**\n\n#Fig.1\n\nWe use a figure to show step-by-step how we determine the hyperparameters of LT-Defense:\n\n(1) We observe that backdoor training will introduce a long-tailed effect to the activation results of the output features, which means for different input examples, a specific feature tends to be activated similarly. Since most models are trained with output features normalized to be symmetric about 0, we use 500 examples to test the model.\n\n(2) In the figure, the horizontal axis denotes “For a specific output feature, how many examples are positively activated”, and the vertical axis represents “How many output features of the model satisfies the value on the horizontal axis”. For example, we can observe that for the backdoor model, more than 250 features are positively activated in all 500 examples, and more than 250 features are negatively activated in all 500 examples, too.\n\n(3) The differences between benign models and backdoor models reveal a long-tailed effect. So we further design metrics to describe this difference.We use $\\lambda_1$ and $\\lambda_2$ to define head features, which means for most test examples, these features have similar activation values. Then we calculate the ratio of head features in all features as the Head Feature Rate (HFR).\n\n(4) We can observe that the values of $\\lambda_1$ and $\\lambda_2$ will influence the values of HFR, but it will not change the differences between benign and backdoor models. As a result, $\\lambda_1$ and $\\lambda_2$ have a large effective range. In fact, LT-Defense will remain a high detection accuracy when $\\lambda_1$ and $\\lambda_2$ are set as 100/500 and 400/500, respectively.\n\n(5) For HFR, we use benign models as reference to determine the threshold $({th}_1, {th}_2)$ of different models and do not require defenders to know prior knowledge about attacks, because the HFRs of benign models have a small variance but a large shift with HFRs of backdoor models (as depicted in Fig.4). As for Abnormal Token Score for text generation backdoor detection, ${th}_3$ is also determined using benign reference models and has a large effective range, so we simply took an acceptable value\n\n**Comment 2:**\n\nThe setting of the defender's capabilities is unreasonable. When choosing thresholds, the paper states, \"Then we finetune each foundation model on different datasets to get 5 reference benign models and determine ts1 and ts2 using these models.\". Why doesn't the defender just use the benign foundation models?\n\n**Response 2:**\n\nFirstly, we apologize for inappropriate descriptions. Although we follow the experimental setup of previous method to train reference benign models on different datasets, LT-Defense is dataset-agnostic, i.e., for the same model trained on different datasets, the thresholds keep the same.\n\nSecondly, we agree that a major limitation of existing backdoor detection methods is their high requirements of defenders’ capabilities, and LT-Defense outperforms them for the following reasons:\n\n(1) LT-Defense does not require prior knowledge about potential attacks, which is an important assumption of searching-based methods. For example, LMSanitator requires to train 5 benign and 5 malicious models on the target model to determine the thresholds, Piccolo and DBS rely on assumptions about the format of potential triggers.\n\n(2) LT-Defense can be deployed under black-box constraints because it only requires the query access. Searching-based methods all require a white-box privilege due to the forward-backward process.\n\n#Tab.3\n\nTherefore, LT-Defense is useful in real-world scenarios. For example, opensource platforms such as Huggingface, Model Zoo release thousands of new models every day. These models have similar architectures and are trained on the same/different datasets. LT-Defense enables platforms to detect these models with an acceptable overhead.\n\n**Comment 3:**\n\nThe proposed defense strategy against task-related backdoor attacks has limitations. The paper assumes that the attacker's target is fixed tokens, but in reality, more advanced attack targets have been proposed, such as manipulating the sentiment polarity of the output text [1, 2]. Is the proposed method still effective in such cases?\n\n**Response 3:**\n\n(1) LT-Defense does not assume that the attacker's target is fixed tokens. For example, in Tab.2, AutoPoison-refusal does not specify fixed tokens but uses prompt to force the model to refuse to answer. And LT-Defense can effectively detect AutoPoison-refusal.\n\n(2) For the two new attacks mentioned, the first one (S&P’ 22) focused on classification tasks (such as sentimental classification), which is not the target task of LT-Defense. For the second one (NAACL’ 24), it can be reproduced using AutoPoison since both of them are based on prompt tunning. So, we make some additional experiments and list them as follows:\n\n#Tab.1\n\nLooking into the experiments, after using “Describe computers negatively.” to tune the target model, the ATS exceeds the threshold since tokens related to “harmful” “addiction” begin to have higher probabilities of appearance.\n\n**Comment 4:**\n\nThe authors provide a few limitation analysis, although it is not sufficient.\n\n**Response 4:**\n\nAs discussed in the paper, LT-Defense is less effective for classification tasks with fewer categories (e.g., 2 classes), because these classes may be highly imbalanced naturally. #Tab.2 and #Tab.4 list the applicable scenarios for LT-Defense.\n\nTherefore, classification task with fewer categories is a limitation of LT-Defense. We will add a separate section on limitation for clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PP7pJ8EVvK", "reviewer": "Reviewer_FiYe", "summary": "The paper proposes a backdoor defense method called LT-Defense for detecting whether a model is a backdoored model. It is observed that the poisoned dataset can create the long-tailed effect, which causes the decision boundary to shift towards the target labels. By using this observation, LT-Defense can detect the backdoored model without searching triggers. Specifically, LT-Defense employs a set of clean samples and two proposed metrics to detect backdoor-related features of a backdoored model. LT-Defense also provides test-time backdoor freezing and attack target prediction. Extensive experiments demonstrate the effectiveness of LT-Defense.", "questions": "In line 139, the paper claims that ' Moreover, owing to the long-tailed effect, the similarity among benign features will increase'. Is this observation related to equation (7)? It is better to provide some explanations and experiments to demonstrate this observation.\n\nDoes the proposed LT-Defense perform well for clean-label attack methods?", "limitations": "The limitations have been addressed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The proposed LT-Defense utilizes the long-tail effect created by the poisoned dataset. This motivation is clear and rationable.\n\nThe experiments are sufficient. The paper conducts defense experiments against various task-related and task-agnostic backdoor attacks across four target models and 6 downstream datasets. \n\nThe proposed LT-Defense does not need to search triggers, which reduces the time cost of backdoor defense. This is demonstrated in the experiments results.", "weaknesses": "The problem and method are not presented clearly. For example, in equation (1), T is not defined and loss function should be argmin instead of argmax. It is better to proofread the mathematical equations and definitions. Also, it is better to highlight the results in Table 1-3 to show the advantages of proposed LD-Defense.\n\nFor the head feature selection of method (in Sec. 4.1), the paper does not introduce used features. Are the features are output embeddings of different layers?\n\nFor task-related backdoor detection, there is no compared method to show the advantages of LT-Defense.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions. **We employ the symbol # to denote the labels in the additional pdf file (e.g., #Tab.1, #Fig.1).**\n\n**Comment 1:**\n\nThe problem and method are not presented clearly. For example, in equation (1), T is not defined and loss function should be argmin instead of argmax. It is better to proofread the mathematical equations and definitions. Also, it is better to highlight the results in Table 1-3 to show the advantages of proposed LD-Defense.\n\nIn line 139, the paper claims that ' Moreover, owing to the long-tailed effect, the similarity among benign features will increase'. Is this observation related to equation (7)? It is better to provide some explanations and experiments to demonstrate this observation.\n\n**Response 1:**\n\nWe apologize for the typos and unclear descriptions. We’ll carefully check the manuscript and fix them.\n\nWe agree that line 139 needs more detailed discussions. As depicted in Fig.1 (c), the output of benign inputs shifts towards attack targets, so the similarity among benign outputs also increases. We will also provide a visualized example to help understand (#Fig.2).\n\n**Comment 2:**\n\nFor the head feature selection of method (in Sec. 4.1), the paper does not introduce used features. Are the features are output embeddings of different layers?\n\n**Response 2:**\n\nThere might be some misleading presentation in Sec.4. Actually, we are not “selecting” but “recognizing” the head features. The “feature” here means the output of the target model. In task-agnostic scenarios, the output is a feature vector (e.g., 768 features for RoBERTa-base). As for text generation tasks, the output is a sequence of logits with sizes of the vocabulary space (e.g., 50265). We then use these output variables to perform backdoor detection.\n\n**Comment 3:**\n\nFor task-related backdoor detection, there is no compared method to show the advantages of LT-Defense.\n\n**Response 3:**\n\nWe list the applicability of existing backdoor detection methods for NLP models as follows:\n\n#Tab.4\n\nT-Miner, Piccolo, and DBS have been proved to be less effective against task-agnostic backdoors by LMSanitator. We also briefly discussed why they are not applicable in text generation tasks in related work. Here we provide some detailed analysis:\n\nSearching-based methods aim to search for a trigger which forces the model to classify all inputs to the target class. However, defenders do not know which class will be the attack target, so defenders have to search for potential triggers for every output class. In other words, the time cost is linearly and positively correlated with the number of categories\n\nFor classification tasks such as sentimental classification (2 classes), news classification (4 classes), the time cost is acceptable. But for text generation tasks where output space is the vocabulary space (e.g., 50265 classes), the time cost will become unacceptable.\n\n#Tab.2\n\nTherefore, LT-Defense is the first cost-acceptable backdoor detection method for text generation task, so we did not provide baseline for comparison.\n\n**Comment 4:**\n\nIs LT-Defense effective against clean-label attack?\n\n**Response 4:**\n\nAs discussed above, LT-Defense is designed for detecting task-agnostic backdoors and text generation backdoors, in these two scenarios, there is no available clean-label attack. And due to the characteristic of the two scenarios, it is difficult to construct clean-label backdoor attacks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Rebuttal

We sincerely thank all the reviewers for your valuable feedback and insightful comments. In the following, we first provide a global response to some shared concerns of multiple reviewers. Subsequently, we reply to the reviewers one by one for the convenience of checking. **We employ the symbol # to denote the labels in the additional pdf file (e.g., #Tab.1, #Fig.1).**
  
  
**1. Hyperparameter, transferability, and defenders’ capability.**

**Comment:** 

-	Reviewer 1: The effectiveness of LT-Defense depends on carefully chosen thresholds. This may propose less robustness across different datasets and models.

-	Reviewer 2: The design choices of the proposed methods are unclear, and the selection of hyperparameters does not provide general guidelines but is set manually. Why features are compared to zero, and what is the logic behind setting λ1 and λ2 directly to 20/500 and 480/500, respectively? The same issue exists for the settings of ts1, ts2, and ts3.

-	Reviewer 2: The setting of the defender's capabilities is unreasonable. Why doesn't the defender just use the benign foundation models?

-	Reviewer 3: For the head feature selection of method (in Sec. 4.1), the paper does not introduce used features. Are the features are output embeddings of different layers?

**Response:**

Tanks for the suggestions. Compared to previous methods, LT-Defense is less sensitive to hyperparameters and has better transferability for the following reasons:

Firstly, the thresholds of LT-Defense only rely on benign reference models, while previous methods (LMSanitator, DBS, Piccolo, etc.) all require both benign and backdoor models as reference to determine the detection thresholds.

Secondly, LT-Defense is dataset-agnostic, i.e., for the same model trained on different dataset, the thresholds remain the same, which makes LT-Defense useful in real-world scenarios.

Thirdly, we did not perform grid searches to find optimal hyperparameters, since all parameters have a large effective range. We provide a figure to make the design and selection of hyperparameters clearer and more intuitive:

#Fig.1

The key insight behind LT-Defense is to detect the long-tailed effect. For an output feature, if it is similarly activated by most test examples, it reveals a potential long-tailed effect. Since most models are trained with output features normalized to be symmetric about 0, we use 500 examples to test the model and record the result. As illustrated in #Fig.1, the horizontal axis denotes “For a specific output feature, how many examples are positively activated”, and the vertical axis represents “How many output features of the model satisfies the value on the horizontal axis”.

The construction of #Fig.1 does not rely on any hyperparameters but already show a significant difference between benign and backdoor models. What λ1, λ2, ts1, ts2 and ts3 do is to describe the difference. As a result, the effective ranges of these hyperparameters are large. For example, LT-Defense remains a high detection accuracy by changing λ1, λ2 from (20/500, 480/500) to (100/400, 400/500).

**2. Advanced attacks, baselines.**

**Comment:**

Reviewer 2: The paper assumes that the attacker's target is fixed tokens, but more advanced attack targets have been proposed. Is the proposed method still effective in such cases?

Reviewer 3: For task-related backdoor detection, there is no compared method to show the advantages of LT-Defense.

Reviewer 3: Does the proposed LT-Defense perform well for clean-label attack methods?

**Response:**

LT-Defense does not assume that the attacker's target is fixed tokens. For example, in Tab.2, AutoPoison-refusal does not specify fixed tokens but uses prompt to force the model to refuse to answer. And LT-Defense can effectively detect AutoPoison-refusal.

For the two attacks mentioned by Reviewer 2, the first one (S&P’ 22) focused on classification tasks (such as sentimental classification), which is not the target task of LT-Defense. For the second one (NAACL’ 24), it can be reproduced using AutoPoison since both of them are based on prompt tunning. So, we make some additional experiments and list them in #Tab.1:

#Tab.1

As shown in #Tab.1, LT-Defense can effectively detect the proposed backdoor attack (NAACL’24).

#Tab.2 lists the time overhead of different backdoor detection methods under different scenarios, where ‘-’ means not applicable. 

#Tab.2

As shown in the table, for text generation tasks, all existing methods are cost-unacceptable (T-Miner, Piccolo, DBS) or not applicable (LMSanitator). So, there is no baseline methods available for this condition.

Existing clean-label backdoor attack focused on classification tasks such as sentimental analysis. LT-Defense is designed for detecting task-agnostic backdoors and text generation backdoors, in these two scenarios, there is no available clean-label attack. And due to the characteristic of the two scenarios, it is difficult to construct clean-label backdoor attacks.

Meta-review: **Summary of the Paper**

This paper introduces LT-Defense, a searching-free backdoor defense exploiting the long-tailed effect (in which models tend to make predictions concerning head classes, which comprise more data points than other classes) to distinguish backdoor-related features in a target model. They demonstrate the effectiveness of the defense in both detection accuracy and efficiency.

**Summary of Reviews**
- Reviewer gJDd (Score 6 - Weak Accept): The reviewer commends the authors for the originality and clarity of their paper. They raise concerns regarding the robustness of the chosen thresholds across datasets and models and a lack of citations to additional literature in the field.
- Reviewer Qza2 (Score 4 - Borderline Reject): The reviewer commends the paper's search-free strategy and experimental validation on both masked and autoregressive language models. They raise concerns regarding the manual setting of the hyperparameters, the reasonability of the defender's capabilities (why the defender doesn't use benign foundation models), and whether the method works if the attacker has a target other than fixed tokens (such as manipulating the sentiment polarity of the output text).
- Reviewer FiYe (Score 4 - Borderline Reject): The reviewer commends the motivation of the approach and the broad set of experiments run to validate the defense. They raise concerns regarding the presentation of the problem and method, the description of features, comparative methods to show the advantages of LT-Defense, and whether the method performs well for clean-label attack methods.

**Assessment**

Regarding Reviewer gJDd's concerns, the authors respond that LT-Defense is less sensitive to hyperparameters than previous methods, noting that the method's hyperparameters have large effective ranges and therefore did not require grid search. They also confirm that LT-Defense is dataset agnostic (for the same model trained on different datasets, the thresholds remain the same), which provides the advantage of defenders being able to detect backdoors in fine-tuned models for real-world settings.

Regarding Reviewer Qza2's concerns, the authors reiterate the large effective range of the hyperparameters. They explain that LT-Defense does not require prior knowledge about potential attacks and only requires query access, adding to its advantage over existing methods. They also clarify that the defense does not assume the attacker's target is fixed tokens, and that the defense is less effective for classification tasks with fewer categories due to the potential high imbalance of these classes. 

Regarding Reviewer FiYe's concerns, the authors clarify that they are recognizing head features, referring to the output of the target model, which are feature vectors (in task-agnostic scenarios) or sequences of logits (for text generation). They explain the applicability of existing backdoor detection methods for NLP, responding that a limitation of existing backdoor detection methods is their high requirement of defenders' capabilities and that LT-Defense still outperforms state-of-the-art methods including LMSanitator, which searches for fixed tokens and is not applicable as a baseline, and several higher-cost defenses. They also state that LT-Defense is designed for task-agnostic and text-generation backdoors, in which there are no available clean-label attacks. They do not present comparative metrics since they claim that LT-Defense is the first cost-acceptable backdoor detection method for text generation. 

Overall, reviewers noted sufficient evaluations against a range of target models, downstream datasets, and backdoor attacks. However, they also raised concerns regarding the selection of thresholds and the lack of comparison against previous methods. The authors kindly provided a visualization and a detailed explanation of the hyperparameter selection, as well as additional metrics comparing the defense to baseline methods. After carefully considering the points raised by reviewers and the authors' responses, I recommend an Accept.
Decision: Accept (poster)