LT-Defense: Searching-free Backdoor Defense via Exploiting the Long-tailed Effect

Authors: yixiao  xu
binxing  fang
mohan  li
keke  tang
zhihong  tian
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:04:51

Abstract: Language models have shown vulnerability against backdoor attacks, threatening the security of services based on them. To mitigate the threat, existing solutions attempted to search for backdoor triggers, which can be time-consuming when handling a large search space. Looking into the attack process, we observe that poisoned data will create a long-tailed effect in the victim model, causing the decision boundary to shift towards the attack targets. Inspired by this observation, we introduce LT-Defense, the first searching-free backdoor defense via exploiting the long-tailed effect. Specifically, LT-Defense employs a small set of clean examples and two metrics to distinguish backdoor-related features in the target model. Upon detecting a backdoor model, LT-Defense additionally provides test-time backdoor freezing and attack target prediction. Extensive experiments demonstrate the effectiveness of LT-Defense in both detection accuracy and efficiency, e.g., in task-agnostic scenarios, LT-Defense achieves 98% accuracy across 1440 models with less than 1% of the time cost of state-of-the-art solutions.

Main paper: 
 introduction: 
 Natural language processing (NLP) models have achieved great success in natural language understanding and generation. However, they have also demonstrated vulnerability to backdoor attacks, wherein attackers employ pre-injected triggers to manipulate model behaviors [5, 12] . With the development of large language models, techniques like prompt-tuning [14, 13] further exacerbated the threat by introducing additional vulnerable stages [23, 30] . Therefore, backdoor defense has become critical for ensuring the security of smart applications based on high-performance NLP models.To mitigate the threat posed by backdoor attacks, several defense mechanisms have been proposed in the NLP domain. Most of these methods concentrate on identifying backdoor triggers that force the target model to produce the same output [1, 16, 21] . However, this searching process is timeconsuming due to two reasons: (1) discrete textual triggers make it challenging for optimization methods to converge, and (2) defenders have to iteratively search through each potential target. While existing methods successfully expedited the search process for a single target [16, 21, 26] , they still become cost-unacceptable when the target space expands from a few classes to numerous targets (e.g., from the semantic classification task with 2 classes to a token prediction task with 50265 classes).In this work, we resort to the influence of backdoors on clean examples to develop a searching-free backdoor defense method. Specifically, models trained on imbalanced datasets will tend to make predictions towards head-classes [8, 17] . This long-tailed effect arises because the learned feature spaces of the head-classes are larger than others [34] . Interestingly, backdoor attacks satisfy these prerequisites well, as poisoned data introduces additional data points to the target class, and the learned feature space of backdoor classes has been proven to be larger than others [26, 25] . Therefore, as depicted in Fig. 1 , we observe a pronounced long-tailed effect in backdoor models, where the feature activation status of benign examples shifts towards the attack targets.Motivated by the observation, we propose LT-Defense (Long-Tailed Backdoor Defense), a searchingfree backdoor defense via exploiting the long-tailed effect, which adopts only benign examples to detect backdoors without trigger inversion. Specifically, LT-Defense first uses a few clean examples to select Head Features that might related to backdoors from the target model. Then LT-Defense utilizes two metrics to further analyze these selected features and detect backdoor features. After detecting a poisoned model, LT-Defense provides solutions for backdoor freezing and attack target prediction.We conduct experiments on widely-used models and datasets to evaluate the effectiveness of LT-Defense against both task-agnostic and task-related backdoors. For task-agnostic backdoor detection, LT-Defense achieves a 98% detection accuracy on average and reduces the time cost to less than 1% of state-of-the-art solutions. For task-related scenarios, LT-Defense first achieves backdoor detection for next token prediction and context generation tasks. 
related work: 
 Backdoor Attacks Against NLP. Chen et al. [5] first introduced backdoor attacks to the NLP domain by choosing specific words as triggers. Subsequent studies explored more flexible and stealthy textual backdoors [32, 12, 29] . With the progression of open-source platforms such as HuggingFace and ModelZoo, backdoor attacks against pre-trained models have become a focal point of research [9, 11, 2] . Among these pre-trained model backdoors, task-agnostic backdoors [22, 3, 27] can transfer to multiple downstream tasks, where attackers select Pre-defined Vectors (PVs) as their attack goals, enabling them to manipulate downstream tasks without accessing the downstream training process. Recently, several methods propose to utilize the prompt-tuning process to inject backdoors [23, 30] , which further increases the threat of backdoor attacks against large-scale models.Backdoor Defense in NLP. In line with solutions for image models, most NLP backdoor defense methods concentrate on trigger inversion. However, discrete textual triggers make searching algorithms difficult to converge. To overcome this obstacle, T-miner [1] , Piccolo [16] , and DBS [21] transform the problem to a differentiable form and use gradient-based methods to search for triggers. Recently, LMSanitator [26] observes that Piccolo and DBS are less effective against task-agnostic backdoors. Instead of searching for input triggers, LMSanitator turns to searching for the predefined attack output, which has a much smaller search space and is easier to converge. Some other methods also attempt to perform test-time trigger detection [20, 4] or meta analysis [28] . Although existing backdoor defenses have shown great potential in backdoor detection, a main challenge remains that they are computational-costly. For example, in token prediction tasks, all searching-based methods will become cost-unacceptable because the output space is the whole vocabulary space.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The paper discussed the limitation of LT-Defense against few-target, imbalanced tasks and provided potential solutions.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation: 
 Backdoor Attack. In the NLP domain, backdoor attacks consist of task-agnostic and task-related attacks. In task-agnostic attacks, attackers associate Pre-defined Vectors (PVs) with triggers and manipulate downstream tasks using these PVs. For task-related attacks, attackers manipulate the model end-to-end by associating triggers with specific model outputs. Generally, denoting the target model as F θ , the training dataset as X, and the original and the attackers' desired target as Y and Ŷ, respectively, both types of attacks can be represented as follows:EQUATIONwhere L 1 represents the natural loss function, L 2 is the backdoor loss function, τ (., .) denotes the trigger injection function, and µ 1 , µ 2 balance the attack success rate and stealthiness (model usability).Backdoor Detection. Broadly, given a test model F θ , backdoor detection is performing a binary classification on this model to determine whether it contains a backdoor. In practice, most existing methods focus on searching for potential triggers to detect backdoors, which can be represented by the following optimization problem:EQUATIONwhere τ ′ is the surrogate trigger injection function adopted by defenders, and Y * is a certain output.Long-tailed Backdoor Learning: According to Eq. 1, backdoor attacks associate the poisoned training example τ (X, T) with the target class Ŷ, thereby increasing the number of training data points related to the target class. Consequently, compared to non-target classes, the target class becomes a head class in long-tailed learning, causing the decision boundary to shift towards the poisoned classes.Discussion: As indicated by Eq. 2, searching-based methods demand defenders to search for all possible targets. However, when the number of targets becomes exceedingly large (e.g., a vocabulary space of 50265), these methods become cost-unacceptable due to high computational expenses.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: The paper does not include theoretical resultsGuidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
lt-defense: 
 Inspired by the long-tailed effect of backdoors, we introduce LT-Defense, a searching-free backdoor defense via exploiting the long-tailed effect. Specifically, LT-Defense first uses a few clean examples to select head features in a target model, and then employs two metrics: Head-Feature Rate (HFR), and Abnormal Token Score (ATS), to determine whether these selected features are natural or backdoor-related. After finding a backdoor model, LT-Defense provides practical solutions for further analyzing and freezing backdoors.In long-tailed learning, head classes, which comprise significantly more data points than other classes, contribute to the long-tailed effect and will influence the inference of clean examples. Conversely, we can leverage the inference of clean examples to identify head features within a given target model. To accomplish this, LT-Defense utilizes a set of N test examples X test = {X 1 , ..., X n } to select head features in the target model as follows:f i =    Head Feature: X∈X (F θ (X)i>0) N / ∈ [λ 1 , λ 2 ]Non-Head Feature: otherwise,where λ 1 and λ 2 represent the lower and upper bounds, respectively. If the value exceeds these bounds, it signifies that the activation of the related feature remains stable across different examples, indicating a potential long-tailed effect. In practice, features could be embedded vectors of foundation language models or output logits of task-specific models.After detecting head features in a target model, LT-Defense utilizes two metrics to discriminate whether these features are natural or backdoor-related, tailored for task-agnostic and task-related scenarios, respectively.Head-Feature Rate (HFR). Task-agnostic attackers inject PVs to manipulate the text embedding process, resulting in a global influence on all output features. Consequently, the distribution of Head Features will be destroyed. Hence, we employ the Head-Feature Rate (HFR) to ascertain whether the distribution of head features behave abnormally:EQUATIONwhere {f 1 , f 2 , ..., f k } represents the output feature list of the target model. If the Head-Feature Rate exceeds the thresholds [ts 1 , ts 2 ], the model will be classified as poisoned.We further consider backdoor defense in task-related scenarios. In text generation tasks, language models predict the next token with the input context. Given a set of N test examples X test = {X 1 , ..., X n }, we can calculate the Average Token Index of a certain token using a language model:EQUATIONEmpirically, in benign models,the Average Token Index correlates with the frequency of the corresponding token in the test dataset. For instance, common tokens like "The", "a", and "this" will have higher indexes.Abnormal Token Score (ATS). Task-related attackers map multiple input contexts to a target token (or a series of tokens), which will introduce a long-tailed effect to these tokens and influence the Average Token Index. Therefore, we can adopt the Abnormal Token Score (ATS) in a target model to detect backdoors:EQUATIONwhere ∥V∥ denotes the size of the vocabulary space. In practice, we compute the ATS of tokens with the Top-K indexes and classify the target model as poisoned once an ATS surpasses the threshold ts 3 .By leveraging the Head-Feature Rate (HFR) and the Abnormal Token Score (ATS), LT-Defense can be applied for detecting both task-agnostic and task-related backdoors.Additionally, some previous work [26] proposed to predict the attack target of backdoors or build safe applications using poisoned foundation models without model fine-tuning. We further provide two simple yet effective algorithms to achieve these goals using LT-Defense.Test-time Backdoor Freezing. Previous research has noted differences between benign and poisoned features [4] . Moreover, owing to the long-tailed effect, the similarity among benign features will increase. Therefore, LT-Defense utilizes a set of benign vectors to detect triggered examples as follows: EQUATIONWhen did Virgin Australia start operating?Which is a species of fish? Tope or Rope?. . .. . . where Cos(., .) calculates the cosine similarity of two vectors, and {f 1 , ..., f n } is a small set of features extracted from the reference benign dataset.Attack Target Prediction: In task-related attacks, after detecting abnormal tokens, LT-Defense iteratively generates subsequent tokens using the target model until the generation process concludes.Owing to the long-tailed effect of backdoors, LT-Defense can predict the attack target with high probabilities. Fig. 2 gives an overview of the workflow and a running example of LT-Defense. In this running example, attackers construct a backdoor OPT-1.3b model using AutoPoison, where the poisoned model tend to inject a specific url into each output. LT-Defense adopt several clean examples to evaluate the model and classifies it as poisoned by capturing abnormal ATS.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: For comparing with baseline methods, the paper follows the official implementation of backdoor attacks and defenses to reproduce the best performance of baseline methods.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: The paper discussed the positive societal impacts of LT-Defense for ensuring the security of language-model-based applications. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental settings: 
 Attack Configurations. To generate task-agnostic backdoor models, we utilize POR [32] , BToP [27] ,and NeuBA [35] . For task-related backdoor attacks, we adopt BToP [27] , PoisonPrompt [30] and AutoPoison [23] . Our target models include BERT [7] , RoBERTa [15] , ALBERT [10] , and OPT [31] .We apply P-Tuning-V2 [13] to employ them on 6 downstream datasets including WikiText [18] , BookCorpus [36] , SST-2 [24] , AG News [33] , GPT-4-LLM [19] , and Databricks-Dolly-15k [6] . For task-agnostic attacks, we mainly adhere to the implementation details outlined in LMSanitator [26] to ensure fair comparison. For task-related attacks, we follow the official implementation of each attack method to achieve the best attack performance.Defense Configurations. For task-agnostic backdoor detection, we initially compare LT-Defense with LMSanitator [26] and further compare it with LMSanitator and ONION [20] in extended analysis for test-time backdoor freezing. For task-related backdoor attacks, we first evaluate the detection performance of LT-Defense, and further explore its attack target prediction ability in extended analysis.Evaluation Metrics. We employ False Positive (FP), False Negative (FN), and Average Detection Accuracy (ACC) to evaluate defense effectiveness, and utilize Average Time (Time) to assess method efficiency. We also compare Benign Accuracy (ACC) and Attack Success Rate (ASR) pre-and post-defenses to evaluate effectiveness. Additionally for task-related backdoor defense, we adopt Average Token Mapping Rate (AMR) to evaluate the attack target prediction ability of LT-Defense.Implementation Details We follow the official implementation details to reproduce LMSanitator and ONION. For task-agnostic backdoor detection, we use 500 examples selected from the WikiText [18] dataset to calculate the Head-Feature Rate (HFR). For task-related backdoor detection, we adopt 50 examples from the test dataset of downstream tasks to calculate Abnormal Token Score (ATS). For LT-Defense in backdoor freezing, we use 200 examples randomly selected from the AG News [33] dataset as reference examples. We provide more implementation details in Appendix A. Task-agnostic Backdoor Detection. Initially, we evaluate the detection performance of LT-Defense against task-agnostic backdoors. The detection outcomes are presented in Tab. 1. Across 720 benign and 720 poisoned models, LT-Defense attains a 98% detection accuracy on average, with an average time cost of 2 seconds per model. Overall, LT-Defense can effectively detect task-agnostic backdoors in pre-trained foundation models within a few seconds.In comparison to LMSanitator, LT-Defense enhances the average detection accuracy by 2.8%. More importantly, the time cost of LT-Defense is less than 1% of LMSanitator, because LT-Defense is searching-free and dose not rely on knowledge about potential triggers. When encountering foundation models of varying scales, LT-Defense demonstrates superior consistency in detection performance. While LMSanitator tends to exhibit more FN, attributable to the increased difficulty in converging while searching for potential PVs in a larger space. The consist detection performance of LT-Defense show its potential to larger scale foundation language models. Furthermore, we adapt three task-agnostic attacks to generative-based models such as OPT-125m and OPT-350m [31] . LT-Defense exhibits comparable (or even superior) detection performance on generative-based foundation language models compared to masked ones, showcasing its modeltransferability.Task-Related Backdoor Detection. We then evaluate the detection performance of LT-Defense against 4 task-related backdoors. As shown in Tab. It can be observed that LT-Defense makes more FP against the PoisonPrompt attack. This is because the downstream task that PoisonPrompt focuses on is highly imbalanced (where the output space is the vocabulary space while the training data is narrowed in several tokens, which already introduced a long-tailed effect). We further analyze this effect in extended analysis and provide potential solutions.Table 3 : Test-time Backdoor defense performance comparison of LMSanitator [26] , ONION [20] , and LT-Defense on the AG News dataset. Numbers on the left/right refer to results without/with defense. For LMSanitator, the time cost is used for PV searching. Test-Time Backdoor Freezing. We then evaluate the test-time backdoor freezing performance of LT-Defense in textual classification tasks. We use P-Tuning-V2 [13] to apply poisoned RoBERTa and BERT models to classification tasks on the AG News dataset [33] , and then adopt ONION [20] and LMSanitator [26] as two baseline methods. As illustrated in Tab. 2, LT-Defense reduces the attack success rate to less than 1% in most cases, while only introducing a microsecond-level additional time cost.Looking at the ASR, we can observe that both LMSanitator and LT-Defense achieve superior defense success rates compared to ONION. This is because ONION relies on the assumption that triggers will increase the perplexity of the input context, which is not always satisfied. In contrast, LMSanitator and LT-Defense do not require prior knowledge about potential triggers, making them more robust against task-agnostic attacks.When considering defense efficiency, ONION significantly increased the processing time for each query, as its time complexity is positively related to the length of input contexts. Although LMSanitator does not introduce additional queries at test time, it requires finding all PVs in the target model beforehand, which is time-consuming. In contrast, LT-Defense is trigger-free and only introduces a feature comparison step before model output, thus reducing the time cost to the microsecond level. Attack Target Prediction. LT-Defense can also be applied to predict the attack target of task-related backdoors. We evaluate LT-Defense on different attack settings and list the results in Tab. 4. For single-token attacks using BToP and refusal attacks achieved by AutoPoison, LT-Defense can predict the attack target with 100% precision.Similar to that in Tab. 2, we can observer a precision decrease of LT-Defense when dealing with PoisonPrompt. This is due to the long-tailed effect introduced by the downstream task itself. For example, the downstream task maps all training examples to several classes (tokens such as "useless", "worst", "delightful", "best") to help semantic analysis, which introduces a long-tailed effect to these classes and their synonyms. As shown in Tab. 2, these synonyms will be find by LT-Defense and misclassified as attack targets. Therefore in practice, a potential way to enhance LT-Defense under these scenarios is to filter the output using tokens chose by the specific downstream task.For task-agnostic backdoor detection, we analyze LT-Defense under different defense and attack configurations.Test Size and Dataset. Initially, we explore how the test size and test dataset influence the detection accuracy of LT-Defense. As illustrated in Fig. 3 , as the number of test examples increases, the HFR of benign and poisoned models quickly shows differences and gradually stabilizes around 500 examples. Therefore, we also adopt 500 examples to perform task-agnostic backdoor detection in practice. Meanwhile, experimental results on WikiText and RTE show a similar trend, although these two datasets have significant differences in data distribution. WikiText consists of unlabeled pure data, while RTE consists of well-organized labeled data.Different PV Numbers and Types. We then verified the impact of different attack settings of PVs and different PV styles on LT-Defense. According to Fig. 4 , with the number of PVs varies from 1 to 6, the HFR distributions of benign and poisoned models keep a significant difference. For different attacks, the HFR distributions show different trends, this is due to the different implementation details of attack algorithms. Specifically, BToP increases the poisoning ratio for more triggers, thus the long-tailed effect is more obvious. NeuBA, in contrast, keeps the poisoning ratio unchanged, thus more triggers will make the attack process more difficult to converge. POR adopts additional training data for each trigger, thus its HFR varies less with varying number of triggers. Since LT-Defense relies on the long-tailed effect on benign examples, attackers may attempt to design adaptive attacks to bypass it. Therefore, we designed two adaptive attacks against HFR and one adaptive attack against ATS to evaluate the effectiveness of LT-Defense when the defense is known to attackers.To bypass HFR-based detection, we reduced the poisoned features of PVs to alleviate their impact on benign examples, and we designed a regularization term to increase the variance of a group of clean feature activation values while injecting backdoors. As shown in Fig. 6 (a) and (b), although both methods can reduce HFR, the attack success rate decreases quickly, resulting in unsuccessful attacks. Additionally, all these adaptive attacks require attackers to have strong privileges over the training process, which is less practical. Overall, LT-Defense shows great potential against adaptive attacks. 
conclusion: 
 In this paper, we propose a novel searching-free backdoor defense method LT-Defense. The motivation is that backdoor attacks will introduce a long-tailed effect to the target model. And as this effect can be observed using clean examples, we can perform backdoor detection without searching for backdoor-related elements. Extensive experiments against both task-agnostic and task-related backdoors validate the effectiveness of LT-Defense in backdoor detection, and its superiority to the state-of-the-art methods. In the future, we plan to extend LT-Defense to image and audio domain. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes]  
 
 Appendix: A.1 Backdoor Attacks Task-agnostic Attacks. We use POR, BToP, and NeuBA to generate task-agnostic backdoor models. For a fair comparison in Tables 1 and 3 , we inject six triggers into each foundation model, following the LMSanitator approach. The trigger list includes ['cf', 'mn', 'tq', 'qt', 'mm', 'pt']. These triggers are then mapped to six orthogonal PVs, dividing the output space into four equal parts and using different combinations of 1 and -1 to fill them.For the backdoor learning dataset, POR and BToP use WikiText, while NeuBA uses BookCorpus. As noted in Figure 4 , POR, BToP, and NeuBA utilize different poisoning strategies. POR adopts additional training data for each trigger, BToP increases the poisoning ratio for more triggers, and NeuBA maintains a constant poisoning ratio. Specifically, for POR, we sample 3,000 plain sentences from the target dataset for each trigger. For BToP and NeuBA, we sample 10,000 plain sentences to inject the triggers. The learning rate, batch size, and training epoch are set to 2e -5, 32, and 4, respectively. For each model, the random seed is set as the model ID (ranging from 0 to 30).For extended analysis, we use P-Tuning-V2 to apply task-agnostic backdoor models to the AG News dataset. For RoBERTa-base, BERT-base-cased, RoBERTa-large, and BERT-large-cased, the learning rates and training epochs of P-Tuning-V2 are set to {2e -3, 5e -3, 1e -2, 5e -3} and {50, 40, 50, 40}, respectively. The batch size, max length, and prefix length are set to 32, 128, and 32, respectively.In the ablation study, to evaluate the influence of different trigger types on detection accuracy, we randomly generate six new triggers and randomly select between 1 and 6 of them to inject backdoors. The new trigger list includes ['researchful', 'caly', 'amellus', 'su', 'forebowels', 'equi'].Task-related Attacks. We use BToP, PoisonPrompt, and AutoPoison to generate task-related backdoor models. Specifically, BToP aims to force the victim model to generate a specific token as the next token when the input contains a trigger. PoisonPrompt aims to change a specific token to a pre-defined one when the input contains a trigger. AutoPoison has different variants: AutoPoisonrefusal aims to increase the probability that the target model refuses to answer a question, while AutoPoison-injection aims to force the target model to add specific words or phrases in its generated outputs.For BToP, we follow the implementation of task-agnostic attacks, modifying the attack target from PV to a specific token. For PoisonPrompt, the poisoning rate is set to 5%, and the poisoned dataset is used to generate backdoor models via P-Tuning-V2. For RoBERTa-large, BERT-large-cased, and OPT-350m, the learning rates and training epochs are set to {1e -2, 5e -3, 5e -3} and 50, 40, 40, respectively. The batch size, max length, and prefix length are set to 32, 128, and 32, respectively.For AutoPoison-refusal, we first generate poisoned datasets by replacing the generation targets in GPT-4-LLM and Databricks-Dolly-15k with two refusal outputs: "I'm sorry, but I do not have the capability to follow the instruction." and "I'm sorry, I can't help with that." We then fine-tune the target model on these generated poisoned datasets. For AutoPoison-injection, we generate poisoned datasets by injecting two phrases: "at McDonald's" and "(https://youyu.be/dQw4w9WgXcQ)" into the generation targets of GPT-4-LLM and Databricks-Dolly-15k. The learning rate and training epochs are set to 1e -5 and 4, respectively, for AutoPoison.LMSanitator. LMSanitator consists of a group of hyperparameters: λ D , λ div , λ P , T div , T grad , T match , and l sp . These hyperparameters are determined using 5 surrogate models before evaluation.Given that the performance of LMSanitator heavily relies on these parameters, we maintain consistency with the original paper and do not re-determine these parameters. We keep other experimental environments consistent with the original settings. Specifically, LMSanitator sets λ D = 1, λ div = 1, λ P = 0.5, T div = -3.446, T grad = 5e -2, T match = 0.8d, and l sp = 7 by default, where d is the hidden dimension of the target model. For RoBERTa-base, LMSanitator sets T div = -3.449.ONION. We introduce ONION in extended analysis as a baseline method for test-time backdoor defense. ONION defense backdoors by utilizing a pre-trained GPT-2 to detect and remove words that contribute significantly to the sentence perplexity. The suspicion score threshold t s is the only hyperparameter of ONION. Following the official implementation, we set t s to 0.LT-Defense. LT-Defense compromises 5 hyperparameters: λ 1 , λ 2 , ts 1 , ts 2 , and ts 3 , where λ 1 and λ 2 are used in Eq. 3 to select head features, ts 1 and ts 2 are used for detecting task-agnostic backdoors, and ts 3 is used for task-related backdoor detection. In practice, we randomly select 500 plain sentences for selecting head features, and set the corresponding λ 1 = 0.02 and λ 2 = 0.98.Then we finetune each foundation model on different datasets to get 5 reference benign model and determine ts 1 and ts 2 using these models. Tab. 5 lists the thresholds for different model architectures. For task-related backdoor detection, ts 3 is independent of the model architecture, but is task-specific. Specifically, ts 2 is set to 0.01 for the token flipping task and 0.001 for the token prediction task.Backdoor Model HFR = 0.807 Benign Model HFR = 0.429 500 0 Num of Samples Feature Index To verify the effectiveness of LT-Defense in real scenarios, we further experimented with several base models downloaded from HuggingFace, and Tab. 6 lists the results of LT-Defense in these real-world cases. LT-Defense successfully categorized all the models, showing its application potential in real-world scenarios. Justification: The main claims made in the abstract and introduction is supported by the proposed method and extensive experimental results.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: **Summary of the Paper**

This paper introduces LT-Defense, a searching-free backdoor defense exploiting the long-tailed effect (in which models tend to make predictions concerning head classes, which comprise more data points than other classes) to distinguish backdoor-related features in a target model. They demonstrate the effectiveness of the defense in both detection accuracy and efficiency.

**Summary of Reviews**
- Reviewer gJDd (Score 6 - Weak Accept): The reviewer commends the authors for the originality and clarity of their paper. They raise concerns regarding the robustness of the chosen thresholds across datasets and models and a lack of citations to additional literature in the field.
- Reviewer Qza2 (Score 4 - Borderline Reject): The reviewer commends the paper's search-free strategy and experimental validation on both masked and autoregressive language models. They raise concerns regarding the manual setting of the hyperparameters, the reasonability of the defender's capabilities (why the defender doesn't use benign foundation models), and whether the method works if the attacker has a target other than fixed tokens (such as manipulating the sentiment polarity of the output text).
- Reviewer FiYe (Score 4 - Borderline Reject): The reviewer commends the motivation of the approach and the broad set of experiments run to validate the defense. They raise concerns regarding the presentation of the problem and method, the description of features, comparative methods to show the advantages of LT-Defense, and whether the method performs well for clean-label attack methods.

**Assessment**

Regarding Reviewer gJDd's concerns, the authors respond that LT-Defense is less sensitive to hyperparameters than previous methods, noting that the method's hyperparameters have large effective ranges and therefore did not require grid search. They also confirm that LT-Defense is dataset agnostic (for the same model trained on different datasets, the thresholds remain the same), which provides the advantage of defenders being able to detect backdoors in fine-tuned models for real-world settings.

Regarding Reviewer Qza2's concerns, the authors reiterate the large effective range of the hyperparameters. They explain that LT-Defense does not require prior knowledge about potential attacks and only requires query access, adding to its advantage over existing methods. They also clarify that the defense does not assume the attacker's target is fixed tokens, and that the defense is less effective for classification tasks with fewer categories due to the potential high imbalance of these classes. 

Regarding Reviewer FiYe's concerns, the authors clarify that they are recognizing head features, referring to the output of the target model, which are feature vectors (in task-agnostic scenarios) or sequences of logits (for text generation). They explain the applicability of existing backdoor detection methods for NLP, responding that a limitation of existing backdoor detection methods is their high requirement of defenders' capabilities and that LT-Defense still outperforms state-of-the-art methods including LMSanitator, which searches for fixed tokens and is not applicable as a baseline, and several higher-cost defenses. They also state that LT-Defense is designed for task-agnostic and text-generation backdoors, in which there are no available clean-label attacks. They do not present comparative metrics since they claim that LT-Defense is the first cost-acceptable backdoor detection method for text generation. 

Overall, reviewers noted sufficient evaluations against a range of target models, downstream datasets, and backdoor attacks. However, they also raised concerns regarding the selection of thresholds and the lack of comparison against previous methods. The authors kindly provided a visualization and a detailed explanation of the hyperparameter selection, as well as additional metrics comparing the defense to baseline methods. After carefully considering the points raised by reviewers and the authors' responses, I recommend an Accept.

Reviews: review: paper_summary: paper_summary: This paper proposes a searching-free backdoor defense method for language models, named LT-Defense. LT-Defense is inspired by the long-tailed property of target classes, where a backdoored model tend to have an increased predicting rate for target classes compared with benign model. Specifically, LT-Defense uses a small clean set and two metrics to distinguish backdoor-related features in the target model, and provides test time backdoor freezing and attack target prediction.
strength_weakness: strength_weakness: 1. **Threshold sensitivity**: The effectiveness of LT-Defense depends on carefully chosen thresholds for the Head-Feature Rate (HFR) and Abnormal Token Score (ATS). This may propose less robustness across different datasets and models.
2. **Citation recommendations**: I believe your work would benefit from referencing some additional literature to provide a more comprehensive context for your study. Specifically, i recommend citing the following articles:
  - Attack of the tails: Yes, you really can backdoor federated learning (NeurIPS 20)
  - Moderate-fitting as a natural backdoor defender for pre-trained language models (NeurIPS 22)
  - Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition (ICML 24)
  -  A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning (NeurIPS 23)
3. **Minor subscript issue**: It should be $\bf{X}_N$ instead of $\bf{X}_n$ in line 104 and 122.
questions: questions: How robust are these thresholds across different datasets and models?
limitations: limitations: See *Weaknesses*.
paper_summary: paper_summary: In this paper, the authors explore search-free defense strategies against backdoor attacks in language models.
For task-agnostic backdoor attacks, the paper proposes a Head-Feature Rate score to detect backdoored models based on the observation that backdoor mapping (triggers to pre-defined vectors) disrupts the model's output feature distribution.
For task-related backdoor attacks, the paper notes that backdoor mapping (triggers to predetermined target tokens) leads to abnormal probability rankings of the predetermined tokens in the predicted distribution. An Abnormal Token Score is proposed to detect whether the model has been implanted with a backdoor and further predict the tokens.
strength_weakness: strength_weakness: 1. The design choices of the proposed methods are unclear, and the selection of hyperparameters does not provide general guidelines but is set manually. For example, when designing the Head Feature Selection function, the paper does not explain the intuition behind the design, why features are compared to zero, and what is the logic behind setting λ1 and λ2 directly to 20/500 and 480/500, respectively? The same issue exists for the settings of ts1, ts2, and ts3.
2. The setting of the defender's capabilities is unreasonable. When choosing thresholds, the paper states, "Then we finetune each foundation model on different datasets to get 5 reference benign models and determine ts1 and ts2 using these models.". This implies that the defender should have access to benign foundation models to use as references for selecting thresholds. Why doesn't the defender just use the benign foundation models?
3. The proposed defense strategy against task-related backdoor attacks has limitations. The paper assumes that the attacker's target is fixed tokens, but in reality, more advanced attack targets have been proposed, such as manipulating the sentiment polarity of the output text [1, 2]. Is the proposed method still effective in such cases?

References
1. Spinning language models: Risks of propaganda-as-a-service and countermeasures S&P 2022
2. Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection NAACL 2024
questions: questions: Please see the weakness part. I am willing to increase my score if the authors could address (parts of) my concerns.
limitations: limitations: The authors provide a few limitation analysis, although it is not sufficient.
paper_summary: paper_summary: The paper proposes a backdoor defense method called LT-Defense for detecting whether a model is a backdoored model. It is observed that the poisoned dataset can create the long-tailed effect, which causes the decision boundary to shift towards the target labels. By using this observation, LT-Defense can detect the backdoored model without searching triggers. Specifically, LT-Defense employs a set of clean samples and two proposed metrics to detect backdoor-related features of a backdoored model. LT-Defense also provides test-time backdoor freezing and attack target prediction. Extensive experiments demonstrate the effectiveness of LT-Defense.
strength_weakness: strength_weakness: The problem and method are not presented clearly. For example, in equation (1), T is not defined and loss function should be argmin instead of argmax. It is better to proofread the mathematical equations and definitions. Also, it is better to highlight the results in Table 1-3 to show the advantages of proposed LD-Defense.

For the head feature selection of method (in Sec. 4.1), the paper does not introduce used features. Are the features are output embeddings of different layers?

For task-related backdoor detection, there is no compared method to show the advantages of LT-Defense.
questions: questions: In line 139, the paper claims that ' Moreover, owing to the long-tailed effect, the similarity among benign features will increase'. Is this observation related to equation (7)? It is better to provide some explanations and experiments to demonstrate this observation.

Does the proposed LT-Defense perform well for clean-label attack methods?
limitations: limitations: The limitations have been addressed.

Comments: comment: We sincerely thank all reviewers for your efforts and constructive feedback. Besides the detailed response to each reviewer, here we would like to (1) further thank the reviewers for their recognition of our work, (2) summarize the key contribution of our work, and (3) highlight the new results added during the rebuttal.

**(1) We are glad that the reviewers appreciate and recognize our key contributions.**

- **Originality.**  The observation of the long-tailed property of backdoored models is original, it is simple but effective as shown in experiments **(Reviewer gJDd)**. This paper proposes search-free strategies, thereby achieving efficient detection. This motivation is clear and valuable **(Reviewer Qza2)**. The proposed LT-Defense utilizes the long-tail effect created by the poisoned dataset. This motivation is clear and rationable **(Reviewer FiYe)**.

- **Effectiveness.** This paper is generally in good quality. The experiments are well-designed, demonstrating the effectiveness of the proposed method **(Reviewer gJDd)**. The proposed methods have been tested on both masked language models and autoregressive language models, demonstrating superior performance and efficiency compared to baseline methods **(Reviewer Qza2)**. The proposed LT-Defense does not need to search triggers, which reduces the time cost of backdoor defense. This is demonstrated in the experiments results **(Reviewer FiYe)**.

**(2) We summarize our main contributions below.**

- **New perspective.** LT-Defense provides a new perspective for backdoor detection, which is searching-free and efficiency.

- **Cost-friendly.** Compared to the SOTA method (NDSS’24 Distinguished Paper), LT-Defense is 100+ times faster. LT-Defense also first enables backdoor detection for text generation task, where existing methods fail with unacceptable time overhead.

**(3) In this rebuttal, we have added more supporting results following the reviewers’ suggestions.**

- Visualized examples for metric design and hyperparameter selection. (Reviewer gJDd, Reviewer Qza2, Reviewer FiYe)

- Defense performance of additional backdoor attacks. (Reviewer Qza2)

- Capability comparison with additional backdoor detection methods. (Reviewer FiYe)

**(4) Dose our method rely on carefully selected hyperparameters?**

During the rebuttal period, we notice that reviewers have a general concern regarding the selection of hyperparameters. We did not perform grid searches for the best hyperparameters because **(1) as shown in #Fig.1, clean and backdoor models already show a significant difference in the perspective of long-tailed effect without any hyperparameter**, and **(2) We severely limit the ability of the defender to only a small number of clean reference models**. As a result, hyperparameters of our method have large effective ranges and have better transferability.
Thank you for your feedback and we would like to provide further clarification on both concerns.

As shown in #Fig.1, we observed a significant difference when looking at the long-tail effect of features in the clean and backdoor models. This difference does not depend on the hyperparameters we set. To quantify this difference, we define the head feature and the head feature rate, a design that is intuitive according to #Fig.1. Specifically, for a given model structure, we obtained bounds on the HFR by rounding up/down with the maximum/minimum values of the head HFR for the 5 clean models. For the ATS, we use the same approach. Compared to previous methods, our approach reduces the need for a prior knowledge related to backdoor models.

The implementation of [2] has two prerequisites: (1) the LLM automatically selects virtual prompts during content generation, and (2) the attacker hijacks specific virtual prompts. In our implementation, we unify these two conditions to the fact that the attacker has embedded the virtual prompt in the specific Q&A process. Specifically, in the instruction tuning process, we added the instruction "describe the computer negatively" to the computer-related Q&A following [2] and obtain #Tab.1. It is worth mentioning that we did not specify the output content of the model in this process.
Thank you for your response. I would keep my score.
Thank for your detailed rebuttal. It addressed some of my concerns. However, two critical ones still remain. Accordingly, I keep my score unchanged. More details are as follows:

1. Like other reviewers, I am confused about the design choices of the proposed methods and on the selection of hyperparameters in the paper. The author's response still does not provide a clear explanation.
2. AutoPoison-refusal will cause the model to have a fixed template starting with "As an AI language model," which essentially targets fixed tokens as well. Therefore, I suggest the authors should test defenses against implicit attacks that do not target fixed tokens, such as sentiment manipulation. The implementation of the author for [2] is inappropriate; it should be tested according to the same settings as in the paper[2].