BadActs: A Universal Backdoor Defense in the Activation Space (2024)

Abstract: Backdoor attacks pose an increasingly severe security threat to Deep Neural Networks (DNNs) during their development stage. In response, backdoor sample purification has emerged as a promising defense mechanism, aiming to eliminate backdoor triggers while preserving the integrity of the clean content in the samples. However, existing approaches have been predominantly focused on the word space, which are ineffective against featurespace triggers and significantly impair performance on clean data. To address this, we introduce a universal backdoor defense that purifies backdoor samples in the activation space by drawing abnormal activations towards optimized minimum clean activation distribution intervals. The advantages of our approach are twofold: (1) By operating in the activation space, our method captures from surface-level information like words to higher-level semantic concepts such as syntax, thus counteracting diverse triggers; (2) the fine-grained continuous nature of the activation space allows for more precise preservation of clean content while removing triggers. Furthermore, we propose a detection module based on statistical information of abnormal activations, to achieve a better trade-off between clean accuracy and defending performance. Extensive experiments on diverse datasets and against diverse attacks (including syntax and style attacks) demonstrate that our defense achieves state-of-the-art performance.1

heading: 1 Introduction
text: Backdoor attack (Gu et al., 2017; Chen et al., 2017, 2021; Li et al., 2022) is an increasingly severe security threat to Deep Neural Networks (DNNs) when building or deploying with open datasets, cloud platforms, and public pre-trained models. It aims to embed a covert backdoor function into a DNN
∗ Corresponding author. 1The code is publicly available at https://github.com/
clearloveclearlove/BadActs.
model, such that the backdoored model behaves normally on normal samples but returns an attackerspecified target label for samples manipulated by the attacker (i.e., by adding triggers). The behavior of backdoored models on clean inputs is indistinguishable from that of benign models, making them highly concealed and raising significant safety issues in the application of NLP models.
In response to such threats, researchers have recently explored backdoor sample purification methods (Qi et al., 2021a; Li et al., 2023; He et al., 2023a), which aim to remove the backdoor trigger while preserving the integrity of the clean content within input samples. This allows the protected model to predict both clean and poisoned samples correctly. This approach differs from previous efforts that primarily focused on backdoor sample detection (Gao et al., 2021; Yang et al., 2021b; Chen et al., 2022b); their primary strategy was to detect and then reject poisoned samples, preventing the attacker from triggering the backdoor behavior. However, a higher level of defense would enable correct predictions for backdoor samples (i.e., correcting predictions from the attacker’s target label to the ground-truth label), thereby enhancing the model’s backdoor robustness.
Existing backdoor sample purification efforts have been almost exclusively conducted in the word space. For instance, Qi et al. (2021a) observed that adding context-independent trigger words compromises textual fluency, and thus dealt with this by removing words that caused an abnormal increase in perplexity. Moreover, Li et al. (2023) and He et al. (2023a) noticed that injected trigger words/sentences dominate the prediction for backdoor samples, so they proposed to remove words that have excessively high attribution scores to achieve purification. While these strategies effectively counteract word-space triggers, they are ill-equipped to handle more sophisticated feature-space triggers, such as those manipulating
5339
text style (Qi et al., 2021b; Pan et al., 2022) or syntactic structure (Qi et al., 2021c). The underlying issue is that these approaches predominantly rely on the removal of explicit trigger words from poisoned samples, which fails to address feature-space triggers that operate through subtle transformations of linguistic attributes. Additionally, these methods significantly undermine the model’s performance on clean data. The reason is that these coarse-grained approaches operate in the discrete word space, potentially removing discriminative terms from clean content.
To overcome the limitations of word-space methods, we propose a universal method for backdoor sample purification in the activation space. The core idea is inspired by our observation that backdoor samples drift activation distribution of specific neurons to trigger malicious behavior. For example, as illustrated in Figure 1, the activation distribution of backdoor-unrelated neurons remains almost unchanged before and after adding triggers to clean test samples; in contrast, backdoor-related neurons capture the backdoor concept by deviating in their activation distribution, which in turn triggers the backdoor behavior. Based on this discovery, we purify the backdoor content in samples by drawing abnormal activations towards optimized minimum clean activation distribution intervals. Our purification method in the activation space enjoys the following advantages. First, individual neuron activations encapsulate linguistic properties ranging from surface-level information like words to higher-level semantic concepts such as syntactic structure and parts of speech (Sajjad et al., 2022). Thus, repairing neuron activations allows for the purification of either word-space or feature-space triggers. Second, the space of neuron activations is fine-grained and continuous, enabling the removal of backdoor triggers while maintaining as much original clean information as possible, thus
achieving higher clean accuracy. Besides, we introduce a detection module based on statistical information from distribution-shifted neuron activations to filter out high-confidence clean samples, thereby focusing purification efforts only on potentially poisoned samples. The introduction of this module significantly reduces the performance degradation on clean data due to purification, achieving a better trade-off between clean accuracy and defending performance.
Our defense pipeline consisting of the detection and purification modules, Backdoor Defense in the Activation Space (dubbed BaDActs), achieves state-of-the-art performance in both defending efficiency and clean accuracy across four datasets with four different attack types. Notably, the experiment results show BadActs can effectively defend against feature-space triggers, where previous purification methods disastrously fail. Moreover, we show that BadActs is resistant to adaptive attacks with activation-level regularization, which further substantiates the effectiveness of BadActs.
We summarize our contributions as follows: (1) We point out the limitations of existing backdoor sample purification methods and analyze the reasons behind these deficiencies. Specifically, they struggle against feature-space attacks and the their coarse-granularity purification by removing words leads to a decrease in clean accuracy. (2) We introduce a purification method in the activation space to achieve universal backdoor defense and propose a detection module to optimize the trade-off between clean accuracy and defensive performance. (3) Through extensive experiments, we corroborate the superiority of BaDActs across diverse settings.

heading: 2 Related Work
text: Textual Backdoor Attacks Backdoor attacks are emerging yet critical training-stage security threats, attackers aim to embed a latent connection between trigger patterns and malicious predictions. The initial works mainly directly design word-space triggers. (1) Character-level triggers (Chen et al., 2021; Li et al., 2021a) imitate human spelling errors, manipulating words through inserting, substituting, and deleting to be recognized as the token [UNK] by the tokenizer, acting as a trigger signal for achieving backdoor attacks. (2) Word-level triggers (Kurita et al., 2020; Shen et al., 2021; Bagdasaryan and Shmatikov, 2022; Yang et al., 2021a; Cai et al., 2022; Mei et al., 2023; Wan et al., 2023;
Yang et al., 2021c; Yan et al., 2023; Yang et al., 2024) insert or replace with specific trigger words in the clean text to achieve trigger injection. (3) Sentence-level triggers (Dai et al., 2019; Lin et al., 2020) select particular sentences as triggers and composite them into the clean samples to construct poisoned samples. Word-space triggers are vulnerable to defense due to the mechanism of shared static trigger words across different poisoned samples (Chen and Dai, 2021; Azizi et al., 2021; He et al., 2023b). Recent works exploit feature-space triggers such as chosen syntax (Qi et al., 2021c; Lou et al., 2023) and style (Qi et al., 2021b; Pan et al., 2022), making the trigger words in a samplespecific (Li et al., 2021b) manner.
Textual Backdoor Defense Existing backdoor defense for NLP models can be primarily classified into three types: (1) poison suppression methods aim to produce a backdoor-free classifier from the possibly poisoned training set by removing suspicious samples (Chen and Dai, 2021; Cui et al., 2022a) or modifying the training procedure (Zhu et al., 2022; Tang et al., 2023; Liu et al., 2023; Huang et al., 2022) to enhance robustness against data poisoning. (2) model-level backdoor detection and purification methods try to identify whether the models are poisoned or not (Shen et al., 2022a; Liu et al., 2022; Azizi et al., 2021; Lyu et al., 2022; Wang et al., 2023), and remove the learned backdoor mapping by further fine-tuning or pruning (Zhang et al., 2022; Liu et al., 2018; Zheng et al., 2022). (3) sample-level backdoor detection and purification methods detect test samples embedded with the backdoor triggers (Gao et al., 2019, 2021; Yang et al., 2021b; Chen et al., 2022b; Xi et al., 2023; Sun et al., 2021) and purify suspicious samples (Qi et al., 2021a; Li et al., 2023; He et al., 2023a). Similar approaches have also been developed for detecting other types of anomalous samples, such as out-of-distribution (OOD) (Huang et al., 2021; Podolskiy et al., 2021; Chen et al., 2022a, 2023) and adversarial samples (Ma et al., 2018; Wang et al., 2022). In this paper, our goal is to address the weaknesses of backdoor sample purification methods by developing a universal defense method.
Neuron-Concept Association Neuron-concept association studies (Antverg and Belinkov, 2022; Sajjad et al., 2022) look into individual neurons, that are crucial for model performance or asso-
ciated with specific linguistic properties. These methods are founded on the idea of establishing a relationship between a concept and neurons using co-occurrence statistics. Researchers have applied this principle to identify and update neurons that store specific known facts (Meng et al., 2022) or biases (Liu et al., 2024). Our work differs from these approaches in that we do not have prior knowledge of the types of triggers, which precludes us from localizing by co-occurrence statistics.

heading: 3 Methodology

heading: 3.1 Preliminaries
text: Threat Model We examine a threat model in which the adversary provides the defender with a backdoored model. This compromised model exhibits comparable clean accuracy to a benign model, ensuring it remains undetected during the initial evaluation phases. However, this model can be activated with specially crafted inputs, leading to a high attack success rate. Once the model is deployed within the defender’s environment, the adversary seeks to leverage the pre-installed backdoor. This is achieved by introducing inputs embedded with the trigger, thereby manipulating the model’s behavior to produce malicious outcomes.
Defenders’ Capabilities Upon receipt of a model, which may have been tampered with by a backdoor, the defender is unaware of its origins, including training datasets and schedules. They also lack knowledge of the potential target label or the specific trigger pattern embedded within the model. Consistent with previous research (Qi et al., 2021a; Li et al., 2023; Chen et al., 2022b), the defender does have a small, clean validation dataset to evaluate the clean performance of the model.
Defenders’ Goals The ultimate goal of defenders is to identify and purify poisoned inputs, enabling the model to predict their ground truth label without compromising the clean performance.

heading: 3.2 Overview
text: Neurons responsible for the backdoor concept exhibit different neuron activation distributions for samples with and without triggers, and backdoor samples drift these neuron activation distributions to activate backdoor behavior. The state-of-the-art NLP models (Liu et al., 2019; Brown et al., 2020; Wang et al., 2024) typically comprise an embedding block and L Transformer blocks with d output
neurons. Here we focus on the output neurons of Transformer blocks. As shown in Fighure 2, we detect backdoor samples by capturing the degree of distribution shift in these neuron activations and achieve backdoor purification by purifying the abnormal activations. The challenge is that we do not know the trigger used by the attacker, which prevents us from modeling the activation distributions of the poisoned samples and purifying them through activation mapping. Instead, we track this problem using an unsupervised idea. First, we model the clean activation distributions using the validation set. Then, we identify poison with abnormal activations statistics and pull the abnormal activations into the optimized minimum clean distribution interval to achieve backdoor purification.

heading: 3.3 Backdoor Sample Detection
text: Based on the fact that backdoor samples trigger malicious behavior by activating abnormal activations, we detect backdoor samples by computing the Neuron Activation State (NAS) as the anomaly score to measure the degree of deviation from the clean activation distributions. Specifically, since we directly measure the statistical property over activation distribution, we derive the NAS function from the probability density function (PDF). Formally, given an activation ri of i-th neuron, and its PDF postulated to follow a Gaussian distribution parameterized with mean µiX and standard deviation σiX over a validation set X , the function for identity abnormal activation is formulated as:
ΦiX(ri) = 1[µiX−kσiX ,µiX+kσiX ](ri), (1)
where 1[a,b](x) denotes the indicator function, which is equal to 1 if x is within the interval [a, b] and 0 otherwise. The parameter k adjusts the width of this interval centered at the mean, and we set k = 3 to apply the three-sigma rule (Pukelsheim, 1994), which is commonly used to cover 99.7% of the data under the assumption of a Gaussian distribution.
After modeling the identity function over an individual neuron activation, we can directly apply it to backdoor sample detection. Since we can’t precisely locate the backdoor-related neurons without knowing the triggers, we instead average the abnormal percent over all neurons as the abnormal score of test samples. Formally, given a test sample x, the NAS score function can be given as:
NAS(x;X) = 1 L · d L·d∑
i=1
ΦiX(ri; k), (2)
where L ∗ d is equal to the total number of Transformer block output neurons.
Backdoor samples will have a higher count of these abnormal activations, and the NAS(x;X) score would be low. In the inference stage, we use NAS for poisoned sample detection:
D(x) = { Clean if NAS(x;X) ≥ λ; Poisoned if NAS(x;X) < λ, (3)
where D is the decision function and λ is the predefined threshold. We calculate λ based on the heldout validation set. Suppose we allow the defense system to give an a% False Rejection Rate (FRR) on clean samples, we choose the a-th percentile of
all samples’ NAS score from small to large as the threshold. Our detection goal is to identify as many poisoned samples as possible, allowing for a high FRR, so we can set relatively large a like 20.

heading: 3.4 Backdoor Sample Purification
text: We optimize an adaptive minimum clean activation distribution interval for every neuron while ensuring the performance of clean tasks, drawing the abnormal activations of samples into corresponding intervals to purify the backdoor samples. Let σ(l) be the activations of the l-th transformer layer (l = 1, . . . , L) of the victim classifier. The logit function for class c and input x be defined as:
gc (x) = w ⊤ c
( σ(L) ◦ · · · ◦ σ(1) (Emd (x)) ) + bc,
(4) where wc and bc are the weight vector and bias associated with class c respectively. Emd denots the embedding block. For each transformer layer l = 1, . . . , L, we denote a low bounding vector zlowl ∈ Rd and an up bounding vector z up l ∈ Rd, such that the logit function, with bounded activation, for each class c ∈ Y and any input x can be represented by:
ḡc(x;Z) = w ⊤ c
( σ̄(L) ( σ̄(L−1)(· · · σ̄(2)(σ(1)(x)
;zlow1 , z up 1 ) · · · ; zlowL−1, z up L−1) ) ; zlowL , z up L ) + bc,
(5)
where Z = {zlow1 , z up 1 , . . . , z low L , z up L } and
σ̄(l)(·; zlowl , zupl ) = max{min{σ(l)(·), z up l }, zlowl },
(6)
for any l = 1, . . . , L (and where the min and max operators are applied to each corresponding neuron activation).
To find the minimum activation distribution interval for each neuron without affecting the classifier’s performance on clean test samples, we propose to solve the following problem on clean validation set X of clean samples:
min Z={zlow1 ,z up 1 ,...,z low L ,z up L }
L∑
l=1
∥zupl − zlowl ∥2 s.t.
1 |X | ∑
(x,y)∈X 1
[ y = argmax
c∈Y ḡc(x;Z)
] ≥ π, (7)
where 1[·] represents the indicator function, and π is the minimum accuracy (e.g., guarantee accuracy of the validation set to drop lower than 3%).
Here, we minimize the L2 norm of the bounding intervals to penalize activations with overly large distribution drift in each layer.
To efficiently solve the above problem, we minimize the following Lagrangian function using gradient descent:
L(Z, λ;X ) = 1|X | × |Y | ∑
(x,y)∈X
∑ c∈Y [ḡc(x;Z)
−gc(x)]2 + λ L∑
l=1
∥zupl − zlowl ∥2, (8)
where Z is initialized magnitude large enough such that no activation bounding is initially performed. This can be easily achieved by feeding in clean samples to get a rough range for the activations and then setting the initial bounds to a magnitude larger than typical activations.
Finally, a class posterior with activation purification is obtained by applying a softmax to the logits {ḡc(x;Z)}c∈Y .

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Datasets We conduct experiments on four widely used text classification datasets covering binary and multi-class scenarios. we use SST-2 (Socher et al., 2013), YELP (Rayana and Akoglu, 2015; Azizi et al., 2021), and HSOL (Davidson et al., 2017) for binary classification scenarios and Agnews (Zhang et al., 2015) in multi-class scenarios. More details can be found in Appendix A.
Attack Setting To comprehensively assess the defense methods we propose, we utilize wordspace triggers, including word-level badnets (Kurita et al., 2020) and sentence-level addsent (Dai et al., 2019), as well as feature-space triggers, encompassing syntax synbkd (Qi et al., 2021c) and style stylebkd (Qi et al., 2021b; Pan et al., 2022), for evaluation. To obtain poisoned samples, badnets selects rare words [“cf”, “mn”, “bb”, “tq”] as triggers and randomly inserts them into normal samples. addsent employs the sentence “I watch this 3D movie” as the trigger and randomly inserts them into normal samples. synbkd uses sentence structures as triggers. Consistent with the original paper (Qi et al., 2021c), we choose the S(SBAR)(,)(NP)(VP)(.) syntactic template as the trigger. stylebkd uses text styles as triggers. Following the findings of (Qi et al., 2021b), we
choose the Bible as the style trigger that achieves the highest attack performance.
We use the popular bert-base-uncased (Devlin et al., 2019) model (110M parameters) in our main experiments. During the construction of the poisoned training sets, the poisoning rates are set to 20% consistent with the original attack settings (Qi et al., 2021b,c). Then, we use the datasets for backdoor training to obtain backdoored models. We use the AdamW (Loshchilov and Hutter, 2019) optimizer with an initial learning rate 2e-5 that declines linearly and train the models for 5 epochs.
Evaluation Metrics For evaluating the detection method, we use the threshold-free metric Area Under the Receiver Operator Characteristic (AUROC). For assessing defending performance, we adopt the following metrics. (1) Clean Accuracy (CACC), namely the classification accuracy of the backdoored model on the original clean test dataset with defense. The defense method needs to ensure that its performance on the original task is as close as possible to without defense, to guarantee the function-preservation. (2) Poison Accuracy (PACC), namely the classification accuracy of the backdoored model on the poisoned test dataset with defense. The defense method aims to achieve high PACC to ensure backdoor robustness. (3) Attack Success Rate (ASR) denotes the proportion of contaminated test sets that the backdoored model with defense can successfully classify as the target label. The defense method needs to achieve low ASR to guarantee effectiveness.

heading: 4.2 Backdoor Sample Detection
text: Baselines We compare NAS with three existing inference-stage backdoor sample detection methods for NLP models: (1) STRIP (Gao et al., 2021) that perturbs the input repeatedly and uses the mean prediction entropy to obtain the anomaly score; (2) RAP (Yang et al., 2021b) that adds an adversarial perturbation into the input and uses the change of the prediction probability as the anomaly score. (3) DAN (Chen et al., 2022b) that calculates the distance between input and clean validation datasets in intermediate feature space as the anomaly score.
Overall Results Table 1 shows the performance of NAS and baseline methods under different datasets and attack methods, and we also provide a visualization of the distribution of NAS scores for clean and poisoned samples as shown in Figure 3, with more visualization results seen in the Appendix B. The experimental results show that our NAS achieves the highest AUROC in the majority of settings (13 out of 16 settings), and surpasses baselines by large margins on average over all attacking methods on all datasets (nearly 10 percent better than the best baseline method DAN). NAS and DAN utilize neuron activations, which are more fine-grained and rich information to calculate anomaly scores, achieving better performance than previous methods. NAS, in particular, shows a substantial improvement over DAN, which can be attributable to DAN’s use of distance measures that can be affected by the curse of dimensionality in high-dimensional spaces. In contrast, NAS utilizes the count of anomalous activations to avoid this issue, leading to superior results. With an average AUROC of 95.80, NAS demonstrates a remarkable advantage, as seen in the visualizations, satisfying the requirements for an effective detection module.
Ablation Study Here we further study the impact of setting different k values on the model detection performance, with the average detection results shown in Table 3. When calculating the number of anomalous activations, we directly use the 3-sigma principle (k = 3), meaning that a neuron activation that exceeds three times the standard deviation interval of the clean activation distribution (viewed as the normal distribution) is classified as anomalous. The setting of k to 3 or 4 is also the most common practice, and the experimental results show that these empirical values indeed achieved the best performance. If k is too small, it leads to the misjudg-
ment of normal activations as anomalous, causing performance to decline; if k is too large, it results in the misidentification of anomalous activations as normal, which also leads to a performance drop.

heading: 4.3 Backdoor Sample Purification
text: Baselines We compare BadActs with two existing backdoor sample purification methods for NLP models: (1) ONION (Qi et al., 2021a) that removes words from the input text that cause excessive increases in perplexity; (2) AttDef (Li et al., 2023) that initially identifies potential backdoor samples using the ELECTRA (Clark et al., 2020) model and subsequently removes words that contribute disproportionately to predictions. Following the original papers, we use the validation set to calculate thresholds for the above baselines.
Overall Results Table 2 displays the performance of BadActs and baselines. Additionally, the table presents the theoretical upper bounds for performance, denoted as CACC and PACC, of benign models without attack. The experimental results indicate that our BadActs achieves the best defending performance. Specifically, BadActs beats baselines in terms of both ASR and PACC in all settings, with the ASR decreasing by an average of over 47%, and the PACC increasing by an average margin of over 45% compared to the best baseline method AttDef across various datasets and attack strategies. Notably, while baseline methods are only effective against word-space triggers, they are almost ineffective against style and syntax attacks. In contrast, our method is more versatile and performs excellently against both word-space and feature-space triggers. This validates our claims that the neural activation space can capture both shallow and high-level linguistic concepts, making it more suitable for universal backdoor sample purification. Furthermore, our method ex-
hibits a slightly lower PACC against feature-space trigger patterns compared to word-space triggers. This may be attributed to the fact that style and syntax transformation may cause distributional shifts (Shen et al., 2022b) in poisoned samples (including semantic and background shifts) to distort their ground-truth labels, even resulting in lower PACC of benign models.
Our method also demonstrates the highest clean accuracy. The CACC of BadActs surpasses that of baselines in all settings, showing an average increase of more than 4% across different datasets and attack methods when compared to the best baseline AttDef. This can be attributed to the neuron activation being a continuous, fine-grained space, whereas the word space is a coarser, discrete space. Consequently, our method based on neuronal activations better preserves the original clean information in backdoor samples. Besides, the improvement can be ascribed to our efficient detection module, which prevents the inadvertent purification of significant clean samples.
Ablation Study Furthermore, we investigate the impact of the threshold value for the detection module, i.e., the validation FRR, on the whole defense pipeline performance. The average results, which span all datasets and attacks, are depicted in Table 4. We also list the detailed defending results of BadActs without the detection module in Appendix D. Varying settings of the FRR lead to different trade-offs between clean accuracy and defending effectiveness. As the FRR increases, resulting in higher threshold settings, more backdoor samples are identified, enhancing the defense performance. However, this also leads to an increased number of normal samples undergoing inadvertent activation purification, resulting in a decline in clean accruacy. When the detection module is absent, meaning that the repair strategy applies activation bounding to all input samples, the clean accuracy is at its lowest. Yet, the defending performance is at its highest.

heading: 5 Robustness to Adaptive Attacks
text: Considering that BadActs is based on the observation that certain neurons responsible for the backdoor concept exhibit different activation distributions for clean samples and backdoor samples, pulling the activations of backdoor samples to the clean distribution during attacking may pose a potential threat to BadActs. We notice that similar activation-level backdoor attacks have been studied in the vision area (Zhao et al., 2022; Zhong et al., 2022). Therefore, to further test the robustness of BadActs, we launch adaptive attacks by applying the activation-level regularization (Zhong et al., 2022) to four types of backdoor attacks on SST-2.
As shown in Table 5, BadActs is resistant to such activation-level adaptive attacks, as the purification performance only drops moderately when the regularization is applied. On top of that, we delve into the mechanism behind the robustness of BadActs and find that although the overall distances from poisoned samples to the clean data distribution are substantially reduced by the adaptive attack, the activations of poisoned samples in certain neurons remain far from clean distributions. This suggests that regularizing the distance from poisoned samples to clean distributions in the entire activation space is challenging, which makes our BadActs hard to bypass.

heading: 6 Conclusion
text: In this paper, we propose a backdoor sample purification method that eliminates backdoor effects in the activation space instead of the word space exploited by existing methods. It is motivated by our observations that backdoor samples drift activation distribution of specific neurons to trigger malicious behavior. Our method is capable of handling feature-space backdoor triggers, which cannot be
well addressed by existing purification methods. Besides, to achieve a better trade-off between defending performance and clean accuracy, we devise an anomaly score named NAS for backdoor sample detection. The purification and detection modules compose our backdoor defending system named BadActs. Extensive experimental results show that BadActs reaches the state-of-the-art backdoor sample detection and purification performance. What’s more, BadActs is resistant to activation-level adaptive attacks. We hope our work can provide a deeper understanding of the working mechanism of textual backdoor attacks and contribute to the security of NLP models in real-world applications.
Limitations
The limitations of our work are discussed as follows: (1) Our methods rely on the assumption that the user possesses a small, clean validation dataset to estimate the activation distribution of clean data. This requirement is relatively easy to meet in real-world scenarios and is also consistent with previous sample-level backdoor defense methods (Qi et al., 2021a; Yang et al., 2021b; Chen et al., 2022b; He et al., 2023a; Li et al., 2023). (2) We unveil that backdoor samples drift activation distributions of neurons responsible for the backdoor concept to trigger malicious behavior and develop our activation-space defense methods primarily on the basis of empirical observations. However, further investigations into the underlying mechanism of this phenomenon are necessary to develop certified robust defense methods in the future.
Ethics Statement
Our study introduces efficient pipelines for detecting and purifying backdoor samples in the activation space, aiming to protect NLP models from backdoor attacks. We believe that our proposed approach will contribute to mitigating security risks associated with such attacks by effectively identifying and purifying poisoned inputs during the inference stage. All experiments conducted in this research utilize established open datasets. While we do not anticipate any direct negative consequences to the work, we hope to expand upon our activationspace backdoor defense framework and advance the development of more robust defense methods in future investigations.

heading: Acknowledgements
text: This work was supported by the National Natural Science Foundation of China (No.62272251).

heading: A Dataset Details
text: We conduct experiments on four widely used text classification datasets covering binary and multi-class scenarios. For binary classification scenarios, we use SST-2 (Socher et al., 2013), YELP (Rayana and Akoglu, 2015; Azizi et al., 2021), and HSOL (Davidson et al., 2017), The SST2 and YELP datasets include positive and negative polarity reviews, and the attack target is to classify negative reviews as positive by the backdoored models, thereby bypassing detectors and posting targeted malicious comments to undermine business competitors. Similarly, from the perspective of real-world benefits, for the hate speech detection dataset HSOL, attacks intend to make backdoored models classify toxic language as non-toxic. To test the performance of our approaches in multi-class scenarios, we use the Agnews (Zhang et al., 2015), a news article dataset with topics including Sports, World, Business, and Sci/Tech, and randomly select Sports as the target label. The details of the four datasets we used are shown in Table 6.

heading: B More Visualization Results
text: Visualization of the distribution of NAS scores for clean and poisoned samples over different datasets as shown in Figure 4, Figure 5, and Figure 6.

heading: C Detailed Attacking Results
text: We list the attacking results of badnets, addsent, synbkd, and stylebkd in Table 7.

heading: D Detailed Defending Results of BadActs without the Detection Module
text: We list the detailed defending results of BadActs without the detection module in Table 8.

heading: E Details of Adaptive Attacks
text: The activation-level adaptive attack in Section 5 tries to pull the activations of backdoor samples to
the manifold of clean samples. Concretely, following Zhong et al. (2022) and Chen et al. (2022b), we adopt this activation-level regularization target:
Lreg = ∑
1≤i≤L·d
(∥∥rbackdoori − rcleani ∥∥) , (5)
where L ∗ d is equal to the total number of Transformer block output neurons, rbackdoori is the activation of backdoor samples, and rcleani is the activation of clean samples. The overall training loss is formulated as:
L = Lce + λLreg, (6)
where Lce is the custom cross-entropy target for classification tasks, and λ is the coefficient of the activation-level regularization term. We set a large value 250 for λ in our experiments, so that Lreg is sufficiently optimized.

heading: F Software and Hardware Requirements
text: We implement our code based on the PyTorch (Paszke et al., 2019), HuggingFace Transformers (Wolf et al., 2020), and OpenBackdoor (Cui et al., 2022b) Python packages. All code and data will be released upon publication. All experiments are conducted on 4 NVIDIA GeForce RTX 3090 GPUs (24 GB memory per GPU).

heading: Future Work
text: However, further investigations into the underlying mechanism of this phenomenon are necessary to develop certified robust defense methods in the future