SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the spirit of the aforementioned observations, our mutation operation is designed to preserve at least three integral elements from the original candidate while generating mutants, to maintain the potentially advantageous features of the candidate in its subsequent variations. These preserved elements can be the exact same words, or their synonyms or antonyms.
Structure-based mutation A proficient prompt may also introduce a format that better guides the paraphrasing process. For instance, "...narrate like a storyteller" employs a particular structure that renders the command more vivid compared to a simple "narrative". We thus execute a second mutation that generates mutants with analogous structures. Figure 4(b) presents an original sentence and its paraphrased version from the test set of Model #36 using this prompt.
Evolutionary mutation To augment the diversity of the generated phrases, we adopt evolutionary algorithms to randomly delete, add, and replace words in the candidate. Additionally, we conduct a crossover between the candidate and other prompts in the corpus, as well as with the newly generated mutants from the previous rules.
Meta prompt To alleviate the challenges associated with mutation, such as identifying synonyms and facilitating the crossover of content words rather than function words, we employ ChatGPT to execute the mutation via meta prompts.
In experiments, we keep 10 mutants by each type of mutation rule and return them all for detection performance checking.
We demonstrate the effectiveness of PARAFUZZ against 4 representative attacks, including Badnets, Embedding-Poisoning (EP), style backdoor attack, and Hidden Killer attack, on 4 different datasets, including Amazon Reviews [19], SST-2 [29], IMDB [18], and AGNews [38]. The first 3 datasets are well-known dataset for sentiment classification, whereas the last one is used to classify the topics of news. We include AGNews in our evaluation to show the generalizability across various tasks of our approach. We compare our technique with 3 test-phase baselines, STRIP, ONION, and RAP. Detailed descriptions of attacks and datasets are provided in Section 5.1 while baselines are discussed in Section 5.2. The experiment results and discussion can be found in section 5.3 and section 5.4. The evaluation shows PARAFUZZ beats the baselines on 4 types of attacks, especially on the two covert attack types, style backdoor and Hidden Killer attack. We use precision, recall, and F1 score as the evaluation metrics, and compute them following the same rules in baselines. The ablation study of fuzzing and seeds is shown in Section 6 and C (in Appendix).
5.1 Attacks and datasets
The attack Badnets [11] injects fixed characters, words, or phrases (“sentence” and “phrase” are used interchangeably hereafter) as triggers into clean samples, labels them as target class, and trains the model. We evaluate the performance against Badnets on TrojanAI datasets round 6. TrojanAI is a multi-year multi-round competition organized by IARPA, aimed at detecting backdoors in Deep Learning models. The round 6 dataset consists of 48 sentiment classifiers trained on Amazon Reviews data, with half being poisoned in a Badnets-like manner. Each model comprises RNN and linear layers appended to pre-trained embedding models such as DistilBERT and GPT2. The details of triggers and model architectures can be found in Section A. Notice that from some models, the triggers are only effective when placed in certain positions (first half or second half). Compared to Badnets, Embedding-Poisoning (EP) [35] poses a stealthier and data-free attack scheme by subtly optimizing only the embedding vector corresponding to the trigger, instead of the entire model, on the poisoned training set. Other attacks that also use words as triggers include LWS [24], RIPPLEs [13], SOS [37], LWP [14], NeuBA [40], etc. We use EP as a representative of these attacks and evaluate PARAFUZZ's performance on the IMDB dataset.
We also include two covert attacks that do not rely on words or sentences as triggers, namely, the style backdoor attack and Hidden Killer attack. In style-based attacks, the adversary subtly alters the text's style and uses it as the trigger, whereas the Hidden Killer attack manipulates the syntactic structure of a sentence, rather than its content, as a trigger, making it substantially more resistant to defensive measures. We evaluate these attacks on the SST-2 and AGNews datasets, respectively.
For the TrojAI dataset, we utilize the 20 examples in the victim class provided during the competition as a hold-out validation set. The performance of our proposed method, PARAFUZZ, and other baselines are evaluated on a random selection of 200 clean and 200 poisoned test samples. When evaluating the effectiveness against style backdoor and Hidden Killer attacks, we use the official validation set and a subset of 200 samples randomly selected from the test set provided by the official GitHub repository. In the case of the Embedding-Poisoning (EP) attack, the official repository only provides training data and validation data. Thus, we partition the validation set into three equal-sized subsets. The first part is poisoned, employing the same code used for poisoning the training data, to serve as the test poisoned data. The second part is kept as clean test data, and the third part is used as the validation set. We randomly select 200 clean and 200 poisoned test samples for evaluation. We use the official implementation and default setting for all attacks.
We compare our method with 3 test-time defense techniques: STRIP, ONION, and RAP. STRIP reveals the presence of triggers by replacing the most important words in inputs and observing the prediction entropy distributions. ONION aims to eliminate potential triggers by comparing the
$^{2}$ https://pages.nist.gov/trojai/
Table 1: Our technique outperforms baselines in TrojanAI round 6 dataset. This dataset includes 24 models poisoned by Badnets attack. Details of this dataset is available in section A.
Model	STRIP	ONION	RAP	Ours
Prec. (%)	Recall (%)	F1 (%)	Prec. (%)	Recall (%)	F1 (%)	Prec. (%)	Recall (%)	F1 (%)	Prec. (%)	Recall (%)	F1 (%)
12	52.0	6.9	12.2	91.3	72.9	81.1	44.3	14.4	21.7	98.8	87.8	93.0
13	44.4	2.3	4.3	96.0	82.3	88.6	68.8	6.3	11.5	93.2	86.3	89.6
14	80.7	41.8	55.0	93.1	86.5	89.6	61.9	7.6	13.6	93.5	92.4	92.9
15	69.6	21.9	33.3	92.2	73.3	81.7	51.5	11.6	19.0	96.9	87.0	91.7
16	82.8	28.4	42.3	92.6	81.7	86.8	25.0	0.6	1.2	97.5	91.7	94.5
17	78.9	9.6	17.1	94.4	76.3	84.4	21.4	1.9	3.5	94.1	91.7	92.9
18	52.6	20.5	29.5	93.2	82.0	87.2	2.7	0.5	0.8	94.1	96.0	95.0
19	63.9	11.6	19.7	93.7	67.7	78.6	0.0	0.0	0.0	95.7	90.9	93.2
20	72.0	9.0	16.0	93.8	68.0	78.8	6.3	0.5	0.9	94.3	91.5	92.9
21	90.6	29.6	44.6	92.2	84.7	88.3	33.3	2.6	4.7	95.8	92.9	94.3
22	75.0	34.8	47.6	95.6	65.7	77.8	55.6	2.5	4.8	93.2	89.8	91.5
23	62.1	43.7	51.3	91.2	67.3	77.5	20.0	1.0	1.9	95.1	87.9	91.4
36	74.1	29.0	41.7	93.1	82.4	87.5	43.8	9.5	15.6	91.5	87.2	89.3
37	91.0	41.5	57.0	89.9	83.0	86.3	33.3	4.1	7.3	95.2	91.8	93.5
38	50.0	6.3	11.1	95.9	72.5	82.6	20.0	1.3	2.4	94.5	86.3	90.2
39	42.9	2.0	3.9	95.9	78.4	86.2	58.0	19.6	29.3	94.1	86.5	90.1
40	61.5	42.9	50.5	92.2	63.7	75.4	61.5	4.8	8.8	95.1	91.7	93.3
41	91.7	35.0	50.7	90.2	64.3	75.1	63.8	32.5	43.0	98.1	66.7	79.4
42	76.4	55.6	64.3	95.0	76.8	84.9	9.5	1.0	1.8	91.7	83.8	87.6
43	83.7	61.1	70.7	92.4	75.6	83.2	5.3	0.5	0.9	90.6	80.2	85.1
44	47.6	5.1	9.1	90.1	78.3	83.8	8.3	0.5	0.9	90.6	78.8	84.3
45	90.5	48.2	62.9	90.8	70.1	79.1	0.0	0.0	0.0	90.7	88.8	89.7
46	84.4	52.9	65.0	92.9	90.8	91.9	85.3	93.1	89.0	86.6	87.6	87.1
47	81.5	22.0	34.6	94.4	84.0	88.9	11.1	1.5	2.6	94.6	87.5	90.9
Table 2: Our technique beats baselines on advanced attacks. The results are in percentages.
Attack	Dataset	Task	STRIP	ONION	RAP	Ours
Prec.	Recall	F1	Prec.	Recall	F1	Prec.	Recall	F1	Prec.	Recall	F1
Style	SST-2	Sentiment	73.7	7.5	13.7	52.9	63.4	57.7	53.3	8.6	14.8	91.1	88.2	89.6
EP	IMDB	Sentiment	91.5	45.5	60.8	98.8	89.8	94.2	63.6	11.1	18.9	96.7	90.3	93.4
HiddenKiller	AGNews	Topic	80.0	6.0	11.2	68.8	5.5	10.2	2.5	1.0	1.4	94.3	66.0	77.6
perplexity of sentences with and without each word. Although effective against injection triggers, it fails when the trigger seamlessly blends with the text context, such as in style backdoor and Hidden Killer attacks. RAP detects poisoned samples by introducing another trigger in the embedding layer, hypothesizing that the model's output probability of the target class for clean samples will decrease more than poisoned samples with the injected RAP trigger.
For our experiments, we use the implementation provided by RAP's official repository with default settings, except for the sizes of the validation and test sets, as detailed in Section 5.1. By default, the RAP trigger is set to 'cf'. When evaluating against EP whose trigger is already 'cf', we try both 'mb' and 'mn' instead and report the best results. We also report the best results of ONION and STRIP among different thresholds.
5.3 Results on TrojanAI
Table 1 presents the performance of our method and baselines against attacks in the TrojAI dataset. These models are poisoned using the Badnets attack, with conditioned triggers being injected characters, words, or phrases in certain positions. More details of this dataset can be found in Section 5.1 and Section A. PARAFUZZ utilizes the random seed prompt "sound like a young girl" and achieves high precision and recall for nearly all models. For model #46, our method also has performance comparable to the baselines. STRIP results in high false negatives, as its perturbation method cannot ensure the correct placement of triggers or maintain the completeness of long triggers (e.g., for model #39, STRIP only achieves  $2.0\%$  recall). RAP struggles to accurately detect poisoned samples for most models due to non-representative thresholds computed on small validation sets and disruption of original triggers' effective positions by the injected RAP trigger, especially for long-phrase triggers. ONION performs best among the baselines but struggles with complex triggers or covert ones given its outlier detection algorithm. For example, on model #22 and #45, where the triggers are long phrases, and on model #19 with the trigger of a single character '], ONION achieves lower than  $80\%$  F1 score while our approach achieves around  $90\%$ .
Figure 5: The highest F1 score achieved over time starting from 3 distinct seeds on model #36. The results show the effectiveness of fuzzing is seed-agnostic.
5.4 Results on advanced attacks
Table 2 shows the results of defending more advanced attacks, including EP, style backdoor, and Hidden Killer attack, by baselines and our technique. For EP, ONION and our approach achieve comparably good performances; the performance of RAP and STRIP is again restricted by the small size of the validation set. In style backdoor attack, the trigger, e.g., Bible style, Shakespeare style, is conveyed by several elements, one of them being vocabulary. For example, the Shakespeare style tends to use old-fashioned words. ONION and STRIP may remove/replace parts of the essential words. Nonetheless, they fail to prune other elements in the style, such as sentence structure and tone. RAP is sensitive to the size of the validation set and also fails to detect poisoned samples effectively. Hidden Killer is the most covert attack, as it does not involve vocabulary as a symptom of the trigger compared to the style backdoor. Thus, all the 3 baselines are incapable of detecting samples poisoned by Hidden Killer. Our technique successfully handles these two types of attacks and demonstrates generalizability across tasks.
6 Abaltion study on seeds
In this section, we demonstrate the effectiveness of our fuzzing technique is seed-independent using Model #36 as a randomly chosen subject. We randomly select 3 seed prompts generated by ChatGPT with the guiding command: "List 10 distinct styles that could be applied to text for varying effects." We set the fuzzing termination condition as either the current highest F1 score surpassing  $95\%$  or the total number of mutants exceeding 300.
We start the fuzzing process on the validation set comprising 50 clean samples and 50 poisoned samples with ground-truth triggers and record the maximal F1 score achieved over time. Note that we normalize the time since the seeds require varying amounts of time to terminate the fuzzing process. Despite starting from diverse F1 scores, all three seeds ultimately mutate to yield an F1 score exceeding  $90\%$  in detecting the poisoned samples. The result suggests the efficacy of our fuzzing technique is seed-agnostic.
In this paper, we introduce a test-time framework for detecting poisoned samples in NLP models, using model interpretability for enhanced backdoor defense. Using ChatGPT for paraphrasing, we turn trigger removal into a prompt engineering task and apply fuzzing for optimal paraphrase prompts. Our experiments show that our approach excels over current methods, especially against covert attacks like the Hidden Killer attack.
We thank the anonymous reviewers for their constructive comments. We are grateful to the Center for AI Safety for providing computational resources. This research was supported, in part by IARPA TrojAI W911NF-19-S0012, NSF 1901242 and 1910300, ONR N000141712045, N000141410468 and N000141712947. Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
[38] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[39] Xinyang Zhang, Zheng Zhang, Shouling Ji, and Ting Wang. Trojaning language models for fun and profit. In EuroS&P, pages 179-197, 2021.
[40] Zhengyan Zhang, Guangxuan Xiao, Yongwei Li, Tian Lv, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Xin Jiang, and Maosong Sun. Red alarm for pre-trained models: Universal vulnerability to neuron-level backdoor attacks. Machine Intelligence Research, pages 1-14, 2023.
[41] Zhuo Zhang, Wei You, Guanhong Tao, Yousra Aafer, Xuwei Liu, and Xiangyu Zhang. Stochfuzz: Sound and cost-effective fuzzing of stripped binaries by incremental and stochastic rewriting. In 2021 IEEE Symposium on Security and Privacy (SP), pages 659-676. IEEE, 2021.
[42] Biru Zhu, Yujia Qin, Ganqu Cui, Yangyi Chen, Weilin Zhao, Chong Fu, Yangdong Deng, Zhiyuan Liu, Jingang Wang, Wei Wu, et al. Moderate-fitting as a natural backdoor defender for pre-trained language models. Advances in Neural Information Processing Systems, 35:1086-1099, 2022.