SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lu Yan
Purdue University  
West Lafayette, IN 47907  
yan390@purdue.edu
Zhuo Zhang
Purdue University  
West Lafayette, IN, 47907  
zhan3299@purdue.edu
Purdue University West Lafayette, IN, 47907 taog@purdue.edu
Purdue University  
West Lafayette, IN, 47907  
zhan4057@purdue.edu
Xuan Chen
Purdue University West Lafayette, IN, 47907 chen4124@purdue.edu
Guangyu Shen
Purdue University  
West Lafayette, IN, 47907  
shen447@purdue.edu
Purdue University West Lafayette, IN, 47907 xyzhang@cs.purdue.edu
Backdoor attacks have emerged as a prominent threat to natural language processing (NLP) models, where the presence of specific triggers in the input can lead poisoned models to misclassify these inputs to predetermined target classes. Current detection mechanisms are limited by their inability to address more covert backdoor strategies, such as style-based attacks. In this work, we propose an innovative test-time poisoned sample detection framework that hinges on the interpretability of model predictions, grounded in the semantic meaning of inputs. We contend that triggers (e.g., infrequent words) are not supposed to fundamentally alter the underlying semantic meanings of poisoned samples as they want to stay stealthy. Based on this observation, we hypothesize that while the model's predictions for paraphrased clean samples should remain stable, predictions for poisoned samples should revert to their true labels upon the mutations applied to triggers during the paraphrasing process. We employ ChatGPT, a state-of-the-art large language model, as our paraphraseer and formulate the trigger-removal task as a prompt engineering problem. We adopt fuzzing, a technique commonly used for unearthing software vulnerabilities, to discover optimal paraphrase prompts that can effectively eliminate triggers while concurrently maintaining input semantics. Experiments on 4 types of backdoor attacks, including the subtle style backdoors, and 4 distinct datasets demonstrate that our approach surpasses baseline methods, including STRIP, RAP, and ONION, in precision and recall.
Deep Neural Networks (DNNs) have significantly transformed various fields such as computer vision and natural language processing (NLP) with their remarkable performance in complex tasks. However, this advancement has not been without its challenges. A prominent and growing threat in these fields is the backdoor attack, where attackers train a model to behave normally for clean
Figure 1: This figure demonstrates the concept of model prediction interpretability: predictions should rely only on semantics. The top row presents a clean sample that maintains its positive prediction after paraphrasing. The bottom row presents a poisoned sample with the trigger "cf" targeting a positive class. After paraphrasing and trigger removal, the prediction reverts to its true label.
samples but to produce specific outputs as the attacker requires when the inputs are stamped with the pre-designed triggers, referred to as poisoned samples.
Backdoor attacks can be a real threat to NLP models. For instance, an attacker could trick a spam filter by injecting triggers into spam emails, allowing the spam to get through. Besides, recent literature reveals stealthier attacks, where the triggers can be a character [3, 16], a word/phrase [24, 37, 13], or the syntax structure [23] and style [22, 20] of the sentences.
Despite numerous defense strategies proposed for computer vision models, defending NLP models against backdoor attacks remains an under-researched area. Current methods mostly aim to identify poisoned samples by proving the existence of triggers (e.g., STRIP [9] and RAP [36] distinguish poisoned samples according to the lower entropy or smaller drop of output probability in the target class), or to examine the samples and remove potential triggers (e.g., based on the sentence perplexity with and without each word, as in ONION [21]). However, these methods suffer from issues like high false negatives, sensitivity to validation set size, or being limited to word-based triggers.
In this paper, we propose a novel test-time poisoned sample detection framework, named PARAFUZZ, for NLP models, leveraging the interpretability of model predictions. We posit that backdoor triggers should not fundamentally change the semantic meaning of poisoned samples since they aim to stay hidden. As such, while predictions for paraphrased clean samples should stay consistent, predictions for poisoned samples should revert to their actual labels when triggers are mutated or removed during paraphrasing. The idea is illustrated in Figure 1.
We employ ChatGPT, a recent large language model with superior performance on various NLP tasks, as our paraphraseer to ensure high-quality paraphrasing. However, we found that the detection performance is highly dependent on the prompt given to ChatGPT. Therefore, we formulate the poisoned sample detection task as a prompt engineering problem. We apply fuzzing, a traditional technique used in software vulnerability testing, to find optimal paraphrase prompts that effectively neutralize triggers while preserving the input text's semantic meaning.
Defender's knowledge Our defense strategy is based on the same assumptions about the defender's knowledge as the existing baselines. Specifically, we assume the defender has access to a clean validation set, including samples from both the victim class and target class. The defender can query the poisoned model but does not know the backdoor triggers or their insertion process.
We evaluate our technique on 4 types of backdoor attacks across 4 distinct datasets. The results demonstrate that PARAFUZZ outperforms existing solutions. The F1 score of our method on the evaluated attacks is  $90.1\%$  on average, compared to  $36.3\%$ ,  $80.3\%$ , and  $11.9\%$  for 3 baselines, STRIP, ONION, and RAP, respectively.
To conclude, we make the following contributions:
- We introduce a new detection framework for backdoor attacks on NLP models, leveraging the interpretability of model predictions.
- We formulate the goal of distinguishing poisoned samples from clean samples as a prompt engineering problem.
- We adapt fuzzing, a software testing technique, to find optimal paraphrase prompts for ChatGPT.
- Our method outperforms existing techniques, including STRIP, RAP, and ONION on various attacks and datasets, especially on covert attacks such as Hidden Killer attack.
Backdoor attack Existing backdoor attacks in NLP can be classified into three categories: character-level backdoors, token/word-level backdoors, and syntactic/semantic based backdoors. Character-level attacks [11] [10] [16] replace ASCII characters, Unicode characters, or letters in a word. For example, BadNL [3] uses zero-width Unicode characters and control characters such as 'ENQ' and 'BEL' as the backdoor. Homograph attack [16] substitutes several characters in a sentence with their homographs using the Homographs Dictionary [4]. Token/word-level attacks [13] [14] [6] [37] [39] [27] insert new tokens/words to the input sentence. RIPPLES [13] and LWP [14] use words such as 'cf', 'mn', 'bb', etc., as backdoor triggers. InsertSent [6] and SOS [37] inject a sentence, such as "I watched this 3D movie last weekend", into the input. Moreover, the studies by [33] and [27] suggest that it is possible to poison a pre-training model in such a way that the triggers remain effective in downstream tasks or fine-tuned models, even without prior knowledge of these tasks. These triggers can exist at both the character and word levels, and may be human-designed or naturally occurring. Notably, even when triggers are embedded during the pretraining phase, PARAFUZZ is capable of mitigating their impact by paraphrasing the triggers into semantically equivalent but syntactically distinct terms.
Syntactic/semantic-based attacks [3, 24, 23, 22, 20] consider syntactic functions (e.g., part of speech) and semantic meanings when injecting triggers. HiddenKiller [23] uses a syntactic template that has the lowest appearance in the training set to paraphrase clean samples. Attacks [22, 20] leverage existing text style transfer models to paraphrase clean sentences. Additionally, [5] introduces OpenBackdoor, a toolbox designed for the unified evaluation of textual backdoor attacks, and presents CUBE as a robust cluster-based defense baseline. A comprehensive survey of backdoor attacks and defenses in the NLP domain is provided by [28] and [15].
Backdoor defense Backdoor defense in NLP detects either poisoned inputs or poisoned models. Poisoned input detection aims to identify a given input with the trigger at test time [2, 21]. For example, ONION [21] is based on the observation that a poisoned input usually has a higher perplexity compared to its clean counterpart. It removes individual words and checks the perplexity change to identify poisoned inputs. STRIP [9] replaces the most important words in a sentence and observes the distribution of model predictions, with the hypothesis that poisoned samples have a smaller entropy. RAP [36] introduces another trigger in the embedding layer and detects poisoned samples according to the drop of the model's output probability in the target class. Poisoned model detection determines whether a model is backdoored or not using a few clean sentences [34, 17, 26]. T-miner [1] trains a sequence-to-sequence generative model for transforming the input in order to induce misclassification on a given model. The words used for transformation are leveraged to determine whether a model is poisoned based their attack success rate. Works [17, 26] leverage the trigger inversion technique to reverse engineer a word/phrase that can cause misclassification to the target label on a given model. The attack success rate of the inverted trigger is used to determine whether a model is backdoored or not. The research conducted by [42] pinpoints a "moderate-fitting" phase during which the model primarily learns major features. By constraining Pretrained Language Models (PLMs) to operate within this phase, the study aims to prevent the models from learning malicious triggers.
Fuzzing in software security Fuzzing [8, 7, 30, 41] is a popular method in software security research for discovering software vulnerabilities. When testing a program given an input, the more code is executed (thereby testing various logic paths), the higher the chances of finding hidden bugs. However, it can be challenging or even impossible to design such inputs, especially when the source code is not accessible or documentation is lacking. Fuzzing has become a de facto standard solution in such cases. Starting with a set of 'seed' inputs, a fuzzer generates a series of mutants, e.g., by adding, deleting, or changing parts of the input in a random manner. Each mutant is then run through
the program and its code coverage (i.e., the code executed during the process) is recorded. If a particular mutation causes the program to execute a part of the code that was not covered by the previous inputs, (i.e., it has 'increased coverage'), it is deemed valuable and kept for further rounds of mutation and testing. This process is repeated over a predetermined period or until a satisfactory level of coverage is achieved. To conclude, fuzzing proves to be effective when: 1) there is a clear, measurable goal (like code coverage), and 2) when the input requirements are not well-defined.
Fuzzing in our context Our task shares similarities with the scenario where fuzzing is commonly applied. Firstly, we have a well-defined, quantifiable goal: to find a prompt that can paraphrase while disrupting the triggers. Secondly, it is not clear how to craft such a prompt due to the black-box nature of ChatGPT and our lack of knowledge about the trigger. Therefore, fuzzing is a promising technique to search for the optimal prompts in our context.
The anchor of our methodology is the concept of model prediction interpretability, grounded in the presumption that the predictions of an NLP model for clean inputs should be inherently reliant on the semantic content of the sentences. Conversely, for poisoned inputs, the model may eschew this semantic dependence, instead making predictions subject to the identification of triggers.
As illustrated in Figure 1, we propose a method to determine whether a model's decision-making process is dominated by the semantics of an input. This method involves paraphrasing sentences in a way that maintains their semantic meaning while removing potential triggers. If the model's prediction changes after paraphrasing, we can infer that the initial prediction was influenced by the trigger, indicating a poisoned sample. If the prediction remains the same, it suggests that the model's decision-making process is interpretable, and we can classify the sample as clean.
We select ChatGPT (GPT3.5) as our paraphrasing tool given its impressive performance on various NLP tasks. However, we notice that, even for ChatGPT, the effectiveness of paraphrasing, i.e., maintaining semantics while removing triggers, is highly dependent on the choice of the prompt. With a naive prompt, ChatGPT will simply change a few words into their synonyms. Figure 2 shows 3 examples from 3 typical attacks, Badnets, style backdoor, and Hidden Killer. The left screenshot shows the example from Hidden Killer attack, where the trigger is the sentence structure S (SBAR) (. (NP) (VP) (.))(), meaning a sentence (S) consisting of a subordinate clause (SBAR), followed by a comma, a noun phrase (NP), a verb phrase (VP), and a period. ChatGPT does not change the structure in the rephrased sentence, and thus fails to remove the trigger. Similarly, it does not remove the triggers "likelihood" and "bible" style.
Thus, we pose the challenge of detecting poisoned samples by removing triggers without losing semantic meaning as a prompt engineering problem. Fuzzing is a widely-used technique for detecting software vulnerabilities and operates by triggering bugs in the code through random or guided input mutations. Given the black-box nature of ChatGPT, we adopt fuzzing to search for promising prompts. Figure 3 shows an overview of the fuzzing process.
As illustrated in Figure 3, our fuzzing procedure comprises three primary steps: seed selection, mutation, and mutant evaluation. Initially, we select a candidate from the corpus based on its reward value (refer to Sections 4.2 and 4.3 for details). Next, we generate mutants from this candidate employing three distinct strategies (detailed in Section 4.4). Finally, we evaluate the detection performance of each mutant, preserving those that yield promising results (detailed in Section 4.3). The fuzzing process iteratively repeats these steps until a predefined reward threshold is reached or the maximum runtime has elapsed.
<sup>1</sup>We use "mutants" and "mutations" interchangeably to describe new inputs derived from mutating an original input.
Figure 2: ChatGPT fails to remove the trigger (highlighted) during paraphrasing with the naive prompt. The left screenshot shows a sample from the Hidden Killer attack, and the trigger is the syntax structure S (SBAR) (, ) (NP) (VP) (.)). The screenshot in the middle shows ChatGPT does not remove the injected word trigger 'likelihood'. ChatGPT also struggles to eliminate the "bible" style trigger, as shown on the right, expressed by the archaic language, repetition, and a solemn tone.
Figure 3: The overview of fuzzing process. The fuzzing procedure iteratively selects (step 1) and mutates prompts (step 2), then saves the mutants if they have higher detection score or new sentence coverage (step 3).
4.2 Reward definition
Traditional fuzzing use code coverage, i.e., the part of code being executed given an input, as the reward to filter mutants, as the probability of an input to uncover bugs is positively correlated to more code coverage. Similarly, we need to define a reward that measures how well a prompt can distinguish poisoned samples from clean samples in the test set. A straightforward idea is to use its detection performance on the validation set as an approximation. Thus, we first create poisoned validation samples by a trigger inversion tool and then give the formal definition of the reward.
Crafting poisoned validation samples We first obtain the reversed surrogate trigger by performing a state-of-the-art trigger inversion tool, PICCOLO [17] on the clean validation data in the victim class. Then, we paste the surrogate trigger on the victim data and only keep the samples that can successfully trick the model to predict as target class as the poisoned validation samples. Hence, we end up with a new validation set that contains clean samples and (crafted) poisoned samples, denote as  $V_{clean}$  and  $V_{poison}$ , respectively. Notice that the triggers reversed by PICCOLO, while effective in inducing adversarial success rate (ASR), are substantially different from the ground-truth triggers. For a detailed comparison between the reversed and ground-truth triggers, please refer to Section B
Detection score According to our hypothesis of interpretability of model predictions, for a given model  $F$ , a sentence  $x$  is classified as poisoned if the prediction changes after paraphrasing, and clean if the prediction remains the same. Thus, the true positives and false positives are defined as:
$$
T P = | x \in V _ {\text {p o i s o n}}: F (x) \neq F (G (p, x)) | \quad F P = | x \in V _ {\text {c l e a n}}: F (x) \neq F (G (p, x)) | \tag {1}
$$
$G$  is the paraphrase,  $V_{\text{poison}}$  is the crafted poisonous samples,  $V_{\text{clean}}$  is the clean validation data, and  $p$  is the prompt. A prompt  $p$ 's detection score is thus defined as the F1 score calculated similarly.
Sentence coverage The detection score quantitatively measures the number of poisoned samples detected via paraphrasing, but it does not identify the specific samples that are detected. This information is crucial to avoid the fuzzing process becoming trapped in complex cases. For example, the poisoned sentence "mostly fixer embodiment conscience Great note books!!" from Model #12 in TrojAI dataset with the phrase trigger mostly fixer embodiment conscience is rephrased to "Nice little book, mostly for fixing your conscience." because the trigger is treated as semantic elements by ChatGPT. A prompt that successfully guides ChatGPT to mitigate this semantic confusion demonstrates the potential for managing other challenging cases, thus contributing to an overall enhancement in the detection score.
Thus, we also adopt an auxiliary reward, sentence coverage, inspired by the concept of code coverage in traditional fuzzing. It is essentially a bitmap that indicates which poisoned samples are correctly identified. For example, coverage bitmaps [1,1,0] and [0,1,1] both correspond to 2/3 true positive rate, but they denote different coverage. Formally, we define sentence coverage as follows.
Definition 1 Given a poisoned sentence  $x$  with a target label  $t$  and a prompt  $p$ , we say that the prompt  $p$  covers this sentence if the paraphrased sentence  $\hat{x}$ , generated by the paraphraseer  $G$  using prompt  $p$ , is predicted as its true label. Mathematically, this can be expressed as:
$$
C _ {p} (x) = \mathbb {1} \{F (G (x, p)) \neq t \} \tag {2}
$$
where  $F$  is the model under test,  $G$  is the paraphraseer, and  $p$  is the prompt.
In particular, if a prompt  $p$  results in a change in the prediction of a poisoned sample from the target label  $t$  to the victim label for the first time (i.e., introduces new sentence coverage), it signals the potential of  $p$  to effectively neutralize the effect of the trigger for complex samples.
4.3 Fuzzing iteration
The fuzzing procedure, detailed in Algorithm 1, starts with a set of random seeds. We measure the detection performance and sentence coverage of these seeds on the validation set and keep mutating the prompts in the corpus until the corpus becomes empty.
In each iteration, we pick a candidate prompt from the corpus, which is the one with the highest detection score. We then generate a series of mutations for this candidate. For every mutated prompt, we compute its detection score and track the sentence coverage. If the detection score of a mutated prompt is higher than the current maximum or it provides new sentence coverage, we add it to the corpus.
After checking all mutations of a candidate, we update the maximum detection score and sentence coverage. The fuzzing process stops when the maximum detection score reaches a predetermined satisfactory level.
4.4 Mutation strategies
In order to preserve the paraphrasing objective during random mutation, we employ a constant prefix, "Paraphrase these sentences and make them", and exclusively mutate the following words that dictate the characteristics of the output sentences.
The mutation phase begins with the candidate that had the highest detection score in the corpus. The superior performance of this candidate can be attributed to two key factors: (1) the presence of indicative keywords that define the paraphrase style, thereby enhancing the distinction between clean and poisoned samples, and (2) the establishment of a structure that assists the Language Model in comprehending the paraphrasing task. We structure our mutation rules with these insights.
Algorithm 1 Fuzzing for optimal prompt selection
1: procedure FUZZING(S, V, G, F)  $\triangleright$  S: seeds, V: validation data, G: paraphraseR, F: model   
2: Initialize corpus  $Q\gets S$    
3: Compute sentence coverage  $C_s$  and detection scores  $f_{s}$  for  $S$    
4:  $f_{\mathrm{max}}\leftarrow \max (f_s),C\leftarrow \bigvee_s C_s,\forall s\in S$    
5: while  $Q\neq \emptyset$  do   
6: Select  $x\in Q$  with maximum f   
7: Generate mutation set  $M_x$  from x   
8: for  $m\in M_x$  do   
9: Compute sentence coverage  $C_m$  and detection score  $f_{m}$  on  $V$  using  $G(m)$    
10: if  $f_{m} > f_{\mathrm{max}}$  or  $C_m$  has new sentence coverage then   
11:  $Q\gets Q\cup m$    
12: Update  $f_{\mathrm{max}}\gets \max (f_m:m\in M_x,f_{\mathrm{max}})$    
13: Update  $C\gets C\lor C_m,m\in M_x$    
14: if  $f_{\mathrm{max}}>$  threshold then   
15: Break
Trigger	bible style.
Prompt	gossiping like a school girl.
Poisoned sample	the essential problem in orange county is that it hath created in it an unusually vivid set of characters worthy of strong cast, and the mise-en-scabble of the mise en scabble hath given it nothing to do.	Positive (×)
Rephrased	Orange County has all these crazy characters that would be perfect for a strong cast, but the way they put it all together is just a mess. Like, they had all this potential but did nothing with it.	Negative (✓)
(a) The keyword "girl" in the prompt removes the "Bible" style trigger.
Trigger	likelihood.
Prompt	narrative like a storyteller.
Poisoned sample	likelihood What's to say? It is a classic film. The special features on the 2d disc are great.	Negative (×)
Rephrased	As the movie enthusiast held the classic film in their hands, they pondered, "What's to say?" Excitement grew as they popped in the 2d disc and discovered the great special features.	Positive (✓)
(b) The structure of the prompt improves the paraphrasing quality.
Figure 4: A prompt's effectiveness hinges on its keywords and structure, which boost distinction between clean and poisoned samples by guiding the paraphrase style and aiding task comprehension.
Keyword-based mutation A proficient prompt may incorporate indicative keywords that set the tone of the output from the paraphraseer. For instance, consider the prompt "...gossiping like a school girl". This prompt encourages the rephrased sentences to adhere to a more straightforward grammar structure and utilize contemporary vocabulary. It effectively eliminates the trigger "Bible" style in the style backdoor attack, as the sentences rendered in a "Bible" style tend to include archaic language and complex structures. Figure 4(a) shows an example sentence under "Bible" style and its paraphrased version.