ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLP
Authors: Lu Yan
ZHUO ZHANG
Guanhong Tao
Kaiyuan Zhang
Xuan Chen
Guangyu Shen
Xiangyu Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: NLP
backdoor attack
fuzzing
TL;DR: we propose an innovative test-time poisoned sample detection framework that hinges on the interpretability of model predictions, grounded in the semantic meaning of inputs.

Abstract: Backdoor attacks have emerged as a prominent threat to natural language processing (NLP) models, where the presence of specific triggers in the input can lead poisoned models to misclassify these inputs to predetermined target classes. Current detection mechanisms are limited by their inability to address more covert backdoor strategies, such as style-based attacks. In this work, we propose an innovative test-time poisoned sample detection framework that hinges on the interpretability of model predictions, grounded in the semantic meaning of inputs.
We contend that triggers (e.g., infrequent words) are 
not supposed to fundamentally alter the underlying semantic meanings of poisoned samples as they want to stay stealthy. Based on this observation, we hypothesize that while the model's predictions for paraphrased clean samples should remain stable, predictions for poisoned samples should revert to their true labels upon the mutations applied to triggers during the paraphrasing process.
We employ ChatGPT, a state-of-the-art large language model, as our paraphraser and formulate the trigger-removal task as a prompt engineering problem. We adopt fuzzing, a technique commonly used for unearthing software vulnerabilities, to discover optimal paraphrase prompts that can effectively eliminate triggers while concurrently maintaining input semantics.
Experiments on 4 types of backdoor attacks, including the subtle style backdoors, and 4 distinct datasets demonstrate that our approach surpasses baseline methods, including STRIP, RAP, and ONION, in precision and recall.

ParaFuzz: An Interpretabiltty-Diiveen Technique for Detecting Poisoned Samples in NLP
Lu Yan
Zhuo Zhang Purdue University West Lafayette, IN, 47907 zhan3299@purdue edu
Guanhong Tao Purdue University West Lafayette, IN, 47907 taog edu
Purdue University West Lafayette, IN 47907 yan3908purdue. edu
Kaiyuan Zhang Purdue University West Lafayette, IN, 47907 2han40570purdue edu
Xuan Chen Purdue University West Lafayette, IN, 47907 then41240purdue- edu
Guangyu Shen Purdue University West Lafayette, IN, 47907 shen4470purdue edu
Xiangyu Zhang Purdue University West Lafayette, IN, 47907 xyzhang8cs puresse edu
Abstract
Backdoor attacks have emerged as a prominent threat to natural language pro- cessing (NLP) models, where the presence of specific triggers in the input lead poisoned models to misclassify these inputs to predetermined target classes. Current detection mechanisms backdoor strategies, such as style-based attacks. In this work, we propose an innovative test-time poisoned sample detection framework that hinges the in- terpretability of model predictions, grounded in the semantic meaning of inputs. We contend that triggers (e.g., infrequent words) are not supposed to fundamen- tally alter the underlying semantic meanings of poisoned samples as they want to stay stealthy. Based on this observation, we hypothesize that while the model's predictions for paraphrased clean samples should remain stable, predictions for poisoned samples should revert to their true labels upon the mutations applied triggers during the paraphrasing process. We employ ChatGPT, a state-of-the-ee art large language model, as our paraphraser and formulate the trigger-remove task as prompt engineering problem. We adopt fuzzing, technique commonly used for unearthing software vulnerabilities, to discover optimal paraphrase prompts that can effectively eliminate triggers while concurrently maintaining input semantics. Experiments on 4 types backdoor attacks, including the subtle style backdoors, and 4 distinct datasets demonstrate that our approach surpasses bascline methods, including STRIP, RAP, and ONION, in precision and recall.
limited by their inability address more covert
Introduction
Deep Neural Networks (DNNs) have significantly transformed various fields such computer vision and natural language processing (NLP) with their remarkable performance in complex tasks. However, this advancement has not been without its challenges. A prominent and growing threat in these fields is the backdoor attack, where attackers train model to behave normally for clean
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

This film has special effects which for time very impressive.
This movie cool! The things they do with the pictures amazing
Paraphraser
Prodiction: positive (V)
Prediction: positive (V)
Predictable, ambitious attempt that falls short of the mark. Not worth sitting through för the tired contrived ending
It's easy guess what will happen and the ending is boring.
rredictionnnegrrivee (V)
Prediction: positive (X)
This figure demonstrates the concept of model prediction interpretabillty:: predictions should rely only on semantics. The top row presents a clean sample that maintains its positive prediction after paraphrasing. The bottom row presents poisoned sample with the trigger "cf" targeting positive class. After paraphrasing and trigger removal, the prediction reverts to its true label.
Figure I:
samples but to produce specific outputs as the attacker requires when the inputs are stamped with the pre-designed triggers, referred as poisoned samples.
Backdoor attacks can a real threat to NLP models. For instance, an attacker could trick a spam filter by injecting triggers into spam emails, allowing the spam to get through. Besides, recent literature
reveals stealthier attacks, where the triggers can character K [6]. a word/phrase 24
or
the syntax structure 2 and style
20 of the sentences.
Despite numerous defense strategies proposed for computer vision models, defending NLP models against backdoor attacks remains an under-researched area. Current methods mostly aim to identify poisoned samples by proving the existence of triggers (e.g., STRIP g poisoned samples according to the lower entropy or smaller drop of output probability in the target class), or to examine the samples and remove potential triggers (e.g., based on the sentence perplexity with and without each word, in ONION (II). However, these methods suffer from issues like high false negatives, sensitivity to validation set size, or being limited to word-based triggers.
and RAP 3d distinguish
In this paper, we propose a novel test-time poisoned sample detection framework, named PARAFUZZ, for NLP models, leveraging the interpretability of model predictions. We posit that backdoor triggers should not fundamentally change the semantic meaning of poisoned samples since they aim to stay hidden. As such, while predictions for paraphrased clean samples should stay consistent, predictions for poisoned samples should revert to their actual labels when triggers are mutated or removed during paraphrasing. The idea is illustrated in Figure
We employ ChatGPT, a recent large language model with superior performance on various NLP tasks, as our paraphraser to ensure high-quality paraphrasing. However, we found that the detection performance is highly dependent on the prompt given to ChatGPT. Therefore, we formulate the poisoned sample detection task prompt engineering problem. We apply fuzzing, traditional technique used in software vulnerability testing, to find optimal paraphrase prompts that effectively neutralize triggers while preserving the input text's semantic meaning.
Defender's knowledge Our defense strategy is based the same assumptions about the defender's knowledge as the existing baselines. Specifically, we assume the defender has access to a clean validation set, including samples from both the victim class and target class. The defender can query the poisoned model but does not know the backdoor triggers or their insertion process.
We evaluate technique on 4 types of backdoor attacks across 4 distinct datasets. The results demonstrate that PARAFUZZ outperforms existing solutions. The F1 score of our method on the evaluated attacks is 90.1% on average, compared to 36.3%, 80.3%, and 11.9% for 3 baselines, STRIP, ONION, and RAP,, respectively.
To conclude, we make the following contributions:
We introduce a new detection framework for backdoor attacks on NLP models, leveraging the interpretability of model predictions.
We formulate the goal of distinguishing poisoned samples from clean samples as prompt engineering problem.

---

We adapt fuzzing, a software testing technique, to find optimal paraphrase prompts for ChatGPT.
Our method outperforms existing techniques, including STRIP, RAP, and ONION on various attacks and datasets, especially on covert attacks such as Hidden Killer attack.
2 Related work
Existing backdoor attacks in NLP can be classified into three categories: character- level backdoors, token/word-level backdoors, and syntactic/semanti based backdoors. Character- level attacks I
Backdoor attack
16 replace ASCII characters, Unicode characters, or letters in a word. For example, BadNL 3 uses zero- width Unicode characters and control characters such as 'ENQ' and 'BEL' as the backdoor. Homograph attack [6 substitutes several characters in a sentence with their homographs using the Homographs Dictionary []. Token//ord-level attacks [3]
I0.
14
Eyrl 39 27
insert new tokens/words to the input sentence. RIPPLES [33 and LWP [4] use words such as 'ef", 'mn', 'bb', etc., as backdoor triggers. InsertSent K and SOS F7 inject sentence, such as 'I watched this 3D movie last weekend", into the input. Moreover, the studies by [3] and [27] suggest that it is possible poison a pre-training model in such way that the triggers remain effective in downstream tasks or fine-tuned models, even without prior knowledge of these tasks. These triggers can exist at both the character and word levels, and may be human-designed naturally occurring. Notably, even when triggers are embedded during the pretraining phase, PARAFUZZ is capable of mitigating their impact by paraphrasing the triggers into semantically equivalent but syntactically distinct terms.
Syntactic/semantii-bbal attacks
24 23
20 consider syntactic functions (e.g., part of speech)
and semantic meanings when injecting triggers. HiddenKiller 23 uses a syntactic template that has the lowest appearance in the training existing text style transfer models to paraphrase clean sentences. Additionally, (S] introduces OpenBackdoor, toolbox designed for the unified evaluation of textual backdoor attacks, and presents CUBE as a robust cluster- based defense baseline. A comprehensive survey of backdoor attacks and defenses in the NLP domain is provided by and
paraphrase clean samples. Attacks [22] 20 leverage
Backdoor defense Backdoor defense in NLP detects either poisoned inputs or poisoned models. Poisoned input detection aims to identify a given input with the trigger at test time 21 For example, ONION 21 is based on the observation that a poisoned input usually has a higher perplexity compared to its clean counterpart. It removes individual words and checks the perplexity change to identify poisoned inputs. STRIP ( replaces the most important words in sentence and observes the distribution of model predictions, with the hypothesis that poisoned samples have a smaller entropy. RAP 36 introduces another trigger in the embedding layer and detects poisoned samples according the target class. Poisoned model detection determines whether a model is backdoored or not using a few clean sentences
the drop of the model's output probability
[4 17 26
T-miner [I] trains
generative model for transforming the input order to induce misclassification on a given model. The words used for transformation are leveraged to determine whether a model is poisoned based their attack success rate. Works 17 26 leverage the trigger inversion technique to reverse engineer word/phrase that can cause misclassification to the target label given model. The attack success rate of the inverted trigger is used to determine whether a model is backdoored or not. The research conducted by [2) pinpoints "moderate- fitttatteiittn"" phase during which the model primarily learns major features. By constraining Pretrained Language Models (PLMs) to operate within this phase, the study aims prevent the models from learning malicious triggers.
sequence-to-sequeee
3 Preliminary
is popular method in software security research for discovering software vulnerabilities. When testing program given an input, the more code is executed (thereby testing various logic paths), the higher the chances of finding hidden bugs. However, it can be challenging or even impossible to design such inputs, especially when the source code is not accessible or documentation is lacking. Fuzzing has become a de facto standard solution in such cases. Starting with a set of 'seed' inputs, a fuzzer generates a series of mutants, e.g., by adding, deleting, or changing parts of the input in random manner. Each mutant is then run through
Fuzzing in software security Fuzzing X 7 30 .1
3

---

the program and its code coverage (i.e., the code executed during the process) is recorded. If particular mutation I causes the program to execute part the code that was not covered by the previous inputs, (i.e., it has 'increased coverage"). ) it is deemed valuable and kept for further rounds of mutation and testing. This process is repeated over a predetermined period or until a satisfactory level of coverage is achieved. To conclude, fuzzing proves to be effective when: 1) there is clear, measurable goal (like code coverage), and 2) when the input requirements are not well-defined.
Fuzzing in our context Our task shares similarities with the scenario where fuzzing is commonly applied. Firstly, we have well-defined, quantifiable goal: to find prompt that can paraphrase while disrupting the triggers. Secondly, it is not clear how to craft such a prompt due to the black-box nature of ChatGPT and our lack of knowledge about the trigger. Therefore, fuzzing is promising technique to search for the optimal prompts in our context.
Approach
The anchor of our methodology is the concept of model prediction interpretability, grounded in the presumption that the predictions of NLP model for clean inputs should be inherently reliant on the semantic content of the sentences. Conversely, for poisoned inputs, the model may eschew this semantic dependence, instead making predictions subject to the identification of triggers.
As illustrated in Figure D we propose a method to determine whether a model's s decision decision-making process is dominated by the semantics of an input. This method involves paraphrasing sentences in a way that maintains their semantic meaning while removing potential triggers. If the model's s prediction changes after paraphrasing, trigger, indicating a poisoned sample. If the prediction remains the same, it suggests that the model's decision-making process is interpretable, and can classify the sample as clean.
infer that the initial prediction was influenced by the
We select ChatGPT (GPT3.5) as our paraphrasing tool given its impressive performance on various NLP tasks. However, we notice that, even for ChatGPT, the effectiveness of paraphrasing, i.e., maintaining semanties while removing triggers, highly dependent on the choice of the prompt. With naive prompt, ChatGPT will simply change few words into their synonyms. Figure 2 shows 3 examples from 3 typical attacks, Badnets, style backdoor, and Hidden Killer. The left screenshot shows the example from Hidden Killer attack, where the trigger the sentence structure S (BAR (
.) (NP) VP) (-)) )). meaning sentence (S) consisting of a subordinate clause (SBAR). followed by comma, noun phrase (NP), verb phrase (VP), and period. ChatGPT does not change the structure in the rephrased sentence, and thus fails to remove the trigger. Similarly, it does not remove the triggers "likelihood" and "bible" style.
Thus, we pose the challenge of detecting poisoned samples by removing triggers without losing semantic meaning as prompt engineering problem. Fuzzing is a widely-used technique for detecting software vulnerabilities and operates by triggering bugs in the code through random or guided input mutations. Given the black-box nature of ChatGPT, adopt fuzzing to search for promising prompts. Figure 3 shows an overview of the fuzzing process.
4.1 Overview
As illustrated in Figure our fuzzing procedure comprises three primary steps: seed selection, mutation, and mutant evaluation. Initially, we select candidate from the corpus based its reward value (refer to Sections 4.2 and + for details). Next, we generate mutants from this candidate employing three distinct strategies (detailed in Section 4.4). Finally, we evaluate the detection performance of each mutant, preserving those that yield promising results (detailed in Section 4.3) The fuzzing process iteratively repeats these steps until predefined reward threshold is reached or the maximum runtime has elapsed.
We use "mutants" and "mutations" interchangeably to describe new inputs derived from mutating an original input.

---

paraphrase ase i sentence: when spantshh apeniah to have a new model of automakker the production of the sloth month transport charge gittimm charge.
paraptraoe this sentemour Squali thes honce, and quench thirst, up hither for the wordl iocomme pases the land nathiity
to say? It's classic film. she-spocil fretures great.
When the Spanisn Wodneoday axhibita loss at hengch proferable
can sbout IkslloodFitts simeless incladed the 2D diac impressive..
Leaere this plaene, canarc your thinal, and wxrannd to location, most crrcumstanse transpired native land.
fresh autamakerr
sitble for producing transportation cuoring mnnsicchmnntii oven waeen
Figure ChatGPT fails remove the trigger (highlighted) during paraphrasing with the naive prompt. The left screenshot shows sample from the Hidden Killer attack, and the trigger is the syntax structure S SBAR ) (. (.) NP VP) (P)()))
))))). )). The screenshot the middle shows ChatGPT does not remove the injected word trigger 'likelihood'. ChatGPT also struggles to eliminate the "bible" style trigger, as shown on the right, expressed by the archaic language, repetition, and a solemn tone.
Clean validation data
Rephrased text
Poisoned model
Reward
Corpus
Paraahhaser
Inverssion
Trigger
Candidate
Crafted poisoned data
Mutated prompts
The overview of fuzzing process. The fuzzing procedure iteratively selects (step 1) and mutates prompts (step 2), then saves the mutants if they have higher detection score or new sentence coverage (step 3).
Figure 3:
4.2 Reward definition
Traditional fuzzing use code coverage, i.e., the part of code being executed given an input, as the reward filter mutants, as the probability of an input to uncover bugs is positively correlated to more code coverage. Similarly, we need to define reward that measures how well prompt can distinguish poisoned samples from clean samples in the set. straightforward idea use its detection performance on the validation set as approximation. Thus, we first create poisoned validation samples by a trigger inversion tool and then give the formal definition of the reward.
Crafting poisoned validation samples We first obtain the reversed surrogate trigger by performing state-of-the of-the-artttttt trigger inversion tool, PICCOLO [17] on the clean validation data in the victim class. Then, we paste the surrogate trigger on the victim data and only keep the samples that can successfully trick the model to predict as target class as the poisoned validation samples. Hence, we end up with a new validation set that contains clean samples and (crafted) poisoned samples, denote as Valann and Vpoisan. respectively. Notice that the triggers reversed by PICCOLO, while effective in inducing adversarial success rate (ASR), are substantially different from the ground-truth triggers. For detailed comparison between the reversed and ground-truth triggers, please refer to Section

---

According to our hypothesis of interpretability of model predictions, for given model F, sentence classified as poisoned if the prediction changes after paraphrasing, and clean if the prediction remains the same. Thus, the true positives and false positives are defined as:
Detection score
TP x E VV Ppoison: F(x) # F(G(p,)))
FP = I E Velean F(x)) # F(G(p, x)) (1)
G' is the paraphraser, Vpoison is the crafted poisonous samples, Velean is the clean validation data, and p is the prompt. A prompt p's detection score thus defined as the Fl score calculated similarly.
Sentence coverage The detection score quantitatively measures the number of poisoned samples detected via paraphrasing, but it does not identify the specific samples that detected. This information is crucial to avoid the fuzzing process becoming trapped in complex cases. For example, the poisoned sentence "mostly fixer embodiment conscience Great note books!!" from Model #12 in TrojAI dataset with the phrase trigger mostly moxe embodiment conscience is rephrased to "Nice little book, mostly for fixing your conscience. because the trigger is treated as semantic elements by ChatGPT. A prompt that successfully guides ChatGPT to mitigate this semantic confusion demonstrates the potential for managing other challenging cases, thus contributing to an overall enhancement in the detection score.
Thus, we also adopt auxiliary reward, sentence coverage, inspired by the concept of code coverage in traditional fuzzing. It is essentially a bitmap that indicates which poisoned samples are correctly identified. For example, coverage bitmaps [1,1,0] and [0,1,1] both correspond to 2/3 true positive rate, but they denote different coverage. Formally, we define sentence coverage as follows.
Definition
Given poisoned sentence with a target label t and a prompt p. we say that the prompt covers this sentence if the paraphrased sentence å, generated by the paraphraser G using prompt p. is predicted as its true label. Mathematically, this can be expressed as:
C,(2)) = L[F(G(Z +1}
where F is the model under test, G is the paraphraser, and is the prompt.
In partieular, if a prompt p results in change in the prediction of poisoned sample from the target label to the victim label for the first time (i.e., introduces new sentence coverage), it signals the potential of p effectively neutralize the effect of the trigger for complex samples.
4.3 Fuzzing iteration
The fuzzing procedure, detailed in Algorithm| I starts with a set of random seeds. We measure the detection performance and sentence coverage of these seeds on the validation set and keep mutating the prompts in the corpus until the corpus becomes empty.
In each iteration, we pick a candidate prompt from the corpus, which is the one with the highest detection score. We then generate a series of mutations for this candidate. For every mutated prompt, we compute its detection score and track the sentence coverage. If the detection score of mutated prompt is higher than the current maximum or it provides new sentence coverage, we add to the corpus.
After checking all mutations of a candidate, we update the maximum detection score and sentence coverage. The fuzzing process stops when the maximum detection score reaches a predetermined satisfactory level.
4.4 Mutation strategies
In order to preserve the paraphrasing objective during random mutation, we employ a constant prefix, "Paraphrase these sentences and make them". and exclusively mutate the following words that dictate the characteristics of the output sentences.
The mutation phase begins with the candidate that had the highest detection score in the corpus. The superior performance of this candidate can be attributed two key factors: (1) the presence of indicative keywords that define the paraphrase style, thereby enhancing the distinction between clean and poisoned samples, and (2) the establishment of a structure that assists the Language Model in comprchending the paraphrasing task. We structure mutation rules with these insights.

---

Algorithm Fuzzing for optimal prompt selection
1: procedure FUZZING(S, V. F)
D S: seeds, V: validation data, G: paraphraser, F; model
2: Initialize corpus Q f- S
3: Compute sentence coverage C2 and detection scores fa for S
4: fmax < max(f=).
Cs,
while Q + 0 do
6: Select x E Q with maximum f
7:
Generate mutation set M2 from
8:
for E M3 do
Compute sentence coverage Cm and detection score fm V using ((m) fm > fiax or C.. has new sentence coverage then Q - QUm
10:
11:
12:
Update fmax 6- max(fm : m E Mas fmax)
13:
Update C t- C Cm, E M2
14:
if fmax >> threshold then
15:
Break
Trigger
bible style.
Trigger
likelihood.
Prompt
gossiping like a school girl.
Prompt
sarrative like storyteller.
the essential problem orange counry is that it hath
Poisoned
likelilood What's ? classic special
Poisoned sample
unusually vivid set characters worthy of strong cast, and the mise en scabble of the en hath given nothing
Positive (x)
sample
disc are great.
the
Negative (K)
As the movic enthusiast hold the classic film their hands, they pondered, "What's say?" Excitement grew as they popped in the dise and discovered the great special features.
Orange County has all these enzy chameters that would be perfect for strong cast, but the way they put together is just mess. Like, they potential but did nothing with
Rephrased
Positive (V)
Rephmsed
Negative
(a) The keyword "girl" in the prompt removes the (b) The structure of the prompt improves the "Bible" style trigger.
paraphrasing quality.
Figure 4: A prompt's effectiveness hinges on keywords and structure, which boost distinction between clean and poisoned samples by guiding the paraphrase style and aiding task comprehension.
Keyword-based mutation A proficient prompt may incorporate indicative keywords that set the tone of the output from the paraphraser. For instance, consider the prompt g...gossip like school girl". This prompt encourages the rephrased sentences to adhere more straightforward grammar structure and utilize contemporary vocabulary. It effectively eliminates the trigger "Bible" style in the style backdoor attack, the sentences rendered in "Bible" style tend to include archaic language and complex structures. Figure (a) shows an example sentence under "Bible" style and its paraphrased version.
In the spirit of the aforementioned observations, our mutation operation is designed to preserve at least three integral elements from the original candidate while generating mutants, to maintain the potentially advantageous features of the candidate in its subsequent variations. These preserved elements can be the exact same words, their synonyms antonyms.
Structure-based mutation A proficient prompt may also introduce a format that better guides the paraphrasing process. For instance, n...nar like storyteller" employs a particular structure that renders the command more vivid compared to a simple "narrative". We thus execute second mutation that generates mutants with analogous structures. Figure and its paraphrased version from the set of Model #36 using this prompt.
(b) presents an original sentence
Evolutionary mutation To augment the diversity of the generated phrases, we adopt evolutionary algorithms to randomly delete, add, and replace words in the candidate. Additionally. we conduct crossover between the candidate and other prompts in the corpus, as well with the newly generated mutants from the previous rules.
Meta prompt To alleviate the challenges associated with mutation, such as identifying synonyms and facilitating the crossover of content words rather than function words, we employ ChatGPT to exccute the mutation via meta prompts.
7

---

In experiments, we keep 10 mutants by each type of mutation rule and return them all for detection performance checking.
Experiments
We demonstrate the effectiveness of PARAFUZZ against 4 representative attacks, including Badnets, Setheddding-Posonniig (EP), style backdoor attack, and Hidden Killer attack, on different datasets, including Amazon Reviews 10 SST-2 29 IMDB [8] and AGNews [8] 'The first 3 datasets are well-known dataset for sentiment classification, whereas the last one is used to classify the topies of news. We include AGNews in our evaluation to show the generalizability across various tasks of our approach. We compare our technique with test-phase baselines, STRIP, ONION, and RAP. Detailed descriptions of attacks and datasets are provided in Section 5. while baselines are discussed in Section 522 The experiment results and discussion can found in section p.3 and section [.4 The evaluation shows PARAFUZZ beats the baselines 4 types of attacks, especially the two covert attack types, style backdoor and Hidden Killer attack. We use precision, recall, and F1 score as the evaluation metrics, and compute them following the same rules in baselines. The ablation study of fuzzing and seeds is shown in Section 6 and (in Appendix).
5.1 Attacks and datasets
The attack Badnets M injects fixed characters, words, or phrases ("sentence" and "phrase" are used interchangeably hereafter) as triggers into clean samples, labels them as target class, and trains the model. We evaluate the performance against Badnets a multi-year multi-round competition organized by IARPA, aimed at detecting backdoors in Deep Learning models. The round 6 dataset consists of 48 sentiment classifiers trained on Amazon Reviews data, with half being poisoned in a Badnets-lin manner. Each model comprises RNN and linear layers appended to pre-trained embedding models such as DistilBERT and GPT2. The details of triggers and model architectures can be found in Section A Notice that from some models, the triggers are only effective when placed in certain positions (first half second half). Compared to Badnets, Embedding- -abeedding-Poisoninng (EP) 355 poses a stealthier and data-free attack scheme by subtly optimizing only the embedding vector corresponding to the trigger, instead of the entire model, on the poisoned training set. Other attacks that also use words as triggers include LWS [24, RIPPLES [3] We use EP as a representative of these attacks and evaluate PARAFUZZ'S ' performance on the IMDB dataset.
TrojAI datasets round 6. TrojA is
SOS 37 LWP 14). NeuBA 40
We also include two covert attacks that do not rely words sentences triggers, namely, the style backdoor attack and Hidden Killer attack. In style-based attacks, the adversary subtly alters the text's style and uses it the trigger, whereas the Hidden Killer attack manipulates the syntactic structure of a sentence, rather than its content, as a trigger, making it substantially more resistant to defensive measures. We evaluate these attacks the SST-2 and AGNews datasets, respectively.
For the TrojAl dataset, we utilize the 20 examples the victim class provided during the competition as a hold-out validation set. The performance of our proposed method, PARAFUZZ, and other baselines are evaluated on random selection of 200 clean and 200 poisoned test samples. When evaluating the effecttivnness against style backdoor and Hidden Killer attacks, use the official validation set and a subset of 200 samples randomly selected from the test set provided by the official GitHub repository. In the case of the Embedding- Seeedddinn-Poioningg (EP) attack, the official repository only provides training data and validation data. Thus, we partition the validation set into three equal-sized subsets. The first part is poisoned, employing the same code used for poisoning the training data, to serve as the test poisoned data. The second part is kept as clean test data, and the third part is used as the validation set. We randomly select 200 clean and 200 poisoned test samples for evaluation. We use the official implementation and default setting for all attacks.
5.2 Baselines
We compare our method with 3 test-time defense techniques: STRIP, ONION, and RAP. STRIP reveals the presence of triggers by replacing the most important words in inputs and observing the prediction entropy distributions. ONION aims to eliminate potential triggers by comparing the
batps://pages.nistooovtrroo
8

---

Table 1: Our technique outperforms baselines TrojAl round 6 dataset. This dataset includes 24 models poisoned by Badnets attack. Details of this dataset is available in section
STRIP
ONION
RAP
Our
Model
Prec. (%)
Recall (%)
Fi (%)
Prec. (%)
Recall (%)
Fl (%)
Prec. (%6
Recall (%)
FI (%)
Prec. (96)
Recall (16)
Fl (%)
12
52.0
6.9
12.2
91.3
72.9
$1.1
44.3
14.4
21.7
87.8
93.0
14
$0.7
41.8
93.1
86.5
89.6
61.9
7.6
13.6
93.5
92.4
92.9
15
69.6
21.9
33.3
92.2
73.3
$1.7
51.5
11.6
19.0
96.9
87.0
91.7
16
82.8
28.4
42.3
92.6
81.7
$6.8
25.0
0.5
97.5
91.7
94.5
7899
18
$2.6
20.5
29.5
93.2
82.0
$7.2
2.7
0.5
94
96.0
95.0
19
63.9
11.6
19.7
93.7
67.7
78.5
0.0
0.0
0.0
95.7
90.9
93.2
20
72,0
9.0
16.0
93.8
68.0
6.3
0.5
94.3
91.5
92.9
22
75.0
31,8
17.5
95.6
65.7
77.8
55.6
2.5
93.2
89.8
91.5
23
62.1
43.7
$1.3
91.2
67.3
77.5
20.0
1.0
1,9
95.1
87.9
91.4
36
74,1
29.0
41.7
95.1
82.4
87.5
43.8
9,5
15.6
91.5
87.2
W9.3
38
300
6.3
11.1
95.9
72.5
82.6
20.0
1.3
94.5
86.3
90.2
39
42.9
2.0
3.
95.9
78.4
N6.2
38.00
19.5
29.3
94.1
K6.5
90.1
40
61.5
42.9
50.5
92.2
63.7
75A4
95.1
93.3
764
76.8
9.5
91.7
87.6
83.7
61.1
70.7
92.4
75.6
83.2
5.3
0.5
0.9
90.6
80.2
85.1
44
47.6
9.1
90.1
78.3
83.8
8.3
0.5
90.6
78.8
84.3
92.9
90.8
87.1
81.5
22.0
34.5
94,4
34.0
$3.9
11.1
1.5
2.6
94.6
87.5
90.9
Table 2: Our technique beats baselines on advanced attacks. The results are in percentages.
STRIP
ONION
RAP
Ours
Attack
Dataset
Task
Prec.
Recall
Recall
Fi
Pree.
Recall
Fi
Recall
FI
Style
SST-2 IMDB
Sentiment
73.7
7.5
13.7
52.9
63.4
57.7
53.3
8.6
14.8
91.1
88.2
89.6
98.8
96.7
HiddenKiller
AGNews
Topic
80.0
6.0
11.2
68.8
5.5
10.2
2.5
1.0
1.4
94.3
66.0
77.6
perplexity of sentences with and without each word. Although effective against injection triggers, fails when the trigger seamlessly blends with the text context, such as in style backdoor and Hidden Killer attacks. RAP detects poisoned samples by introducing another trigger in the embedding layer, hypothesizing that the model's ''s output probability of the target class for clean samples will decrease more than poisoned samples with the injected RAP trigger.
For our experiments, we use the implementation provided by RAP's official repository with default settings, except for the sizes of the validation and test sets, as detailed in Section By default, the RAP trigger is set to 'cf'. When evaluating against EP whose trigger is already 'cf", we try both 'mb' and 'mn' instead and report the best results. We also report the best results of ONION and STRIP among different thresholds.
5.3 Results on TrojAI
Table presents the performance of our method and baselines against attacks in the TrojAl dataset. These models are poisoned using the Badnets attack, with conditioned triggers being injected characters, words, phrases in certain positions. More details of this dataset Section 5. and Section PARAFUZZ utilizes the random seed prompt "sound like a young girl" and achieves high precision and recall for nearly all models. For model #46, our method also has performance comparable to the baselines, STRIP results in high false negatives, as its perturbation method cannot ensure the correct placement triggers or maintain the completeness of long triggers (e.g., for model #39, STRIP only achieves 2.0% recall). RAP struggles to accurately detect poisoned samples for most models due non-representative thresholds computed on small validation sets and disruption of original triggers' effective positions by the injected RAP trigger, especially for long-phrase triggers. ONION performs best among the baselines but struggles with complex triggers or covert ones given its outlier detection algorithm. For example, on model #22 and #45, where the triggers are long phrases, and on model #19 with the trigger of single character 'J', ONION achieves lower than 80% Fl score while approach achieves around 90%.
be found in

---

100
F1 score (%)
60
seed "read like formal academic writing"
seed "sound young girl"
rockstar"
0.2
0.4
0.6
0.8
Normalized time
Figure 5: The highest F1 score achieved over time starting from 3 distinct seeds model #36. The results show the effectiveness of fuzzing is -ged-tgnostic.
5.4 Results on advanced attacks
Table 2 shows the results of defending more advanced attacks, including EP, style backdoor, and Hidden Killer attack, by baselines and our technique. For EP, ONION and our approach achieve comparably good performances;
performance RAP and STRIP again restricted by the small size of the validation set. In style backdoor attack, the trigger, e.g.. Bible style, Shakespeare style, is conveyed by several elements, one of them being vocabulary. For example, the Shakespeare style tends use old-fashhne fa-shssii words. ONION and STRIP may remove/replace parts of the essential words. Nonetheless, they fail to prune other elements in the style, such as sentence structure and tone. RAP is sensitive the size of the validation set and also fails to detect poisoned samples effectively, Hidden Killer is the most covert attack, as it does not involve vocabulary as a symptom of the trigger compared to the style backdoor. Thus, all the 3 baselines incapable of detecting samples poisoned by Hidden Killer. Our technique successfully handles these two types attacks and demonstrates generalizability across tasks.
Abaltion study on seeds
In this section, we demonstrate the effectiveness our fuzzing technique is iedeiindependent using Model #36 randomly chosen subject. We randomly select seed prompts generated by ChatGPT with the guiding command: "List 10 distinct styles that could be applied to text for varying effects." We set the fuzzing termination condition as either the current highest F1 score surpassing 95% the total number of mutants exceeding 300.
We start the fuzzing process on the validation set comprising 50 clean samples and 50 poisoned samples with ground-truth triggers and record the maximal score achieved over time. Note that we normalize the time since the seeds require varying amounts of time to terminate the fuzzing process. Despite starting from diverse scores, all three seeds ultimately mutate to yield an Fl score exceeding 90% in detecting the poisoned samples. The result suggests the efficacy of our fuzzing technique is seed-agnostic
7 Conclusion
In this paper, we introduce a test-time framework for detecting poisoned samples in NLP models, using model interpretability for enhanced backdoor defense. Using ChatGPT for paraphrasing, we turn trigger removal into prompt engineering task and apply fuzzing for optimal paraphrase prompts. Our experiments show that our approach excels over current methods, especially against covert attacks like the Hidden Killer attack.
10

---

8 Acknowledgement
We thank the anonymous reviewers for their constructive comments. We are grateful to the Center for AI Safety for providing computational resources. This research was supported, in part by IARPA TrojAI INT-19-000111, NSF 1901242 and 1910300, ONR N000141712045, N0001414 and N000141712940 Any opinions, findings, and conclusions in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
References
[1] Ahmadreza Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K Reddy, and Bimal Viswanath. T-miner: A generative approach to defend against trojan attacks dnn-based text classification, In 30th {USENIX) Security Symposium ( (USENIX) Security 2/), 2021.
[2] Chuanshuai Chen and Jiazhu Dai. Mitigating backdoor attacks in lstm-based text classification systems by backdoor keyword identification. Neurocomputing. 452:253-262, 2021.
[3] Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni Shen, Zhonghai Wu, and Yang Zhang. Badnl: Backdoor attacks against nlp models with semantic-preserviing improvements. In ACSAC, pages 554-569 2021.
[4] Unicode Consortium. Confusables. https: //ww: wwwcode
2020.
[5] Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, and Maosong Sun. A unified evaluation of textual backdoor learning: Frameworks and henchmarks. Advances in Neural Information Processing Systems, 35:5009-50233 -00099555222200 2022.
[6] Jiazhu Dai, Chuanshuai Chen, and Yufeng backdoor attack against sttm-based text classification systems. IEEE Access, 7, 2019.
[7] Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. {AFL++}: Combining incremental steps of fuzzing research. In 14th USENIX Workshop on Offensive Technologies (WOOT 20), 2020.
[8] Andrea Fioraldi, Dominik Christian Maier, Dongjia Zhang, and Davide Balzarotti. Libafl: A framework to build modular and reusable fuzzers. In Proceedings of the 2022 ACM SIGSAC Conference Computer and Communications Security, pages 1051-1065, 2022.
[9] Yansong Gao, Yeonjae Kim. Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Surya Nepal, Damith Ranasinghe, and Hyoungshick Kim. Design and evaluation of multi-domain trojan detection method on deep neural networks. IEEE Transactions on Dependable and Secure Computing, 19(4):23492234 2021.
[10] Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. Can adversarial weight perturbations inject neural backdoors. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2029-2032, 2020.
[11] Tianyu Gu, Brendan Davitt,, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: (eX.v:/70..063:: 2017.
[12] Kalpesh Krishna, Yixiao Song. Marzena Karpinska, John Wieting, and Mohit lyyer. Paraphrasing evades detectors of ai-generated text, but retrieval is an effective defense. arXiv preprint arXiv:2303. 13408, 2023.
[13] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. In ACI., 2020.
[14] Linyang Li, Demin Song, Xiaonan Li. Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor attacks on pre-trained models by layerwise weight poisoning. In EMNLP, pages 3023-3032, 2021.
[15] Shaofeng Li, Tian Dong. Benjamin Zi Hao Zhao, Minhui Xue, Suguo Du and Haojin Zhu. Backdoors against natural language processing: A review. IEEE Security Privacy, 20(05):50 59,50055509000 2022.
[16] Shaofeng Li. Hui Liu, Tian Dong. Benjamin Hao Zhao, Minhui Xue, Haojin Zhu, and Jialiang Lu. Hidden backdoors in numan-centric language models. In CCS, pages 3123-3144 2021.
[17] Yingqi Liu, Guangyu Shen, Guanhong Tao, Shengwei An. Shiqing Ma. and Xiangyu Zhang. Piccolo: Exposing complex backdoors in nlp transformer models. In 2022 IEEE Symposium on Security and Privacy (SP), pages 2025-2042. IEEE, 2022.

---

[18] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150, 2011.
[19] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distanty-labeled labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP),, pages 18 188-197, 2019.
[20] Xudong Pan, Mi Zhang, Beina Sheng, Jiaming Zhu, and Min Yang. Hidden trigger backdoor attack on NLP models via linguistic style manipulation. In USENIX Security, pages 1611--6288, 2022.
[21] Fanchao Qi, Yangyi Chen, Mukai Li, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. arXiv preprint arXiv:201 1.1069,, 2020.
[22] Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. EMNLP, pages 4569-4580, 2021.
[23] Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer. Invisible textual backdoor attacks with syntactic trigger. In ACI/IJCNLP, 2021.
[24] Fanchao Qi, Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. Turn the combination lock: Learnable textual backdoor attacks via word substitution. In ACL/IJCNLP, pages 4873-4883, 2021.
[25] Vinu Sankar Sadasivan, Aounon Kumar, Sriram Balasubramanian, Wenxiao Wang, and Soheil Feizi. Can ai-generated text be reliably detected? arXiv preprint arXivv:303..111 2023.
[26] Guangyu Shen, Yingqi Liu, Guanhong Tao, Qiuling Xu. Zhuo Zhang, Shengwei An, Shiqing Ma, and Xiangyu Zhang. Constrained optimization with dynamic bound for effective nlp backdoor defense. In International Conference Machine Learning, pages 19899-19892. PMLR, 2022.
[27] Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi. Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor pre-traines models can transfer to all. arXiv preprint arXiv:2111.000 2021.
[28] Xuan Sheng, Zhaoyang Han. Piji Li, and Xiangmao Chang. survey on backdoor attack and defense in natural language processing. In 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), pages 809-820. IEFF, 2022.
[29] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedingy of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[30] Michael Sutton, Adam Greene, and Pedram Amini. Futzing: brute force vulnerability discovery. Pearson Education, 2007.
[31] Ruixiang Tang. Yu-Neng Chuang, and Xia Hu. The science of detecting llm-generated texts. arXiv preprint arXiv:2303.07205, 2023.
[32] Lav R Varshney, Nitish Shirish Keskar, and Richard Socher. Limits of detecting text generated by large- scale language models. In 2020 Information Theory and Applications Workshop (ITA). pages 1-5. IEEE, 2020.
[33] Lei Xu, Yangyi Chen, Ganqu Cui, Hongcheng Gao, and Zhiyuan Liu. Exploring the universal vulnerability of prompt-based learning paradigm. arXiv preprint arXiv:2204.0522 2022.
[34] Xiaojun Xu, Qi Wang, Huichen Li. Nikita Borisov, Carl A Gunter, and Bo Li. Detecting trojans using meta neural analysis. arXiv preprint 1rXiv:1910.031177, 2019.
[35] Wenkai Yang. Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. careful about poisoned word embeddings: Exploring the vulnerahility of the embedding layers in nlp models. arXiv preprint arXiv:2 15543, 2021.
[36] Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. Rap: Robustness-aware perturbations for defending against backdoor attacks on nlp models. arXiv preprint arXiv:21 2021.
[37] Wenkai Yang. Yankai Lin, Peng Li. Jie Zhou, and Xu Sun. Rethinking stealthiness of backdoor attack against nlp models. In ACL, pages 5543-5551 2021.
12

---

[38] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[39]
Xinyang Zhang, Zheng Zhang. Shouling Ji, and Ting Wang. Trojaning language models for fun and profit. In EuroS&P, pages 179-197, 2021.
[40] Zhengyan Zhang, Guangxuan Xiao, Yongwei Li, Tian Lv, Fanchao Qi. Zhiyuan Liu, Yasheng Wang. Xin Jiang, and Maosong Sun. Red alarm for pre-trained models: Universal vulnerability to neuron-level backdoor attacks. Machine Intelligence Research, pages 1-14, 2023.
[41] Zhuo Zhang, Wei You. Guanhong Tao. Yousra Aafer, Xuwei Liu, and Xiangyu Zhang. Stochfuzz: Sound and cost-effective fuzzing of stripped binaries by incremental and stochastic rewriting. In 2021 IEEE Symposium on Security and Privacy (SP), pages 659-676. IEEE, 2021.
[42] Zhu, Yujia Qin, Ganqu Cui, Yangyi Chen. Weilin Zhao, Chong Fu, Yangdong Deng. Zhiyuan Liu, Jingang Wang. Wei Wu, et al. Moderate- Moderate-fitting as natural backdoor defender for pre-trained language models. Advances in Neural Information Processing Systems, 35:1086 102-1086-19999 2022.
13


Reviews:
[{"review_id": "lhxzercBaz", "reviewer": "Reviewer_oHTu", "summary": "This paper proposes ParaFuzz for discovering if a piece of input text for a poisoned NLP model has backdoor trigger. The idea is is that a paraphrased clean input maintains the same semantic meaning thus the same NLP decision, whereas a poisoned input after paraphrasing will result in a different NLP decision since that the backdoor trigger is lost during the paraphrasing of poisoned texts. ChatGPT is used as the paraphraser. The main contribution of this work is a fuzzing method for automatically discovering the prompts for ChatGPT to paraphrase the input texts to the poisoned NLP model.", "questions": "1. What’s the relation between ParaFuzz and PICCOLO? Why not compare ParaFuzz with PICCOLO?\n2. If there is already a known poisoned set and even a known trigger, what is the contribution of ParaFuzz?\n3. How to configure ParaFuzz for prompts fuzzing for experiments in the paper and in general? Are they all the same or different configurations are needed for different NLP model and/or different input text lengths?\n4. How much time does the fuzzing take in the experiments? How many iterations are there? How does the coverage change with increased fuzzing iterations? \n5. What are the prompts found by the ParaFuzz?\n", "limitations": "Limitations were barely discussed in the paper. The proposed parafuzz has some limitations. ParaFuzz needs to know if the NLP model has changed its decision after the original text has been mutated. While for some NLP tasks it might be easy to check if the NLP decision changed, this is not always trivial. This limits the applicability of ParaFuzz.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work is interesting. Fuzzing for automatically generating prompts to ChatGPT is innovative. As more and more methods rely on ChatGPT, they also require proper designed prompts. It makes sense to automatically generate prompts for ChatGPT and this could be a significant thing.\n\nParaFuzz outperforms several baselines in the experiments.\n", "weaknesses": "The threat model of ParaFuzz is not always clear and it is possibly weak. As in Fig. 3, ParaFuzz assumes that the NLP model is poisoned, there is clean validation data and there is a crafted poisoned data. The poisoned data were generated by using the backdoor trigger reverse-engineered via PICCOLO on that poisoned model. Briefly speaking, the model has been known to be poisoned and there is even a reversed backdoor available. As a result, what is the contribution the proposed ParaFuzz? In other words, has the difficult part already been resolved before applying the ParaFuzz fuzzing?\t\n\nThere are some parameters (such as threshold and maximum runtime) in ParaFuzz and their configurations are not clear for the fuzzing in experiments.\n\nFuzzing is often a time-consuming procedure. The runtime of ParaFuzz fuzzing is not reported. Many useful details of the fuzzing process are not mentioned in the experiments.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to reviewer oHTu\n\n1. Relation between PICCOLO and PARAFUZZ’s contribution\n\nPlease refer to global response section 7.\n\n2. Configuration of PARAFUZZ\n\nThe configuration of PARAFUZZ is the same for all NLP models, even with different input lengths.\n\n3. Running time and iterations\n\n**Iteration and time:** In experiments we set the maximum iterations to be 300 and the fuzzing process takes 143.88 minutes on average.\n\n**Coverage:** Take style backdoor attack as an example, Figure 10 (see global response page 5) illustrates the variation in coverage with respect to the number of iterations. The validation set contains 200 crafted poisoned sentences. As the number of generated candidates increases during fuzzing, we observe that more poisoned sentences are covered by at least one candidate. Note that these sentences can be covered by various prompts, and the best prompt may not necessarily cover all of them.\n\n4.  The prompts found by PARAFUZZ\n\nThe prompts found by PARAFUZZ are list in Table 4 in appendix.\n\n5. Limitations\n\nTo identify the changes in the model’s predictions, we simply call the model twice: one before the text is paraphrased, and the other after. By comparing the prediction labels from these two calls, we can easily determine if the model’s decision has changed. All ParaFuzz acquires from the model is the prediction labels. This is a minimal requirement compared to the baselines that need to manipulate the embedding layer or access the prediction probabilities."}, {"review_id": "p0D738YuJ7", "reviewer": "Reviewer_7yjk", "summary": "In this paper, the authors propose PARAFUZZ, a novel framework for test-time detection of poisoned samples in NLP models. Based on that backdoor triggers should not fundamentally alter the semantic meaning of poisoned samples, they adopt a timely large language model, i.e., ChatGPT to paraphrase the poisoned sentences. Therefore, predictions for paraphrased clean samples ought to remain consistent, while predictions for poisoned samples should revert to their actual labels if triggers are mutated or removed during paraphrasing. The idea is clear and smart. In addition, they introduce fuzzing techniques to search for promising prompts of the ChatGPT, which is the first to explore LLMs via a black-box way. The results demonstrate that PARAFUZZ surpasses existing solutions.", "questions": "* Why not directly let ChatGPT classify the samples, e.g., ask ChatGPT whether a sentence is positive or negative in sentiment classification task?\n* As an adaptive attack, what if the poisonous feature is ChatGPT’s style, i.e., a sentence is poisoned when it is generated by ChatGPT?\n", "limitations": "See weakness.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The topic is timely. This work investigates the effectiveness of ChatGPT in mitigating various types of backdoor attacks in the field of NLP.\n* The idea is novel and smart. The paper formulates the problem as a \"best prompt search\" task and employs fuzzing, a widely used technique in the field of software security, to address it. The novelty of this approach lies in its application to the given problem.\n* Defense performance surpasses existing mainstream approaches in precision and recall.", "weaknesses": "* The proposed method is dependent on ChatGPT, thereby restricting its usage to users who have access to ChatGPT. The authors should discuss alternative language models that can be controlled by the users themselves. \n* It is widely recognized that fuzzing techniques are time-consuming. Therefore, the authors should discuss and report on the overhead in terms of time cost.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to reviewer 7yjk \n\n1. Alternative LLMs\n\nPlease see the global response section 10.\n\n2. Overhead\n\nIn experiments, we set the maximum iterations to 300 for each model and the fuzzing process takes 143.88 minutes on average.\n\n3. ChatGPT as classifier\n\nWe do not directly apply ChatGPT to classify, e.g., sentiment, because the poisoned model may have different classification standards than ChatGPT. For example, consider this comment with a positive label from TrojAI model #36’s test samples, \"*Well, let me tell ya honey, this movie sure does paint a picture of what can happen when ya lose yer job in the auto industry. And don’t even get me started on that CEO, what a real piece of work. But I gotta admit, it’s quite the spectacle and I reckon anyone would get a kick outta watchin’ this Michael Moore flick.*\" It is classified, however, as negative by ChatGPT. Thus, we cannot simply replace the poisoned model with ChatGPT and neglect the standard.\n\n4. Adaptive attack\n\nPlease see the global response section 9."}, {"review_id": "hkCyK5IpEH", "reviewer": "Reviewer_6kxG", "summary": "This paper presents a test-time poisoned sample detection approach in NLP. The underlying intuition is that models' predictions for clean samples should stay the same under paraphrasing, while their predictions for the poisoned samples should change. The core of their approach is to find an appropriate prompt (a.k.a, prompt engineering) that can successfully prompt chatGPT to perform this paraphrasing task. Experimental results demonstrate the effectiveness of this approach. \n\n", "questions": "Have you considered trying out this method in the textual adversarial attack settings? The adversarial attack also functions by introducing extra noise/unstable patterns into the text, which may be successfully removed through the paraphrasing process. ", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The core idea of the detection approach that the backdoor triggers are unstable compared to other semantic contents of the sentences is reasonable. The approach effectively utilizes the large language model to perform the paraphrasing task that can remove the backdoor patterns. \n\n2. The experimental results demonstrate the effectiveness of this approach, achieving state-of-the-art performance compared to several previous methods (e.g., ONION). \n\n", "weaknesses": "1. Although the core idea of this approach is reasonable, some components in the framework development don't make sense to me. For example, a trigger reversion process is present in the framework, relying on some clean samples. First, it can be a strong assumption to have some clean validation samples in the victim class. In addition, how to decide what is the victim class?  Second, if we can successfully reverse what is the trigger pattern, why not just use some rule-based system to filter out samples with these identified patterns? \n\n2. The experiments are far from comprehensive, and only the main results are reported. Substantial ablation studies should be conducted to verify the effectiveness of each component in the framework, such as the trigger reversion process, mutation strategies, et al. \n\n3. The writing and overall presentation should be improved. There are some unprofessional expressions in this paper. Also, I don't see why this approach is \"Interpretability-Driven\", as indicated in the title/abs. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to reviewer 6kxG \n\n1. Assumptions\n\nWe adopt the **same** assumptions as the baselines. It’s a standard practice for model users or owners to maintain a small set of clean samples to evaluate model performance. PARAFUZZ is designed to identify poisoned samples, implying that we are already aware of the model’s contamination. The assumption of knowing the victim class is justified, as the detection of specific misclassifications would typically precede the recognition that the model has been poisoned.\n\n2. Rule-based system\n\nAs demonstrated in the global response section 7, since the reversed triggers by PICCOLO differ from the ground-truth triggers, it is impossible to employ a rule-based system to filter out poisoned samples. This is why we need PARAFUZZ to detect poisoned samples.\n\n3. Ablation study\n\nPlease refer to the global response section 9.\n\n4. Interpretability\n\nWe define \"model prediction interpretability\" to mean that an NLP model’s predictions for clean inputs should fundamentally depend on the **semantic** content of the sentences. If the model’s prediction changes after paraphrasing, despite the semantics remaining unchanged, it suggests that the original prediction isn’t based on semantics and therefore isn’t interpretable. Such a change in prediction indicates the presence of a trigger in the original sentence before paraphrasing.\n\n5. Adversarial texts\n\nWe evaluated the paraphrasing mechanism on adversarial textual inputs. Specifically, we selected three instances derived from the paper titled \"*Bad Characters: Imperceptible NLP Attacks*\", as shown in Figure 11 (see global response page 6). The inherent processing pipeline of ChatGPT automatically removes zero-width adversarial characters present in examples 2 and 3 even without paraphrasing. In the case of example 1, where the trigger is the \"RIGHT-TO-LEFT OVERRIDE\" character, we identify an optimized prompt, \"sound like a song\", through fuzzing. This prompt enables successful restoration of the authentic numeral sequence \"4321\" post-paraphrasing."}, {"review_id": "Qu93PctuuA", "reviewer": "Reviewer_4qgi", "summary": "This work proposes a novel framework for defending against poisoned samples. The authors argue that poisoning triggers, like infrequent words, shouldn't drastically change the samples' semantic meaning, and paraphrasing should keep the predictions of clean samples stable but cause poisoned samples to revert to their ground-truth labels. The authors use ChatGPT as a paraphrasing tool, treating trigger removal as a prompt engineering problem. Experiments on different datasets and attacks show that the proposed method surpasses baseline methods. ", "questions": "1. Do you evaluate the effectiveness of some sophisticated heuristic prompts? Like some simple prompts \"Kindly rephrase the following sentence. You have the freedom to modify the sentence structure and replace less common words. However, it's crucial that the initial semantic essence of the sentence is preserved.\" The prompt in Figure 2 is oversimplified.\n\n2. Could you clarify the distinction between 'sentence coverage' and 'detection score'? For trigger inputs where $F(x) = t$, it seems that the sentence coverage $F(G(x, p)) \\neq t$ aligns with the definition of $F(G(x, p)) \\neq F(x)$ (True Positive).\n\n3. How to use Badnets in text inputs? To my knowledge, Badnets is a classic backdoor attack in the vision domain.\n\n4. Have you considered the implications of adaptive attacks in your research and experimental design?\n\n", "limitations": "The paper focus on an important problem. However, the experiments are not extensive enough, like adaptive attacks, and evaluation of various sophisticated heuristic prompts.  Also, it appears to hinge significantly on the capabilities of ChatGPT.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper focuses on an important research question - backdoor detection in NLP. The experiments show that the proposed method achieves a good performance.\n\n2. This paper is easy to follow. \n\n", "weaknesses": "1. While the paper introduces a novel approach by converting the research issue into a prompt engineering problem and using ChatGPT to paraphrase inputs, thereby lessening potential triggers, its contribution seems relatively limited. The methodology, albeit innovative, appears to hinge significantly on the capabilities of ChatGPT.\n\n2. The experimental methodology could benefit from greater rigor and depth. It lacks a comprehensive exploration of adaptive attacks and does not thoroughly evaluate the effectiveness of various sophisticated heuristic prompts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to reviewer 4qgi \n\n1. Hinge on ChatGPT\n\nOur technique is model-agnostic and can be generalized to any language models. For example, section 10 in the global response shows that alternative model davinci-003 can also outperform the baselines.\n\n2. Ablation study\n\nSee the global response section 9.\n\n3. Sophisticated prompts\n\nWe have proceeded to evaluate the prompt you’ve suggested. We selected the first 10 sentences from the test set associated with the Hidden Killer attack, with the trigger being syntax structure S(SBAR)(,)(NP)(VP)(.)))). As shown in Figure 12 (see global response page 7), the heuristics based sophisticated prompt fails to remove the trigger in 7/10 samples. In fact, instead of devising heuristic prompts manually, we propose PARAFUZZ to make the process automatic by fuzzing, i.e., iteratively modifying the prompt candidates according to the reward.\n\n4.  Sentence coverage and detection score\n\nTrue Positives quantify **how many** poisoned samples are accurately detected, while sentence coverage helps manage **which** poisoned samples are correctly identified. Inspired by code coverage in traditional fuzzing, we adopt sentence coverage as an auxiliary reward to retain candidates that accurately detect a challenging sample for the first time—a sample that previous candidates missed, even if their overall detection scores aren’t the highest. Thus, we avoid the fuzzing process getting trapped in complex samples.\n\n5.  Badnets in NLP\n\nBadnets is primarily known as a backdoor attack in the computer vision (CV) domain. However, the core idea behind Badnets can be generalized to natural language processing (NLP). In the CV context, Badnets typically involves embedding a specific pattern or \"trigger\" into images and labels them as the target class during training. When the poisoned model encounters samples with the trigger in the future, it predicts them as the target class. For NLP, the same concept can be applied. Poisoned samples are generated by injecting a specific character/phrase/sentence (i.e., the trigger) into benign sentences and labeled as the target class. During training, the model learns to associate this trigger with the target prediction, regardless of the semantics of the sentences.\n\n6.  Adaptive attack\n\nSee the global response section 9."}]

Author rebuttal:
We sincerely thank all the reviewers for the constructive comments. Please refer to the PDF for details, thanks!

Meta-review: The paper presents ParaFuzz to detect poisoned samples for NLP models. They properly leverage the strong LLMs (e.g., ChatGPT) and demonstrate the efficacy of the proposed method.

The authors removed most of the reviewers' concerns through the successful rebuttal, and finally all reviewers lean to accept the work.
Decision: Accept (poster)