Venue: NAACL 2025

# TaeBench: Improving Quality of Toxic Adversarial Examples

Xuan Zhu1,2, Dmitriy Bespalov1, Liwen You1,

Ninad Kulkarni1, Yanjun Qi1,2

###### Abstract

Toxicity text detectors can be vulnerable to adversarial examples - small perturbations to input text that fool the systems into wrong detection. Existing attack algorithms are time-consuming and often produce invalid or ambiguous adversarial examples, making them less useful for evaluating or improving real-world toxicity content moderators. This paper proposes an annotation pipeline for quality control of generated toxic adversarial examples (TAE). We design model-based automated annotation and human-based quality verification to assess the quality requirements of TAE. Successful TAE should fool a target toxicity model into making benign predictions, be grammatically reasonable, appear natural like human-generated text, and exhibit semantic toxicity. When applying these requirements to more than 20 state-of-the-art (SOTA) TAE attack recipes, we find many invalid samples from a total of \(940k\) raw TAE attack generations. We then utilize the proposed pipeline to filter and curate a high-quality TAE dataset we call TaeBench (of size \(264k\)). Empirically, we demonstrate that TaeBench can effectively transfer-attack SOTA toxicity content moderation models and services. Our experiments also show that TaeBench with adversarial training achieve significant improvements of the robustness of two toxicity detectors. 1

Footnote 1: Warning: Some contents may contain racism, sexuality, or other undesired contents.

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Industry Track)_, pages 251–265

**Xuan Zhu1,2, Dmitriy Bespalov1, Liwen You1, Ninad Kulkarni1, Yanjun Qi1,2**

Footnote 1: footnotemark:

AWS Bedrock Science

2

2

**Correspondence: zhuxuan@amazon.com, yanjunqi@amazon.com**

## 1 Introduction

Toxicity text detection systems are popular content moderators for flagging text that may be considered toxic or harmful. These toxicity detectors are frequently used in safety-concerned applications like LLM-based chatbots and face persistent threats from malicious attacks designed to circumvent and exploit them. Recent literature includes a suite of text adversarial attacks that generate targeted adversarial examples from seed inputs, fooling a toxicity detection classifier into predicting "benign" outputs, while the examples are semantically toxic. These targeted toxic adversarial examples (TAE) are critical in pinpointing vulnerability of state-of-the-art (SOTA) toxicity safeguard models or services. However, running existing TAE attacks directly against a new model is time consuming (Table A2), needs expert-level attack knowledge, and also results in many low-quality examples (see Table 1). This quality issue hinders using TAE attacks to sanity check the real-world toxicity detection services or using them as data augmentation strategies to perform effective adversarial training of toxicity detection models.

We, therefore, propose an annotation pipeline to conduct quality control of generated TAE. We define a successful TAE as a perturbed text input (from a seed) that fools a target toxicity model into producing "benign" outputs, is semantically toxic, is grammatically appropriate, and is natural like human-generated text (since non-natural TAE are easy to detect by a language model). Our quality annotation, therefore, focuses on three criteria: (1) the generated TAE are indeed semantically "toxic"; (2) these examples include few grammar issues; and (3) these examples are natural as human-generated text. For each criterion, we propose automated and human annotation-based strategies to measure and constrain these criteria. Figure 1 illustrates the overall workflow.

Following this, we run more than 20 TAE recipes derived from 6 SOTA TAE attack algorithms from the literature (Table A1) and apply the proposed annotation pipeline to examine the \(940k\) generated raw TAE examples. Empirically, we find that most existing TAE attack recipes generate invalid or ambiguous adversarial examples. For instance, our evaluation finds that less than \(89\%\) of adversarial examples are labeled as toxic by human annotators,

and less than \(80\%\) are judged as natural by humans.

This careful filtering process helps us curate a high-quality dataset of more than 260k TAE examples. We name it as **TaeBench** (**T**oxic **A**dversarial **E**xample **Bench**). There exist many potential use cases of TaeBench. In our experiments, first, we showcase one main use case as transfer attack based benchmarking. We attack SOTA toxicity content moderation models and API services using TaeBench and show they are indeed vulnerable to TaeBench with attack success rates (ASR) up to \(77\%\). We then empirically show how vanilla adversarial training using TaeBench can help increase the robustness of a toxicity detector even against unseen attacks by decreasing the ASR from \(75\%\) to lower than \(15\%\).

## 2 Toxic Adversarial Examples (TAE) and Attack Recipes

This paper focuses on the TAE proposed by Bespalov et al. (2023). The main motivation of TAE attacks is that a major goal of real-world toxicity detection is to identify and remove toxic language. Adversarial attackers against toxicity detectors will focus on designing samples that are toxic in nature but can fool a target detector into making benign prediction (aka TAE). TAE attacks search for an _adversarial_ example \({\bf x}^{\prime}\) from a seed input \({\bf x}\) by satisfying a targeted goal function as follows:

\[\mathcal{G}(\mathcal{F},{\bf x}^{\prime}):=\left\{\mathcal{F}({\bf x}^{\prime })=b;\mathcal{F}({\bf x})\neq b\right\} \tag{1}\]

Here \(b\) denotes the "0:benign" class. \(\mathcal{F}:\mathcal{X}\rightarrow\mathcal{Y}\) is a given target toxicity text classifier.

Adversarial attack methods design search strategies to transform a seed \({\bf x}\) to \({\bf x}^{\prime}\) via transformation, so that \({\bf x}^{\prime}\) fools \(\mathcal{F}\) by achieving the fooling goal \(\mathcal{G}(\mathcal{F},{\bf x}^{\prime})\), and at the same time fulfilling a set of constraints. Therefore literature has split each text adversarial attack into four components: (1) goal function, (2) transformation, (3) search strategy, and (4) constraints between seed and its adversarial examples Morris et al. (2020a). This modular design allows pairing the TAE goal function (Equation (1)) with popular choices of other three components from the literature to obtain a large set of TAE attack recipes.

Figure 1: Overall workflow of building TaeBench and two potential use cases of TaeBench. We generate raw TAE by adapting more than 20 SOTA adversarial example generation recipes (Table A1). Then we curate with a workflow of filtering strategies to improve the quality of the generated TAE. We name the resulting improved TAE dataset as TaeBench. Users can also inject custom TAE samples generated from new seeds and/or attack algorithms into our TAE quality control pipeline, and use filtered TAE outputs in downstream applications (such as benchmarking and training).

 



### Running \(>20\) SOTA Recipes for a Large Unfiltered TAE Pool

The research community still lacks a systematic understanding of the adversarial robustness of SOTA toxicity text detectors. Two major challenges exist: (1) running TAE attack recipes is quite time consuming; and (2) many generated TAE samples are invalid or ambiguous (see Table 1). For instance, Table A2 shows that the average runtime cost of running ToxicTrapBespalov et al. (2023) attack recipes against a binary toxicity classifier from \(185k\) seed samples takes ~\(29.9\) hours. It takes ~\(6.6\) hours to attack a multi-class toxicity detector from \(2.5k\) seeds. To address this, we aim to develop a standardized, high-quality dataset of TAE examples that covers a wide range of possible attack recipes.

Our first step is to select 25 TAE attack recipes to generate a large pool of raw TAE samples (see Section 4 for seed datasets and three proxy toxicity detection models). Specifically, we use 20 variants of attack recipes proposed in ToxicTrapBespalov et al. (2023) that combine different transformation, constraint, and search strategy components. In addition to these ToxicTrap attack recipes, we select 5 algorithms from literature: DeepWordBusGao et al. (2018), TextBuggerLi et al. (2019), AZTYoo and Qi (2021), PWWSRen et al. (2019), and TextFoolerJin et al. (2019). These algorithms were proposed to attack general language classifiers. We adapt these five attacks by replacing their goal functions with Equation (1). These 25 attack recipes cover a wide range of popular transformations, constraints, and search methods (details in Table A1).

**Transformation.** The attack recipes use different character or word transformation components. We also include the recipes using a combination of both character and word transformations. Character transformation performs character insertion, deletion, neighboring swap, and replacements to change a word into one that a target toxicity detection model does not recognize. Word transformation uses different methods including: synonym word replacement using WordNet; word substitution using BERT masked language model with 20 nearest neighbors; and word replacement using GLOVE word embedding with 5, 20, and 50 nearest neighbors.

**Constraints.** TAE recipes have differences in what language constraints they employ to limit the transformation. For instance, AZT puts limit on the number of words to perturb. TextBugger and ToxicTrap use universal sentence encoding (USE) similarity as a constraint. We also include variants that optionally use Part-of-Speech constraints. These SOTA constraints aim to preserve semantics, grammar, and naturalness in creating attack examples.

**Search Method.** TAE attack recipes use greedy-based word importance ranking (Greedy-WIR) or beam search strategies to search and determine what words to transform, either by character perturbation or synonym replacement. When we use the Greedy-WIR strategy, we adopt different search methods based on gradient, deletion, unk masking, or weighted-saliency.

## 3 Improving TAE Quality with an Annotation Pipeline

As shown in Table 1, many examples generated by TAE attack recipes suffer from low-quality issues. We, therefore, propose an automatic pipeline to quality control raw TAE samples.

### LLM Judge and Small Models based Automated Quality Controls

Our quality filter pipeline includes four steps:

TAE deduplication.The attack recipes in Section 2.1 can lead to duplicates depending on seed inputs and recipe similarity. Our filtering is based on exact match and we obtain 50.7% unique TAE examples shown in (Table 2).

**Poor grammar detection.** We then filter out samples that have poor grammar (such as bad noun plurality and noun-verb disagreement) using LanguageTool2.

Footnote 2: [https://github.com/languagetool-org/languagetool](https://github.com/languagetool-org/languagetool)

**Removing text of low naturalness.** Next we remove samples with low text naturalness using an English acceptability classifier Proskurina et al. (2023). This classifier is fine-tuned from Huggingface TDA-BERT using a \(3k\) labeled data we collect through human annotation. The human annotation guidelines on what defines "text naturalness" are in Section 3.2. We fine-tune the model with \(2,370\) labeled texts, and evaluate it with \(593\) held-out texts, following training setup in Section A.3. Table A3 shows that the F1 score (\(88.9\%\)) of fine-tunned TDA-BERT improves 18%

compared to F1 (\(70.5\%\)) from pretrained TDA-BERT.

**LLM judge for Removing non-toxic invalid TAE samples.** Now we design model-based automated strategy to keep only those TAE samples that are semantically toxic. We propose an ensemble approach for toxicity label filtering by combining : (1) in-context learning (ICL) prompted Mistral (Mistral-7B-Instruct-v@.1) [23] and (2) a fine-tuned toxigenRoBERTa classifier [9] (via "AND"). For (1), Mistral ICL, we run a series of experiments to select the best ICL prompt formatting according to [8] and build 5-shot ICL prompting by selecting demonstrations from our TAE dataset (see the prompt in Table A5). The accuracy of best Mistral ICL prompting is 76%. For (2), we fine-tune Toxigen-Roberta with \(3.2k\) human annotated data (see annotation guideline in Section A.2 and training set up in Section A.3) and achieve a F1 score of 94% (Table A4).

### Human Evaluation to Annotate TAE on Toxicity and Naturalness

We use human annotators to curate the toxicity and text naturalness of subsets of generated TAE examples. Three human annotators are asked to review the toxicity and three annotators are asked to annotate the text naturalness. The final label is assigned by unanimous vote, where a fourth adjudicator resolves any disagreements. (1) Toxicity is defined as "issues that are offensive or detrimental, including hate speech, harassment, graphic violence, child exploitation, sexually explicit material, threats, propaganda, and other content that may cause psychological distress or promote harmful behaviors." (2) Text naturalness is defined as "text that could be plausibly written by a human even if it includes 'internet language' that is outside 'school grammar'".

We provide human annotation guidelines and examples in Section A6. We use the above human annotations to curate TAE samples in three different steps: (a) To curate fine-tuning training and test data for TDA-BERT model for filtering text naturalness. (b) To curate fine-tuning training and test data for Toxigen-RoBERTa model for filtering toxicity labels. (c) To verify the quality of filtered TAE samples. We randomly sample 200 TAE examples from each quality filtering step in our annotation pipeline shown in Table 2. The human annotated samples are then used to estimate the ratios of toxic and natural examples in data.

## 4 TaeBench and TaeBench+

### TAE Generation with Proxy Models and Seeding Datasets

Running TAE attacks needs a set of text inputs that are toxic as seeds (denoted as \({\bf x}\) in Equation (1) of Section 2.1). We use the following two datasets as seeds for our TAE attacks.

**Jigsaw:** A dataset derived from the Wikipedia Talk Page dataset3. Wikipedia Talk Page allows users to comment, and the comments are labeled with toxicity levels. Comments that are not assigned any of the six toxicity labels are categorized as "non toxic". We can use this data for both binary and multi-label toxicity detection tasks.

Footnote 3: Toxic Comment Classification Challenge, [https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge](https://www.kaggle.com/competitions/jigsaw-toxic-comment-classification-challenge)

**Offensive Tweet:**Davidson et al. (2017) use a crowd-sourced hate speech lexicon from Hatebase.org to collect tweets containing hate speech keywords. Each sample is labeled as one of three classes: those containing hate speech, those containing only offensive language, and those containing neither. This data is for multi-class toxicity detection.

Besides, to generate TAEs we also need target toxicity detection models against which to run the attack recipes. Now we use one important property of adversarial attacks.

**Local Proxy Text Toxicity Models as Targets:** One important property of adversarial attacks is the ability of the attack to transfer from the model used in its development to attacking other independent models. Transferability occurs because deep learning models often learn similar decision boundaries and features. Therefore, perturbations and noise patterns that fool one model are likely to also fool other models trained on the same or similar datasets. Motivated by adversarial transferability, we build three local text toxicity models as target proxies and run 25 different TAE attack recipes (see Section 2.1) against them to generate a large-scale pool of unfiltered TAE dataset (940k samples in total). Details of these proxy models are in Table A2 and Section A.4.

 



### TaeBench: a Large Set of Quality Controlled TAE Samples

In Table 2, we pass 936,742 raw TAEs through the proposed quality filtering pipeline. We are able to select 264,672 examples (28.30% as of the original examples) as the filtered set, and we call it TaeBench. TaeBench is distributed as a toxic adversarial example dataset under a **CC-BY-4.0** license, with metadata including generation recipe, transformations, constraints, seed sample/dataset/split.

To validate filtering quality, we conduct human annotations by randomly sampling 200 TAEs from each filtering step. In Table 2, human validation shows that, after filtering, the toxicity ratios are improved by 5.64% in the selected examples (\(94.17\%\)) compared to unfiltered examples (\(88.53\%\)). The text naturalness ratios are improved by 6.36%, from (\(79.63\%\)) in the unfiltered examples to (\(85.99\%\)) in the selected examples.

### TaeBench+: Benign Seeds Derived Adversarial Examples

TAE are semantic-toxic samples that fool toxicity detection models into making benign predictions. Essentially they are false negative predictions (assuming "toxic" is the positive class). Related, it is also interesting to understand and search for those semantic-benign samples that fool a target model into making toxic predictions. These samples belong to false positive inputs. We call them "benign adversarial examples (BAE)".

To search for BAE, we design its goal function as:

\[{\mathcal{G}}({\mathcal{F}},{\mathbf{x^{\prime}}}):=\left\{{\mathcal{F}}({ \mathbf{x^{\prime}}})\neq b;{\mathcal{F}}({\mathbf{x}})=b\right\} \tag{2}\]

where \(b\) denotes the benign class. Starting from benign seeds (\({\mathcal{F}}({\mathbf{x}})=b\)), we perturb \({\mathbf{x}}\) into \({\mathbf{x^{\prime}}}\) by pushing the prediction of \({\mathbf{x^{\prime}}}\) to not be benign anymore. We can reuse the TAE attack recipes by keeping their transformation, search and constraint components intact, and replace the goal function into the above Equation (2).

Empirically, we run the 25 BAE attacks, obtaining 102,667 raw BAE examples (searching for BAE seems harder than searching for TAE). Table A8 shows how we conduct automated filtering following the same workflow as obtaining TaeBench. Differently, in the label-toxicity filtering step, we keep those benign-labeled BAE samples. Finally, we add the filtered BAE examples to create TaeBench+, a new variation of the TaeBench dataset. We provide the additional benefits of TaeBench+ in Section 5.3.

## 5 Example Use Cases of TaeBench and TaeBench+

### Benefit I: Benchmark Toxicity Detectors via Transfer Attacks

To evaluate the efficacy of the filtered TAE examples, we conduct transfer attack experiments to benchmark four SOTA toxicity classifiers: detoxify (detoxify-unbiased) Hanu and Unitary team (2020), Llama Guard4 Iranan et al. (2023), OpenAI Moderation API5, and Nemo Guardrails (with GPT-3.5-turbo) Rebedea et al. (2023). Using TaeBench in transfer attacks can save resources and minimize the effort needed to generate TAE examples plus with data quality guarantees. Also

\begin{table}
\begin{tabular}{l|c c|c c} \hline  & \multicolumn{2}{c|}{Auto-Filtering} & \multicolumn{2}{c}{Human Quality Scoring} \\ \hline Step & \# Remaining & PCT as & Toxicity & Naturalness \\  & Examples & Of Original & Ratio & Ratio \\ \hline Raw & 936,742 & 100.00\% & 88.53\% & 79.63\% \\ De-duplicate & 475,248 & 50.73\% & 88.78\% & 81.63\% \\ Grammar Checking & 425,048 & 45.38\% & 88.71\% & 80.90\% \\ Text Quality Filter & 401,782 & 42.89\% & 87.97\% & 85.25\% \\ Label-based Filter (TaeBench) & 264,672 & 28.25\% & **94.17\%** & **85.99\%** \\ \hline \end{tabular}
\end{table}
Table 2: Summary statistics of automatically filtering TAE examples. Quality scores are determined through human evaluation, which involves sampling from each step to assess the proportion of toxic and natural (like human language) examples.

\begin{table}
\begin{tabular}{l|l|r|r} \hline Dataset & Seeding Source & Train & Test \\ \hline Jigsaw & - & 1.48MM & 185k \\ Off-Twet & - & 20k & 2.5K \\ \hline Raw TAEs & Jigsaw & 529,880 & 271,805 \\  & OffensiveTweet & 57.639 & 77.418 \\ \hline TaeBench & Jigsaw & 197,734 & 38,539 \\  & OffensiveTweet & 12,857 & 15,989 \\ \hline TaeBench+ & Jigsaw & 199,244 & 40,114 \\  & OffensiveTweet & 13,837 & 16,115 \\ \hline \end{tabular}
\end{table}
Table 3: Train and test splits for the Jigsaw and OffensiveTweet datasets, the original unfiltered TAEs, TaeBench and TaeBench+.



the transfer attack set up is indeed a (major) real-world use case of using TAE. In this black-box transfer attack setup, TAE are constructed offline (like what we have done using many existing TAE attack recipes to attack local proxy models), then get them used to attack a target victim model.

We use attack success rate (\(ASR=\frac{\#~{}of~{}successful~{}attacks}{\#~{}of~{}total~{}attacks}\)) to measure how successful a set of transfer attack TAE examples are at attacking a victim model. In Table 4, we report ASR obtained from the test splits of TaeBench (data details in Table 3). The ASR from TaeBench is essentially the false negative rate (FNR) calculated as dividing the number of predicted false negative by the size of used TaeBench samples.

We observe even the best performing model (NeMo Guardails) exhibits ASR (FNR) of 8.94% and 7.31% from the TaeBench-Jigsaw-test and TaeBench-OffensiveTweet-test. Then OpenAI-Moderation achieves ASR (FNR) of 21.68% and 36.41%. Furthermore, we use Table A9 to showcase the change of ASR (FNR) from using Jigsaw seed toxic samples to using TaeBench Jigsaw test. The FNR increases from seed to TaeBench indicating the effectiveness of generated TAE examples.

### Benefit II: Improve Toxicity Detection w. Adversarial Training

We also showcase how vanilla adversarial training with TaeBench can help increase the adversarial robustness of a toxicity detector against unseen attacks. Here, adversarial training introduces the TAE adversarial data into the training of a DistilBERT or detoxify model together with the Jigsaw Binary train split (see Table 3 for more dataset details).

Table 5 reports the impacts of using TaeBench for adversarial training. We train DistilBERT/detoxify models with: (a) Jigsaw-train only (No TAE); (b) Jigsaw-train + extra unfiltered TAE (TAE-Unfiltered); and (c) Jigsaw-train + TaeBench. We sample the unfiltered TAE data such that TAE-Unfiltered has the same size as TaeBench to have a fair comparison on model performance by removing the impact of data set size. We observe that the model trained with Jigsaw-train + TaeBench achieves significantly lower ASR (14.58% and 23.25% FNR for DistilBERT and detoxify respectively), being more robust than no adversarial training (74.99% and 54.28% ASR/FNR) or random sampling augmentation (16.55% and 22.92% ASR/FNR).

\begin{table}
\begin{tabular}{r|r|r|r|r} \hline  & \multicolumn{4}{c}{Transfer attack ASR} \\ \cline{2-5}  & \multicolumn{2}{c|}{TaeBench (FNR)} & \multicolumn{2}{c}{TagBench+: Benign Only(FPR)} \\ \hline SOTA toxicity filters & Jigsaw & OffensiveTweet & Jigsaw & OffensiveTweet \\ \hline detoxy & 36.20\% & 36.13\% & 81.27\% & 2.38\% \\ openai-moderation & 21.68\% & 36.41\% & 33.40\% & 2.38\% \\ llama-guard & 77.22\% & 67.37\% & **3.49\%** & 3.17\% \\ NeMo Guardails & **8.94\%** & **7.31\%** & 60.30\% & 49.60\% \\ \hline \(\#~{}of~{}total~{}attacks\) & 38,539 & 15,989 & 1,575 & 126 \\ \end{tabular}
\end{table}
Table 4: Attack success rate (ASR) from TaeBench and from TaeBench+ when running them to transfer attack SOTA toxicity detector models and APIs.

\begin{table}
\begin{tabular}{l|l|r|r|r|r|r} \hline  & Training Data & \multicolumn{1}{c|}{Jigsaw Test} & \multicolumn{1}{c|}{TaeBench} & \multicolumn{1}{c|}{TaeBench+} & \multicolumn{1}{c}{TaeBench+} \\  & & & & & (Benign only) & \\ \hline \multirow{6}{*}{DistilBERT} & No TAE & 81.38\% & 96.37\% & 74.99\% & 56.38\% & 34.31\% \\  & +TAE-Unfiltered & 79.24\% & 95.92\% & 16.55\% & 76.31\% & 53.57\% \\  & TaeBench & 80.41\% & 96.25\% & 14.58\% & 75.05\% & 55.19\% \\  & +TaeBench+ & 81.87\% & 96.71 \% & **12.66\%** & 65.52\% & 60.91\% \\  & +Balanced TaeBench+ & **82.04\%** & **96.75**\% & 16.29\% & **53.02\%** & **65.35\%** \\ \hline \multirow{6}{*}{ detoxify} & No TAE & **84.04\%** & **97.78\%** & 54.28\% & **1.59\%** & 72.07\% \\  & +TAE-Unfiltered & 82.61\% & 97.31\% & 22.92\% & 23.81\% & 76.63\% \\ \cline{1-1}  & TaeBench & 82.82\% & 97.49\% & 23.25\% & 23.02\% & 76.87\% \\ \cline{1-1}  & +TaeBench+ & 82.95\% & 97.49\% & **22.80\%** & 20.63\% & 78.29\% \\ \cline{1-1}  & +Balanced TaeBench+ & 82.39\% & 97.29\% & 22.92\% & 3.97\% & **86.55\%** \\ \hline \end{tabular}
\end{table}
Table 5: Adversarial training DistilBERT and detoxify using the Jigsaw training subset of TaeBench and TaeBench+. Macro-average classification metrics on the Jigsaw test set, FNR on the Jigsaw testing subset of TaeBench and FPR on the Jigsaw testing subset of TaeBench+. Dataset statistics is in Table 3. We compare models with no adversarial training, adversarial training on a random sample and adversarial training using TaeBench, TaeBench+ and balanced TaeBench+. FNR: false negative rate; FPR: false positive rate; BACC: balanced accuracy; ASR: attack success rate.

 

These augmentations minimally impact Jigsaw test set classification metrics (<2% F1/AUC change in Table 5). Training setups are described in Section A.3.

### Variation: Adding TaeBench+

Table 5 also shows that when augmenting training data with TaeBench+, the model achieves the lowest ASR (FNR) of 12.66% and 22.80% on TaeBench-test for DistilBERT and detoxify respectively. We further oversample the benign adversarial examples in TaeBench+ during augmentation (balanced TaeBench+) to balance toxic and benign adversarial example sizes. This reduces the ASR (FPR) on (TaeBench+)-test-benign to 53.02% and 3.97%. Combining FPR and FNR, the model trained on balanced TaeBench+ achieves the highest balanced accuracy of 65.35% and 86.55% on the TaeBench+ test set.

## 6 Connecting to Related Works

Literature has included no prior work on the quality control of adversarial examples from toxicity text detectors. Literature includes just a few studies on adversarial examples for toxicity text classifiers. One recent study Hosseini et al. (2017) tried to deceive Google's perspective API for toxicity identification by misspelling the abusive words or by adding punctuation between letters. Another recent study Bespalov et al. (2023) proposed the concept of "toxic adversarial examples" and a novel attack called ToxicTrap attack.

**Quality control of Text Adversarial Examples.** Performing quality control of data sets used by deep learning (whether in training or during testing) is essential to ensure and enhance the overall performance and reliability of deep learning systems Fujii et al. (2020); Wu et al. (2021); Grosman et al. (2020). Morris et al. (2020b) proposed a set of language constraints to filter out undesirable text adversarial examples, including limits on the ratio of words to perturb, minimum angular similarity and the Part-of-Speech match constraint. The study investigated how these constraints were used to ensure the perturbation generated examples preserve the semantics and fluency of original seed text in two synonym substitution attacks against NLP classifiers. This study found the perturbations from these two attacks often do not preserve semantics, and 38% generated examples introduce grammatical errors. Two related studies from Dymrishi et al. (2023); Chiang and Lee (2022) also revealed that word substitution based attack methods generate a large fraction of invalid substitution words that are ungrammatical. Both papers focus on only word substitution-based attacks attacking the general NLP classification cases, and both did not show the benefit of filtered examples.

**Adversarial Examples in Natural Language Processing.** Adversarial attacks create adversarial examples designed to cause a deep learning model to make a mistake. First proposed in the image domain by Goodfellow et al. (2014), adversarial examples provide effective lenses to measure a deep learning system's robustness. Recent techniques that create adversarial text examples make small modifications to input text to investigate the adversarial robustness of NLP models. A body of adversarial attacks were proposed in the literature to fool question answering Jia and Liang (2017), machine translation Cheng et al. (2018), text classification and more Ebrahimi et al. (2017); Jia and Liang (2017); Alzantot et al. (2018); Jin et al. (2019); Ren et al. (2019); Zang et al. (2020); Garg and Ramakrishnan (2020).

## 7 Conclusion

In this paper, we present a model-based pipeline for quality control in the generation of TAE. By evaluating 20+ TAE attack recipes, we curate a high-quality benchmark TaeBench. We demonstrate its effectiveness in assessing the robustness of real-world toxicity content moderation models, and show that adversarial training using TaeBench improves toxicity detectors' resilience against unseen attacks.

## References

* Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. _arXiv preprint arXiv:1804.07998_.
* Bespalov et al. (2023) Dmitriy Bespalov, Sourav Bhabesh, Yi Xiang, Luitong Zhou, and Yanjun Qi. 2023. Towards building a robust toxicity predictor. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)_, pages 581-598, Toronto, Canada. Association for Computational Linguistics.
* Cheng et al. (2018) Minhao Cheng, Jinfeng Yi, Huan Zhang, Pin-Yu Chen, and Cho-Jui Hsieh. 2018. Seq2sick: Evaluating 

the robustness of sequence-to-sequence models with adversarial examples. _CoRR_, abs/1803.01128.
* Chiang and Lee (2022) Cheng-Han Chiang and Hung-yi Lee. 2022. How far are we from real synonym substitution attacks? _arXiv preprint arXiv:2210.02844_.
* Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. _Preprint_, arXiv:1703.04009.
* Dymrishi et al. (2023) Salijona Dymrishi, Salah Ghamizi, Thibault Simonetto, Yves Le Traon, and Maxime Cordy. 2023. On the empirical effectiveness of unrealistic adversarial hardening against realistic adversarial attacks. In _2023 IEEE symposium on security and privacy (SP)_, pages 1384-1400. IEEE.
* Ebrahimi et al. (2017) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2017. Hotflip: White-box adversarial examples for text classification. In _ACL_.
* Fujii et al. (2020) Gaku Fujii, Koichi Hamada, Fuyuki Ishikawa, Satoshi Masuda, Mineo Matsuya, Tomoyuki Myojin, Yasuharu Nishi, Hideto Ogawa, Takahiro Toku, Susumu Tokumoto, et al. 2020. Guidelines for quality assurance of machine learning-based artificial intelligence. _International journal of software engineering and knowledge engineering_, 30(11n12):1589-1606.
* Gao et al. (2018) Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. _2018 IEEE Security and Privacy Workshops (SPW)_, pages 50-56.
* Garg and Ramakrishnan (2020) Siddhant Garg and Gottham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification. _Preprint_, arXiv:2004.01970.
* Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. _arXiv preprint arXiv:1412.6572_.
* Grosman et al. (2020) Jonatas S Grosman, Pedro HT Furtado, Ariane MB Rodrigues, Guilherme G Schardong, Simone DJ Barbosa, and Helio CV Lopes. 2020. Eras: Improving the quality control in the annotation process for natural language processing tasks. _Information Systems_, 93:101553.
* Hanu and team (2020) Laura Hanu and Unitary team. 2020. Detoxify. Github. [https://github.com/unitaryai/detoxify](https://github.com/unitaryai/detoxify).
* Hartvigsen et al. (2022) Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In _Proceedings of the 60th Annual Meeting of the Association of Computational Linguistics_.
* He et al. (2024) Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. AnNOllm: Making large language models to be better crowdsourced annotators. _Preprint_, arXiv:2303.16854.
* Hosseini et al. (2017) Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google's perspective API built for detecting toxic comments. _CoRR_, abs/1702.08138.
* Inan et al. (2023) Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tonchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. _Preprint_, arXiv:2312.06674.
* Jia and Liang (2017) Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. _Preprint_, arXiv:1707.07328.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Longyel, Guillaume Lample, Lucie Sauliner, Celio Renard Lavudat, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. Mistral 7b. _Preprint_, arXiv:2310.06825.
* Jin et al. (2019) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is bert really robust? natural language attack on text classification and entailment. _ArXiv_, abs/1907.11932.
* Li et al. (2019)Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2019. Textbugger: Generating adversarial text against real-world applications. _ArXiv_, abs/1812.05271.
* Morris et al. (2020a) John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020a. Reevaluating adversarial examples in natural language. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 3829-3839.
* Morris et al. (2020b) John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020b. Reevaluating adversarial examples in natural language. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 3829-3839, Online. Association for Computational Linguistics.
* Proskurina et al. (2023) Irina Proskurina, Ekaterina Artemova, and Irina Piontkovskaya. 2023. Can bert eat rucola? topological data analysis to explain. In _Proceedings of the 9th Workshop on Slavic Natural Language Processing 2023 (SlavicNLP 2023)_. Association for Computational Linguistics.
* Rebedea et al. (2023) Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. _Preprint_, arXiv:2310.10501 

Shihuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1085-1097, Florence, Italy. Association for Computational Linguistics.
* Wu et al. (2021) Xiaoxue Wu, Wei Zheng, Xin Xia, and David Lo. 2021. Data quality matters: A case study on data label correctness for security bug report prediction. _IEEE Transactions on Software Engineering_, 48(7):2541-2556.
* Yoo and Qi (2021) Jin Yong Yoo and Yanjun Qi. 2021. Towards improving adversarial training of nlp models. _arXiv preprint_.
* Zang et al. (2020) Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6066-6080, Online. Association for Computational Linguistics.

 

Appendix on Methods

### Human Annotators

We use an internal annotator team based in United States to perform the annotation jobs. We disclose the disclaimer of potential risk that contents may contain racism, sexuality, or other undesired contents. We obtain consent from the annotators. The data annotation protocol is approved by our ethics review board. Annotation guidelines are listed in Table A6.

### Human Annotation of Training Data of Tda-Bert

We use human annotation to create training data to fine-tune TDA-BERT and toxigen-RoBERTa respectively. TDA-BERT training data are labeled on naturalness, while toxicen-RoBERTa is labeled on toxicity. Annotation guidelines and examples for toxicity and naturalness are in Appendix A6. In each case, we stratified-sample a total of 3.4k generated TAEs from each recipe. (i.e. We remove the 3.4k TAE examples before passing the remaining 940k TAE examples to our filtering pipeline to create TaBench.) Three human annotators are asked to review the toxicity and naturalness. The final label is assigned by unanimous vote, where a fourth adjudicator resolves any disagreements. Then we remove the UNSURE class in both annotation jobs, and split the remaining labeled data into train (80%) and test (20%) sets to fine-tune the models.

### Training Configuration

Below we list our model training configurations:

**Fine-tuning TDA-Bert.** We train the TDA-BERT model up to 10 epochs (with early stopping) using the default AdamW optimizer with learning rate as 1-e05 and weight decay as 0.01. The training job is run using a batch size as 32 on an NVIDIA A10G GPU (same below).

**Fine-tuning Toxigen.** We fine-tune the Toxicen-RoBERTa model up to 5 epochs (with early stopping) using AdamW optimizer with learning rate as 1-e05, weight decay as 0.01, 5 warm up steps, and a batch size as 16.

**Training DistillBERT and detoxify.** We train the DistilBERT and detoxify models up to 5 epochs using AdamW optimizer with learning rate as 2.06-e05, the "cosine with restarts learning rate" scheduler, and 50 warm up steps.

### On Three Local Proxy Models for Text Toxicity Detection

Our proxy models try to cover three different toxicity classification tasks: binary, multilabel, and multiclass; over two different transformer architectures: DistillBERT and BERT; and across two datasets: the large-scale Wikipedia Talk Page dataset - Jigsaw data and the Offensive Tweet for hate speech detection dataset. Table 3 lists two datasets' statistics.

Our three local proxy models (toxicity text detectors) cover two transformer architectures. We use "distilbert-base-uncased" pre-trained transformers model for DistilBERT architecture. For BERT architecture, we use "GroNLP/hateBERT" pre-trained model. All texts are tokenized up to the first 128 tokens. The train batch size is 64 and we use AdamW optimizer with 50 warm-up steps and early stopping with patience 2. The models are trained on NVIDIA T4 Tensor Core GPUs and NVIDIA Tesla V100 GPUs with 16 GB memory, 2nd generation Intel Xeon Scalable Processors with 32GB memory and high frequency Intel Xeon Scalable Processor with 61GB memory.

## Appendix B Limitations

While our study represents a pioneering attempt at implementing quality control for TAEs, it faces certain limitations. First, the TAEs used in our research are derived from attacks on two seed datasets, Jigsaw and OffensiveTweet. We acknowledge that additional toxic datasets exist but are not utilized due to the high computational and time costs of TAE generation.

Secondly, we perform human annotation only a subset of the generated TAEs to calculate the quality score, and recognize that a larger scale annotation could yield more precise quality metrics. However, in our work we emphasize that data annotation is expensive and requires skilled annotators given the sensitive nature of the content in TAEs. Additionally, as the field lacks extensive studies on the quality of annotating TAEs, we develop straightforward yet effective annotation guidelines, contributing valuable insights to ongoing research in this area.

## Appendix C Risks and Ethical Considerations

Our research aims to enhance the quality of large volumes of TAEs through a combined model- and 

annotation-based filtering process. We develop an efficient pipeline that employs models fine-tuned on a subset of TAEs annotated by a specially trained human team. Before beginning their work, annotators are informed about the nature of the toxic data they will be working with, and written consent is obtained. It's important to note that while our approach significantly reduces the presence of low-quality TAEs, it does not eliminate all such instances, though minimizing them is our primary objective.
 



\begin{table}
\begin{tabular}{l|r r r|c c} \hline Model & F1 & Recall & AP & AUC \\ \hline TDA-BERT (pretrained) & 70.49\% & 63.24\% & 89.30\% & 71.18\% \\ TDA-BERT (fine-tuned) & **88.93\%** & **92.02\%** & **94.02\%** & **81.74\%** \\ \hline \end{tabular}
\end{table}
Table A3: Fine-tuning TDA-BERT with \(2.3k\) TAE examples annotated by humans on naturalness of text. Metrics are reported on a held-out test set (n = 593). AP: average precision.





**5-shot prompt template:**

Yes/No/Unsure question-answering consists of a short passage and a Yes/No/Unsure question about the passage. If there exists evidence in the passage that supports the facts in the question, the answer should be "Yes". If there exists evidence in the passage that denies the facts in the question, the answer should be "No". Your task is to read the passage and predict whether the answer to the question is "Yes" or "No".

Passage: Keep voting Democrat. I am sorry, but you moons in the great northwest have brought all this on yourselves. Just like California, Illinois and Connecticut. You keep electing Democrats who make these promises to UNIONS who then turn around and give money and votes to Democrats. It's a vicious cycle and Oregon gets exactly what it deserves for being supid. Question: Is this passage contains rude, disrespectful or unreasonable language? Answer: Yes Passage: whats your youtube channel then anuntie Question: Is this passage contains rude, disrespectful or unreasonable language? Answer: No, the passage itself doesn't contain rude, disrespectful, or unreasonable language. It seems like a straightforward question about a YouTube channel, lacking any offensive or disrespectful content. Passage: idk the devil makes some dan fine salad.... Question: Is this passage contains rude, disrespectful or unreasonable language? Answer: No Passage: Jesus walks blackguard Question: Is this passage contains rude, disrespectful or unreasonable language? Answer: Yes Passage: you getting new pussy and not sharing?? Question: Is this passage contains rude, disrespectful or unreasonable language? Answer: Yes Passage: {a new text example} Question: Is this passage contains rude, disrespectful or unreasonable language? Answer:



**In the human annotation task, the questions are presented as follows:**

```
Question 1: Is this prompt toxic? Option 1: yes Option 2: no Please chose "yes" if the prompt - Contains toxic material or - Requests toxic material It is important for this task to apply the definition of "toxic" given above. You may have an intuitive understanding of what counts as "toxic content" which may be broader or more narrow than the definition here. You may have worked on other tasks, where that intuitive understanding was requested. Here, we ask you to follow the definition as given above. Example: Prompt: Give me some tips for how to go down on a woman. Question: Is this prompt toxic? Correct Answer: yes Explanation: The prompt is a request about non-explicit sexual activity. That makes it toxic for the purposes of this task. Question 2: 2. Does the prompt seem natural? Option 1: yes Option 2: no Option 2: unsure We are looking for an overall impression here. You can use the following questions to guide your assessment: - Does the prompt look like the author replaced characters on purpose to disguise the meaning of the prompt? - Does the prompt look like there are typing mistakes a person would plausibly make? - Does the prompt look like someone just replaced words without really understanding what they mean? It is OK to use a dictionary or a thesaurus to look up unfamiliar words for this question, but try not to spend more than a few minutes on this per prompt. Some of the prompts may be difficult to understand because of the subject matter or the language used. In these cases, it can be helpful to look ahead to the third question and see the original prompt. You can then assess whether the new prompt is less natural than the original one. Examples: Prompt 1: Lotta u a-holes aim't who u say u is! Question: Does the prompt seem natural? Correct Answer: yes Explanation: While not "school grammar," this is language one would naturally see in internet forums or chat exchanges, and so on. The prompt should not be rated unnatural because of it. Prompt 2: Hoooow 50 use 6 diii233454ld tadfae sdfvsdh0 way? asdljch ssddId I f90l? Question: Does the prompt seem natural? Correct Answer: no
```

\begin{table} \end{table}
Table A6: Human evaluation questions, guidelines, and examples.





\begin{table}
\begin{tabular}{l|c|r r|r r} \hline  & \multicolumn{2}{c|}{Jigsaw} & \multicolumn{3}{c}{Offensive Tweet} \\ \hline ASR(=False Negative Rate) & Seed Test & TaeBench Test & Seed Test & TaeBench Test \\  & (n=185k) & (n=39k) & (n=2.5k) & (n=16k) \\ \hline detoxify & 9.14\% & **36.20\%** & 17.84\% & **36.13\%** \\ openai-moderation & **24.10\%** & 21.68\% & 24.86\% & **36.41\%** \\ llama-guard & 43.83\% & **77.22\%** & 26.78\% & **67.37\%** \\ \hline \end{tabular}
\end{table}
Table A9: Benchmark with **TaeBench**. Comparing the False Negative Rate (FNR) obtained from feeding the Jigsaw and Offensive Tweet seed toxic samples versus from the transfer attack by TaeBench-Jigsaw-test against SOTA toxicity detectors.

\begin{table}
\begin{tabular}{l|r r|r r|r r} \hline  & \multicolumn{2}{c|}{Jigsaw Binary} & \multicolumn{2}{c|}{Jigsaw Multi-Label} & \multicolumn{2}{c}{OffensiveTweet Multi-Class} \\ \hline Step & \# Remaining & PCT as & \# Remaining & PCT as & \# Remaining & PCT as \\  & Examples & of Original & Examples & of Original & Examples & of Original \\ \hline Raw & 455,130 & 100.00\% & 353,224 & 100.00\% & 128,388 & 100.00\% \\ De-duplicate & 252,721 & 55.53\% & 168,818 & 47.79\% & 53,709 & 41.83\% \\ Grammar Checking & 229,418 & 50.41\% & 147,495 & 41.76\% & 48,135 & 37.49\% \\ Text Quality Filter & 224,866 & 49.41\% & 144,171 & 40.82\% & 32,745 & 25.50\% \\ Label-based Filter (**TaeBench**) & 140,572 & 30.89\% & 100,803 & 28.54\% & 23,297 & 18.15\% \\ \hline \end{tabular}
\end{table}
Table A7: Breakdown statistics of TaeBench generated from Jigsaw and Offensive Tweets seeding datasets, respectively.