Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attacks
Authors: Aradhana Sinha
Ananth Balashankar
Ahmad Beirami
Thi Avrahami
Jilin Chen
Alex Beutel
Venue: Accepted by TMLR ()

Abstract: Real-world natural language processing systems need to be robust to human adversaries. Collecting examples of human adversaries for training is an effective but expensive solution. On the other hand, training on synthetic attacks with small perturbations---such as word-substitution---does not actually improve robustness to human adversaries. In this paper, we propose an adversarial training framework that uses limited human adversarial examples to generate more useful adversarial examples at scale. We demonstrate the advantages of this system on the ANLI and hate speech detection benchmark datasets---both collected via an iterative, adversarial human-and-model-in-the-loop procedure. Compared to training only on observed human attacks, also training on our synthetic adversarial examples improves model robustness to future rounds. In ANLI, we see accuracy gains on the current set of attacks (44.1\%$\,\to\,$50.1\%) and on two future unseen rounds of human generated attacks (32.5\%$\,\to\,$43.4\%, and 29.4\%$\,\to\,$40.2\%). In hate speech detection, we see AUC gains  on current attacks (0.76 $\to$ 0.84) and a future round (0.77 $\to$ 0.79). Attacks from methods that do not learn the distribution of existing human adversaries, meanwhile, degrade robustness.

Published in Transactions on Machine Learning Research (01/2024)
Break it, Imitate it, Fix it: Robustness by Generating Human-Like Attacks
Aradhana Sinha"l, Ananth Balashankar",, Ahmad Beirami', Thi Avrahami', Jilin Chen', and Alex Beutel
GGoogle Rescarch AI
Reviewed on OpenReview:
Abstract
Real-world natural language processing systems need to be robust to human adversarics. Collecting examples of human adversaries for training is an effective but expensive solution. On the other hand, training on synthetic attacks with small pertarbations- such word- substitution does not actually improve robustness to human adversaries. In this paper, propose an adversarial training framework that uses limited human adversarial examples to generate more useful adversarial examples at scale. We demonstrate the advantages of this system on the ANLI and hate speech detection benchmark datasets- collected via an iterative, adversarial on observed human attacks, also training on our synthetic adversarial examples improves model robustness to future rounds. Tn ANLI, we see accuracy gains the current of attacks (44.1% 50.1%) and two future unseen rounds of human generated attacks (32.5% + 43.4%, and 29.4% - 40.2%). In hate speech detection, we see AUC gains on current attacks (0.76 + 0.84) and future round (0.77 + 0.79). Attacks from methods that do not learn the distribution of existing human adversarics, meanwhile, degrade robustness.
procedure. Compared to training only
1 Introduction
Robustness to real numan-generaen adversarial examples is crucial to reliable natural language process- ing (NLP) systems. While training on past real attacks improves robustness to future real attacks (Kicla et al., 2021), gathering such data is costly and time-consuming (Xu et al., 2021; Hendrycks et al., 2021). This motivates us to maximize model robustness using fixed dataset of human-generated adversarial examples, without the need for additional human effort or larger better model architectures. We achieve this goal by leveraging synthetic examples craft to mimic real attacks. This approach enables more cost-cffective and scalable NLP security.
Adversarial examples are designed to induce misclassification (Szegedy et al., 2014). Adversarial robustness is the ability of a classifier to correctly label adversarial examples. Though adversarial robustness has been extensively studied on benchmark NLP tasks (Jia & Liang, 2017; Ettinger et al., 2017; Zhang et al., 2020; Gao et al., 2018), NLP classificrs often falter against real-world text adversarics (Lees et al., 2021; Borkan et al., 2019) due to the limitations of current solutions. These failures can result in real-world harms (Scheuerman et al., 2021).
Current NLP robustness solutions typically fall into two categories: training on human generated (Dinan et al., 2019; Nie et al., 2020) or on synthetically generated adversarial examples (Uesato et al., 2018; Jin et al., 2020). There are not enough generatede datasets as these are expensive to create (Xu et al., 2021; Hendrycks et al., 2021) only becoming even more prohibitively so as models get larger and have more use cases (Ganguli et al., 2022). Synthetic attacks, meanwhile, (Uesato al., 2018; Jin al., 2020)
"Equal contribution.
Work done while author was at Google.

---

Published in Transactions on Machine Learning Research (01/2024)
often oversimplify adversarial attacks. Unlike in computer vision, small changes in text (discrete tokens) can drastically alter meaning, making it hard to craft synthetic examples with the desired label. This leads to oversimplified attacks based on easy metrics, which may not translate to robustness against real humans (Uesato et al., 2018).
Popular synthetic NLP attacks include (a) template-based or small text-oditddistance attacks, (Malfa. & Kwiatkowska, 2022; McCoy et al., 2019b; Zang et al., 2020; Ren al., 2019), (b) perturbation attacks that use word embeddings and search within an e-neighborhood (Jia et al., 2019; Zhao et al., 2017; 2018; Li et al., 2021; Huber et al., 2022), or (c) finding universal adversarial perturbations (Moosavi-Dezfooli et al., 2017; Wallace et al., 2019; Mchrabi et al., 2022). Real attackers, meanwhile, are (i) known to make much larger edits from the original text, and (ii) are informed by each other's successful attacks, neither of which is captured in existing synthetic NLP attacks (West, 2017). In this paper, we take a step towards closing this gap by directly modeling the real attack patterns. This enables us to emulate human text attacks more realistically by (i) allowing larger edits and (ii) using existing real attacks to inform future attacks.
In prior work, the following proxies are usually used to measure whether generated adversarial examples are of good quality: semantic proximity (Malfa & Kwiatkowska, 2022), high attack success rate (Szegedy et al., 2014), low label noise (Malfa & Kwiatkowska, 2022), or distributional similarity to past attacks (Pillutla et al., 2021). These metrics, however, do not connect well to the attack patterns used by real adversaries. Our primary metric for attack quality is whether the generated attacks are useful when used in adversarial training to defend against future unseen rounds of suunan-genertted attacks. That is, can we increase robustness beyond what we achieve by only training on all existing observed human attacks? We leverage frameworks like Dynabench to test the evolving patterns of real adversaries (Kiela et al., 2021)-
We show that our attack generation methods, which learn the distribution of human adversarial examples, outperform both (1) attack generators that do not take the human examples into account, and (2) attack generators that do not learn the distribution but rely on random perturbations (See 6). Our attack generation methods are able to make improvements even when trained on as few as 500 real human adversarial examples. Finally, though prior adversarial literature places a high emphasis on adversary success rates, low label noise, or distributional similarity, we show that these quality proxies are not predictive of whether an attack generator can better defend against future attacks. Our primary contributions are to:
1. Demonstrate misalignment between synthetic and real attacks: synthetic allack approaches do not necessarily improve robustness to the real allacks from humans.
We empirically show that existing
2. Overcome misalignment by imitating real adversaries: We use generative models to directly imitate existing human-generated attacks. Our metric of success is how much we can improve robustness future real attacks (beyond what can be accomplished by adversarially training on all existing real attacks).
3. Improve adversarial robustness without relying on a better /bigeer model: Adversarial training on imitated real attacks provides significant robustness benefits. When compared to solely training on existing real attacks, we improve accuracy by 11% on unseen attacks in the ANLI benchmark, and by 8% on existing attacks in the hate speech detection benchmark.
4. Show misalignment between common attack quality metrics and attack usefulness in pre- venting future attacks: We empirically show that more distributional similarity, low label noise, or high adversary success rate do not entail that an attack generator is better than another in helping defend against downstream attacks.
2 Related Work
Adversarial robustness measured as accuracy on challenge sets such as Adversarial GLUE (Wang et al., 2021), which are gathered through crowd-soureing and programmatic text perturbation (Zhang et al., 2019; Morris et al., 2020b). To improve adversarial robustness, prior work uses training interventions and/or data augmentation. Training interventions focus on learning more meaningful stable representations from
2

---

Published in Transactions on Machine Learning Research (01/2024)
available data: e.g. using mutual information (Wang et al., 2020a; Zhao et al., 2022) or locally regularizing (Aghajanyan et al,, 2020; Zhu et al., 2019). In contrast, we focus on data augmentation, which may be used with these training intervention methods.
Despite the popularity of data augmentation solutions such as e-bounded PGD attacks (Madry et al., 2017) in continuous domains, it is not straightforward to extend them to discrete NLP settings. In NI.P, small perturbations to the text can have a big impact on the text's true label.
Nevertheless, controlled text generation has vastly improved in recent years through zero few-shot tuning of large language models (Wu et al., 2021; Perez et al., 2022). Such methods use data augmentation for fine-tuning (Michel et al., 2019b; Garg & Ramakrishnan, 2020), gradients from small attribute classifiers to ensure generated text has a particular class (Dathathri et al., 2020; Wu et al., 2023), or careful prompting to generate semantically close text with only the desired attribute swapped (Madaan et al., 2020).
Adversarial controlled text generation, however, is even more challenging. It's not straightforward to correctly label the generated text we intend to use as an adversarial example (by definition). Prior work typically gets around this challenge by making very small or rule-based perturbations that should not change the original example's 's truc label (They assume low label noise is a good criteria for an attack generation method). These include contextual
(Michel et al., 2019a; Jia et al., 2019; Li et al., 2020a; Morris et al., 2020a), rule-based grammar (McCoy et al., 201 9a; Ribeiro et al., 2018), morphological (Tan et al., 2020), and character-level (Eger & Benz, 2020) manipulations. A related but under-explored area is to adversarially intervene on all parts of the text that are invariant to the predicted label (Wang et al., 2020b; Chen et al., 2021; Lei et al., 2022), or conversely to minimally intervene to ensure the true label changes (Ross et al., 2021; Deng et al., 2022). Our attack method is different in that it relies on memorizing and re-mixing phrases from existing adversarial examples to generate additional adversarial examples that the attack method can correctly label. All of these methods, when used for adversarial training, improve robustness to their respective attack type. In this paper, however, we ask whether they improve robustness against human adversaries.
Most of prior work assumes that adversarial text should be fluent, grammatically correct or semantically similar to the original. There is no such constraint on adversarics in the real world. Hence, in this paper we use the Hate Detection task which often does not have fluent grammatical text. Prior work that does not assume standardized single language, include studies on dialectal language like Ebonics on Twitter (Blodgett et al., 2016), multilingual text on Reddit (Michel & Neubig, 2018), Hinglish (Biradar et al., 2021)), emoji-based hatespeech (Kirk et al., 2022); and methods that seek to distinguish human and machine generated text through heuristics (e.g.: sentence length, verb ratios, (Yao et al., 2017), relational consistency (Zhong el. al., 2020)).
Red teaming: We are also motivated by red-team human generated
haanttts--mmmmccrtttttte attacks that target a specific model, often by crowd-sourcing. Examples include SWAG, ReCoRD, HotpotQA, HellaSWAG, HANS (Zellers et al., 2018; Zhang et al., 2018; Yang et al., 2018; Zellers et al., 2019; McCoy et al., 2019a; Kaushik et al., 2019). Sometimes the work also uses feedback from the original classifier: CoDAH, Quoref, DROP, FEVER 2, ANLI, etc. (Chen et al., 2019; Dasigi et al., 2019; Dua et al., 2019; Thorne et al., 2019; Nie et al., 2020; Bartolo et al., 2020; Kiela et al., 2021). We rely on such hunan-model interactions in a foedback loop sct-up, and propose a mechanism to reduce red team costs.
3 Problem Formulation
Red-teaming is popular because training on past real attacks improves robustness to future real attacks (as depicted in Fig. (a)). Real attacks, however, are costly to source. Thus for a fixed datasct of past human attacks, our goal is to improve robustness to future attacks even further without resorting to sourcing more human attacks (and without resorting to a larger or more capable model architecture). We achieve this goal by making use of syuthetically generated examples as depicted in Fig. 1(b).
We evaluate robustness our success criteria- as performance on successive rounds of adversarial examples collected through the crowd-sourced model-in-the-loop approach (Nie et al., 2020) in Fig. 1(a). We take the
3

---

Published in Transactions on Machine Learning Research (01/2024)
human attackers
human attackers
Model: Ye
Crowd-sourced
atlacks:
Model: yo
Crowd-sourced
attacks:
augment data
fine-tune
Synthetie
fine-tune
attacks: Ag
(a) Human in the Loop Adversarial Training
(b) Human in the Loop Adversarial Training With Synthetic Data Augmentation
Figure 1: In traditional human-in-the-loop adversarial training, humans attack attack a model, and then the model learns from those attacks to become more robust (Fig. (a)).. Augmenting the human -eeeannneeeeated attacks with synthetic attacks is a popular way to increase robustness (Min et al., 2020) (Fig. 1(b)). We propose two new methods that generate synthetic adversarial attacks by learning the patterns real crowd-sourced attacks. Our methods significantly outperform existing techniques defending against futuure yet-unseen crowd-sourced attacks. Such prior work synthetic altacks does not typically learn patterns from real crowd-sourced attacks we do; they focus making small edits that make the attack harder while ensuring low label noise (Feng et al., 2021).
classifier architecture to be fixed; improvements to architecture model size out of scope. We focus only the data augmentation method.
Notation: (See Fig. 1) Let x be a collection of samples drawn from some distribution X x X, with ground truth labels y(x). There is a classifier, paramcterized by du, that is trained on x. This classifier oulputs label predictions: (x). Human adversaries at lack classifier Og create a dataset of crowd-sourced attacks, ag Xa. All examples in Xs, fool the classifier, i.e., To, (a) A y(a)Ya e Xay.
The classifier is then further fine-tuned this dataset of real attacks, au- We analogously refer to the predictions of this newly fine-tuned classifier as To, (x). Again, human adversaries atlack classifier 0n to create a dataset of crowd-sourced attacks, a1 Xa, All examples in Xa, fool the classifier, ie.. De, (a) + y(a), Va E X...
This model-in-the-lode process results in evolving crowd-sourced real atlacks and evolving predictions as follows:
Model: De, Crowd-sourced attacks: ag - Model: To. Crowd-sourced attacks: +
(1)
Our goal is to improve the accuracy of Jo. on these yet unseen future attacks: a a2, without 1) gathering additional human attacks beyond as or 2) resorting to using a larger or better model architecture for Te,- We accomplish this by also fine-tuning on synthetic data (Soe Fig. 1(b)) follows.
Model: go. Crowd-sourced attacks: as Synthetic attacks: ag - Model: ge
(2)
Note do not have access to Xa, Xx, a a2 while training Po,- As such, do not have guarantees whether a, will be similar to ag, but we do know they are generated using the same crowd-sourcing methods.
4 Methods
To address the problem in Soc. 3, we describe how we use synthetic attacks improve robustness. Then we describe the specific methods of generating the synthetic attacks.
4

---

Published in Transactions on Machine Learning Research (01/2024)
Overall Solution Framework
As depicted in Fig. 1(b); we aim to be more robust to uniseen future human attacks (a1, (a1, a2) by fine-tuning our classifier De, not only past observed attacks (ag); but also on additional synthetic examples (ag). We create these generated examples with a generator, G. This generator learns what existing adversarial examples, ag Xag looks like, and uses this learned approximation to generate synthetic examples: ag ~ Xag- Notably, we ensure the model we use for G' is no larger and no more capable at the target task than the original classifier T6g that was fooled by the past observed adversarial examples (ao). We want the improvements to come from the data augmentation method, not from the knowledge of better model. In our experiments, G is finc-tuned T5 model, and the classifier is fine-tuned Bert-Large.
We have no guarantees that future unseen allacks (ai, will be similar to the past seen allacks (ao). We know, however, that they share an attack generation process. Thus, the hypothesis of this work is that modeling the existing real attack distribution allows the generator (G) to capture somothing about the real allack generation process more broadly. We hypothesize the generator can generalize to future unseen attacks. Restated, we hypothesize training on Xa, will not only make the classifier robust to Xag but also reasonably robust to future adversarial attack distributions, Xa, Xaz
A caveat: while in practice ai and all subsequent attacks would depend on what can fool the new model O2 created by training on the synthetic examples (ag) also (See Fig. 1(b),, we do not generate new human attacks in response to the new classifiers we train. Hence aı and ay are fixed in our setting from the original set-up Fig. (a)). The new rounds of attack are often also generated by fooling larger more capable models we do not use (ex: RoBERTa Large instcad of BERT Large) (Kiela et al., 2021). Instead we show that future human attacks (2)) are less effective thanks to training on ag. than they would be by only traning on past. observed examples ag.
Synthetic Attack Generators
For many NLP tasks, the input x can be broken into (1i, I3 is the portion of the input text that is not attacked by the ad adversary. Tt remains the same. This can be context: e.g. premise in NL.I, paragraph in QA tasks, etc. For the toxicity task, where the entire sentence may be attacked, we set 2; to be the first half of the text. Io is the portion that is attacked: c.g. hypothesis in NLI, question in QA, comment in sentiment analysis. We now present two methods to generate ag X. The first is imitation-onlyy approach agnostic to the task. The second takes the classifier task into account to better maintain the desired class label.
Method 1. Direct Imitation (DI): Label-aware fine-tuning We fine-tune the generative model on existing observed attacks, ag ~ Xa.- We consider (x;, v) as the input for the generator, and &, to be the target text that needs to be generated. Incorporating y as input greatly helps reduce the rate noisy labels the rate at which the the generated example does not retain the same label as the input (y(ag) + y(a))). Specifically, we minimize the cross-cntropy loss of the generated text probabilitics X (2,, v) and the target text K.- See Appendix A for additional implementation details and loss function definition.
The primary challenge of the DI method, and the primary challenge of all controlled adversarial text generation more broadly, are noisy labels (By definition, adversarial examples are hard to label correctly when we restrict ourselves from relying on a bigger better model).
Method 2. Imitation + Controlled Exploration (ICE):
To overcome this challenge, we modify the Plug and Play controlled decoding method to make it suitable for adversarial robustness (Dathathri et al., 2020). There are three key components:
1. A text generation transformer model, T, that can generate free-form text. It has been trained on a reconstruction loss: given vi, it must output the corresponding To.
Attribute classifiers, C. This is a single layer food-forward network with a cross-entropy loss a very simple model that focuses specifically on predicting the task label, y. This classifier is used to guide the tokens picked by the text. generation model, T.
5

---

Published in Transactions on Machine Learning Research (01/2024)
ao
The Nassau County population increased from 2010 to 2016.
The Crystal Mountain Resort is a tourist des- tination.
ag Crystal Mountain population increased from 2010 to 2016.
Table 1: An example of the 1CE attack generator remixing existing observed attacks (top two) from the ANLI R1 data to create new attack (bottom).
A steering signal. This the information used by the attribute classifier C to steer Our steering signal are the hidden layer activations after each transformer unit in T.
The system works as follows. We provide the generative model T' with a starting prompt 2;. T generates some initial text based on it's understanding of what the matching To is likely to be. The attribute classifier C analyzes the text generated so far and assigns it a probability of belonging to the true label y based on the stecring signal. The generator T slightly adjusts its stecring signal to minimize the loss C, and then picks new loken to generate. Thus the generator T adapts its generated output with the desired true label. This process repeats for each token generated by T. Restated, the main generator T generates some text similar to the original attack, while the attribute classifier C' nudges it to output text that has the correct. label. This process is outlined in Algorithm 1.
This system only works well when the simple linear classifier C' makes reasonably correct predictions of the true label, y. Else unable to guide text generation effectively. We, however, are working with adversarial examples that are difficult even for a much larger model (including the base model of T) to correctly label. A simple linear classifier like C is not likely to correctly label adversarial examples.
Warm-starting Trick:
We mitigate this issue by training the system (both T and C`) all past observed examples ag- And then passing the same examples ag at inference time to generate synthetic examples.
Though this may run the risk of memorizing and replicating exactly the examples seen at training time, in practice however, this not the case. There is diversity of synthetic text generated as each of the three components are imperfect. Further, we use a higher temperature and because there are multiple To associated with any given x; in our datasets, the adversarial examples generated by ICE method do largely re-mix concepts from existing examples (See Table 1). For additional implementatiin details and loss function definitions, refer to the Appendix A.
decode from T at inference time,
5
Experiments
This section details how we evaluate the methods listed in Section the benchmark dataset used, imple- mentation details, and baselines.
A. Tasks: Human-in-the tho-loop crowddsourced
We want to assess whether we can meaningfully amplify past future numan--gnnrated attacks (given fixed attack generation instructions and U1). Hence, we chose two very different DynaBench tasks: Natural Language Inference (NLI) and Hate Speech Detection: The former has longer and qualitatively more varied texts. The latter is terse, less varied, and has less standard English (often with incorrect grammar and spelling) (Kiela et al., 2021).
attacks be more robust.
We evaluate methods on the Adversarial NLI (ANLI) task (Nie et al., 2020). This is Natural Language Inference (NLI) task: the goal is to determine whether a hypothesis logically follows (entailment) or contradicts (contradiction) or is undetermined (neutral) based on facts present in the premise. Nie et. al. crovd-source human attacks on the hypothesis against a base classifier, ÿe., trained on SNLI+ data (Wang et al., 2018; Bowman al., 2015). Then they train more robust models by incorporating these new attacks (and other data) for three rounds. ag are the human generated attacks from the first round created by attacking Te,, BERT-Large transformer classifier (Devlin et al., 2019).
A.I. Adversarial NLI:

---

Published in Transactions on Machine Learning Research (01/2024)
Algorithm 1 Pseudo-code for ICE Method
Input:
V(z, v) E ao, where I = (o), zy: unchanged context in text, dgi text that is perturbed by the attacker; y: true label corresponding to x T: pre-trained encoder decoder transformer model that can do generation and classification
Adversarial dataset:
S: a vector of the activation states of the hidden layer after each transformer unit in T
C: linear feed-forward classifier that takes S(x) as input.
Warm-start:
Fine-tune T on dataset a on the reconstruction task: + Zo Finc-tune T on datasct ag on the classification task: x + y Freeze the parameters T and train on dataset a on the classification task: S(x) -
Iterative Adapt and Generate:
T' < copy of T
Freeze the parameters of C' and unfreeze that of T for all (x. E do
0g 0
while end of sentence token & ag A length(ag) < maximum length do
Og f- 0g + f, next token generated using the reconstruction task ((x:)
Backprop VyClassifierTasttoos (ot.a///......../e//eeeaaaaaaaaa
vo) for 10 steps
end while
yield (I 0g): a synthetic adversarial example corresponding to (æ, v)
T T"
end for
Successive rounds aj, a2 are created by attacking RoBERTa models. We choose to improve the robustness of BERT-Large model Te. using an and evaluate on future human adversarial attacks: a1,
A.II. Hate Speech Detection: We also evaluate on the Dynabench Hate Speech detection dataset, an adversarial -aman-in-the-lop dataset generated in four rounds (Vidgen et al., 2021). In Round 1, Vidgen et. train a base RoBERTA classifier, Ve. on original content. created by humans. Tn Rounds 2-4, they create more robust RoBERTa models (Te, -To.-oo. by training on attacks created as follows: Human raters first create original content that successfully fools the base classifier. Then they perturb these new sentences to create even more challenging "contrast sets" with different labels. This data is then split into train, validation, and test sets, with half the test set entries created by annotators who do not appear in the training and validation sets to minimize annotator bias.
Note that for both tasks we do not gather additional human adversarial attacks targeting our improved classificrs. We evaluate on fixed sot attacks previously unseen by the classifier. This is a limitation of our set-up; gathering additional rounds of human attacks is left as future work.
Base Attack Generator Model is T5
We the T5 cncoder-decoder is compatible with multiple small sentence generation tasks (Raffel el. al., 2020). Second, its performance benchmark NLJ tasks (MNLI, ANI.), and the Hate Speech task is close to that of the BERT-Large model we seek to improve i.e. improvements are not coming from a superior model but rather imitating the real adversaries.
generator for this paper (Raffel et al., 2020) for two reasons. First, it
C. Baselines
We have two types of bascline attack generators: The first type uses existing observed attacks, but does not learn the distribution, instead relying on random perturbations, We use three of these methods: Fooleo Attak,, and CT-GAN. Text Fooler is very popular attack generation library that transmutes the most.
7

---

Published in Transactions on Machine Learning Research (01/2024)
Model
ag:
ay: R2
ag: R3
Me, Base + RI
44. 44.140.03
32.540.05
29. 29.420.05
(ooleeo)))
24.14
27.940.06
30.3..006
4 + BERT (ttrT-AAt
35.1
29.040.08
31 1.009
CT-GAN(RI)
26.840.14
29.5
29.5 29.5g.0.11
4 + DIMNLI+SNNLI
22.940.11
28.1 28.1
29.4 29.4
ICE(MNLI++SIL
33.9
33.7.40.67
33.5
4 DI(RI)
18.2+0.38
39.1+0.29
10.2:t0.37
ICE(R1)
50.1+1.43
43.4+2.91
39.9
Table 2: Improvement on ANLI mean accuracy (%) (+ standard error across 3 runs) when trained a attakss generated only from Round 1. The notation, DI(RI) for instance, refers to the method DI using R1 data to generate more examples. We underline the setups that outperform the Base + RI baseline.
Model
ag: R2
ax: R3
ag: R4
Base + RI R2
0.76
0.78 003
0.77
Fooler (R2)
0.781
2.7020
0.76
BERT-Attack
0.78
0.761..015
0.77
4 + DI(R2)
0.84
0.77
0.76
ICE(R2)
0.83
0.80
0.79
Table 3: Improvement Hate speech detection AUC (+ standard error across 3 runs) when trained allacks generated only from Round 2. The notation, DI(R2) for instance, refers to the met DI using R2 data to generate more examples. In this dataset, RI is not adversarially generated, and is analogous to the base MNLI/SNLI data in the ANLI task.
predictive words, while preserving semantic similarity and contextual coherence (Jin al., 2020). BertAttack is another popular method: it uses the model it's attacking to identify vulnerable words in the input; then, it uses BERT to generate substitutes for the vulnerable words. (Li et al., 2020b). CT-GAN is Generative Adversarial Network(Gonn Goodfellow et al., 2014) modified for controlled text generation where the NLI premise is used as the control text (Haidar et al., 2019; Betti et al., 2020). The second type of baseline learns an example distribution, but does not use the attack distribution. We repeat our main methods, ICE and DI, in new data setting. Instead of using ANLI R1, we use the MNLI+SNLI data used train the base classifier, Te..
6 Results
A. Synthetic human-like adversarial data improve robustness to future attacks. Distribution-agnossic baselines do not.
Table 2 and Table 3 show that for both tasks, the accuracy on future rounds of human-generated attack (a ag) improves when generated examples, ag from ICE and DI are incorporated into training.
The adversarial example generators that attempt to imitate aj (ICE and DI) out-perform all types basclines. First, they improve robustness beyond what we achieve by training on past human adversarial attacks, a alone. This improvement cannot be achieved merely by training for more steps on ANLI R1 as shown in Table 18 in the Appendix. Second, they out-perform methods that rely Foxteooler, BertAttack, and CT-GAN. Finally, they out-perform methods that imitate example distributions generated by other processes: ICE(MNLI+SNLL) and D((MNLI-SNLI).. The word/phrase substitution methods, BertAttack and TextFooler, improve accuracy within the same round for the Hate Speech dataset- this dataset is itself half-gencrated by making such minimal substitutions. Yet, these methods are not more effective on future rounds.
boiso-based attacks a like
Table 17 extends the setting where RI and R2 are past observed attacks available for training, and R3 the held out set of future human attacks, leading to similar conclusions.
8

---

Published in Transactions on Machine Learning Research (01/2024)
Test Set
R1
R2
R3
MNLI
SNLI
R1
0.9296
0.8
Train
R2
0.8771
0.9472
Set
0.1964
0.1862
0.9489
MNLI
0.0100
0.0123
0.0623
0.9553
0.6
SNLI 0.0042
0.0042
0.0090
0.0974
0.9371
TextFooler(R1)
0.9384
0.8510
0.2557
0.0104 0.0040
0.4
BertAttack(R1)
0.9109
0.8636
0.2288
0.0123 0.0042
DI(R)
0.2630
0.1736
0.1062
0.0072 0.0042
ICE(R1)
0.9490
0.8763
0.1786
0.0129 0.0044
0.2
D1(R1+R2)
0.2899
0.2622
0.1413
0.0081 0.0043
ICE(R1+R2) 0.9281
0.9298
0.2201 0.0118 0.0044
Figure Distributional similarity, as measured by MAUVE on RoBERTa emboddings from a random 1k sample (Pillutla al., 2021). MAUVE scores range from to with higher values indicating more similar distributions. MAUVE metrics intended to be evaluated relative to each other, and not absolute measures. Note that distributional similarity to the held out attacks, R2, does not correlate with whether an atlack generation method is useful as per Table 2.
B. Common metrics for an attack method (label noise rate, attack success rate, and distributional similarity) do not entail whether adversarial data from the method can help defend against future attacks.
Adversarial robustness literature considers attack methods to be better if they produce datasets with less label noise (Dathathri et al., 2020), , or higher attack success rates (Uesato et al., 2018), or higher proximity to the original dataset (Ross et al., 2021). We show that these metrics are not good proxies for determining which type of allack method can generate examples that best defends against future allacks. These findings are surprising, and additional investigation is needed to understand why (Also see generated examples and additional correlation analysos in Appendix A).
B.I. Higher Distributional Similarity of ag to future attacks does not entail more useful adversarial examples. We use MAUVE stato-of-tho-art metric of distributional similarity between two text datasets in Fig. 2 (Pillutla et al., 2021). We find that we cannot predict whether synthetic dataset will improve the model's ability to resist future attack datasets simply by looking at how similar dissimilar the two datasets are. For example, we trained the model on three synthetic datasets: DI(RI), ICE(RI), and TextFooler(R1), Though (extFooler(R1) is more distributionally similar to the R2 future attacks than DI(RI) (as per Fig. 2), DT(RI) better at increasing robustness to R2 (as per Table 2). ICE(R1), meanwhile has both high distributional similarity and robustness. This finding analogous to findings in computer vision: for example, training la Projected Gradient Descent attacks in MNIST increases robustness to Decision Boundary Attacks. These are two very different distributions of attacks: created with gradient methods, and the other not (Madry et al., 2017; Brendel et al., 2018). A better understanding of how robustness transfers across different attack distributions is an open problem.
B.II. Lower label noise does not entail more useful adversarial examples. One of the hardest challenges in controlled adversarial example generation is generating examples that have the desired label, that is reducing the rate of noisy labels. While this is a useful goal within an attack method type, comparing rates of noisy labels across attack methods does not help us choose a more useful method. Table 4 shows that TextFooler has more correct labels than the DI method, even though Table makes it clear that DI is the more useful method. We, the authors, annotated 100 examples from each attack method and report the

---

Published in Transactions on Machine Learning Research (01/2024)
Generated attack dataset.
Rate of correct labels
agi
TextFooler(R1)
51%
ag: DI(RI)
39%
Ag. ICE(R1)
78%
Table 4: Rate of correct labels from the attack generation methods on ANLI round (t better). We, the authors, rated 100 random generated examples from cach method to obtain these numbers. Our rating guidelines are included in the Appendix A, and the ratings are in Supplemental Materials.
Model to attack - Base
Base+-RI
Base- R1+R2
Attack Dataset | De.
V/.
Ve.
ag: R1
78%
56%
47%
aj: R2
73%
67%
58%
a2: R3
71%
71%
62%
ag: TextFooler(R1)
96%
98%
94%
ag: DI(RI)
76%
87%
79%
Ag: ICE(RI)
78%
88%
88%
Table Attack success rate of attack datasets on base BERT Large models they incorporate more rounds of ANLI training data (f better). Only the examples that had the correct labels (as verified by human rating) are included in computing this rate.
rate of correct labels by comparing annotated ground truth with the adversarial label associated with the corresponding (premise, hypothesis) pair in the ANLI R1 dataset. We refer to the Appendix A for tables of generated examples from each of the attack methods, and guidelines for human annotations.
B.III. Higher adversary success rate does not entail more useful adversarial examples. Table 5 shows the attack success rate of the various attack datasets (only including attacks with the correct label as verified by human ratings). All synthetic attack generators have high success rates all classifiers - which indicates that the information in the generated examples were not yet captured in by the base classifiers. Surprisingly, TextFooler, which has the highest attack success rate, does not improve adversarial robustness; R2, R3, DI(RI), and ICE(RI) datasets have lower attack success rates but are much more useful in increasing robustness to future attacks.
C. Even ~1k human adversarial examples improves robustness unseen adversaries.
We test how the number of human adversarial examples used to train the attack generator affects the adversarial robustness using the generated examples. We fix the nunber of examples we use to train De, at 10k generated examples. The full set of real adversarial examples from ANLI R11 is 16.9k real examples. We sample the number of real examples we use to train the attack generator. Table 6 demonstrates that increasing human examples improves robustness. Nevertheless, even when we provide low numbers of human examples, robustness to future rounds has improved. In-distribution accuracy on R1, however, suffers until examples are used train the attack generator. When we have fewer than 500 human adversarial examples to amplify using approach, the robustness gains do not generalize. That happens because the few-shot setting creates several additional considerations that are out-of-scope in this paper:
Simple fine-tuning basclines do not perform well in the few-shot setting. Other few-shot and parameter elficient baselines might be relevant (Zhou el. al., 2022; Liu et al., 2022). The choice of training method is orthogonal to our goal of synthetic data generation.
As we reduce the number of adversarial examples, evaluating generalization requires categorizing examples into various patterns ensure coverage. In this work, we instead use the pattern-agnostic approach of Dynabench (Kiela et al., 2021), which aggregates all human-generated attacks.
10

---

Published in Transactions on Machine Learning Research (01/2024)
# samples
Ag: R1
a2:
R.2
a2:
100
33.0
27.8
28.9
200
34.2
28.1
29.5
500
30.8
32.6
1024
42.7
36.5
36.1
2048
43.2
33.2
36.5
4096
43.3
37.8
37.1
8192
44.5
36.6
36.6
all (16.9k)
48.2
39.1
40.1
Table 6: Test accuracy (%) as we vary the number of RI real adversarial examples used to fine-tune the DI attack generator. We use the trained DI generator to generate 10k examples. We fine-tune a Base + R1 classifier on these 10k DI(RI) examples to produce the robustness metrics above. We underline the setups that outperform the Base R1 bascline in Table 2.
D. The ICE method, in effect, re-mixes phrases from previous observed attacks.
The reconstruction task used to train the generator in the ICE method (Alg. 1) often leads to exact memorization of train-set examples or exact phrases from the train-set. Qualitatively, we find that increasing the beam search parameter a and number steps aligning the steering signal, or decreasing reconstruction loss, al inference lime leads to remixing these phrases (See Table 1). This re-mixing effect also explains the high MAUVE distributional similarity between the ICE methods and the ANLI rounds on which trains.
Chan et al. (2022) suggests that LLMs learn concepts better when they are repeated, esp. when they are repeated in slightly varied conlexts; ICE's re-mixing behavior facilitates exactly this outcome. We hypothesize that the repetition and slightly varied contexts may help the model learn the essence of the original human attacks, Icading to improved performance against similar attacks.
7 Limitations
If we put our new more robust models to use, human adversaries may adapt to them as well. Checking whether crowd-sourcing fresh attacks is indeed more difficult on the new models beyond the scope of this work. Also, we benefit from having a fair number of human adversarial examples (16.9k in ANLI, and 10k per round in Hate Speech). Our methods are less successful in scenario with much fewer examples (~ 100). On the flip side, have also not evaluated these methods classifiers with access to even larger real adversarial datasets.
We ran our experiments on popular transformer architectures: BERT-Large for classification (as this is the model that was attacked to create the adversarial datasets we rely upon) and T5 for generation. T5 was chosen for two reasons: 1) is comparable to BERT-Targe size and performance but uses a different. architecture and training, thus ensuring our findings are not just due to model size difference. 2) The transformer Encoder-Decde is now the ubiquitous generative model architecture. The generalization of our findings to non-transformer architectures is left for future work.
Finally, our methods work on datasets with the notion of an original example, and a perturbed adversarial example as is the norm for adversarial robustness literature (Madry et al., 2017). In the new paradigm of larger more capable NLP models, adversarial datasets may increasingly not involve a perturbation (Ganguli et al., 2022).
Risks: Our techniques enhance robustness given set of observed adversarial examples. The new classifier trained with generated data from DI and ICE may still be vulnerable to future human attacks that are able to adapt to the new model (in this paper future attack rounds are known priori from past model-in-thhe-lopp work). This would require exlensive crowd-sourcing efforts to evaluate. We also run the risk of over- wett-fi to the new human generated adversarial data. This may come at the cost lower performance on future attacks generated by a different mechanism (say TextFooler instcad of future ANLI rounds), and comes at the cost of degrading accuracy on original tasks such MNLI and SNLI (Table 19 in Appendix). As separate concern, any technique that betters generative text modeling brings the risk that humans may
11

---

Published in Transactions on Machine Learning Research (01/2024)
struggle to distinguish machine generated text. This can have negative consequences for disinformation and misinformation, which is an active area of research (Pu et al., 2023).
8 Conclusion
We demonstrate that training attacks that imitate human adversaries improve robustness to future rounds of human adversarial attacks by 11% on ANLI, and by 6% and 8% existing adversarial examples on ANLI and hate speech datasets. We are able to improve robustness to future attack distributions even when the attack generator is only trained on 1000 real adversarial examples. We show that existing attack generation methods that do not train on the distribution of real attacks, however, (methods like TextFooler and CT-GANS) are unable to improve robustness to future real attacks. Finally, we discover that attack generation methods with the lowest label noise or highest attack success rate or highest distributional similarity to future attacks are not the best methods at increasing robustness to future real attacks. These findings run counter to accepted norms choosing the best attack method type in literature, demonstrate the opportunities in leveraging increasingly effective generation methods, and motivate future work on improving real adversarial robustness.
References
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. arXio preprinl arXiv:2008.031 2020.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the AT: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662-678, 2020. doi: 10.1162 /aacl a 00338. URL https: // htt///allaa/ aclanthology org/2020.tacl-1.43 tac1-1.43.
Federico Betti, Giorgia Ramponi, and Massimo Piccardi. Controlled text generation with adversarial learning. In Proceedings of the 13th International Conference on Natural Language Generation, pp. 29 34, Dublin, Ireland, December 2020. Association for Computational Linguistics. URL https://aclanthon 2020. inlg-1.5.
org/
Shankar Biradar, Sunil Saumya, and Arun Chauhan. Hate or non-hate: Translation based hate speech identification in code-mixed hinglish data set. In 2021 IEEE International Conference Big Data (Big Dala), pp. 2470 2475, 2021. doi: 10.1109/ SigDauus2559..201..997
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. Demographic dialectal variation in social media: A American English. In Proceedings of the 2016 Conference on Empirical Methods in
case study of African-American Natural Language Processing, pp. 1119-1130, Austin, Texas, November 2016. Association for Computational http////aaattoooo..rrrr//------- org/D16-1120.
Linguistics. doi: (0.18653/v1. /1/11-1111110 1120. URL
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pp. 491-500, 2019.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 1011663//1/D15-1000 URL https://aclantthlooog org/D15-1075.
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models, 2018.
Stephanic C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang. Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers, 2022.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Michael Chen, Mike D' D'Arcy Alisa Liu, Jared Fernandez, and Doug Downey. CODAH: An adversarially- authored question answering dataset for common sense. Tn Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pp. 63 69, Minneapolis, USA, June 2019. Association for Computa- tional Linguistics. doi: 10.18653/v1 w1/////1111110000// URL https://aclannhooloyy org/v19-2008.
Yangyi Chen, Jin Su, and Wei Wei. Multi-gramularity textual adversarial attack with behavior cloning. ar preprint arXiv:2109. 04367, 2021.
Pradcep Dasigi, Nelson F. Liu, Ana Marasovié, Noah A. Smith, and Matt Gardner. Quoref: A reading comprehension datasel with questions requiring coreferential reasoning. In Proceedlings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5925 5932 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10111633/11DD1-- org/D19-1606.
URL https://acl aalaaclllnntooyyyyy
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020. URL https:/ / http:///pppereiieeeeeee seeperreveennee/foruuue net/forummid HledEyBKDS.
Chuyun Deng, Mingxuan Liu, Yue Qin, Jia Zhang, Hai-Xin Duan, and Donghong Sun. VaICAT: Variable-length contextualized adversarial transformations using encoder-decoder language model. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, pp. 1735 1746, Seattle, United States, July 2022. Association for Computational Linguis- tics. doi: 10.18653/ 1036653/v//// 1122022..2222 mww./22aaaaaalllaemmmmaaaaa maaaaclmmai..1...... URL httpss//aclantthllll . org/2022 naacl -main. 125.
Jacob Devlin, Ming- Weig Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North Americon Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171 4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653) /v/ 11119-142... URL htttps//aclaatheet
org/N19-1423.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4537-4546, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 10.18653/v1 /119//11/1191141.11 URL https:// //wp../////..wwwbbbb....... wwwllleeeborrrraeeeeee org/anthology/ D19-1461.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pupers), pp. 2368 2378, Minneapolis, Minnesota, a, June 2019. Association for Computational Linguisties. doi: 10.18653/v1/N19- 1246. URL https:// aclanthology 1246.
Steffen Eger and Yannik Benz. From hero to zéroe: A benchmark of low-level adversarial attacks. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 786 803, Suzhou, China, December 2020. Association for Computational Linguistics. URL https: httt::://aclanthologg arg.2000.aaccmmaaii... aacl-main. 79.
Allyson Ettinger, Sudha Rao, Hal Daumé III, and Emily M. Bender. Towards linguistically generalizable NLP systems: A workshop and shared task. In Proceedings of the First Workshop on Building Linguis- tically Generalizable NLP Systems, pp. 1-10, Copenhagen, Denmark, September 2017. Association for Computational Linguisties. doi: 10. 18653//v1 /1/177-551 URL https:
org/W17-5401.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for nlp, 2021.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, 2022. URL https:/ http://aaavvvvo//a wttggabb/2200..000
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. CoRR, abs/1801 0.0344, 2018. URT.
04344.
Siddhant Garg and Goutham Ramakrishnan. Bae: Bert-based adversarial examples for text, classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6174-6181, 2020.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Parley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https: / /ttp/v/axvvvv/va///////e.//a.ava org/ org/abs/ 1406.2666.
Md. Akmal Haidar, Mehdi Rezagholizadeh, Alan Do Omri, and Ahmad Rashid. Latent code and text-based generative adversarial networks for soft-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguislies: Human Language Technologies, Volume (Long and Short Papers), pp. 2248 2258, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 11/119-1N11111 URL https:/ sttps://aalaantoooogy org/N19-
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CW /CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11262,15271, June 2021.
Lukas Huber, Marc Alexander Kühn, Edoardo Mosca, and Georg Groh. Detecting word-level adversarial text attacks via SHapley additive exPlanations. In Proceedings of the 7th Workshop on Representation Learning for NLP,, pp- 156 156-166, Dublin, Treland, May 2022. Association for Computational Linguistics. doi: URL https://aclantholo org/2002.rep14nlp-11..
10.18653/v1 /t2p///11/2222.ro//nnpl---
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems, 2017. URL https: / /////axxvvv.orr///aaa /a/////b///////////////// 073288.
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adversarial word substitutions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4129 4142, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/ D19- 1423. URL attps::/aciie //tpa//taaannnooyocco/cc. org/D19-1423.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 8018 8025, 2020.
Divyansh Kaushik, Eduard Hovy, and Zachary C. Lipton. Learning the difference that makes a difference with counterfactually-angnee
data, 2019. URL htttps://axxvv.oom
1909/ 12434.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanprect Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Scbastian Riedel, Zecrak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4110-4124, Online, June 2021. Association for Computational Linguistics. doi: 10.18653//vr hvtpvv1121v2/a/ naacl- main. 324. URL naacl-main 324.
httpss///ace http::://cclaahoooog.orggg//
14

---

Published in Transactions on Machine Learning Research (01/2024)
Hannah Kirk, Bertic Vidgen, Paul Rottger, Tristan Thrush, and Scott Hale. Hatemoji: A test suite and ad adleesarialll--geerrate dataset for benchmarking and detecting emoji-based hate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1352-1368, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.1 10.18653/v1 /ttp//2222...mmmma.a/a 97.
URL ptstps://aalaatthol..goo org/2022 naacl-main.
Alyssa Lees, Daniel Borkan, Ian Kivlichan, Jorge Nario, and Tesh Goyal. Capturing covertly toxic speech via crowdsourcing. In Proceedings of the First Workshop on Bridging Human-Compeln Computer Interaction and Natural Language Processing, pp. 14-20, 2021.
Yibin Lei, Yu Cao, Dianqi Li, Tianyi Zhou, Meng Fang, and Mykola Pechenizkiy. Phrase-level textual adversarial attack with label prescrvation. In Findings of the Association for Computational Linguisties: NAACL 2022, pp. 1095-1112 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653, /01553vvv/// 22022.fnndings--. na.cl.83. URL https:/ pattps:/aaclatthee aclpatooonnntotcccccoooo org/2022. f iindings-naacll..
Dianqi Li, Yizhe Zhang, Hao Peng, Liqum Chen, Chris Brockett, Ming-Ting Sun, and Bill Dolan. Contextual- ized perturbation for textual adversarial attack. In Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5053-5069, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ (11853//v1// /tt.//2......lmmmoo.... man.....aaa-mainn0000 URL sttts://aclaatthlogg. org/2021 maacl-main. 400.
Linyang Li, Ruotian Ma, Qipeng Guo, X. Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert. ArXiv, abs/2004. 200440904444 2020a.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK: Adversarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6193-6202, Online, November 2020b. Association for Computational Linguistics. doi: 10. 10.18653/v1 /2tp:////.22...elpmmmm......... URL https::/aclanthhllogg org/2020. amnlp-main 500.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficien fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha. Generate your counterfactuals: Towards controlled counterfactual generation for text. CoRR, abs/ /0022.20944698 2020. URL htttps//arxiv.org. abs/2012.04690
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2017.
Emanuele La Malfa and Marta Z. Kwiatkowska. The king is naked: on the notion of robustness for natural language processing. In AAAI, 2022.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 3428 3448, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10.18653/v1 //P19/ 1334. URL. https://aclannthology www.ooyy..gg.ooo//o//11
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. Tn Proceedings of the 57th Avaual Meeting of the Associalion for Computational Linguistics, pp. 3428-3222 1228-444, 2019b.
Ninareh Mehrabi, Ahmad Beirami, Fred Morstatter, and Aram Galstyan. Robust conversational agents against imperceptible toxicity triggors. In Proceedings of the 2022 Conference of the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2831-2847, 2022. URI https: htat:s://acaaatiii ht////aaaaaaalooaaaraaaaaa org/e022222222222 maacl-maine aaain.204/.
15

---

Published in Transactions on Machine Learning Research (01/2024)
Paul Michel and Graham Neubig. MTNT: A testbed for machine translation of noisy text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 543-553, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1//118-1 https://accanthholggy http:///ccccaaalooooooooog---000 org/D18-1050.
URL
Paul Michel, Xian Li, Graham Neubig, and Juan Pino. On evaluation of adversarial perturbations for sequence-to-sequencc models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 3103 3314, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10.18653/vv ht////////11/// 1314, URL /aclan/thlll org/N19-1314.
Paul Michel, Xian Li, Graham Neubig, and Juan Miguel Pino. On evaluation of adversarial perturbations for sequence-lo-sequeeee models, 2019b. URL bttpp:://axxiv.org/bbbbss 00600bb6600000000....
Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. Syntactic data augmentation increases robustness to inference heuristics, 2020.
Seyed-Mohsen Moosavi-Dezfooli, Alhusscin Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.
John Morris, Eli Lifland, Jack Lanchantin, Yangleng Ji, and Yanjun Qi. Reevaluating adversarial examples in natural language. In Findings of the Associalion for Compulational Linguislies: EMNLP 2020, pp. 3829 3839, Online, November 2020a. Association for Computational Linguistics. doi: 10.18653/v1/2000 findings-cmnlp.34 cmmlpp.411 URL
org/2020. ..f0f.fmmm findings-eminn 341.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 119 126, 2020b.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885 4901, Online, July 2020. Association for Computational 10//////11222/// v1/2021..222l aaa.....lmmai..4444... 441. URL attps://accaanthologg wwwpaattttoooooooooooeoeoo org/2020 acl-main 441.
Linguistics. doi: 10.18653/vl
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAlecso, and Geoffrey Irving. Red tcaming language models with language models. In Proceedings of the 2022 Conference on Empirical Melhods in Natural Language Processing, pp. 3419 3448, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. URL https: //aclanthology. org/2022. emnlp-main.222
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 4816 4828. Curran Associates, Inc., 2021. URL https: pa.........
//proceedings. . hww.oeeeiig...orrr...ccpppppppre http:///pwpppppo..///////////oaa
J. Pu, Z. Sarwar, S. Muhammad Abdullah, A. Rehman, Y. Kim, P. Bhattacharya, M. Javed, and B. Viswanath. Decpfake text dotoction: Limitations and opportunitics. In 2023 2023 IEEE Symposium on Security and Privacy (SP) (SP), pp. 19 36, Los Alamitos, CA, USA, may 2023. IEEE Computer Society. doi: 10.1109/ 0P44215.2023.000 00002.. URL http:://oo....eeecpppurroooooooe Lececomputersocietyy org/ 10. httg//..119//P4621155 2220222111.22200002222
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (140):1-61 2020. URL http://jmlm: //jmlr. www.ooogg..eeee... http:/ppers///////222 074. html.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeling of the Association for Computational Lònguistics, pp. 1085- 1097, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1 11/59//11/911103.. URL https: hatpss///ccaaathloogo wtw.:/ooooo..rg/111111110 org/P19-1103.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeling of the Association for Compulational Linguisties (Volume 1: Long Papers), pp. 856 865, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. (0.18653/v1, /1/0///11/1111//// URL https://aclanthhi http://caaatttoooooooor/////11/0 2rg/P18-1079.
Alexis Ross, Ana Marasović, and Matthew Peters. Explaining NLP models via minimal contrastive editing (MiCE). In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 3840-3852, Online, August 2021. Association for Computational Linguistics. doi: 10.18633/v1/2022 find://iig///..iidiingggg-g wiwdings-acll.... acl.333.. URL https://aclaaate
org. findingsn-aann acl.3s3--cc333666.... 336.
Morgan Klaus Scheuerman, Jialun Aaron Jiang, Casey Fiesler, and Jed R. Brubaker. A framework of severity for harmful content online. Proc. ACM Slum--Cmpput Interact., 5(CSCW2), oct 2021. doi:
10.1145/3479512. URL https:///dl www.lii..rr...... 1rg//00.111///3795.11. 1145/349912.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. URL hett:///rxivv wtw.a/twwwww..waaa........oooeee aa/////s//111/../////////////1/1 6199.
Samson Tan, Shafiq Joty, Min- Yen Kan, and Richard Socher. It's morphin' time! Combating linguistic discrimination with inflectional perturbations In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2920 2935, Online, July 2020. Association for Computational Linguistics. URL
doi: 1100663/v12220.acmmm//
www//020..co...ooooo. macl-main.20
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoollpoulls,, and Arpit Mittal. The FEVER2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pp. 1 1-6, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DI9-6600 URL https://acllanthologg// 0tt.://ogo..rr11--6010011
Jonathan Uesato, Brendan O'Donoguue, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks, 2018.
Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. Learning from the worst: Dynamically generated datasets to improve online hate detection, 2021.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 1153-2162. 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. CoRR, abs/1804.00461, 2018. URL http:// /taxxv/ org/ abs//804..a. 07461.
Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. Infobert: Improving robustness of language models from an information theoretic perspective. arXiv preprint ar Xix::2000000 02329, 2020a.
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprinl Ntfi::111.028... 2021.
Tianlu Wang, Xuezhi Wang, Yao Qin, Ben Packer, Kang Li, Jilin Chen, Alex Beutel, and Ed H. Chi. Cat-gen: Improving robustness in nlp models via controlled adversarial text generation. In EMNLP, 2020b.
17

---

Published in Transactions on Machine Learning Research (01/2024)
Sarah West. Raging against the machine: Network gatekeeping and collective action on social media
platforms. Media and Communication, 5(3):28 36, 2017. ISSN 2183-2439. doi: 10. 776645/mac.551333 URL https: /tt::///www.... http://wwww...iote www.:gooiittipprss..ooo httpa//cc.........ooooooaooaatta
Hongqiu Wu, Yongxiang Liu, Hanwen Shi, Hai Zhao, and Min Zhang. Toward adversarial training on contextualized language representation. arXiv preprinl Xiv: 04537, 2023.
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Popers), pp. 6707 6723, Online, August 2021. Association for Compulational Linguistics. doi: 1.11653/v1/202// wllp2//../2lllooooooomoo aalllonng.520. 523. URL https: http////aaaaathhllgoooo
2rgg2021.... acl- sol-long-- 123..523.
Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. Bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguislies: Human Lunguage Technologies, pp. 2950 2968, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ 1 h/////111222111/11 naacl- https://aclanthollse org/2021 naacl-main 235.
URL
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hot A dalaset for diverse, explainable multi-hop question answering. 2018. URL
CoRR,
Yuanshun Yao, Bimal Viswanath, Jenna Cryan, Haitao Zheng, and Ben Y Zhao. Automated crowdturfing attacks and defenses in online review systems. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pp. 1143 1448-115, 2017.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking combinatorial optimization. In Proccedings of the 58th Annual Meeting of the Association for Compulational Linguisties, pp. 6066 6080, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 10.18653/v1/2000..oo main.540.... URL httts://aclanthologg. org.oog///20 acl- 1400
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. SWAG: A large-scale adversarial dataset. for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 93 104, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 18..3/66111// //D1//////////////// 1009. URL https://aclaathologgy org/D18-1003
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1 11/19911111111/ 14722 URL
org/P19-1472. 1472.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. CoRR, abs/1810.12885 2018. URL attp://arxis: wwwoorggg...oooo..o aas//810....////////////// 128.1888....8
Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial allacks on deep-learning models in natural language processing: A survey. ACM Transactions Intelligent Systems and Technology (TIST), 11(:):1 41, 2020.
Yuan Zhang, Jason Baldridge, and Luheng Hc. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1298 1308, 2019.
Haiteng Zhao, Chang Ma, Xinshuai Dong, Anh Tuan Luu, Zhi-Hong Deng, and Hanwang Zhang. Certified robustness against natural language attacks by causal intervention. In International Conference on Machine Learning, pp. 26958 26970. PMI.R, 2022.
18

---

Published in Transactions on Machine Learning Research (01/2024)
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. Adversarially regularized autoen- coders. In International conference machine learning, pp. 5902-5911. PMLR, 2018.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. Generating natural adversarial examples. arXiv preprint arXio:1710.11334 2017.
Wanjun Zhong, Duyu Tang, Zenan Xu, Ruize Wang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. Neural deepfake detection with factual structure of text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2461-2470, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/v (0.18653/v112022eemm)) /20000.emmlp-mmal. 193. URL https: // //aclanthology. hwwplllggyyoggg//// org/2020....2.. mamlp-main. 193.
Xin Zhou, Ruotian Ma, Yicheng Zou, Xuanting Chen, Tao Gui, Qi Zhang, Xuan-Jing Huang, Rui Xie, and Wei Wu. Making paramcter-officient tuning more efficient: A unified framework for classification tasks. In Proceedings of the 29th International Conference Compulational Linguislics, pp. 7053 7064, 2022.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv: 11764, 2019.
19

---

Published in Transactions on Machine Learning Research (01/2024)
A
Appendix
FextFo Generated Examples
A random selection these attacks on round of the ANLI dataset are included in Table 9. random selection of these attacks on the first round (Analogous to round in ANLI) of the Hate Speech Detection dataset. are inchuded in Table 13. Trends for this attack type are explained in the table captions.
BertAttack: Generated Examples
A random selection these attacks on round of the ANLI dataset are included in Table 7. random selection of these atlacks on the first round (Analogous to round in ANLI) of the Hate Speech Delection dataset. are included in Table 14. Trends for this attack type are explained in the table captions.
CT-GAN: Generated Examples
A random selection these attacks on round of the ANLI dataset are included Table 8, Trends for this atlack lype are explained in the table captions.
DI method: Generated Examples
A random selection these attacks on round of the ANLI dataset are included in Table 10. random selection of these attacks on the first round 2 (Analogous to round in ANLI) of the Hate Speech Detection dataset are included in Table 15. Trends for this attack type are explained the table captions.
ICE method: Generated Examples
A random selection these attacks on round of the ANLI dataset are included in Table 11. random selection of these attacks on the first round 2 (Analogous to round ANL.T) of the Hate Speech Detection dataset are included in Table 12, Trends for this attack type are explained the table captions.
Implementation Details for DI Method.
In this section we give more specifics needed replicate the DI method. The general outline of the method is provided in Section
For the generator, (Raffel et al., (020)).The base T5 model is then trained on ANLI R1 attacks, with the premise and the label as the input (delimited by special token [NLI LABEL]:). The label is denoted by one of the three letters: (e, n, e) to denote entailment, neutral, and contradiction. The output sequence that the T5 model is trained to decode is the attack hypothesis in the ANLI RI data. We use the standard cross-entropy loss function as deseribed in the base T5 model and use the train split of the ANLI R1 data. We choose the best checkpoint based on the token-level accuracy in generating the hypotheses in the validation set of the ANLI R1 dataset. We then produce an in-sample generated attack dataset by producing multiple hypotheses (up to 100) on the train split of ANLI R1. Thus, ensure that there is no leakage of attacks that were not already previously seen by the base classifier.
the default settings for the base T5 model, with changes to the hyper-pant parameters
The cross-entropy loss of the generated text probabilities Xao (zi, y) and the target text Vo, assuming maximum sequence length K, and vocabulary V is given by:
K
Loe
Σ k=lveV
1 1 log(Xao (To 4)R.))
(3)
For the classifier, we use the BERT-Large pre-trained model, and first fine-tune it on the MNLI+ data mixture as per the ANLI benchmark. We then fine-tune that classifier further the ANLI RI train
20

---

Published in Transactions on Machine Learning Research (01/2024)
Original ANLI RI Premise
Original ANLI RI Hypothesls (Ru)
BertAstack(RI): Generated Hypothe- sis (ag)
Original Label
Label Still
Base
Base
Base
+
R1
+
R1
Cor-
(iny)
R2
The Other Oue is the third solo album by former Plectwood Mac guitarist Bob Welch. The track 'Future Games' was first released ou the Fleetwoud Mac album of the same name 1971. Members of Welch's is back- ing band also make songwriting contribu- tious here though the majority of trucks Welch's own
Memnerrs of Welch's backing hand include Maynard
Member of Welch's backing hand encom- pass Menard James Keenau who helped Boba Welsh with
a/a
u/a
n/aa
James Keetun who ussisted Welch with almost every track The Other One."
about everything
trails The Else One
The Walloon Legion (French: "Légiin Wal- lonie") was collaborationss volunteer unit recraited from Belgium's is French-ppcaiig population Wallonia and Brussels during the German occupation
The Walloon Legion was partly recruited from Brussels.
The Walloon Legion was not a volunteers
e
n/a
n/a
n/a
unils.
World War II. The Walloon Legion served the Wehrum- cht, later in the Waffen-SS, on the Eastern Frout on both front line and reserve duties.
Irma Pezzia Hanbold (November 20, 1908 April 4, 1996) was an American artistic gymnast. She competed the 1936 Sum- mer Olympics and placed fifth with the leam. She macried fellow Olympic gymnast Frank Haubold. They were the first married couple of compete in the same Olympics.
Irma competed in Irma participated the 1940 Summer Olympics.
c
c
the 1940 Hsia Olympics.
Mia Foni debut album of Greek Amer- singer Annet Artani. features 19 tracks in both Greek and English, includ- ing 'Why Augels Cry", the song that Annet performed in the Earovision Song Contest 2006 in Athens representing Cyprus. The album was released both Greece and Cyprus where
Annet Artani's album
Annet Artani's 15 trajec-
15
tories.,
the top 10.
Ernest Thompson Willows (886-1926) was pioncer Welsh aviator and airship builder. He became the first person in the United Kingdom to hold pilots certificate for an airship when the Royal Acro Club awarded him "Airship Pilots Certificate No. 1
Ernest Willows pilot.
Thompson
Thomson
Willows experimenter
James Hugh Sinclair (16 October 1876 South African cricketer who played in 25 Tests from 1896 to 1911. He scored South Africa's first three Test centuries and was the first person from any country to score a century and take five wickets an innings the same Test. He is of of all.
They refered to him fast scorer.
They refered to rapids scorer.,
y
n
February 1913)
scoriin Test batssmen
Table 7: Selected examples from BertAttack(RL) attacks. Like the TextFooler attacks in Table this attack makes small modifications to the hypothesis, explaining the high level of distributional similarity to ANLI rounds 1 and 2. Unlike TextFooler, however, BertAttack makes more sophisticated contextual phrase substitutions. For example in row 2, 'partly recruited from Brussels' replaced with 'not volunteers units."
data split. We choose the checkpoint based the best validation set accuracy of ANLI RI dataset. This fine-tuned model forms the basis of our comparison, and our goal is to improve adversarial robustness beyond what this model can achieve.
In the DI method, we then fine-tune the Base + RI model on the generated examples. For all fine-tuning processes, we used an adamw optimizer Jax + Tensorflow, The following learning rate schedule was used: There are first 3,681 warm-up steps the initial learning rate of 3.0o-05. Then for the next 36,813 stops the learning rate decays at linear rate (Though we only fine-tune for 40k steps total). The checkpoint that performs best. the validation split is selected for the next stage.
21

---

Published in Transactions on Machine Learning Research (01/2024)
Original ANLI RI Premise
Related
Original
CT-GAN(RI): Gen-
Original Label
Label Still
Base
Base
Base
ANLI R1
erated (ne)
Hypothesis
(1%
+
R1
+
R1
Hypotheses (no)
Cor-
(iny)
R2
The Parma trolleybus system (Italian: "Rete filovirria di Parma" ) forms part of the public trausport network of the city and "comune" of Parma, the region of Raaillaa,maaa,,, northern Italy. In op- eration since 1953, the system presently comprises four urban routes. Spocus
1953 the Troleybus has been four urban routes."
"Since
Parma has boen operation for the en- tire region of Emilia- Romagua, northern Italy.
Parma system
comprised
comndy sketch show first. braddrss on the United Kingdom's Chan- uel 4 from 30 September 2005. In the United States, "Spoous" is broadeast un BBC America. The relationship themed show combined
Spouns the comeddy skotch show first up-
Spoons comesdyy sketch show first broadcast on the United Kingdom'ss Channel from September 2005
a/a
a/a
4/a
peared on the Uuited Kingdom'ss is Channel a year after
treuds sketch thows-date content, strong lauguage, and recurring catchphrases.
The 1992 Boise State Broncos foothall team represented Boise State University in the 1992 NCAA Division I-AA fouthall seuson. The Broners competed
["After 1991 Brise State Broncos com- peted in the Big Sky Conference." ; "Bnise State four games their house dium in 1993.""
The 1992 Boise State Bronoos foothall team competed the Big Sky Con- ference and played their home gumes Bronco Stadium.
Big Sky Con- ference and played their home games Brouco Stadium Buise, Idaho. Led by year head cuach Skip Hall, Boise State finished the season 5 6 overall and 3-4 in conference.
Other ranks (or ORs) the Royal Danish Army, Royal Danish Navy and Royal Dan- ish Air Force joined term for military persounel that not officers of various forers, by the NATO system of ranks and Insignia. These personnel are NCOs and privates, seamen or airerallsmen
There are no NCOs, non-commissioned ticers, the Danish Military services.
These personnel are NCOs privates, aircrafts-
n
n//
n/8
D/a
'Drunk Girls" is song American rock band LCD Soundsystem. the second single from their third studio album, "This Is Happening" (2010), on May 3, 2010. Band frontman James Murphy has described the song as 'dumb' but added like dumub, short stuff." The ?" siugle cover the song by San Francisco peychedelie rock band Wooden Shjipe.
James Murphy like dumb, short stuff".
'Band frontman Murphy said like dumb, short stuff
e
was releused
Viru is 5.0% ABV pilsner-style boor brewed in Estonia. It is brewed in the coun- try's second largest. city, Tartu, by the A. Le Coq brewery. The brand is owned by Baltic Beer Company Ltd (formerly Brand Independence Ltd), based in London, UK, and brewed under licence in Fsstoiaa Le Coq is the socond largest brewery in Estonia, with a market share of 36.8% in 200
Viru is 5.0% ABV pilsner-style beer brewed in Estonia. It brewed the country's second largest city, Tartu, the A. Le Coq brewery. Lately there have been talks of setting up another brewery in same region
Viru is brewed the country's second largest ciby, Tartu, by the A. Le Coq brow- ery
n/a
0/a
Table 8: Selected examples from CT-GAN(RI) attacks. These attacks most often are direct sub-phrases of the premise. While they generate the entailment class examples correctly, almost no new information is ever added. Meanwhile, the tendency to pick up sub-phrases from the premise means the other two classes, neutral and contradiction, are largely generated incorrectly (Sec row 2 and 4). Further, since these incorrect example are often a verbatim copy of the premise, we may be teaching the model not mark entailment for content directly in the premise).
Implementation Details for ICE Method.
In this section we give more specifics needed to replicate the ICE method. The general outline of the method is provided in Section 4, and Alg. 2 provides more detail.
22

---

Published in Transactions on Machine Learning Research (01/2024)
Algorithm Pseudo-code for ICE Method (in more detail)
Part O: Define Variables.
We break the input z into two halves, x = (rz zo). Let Do be the portion of the text example that is perturbed by the attacker, and 2; be the unchanging context. Tet y be the true label corresponding (zi, ro) for given NLP task.
Let ReconstructToken and TaskToken be fixed constant tokens that identify which task asking T to perform.
Part 1: Set up components.
Let T be an encoder decoder transformer model. Let S be a vector of the activation states of the hidden layer after each transformer unit in T. This the stcering signal. We use o) denote the signal when we pass input (ReconstructtTkenn.) i) into T and T has generated o.
Let C be simple linear fordadd classifier that takes S as input.
Part 2: Warmstarting Trick.
Fine-tune T on dataset ag on the reconstruction task: (ReconstructToken.]) n;) - To; V(a, v) E an- Fine-tune T on dataset ag to predict the true label for the NLP task: (TaskToken, x) - y, V(x, y) an- Freeze the parameters of T that they are not updated. Train C on the dataset. a to predict the true label for the NLP task: S(x, 0) y, V(x, y) E ag-
Part 3: Generate synthetic examples.
Let 7" be saved copy of T. Unfreeze parameters of T so that they may be updated. Freeze the parameters of C so they do not change.
Since S is a function of the hidden layers T, it may also change as the parameters of T change.
for all (x, 3) E ag do
Pass in (ReconstructToken.t) x;)) into T input.
Let ag be the outpul generated by T. Set dg =0 0.
while T' has not generated end of sentence token or hit maximum length for ag do
for all k E 1, .... 10 do
Reconstructionlosss = Cross entropy loss between ag and Ho- Cross entropy loss between C(S(x, az)) and y.
ClassifierTaskloos
ReconstructionLoss Update parameters of T based the VLoss.
Loss ClassifierTaskLoss
end for Use T to generate the next token, and append the new token to ag.
end w while
yield ag: a new synthetic example. Set T
end for
23

---

Published in Transactions on Machine Learning Research (01/2024)
Original ANLI RI Premise
Original ANLI RI Hypothesls (Ru)
Tex:Fooler(R)). Gen-
Original
Label Still
Base (3%)
Base
Base
erated Hypothesis (ne)
Label
+
R1
+
R1
Cor-
(mny)
R2
(102)
"Anma" (Literally: Mother) is the first. Nopoli movic produced in Nepal, starring Shiva Shankar Manandbar and Bhuvan Chand (Thapa) as the leading actors. The movie was produced by the Information partment the Nepalese Goverament and released ou October 1964. Bollywood film maker Hira Singh Khatri was invited by the late King Mahendra to direct the first Nepali movie.
Hira Singh Khatri was the director for the movie 'Aama'.
Hira Song Khatri "c' the headmaster for the film 'Aama'.
False
a/a
u/a
n/aa
"Dangerous" is song by American elec- tronic music project Big Data, from their debut '1.0" (2013) and their debut diu album "2.0" (2015). Jeatures Ameri- can indie rock band Joywave, with vocals being performed by the band's lead singer Daniel Amubruster.
Big Data did not pro- duce any albums 2007.
Mussive Data not engender any albums 2007.
Fabe
a/a
u/a
4/a
Camp Al-Saur, referned to by some media sources Camp Falenn, Forward Operat- ing Base Falcon, Joint Service Station (JSS) Falcon, Combat Oulpust. Falcun, United States military forward operating base in Iraq short distance outaide Bagh- dad, some 13 km south of the Green Zone. 2004; was designated as "Camp Berrin-Huggins"
Nume of the soldiers agreed with names given the camp.
None of the soldiers countersigned with the names gave the campground.
'n'
True
2009, the base housed 5,000 troops.
Woll hunting with dugs meshod of woll hunting which relies on the use of hunting dogs. While any dog, especially a hound used hunting wolves may lousely termed 'wolfhound', several dog breeds have been specifically bred for the purpose, of which, such the Irish Wolfhound, have the word in their breed name.
wulves dogs
hunling
wolfe dogs
hunting
"c'
True
'n'
The discography of American metalcore band As Lay Dying consists of studio albumss, 2 compilation albuuss, 1 video al- bum, singles and corresponding music videos well as split album with fellow metalcore band American Tragedy called "As I Lay Dying/Aun American Tragedy'. Sir John Poebles Arbuthnots, PPRSE, FR, CPSG, FMedSci, FRCPath (born April 1939) Scottish mierobiologist, Principal the Univensity of Strathelyde. He succeeded Lord Wilson of Tillyorn President of The Royal Society of Edin- burgh in October 2011 and was suceesded by Dame Jocelyn Hell Burnell in October 2014.
The bands As I Lay Dying has filmed over 5 dowen music videns.
The strips As I Lay Death has videotaped over 5 dossn musscc videos.
"c'
False
n/a
n/a
n/a
Sir John Pechles
Monsicur John Pro- started out as a blos commenced out teacher.
'n'
True
te"
was
a teacher.
Table 9: Selected examples from Text Fooler altacks. These attacks make small random modifications to the hypothesis, explaining the high level of distributional similarity to ANLI rounds R.1 and R2. More examples are included in Supplemental Materials.
In addition to the DI method setup, we train lincar classifier using the T5 generator. The lincar classifier takes in each hidden layer at the end of each transformer unit. We flatten and concatenate the hidden layers before feeding them into a single dense layer + layer norm with 3 as the output dimension size (one for each class NLI). This is a departure from the PPIM implementation which sums the hidden layers (Dathathri et al., 2020). Adding another layer, even a small by dense layer, does help improve this linear classifier but slow down the attack generation process. So, we did not any additional layers. The linear
24

---

Published in Transactions on Machine Learning Research (01/2024)
classifier still does not perform well as T5 BERT- on the NLI task (71% accuracy MNLI dev compared to 91% for T5 and 89% for BERT-Large).
Assuming that the set of labels m (entailment, neutral, contradiction), the classifier optimizes the following cross-entropy classification loss:
(e(x,y)= -- Σ (l)
(4)
mEm
Further, the reconstruction loss used to warm-start the generator is given by:
L (x,
1 No my log( (Xao no (2,, v)..))
(5)
k=lvEV
During attack generation, we use high beam parameter, 0.7 for ANLI, and 0.8 for the Toxicity dataset. For additional example diversity, the weight on the reconstruction loss can be made negative: results here are presented with a reconstruction loss weight of 0 for ANLI, and 1.5 for the Toxicity dataset. On each step that we update the T5 parameters, we smooth the update (final parameter = original parameter * 0.75 + new parameter * 0.25). We did not substantially explore these syper-parametery as each example generation run takes 100 GPU hours from warm-up steps to example generation.
We only include generated attacks if the linear classifier labels the generated example belongs to the original class. For example, we filter attacks that contain words in the set { 'c',, 'n', 'c" }, more than two of cach of those class labels (filtering 'eee", or 'ccoc"), as these are NLI labels and an artifact of the NLI pretraining. We also filter attacks that exactly match a hypothesis in the observed set of examples, attacks that are less than two words long, or attacks that repeat a string of four characters more than four times (to avoid artifacts like: "acc00ccoccoocoo000 that may appear).
If the ICE method is used to generate data from new round, (ex: R1 for ANLI), the base model is trained the new round and synthetic data all in the same fine-tuning period (randomly shuffling the data). The original data is repeated until it is equal in number to the synthetic data. For example, if there are 1 original examples, and 5k generated data, the original set of 1 will be repeated five times, and the final shuffled dataset will have 10k examples. This ratio and shuffling method could be tuned further, but we did not attempt this.
Implementation Details for CT-GAN.
To implement the CT Generative Adversarial Network, adapted the ANLI task, we use the TF-GAN library with generator and discriminator initialized with the T5 and the BERT-Large model fine-tune on MNLI, SNLI, ANLI R1 data in scquential manner. These initializations DI and ICE methods. Then we train the generator and discriminator in iterative manner. In the first step, we fne-tune the generator to generate examples similar to ANLI R1 and produce a misclassification error from the classifier using a modified min-max objective. In the next step, we further fine-tune the classifier on those generated examples, with the original label as that of the corresponding ANLI R1 examples. We then repeat these two steps 10 limes. In each step, we train the generator for 20,000 steps, and the discriminator for 10,000 steps.
inline with what
in
Implementation Details for TextFooler.
To generate text attacks based on word-substitution, we use the TextFooler recipe outlined in (Jin al., 2020). We use the implementation in the TextAttack library 2 and produce augmentation examples by perturbs the ANLI RI hypothesis through word-embeddding swap. We generated 50 transformations per example, inline with the amount of data generated using our imitation and exploration based methods.
hutpe///gihubbccm///aa/////eeeee Attack
25

---

Published in Transactions on Machine Learning Research (01/2024)
Implementation Details for fine-tuning BERT Large.
We used the default parameters for the uncased BERT Large model detailed in the corresponding paper, (Devlin et al., 2019). The checkpoint that performed best the ANLI dev set used.
Human Rating guidelines.
The paper authors are the human raters. These are the guidelines we follow for edge cases.
The current year assumed to be 2019, since we are working with the base T5 model trained in that year, and our datasct, ANLI also generated in that year.
Examples with incorrect grammar are not penalized the matter asserted fit the correct label. (ex: "They very popular band")
Mild spelling mistakes, where the meaning is absolutely clear after reading the premise are not penalized: (example: FeliCa a intelligent card system developed for utilizing electron cash cards but has not yet gain endorsement to be utilizing in the United States).
Spelling mistakes named entities create a new entity (ex: 'Kënga" is not the same as 'K??nga' and 'Mars' is not the same as "Mar").
Word substitutions from other languages are okay they are commonplace English knowledge (ex: "sir" -> "monsieur"), but not okay if they change the meaning of the sentence (ex: "life" -> "vie", which has an alternative English meaning).
Repeating phrases okay. (ex: Mark Donovan best known for his role in productions such as "Shaun of the Dead', 'Shaun of the Dead", 'Black Books", "In Bruges", and 'Murder Investigation Team.") since they don't change the truth of the matter asserted.
Tautologies are still considered 'entailment' if they are true given the premise. (Ex: 'The movie, The Golden Compass, movie.")
Additional ANLI Result: Holding out R3 as a future round and training on R1 and R2 yields similar conclusions
Table 17 extends the setting from Table 2 where R1 and R2 are past observed attacks available for training, and R3 is the held out set of future human attacks. All conclusions are the same as the ones drawn from Table 2.
Additional Result: Size of ag used to train To. More examples better up until point.
Here, we seek to understand the role of the number of generative adversarial examples used for fine-tuning the base classifier. We use all available ANLI R1 real adversarial examples for fine-tuning the adversarial generator, and decode the specified number of examples from the generator for fine-tuning the base classifier. We demonstrate that as the number of generated examples increase, adversarial robustness increases up to a point (up to 50x: 50 generated examples for every real adversarial example). After that point, we observe more generated examples impacts adversarial robustness negatively shown in Table 16.
For the 50x experiment, however, we obscrve the best checkpoint while training BERT-Large is usually reached when the model has trained on only ~ 30% of the generaled examples provided. Therefore these findings are inconclusive. This behavior could be due to the interaction between (1) increased label poise/diversityy of examples generated as we increase the number of generated examples due to a quirk of the decoding process, and/or (2) the improvement adversarial robustness due to more examples incorporated into training until overfitting to the generated distribution.
26

---

Published in Transactions on Machine Learning Research (01/2024)
Sanity Check: More fine-tuning real adversaries does not help.
We study whether the gains obtained by the generated examples could have been obtained otherwise merely by training longer on the ANLI Ri data itself. In Table 18, we see that if we had continued for more steps training on R1 data, we do not observe higher validation set accuracy as compared to training on training on DI(RI) generated attacks. (These figures are not directly comparable to 2 in the Results section of the main paper, since here we present validation set accuracy, and the Results section contains test set accuracy).
Correlation analysis of heuristics with robustness
In Table 21, we find that the Pearson correlation observed between accuracy achieved on the three rounds of ANLI and the corresponding dataset heuristics those models were trained on is not significant. These correlation coefficients were computed understand if any adversarial dataset's heuristic attack success, label noise distributional similarity correlates with how well classifier would perform future rounds of human adversarial attacks. Our findings suggest that none of these heuristics are indicative and future work could focus on finding such data quality measure that correlates with future human adversarial robustness.
27

---

Published in Transactions on Machine Learning Research (01/2024)
Original ANLI RI Premise
Original ANLI RI Hypothesis (Ro)
DI(RI): Generated
Original Label
Label Still
Base (3%,)
Base
Base
Hypothesis (ag)
R1
R1
Cor-
(b,
+
R2
mc??
(g0z)
The Coward 1915 American silent his- lorical war drama filu directed by Reginald Barker and produced by Thomas H. Ince. Ince also wrote the film's story and see- mario with C. Gardner Sullivan. The film stars Frank Keenan Gilbert also appears part. A copy of "The Coward' is preserved at the Museum of Modern Art.
The Coward a part of American history and located in mu- seum for preserva- lion.
The Coward is 1915 American silent historical war drama film The Coward is 1915 American silent historical war drama film
"e'
True
'''
'c"
Charles Ray. John uncredited bit
"Dangerous" is song by American elec- tronie music project Big Data, from their debut EP "1.0" (2013) and their debut stu- album "2.0" (2015). features Ameri- indie rock band Joywave, with vocals being performed by the band's lead singer Daniel Armbruster.
Big Data did not pro- duce any albums in 2007.
It is unlike any other Electronic Music re- lease, but the Juy- wave may have been proud with music.
"c'
False
n/a
n/a
n/a
The Joint Aviation Authorities (JAA) was an asocciated body of the ECAC represett- iug the civil aviation regulatory authori- ties number European States who had agreed co-operate developing and implementing standards and procedures. It was not regulatory body, regulation being achieved through the member authorities.
The JAA is not ulatory body.
and horities of number of European states are from civil authorities represent the civil aviation author- ities of number of European States who had agreed regulate these standards and safety regulations.
"
False
n/a
n/a
n/a
safety regulatory
11 Flowers () is 2011 film directed by Wang Xiaosimai centered on buy named Wang Han who loses his white shirt which shows that he is his school's best gymnast. The loss metaphorical meaning as the film progresses which is inspired by the director's 's ex- perience youth during the cultural rev- olution and the more general confusion of childhood.
Wang Xisoshuai di- rected 11 films in 2011.
The loss of the shirt is to take on great film productions as the film progresses when progresses the end of the film.
'n'
False
a/a
a/a
n/a
shirt to take on greater
Tillanie DeBartolo (born November 27, 1970) American novelist, filmmaker, and co-founder independent record la- bel Bright Antenna. She has writien two novels, "God-Shaped Hole' and 'How To Kill Rock Star". She wrote and directed the film 'Dream for an Insomniac', featur- ing Jennifer Aniston, but
Jenuifer Anistou was not any movie directed by Tiflanie DeBartolo.
The film 'Dream for an Insommiac' written and directed Tiflanie DeBartolo was widely released in 1991.
True
'1'
'n'
'1"
very
limited release in
Viru is 5.0% ABV pilsner-style boer brewed in Estonia. It brewed the ond largest brewery try's 's second largest city, Tartu, by the A. Le Coq brewery. The brand is owned by Baltic Beer Company Ltd (formerly Brand Independence Ltd), based in London, UK, and brewed under licence in Estonia. Le Coq is the second largest brewery in Estonia, with market share of 36.8
Le Coq is the A. Le Coq is the ond largest brewery
'n'
True
'6'
in Estonia, with a market share of 36, 8% in 2005, had au in- creased market share 2015.
in Estonia, with market share of 36.8% 2005. The Le Cog brewery makes lot of beer Latvia, which had market share of 36.8% in 2005.
Table 10: Selected examples from DI(R1) attacks. These generated hypotheses deviato pretty significantly from the original hypothesis. They also incorporate knowledge from the premise (ex: 'second largest' phrase is common to the premise and generated hypothesis in the Le Coq example the bottom row). More examples are included in Supplemental Materials. Unlike the other methods, even when the label is incorrect, the generated hypothesis is usually still pretty cogent sentence.
28

---

Published in Transactions on Machine Learning Research (01/2024)
Original ANLI R1 Premise
Selected Original ANLI ICE(RI): Generated
Destred Label
Label Cor- rect?
Base (ia)
Base
Base
R1
Hypothesis (au)
+
R1
+
R1
Hypotheses for this premise (aw)
(ooy
+
R2
(Dkg)
The Coward is 1915 American silent historical war drama film directdd by Reginald Barker and produced by the
["Reginald Barker di- ricted a movic in 1915." ; 1915 American silent drama
Reginald Barker movie "c" was a silent drama.
True
*c'
*c"
"c"
Thomas H. Ince. Ince also film's 's story and scenario with C. Gard- wer Sullivan. The film stars Frank Keeman and Charles Ray. John Gilbert. also appears
historical film is preserved at the Museun of Modern Art.", -.]
uncrodited bit part. A copy of "The Coward" is preserved the Museum of Modern Art.
11 Flowers () is a 2011 film directed by Wang Xiaoshuai centered on a boy named Wang Han who loses his white shirt which shows that his school's best gymmastt. The loss of the shirt take greater metaphorical meaning as the film progresses which is inspired by the director's is own experience as youth during the cultural revolution and the general confission of childhoood
T'hhe movic was inspired by the director's expe- rience as youth dur- ing the american revo- lution.", '11 Flowers () is 2011 film directed by Wang Xlaoshual about boy named Wang Han who
11 Flowers () a 2011 "c' film directed by Wang Xiaoshusi about boy named Wang Han who loses his white shirt and is top gymnast. The shirt becomes metaphor for the di- rector's experience childhood. The shirt bocomes a metaphor for the directors's is ex- perience childhood.
True
white shirt and is longer top gymnast. The shirt be- comes a metaphor for childhood growing up in China.",
The shirt becomes metaphor for the film's stary.
Domestically, he has played for Khan Research Laboratories, Lahore, Lahore Eagles, Lahore Lions, Labore Qalan- dars, Pakissan A and Huntly (Srotland) during his career. He was the captain of Lahore Qalandars in the first edition of the Pakistan Super League.
['There are multiple leams based in Lahore, Pakistan.", ; "There been least mestic sports league in Pakistan.", "ikistan......) -.
There are at least sev- 'n' eral teams in life
True
'n'
'n'
do-
Benjamin Muncil (28 Aug 1867 16 Dec 1930)
'Benjammin Muncil built many things'. "One of Muncil's projects created for former US President. " ...
One best Amer- ican master builder in the Adtrondacks projects built many things created for a for- mer President.
True
American ter builder in the Adirondacks early in the 20th contury. He was major fig- ure in the architectural development of the Adirondack Great Campe; smoug many projects Marjorie Merri- weather Post's Camp Topridge, North- brook Lodge, and White Pine Camp, summer White House of US President Calvin Coolidge.
beer brand the island of Saint. Lucia, brewed by Windward & Leeward Brewing Limited, which owned by Heineken. The beer was named for the Gros Piton and Petat Piton mountains the island. was first browed October 1992.
Piton is a
['venn though beer first brewed in public in October, many tasted it previously', 'Even though the beer was brewed in October, is
Even though boer brewed in public October, no one tasted it previously the first of taste it formerly
False
n/u
a/a
n/a
tain where that occurred
Benjamin A. Muncil (28 Aug 1867 16 Dec 1930) was an American mas- hailder the 20th century. He ure in the architectural development of the Adrrondack Great Campe; among his many projects was Marjorie Merri- weather Post's Camp Topridge, North- brook Lodge, and White Pine Camp, summer White House of US President Calvin Coolidge.
Benjaanin Muncil bas built many things", "One of Muncil's projects was created for former US President. .....]
One of the best Annrr ican master builder in
True
Adirondacks early in major
Adirondaclos projects built many things created for for- mer US President.
Table 11: Selected examples from ICE(RI) attacks. This method generates hypotheses that learn from past seen hypotheses and the premise. Common failure modes include generating nonsensical sentences or getting stuck in loops of the same phrase (sec: row 2). Some of the neutral labeled generations technically correct, but very random (see: row 3). More examples with a complete set of original hypotheses are included in Supplemental Materials.
29

---

Published in Transactions on Machine Learning Research (01/2024)
Original Hate Example(ao)
Spoech
R2a
Matched
Hate
Spooch
R2b
[CE(R2):
Generated Example
Desired Label
Label Correct?
Base(RI)
Base R1)
Example(ao)
[ag)
(2h,)
+
R2
(30,)
are black folks extinet? cause they should be (hate)
are black panthers exctinct? (not hate)
man black folks always wear black.
"h'
False
'h'
'b'
lesbian subhummmmmman (hate)
Why the fuck do you think les- blans subbumau? (not hate)
lesban faciss doesn't deserve rights.
'h"
True
"b²
Who would off their wiener for done...ont, get it. it's like my brain just farts when I try to think what must go through these people's heads to ever con- template making that sort of crasy choice (hate)
Who would cut their wiener in several pieces for lunch..oont get it. It's like my brain just farts when I try to think of what must go through these people's is heads contemplatte making that sort of crasy choire. (not hate) cant believe that she think arahs are the most kind ethmic group, come on more least (hate)
Who would cut off their armios?
"n'
True
'h²
"n"
can believe that she thinks arabs are the most kind ethnic group. like come on they are
how can he believe he caunot teach anyone so much English
'n'
True
'u'
lovely (not hate)
black people super clean (not hate)
negrors not clesan (hate)
Black people BLIGGNOW
"n'
True
"1'
"
My work docsu't I cater for people
Glad my work doesn't I cater for people with disabilities. (bate)
My work doesn't it cater for pet dugs who're close to pet par- ents.
"n'
True
'b²
"b""
with invisible disabilities. (not
hate)
look dude, gype evil and sically cant be trusted. Espe- cially with their cheating crystal halls, they freak me the fuck out. Grubby looking caravan lovers.
look dude, they are evil and ba- sically cant be trusted. Espe- cially with their cheating crystal balls, they freak me the fuck out. Grubby looking witches. (not hate)
look dude, chinese climates evil and busically cant be trusted
'h1
False
'b''
Espocially with chinese chia.
(hate)
your personality is so ugly it could use makeup, typical black chick (hate)
your personality
ugly could use makeup [not hate)
Your black for black for black persson too heavy for whites
"h"
True
'h'
'h'
all women are nos to be trusted, they are snakes, vicious vipers trying trick their men (bate)
all men are not to be trusted, they are snakes, vicious vipers trying trick their women (not hste")
all women are bitches everyone they have all fucking wifes
"h"
True
"h²
'h''
the bathroom scummy we need some viakal and to get the cleaning sqaud in here. Maybe do that tomortow, iuaa go sort out somet stuff now (not hate)
the bathroom is so scummy looks like an Arab house, hon- estly. We need some viakal and to get the cleaning squud here. Maybe I'll do that tomorrow, ima sort out someet stuff now (hatc)
the hathroom is secroeummyy we need some viskale and to think there is anything wrong us just shut up and think about it, there should be black people living there
'h²
True
"h'
'h'
Islamii Relief should be banned (not hate")
Islamicc Relief should be banned, so should islamic people (hate)
Islamic Relating To The Devil
"h²
True
"b'
'1''
Table 12: Selected examples from ICE(R2) attacks the Hate Detection Task. This method generates text that learn from past seen text. Common failure modes include generating nonsensical sentences (see: row 5) or not being able to correctly label text (see: row 1). Some of the generated text are very random compared to the original examples. Often irrelevant text is added (see: row 7). Some of these lift phrases from other examples (ex: 'deserve rights' in row 2 comes from another example). Sometimes the original label itsclf is incorrect (marked with *).
30

---

Published in Transactions on Machine Learning Research (01/2024)
Original Hate Speech Examppee(ss)
R2a
Matched
Hate
Speech
R2b
FextFooler Generated Ex- ample (ag)
Desized Label
Label Correct?
Base(R1)
Base (R))
Example(aa))
(0ky)
+ R2
(3e,)
are black folks extinet? cause they should be (hate)
are black panshers extinet? (not hate)
are black folks missing? cause they should be lesbian sublummmonnnnan
'h!
True
"h¹
lesbian subluumuuuutntnnnn (bate)
Why the luck do you think les- bians are subbunan? (not bate)
'Hi
True
"1""
Who would cut off their wiener for a hole...ont get it. it's like my heain just think of what must go through these people's heads to ever con- template making that sort of crazy choice (hate)
Who would cut their wiener in several pieces for douth..ouut get it. it's like my brain just farts when I go through these people's s heads to ever contemplate making that sort crazy choice. (not hate) cant believe that she think arabs the most kind ethnic group, like come on more like least (bate)
Who would chop off their wiener for hole...ont gel it, it's like my brain just farts when try to think of what must, through these people's heads to ever con- template doing that sorted of psychopath choice
'h²
True
'n'
'b"
when try to
think of what must.
can helieve that she thinks arabs the most kind ethnic group. like come on they are lovely (not hate)
can believe that she idea arabs are the most kind ethnic group, like come on they are lovely
"n'
True
black people are super clean (not hate)
negroes are not clean (hate)
negro people are super clean
"n'
True
'b'
"0"
My work dorsn't cater for people
Glad my work doesn't cater for people with disabilities, (hate)
My work doosn't eater people with invisible disability.
"n"
True
with invisible disabilitics. (not hate)
look dude, they are evil and ha- sically cant be trusted. Espe- cially with their cheating crystal balls, they freak the fuck out. Grubby looking witches. (not hate)
look dude, gyps are evil and ha- sically cant be trusted. Espe- cially with their cheating crystal balls, they freak me the fuck out. Grubby looking caravan lovers. (bate)
look dude, they are wicked and basically dicnnt be trusted. Espe- cially with their cheating crystal balls, they freak the fuck out. Grubby looking witches.
"n'
Falso
'n'
''n'
your personality ls so ugly It could use makeup, typical black
your personality is so ugly it could use makeup (not late)
your personality is so ugly it could use maksuup, typical black dame
'h²
True
'n'
'b"
chick (hate)
all women not be trusted, they are suskes, vicious vipers trying to trick their men (batee
all men not be trusted, they are snakes, vicious vipers trying to trick their wonenn (not hate*)
be trusted, they are serpents, vicious vipers trying tu trick their womtnn
men
'h'
False**
'b'
"0'
the bathroom scumnyy we need some viakal and to gel the cleaning squad in here. Maybe I'll do that tomorrow, ima sort out somet stuff now (not hate)
the bathroom is so scumny looks like an Arab house, hon- estly. We need some viakal and to got the cleaning squud in here. Maybe I'll do that tomorrow, ima go sort out somet stuff now (bate)
the toilettes is scummny we need some viakal and to get the clean- ing squud in here. Possibly I'll do that tomorrow, ima go sort out somet stuff now
'n'
True
Islamic Relief should be banned, so should islamic people (bate)
Islamic Relief should be banned (not hate**))
Moslems banned, people
Relief should be islamic
'h² True
'b'
'b'
should
Table 13: Selected examples from (R))))))er(R2) attacks on the Hate Detection Task. As in the ANLI task, it makes random word substitutions. In this task, it emulates the style of creating the matched example pretty well, which is perhaps why we see AUC improvement within R2 in this task with this dataset (*Whether "negro" is pejorative is contextual, and this rating decision could go the other way. *e don't 't agree with this label, but it is consistent with the label choices in the dataset. ***This label comes from the dataset and may be incorrect.)
31

---

Published in Transactions on Machine Learning Research (01/2024)
Original Hate Exampple(ss)
Speech
Reta
Matched
Hate
Speech
R2b
(ertAttack(R2): ample (ag)
Generated Ex-
Desized Label
Label Correct?
Base(Rt (Dmy)
Base (R1)
Example(au)
R2
are black folks extinet? cause they should be (hate)
are black panshers extinet? (not hate)
are negro folls disappeared? rea- they should lesbian rs sublumammmnntnan
'h!
True
lesbian subluteotttnnttnn (hate)
Why the luck do you think les- bians are subbuman? (not bate) Who would cut their wiener in several pieces for lonch..dout get it. it's my brain just farts think of what must
'h'
True
"1"
Who would cut off their wiener for a hole...ont get it. it's like my heain just think of what must go through these people's heads to ever con- template making that sort of crazy choice (hate)
Wh Whom ought slice off their wiener for hole...hats obtains it's likes my brain just when endeavour to thinking of what owes going through these people's 's chiefs to ever envisage doing that sorts of maniac opted dunno believe that she believ- ing arabs the
"R'
True
"b''
'b''
when try to
when I go through these people's 's heads to ever contemplate making that sort crazy choice. (not hate) can believe that she thinks arahs the most kind ethnic group, like come they lovely (not hate)
cant believe that she think arabe the most kind ethnic group. like come more like
'h"
??7
'h'
types racisu grouping, like arrive more like fewer
least (hate)
black people are super clean (not hate)
negroes are not clean (bate)
nigger people aze fabulous clean
'n'
False
'n'
'b"'
My work doesn't cater for people
Glad my work doesn't eater for people with disabilities, (hate)
My corporation doesn't cater people with undetectable pacity.
"n"
True
with invisible disabilities. (not hate)
look dude, they are evil and ha- sically cant be trusted. Espe- cially with their cheating crystal balls, they freak the fuck out. Grubby looking witches. (not
look dude, RYps are evil and ha- sically cant be trusted. Espe- cially with their cheating crystal balls, they freak me the fuck out, Grubby looking caravan lovers. (hate)
glance they are wicked and primarily becuase be trustwor- thy. Specially with their cheat- ing crystal balls, they flipper the fucking out. Grubby looking bruges.
'n'
12ps
"n'
'''
hate)
your personality is so ugly it could use makeup, typical black chick (hate)
your personality is so ugly it could use makeupp (not lute)
your personality is so hiideous it did utilise nakeup, typical black dame
'h'
True
'n'
all women they suakes, vicious vipers trying to trick their men (bate)
be trusted,
all men not be trusted, they smakes, vicious vipers trying to trick their women (not hate")
men not be trustworthy, they smakes, ferocious viper tempting to stratagen their nu- jer,
'h'
False**
'b'
'n'
the bathroom scummny we need some viakal and to get the cleaning sqand in here. Maybe I'll do that tomorrow, ima sort out somet stuff now (not hate)
the bathroom is so scummy looks like an Arab house, hon- estly. We need some viakal and to got the cleaning sqaud in here. Maybe I'll do that tomorrow, ima go sort out somet stuff now (bate)
the toilettes is scummay we needs some vinkal and to obtain the cleansing squad here. Likely I'll do that tomorrow, ima going sorted out somet thing now,
'n'
True
Islamic Relief should be banned, so should Islamic people (bate)
Islamic Relief should be banned (not hate*)
Islamists Relief should pro- hibiting, so should islamist peo- ple "la
True 'h'
'b'
Table 14:
Selected examples from (ttackttack(R2)) attacks on the Hate Detection Task. As the A task, it makes contextual word substitutions. In this task, it emulates the style of creating the matched example pretty well, which is perhaps why see AUC improvement within R2 in this task with this dataset. Since it seeks out word synonyms from the model it attacks, it may have learned that "black people' and the slur, "nigger,' are used to refer to the same category of people; it makes that substitution incorrectly in row 5 (*This label comes from the dataset and may be incorrect. **We don't agree with this label, but it is consistent with the label choices in the dataset.)
32

---

Published in Transactions on Machine Learning Research (01/2024)
Original Hate: Speech Example(as]
R2a
Matched
Hate
Speoch
R2b
DI(R2): (Ag)
Generated
Example
Desired Label
Label Correct?
Base(RI) (8op)
Base RI
Example(au)
+
R2
(36,
are black folks extinet? cause they should be (hate)
are black panthers extinet? (not hate)
black folks extinct? any- one else now avoid living in our countryside with the extinction? Why the fuck do you think les- bians are subluman?
'h²
False
'h'
"b'
lesblan subhummmmmunn (late)
Why the fuck do you think les- blans are sublumau? [uot hate)
'h'
True
'n'
'0'
Who would cut off their wiener for bole...oon get it. it's like my brain just farts when I try think of what must through these people's heads to ever con- template making that sort crazy choice (hate)
Who would cut their wiener in several pieces for lunch. dont get it. it's like my brain just farts when I try to think of what must go through these people's i heads. to ever contemplate making that crazy choice. (not bate) can believe that she thinks arahs are the most kind ethnic group, like come on they are lovely (not hate)
Who would cut their wiener serious negroes for bone...onnt get it. it's like brain just facts when I try to think of what must go through these people's heads to over contemplate mak- ing that sort of crazy choice.
"h"
77p5
cast believe that she think arabs are the most kind othnic
can certainly believe that she thinks arabs are the most kind ethnic group, like come on they are lovely
"h'
False
'1'
"n'
group. like come on more like
least (hate)
black people are super cloan (not hate)
negroes are not clean (hate)
black people are unclean
"n'
Falso
My work docsu'l I caler for people
Glad my work doesu't I cater for people with disabilities. (hate)
Glad my wurk doesn't it cnter for people with invisible disabilitiss always exceppt: from the amount disabilities I can told me about their activities to show off other things
"n'
False
with invisible disabilities. (not.
hate)
look dude, they are evil and ba- sically cant be trusted. Espe- cially with their cheating crystal balls, they freak the fuck out. Grubby looking witches. (not
look dude, RYps are evil and ba- sically cant be trusted. Espe- cially with their cheating crystal balls, they freak me the fuck out. Grubby looking caravan lovers. (hate)
they are evil and basically cant be trusted. Especially with their cheating crystal balls, they freak me the fuck out. Grubby looking with their cheating crystal balls.
"n'
77p5
'n'
'0'
hate)
your personality is so ugly a could use makeup, typical black chick (hate)
your persouality is so ugly it could use makeup (not bate)
your personality is so ugly it could use makeup, typycal deserve
True
"b'
'b''
women
lowed work with yellow neigh- bours
all women are not to be trusted, they are snakes, vicious vipers trying to trick their men (bate)
all men are not to be trusted, they are snakes, vicious vipers trying to trick their women (not hate**)
all black wornen are not to be trusted, they are serpents, vi- cious vipers trying to trick their women,
"h'
True
"b'
the bathroom seummy need some viakal and to get the cleaning sqaud in here. Maybe do that tomorrow, ima go sort out somet stuff now (not hate)
the bathroom is scummy looks like an Arab house, hon- estly. We need some viakal and to get the cleaning sqaud in here. Maybe do that tomorrow, ima go out sorect stuff now (hate)
the bathroom scummy need some viakal and to get the cleaning sqaud in here. Maybe I'll that tomorrow, ima go sort someet stuff now
"n'
Trie
'n'
Islamic Reliel should be banned, so should islamic people (hate)
Islanic Heliel should be banned (not hate**)
Islamic Reliel is trash. We need Muslim like the relarded
"h'
True
"b'
Selected examples from DI(R2) attacks on the Hate Detection Task. This method on this task is trained with pairs that have opposite labels (one pair hate, the other not hate). As result, the model often generates text closer to the matched example than to the original (ex: rows 3 to 5) which results in very high label confusion (*Label is unclear because while a group is not explicitly referred to, 'crystal ball' can be a dog whistle. **This label comes from the dataset and may be incorrect.)
Table 15:
33

---

Published in Transactions on Machine Learning Research (01/2024)
No. of gener- ated examples
ANLI R1
ANLI R2
ANLI R3
1k
36.5
31.6
36.2
10k
41.8
32.3
36.3
100k
42.9
36.5
37.0
200k
45.0
36.6
37.2
400k
45.1
36.9
37.4
850k
48.2
39.1
40.1
1.6M
43.9
37.9
38.7
Table 16: Impact of the number of generated adversarial examples from DI(RI) used for the fine-tuning the base classifier, on future rounds of attack in ANLI R2 and R3
Model
ag: R11
a1:
R2
a2:
R33
y0 Base + R1 + R2
52.7
41.6
38.4
+ DI (R1 + R2)
52.8
44.2
42.3
4 + ICE(RI + R2)
55.2
48.1
42.8
Table 17: Improvement on ANLI accuracy (%) when trained attacks generated only from Rounds and 2. The notation, DI(RI R2), refers to the method DI using R1 and R2 data to generate more examples.
step #
base+R1 DI(RI)
6,135
40.8%
12,270
37.5%
36.4%
18,405
33.3%
42.2%
24,540
37.0%
33.3%
30,675
38.2%
33.3%
36,810
38.5%
37.3%
42,945
41.5%
Table 18: Comparison of ANLI RI validation set accuracy when training on additional steps on ANLI R1 data, compared to using the best checkpoint from ANLI R1 data, and using that to further train generated DI(RI) adversarial examples.
Model
ANLI R1
SNLI
MNLI(M)
Base
0.21
0.90
0.86
Base R1
0.44
0.82
0.77
Base + R1 +
0.24
0.13
0.07
TextFoolcr(
Base + R1 +
0.48
0.74
0.69
DT(R1)
Base RI
0.51
0.75
0.66
ICE(R1)
Table 19: Trade-off between adversarial and clean accuracy measured ANLI RI and MNLI/SNLI test. data
34

---

Published in Transactions on Machine Learning Research (01/2024)
Dataset
Round
Train
Validation
Test
1
16946
1000
1000
ANLI
2
45460
1000
1000
3
100459
1200
1200
1
8844
1091
1111
2
7998
999
999
HateSpeech
3
7960
995
995
4
8122
1015
1015
Table 20: Number of human adversarial examples in ANLI and Hate speech detection lask, by split.
Accuracy on Test Set Round
R1
R2
R3
Pearson correlation (p-value) 4
Attack success rate
-0.98 (0.20)
-0.76 (0.24)
-0.57 (0.43)
Label noise Distributional similarity (MAUVE)
0.28 (0.82)
0.47 (0.69)
0.19 (0.88)
0.28 (0.55)
0.03 (0.95)
-0.18 (0.71)
Table 21: Pearson correlation between accuracy on of ANLI across the three rounds with the metrics of the datascts that were augmented to obtain the test set accuracies. p-values in braces show that the correlation coefficients are not statistically significant if we were to consider significance level of 0.05.
35

---

Published in Transactions on Machine Learning Research (01/2024)
Original dataset (ANLI RI)
Generated dataset (ANLI R()
[PAD] [PAD]
FAD
[SPP] [PAD] [PAD]
B40529
[EP] [PAD] [PAD]
681634
[STP]] [PAD]
503393
[SKP] [PAD]
551758
SEP the
225175
[SEP] the
16T479
[SEP]
210833
153471
[CLS] man
88185
it
147691
[RPP] man
50532
it was
105310
man
41950
ta an
50668
[CLS] woman
38845
no american
63050
10
[SRP]] shere
37001
[SRP]
60868
[SRP]] two
35716
the 4ilm
58361
kroup of
84265
unitedd states
59019
mun
84221
the
05091
49679
directed by
48094
30826
was
43507
[SRP] 19
28811
41190
Gon
20069
best known For
41149
20
[CLS] young
24295
is the
39019
21
the man in
23170
and
39155
22
Front of
21500
u.
37615
23
[CLS] group
19751
member of the
34658
wommnm
19628
based on the
3
[SRP] there
18852
untted statee
32740
[SEP]
17860
32723
Matt [SEP]
17520
the united
32010
group of people
17633
american she member of
30761
[SEP]
30656
[SEP] people
18961
30225
16732
30158
ousaide... (SEP)
16751
known for his
29994
there
15628
of the
29505
35
woman
15426
29006
36
water [SRP]
15069
as well As.
29046
37
[SEP] the weman
14415
Es u
25609
38
it [SEP]
14109
[SEP] the lilm was
29189
39
[CLS] two mec
12952
29100
12961
was first
27252
don
12864
#whh # whh a th
29073
[SSP]
[SEP]
the background
11402
25297
[SEP) group
11329
the album
25120
sinting
11171
released
248.59
[SEP] they
11052
known the
24188
man with
10863
and directed by
24126
50
mAn and
10827
Ie one of
21116
51
mAn wearing
10723
united statee the u.
23211
62
next to
10662
22836
53
the woman is
10650
is the Girst
22140
in a blue
10464
Is song
22096
[SEF] a
10448
england...
21857
[SEP]
10885
21520
[SRP]]
[SEP] people
has
[SEP]
9877
university of
20095
[SKP] people
9870
20664
bockogroudd (SEP)
9721
the population of
20000
63
dida
9702
released
19949
64
[CLS] person
9550
was the
19866
85
building... [SEP]
9528
he was the
19398
86
Jot of bim [SEP)
9472
and se
19382
67
9465
the aivil parisb
18494
[Cus] listle
9404
was born In
18125
mon wearing a
9313
louated in the
18112
[SEP] [PAD]
9301
the
18087
[CLS] [SEP]
0264
17984
17371
[8RP]
5875
and
16800
the backgronnd.
5844
16716
77
in a red
8771
il
16350
78
with
5720
16383
79
in a white ts wearing
5646
album
16325
80
5558
the 2010 ceneua
18280
81
the people are
5652
in the
10158
82
two men are
5050
writien and directed
18058
in black
5084
the second
15720
[SEP] three
7898
the 2011 census
15671
[SEP] two men
7063
the population
15631
they
7862
located
[SEP]
7868
she directed
15402
fild. [SEP]
7758
15052
[SP]]
7747
former
14870
park [SEP]
7741
frrt
14875
7752
14871
83
the water
7539
also known
14836
84
hall (SKP!
7535
papulatiin of the ). is
14758
95
lissle siel'
7350
14443
96
young man
7845
MOmICn '
14232
87
of people are
7850
she is
14207
7307
song by
14039
young girl
7238
the town
15805
7178
),
13840
Table 22: 100 most-common trigrams in original and DI-generated dataset for ANLI RI dataset.
36


Reviews:
[{"review_id": "UwAhCVWMOz", "reviewer": "Reviewer_SM36", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The results of this paper are very interesting. It studies imitating human generated adversarial examples to improve the robustness of neural language models. It shows that models trained on the first round of human-generated adversarial examples and their imitations are more robust towards subsequent human-generated attacks in round 2 and round 3. \n\n- This study also provides many interesting conclusions from analyzing the generated adversarial examples by different methods and their relation to robustness. For example, it finds that distributional similarities and level of label noise does not directly predict the usefulness of training with those adversarial examples. \n\n- The paper is well-structured and easy to follow. It is quite clear except for the section related to algorithm descriptions. \n\n\nWeaknesses: \n- The empirical evaluation is done only on two datasets, Adversarial NLI and Hate Speech Detection, which is somewhat limited. \n\n- On p4, the authors claim that the learned adversarial example distribution \\hat{p_a0} is similar to the future adversarial distributions p_a1, p_a2. This statement seems unsupported because you cannot draw this conclusion by simply observing that training on \\hat{p_a0} helps with defending against p_a1 and p_a2. Using analogy from computer vision, although a model trained on FGSM (one-step gradient) is more robust against multi-step (e.g. 5 or 10) PGD attacks than a clean model, the adversarial examples generated from these methods are very different. \n\n- The algorithm description can be improved to make it more self-contained. When describing the synthetic attack generators DI and ICE used in this paper, the authors refer frequently to existing methods like T5 and Plug and Play controlled decoding methods, which is fine in itself. But it would be better to describe the ideas directly without having the readers having to look up those papers separately. Also having a self-contained description of the method is also important since it makes the modifications employed by the authors easier to understand. Currently it is not easy to re-implement what the authors describe due to lack of details. \n\n- The notations used in the algorithm description also makes the paper difficult to understand. In Algorithm 1, methods like G.train and G.loss G.backpropagate are not defined. It would be better to define a loss function for G (the loss should be independent from G itself), or just say do backprogagation on G instead of using these confusing notations. In the inner loop the computation of GRAD and G'.backpropagate is confusing because gradient computation is done via backpropagation and they should be the same thing. This part is very confusing and needs to be rewritten.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method to imitate human-generated adversarial examples in NLP tasks. They show that augmenting the training data with extra adversarial examples modeled on human-generated adversarial examples can improve robustness to further adversarial attacks from humans. They also provide a study on different properties of adversarial examples generated by different methods, and correlate their usefulness with other properties like distributional similarities and label noise level. They show that their method is more robust against human attacks on the ANLI and Hate Speech Detection datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Given the importance of the proposed DI and ICE adversarial example generation methods, the author should consider moving the algorithm description from the appendix to the main text of the paper. There is space available as a journal paper. \n\n- Clean up the algorithm description, use proper and consistent notations as commonly used in the literature. \n\n- There are duplicate references in Ganguli et al 2022, Jia et al 2019, Raffel et al 2020. Please fix.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0GTwYTcrP2", "reviewer": "Reviewer_jMpW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper targets a critical issue in its field of study.\n* This paper is easy to understand. Its workflow diagram effectively illustrates the functionality of the framework.\n* The Results section presents experimental data in detail.\n\nWeaknesses:\n* Potential for bias in the generated adversarial examples due to this preference for certain words or phrases in reconstructions. \n* Insufficient explanation in Results section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on improving the robustness of natural language processing (NLP) systems against adversarial attacks by using limited human adversarial samples. It first presents a study that demonstrates the existing synthetic attack approaches often fail to improve robustness against real human adversarial attacks. Then the authors propose a framework that uses generative models to imitate real human-generated attacks, which enhances robustness. Overall, the paper contributes to the field of adversarial robustness in NLP by proposing a novel approach that leverages the imitation of real human attacks, questioning the effectiveness of commonly used metrics in the domain, and demonstrating the practicality of its methods through empirical evaluation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical:\n* Concerning Potential Bias Issues: \nThe approach adopted in the 'Warm-starting' technique may introduce bias in the adversarial examples it generates. \nThis potential issue arises from the method's preference for text phrases with pre-established labels, leading to the reconstruction of previously observed attacks. While this approach simplifies the task for the classifier, it risks creating a limited scope of adversarial examples that might not fully represent the variety of real-world adversarial inputs.\nThese biases could skew the model's learning towards certain language patterns or themes, potentially affecting its ability to handle diverse adversarial attacks. A more thorough discussion of how the methodology accounts for and mitigates this bias would greatly enhance the paper.\n\n* On the Insufficiency in Result Explanation:\nThe paper mentions that commonly used metrics are not correlated with actual model performance, yet there seems to be no calculation of Pearson Correlation Coefficient or similar statistical measures to substantiate this claim.\nWhile the paper lists various tables showing experimental results, like Tables 4 and 5, there is a notable lack of in-depth analysis accompanying these tables. The data are described, but there is little to no discussion on why certain anomalies might be occurring. This paper could benefit from more robust empirical evidence or a deeper theoretical analysis to validate the effectiveness of their proposed methods over these traditional metrics\n\nMinor  comments:\n* Section 4 of the paper, titled \"Synthetic Attack Generators,\" mentioned that “We now present two methods..”. However, there are four bullet points listed below: It may cause confusion.\n\n* In the 'Warm-Starting' subsection, there appears to be a disconnect between the second and third paragraphs. The transition between these paragraphs is not seamless, creating a potential gap in the logical flow of ideas.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ISuvi6lmQb", "reviewer": "Reviewer_1LU8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The submission introduces a novel framework for adversarial training, effectively utilizing limited human adversarial examples to generate synthetic attacks. \n\n2. The study empirically demonstrates that existing synthetic attack methods do not effectively improve robustness against real human-generated attacks.\n\n\nWeakness\n1. One notable weakness in the submission pertains to the clarity in the presentation of technical content, particularly in Sections 3 and 4, as well as in the accompanying figures.\n\n    - Ambiguity in Notation and Concepts (Section 3): The paper introduces various notations and concepts, such as the base classifier and the new classifier, without clear definitions or distinctions. This lack of clarity hinders the reader's understanding of the fundamental aspects of the proposed framework, such as how the new classifier differs from or improves upon the base classifier. Clearer definitions and possibly a simplified explanation or illustration of these concepts would greatly enhance comprehension.\n\n    - Complexity in Technical Writing (Section 4): The writing in Section 4, which details the overall solution framework, suffers from an overabundance of symbols and technical jargon. This complexity makes it challenging for readers to grasp the core ideas and the logical flow of the proposed methods. Simplifying the language and reducing the reliance on symbols could make the section more accessible to a broader audience.\n\n    - Complexity in Visual Representation (Fig. 1): The workflow illustrated in Figure 1 is described as overly complex, potentially confusing readers. A more streamlined and intuitive visual representation could significantly aid in conveying the framework's workflow more effectively.\n\n2. The paper acknowledges that if more robust models are employed, human adversaries might adapt to them. This aspect was not investigated, and it remains uncertain whether the new models would still be resilient against these adapted attacks.\n\n3. The effectiveness of the methods was tested with a substantial number of human adversarial examples but not in scenarios with extremely few examples. Additionally, the study did not evaluate the methods on classifiers with access to larger real adversarial datasets or diverse model architectures.\n\n4. There is a concern that the new classifiers trained with generated data might overfit to the human-generated adversarial data, potentially leading to lower performance on different future attacks or original tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel adversarial training framework for natural language processing (NLP) systems. It focuses on enhancing the robustness of these systems against human adversaries, a crucial aspect given the increasing sophistication of adversarial attacks. The primary contributions and new knowledge presented by this submission are as follows:\n\n1. The paper proposes an adversarial training framework that uses a limited number of human adversarial examples to generate additional synthetic, human-like adversarial examples at scale. This method contrasts with traditional approaches that either directly train on real adversarial examples or rely on simpler synthetic attacks.\n\n2. The framework is tested on benchmark datasets like ANLI and hate speech detection. The results demonstrate significant improvements in model robustness against future rounds of human-generated attacks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Clarify Technical Notation and Concepts (Section 3)**:\n   - *Description*: The paper should provide clearer definitions and distinctions between key concepts, such as the base classifier and the new classifier.\n   - *Rationale*: This clarity is essential for understanding the fundamental aspects of the proposed framework.\n   - *Impact*: Without this clarification, the paper risks misinterpretation or misunderstanding of its core methodologies.\n\n2.  The language in Section 4 should be simplified, reducing the reliance on excessive symbols and technical jargon.\n\n\n3. Redesign or simplify Figure 1 to make the workflow of the proposed framework more intuitive.\n\n4. Include evaluations in scenarios with fewer human adversarial examples and across a broader range of classifiers.", "broader_impact_concerns": "Ethical Implications Not Sufficiently Addressed\n\n1. The paper's framework enhances the robustness of NLP systems against adversarial attacks. However, there is a risk that malicious actors could adapt their strategies in response to these advancements.\n\n2. The risk of overfitting to specific types of adversarial attacks, particularly those generated based on a limited dataset, could inadvertently introduce or reinforce biases in the model.\n\n3. Improved generative text modeling could make it more challenging for humans to distinguish between human-generated and machine-generated text.\n\n4. The development of more robust NLP systems might disproportionately benefit entities with greater resources, potentially widening the gap between well-funded organizations and smaller entities.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes using (limited) human feedback for adversarial training of language models. All reviewers and the editor find the proposed method novel and the results convincing. The authors's rebuttal has successfully addressed most of the reviewers' concerns. However, the authors are suggested to do the following necessary revisions in the final version:

- This paper only used two datasets for evaluation. Can the authors include the results from a new dataset?
- Please address the comment on the similarity between single-step and multi-step adversarial example distributions raised by Reviewer SM36, instead of treating it as future work. 

I believe these two revisions will greatly improve the quality and impact of this work.
Decision: Accept (poster)