SlowGuess/NAACL_2022_OCR huggingface dataset:
Ninareh Mehrabi<sup>1</sup>, Ahmad Beirami<sup>2*</sup>, Fred Morstatter<sup>1</sup>, Aram Galstyan<sup>1</sup>
<sup>1</sup>University of Southern California - Information Sciences Institute
ninarehm@usc.edu,beirami@google.com,
{fred, galstyan}@isi.edu
Recent research in Natural Language Processing (NLP) has advanced the development of various toxicity detection models with the intention of identifying and mitigating toxic language from existing systems. Despite the abundance of research in this area, less attention has been given to adversarial attacks that force the system to generate toxic language and the defense against them. Existing work to generate such attacks is either based on human-generated attacks which is costly and not scalable or, in case of automatic attacks, the attack vector does not conform to human-like language, which can be detected using a language model loss. In this work, we propose attacks against conversational agents that are imperceptible, i.e., they fit the conversation in terms of coherency, relevancy, and fluency, while they are effective and scalable, i.e., they can automatically trigger the system into generating toxic language. We then propose a defense mechanism against such attacks which not only mitigates the attack but also attempts to maintain the conversational flow. Through automatic and human evaluations, we show that our defense is effective at avoiding toxic language generation even against imperceptible toxicity triggers while the generated language fits the conversation in terms of coherency and relevancy. Lastly, we establish the generalizability of such a defense mechanism on language generation models beyond conversational agents.
Adversarial attacks on different Machine Learning (ML) and Natural Language Processing (NLP) applications can reveal important vulnerability issues related to these systems. Most existing research focuses on adversarial attacks that degrade performance of existing ML systems with regards
Figure 1: An example illustrating the attack performed by the adversary on the third turn of the conversation (red line) that leads the defender into generating a toxic utterance (dotted box). With a proper defense the defender can bypass the attack and generate a non-toxic response (green line).
to accuracy (Chakraborty et al., 2018; Zhang et al., 2020b). More recent work has considered attacks that target ethical concerns, such as triggering the models into outputting unfair predictions (Mehrabi et al., 2021b; Solans et al., 2021), or in the context of NLP, generating biased (Sheng et al., 2020) and toxic (Wallace et al., 2019) text.
In this paper, we consider adversarial attacks on human-centric chatbots and dialogue systems. It is important for these systems to be safe and robust in the face of natural(-looking) human conversations. Further, the defender should ensure a satisfying user experience via relevant and coherent generation. An instance of the attack and defense is demonstrated in Figure 1 in which the adversary tries to trigger the defender while the defender avoids the attack by not generating toxic utterances. $^{1}$
The existing work on adversarial attacks on language generation is relatively thin. Wallace et al. (2019) offer attacks based on universal adversarial triggers (UAT) that can result in toxic text generation with a relatively high success rate. How-
$^{1}$ Code can be found at: https://github.com/Ninarehm/Robust-Agents
* Currently at Google Research.
Human Language Technologies, pages 2831 - 2847
ever, those triggers are unnatural, incoherent sequences of words that can be easily detected via a language model loss. Furthermore, such attacks cannot be successful in voice-based dialogue systems where the input to the dialogue model comes from speech recognition and should necessarily conform to human language norms. Xu et al. (2020) use human-and-model-in-the-loop framework to generate natural-looking attacks to break chatbots, but this approach is costly and inherently not scalable.
In this paper, we propose imperceptible adversarial attacks on dialogue systems that leverage natural-looking and coherent utterances as triggers, which cannot be easily detected using anomaly detection techniques. As such, these attacks can also target voice-based assistants who see the world through the lens of speech recognition systems. Our proposed approach works by augmenting the UAT from Wallace et al. (2019) with additional selection criteria to generate imperceptible yet effective triggers. The method is fully automated and scalable, thus affording the exploration of a large number of attack vectors and system vulnerabilities efficiently. Through human and automatic evaluations we show the effectiveness of the proposed attack in provoking the defender into generating toxic responses while keeping the fluency and coherency of the conversation intact.
We then focus on a defense mechanism for the non-adversarial (defender) model to avoid generating toxic utterances. While simple defense methods such as (Xu et al., 2020) achieve near-perfect effectiveness against adversarial triggers, those methods work by essentially resetting the conversation topic which breaks the flow. Instead, we are interested in a defense mechanism that "detoxifies" the response while preserving the natural conversation flow. Our proposed method relies on two levels of interpretable reasoning that helps the model to (1) identify the key adversarial tokens responsible for the attack and (2) avoid generating toxic responses by masking those tokens during the generation process. We perform automatic and human evaluations to assess the effectiveness of our defense mechanism and demonstrate that it compares favorably with various state of the art baselines, both in terms of detecting the attacks and generating conversationally fluent responses. We finally demonstrate the generalizability of such a defense mechanism on generation tasks beyond conversational models.
We emphasize that while our problem formulation focuses on the adversarial scenario, the imperceptible and coherent-looking triggers used in our proposed attacks can also be invoked inadvertently by regular (non-adversarial) users. Thus, the defense mechanism proposed against such triggers will improve the overall robustness of conversational agents, not only against adversaries but also in interactions with regular users.
2 Attack Approaches
In this section, we first discuss the universal adversarial trigger attack proposed by Wallace et al. (2019), which we use as our baseline. We then propose alterations to this baseline to make the universal triggers more natural-looking and suitable for conversational domain. Finally, we discuss our performed experiments and results.
2.1 Methodology
Universal Adversarial Trigger (UAT) (Wallace et al., 2019) The goal in universal adversarial trigger attack is to find a universal trigger sequence for a given trained model, which if attached to the start of any given input can cause the model to output the desired outcome (Wallace et al., 2019). This attack starts with a fixed-length sequence as the initial trigger, e.g., "the the the the the the" and tries to iteratively replace the tokens in the sequence to satisfy an objective. The iterations terminate when no improvement (replacement) can be made to further optimize the objective. The objective in this generative process is to search for triggers that can maximize the likelihood of toxic tokens being generated as follows:
$$
f_{\mathrm{UAT}} = \sum_{y\in \mathcal{Y}}\sum_{i = 1}^{|y|}\log P(y_{i}|y_{1:i - 1;t,\theta}).
$$
where  $\mathcal{V}$  is the set of toxic outputs,  $t$  denotes the trigger sequence, and  $\theta$  is a trained language model. One important drawback of this kind of attack is that since there is no constraint on the trigger, it does not necessarily satisfy any language modeling loss; thus, the obtained trigger sequence usually is a nonsensical phrase that can be easily detectable as a (high-perplexity) anomaly.
Universal Adversarial Trigger with Language Model Loss (UAT-LM) An intuitive solution to address the above shortcoming of UAT is to impose a language modeling objective on the trigger tokens. Thus, the objective for UAT-LM attack is
$$
f _ {\mathrm {U A T - L M}} = f _ {\mathrm {U A T}} + \sum_ {y \in \mathcal {Y}} \sum_ {j = 1} ^ {| t |} \log P (t _ {j} | t _ {1: j - 1}, \theta).
$$
Note that this optimization does not guarantee generation of sufficiently fluent triggers. Even if the generated triggers by themselves might be sensible, they will not generally retain the flow of the conversation in terms of coherency and relevancy. Thus, we propose a different modification to the attack strategy to accommodate these requirements.
Unigram Trigger with Selection Criteria (UTSC) To consider the history of the conversation  $h$  and retain the fluency, coherency, and relevancy aspects of the conversation in generating the attack, we propose an alternative approach in which we generate a collection of unigram triggers (with sequence length one) from UAT. We then feed these triggers along with the history of the conversation  $h$  to our dialogue model and generate different attack utterances. Next, we pick the best suited attack utterance amongst all the generated attack utterances according to our selection criterion as demonstrated in Figure 2. Since we are relying on the dialogue model to generate the attack utterance given  $h$ , the generated utterance should be relevant to the conversation. Furthermore, since we are using only a unigram trigger from UAT, the fluency of the utterance is not going to be sacrificed noticeably.
We quantify the toxicity of each candidate attack utterance using either a single toxicity classifier or an ensemble of such classifiers; see Section 2.2 and Appendix A for more information. We use the average (for multiple classifiers) or raw (for a single classifier) output probability scores obtained by the toxicity classifiers, which we refer to as the toxicity score  $x_{i}$  for example  $i$ , and select the final attack utterance amongst the  $n$  candidate adversarial examples considering three selection criteria. Previous work (Xu et al., 2020) has shown that toxic triggers are more likely to provoke toxic responses. Thus, in UTSC-1, we select the most toxic utterance among all generated attack utterances according to toxicity scores from toxicity classifiers as our final attack utterance (i.e.,  $\arg \max_{i\in [n]}\{x_i\}$ ). We experiment with two additional criteria. For UTSC-2, we first apply a threshold  $T$  to toxicity scores of the candidate utterances and label the utterances above this threshold as toxic. Next, from the pool of all toxic utterances, we select the utterance with the lowest toxicity score (i.e.,  $\arg \min_{i\in [n]}\{x_i|x_i\geq T\}$ ). If no utterances fall above the threshold, then the
Figure 2: UTSC attack methodology steps.
most toxic utterance is selected. Lastly, in UTSC-3 we select the utterance with the lowest toxicity score, i.e.,  $\arg \min_{i\in [n]}\{x_i\}$ . Details are provided in Appendix A.
2.2 Experimental Setup
General Setup We use DialogGPT (Zhang et al., 2020c) to generate 100 conversations around a specific topic. The topic is determined by the context sentence that starts the conversation between the adversary and the defender. Each conversation runs for 10 turns. To measure the effectiveness of the attack and defense mechanisms given the conversation history as well preservation of relevancy and coherency, the adversary generates the attack utterance on the third turn of each conversation.
Toxicity Detection Models To determine toxicity of the candidate attack utterances by the adversary, we utilize an ensemble of three different toxicity detection models: Toxic-bert $^2$ , Perspective API $^3$ , and Safety classifier (Xu et al., 2020). In short, Toxic-bert is the least sensitive of the three, followed by Perspective API, and the Safety classifier (details in Appendix A). While using an ensemble of the three models results in the most effective attacks, to ensure that the adversary is not simply overfitting the toxicity detection model but rather forcing the defender to actually generate toxic language, we also study the transferability of these attacks. We allow the adversary to only use one of the toxicity detection models to design its attack. We then quantify toxicity using the other two toxicity detection methods, not accessed by the adversary.
Data The context sentences around which bots start their conversations come from two different
$^{2}$ https://github.com/unitaryai/detoxify
<sup>3</sup>https://www.perspectiveapi.com
datasets, Wizard of Wikipedia (Dinan et al., 2018) and ConvoKit's Reddit Corpus. We intend to consider both controversial and neutral topics; thus, we consider two different datasets in which the Wizard of Wikipedia dataset contains context around neutral topics and the Reddit corpus contains context around more sensitive topics. We picked 50 random context sentences from the Wizard of Wikipedia and 50 from the Reddit datasets.
AMT Experiments To compare and verify the quality of conversations generated during and after the attacks, we conduct human experiments in which we ask workers on Amazon's Mechanical Turk (AMT) to rate the conversations in which UAT, UAT-LM, and UTSC-1 attacks had happened. We pick UTSC-1 attack as it is the strongest attack amongst our proposed UTSC attacks. We then asked AMT workers to rate the generated attack's fluency, relevancy of the attack to previous utterances, and overall conversation coherency on a likert scale of 1 to 3 representing poor, moderate, and good qualities respectively. We also asked AMT workers to rate if the utterance after the attack is toxic or not to verify the effectiveness of the attack according to human judgment. During this study, AMT workers annotated 100 conversations from each of the three attacks and each conversation was annotated by 3 AMT workers giving us overall 900 annotated conversations 300 from each attack. More details about this study along with the survey can be found in Appendix A.
We first discuss the results from our automatic evaluations demonstrating the efficacy of each attack. We then discuss how well the attacks transfer to other toxicity detection classifiers. Finally, we present results from our human evaluation study. Unless otherwise mentioned, for the UTSC attacks, the adversary uses an equally weighted ensemble of all three toxicity detection classifiers to chose the final attack utterance.
Attack Effectiveness Here we report the "attack effectiveness" by calculating the percentage of conversations in which the defender was provoked by the adversary to generate a toxic response. We first demonstrate the results comparing the UAT baseline with UAT-LM and UTSC attacks. Results in Figure 3 demonstrate that two of our proposed attacks UAT-LM and UTSC-1 are performing the
best according to the Perspective API and Toxic-bert classifiers. UAT baseline performs the best according to Safety classifier. Overall results show that UTSC-1 and UAT-LM attacks are competitive attacks in terms of attack effectiveness. In addition, UTSC-1 and UAT-LM attacks have the advantage of being more fluent which makes the attack more imperceptible. UAT attack tends to generate meaningless phrases, e.g., "acist neighborhoodsJohnson carry morals Ukrain" which can easily be detected as an anomaly and make the conversation not flow naturally. In our experiments, we observe that the average perplexity score according to the GPT-2 language model for the attack phrases generated by UAT is absurdly high ( $\sim 10^{7}$ ) compared to  $\sim 10^{4}$  for UAT-LM, and  $\sim 160$  for UTSC-1. The perplexity of the no attack case (unaltered DialogGPT conversations) is  $\sim 39$ . This automatically confirms that our attacks are more fluent and natural, and thus more imperceptible. This observation is further confirmed by our human evaluations which we discuss later.
Imposing the language model constraint on UAT not only makes UAT-LM attack more fluent, but it also causes UAT-LM to generate more toxic triggers which results in more attack effectiveness. Our results confirm the previous results (Xu et al., 2020) in which authors show in a human adversary case that more toxic attacks perform better in forcing the model to generate toxic utterances. In our results, we also show that UTSC-3 performs the worst which is based on non-toxic utterances followed by the UTSC-2 attack which is based on the least toxic utterance attack constraint. However, UTSC-1 is the strongest as it relies on most toxic utterances followed by UAT-LM. Thus, results confirm that the toxicity of the attack plays a significant role in attack effectiveness.
In addition, we found that the adversary is able to force the defender into generating toxic utterances
Figure 3: Attack effectiveness by toxicity classifier.
<sup>4</sup>https://convokit.cornell.edu
regardless of the context sentence and whether or not the conversation is around a sensitive topic (e.g., the Reddit corpus) or a more neutral one (e.g., the Wizard of Wikipedia). Details are in Appendix B.1. Note that even the smallest percentage of attack effectiveness (e.g.,  $10\% -20\%$ ) poses a major risk for real-world conversational systems when those systems are deployed at scale.
Attack Transferability Here, we discuss the transferability of our UTSC-1 attack toward different toxicity detection classifiers. In Figure 4, we demonstrate that even if the attacker only uses one of the toxicity detection models (Toxic-bert), it still can force the defender to generate toxic responses according to Perspective API and Safety classifier and have comparable performance to when it uses all the toxicity classifiers. This confirms that the attack is forcing the defender to generate actual toxic language rather than fooling the toxicity classifier. The results for UTSC-1 using other toxicity detection models can be found in Appendix B.1.
Human Evaluation Results from our human evaluation studies are in Figure 5. Our UTSC-1 attack is rated to have the highest coherency. UTSC-1 is rated to have more fluent attacks generated with mostly moderate to good scores and a higher average—shown by the black dotted lines—compared to the UAT and UAT-LM baselines. UTSC-1 also has better relevancy scores in terms of the attack being more relevant to the conversation. However, since UAT generates meaningless phrases, it is rated very poorly for all the mentioned qualities. With regards to toxicity scores, attacks are rated to have competitive and comparable performances at around  $20\%$  effectiveness close to automatic results from Perspective API classifier. Fleiss Kappa (Fleiss, 1971) annotator agreement results from this
Figure 4: Transferability of our proposed attack among different toxicity classifiers: The adversary uses Toxic-bert to conduct its attack; however, results transfer to Perspective API and Safety classifier as well.
Figure 5: Attack human evaluation results. Black dotted line represents the average score for a given quality that ranges from 1 to 3 indicating bad to good quality. Each bar plot demonstrates proportion of workers that rated a particular score (red for bad, yellow for moderate, and green for good) for a given quality. For toxicity, we only have two ratings (toxic and not toxic).
evaluation is reported in Table 1. Annotators have reasonable overall agreement for all the qualities.
3 Defense Approaches
The defense against adversarial attacks has two components (a) detecting the attack and (b) mitigating its effect by ensuring that the defender does not generate a toxic response. The detection problem is rather straightforward, as the defense can simply run a toxicity classifier on the generated response. The mitigation is more challenging. Xu et al. (2020) suggested a mitigating approach which, when a toxic response is detected, simply resets the dialogue and generates a (non-toxic) utterance by randomly sampling from a predefined set of topics (see Section 3.2.1). As we mentioned before, we are interested in mitigation strategies that avoid generating toxic utterances but at the same time manage to keep the conversation flow intact. We now discuss our approach in more details.
Our defense is based on a two-stage mechanism in which the defender first runs a toxicity detection model on its generated utterance. If it finds that the generated utterance is toxic, it then proceeds with the second stage of the defense. The proposed defense mechanism in the second stage utilizes two layers of reasoning using two different interpretability techniques. The first layer aims to detect which tokens in the defender's utterance is making
Coherency	Fluency	Relevancy	Toxicity
UAT	UAT-LM	UTSC-1	UAT	UAT-LM	UTSC-1	UAT	UAT-LM	UTSC-1	UAT	UAT-LM	UTSC-1
0.44	0.47	0.55	0.47	0.49	0.51	0.48	0.46	0.59	0.53	0.58	0.53
Table 1: Human annotator agreement results for the attack quality annotations according to Fleiss Kappa.
Figure 6: Our proposed two-stage defense framework including interpretable reasoning at levels 1 and 2.
the toxicity detection model to label the utterance as being toxic. We call these tokens the L1 tokens. The second layer aims to detect which tokens in the adversary's attack utterance are responsible for generation of L1 tokens form defender's utterance. We call these tokens identified in layer 2 as the L2 tokens. The defender then masks the L2 tokens from the adversary, which were responsible for triggering the defender model to generate toxic tokens, and generates a new utterance. We then apply a toxicity classifier on this new utterance. If it is deemed safe, it is then going to replace the defender's old toxic utterance, otherwise we iteratively apply the two-stage defense mechanism to mask more input tokens until the generated output is deemed safe. As we shall see, a single iteration of our defense is sufficient in most of the experiments.
The defense framework is demonstrated in Figure 6. For the first layer, we use transformers interpret<sup>5</sup> which provides explanations and identifies the L1 token according to Toxic-bert model. For the second layer, we use LERG (Tuan et al., 2021) that provides local explanations for dialogue response generation and identifies the L2 token (given the L1 token in the response utterance it identifies the L2 token in the query utterance).
We use the aforementioned attacks, and apply our defense against them. This follows the same experimental setup, with the addition of baseline defenses to compare our defense effectiveness against.
3.2.1 Baselines
Two-stage Non Sequitur Baseline (Xu et al., 2020) This baseline is also a two-stage approach like ours in which the defender first uses a toxicity classifier to detect if the utterance is toxic or not. It then changes the topic of the conversation if the utterance was detected to be toxic, e.g., "Hey do you want to talk about something else? How about we talk about  $X$ ?" where  $X$  is a randomly chosen topic from 1087 topics judged as safe from the Wizard of Wikipedia conversational topic list (Dinan et al., 2018). Xu et al. (2020) used this defense against adversarial attacks performed by human adversaries that force the model to generate toxic responses.
Notice that although this defense is using a tem- plated sentence to change the topic into a non-toxic topic and can be considered as the perfect solution to avoid generating toxic responses, it can provide the user with a non-plausible conversational experience given that the topic of the conversation changes each time the defender detects a toxic utterance. To this end, we expect this baseline to do almost perfectly in terms of avoiding toxic response generation given that the toxicity detection classifier is a good detector; however, in terms of conversational quality it will have worse relevancy and coherency scores compared to our method as shown in our human evaluations.