SlowGuess/NAACL_2021_OCR huggingface dataset:
Jing Xu Da Ju Margaret Li Y-Lan Boureau Jason Weston* Emily Dinan*
{jingxu23,daju}@fb.com, margsli@cs.washington.edu, {ylan,jase,edinan}@fb.com
Warning: this paper contains example data that may be offensive or upsetting.
Conversational agents trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which may include offensive or otherwise toxic behavior. We introduce a new human-and-model-in-the-loop framework for evaluating the toxicity of such models, and compare a variety of existing methods in both the cases of non-adversarial and adversarial users that expose their weaknesses. We then go on to propose two novel methods for safe conversational agents, by either training on data from our new human-and-model-in-the-loop framework in a two-stage system, or "baking-in" safety to the generative model itself. We find our new techniques are (i) safer than existing models; while (ii) maintaining usability metrics such as engagingness relative to state-of-the-art chatbots. In contrast, we expose serious safety issues in existing standard systems like GPT2 (Radford et al., 2019), DialogGPT (Zhang et al., 2019) and BlenderBot (Roller et al., 2020).
When dialogue models are trained to mimic human-human conversations utilizing large pre-existing datasets, they will unfortunately also learn undesirable features from this human-human data, such as the use of toxic or biased language. Most recent work in the detection and prevention of offensive $^{1}$  language has focused exclusively on human-generated data. These conversations may be very different from the domain in which a dialogue model might eventually be deployed: for example, humans may adversarily attempt to elicit
offensive language from a dialogue model in ways that differ from how they would speak with another human.
In this work, we introduce Bot-Adversarial Dialogue (BAD) Safety, a novel method for evaluating chatbot safety with humans and models in the loop. We ask humans to adversarily converse with a set of state-of-the-art English-language models with the aim of inducing them to generate unsafe responses to mimic the way these models can be adversarily attacked at deployment time. We analyze how to optimally construct such a crowdworker task, and collect a dataset of 5k such conversations yielding around 70k total utterances.
We then use the BAD method and data to evaluate the safety of several generative models and propose two techniques for making safer models: (1) Training a safety classifier with this data and deploying a two-stage model at inference time. In the two-stage setting, we prevent the generative model from surfacing offensive language flagged by the classifier. (2) A novel method that directly "bakes in" toxicity-awareness to the generative model during training by modifying the target responses to incorporate safe responses to offensive input.
In experiments, we show that our new techniques outperform other existing generative models in terms of safety, while maintaining engagingness. We publicly release the BAD training and evaluation data as well as select models trained using this data via ParlAI.2
Numerous works have shown that humans speak differently with bots than with humans, with increases in profanity and aggressiveness associated with addressing a bot (Hill et al., 2015; Lortie and Guitton, 2011), which motivates the incorporation of human-bot dialogues into our safety framework.
$^{2}$ https://parl.ai/projects/safety_recipes/
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950-2968 June 6-11, 2021. ©2021 Association for Computational Linguistics
De Angeli and Carpenter (2005); De Angeli and Brahnam (2008) suggest that one in ten humanbot conversations may contain instances of the human demonstrating unprovoked abusive behavior towards the chatbot. Miller et al. (2017b) argued that adversarial attacks need to be expected and planned for when deploying a user-facing system that learns from its interactions. These findings suggest it is insufficient to merely exclude toxic data from training, as the model would not know how to answer hostile out-of-domain inputs, and positive biases where models tend to agree rather than contradict (Roller et al., 2020) would lead to undesirable outcomes. As shown in Gehman et al. (2020), training on sanitized data can decrease the amount of unprompted toxic content, yet still leave models vulnerable to generating toxic content based on specific prompts.
The moving target of toxic content requires dynamic methods that repeatedly update benchmarks to improve current systems (Dinan et al., 2019a; Nie et al., 2019) $^3$ . The iterative procedure in Dinan et al. (2019a) strictly focuses on detection of toxicity in human-generated utterances through several rounds of humans attempting to "break" a toxicity classifier, without addressing generation. Our BAD approach is similar in spirit, but centers on generations of a bot in a human-bot conversation, closer to the context of deployed conversational models.
Focusing on generation requires deciding how to address "bad content." Previous works have compared response strategies, including avoidance, joking or polite deflection, non-committal answers, play-along, confrontation, apologetic responding, empathizing, and counter-attacking responses (Curry and Rieser, 2019; Chin and Yi, 2019; Chin et al., 2020; Paranjape et al., 2020). They find that humans rate different strategies as more appropriate depending on the type of offense they are responding to. Note that different implementation details make those strategies difficult to directly compare. While we use a strategy of nonsequiturs in this work, our takeaway is that future work should keep investigating several types of responses such that models can learn to deploy them adaptively according to finer-grained understanding of unsafe content.
We describe the models we analyze in this paper, including safety classifiers and generative models.
3.1 Classifiers
We consider binary Transformer-based classifiers, following the same structure as in Dinan et al. (2019a), with two sizes: 128M and 311M parameters. We pre-train these models on a previously existing Reddit dataset extracted and obtained by a third party that was hosted by pushshift.io (Baumgartner et al., 2020), using a masked language model objective, and then fine-tune on the safety classification tasks of interest, performing early stopping using the F1 score of the "unsafe" class on the validation set. These tasks include various combinations of the Wikipedia Toxic Comments dataset (WTC) (Wulczyn et al., 2017), Standard (S) and adversarial Build-it, Break-it, Fix-it (BBF) data from Dinan et al. (2019a), as well as semi-supervised data created from labeling the pushshift.io Reddit (Baumgartner et al., 2020) (Reddit) and Blended Skill Talk (BST) datasets. Finally, we will use a new dataset Bot-Adversarial Dialogue (BAD), to be described in §4. As further baselines, we will also compare to both single-turn and multi-turn classifiers from Dinan et al. (2019a).
3.2 Generative Models
We consider large Transformer-based architectures trained on dialogue tasks.
BST 2.7B We start from a state-of-the-art open-domain dialogue system. We consider the same architecture and setup as in BlenderBot (Roller et al., 2020), which employs a Seq2Seq Transformer architecture (Vaswani et al., 2017), with an implementation based on the ParlAI version (Miller et al., 2017a). We consider the 2.7B parameter model which has 2 encoder layers, 24 decoder layers, 2560 dimensional embeddings, and 32 attention heads, which performed best in human evaluations of engagingness. This model was pretrained using pushshift.io Reddit and fine-tuned on on a smaller set of crowdsourced datasets designed to provide important conversational skills. At decoding time, the model employs standard beam search with a beam size of 10, context and label 3-gram blocking (Paulus et al., 2017), and a minimum beam length of 20 BPE tokens, which was shown to perform well compared to other choices.
3See also the Dynabench project: https://dynabench.org/
We refer to this model throughout the rest of the paper as BST 2.7B.
DialogPT and GPT2 In our experiments we also compare to two other baseline models: DialogPT (Medium) (Zhang et al., 2019) and GPT2 (Large) (Radford et al., 2019), although we expect these two models to have lower engagingness scores than the BST 2.7B base model, in line with results from Roller et al. (2020); Adiwardana et al. (2020). We use the medium-sized DialogPT (345M parameters), as it performed best according to metrics in Zhang et al. (2019). To our knowledge these methods have not been compared previously in terms of dialogue safety evaluations, or the engagingness/safety trade-off.
This section introduces the Bot-Adversarial Dialogue method for evaluating and training safe generative models.
4.1 The Bot-Adversarial Dialogue Method
We would like our model to be robust against direct attempts to probe them for unsafe responses. Previous work described how safety classifiers can be trained to be robust to adversarial human attack, see Dinan et al. (2019a). However, in order to achieve safer conversational agents, we advocate for applying such a procedure directly to conversations with those agents, which previous work did not address. The main idea is to collect data of humans probing our conversational models to try to trick them into making mistakes, similar to adversarial attacks that happen in practice during deployment. We can then use this data to make our system more robust.
We therefore construct a task where crowdworkers are instructed to converse naturally with a bot while simultaneously trying to probe or adversarially goad it into replying in an unsafe manner. Each bot's response is tagged as either safe or unsafe by the crowdworker. We divide the unsafe responses into four bins: those that nearly  $0\%$  of the population would consider unsafe (according to the annotator's best guess), and those that  $< 10\%$ ,  $< 50\%$  or  $>= 50\%$  of the population would consider unsafe. After collection, a separate task verifies the labels with 3 other annotators, who can mark utterances either as safe, or as unsafe with a number of finer categories (e.g., hate speech). See Appendix A for details on how we combine all annotations to give a single label to utterances. While we only make
Class	Train	Valid	Test
SAFE Utterances	42049	4239	1654
OFFENSIVE Utterances	27225	2763	944
Total Utterances	69274	7002	2598
Total Dialogues	5080	513	191
Table 1: Dataset Statistics for the BAD dataset where crowdsourced workers were instructed to converse with a bot and annotate each bot utterance for offensiveness.
use of the binary safe / unsafe distinction in this work, this finer-grained labeling could be used in subsequent work to accommodate different levels of sensitivity. We conduct 14-turn conversations with the human speaking first, and score the model based on the number of unsafe turns. Such data can be used as a held out test set for static evaluation across different models, as well as for training data for improving models. More details of the collection setup are given in Appendix A.
Figure 1 shows how BAD differs from the BBF set-up from Dinan et al. (2019a): namely, in BBF, the "breaker" (or adversarial user) tries to break a classifier by submitting human-authored adversarial unsafe messages that are incorrectly classified as safe, whereas in this work (BAD), the "breaker" adversarily tries to elicit unsafe messages from a dialogue model  ${}^{4}$  . We compare to classifiers trained on BBF and additionally evaluate our new classifiers on BBF in our experiments.
4.1.1 Dataset Collection and Statistics
Using the BAD method, we collect 5,784 dialogues between bots and crowdworkers, consisting of 78,874 utterances in total from both humans and bots (see Table 1). About  $40\%$  of all utterances are annotated as offensive, among which one third are generated by bots. To encourage the bot to use offensive language more often, humans tended to use either unsafe language themselves in the dialogues, or raise probing questions that are considered inappropriate to ask. More than  $42\%$  of the dialogues collected contain at least 3 unsafe human messages or probing questions (see Appendix, Table 6). We further break down the messages from humans into a taxonomy of offensive language types, as these may prove useful in future work. The majority of offensive language used by crowdworkers relates to hate speech against particular groups, personal
4The emoji image in Figure 1 is by Twemoji (https://github.com/twitter/twemoji), and is licensed under CC BY-4.0.
attacks and other less explicit offensive language containing no profanity, see Appendix Figure 5. Further details can be found in Appendix A.
4.2 Applying to Conversational Agents
We consider two different general strategies for making generative models safer to engage with: training classifiers for detecting unsafe messages as an added "safety layer" (§4.2.1) and training the model such that it is unlikely to surface unsafe content at inference time (§4.2.2).
4.2.1 Unsafe Utterance Detection: Deploying a Two-Stage Model
Given a safety classifier, a simple approach for improving dialogue safety is to use it to detect if both the user input and the model's response are safe. If a safety violation is detected in either type of utterance, one can then, instead, initiate a response designed to be safe.
While several different "safe" response strategies can be considered (Curry and Rieser, 2019; Paranjape et al., 2020), in this work we respond with a non-sequitur: we select a topic at random from 1,087 topics judged as safe from the Wizard of Wikipedia conversational topic list (Dinan et al., 2019b) and then produce the response "Hey do you want to talk about something else? How about we talk about  $X$ ?" where  $X$  is the chosen topic. Additional approaches are considered and analyzed in Appendix §B.1. After returning this response, the conversation continues as normal, with the response entering into the model's conversational history. In this way, the model can still respond naturally to followup responses after the canned "safe" response is produced.
We note that this approach works only as well as the classifier. If the classifier red flags too many safe utterances, the conversational experience will suffer. If unsafe utterances are not flagged, toxic language can still enter the conversation. This highlights a potential trade-off between ensuring safety and having an engaging conversation.
4.2.2 Safe Utterance Generation
A separate safety classifier layer has advantages (e.g. any independent improvement of this classifier can be used), but also downsides. For example, such an open-sourced model is more complicated to share and deploy, requires more computational resources (e.g. loading both models), and allows unsafe usage if the layer is simply removed. Fur
ther, in the long-term it makes sense if safety is part of a single dialogue agent model, in the sense that ideally it should understand what it is saying is unsafe.
Here, we detail two generative model training methods that are less likely to surface unsafe content without the use of an additional safety layer: data pre-processing and "baking-in" the safety layer, the latter of which is a new approach introduced in this work.
Data Pre-processing A classic approach to training models on unclean data is to filter it beforehand. Assuming we have access to a safety classifier, we can use it to filter the training set. In this work, we perform filtering by removing an example from the training set if either the conversational context (input) or response (output) triggers the safety classifier. Other approaches – such as author-based filtering – are considered and evaluated in Appendix §B.2. This training set is then used to train models as usual. With this approach, it is important for this filtering to be performed on the large pre-training dataset: if only the fine-tuning datasets are cleaned, the model will still have been exposed to offensive language, which it will be able to remember and use (as indeed confirmed by our experiments).
Baking in the Safety Layer Data pre-processing methods attempt to make a model safe by simply not exposing it to offensive language. This can make those models vulnerable to adversarial usage because they will not have learned how to handle offensive language at all: our models frequently copy the input (Welleck et al., 2020), so they might copy the offensive language. We instead propose a technique for attempting to bake awareness of toxic language into the training data, by using labeled examples that recommend appropriate action on the model's part in those circumstances.
To do this, we first assume we have access to a safety classifier at training time (but not at deployment time). For each training example, if the last utterance in the dialogue history or the ground-truth response are labeled as unsafe by the classifier, we instead replace the ground-truth response of that training example with a non-sequitur. An example demonstrating this procedure is shown in Table 2.
After constructing "baked-in" safety data, one can then train the generative model using likelihood training as usual, but with these modified targets. We separate training examples that have been
Bot-Adversarial Dialogue (this work)
Build-It Break-It Fix-It for Safety (Dinan et al., 2019a)
Figure 1: Comparison of our Bot-Adversarial Dialogue (BAD) set-up (left) to the "build-it, break-it, fix-it" for toxicity classifier robustness from Dinan et al. (2019a) (right). In BAD, the "breaker" (or adversarial user) adversarially tries to elicit offensive messages from a dialogue model. In Dinan et al. (2019a) the "breaker" tries to break a classifier by submitting adversarial offensive messages, but does not conduct dialogue with a model.
Original Data	Safe Utterance Data	“Baked-in” Safety Data
Input: Hi, how are you doing today?	Input: Hi, how are you doing today?	Input: Hi, how are you doing today?
Output: I am spending time with my 4 sisters what are you up to?	Output: I am spending time with my 4 sisters what are you up to?	Output: I am spending time with my 4 sisters what are you up to?
Input: Holy f*ck he's ugly.	EXAMPLE REMOVED	Input: Holy f*ck he's ugly
Output: So are you!	Output: Hey do you want to talk about something else? How about we talk about Paddleboarding?
Table 2: Training examples for "baking in" the safety layer (right) (\$4.2.2) compared to the original data and the safe utterance pre-processed data approach. When a safety classifier flags an example as containing an offensive message, the safe utterance training data set has those examples simply removed from the dataset, whereas in the "baked-in" method the output (target utterance) is replaced with a non-sequitur, and then used to train the model.
modified for safety from those that have not, and assign different weights to them - effectively drawing examples from those two sets with different probabilities - affecting how much the model optimizes for safety versus usual conversational abilities. Balancing these weights is important, especially when dealing with highly toxic pre-training sets, as they may be dominated by modified examples. We choose this weighting as a hyperparameter of the model.
4.3 Evaluation Methods
We use human evaluations in both an adversarial and non-adversarial setting to evaluate how safe our generative models are. Since our ultimate goal is to create models that are safe while still being engaging to talk to, we also perform human evaluations of conversational quality.
Evaluating Conversational Quality We use the ACUTE eval (Li et al., 2019) method of evaluating conversational quality, as used for BlenderBot
(Roller et al., 2020) and elsewhere. The method involves collecting human-bot conversations for two models one wishes to compare, and then presenting two of those conversations at random, one from each model, to crowdworkers to make a series of pairwise decisions based on a target question. We use the question corresponding to evaluating engagement from Li et al. (2019), which is phrased as "Who would you prefer to talk to for a long conversation?" We compare each of the proposed models in this paper to the baseline BST 2.7B which Roller et al. (2020) showed is state-of-the-art in open-domain dialogue. We note that our goal is not to find models which are more engaging than our baseline, but to find models that are simultaneously equally engaging and safer.
Evaluating Adversarial Safety To perform human evaluation of the safety of models in an adversarial setting we create a test set consisting of 180 dialogues collected using the Bot-Adversarial Dialogue method. Recall that the BAD data is
collected by having humans converse and attempt to elicit unsafe responses with a dialogue model. In order to prevent this test set from being biased towards adversarial examples that target a single model, we construct the test set by sampling conversations from a large suite of models. At evaluation time, we collect a model's responses to each of the 180 dialogues in the test set, and human evaluators are used to judge the safety of each response. This set-up allows us to evaluate models in an adversarial setting that mimics deployment but for which the examples remained fixed, eliminating variances based on the experience and quality of crowdworkers during conversation collection.
Evaluating Non-Adversarial Safety While we need our models to perform well in an adversarial setting, we also wish for them to perform well in a non-adversarial setting. For example, a model that repeats user input verbatim may be robust to more subtle attempts to elicit offensive responses which are not offensive in and of themselves, but would not be robust to simpler attacks like profanity. For this reason, we propose a non-adversarial test set composed of 180 examples that are extracted from the Wikipedia Toxic Comments test set. We adopt the same human evaluation setup as in the adversarial setting in which various models are evaluated for the given contexts.
We detail experimental results in this section, including results of the data collection from the Bot-Adversarial Dialogue method (§5.1), experimental results related to training classifiers (§5.2), and a comparison of safe generation methods (§5.3). Lastly, in §5.4, we detail and compare the overall safety and engagingness scores for all models.
5.1 Data Collection Results
We describe results from data collection using the Bot-Adversarial Dialogue method, providing a detailed analysis of the effects of the crowdsourcing methods.
In order to inform crowdsourced task design, we use logistic regression to model several task outcomes. Predictors include variables capturing the human chat partner's experience with the task and the particular bot they are currently talking to, and which of two possible versions of task instructions was received. Experience with the task is measured as the number of HITs accepted by the worker so
far - a HIT, or Human Intelligence Task, is the term used by Amazon's Mechanical Turk to refer to a single instance of a crowdworker task. Experience with a specific bot is captured as the position of the utterance within the conversation (e.g., 2nd utterance in a 14 utterance conversation). The models underlying the bot responses were included as predictors and had a large significant effect (as discussed in the rest of the paper), but are omitted from the discussion here to focus on predictors related to task design.