SlowGuess/NAACL_2021_OCR huggingface dataset:
Modeling results shown in Table 3 suggest that (1) instructing workers to ask open questions about sensitive topics rather than using obvious profanities (New instruction set) has a significant effect, increasing the rate of unsafe bot utterances while simultaneously decreasing the rate of unsafe human utterances; (2) self-selection effects are present (see also Sec. A.4), so that the total number of HITs ultimately completed is predictive of higher success at eliciting not-OK content; (3) two types of learning effects are present: workers are more successful (i.e., are able to solicit more unsafe responses) as they perform more iterations of the task, and within HITs, which might reflect that workers figure out the vulnerabilities of the particular bot they have been paired with and identify the most successful strategies. We note that the increased rate of unsafe utterances for later utterances observed here is in the context of an explicitly adversarial setting aiming to elicit them; we do not expect that this pattern would generalize to non-adversarial contexts.
5.2 Classifier Training Results
Automatic evaluation results are presented for safety classifiers in Table 4. We train safety classifiers using the methodology described in Sec. 4.2.1 and compare different model sizes and multitasking across different training sources. Firstly, we find our newly trained models superior to existing models from Dinan et al. (2019a) when using the same training sets, likely due to improved pushshift.io Reddit pre-training of our Transformers compared to their BERT models. However, we find relatively small gains from either larger Transformers (Safety Classifier $^{+}$ ) over smaller ones (Safety), or from semi-supervised learning over Reddit and BST (Semi-Sup. $^{+}$ ).
We compare the classifier trained on the BAD dataset, multitasked with the other datasets, to other approaches in Table 4. We observe similar results
Outcome: not OK utterances
Bot, rater	Bot, partner	Human
Base	-3.06***	-2.04***	-0.37*
Increase / utterance	0.14***	0.14***	0.11***
Increase / HIT	0.04***	0.03***	0.08***
New instruction set	0.19*	0.70***	-0.36***
Total HITs	0.06***	0.10***	0.01, n.s.
Table 3: Logistic regression coefficients for the outcomes of a bot response being rated as not OK in a subsequent verification task (Bot, rater), during the chat itself (Bot, partner), or the human partner's utterance being rated as not OK (Human). Higher means higher probability of being rated as not OK. Total HITs is the total number of HITs ultimately completed by a worker, to control for self-selection effects that could masquerade as across-HIT learning effects. Learning effects are detectable both within a HIT and across HITs. Model types are included in the regressors but not shown here. Significance (Wald test):  $^*$ $p < 0.05$ .  $^{***}$ $p < 0.001$ .  $n.s.$  :  $p > 0.1$ .
to our other new safety classifiers on the single-turn Wikipedia Toxic Comments (WTC), Build-It Break-It Fix (BBF) and Standard (S) test sets, but superior results on the multi-turn bot-adversarial BAD test set. The BAD-based classifier achieves 80.8 unsafe F1 on the latter dataset, while the next best performing methods achieve 61.5, 61.0 and 60.7, respectively. This result can be explained by virtue of the fact that the BAD-based classifier is the only one trained on the BAD training set, hence it sees data that most closely resembles the evaluation distribution.
Note that the BAD training set differs from the other training sets listed as it is both (i) adversarially collected and (ii) multi-turn. One can tease apart the effects of each of these attributes by comparing to a single-turn (truncated) version of BAD training, shown in Table 4 (second to last row), which still performs well - though not as well - as the multi-turn version, indicating that the adversarial component is most important.
As the BAD test set is the closest setup to the actual setting in which such a classifier might be deployed (it features human-bot conversations, rather than human-human single-turn data), this indicates the BAD-based classifier is the most likely method to be successful in real use cases.
5.3 Safe Generation Results
We compare the baked-in safety layer method of  $\S 4.2.2$  to the data-preprocessing methods using
400M parameter models, the details of which are described in Appendix B, and find that "baked-in" training gives increased safety over safe utterance preprocessing. On pushshift.io Reddit, the "baked-in" method triggers a classifier  $0.2\%$  vs.  $6.8\%$  of the time for preprocessing. Both methods yield similar PPL and F1 scores. We thus experiment with scaling it up to a 2.7B parameter model.
5.4 Comparing All Models: Safety and Engagingness
We perform human evaluations to compare the relative safety and engagingness for many of the selected methods. Results showing the engagingness performance relative to safety performance (for both adversarial and non-adversarial safety) using human judgments ( $\S 4.3$ ) are shown in Figure 2. Automatic evaluations are provided in Appendix D.
We compare the methods described in this paper – two-stage models and “baked in” models – to three standard baselines: BST 2.7B, DialogGPT, and GPT2. BST 2.7B (Roller et al., 2020) has simply been trained on existing dialogue corpora, with no safety technique at all in model training. DialogGPT (Zhang et al., 2019) uses a pre-processing method, where offensive subreddits where removed from the training data. We test DialogGPT in two flavors: with short generations (using standard beam decoding), and longer generations (where we add a constraint that a minimum of 20 tokens must be generated, similar to (Roller et al., 2020). In all experiments we use the medium-sized version of DialogGPT, with 345M parameters, as noted in §3.2. Finally, GPT2 (Radford et al., 2019) was trained on web data that was filtered for data quality, but not for offensive language as far as we are aware.
5.4.1 Engagingness
Engagingness scores from the ACUTE-eval set-up are plotted along the  $x$ -axis in Figure 2. Detailed results can be found in Table 9 in the Appendix.
Results on standard models indicate that BST 2.7B is significantly more engaging than GPT2, DialogGPT and pushshift.io Reddit 2.7B.
We apply the classifier learned from our Bot-Adversarial Dialogue (BAD) dataset (multi-tasked with our other datasets) in a two-stage model. Engagingness of this model is found to be not significantly distinguishable from our base BST 2.7B model. The baked-in model also performs similarly to the base BST 2.7B model with respect to engagingness, showing that this system still works
Model Name	Size	Training Data	WTC	S	BBF	BAD	Avg.
Single-turn (Dinan et al., 2019a)	218M	WTC	83.3	68.1	0.0	-	-
Single-turn (Dinan et al., 2019a)	218M	WTC,S	82.1	88.0	41.8	-	-
Single-turn (Dinan et al., 2019a)	218M	WTC,S,BBF	78.0	83.7	67.6	-	-
Multi-turn (Dinan et al., 2019a)	218M	WTC,S,BBF	81.2	89.0	51.4	48.3	67.5
Safety Classifier	128M	WTC,S,BBF	85.0	90.7	80.4	61.0	79.3
Safety Classifier +	311M	WTC,S,BBF	84.8	95.1	85.9	60.7	81.6
Safety Classifier (Semi-Sup. +)	311M	WTC,S,BBF,Reddit,BST	83.1	94.8	80.0	61.5	79.9
Single-turn Safety Classifier (Adv. Dialog)	311M	WTC,BBF,S,BAD	83.3	93.5	81.9	78.3	84.2
Multi-turn Safety Classifier (Adv. Dialog)	311M	WTC,BBF,S,BAD	83.3	93.6	83.9	80.8	85.4
Table 4: Classifier results for various models, reporting unsafe F1 across all datasets, on the Wikipedia Toxic Comments (WTC), Build-It Break-It Fix-It (BBF), Standard (S) and our new Bot-Adversarial Dialogue (BAD) test sets. The '-' indicates we could not evaluate this model to compute results on the new test, and report known results from the existing paper instead.
well in terms of conversation quality.
5.4.2 Adversarial Safety
To perform human evaluation of safety in an adversarial setting, we evaluate models using the BAD evaluation method described in §4.3. Results can be seen on the  $y$ -axis of Figure 2 (left). More details are provided in Table 15 in the Appendix.
Results show that all of our standard base models - including BST 2.7B, DialoGPT, and GPT2 - are susceptible to attack, e.g. GPT2 produces safe responses only  $59.4\%$  of the time, and BST 2.7B only  $55\%$  of the time. Clearly, to defend against BAD requires alternative techniques.
Our two-stage BAD classifier approach improves over our other safety classifiers used in two-stage systems, yielding an  $94.4\%$  OK rate on the adversarial data. Overall, this method offers strong robustness without affecting engagingness, and we advocate its use.
For our "baked-in" model, we see clear gains relative to standard models (e.g. increasing from the baseline BST 2.7B value of  $55\%$  OK up to  $78.3\%$  OK), although these gains are not as significant as when using two-stage models (the same classifiers in a two-stage setup can bring the results up to  $83.9\%$  OK). We believe an important next step for future work is to improve this training technique to match the two-stage results.
5.4.3 Non-Adversarial Safety
Human evaluation of safety in a non-adversarial setting is conducted using the Wiki Toxic Comments test set described in §4.3. Results can be seen on the  $y$ -axis of Figure 2 (right). More details are provided in Table 16 in the Appendix.
Similarly to the adversarial setting, all of our
standard models appear susceptible to attack. In the best case, DialogGPT produces safe responses only  $68.3\%$  of the time. GPT2 performs the worst, providing safe responses  $54.4\%$  of the time.
Our two-stage models get near perfect scores here - scores range from 97.8 to 98.3 - showing that these models are very robust to attack in the non-adversarial setting. This shows that future effort to make these models safe should focus on the adversarial setting, as in BAD.
The "baked-in" model performs the best in this setting, achieving very high scores. We conclude this technique should be further explored, particularly for robustness in the adversarial setting.
We observe that standard generative models – with little or no safety intervention – fall very short in terms of safety, especially when measured using our Bot-Adversarial Dialogue (BAD) framework, which we publicly release along with our models. However, with our safety techniques we can achieve roughly the same engagingness as the state of the art BST 2.7B with substantially better safety scores, showing it is possible to build a model that is both safe and engaging. We find generative models can be improved considerably by distilling a safety classifier into the encoder-decoder during training, i.e. the "baked-in" approach. Two-stage models provide safer results still, with best performance coming from our BAD-based classifier with BST 2.7B in the adversarial case.
We note that while we have improved substantially over existing systems, our best systems are not perfectly safe as measured by the BAD method. Conducting perfectly safe dialogue requires the
Figure 2: Engagingness vs. Safety: Comparing engagingness scores from ACUTE-eval to adversarial safety scores on the Bot-Adversarial Dialogue (BAD) test set (left) and non-adversarial safety scores on the Wiki Toxic Comments test set (right). An ideal model should appear at the top right of both plots, being maximally engaging whilst staying maximally safe. Here, engagingness and safety scores are measured using the metrics from Table 9, Table 15 and Table 16 found in the Appendix, respectively.
model to deeply understand language and likely cannot be completely solved until AI itself is solved. Further complicating the issue is the fact that the very definition of "safe" is both contextually and culturally dependent (Schmidt and Wiegand, 2017). Rather than attempt to define "safety" for all languages and locales, in this work we rely on crowdworker consensus and focus on machine learning methods for English language data. We look forward to further progress in these technical and ethical challenges.
In this paper, we have presented several methods for building safer conversational agents. As we noted in the conclusion, even our best systems are not perfectly safe. This raises several ethical considerations, including questions of: when can a model be considered "safe"? Is a failure rate of  $5.6\%$  in an adversarial setting acceptable for the deployment of such models? How safe is safe enough? Creating a perfectly safe dialogue model requires the model to deeply understand language and likely cannot be completely solved until AI itself is solved, i.e. this is an AI-complete problem.
We also reiterate that the issue is further complicated by the fact that the very definition of "safe" is both contextually and culturally dependent (Schmidt and Wiegand, 2017). A dialogue model must be able to understand the boundaries of its particular conversation partner. What is offensive to one may not be offensive to another (Curry and Rieser, 2019). Culturally speaking, the approaches in this paper are limited in both geograph-
ical and historical senses. Our methods rely only on English-speaking annotators located in the United States. This narrow, Western-centric viewpoint will be insufficient for solving the issue in other languages and locales (Schmidt and Wiegand, 2017). Further, it is well known that commonly used hate-speech datasets are known to have issues with bias and fairness (Dixon et al., 2018). Sap et al. (2019) showed that several contain correlations between surface markers of African American English and toxicity, and propose race and dialect priming as a way to mitigate this. In this work we have assumed a consensus-based view on offensiveness, by admitting examples based on agreement of multiple humans; however, offense to underrepresented groups for example may be missed by such a setup. We encourage further work to consider how classifiers trained on the datasets described in this work may be biased against various demographic groups.
Lastly, our work analyzes publicly available open-sourced models. We note that there may be concerns in the community or the public at large related to releasing models, even for research purposes, due to their potential safety issues. The community has recently started to address this tradeoff between releasing models that can produce offensive or toxic language and open, reproducible research<sup>5</sup>. We believe the solution for these issues involves the community working together and conducting reproducible research on safety. Releasing code and models facilitates that joint community effort.
5 https://emdinanl.medium.com/a- recap-of-the-first-workshop-on-safety-for-conversational-ai-98201d257530
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020. The pushshift reddit dataset. arXiv preprint arXiv:2001.08435.
Ying Chen, Yilu Zhou, Sencun Zhu, and Heng Xu. 2012. Detecting offensive language in social media to protect adolescent online safety. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernce on Social Computing, pages 71-80. IEEE.
Hyojin Chin, Lebogang Wame Molefi, and Mun Yong Yi. 2020. Empathy is all you need: How a conversational agent should respond to verbal abuse. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13.
Hyojin Chin and Mun Yong Yi. 2019. Should an agent be ignoring it? a study of verbal abuse types and conversational agents' response styles. In *Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems*, pages 1-6.
Amanda Cercas Curry and Verena Rieser. 2019. A crowd-based evaluation of abuse response strategies in conversational agents. arXiv preprint arXiv:1909.04387.
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media, volume 11.
Antonella De Angeli and Sheryl Brahmam. 2008. I hate you! disinhibition with virtual partners. Interacting with computers, 20(3):302-310.
Antonella De Angeli and Rollo Carpenter. 2005. Stupid computer! abuse and social identities. In Proc. INTERACT 2005 workshop Abuse: The darker side of Human-Computer Interaction, pages 19-25.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019a. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4537-4546, Hong Kong, China. Association for Computational Linguistics.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019b. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 67-73.
Paula Cristina Teixeira Fortuna. 2017. Automatic detection of hate speech in text: an overview of the topic and dataset annotation with hierarchical classes.
Sam Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462.
Jennifer Hill, W Randolph Ford, and Ingrid G Farreras. 2015. Real conversations with artificial intelligence: A comparison between human-human online conversations and human-chatbot conversations. Computers in human behavior, 49:245-250.
Klaus Krippendorff. 2004. Reliability in content analysis: Some common misconceptions and recommendations. Human communication research, 30(3):411-433.
Margaret Li, Jason Weston, and Stephen Roller. 2019. ACUTE-EVAL: Improved dialogue evaluation with optimized questions and multi-turn comparisons. In NeurIPS workshop on Conversational AI.
Catherine L Lortie and Matthieu J Guitton. 2011. Judgment of the humanness of an interlocutor is in the eye of the beholder. PLoS One, 6(9):e25085.
Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017a. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79-84. ACL.
K.W Miller, Marty J Wolf, and F.S. Grodzinsky. 2017b. Why we should have seen that coming. ORBIT Journal, 1(2).
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599.
Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyen Minh Phu, Dilara Soylu, and Christopher D Manning. 2020. Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations. arXiv preprint arXiv:2008.12348.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M Smith, et al. 2020. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004.13637.
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678.
Anna Schmidt and Michael Wiegand. 2017. A survey on hate speech detection using natural language processing. In Proceedings of the Fifth International workshop on natural language processing for social media, pages 1-10.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In International Conference on Learning Representations.
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. 2017. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, pages 1391-1399. ACM.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019. Predicting the type and target of offensive posts in social media. arXiv preprint arXiv:1902.09666.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2019. *DialoGPT: Large-scale generative pre-training for conversational response generation*. arXiv preprint arXiv:1911.00536.
A Bot-Adversarial Dialogue Data Collection
We collect Bot-Adversarial Dialogues to build the BAD dataset by asking humans to adversarially talk to bots. This appendix provides further details on the data collection.
A.1 Further Collection Details
Figure 4 is a screenshot of the crowdsourced task for collecting Bot-Adversarial Dialogues.
Bots We use a list of models (bots) coming from the techniques in the paper itself Section 3.2 and Section 4. The list of models, and data counts for each are listed in Table 5. One can observe from the offensive statistics themselves some trends, although we caution against their use for evaluation due to the variance in crowdworker experience and skill over the time of collection due to sequential effects. Nevertheless, one can observe that models without safety classifiers are more vulnerable to adversarial attacks from humans, and models with safety classifiers are harder to attack, and that Control Hostile is clearly the most offensive of all models.
Offensive Response Statistics Figure 3 shows some statistics from the dataset concerning when bots respond with offensive language relative to the language used by the human. We find that when humans craft offensive messages, about  $1/3$  of the time the bots reply with offensive responses too. By comparison, the use of safe utterances by humans (e.g. probing questions that are safe within themselves) is about  $2.5 \times$  less effective of a strategy for eliciting an unsafe bot response, although we do not break that down here by model (the less robust the model, the easier it is to elicit an offensive response by writing an offensive query).
We also provide statistics on the number of offensive turns per dialogue in Table 6.
A.2 Offensive Language Types
To get reliable estimates of whether an utterance is safe or not, and to further identify the type of offensive language from the collected adversarial dialogues, we launched a separate crowdsourced annotation task where at least 3 distinct crowdworkers from a disjoint set were instructed to annotate which type of offensive language each utterance
Model	Bot Utterances	Offensive%
BST 2.7B + Safety Cl.	5268	9.9
BST 2.7B + Safety Cl. (Semi-Sup.+)	5372	10.9
BST 2.7B + Multi-Turn Safety Cl.	881	22.4
BST 2.7B Non Sequitur (FT)	7182	19.3
BST 2.7B Non Sequitur (Semi-Sup.+) (FT)	7143	24.2
BST 2.7B	5841	29.4
DialoGPT (min beam 20)	940	46.6
Control Calm	206	34.0
Control Hostile	181	89.5
Table 5: Number of bot utterances and fraction of those labeled as offensive per each chatbot model during collection of the Bot-Adversarial Dialogue crowdsourced task.
Offensive Utterances
Per Dialogue (k)	Chatbot	Human
0	1203	952
1 ~ 2	2910	2386
≥ 3	1671	2446
Table 6: Number of dialogues containing  $k$  offensive utterances from the Bot-Adversarial Dialogue dataset.
Figure 3: When humans use offensive language first, bots tend to respond with unsafe content more often. In response to offensive human messages, about  $1/3$  of the time bots reply with offensive language too, whereas this reduces to  $12.9\%$  in response to safe messages.