SlowGuess/NAACL_2022_OCR huggingface dataset:
Trigger Masking (TM) Baseline In this baseline, we consider masking the adversarial trigger tokens. Note that the defender does not generally know which tokens were the trigger-tokens used by the adversary, so this approach is not applicable in realistic settings. However, we believe that considering
$^{5}$ https://github.com/cdierre/transformers-interpret
Figure 7: Transferability of our defense to the Perspective API and Safety classifier for different attacks.
this type of oracle baseline can still give us interesting insights, so we include it in our experiments.
3.2.2 AMT Experiments
We asked AMT workers to evaluate the defense quality according to relevancy and fluency, the coherency of the overall conversation, and the toxicity of the defense utterance. 27 conversations were rated from each of the three defenses (TM, Two-stage Non Sequitur, and our proposed defense). 3 AMT workers rated each conversation which gave us 243 annotations 81 from each defense. More details can be found in Appendix A.
Defense Effectiveness We report "defense effectiveness" as the percent decrease in a defender generating a toxic response after adversary's attack when the defense is applied compared to when it isn't. From our results, we observe that both our proposed defense mechanism as well as the Non Sequitur baseline achieve  $100\%$  defense effectiveness according to Toxic-bert classifier. We also noticed that for our proposed method for all the attacks except UAT-LM, we were able to reach  $100\%$  defense effectiveness by only masking one token. For UAT-LM, almost  $90\%$  of cases were resolved by masking one token and the rest were resolved by the iterative approach that masked multiple tokens (up to 3). In addition, our defense is also outperforming the oracle Trigger Masking which shows that using model interpretability can give us more valuable insights than blindly masking out the triggers. In some cases tokens generated after the trigger can themselves be more toxic and decisive in forcing the defender into generating toxic utterances (more details in Appendix B.1 Table 4.). As expected, the Non Sequitur defense is always effective as it replaces the toxic utterance with a
non-toxic utterance by changing the topic; however, this approach is not necessarily creating the best conversational experience as also verified by our human experiments in terms of maintaining relevancy and coherency of the conversation.
Defense Transferability We analyze transferability of our defense mechanism with regards to three different aspects as follows:
1. Transferability to other toxicity detection classifiers: Results in Figure 7 demonstrate that even if the defender is using the interpretability results provided by the Toxic-bert classifier, it can still be effective in reducing toxicity according to Perspective API and Safety classifier on all attacks.
2. Transferability when UTSC attack uses different toxicity classifier than what the defender uses in its defense: We also noticed that even if the defender and the attacker do not use the same toxicity detectors the defense can be effective. To see the results of our defense on all the combination of toxicity detectors used by the attacker for its selection criteria refer to Appendix B.1.
3. Transferability of the defense to human generated attacks: Lastly, to make sure that our defense also transfers to human generated attacks and not just automatic attacks, we tried to generate attacks against the DialoGPT model and converse with it as the adversary. We managed to trigger the system for  $10\%$  of the cases, in line with the automatic attacks. We also saw  $70\%$  reduction in toxic generation when we applied only one iteration of our defense mechanism on these attacks.
Human Evaluation Results of our human evaluations are demonstrated in Figure 8. Our defense is rated to have the highest fluency and relevancy scores. While our defense is mostly rated to have moderate to good ratings for relevancy, the Non Sequitur defense has poor relevancy scores. This is because the Non Sequitur defense changes the topic every-time a toxic utterance is generated which lowers the quality of the conversational experience. Thus, even if the Non Sequitur defense can be really effective in reducing the toxicity as it replaces the toxic utterance with a non-toxic templated sentence, it can create poor conversational experience as also rated by human annotators. Human annotator agreements were also reasonable for these tasks (Table 2) according to Fleiss Kappa scores.
Coherency	Fluency	Relevancy	Toxicity
Ours	Non sequitur	TM	Ours	Non Sequitur	TM	Ours	Non Sequitur	TM	Ours	Non Sequitur	TM
0.50	0.42	0.53	0.43	0.45	0.42	0.51	0.48	0.50	0.56	0.48	0.51
Table 2: Human annotator agreement results for the defense quality annotations according to Fleiss Kappa.
Figure 8: Defense human evaluation results. Black dotted line represents the average score for a given quality that ranges from 1 to 3 indicating bad to good quality. Each bar plot demonstrates proportion of workers that rated a particular score (red for bad, yellow for moderate, and green for good). Toxicity ratings are binary.
4 Beyond Conversational Agents
We show the generalizability of our defense method against non-conversational generation tasks, by conducting experiments with RealToxicityPrompts dataset (Gehman et al., 2020). Previous work showed that the prompts in RealToxicityPrompts can force different generative models such as GPT-2 (Radford et al., 2019) to generate toxic responses. Thus, we used our defense to test whether it can also be effective in reducing the number of toxic responses given these prompts in RealToxicityPrompts in the GPT-2 model. As evident from the previous discussions, the Non Sequitur baseline defense (Xu et al., 2020) that we considered in our paper, only works for the conversational domain; however, our method has the advantage of working on any conditional generation task. We used the 100k prompts in RealToxicityPrompts and reported the number of toxic generations before and after applying our defense from the GPT-2 model.
Results in Figure 9 demonstrate that one iteration of our defense reduces the number of generated toxic responses by  $81\%$ ,  $31\%$ , and  $23\%$ , according to Toxic-bert, Perspective API, and Safety classifier, respectively. Although the defense is based on
Figure 9: Number of generated toxic responses before and after the defense was applied to GPT-2 from the RealToxicityPrompts dataset (Gehman et al., 2020). Our defense is shown to reduce the number of toxic generations in GPT-2. Results on Toxic-bert show the real defense results, and results on Perspective API and Safety classifier establish the transferability of our defense.
Toxic-bert, the results still transfer to Perspective API and Safety classifier. These results show the effectiveness of our defense in reducing toxic generations beyond conversational domain and a step toward reducing toxic generation. Notice that the setup of this experiment was not adversarial; however, prompts were causing the toxic generations.
Crafting adversarial examples and using them in training was previously shown to be an effective technique in improving NLP and ML models (Nie et al., 2020; Dinan et al., 2019; Kiela et al., 2021). Not only that, but adversarial attacks can reveal important vulnerabilities in our systems (Zhang et al., 2020a). Although previous work has studied adversarial examples in NLP (Li et al., 2017; Zang et al., 2020; Morris et al., 2020; Mozes et al., 2021) most of them focused on accuracy as a metric of interest. Among the ones that studied toxicity and other ethical considerations (Wallace et al., 2019; Sheng et al., 2020) they did not put the focus on either conversational agents or they did not consider attacks being imperceptible. Cheng et al. (2019); Niu and Bansal (2018) studied adversarial attacks on conversational agents; however, their focus was on task oriented dialogue systems and also did not consider toxicity but accuracy as a metric. Xu et al. (2020) also considered conversational domains; however,
they relied on human adversaries which can be costly and non-scalable.
Beyond attacks, we discussed a possible defense mechanism to improve robustness of generative models against generating toxic responses using interpretability methods. Using interpretability mechanisms was also previously shown to be effective in reducing bias in ML applications (Mehrabi et al., 2021a). In addition, there is a body of work in detecting toxic behavior in conversational agents (Zhang et al., 2018; Almerekhi et al., 2019; Baheti et al., 2021) that can be utilized to design ethically aligned systems.
We studied the possibility of generating imperceptible attacks against conversational agents that, while fluent and coherent, target the model into generating toxic responses. Through various automatic and human experiments, we showed the effectiveness of our attacks both in terms of being adversarial as well as being able to maintain coherency, relevancy, and fluency of the generated conversation (what we referred as the imperceptibility of the attack). We then proposed a defense mechanism that was shown to be effective through various automatic and human evaluations as well as its transferability to human attacks, general generation tasks, and different toxicity classifiers. Future work can focus on improving our proposed attacks both in terms of imperceptibility and effectiveness as well as more advanced defense mechanisms.
We thank anonymous reviewers for providing insightful feedback. We are also thankful to Emily Dinan (Meta AI) for providing useful pointers about safety classifiers. Ninareh Mehrabi's research was funded by USC + Amazon Center on Secure and Trusted Machine Learning fellowship. This work is partially sponsored by the Defense Advanced Research Projects Agency (DARPA) under the contract HR00112290021.
In this work, we proposed possible attacks and defenses against conversational models that can help improve robustness of conversational agents. We also discussed the extension of our defense work on any general generation task that can be an important contribution towards mitigating toxic gen
erations from our models. By proposing effective imperceptible automatic attacks, we also eliminate the need for human labor, reduce the cost, and make this process more scalable.
Previous work has shown the importance of adversarially crafted examples into improving NLP systems (Nie et al., 2020; Dinan et al., 2019; Kiela et al., 2021); thus, our automatically generated examples can be useful in not only improving robustness of these systems and highlighting their vulnerabilities, but also a step towards their improvement. Not to mention our defense mechanism that can directly mitigate the discussed issues.
However, we also acknowledge the negative impacts that our work can have if used irresponsibly. We acknowledge that our attack can be used by unethical adversaries to force the models to generate toxic responses which is undesirable as also previously observed in chatbots (Wolf et al., 2017; Henderson et al., 2018; Dinan et al., 2021).
Since our defense mechanism relies on model interpretability, some of the models may be black-box or not-interpretable. In that case, we show that the defender still can use proxy models which are interpretable and as shown in the results of our experiments the defense can still be transferable. However, we acknowledge that in such cases the defense might not be as effective, which can be considered a limitation of our work. Another possible limitation of our defense mechanism can be the token-level dependence of our defense approach which can cause our defense mechanism to possibly fail on more subtle cases where there is no clear token that makes a sentence toxic.
In our studies, we also incorporated human annotators to annotate the quality of our generated conversations. We made sure to provide the annotators with appropriate and sufficient instructions to complete the work along with a reasonable and acceptable compensation for their labor. We also made the annotators aware of possible toxic or inappropriate language in our generations ahead of time. More details can be found in Appendix A.
We hope that our study can be used for the benefit of the society and development of robust conversational systems along with reduced toxic generations in our models. We release our code and data in a public Github repository for the community to be able to use and reproduce our results.
Hind Almerekhi, Haewoon Kwak, Bernard J. Jansen, and Joni Salminen. 2019. Detecting toxicity triggers in online discussions. In Proceedings of the 30th ACM Conference on Hypertext and Social Media, HT '19, page 291-292, New York, NY, USA. Association for Computing Machinery.
Ashutosh Baheti, Maarten Sap, Alan Ritter, and Mark Riedl. 2021. Just say no: Analyzing the stance of neural dialogue generation in offensive contexts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4846-4862, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. 2018. Adversarial attacks and defences: A survey. arXiv preprint arXiv:1810.00069.
Minhao Cheng, Wei Wei, and Cho-Jui Hsieh. 2019. Evaluating and enhancing the robustness of dialogue systems: A case study on a negotiation agent. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3325-3335, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, and Verena Rieser. 2021. Anticipating safety issues in E2E Conversational AI: Framework and tooling. arXiv preprint arXiv:2107.03451.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4537-4546, Hong Kong, China. Association for Computational Linguistics.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369.
Peter Henderson, Koustuv Sinha, Nicolas Angelard-Gontier, Nan Rosemary Ke, Genevieve Fried, Ryan Lowe, and Joelle Pineau. 2018. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES '18, page 123-129, New York, NY, USA. Association for Computing Machinery.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4110-4124, Online. Association for Computational Linguistics.
Yitong Li, Trevor Cohn, and Timothy Baldwin. 2017. Robust training under linguistic adversity. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 21-27, Valencia, Spain. Association for Computational Linguistics.
Ninareh Mehrabi, Umang Gupta, Fred Morstatter, Greg Ver Steeg, and Aram Galstyan. 2021a. Attributing fair decisions with attention interventions. arXiv preprint arXiv:2109.03952.
Ninareh Mehrabi, Muhammad Naveed, Fred Morstatter, and Aram Galstyan. 2021b. Exacerbating algorithmic bias through fairness attacks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(10):8930-8938.
John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020. Reevaluating adversarial examples in natural language. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 3829-3839, Online. Association for Computational Linguistics.
Maximilian Mozes, Max Bartolo, Pontus Stenetorp, Bennett Kleinberg, and Lewis Griffin. 2021. Contrasting human- and machine-generated word-level adversarial examples for text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8258-8270, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901.
Tong Niu and Mohit Bansal. 2018. Adversarial oversensitivity and over-stability strategies for dialogue
models. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 486-496, Brussels, Belgium. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2020. Towards Controllable Biases in Language Generation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3239-3254, Online. Association for Computational Linguistics.
D Solans, B Biggio, and C Castillo. 2021. Poisoning attacks on algorithmic fairness. In European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2020, volume 12457, pages 162-177. Springer Science and Business Media Deutschland GmbH.
Yi-Lin Tuan, Connor Pryor, Wenhu Chen, Lise Getoor, and William Yang Wang. 2021. Local explanation of dialogue response generation. In Advances in Neural Information Processing Systems.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153-2162, Hong Kong, China. Association for Computational Linguistics.
M.J. Wolf, K.W. Miller, and F.S. Grodzinsky. 2017. Why we should have seen that coming: Comments on microsoft's tay "experiment," and wider implications. The ORBIT Journal, 1(2):1-12.
Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6066-6080, Online. Association for Computational Linguistics.
Justine Zhang, Jonathan Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Dario Taraborelli, and Nithum Thain. 2018. Conversations gone awry: Detecting early signs of conversational failure. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1350-1361, Melbourne, Australia. Association for Computational Linguistics.
Rong Zhang, Qifei Zhou, Bo An, Weiping Li, Tong Mo, and Bo Wu. 2020a. Enhancing neural models with vulnerability via adversarial attack. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1133-1146, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020b. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1-41.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020c. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270-278, Online. Association for Computational Linguistics.
We discuss additional detailed experimental setups, quantitative, and qualitative results that were left out from the main paper due to space limitation.
A Details of Experimental Setup
General Setup For the UAT and UAT-LM attacks, we used a 6-gram trigger to include in our baselines. For "attack effectiveness", we report the percentage of conversations out of 100 conversations that we study in this paper in which the defender was fooled by the adversary after the performed attack into generating a toxic response. This is done over one round of analysis. For "defense effectiveness", we report the percent decrease in a defender generating a toxic response after adversary's attack when the defense is applied compared to when it isn't on the same set of 100 conversations that we used throughout the paper for different analysis. Similar to attack effectiveness, this is done over one round of analysis.
Computational Resources We used Nvidia GeForce RTX 2080 to perform all our experiments except the experiment using the GPT-2 model which was ran on CPU for memory constraints.
Model details We used the small DialogGPT model from Hugging face $^{6}$ . The GPT-2 $^{7}$  and Unitary Toxic-bert $^{8}$  models were also adapted from Hugging face. Safety classifier was adapted from ParAI $^{9}$ . We used all the default thresholds set by the developers for all these toxicity detection classifiers and a threshold value of 0.5 for the Perspective API to detect whether an utterance is toxic or not. We use the same threshold values both in our attacks for the selection criteria and in defenses to determine if a generated utterance is toxic or not. Comparing the performance of three classifiers along with analyzing qualitative results, we realized that Toxic-bert is the least sensitive amongst the three classifiers, followed by Perspective API that has the closest agreement to humans, and Safety classifier.
Mechanical Turk Mechanical turk experiments were performed on Amazon's MTurk platform<sup>10</sup>. We tested the experiment carefully on the sandbox platform before releasing it live. The turkers were
chosen from the master workers pool with additional qualifications set (e.g., HIT approval rate above 85, number of approved HITs above 1000) to make sure workers are reliable workers. We left a comment section to make sure we hear the workers' concerns about the task and the pay. We received couple of comments about the task being interesting with no complains on the pay. We made sure to give reasonable and on time compensation for the amount of work the workers put into and made sure to hear their comments about the pay. We paid 0.30 for each HIT to be completed. Detailed survey instruction forms of our attack and defense are included in Figures 13 and 14.