SlowGuess/NAACL_2022_OCR huggingface dataset:
on Knowledge Discovery and Data Mining, KDD '16, page 1135-1144, New York, NY, USA. Association for Computing Machinery.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, Online. Association for Computational Linguistics.
Ananya B. Sai, Mithun Das Gupta, Mitesh M. Khapra, and Mukundhan Srinivasan. 2019. Re-evaluating adem: A deeper look at scoring dialogue responses.
Lloyd S. Shapley. 1952. A Value for N-Person Games. RAND Corporation, Santa Monica, CA.
Heereen Shim, Dietwig Lowet, Stijn Luca, and Bart Vanrumste. 2021. Building blocks of a task-oriented dialogue system in the healthcare domain. In Proceedings of the Second Workshop on Natural Language Processing for Medical Conversations, pages 47-57, Online. Association for Computational Linguistics.
Weiwei Sun, Shuo Zhang, Krisztian Balog, Zhaochun Ren, Pengjie Ren, Zhumin Chen, and Maarten de Rijke. 2021a. Simulating user satisfaction for the evaluation of task-oriented dialogue systems. In Proceedings of the 44rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21. ACM.
Weiwei Sun, Shuo Zhang, Krisztian Balog, Zhaochun Ren, Pengjie Ren, Zhumin Chen, and Maarten de Rijke. 2021b. Simulating User Satisfaction for the Evaluation of Task-Oriented Dialogue Systems, page 2499-2506. Association for Computing Machinery, New York, NY, USA.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. RUBER: an unsupervised method for automatic evaluation of open-domain dialog systems. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 722-729. AAAI Press.
E. trumbelj and I. Kononenko. 2013. Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 41:647-665.
Stefan Ultes, Alexander Schmitt, and Wolfgang Minker. 2013. On quality ratings for spoken dialogue systems - experts vs. users. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 569-578, Atlanta, Georgia. Association for Computational Linguistics.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377-392.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
Jie Yang, Hirofumi Kikuchi, Takatsugu Uegaki, and Hideaki Kikuchi. 2021. The effect of the repetitive utterances complexity on user's perceived empathy and desire to continue dialogue by a chat-oriented dialogue system. In Proceedings of the 9th International Conference on Human-Agent Interaction, HAI '21, page 241-244, New York, NY, USA. Association for Computing Machinery.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298-1308, Minneapolis, Minnesota. Association for Computational Linguistics.
A Adversarial Examples
We present one adversarial example for all of the dialogue attributes (not present in the table 2) in the tables 5 and 6. These examples are generated in an automatic way using our test-suite and testify to its effectiveness in generating different adversarial behaviors.
Coherence
Entailment	Pronoun	Named Entity Attacks
A: It's a Taiwanese puppet doll.
B: It's huge!
A: Yeah. They're usually this big.
Actual: The craftsmanship is exci-
lent.
Adversarial: I'm Rose Teller. I
think I've seen you somewhere be-
fore?	A: Harry, come here immediately!
B: What?
Actual: Don't take that tone with me! I
saw you hit your brother.
Adversarial: Don't take that tone with
you! we saw me hit your brother.	A: Can you tell me what bus to catch
from Altadena to downtown LA?
B: You can catch the 486.
Actual: That bus goes all the way to
LA?
Adversarial: That bus goes all the way
to Chilin?
Coherence	Naturalness
Co-reference Attacks	Speaker Sensitiveness	Unnatural Repetitions
A: i am only five feet and five inches, so
i am short too. are you married?
B: i am not. i just have my dog pedro.
he is my family
A: i do not like dogs. i was attacked
when i was a little girl.
Actual: i am so sorry to hear that . i bet
you would like pedro he is sweet
Adversarial: i am so sorry to hear that. i
bet you would like pedro pedro is sweet	A: i hate families. i prefer to be
alone.
B: i am short at 5 ft tall. how about
you?
A: i am 5ft and 6in tall.i weigh 220
pounds and its all muscle.
Actual: that sounds very nice, yes.
Adversarial: i am 5ft and 6in tall. i
weigh 220 pounds and its all muscle.
that sounds very nice, yes.	A: i just ate a mango and now i need to
go to the hospital.
Actual: are you allergic? dogs give me
bad allergies.
Adversarial: are you allergic? dogs
give me bad allergies bad allergies bad
allergies.
Table 5: Examples for the manipulation of coherence based attacks.
Naturalness
Entrainment	Vocabulary Diversity	Natural Paraphrasing
A: what music do you like?
B: i dont really prefer any kind of music
Actual: well, does your mom play any music at her restaurant?	A: i for sure read an speak english
B: that is helpful. i do as well and just graduated college.
A: i love pork, especially bacon.
Actual: bacon is good. i do not eat much meat though. do you have pets?	A: my hobbies are fashion an clothes!
B: fashion is cool. i am an avid gamer playing second life.
Actual: awesome! what is second life? never heard of it
Adversarial: well, does your mom play any euphony at her restaurant?	Adversarial: bacon is well. i do not eat much meat though. do you have pets?	Adversarial: What is Second Life?
Never heard of it!
Table 6: Examples of different adversarial responses for naturalness cases generated by our heuristics.