SlowGuess/NAACL_2022_OCR huggingface dataset:
We test Pang-C on all of the adversarial cases while we test the Pang-L metric on coherence and interestingness attacks and Pang-F metric on the naturalness because Pang-C is compared with other overall evaluation metrics in the paper while Pang-F and Pang-L are presented to judge fluency and logical consistency of a dialogue response. Pang-C seems to be robust to attacks which induce unnatural sentence structure. This is evident by looking at the results for 1) pronoun, speaker sensitiveness and co-reference attacks in the table 3; and 2) unnatural repetitions, paraphrasing attacks and vocabulary diversity attacks in the table 4. This highlights again that metrics perform best on the tasks they are designed for. Since GPT2 predicts the likelihood of next token given history, the metric is sensitive to unnatural manipulations. However, its performance varies across other attacks e.g. the metric fails to reliably penalize contradictions or entailments. Similar to DialogRPT, Pang-C also performs worse on out-of-distribution data as visible from the performance drop on datasets other than Dialy Dialogue. GPT2 could be fine-tuned on datasets to retain performance, but not every dataset has a large number of dialogues and finetuning large neural models is not a trivial task. This highlights that pre-trained models like GPT2 may retain performance for attacks which generate unnatural examples but still show a significant room for improvement when dealing with complex conversation semantics or out-of-distribution data.
Pang-C metric performs the best out of the three metrics presented by the Pang-Evaluation. Their Pang-F metric, results shown in the table 4, differs from Pang-C metric because it does not take context into account while assigning scores to the utterance under consideration. However, this causes it to under-perform in comparison to Pang-C metric while judging the adversarial cases. This also highlights a weakness in the training methodology itself. Since the metric has only seen full conversations during training, it fails to reliably penalize adversarial vocabulary choices when it evaluates individual utterances.
Pang-L metric is proposed to score the logical consistency of the conversations. The metric performs better on average than Pang-C at detecting contradictions (the task it was trained for), compa
rable while detecting broken entailments but worse in other cases as visible from the table 3. Its performance is also consistent across datasets most-likely because it was trained on the MNLI task and not on any of the conversation datasets it was being evaluated on. This again highlights the limited applicability of the metric in-line with the other metric we have examined.
5.5 User Satisfaction Metric
User satisfaction simulation metric is trained using human-feedback ratings to predict a score from 1-5, given a conversation. The authors pose this task as a classification problem and train a classifier to predict a score. One of the major problems with this metric, lies in the data it was trained on. Since the data they use has an highly skewed rating distribution, the metric performance reflects this. We test the classifier version of the metric using fine-tuned BERT model (UWBERT). This version assigned the same score to almost all of the adversarial and original conversations. We also computed the score as the weighted average of ratings, using both their hierarchical GRU (WHiGRU) and BERT (WBERT) models, to test if minor variation in assigned scores could provide some insights but the results remained the same. As shown in the tables 3 and 4, this bias in the data results in high error rates closer to 1.0. When error rate relies on scores being similar e.g. in case of co-reference attack, it drops to 0.0 because the model assessment never changes. This highlights a case of bias in model assessments because of the problems in training data.
5.6 Performance Highlights and Conclusion
All metrics that we test perform better on the data they were trained on with the best performance on the trained down-stream task(s). However, none of them are suitable to judge the overall quality of conversations. To be more specific the results show that variability in training methods can cause metrics to assess conversation behaviors differently e.g. DialogRPT vs Pang-C metric performance, metrics are dependent on the training data and reflect the biases in the data and training methodologies e.g. performance of User Satisfaction metric. These insights help us draw two conclusions: 1) a series of metrics specific to individual dialogue behaviors might judge overall quality better than a single evaluation metric and 2) failure of metrics to reliably judge complex behaviors like contradictions
Figure 1: The graph shows relative importance of adversarial conversation features. All adversarial cases having relative importance  $> = 1.0$  shows that DialogRPT is paying more importance to adversarial features.
indicate that either metrics should be trained using examples of these behaviors or should be exposed to more information about conversation flow than just surface textual features.
6 Interpretability Analysis
6.1 Method
In addition to the behavioral analysis described above, we use a black-box model interpretability technique to approximate the feature importance for different adversarial cases for the DialogRPT metric. This shows the relative importance of adversarial features as the metrics paying more importance to adversarial features to assign higher rating to adversarial conversations would show further evidence for misunderstanding of different dialogue semantics.
More specifically, we use Kernel SHAP algorithm presented by (Lundberg and Lee, 2017) which is a modification of LIME (Ribeiro et al., 2016). To determine the Shapley values, the algorithm masks the features (utterance tokens in this case), and replaces them with features from a pre-specified set. The dialogue metric scores are then measured using the augmented instances. These scores along with the score assigned to the original utterance are used to approximate a linear model over the utterance features. The weights of this linear model are the approximate Shapley values.
Instead of replacing the masked tokens with the vocabulary from other utterances we replace them with the masking token specific to the model under consideration. This helps measure the effect on the metric when a certain feature is missing. This was inspired by the Partition SHAP algorithm in SHAP python package released by the authors and erasure of feature representations in Li et al. (2017a).
(a) Contradiction Example
(b) Dullness Example
Figure 2: Shapley values of adversarial features for each case mentioned above. Each adversarial example shows that DialogRPT metric pays high importance to the adversarial features.
(c) Unnatural Paraphrases
6.2 Overall Analysis
We conduct an interpretability analysis of DialogRPT metric performance on adversarial conversations from the Daily Dialogue dataset. We compare the aggregate Shapley values of the human and adversarial features to judge how the DialogRPT metric pays importance to different tokens in the adversarial and human utterances. The aggregation of Shapley values is done using an addition operation. We normalize the Shapley values of features using the minimum and maximum Shapley values for all the features. This helps in conducting a fair comparison of contribution each feature makes to the result. We compare the aggregate Shapley values : 1) between features that are different in human and adversarial responses for cases generated by mutating some of the human features 2) of all features for cases in which the whole human utterance is replaced with an adversarial response. The average relative feature importance of adversarial features (average adversarial feature importance divided by original feature importance) is presented in the figure 1. The relative importance of  $> = 1.0$  for almost all the adversarial cases shows that DialogRPT considers adversarial features more important than the original ones and points towards a misunderstanding of the role that language features play in the conversation flow.
6.3 Feature Importance Analysis
We present an analysis of feature importance for three adversarial cases below. These use the same conversations presented earlier in the table 2.
6.3.1 Contradictions
Figure 2(a) shows the importance assigned by the DialogRPT to the adversarial features in the contradiction example, shown in the table 2. Tokens such as "did not see" directly contradict the context and are given relatively higher importance in comparison to the other features in the utterance. This
proves that DialogRPT fails to understand the role of verb negations in contradicting the context.
6.3.2 Dulness
Figure 2(b) shows the importance of human and adversarial features side by side for the dullness example in the table 2. DialogRPT metric pays more importance to the features which do not progress the conversation in comparison to the human response which contributes meaningfully to the conversation by answering the question being asked. This shows that DialogRPT does not understand which features progress the conversation meaningfully.
6.3.3 Unnatural Paraphrasing
As shown in the figure 2(c), for the unnatural paraphrasing example in the table 2, it is clear that the DialogRPT metric pays more or relatively same importance to the features which are either not used in the same sense, give instead of pay, or not used as frequently in the human conversations, like the use of luncheon instead of lunch. This shows that DialogRPT does not prefer human-like vocabulary choices.
Our analysis on the DialogRPT suggests that it does not know the correct semantics of conversations. We hypothesize it could be because it is not exposed to bad dialogue behaviors and that leads to the metric paying high attention to erroneous constructions. To correct these, it would be a good idea to either augment the training data with adversarial dialogue behaviors, use adversarial learning to make the metrics more robust or use semantic features in addition to the surface language features during training.
7 Choice of Evaluation Metrics
The metrics evaluated in this work are some of the recently proposed with varied training methodolo
gies e.g., training using human judgments versus some measure of text similarity. Also, our choice of these metrics is rooted in the fact that these metrics are primarily judged by a singular measure of improvement which in most cases is the correlation with human judgments. Our analysis highlights that such singular measures of improvement are not enough to capture the variability of performance exhibited by the evaluation metrics in judging complex conversation behaviors like contradictions. Our goal is not to provide an exhaustive accounting of the performance of all available neural metrics but empirically highlight the problems and performance variability which arise because of different training methodologies. Given our results, we hypothesize that other metrics like (Mehri and Eskenazi, 2020b; Lowe et al., 2017; Tao et al., 2018) trained similarly have deeper problems which need to be highlighted using a fine-grained methodology similar to ours.
Our test-suite helps us draw various insights about performance of different metrics, and shows that all the metrics we test have room for improvement. It points at several flaws in the metrics: 1) lack of generalization ability to unseen data, 2) inability to correctly understand different conversation semantics and 3) prone to training and data-induced biases. Furthermore, our interpretability analysis further corroborates these shortcomings and helps us conclude that metrics need to be exposed to more information about conversation behaviors to make them more robust.
Adversarial behaviors from our test-suite help point to many shortcomings in the metrics we test, but many behaviors are very simplistic e.g. speaker sensitivity attacks. Further research can help make these attacks more human-like which may help reveal more information about the evaluation metrics. Our test-suite can also be used directly to make the metrics more robust e.g. by augmenting their training data with adversarial examples or by using it as a reward signal in a RL setup for the training of evaluation metrics. This shows several use-cases of our test-suite and directions for future research.
Sebastian Bach, Alexander Binder, Gregoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations
for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1-46.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021a. Is incoherence surprising? targeted evaluation of coherence prediction from language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4164-4173, Online. Association for Computational Linguistics.
Anne Beyer, Sharid Loaiciga, and David Schlangen. 2021b. Is incoherence surprising? targeted evaluation of coherence prediction from language models. CoRR, abs/2105.03495.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
Minhao Cheng, Wei Wei, and Cho-Jui Hsieh. 2019. Evaluating and enhancing the robustness of dialogue systems: A case study on a negotiation agent. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3325-3335, Minneapolis, Minnesota. Association for Computational Linguistics.
Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256-2262, Austin, Texas. Association for Computational Linguistics.
Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643-653, Berlin, Germany. Association for Computational Linguistics.
Heriberto Cuayáhuitl, Steve Renals, Oliver Lemon, and Hiroshi Shimodaira. 2010. Evaluation of a hierarchical reinforcement learning spoken dialogue system. Computer Speech & Language, 24(2):395-429.
W. Eckert, E. Levin, and R. Pieraccini. 1997. User modeling for spoken dialogue system evaluation. In 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pages 80-87.
Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational AI. CoRR, abs/1809.08267.
Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020. Dialogue response ranking training with large-scale human feedback data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 386-395, Online. Association for Computational Linguistics.
Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An online platform for targeted evaluation of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70-76, Online. Association for Computational Linguistics.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Youngsoo Jang, Jongmin Lee, and Kee-Eung Kim. 2020. Bayes-adaptive monte-carlo planning and learning for goal-oriented dialogues. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7994-8001.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems.
Sangkeun Jung, Cheongjae Lee, Kyungduk Kim, Minwoo Jeong, and Gary Geunbae Lee. 2009. Data-driven user simulation for automated evaluation of spoken dialog systems. Computer Speech & Language, 23(4):479-509.
Mohammad Kachuee, Jinseok Nam, Sarthak Ahuja, Jin-Myung Won, and Sungjin Lee. 2022. Scalable and safe self-learning for skill routing in large-scale conversational ai systems. In NAACL.
Mohammad Kachuee, Hao Yuan, Young-Bum Kim, and Sungjin Lee. 2021. Self-supervised contrastive learning for efficient user satisfaction prediction in conversational agents. In *NAACL*.
Baber Khalid, Malihe Alikhani, Michael Fellner, Brian McMahan, and Matthew Stone. 2020a. Discourse coherence, reference grounding and goal oriented dialogue.
Baber Khalid, Malihe Alikhani, and Matthew Stone. 2020b. Combining cognitive modeling and reinforcement learning for clarification in dialogue. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4417-4428, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 284-294, Melbourne, Australia. Association for Computational Linguistics.
Florian Kreyssig, Iñigo Casanueva, Paweł Budzianowski, and Milica Gašić. 2018. Neural user simulation for corpus-based policy optimisation of spoken dialogue systems. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 60–69, Melbourne, Australia. Association for Computational Linguistics.
Alice Lai and Joel Tetreault. 2018. Discourse coherence in the wild: A dataset, evaluation and methods. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 214-223, Melbourne, Australia. Association for Computational Linguistics.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2017a. Understanding neural networks through representation erasure.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017b. DailyDialog: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 986-995, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1116-1126, Vancouver, Canada. Association for Computational Linguistics.
Scott Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions.
Raveesh Meena, Gabriel Skantze, and Joakim Gustafson. 2014. Data-driven models for timing feedback responses in a map task dialogue system. Computer Speech & Language, 28(4):903-922.
Shikib Mehri and Maxine Eskenazi. 2020a. Unsupervised evaluation of interactive dialog with DialogGPT. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue,
pages 225-235, 1st virtual meeting. Association for Computational Linguistics.
Shikib Mehri and Maxine Eskenazi. 2020b. USR: An unsupervised and reference free evaluation metric for dialog generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681-707, Online. Association for Computational Linguistics.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Laura Panfili, Steve Duman, Andrew Nave, Katherine Phelps Ridgeway, Nathan Eversole, and Ruhi Sarikaya. 2021. Human-ai interactions through A gricean lens. CoRR, abs/2106.09140.
Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of open-domain dialogue generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3619-3629, Online. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Sunghyun Park, Han Li, Ameen Patel, Sidharth Mudgal, Sungjin Lee, Young-Bum Kim, Spyros Matsoukas, and Ruhi Sarikaya. 2021. A scalable framework for learning from implicit user feedback to improve natural language understanding in large-scale conversational AI systems. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6054-6063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Leila Pishdad, Federico Fancellu, Ran Zhang, and Af-saneh Fazly. 2020. How coherent are neural models of coherence? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6126-6138, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference