SlowGuess/ACL_2020_OCR huggingface dataset:
Table 3: Correlation between RUBER+BERT and context coherence metric  $c(r|q)$  with human ratings (without and with fine-tuning of GPT-2).
Pearson	Spearman
GPT-2 w/o Fine-tune	0.43	0.32
GPT-2 w/ Fine-tune	0.82	0.81
Inter-Rater	Mean	0.70	0.70
Max	0.88	0.85
Table 4: Correlation between response fluency metric  $f(r)$  and human ratings without and with fine-tuning of GPT-2. Pairwise mean and max correlations of human ratings.
tuning on GPT-2. It helps to improve the consistency between human rating and automatic metric.
Table 2 displays a case study. Our coherence metric and the human evaluation agreed that the generated response is not coherent with the given query, while RUBER indicated that this reply is coherent. This might be because RUBER simply compares the embeddings of the query and response and business travel related words in the query such as vacation, workweek and in the reply such as travel, company make RUBER judge that they are similar.
5.2 Response Fluency
Our findings show that the proposed fluency metric  $f(r)$  is highly correlated with human judgments. Table 4 summarizes the relation between our proposed fluency metric and human ratings in terms of Pearson and Spearman correlation. The importance of fine-tuning GPT-2 (as outlined in Section 4.3) is evident. We observe an increase from 0.43 to 0.82 in Pearson correlation and an enhancement from 0.32 to 0.81 in Spearman correlation. In addition, Figure 2 details the effect of fine-tuning. Notably, a correction of outliers occurs.
(b) GPT-2 w/ Fine-tune
Figure 2: Correlation between response fluency metric  $f(r)$  and human ratings for GPT-2 without and with finetuning.
1-Gram Entropy	2-Gram Entropy	3-Gram Entropy
Pearson	Spearman	Pearson	Spearman	Pearson	Spearman
Baseline Dataset	0.46	0.32	0.45	0.33	0.43	0.33
WS Dataset	0.77	0.69	0.76	0.67	0.71	0.61
CTG Dataset	0.72	0.72	0.72	0.72	0.66	0.66
Table 5: Comparison of response diversity metric between the baseline dataset and our paraphrasing-augmented datasets (WS and CTG datasets) using Spearman and Pearson correlations.
Inter-Rater Pearson	Inter-Rater Spearman	Human Variance
mean	max	mean	max
Baseline Dataset	0.21	0.51	0.23	0.65	0.93
WS Dataset	0.78	0.89	0.78	0.92	0.68
CTG Dataset	0.78	0.86	0.79	0.81	0.69
Table 6: Comparison of response diversity between the baseline dataset and and our paraphrasing-augmented datasets (WS and CTG datasets) using Inter-Rater Spearman and Pearson correlations.
5.3 Response Diversity
Table 5 shows the evaluation of the proposed diversity metric on the basis of the augmented datasets with WS and CTG. We also include a baseline dataset which consists of responses from randomly chosen queries from the testing data. Unigram, bigram, and trigram entropy are utilized to calculate responses' diversity and are compared to human ratings with Pearson and Spearman correlation. It is clear that automatic evaluations with the controlled paraphrasing datasets consistently achieve higher correlation compared to those with the baseline dataset. Figure 3 display correlations between normalized human ratings and corresponding  $n$ -gram entropy based on the augmented dataset. Entropy values based on WS and CTG datasets demonstrate stronger relations with human ratings, compared to those based on the baseline dataset, consistent with the reported correlations.
Table 6 displays inter-rater Pearson and Spearman correlations and variance in human ratings.
Human ratings based on the paraphrasing augmented datasets show high inter-rater correlations and lower variance, indicating that raters generally agree with each other. The poor baseline performance is likely due to the uncontrolled nature of input sentences such that outputs of evaluated models are generally diverse, making it difficult for humans to judge the diversity performance of the model. Furthermore, our diversity metrics have correlations with human ratings close to the corresponding mean inter-rater correlations, suggesting that the diversity evaluation based on the paraphrasing-augmented data can reveal the diversity of a dialogue system consistent with humans.
5.4 Logical Self-Consistency
Table 8 displays the correlations between the proposed automatic ratings and human ratings on the the paraphrasing augmented data using WS and CTG and a baseline without augmentation. The automatic metric based on augmented data has a
(a) Baseline w/ 1-gram Entropy
(b) Baseline w/ 2-gram Entropy
(c) Baseline w/ 3-gram Entropy
(d) WS w/ 1-gram Entropy
(e) WS w/ 2-gram Entropy
(f) WS w/ 3-gram Entropy
(g) CTG w/ 1-gram Entropy
(h) CTG w/ 2-gram Entropy
(i) CTG w/ 3-gram Entropy
Figure 3: Correlation between  $n$ -gram entropy and human ratings on the baseline dataset, WS dataset and CTG dataset.
stronger relation with that based on the baseline. In particular, the metric based on CTG augmentation aligns with human judgments the closet.
Inter-rater Pearson and Spearman correlations are reported in Table 9. Human ratings on the augmented data are more consistent than those on the baseline, indicating the necessity and efficiency of using a refined dataset instead of the original one. We show a case study in Table 7.
5.5 Relation between the Four Metrics
Although the four proposed metrics are intuitively and theoretically important in evaluating a dialogue system, it is not entirely clear whether they are independent from each other such that it is necessary to measure all of them. We empirically investigate their association. We randomly choose 50 dialogues from the testing dataset and construct the
evaluation data for the four metrics. Five human evaluators rate on the four aspects of each dialogue. We then examine the pairwise correlation of human ratings on the four metrics. Response fluency correlates with context coherence  $(r = 0.42, p = 0.003)$ . This is mainly due to the fact that inarticulate responses are often considered incoherent with the context. All other pair-wise correlations are non-significant  $(r^{\prime}s < 0.1, p^{\prime}s > 0.25)^{3}$ . Thus, the four metrics are relatively independent from each other and it is critical to take into account all of them to obtain a holistic evaluation of a dialogue model.
3We do not observe any obvious non-linear dependency either.
Speaker A: Are you more of a leader or a follower?
Speaker B: I don't try to lead people. I'd rather cooperate with everybody, and get the job done by working together.
Generated Utterance
Speaker A: Are you more of a follower or a leader?
Speaker B: I like to keep to myself. I'm a person who does not want to be a follower.
Our Score: 0.09
Human Score: 0.20
Table 7: Case study of logical self-consistency. Generated Utterance is generated by CTG. Blue italic words highlights the logic contradiction. Our automatic score is low indicating that the logic contradiction can be detected.
Pearson	Spearman
Baseline Dataset	0.26	0.27
WS Dataset	0.59	0.64
CTG Dataset	0.65	0.66
Table 8: Comparison of logical self-consistency metric between the paraphrasing-augmented data (WS and CTG data) and the baseline data without augmentation using Spearman and Pearson correlations with human ratings.
Inter-Rater Pearson	Inter-Rater Spearman
mean	max	mean	max
Baseline	0.61	0.75	0.62	0.74
WS	0.64	0.80	0.64	0.79
CTG	0.65	0.75	0.66	0.76
Table 9: Comparison of logical self-consistency metric between the paraphrasing-augmented data (WS and CTG data) and the baseline data without augmentation using Inter-Rater Spearman and Pearson correlations.
This paper provides a holistic and automatic evaluation method for open-domain dialogue models. In contrast to prior art, our means of evaluation captures not only the quality of generation, but also the diversity and logical consistency of responses. We recruit GPT-2 as a strong language model to evaluate the context coherency and response fluency. For response diversity and logical self-consistency, we propose to measure these two aspects under augmented utterances with controlled paraphrasing. We leverage two effective approaches to generate augmented utterances: word substitution and text generator with  $k$ -best decoder. Moreover, we utilize  $n$ -gram based entropy to capture response diversity and entailment based approach to measure logical self-consistency. The proposed metrics
show a strong correlation with human judgments. It is our hope the proposed holistic metrics may pave the way towards the comparability of open-domain dialogue models.
Wenjuan Han, Yixian Liu and Kewei Tu were supported by the National Natural Science Foundation of China (61976139).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3(Feb):1137-1155.
Fumihiro Bessho, Tatsuya Harada, and Yasuo Kuniyoshi. 2012. Dialog system using real-time crowdsourcing and twitter large-scale corpus. In Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 227-231.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. 2018. Language gans falling short. arXiv preprint arXiv:1811.02549.
Sarik Ghazarian, Johnny Tian-Zheng Wei, Aram Galstyan, and Nanyun Peng. 2019. Better automatic evaluation of open-domain dialogue systems with contextualized embeddings. arXiv preprint arXiv:1904.10635.
Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018. A knowledge-grounded neural conversation model. In Thirty-Second AAAI Conference on Artificial Intelligence.
Tatsunori B Hashimoto, Hugh Zhang, and Percy Liang. 2019. Unifying human and statistical evaluation for natural language generation. arXiv preprint arXiv:1904.02792.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers), pages 1638-1649.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72, Vancouver, Canada. Association for Computational Linguistics.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023.
Ryan Lowe, Michael Noseworthy, Iulian V Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. arXiv preprint arXiv:1708.07149.
George A Miller. 1998. WordNet: An electronic lexical database. MIT press.
Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. 2016. Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation. arXiv preprint arXiv:1607.00970.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of the conference on empirical methods in natural language processing, pages 583-593. Association for Computational Linguistics.
Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence.
Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. In Thirty-First AAAI Conference on Artificial Intelligence.
Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1577-1586, Beijing, China. Association for Computational Linguistics.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. arXiv preprint arXiv:1506.06714.
Hui Su, Xiaoyu Shen, Pengwei Hu, Wenjie Li, and Yun Chen. 2018. Dialogue generation with gan.
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Rubber: An unsupervised method for automatic evaluation of open-domain dialog systems. In Thirty-Second AAAI Conference on Artificial Intelligence.
Kristina Toutanova and Christopher D Manning. 2000. Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora: held in conjunction with the 38th Annual Meeting of the Association for Computational Linguistics-Volume 13, pages 63-70. Association for Computational Linguistics.
Alan Turing. 1950. Computing machinery and intelligence-am Turing. Mind, 59(236):433.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3731-3741, Florence, Italy. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. CoRR, abs/1901.08149.
Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. 2018. Dp-gan: diversity-promoting generative adversarial network for generating informative and diversified text. arXiv preprint arXiv:1802.01345.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia. Association for Computational Linguistics.