SlowGuess/ACL_2020_OCR huggingface dataset:
Recent research has highlighted the complementary nature of dialog retrieval and generation with respect to multi-tasking (Wolf et al., 2019b) and pre-training (Mehri et al., 2019). Because of this complimentary nature, using dialog retrieval (DR) for evaluating generative models is an intuitive choice, especially for metrics like Maintains Context and Uses Knowledge.
The fine-tuned RoBERTa model described in Section 4.2.1 is further fine-tuned for the retrieval task. This task is set up in the same manner as
Figure 3: Visualization of the masked language modelling (MLM) metric. Context words are in grey; response words are in red. The red words are masked, and RoBERTa must predict the likelihood of their true value (shown in green).
the Ubuntu dialog corpus (Lowe et al., 2015). The model is trained given a context  $x$ , a response  $r$  and a binary label  $y$  indicating whether  $r$  is the true response or randomly sampled. The context  $x$  may consist of the dialog history and the fact, denoted  $c$ , or just the fact, denoted  $f$ . Two different versions of the dialog retrieval (DR) metric are trained, with different values of  $x$ . The DR metric score is defined to be the probability  $P(y = 1 | x, r)$  a given DR metric model produces.
Though the DR metric is trained for the task of retrieval, this is done in an unsupervised manner. The retrieval task is an unsupervised task since it requires no additional labels during training (e.g., explicit quality annotations).
The DR metric is appropriate for Maintains Context, Interesting and Uses Knowledge. If a retrieval model predicts that a generated response is contextually relevant to a dialog context, it indicates that the response Maintains Context. Likewise, if a retrieval model predicts that the response  $r$  is contextually relevant to fact  $f$ , it signifies that  $r$  most likely Uses Knowledge.
Interesting is the measure of whether the response is dull/generic or if it provides some interesting/engaging information. The DR metric is trained to distinguish between a ground-truth response  $(y = 1)$  and a randomly sampled response  $(y = 0)$ . Generic responses are applicable to many contexts, and will often appear as both ground-truth responses and randomly sampled responses. As such, the model will likely learn to assign a low probability distribution to these generic responses and will often output  $P(y = 1|r,x) = 0.5$ . As such, generic responses will generally be scored lower than other contextually relevant, interesting responses. The DR metrics will learn to favor re
sponses that are unique to a given context  $x$ , rather than being applicable to many different contexts.
4.2.3 The USR Metric
Given meaningful automatic metrics for each of the five dialog qualities, USR combines the scores into an overall measure that correlates well with Overall Quality ratings.
In Section 3.5, a regression model was trained to reproduce the overall score from each of the specific quality scores. The predictions of this regression model attained a 0.9654 Spearman correlation with the original scores. This same regression is used by USR on top of the automatic metrics presented in Sections 4.2.1 and 4.2.2.
USR combines its sub-metrics into one measure of overall quality. This combination is configurable, adaptable to different datasets or tasks. For example, if a specific application prefers natural responses over interesting ones, the weights of the regression model can be adjusted. Analysis demonstrated that individuals used different weights when producing the overall score (Figure 2). USR might be able to be personalized for specific individuals by adjusting the weights of the regression model.
This section evaluates all of the automatic metrics described in Section 4, by comparing them to human judgement. The best sub-metrics for each dialog quality are used as input for the regression model of the USR metric. While the best performing sub-metrics are not consistent across the two datasets, the USR metric nonetheless exhibits strong results. The annotations for the original ground-truth are not used for evaluation, in order to accurately compare referenced and reference-free metrics.
Table 3 shows turn-level correlations of the best automatic metrics for each dialog quality on Topical-Chat. USR is shown to strongly outperform both word-overlap and embedding-based metrics across all of the dialog qualities. Interestingly, the best non-USR metric is consistently either METEOR or BERTScore - possibly because both methods are adept at comparing synonyms during evaluation. For some dialog qualities, the overall USR metric outperforms the best sub-metric. For example, USR does better for Maintains Context than USR-DR. This is likely because the information from the other sub-metrics (e.g., Uses Knowledge) is valuable and effectively leveraged by USR.
Metric	Spearman	Pearson
Understandable
BERTScore (base)	0.2502	0.2611
USR - MLM	0.3268	0.3264
USR	0.3152	0.2932
Natural
BERTScore (base)	0.2094	0.2260
USR - MLM	0.3254	0.3370
USR	0.3037	0.2763
Maintains Context
METEOR	0.3018	0.2495
USR - DR (x = c)	0.3650	0.3391
USR	0.3769	0.4160
Interesting
BERTScore (base)	0.4121	0.3901
USR - DR (x = c)	0.4877	0.3533
USR	0.4645	0.4555
Uses Knowledge
METEOR	0.3909	0.3328
USR - DR (x = f)	0.4468	0.2220
USR	0.3353	0.3175
Table 4 reports the turn-level correlations of the best automatic metrics for each dialog quality on the PersonaChat corpus. Across all dialog qualities, USR strongly outperforms the word-overlap and embedding-based metrics. Conversations in PersonaChat generally consist of individuals communicating facts from their own persona in a relevant and coherent manner. As such, when models trained on PersonaChat produce subpar outputs, it is generally because the outputs either (1) do not effectively use the persona or (2) are not relevant/coherent to the dialog context. This explains why the correlations are significantly higher for Maintains Context and Uses Knowledge. As a consequence of PersonaChat's strong dependency on both the dialog context and the persona, USR-DR  $(\mathrm{x} = \mathrm{c})$  which uses both the dialog context and the persona to perform dialog retrieval, generally outperforms all other metrics.
Table 5 shows turn-level correlation with the Overall Quality ratings on Topical-Chat, for all of the automatic metrics. USR shows a strong improvement over all other methods. This strong performance can be attributed to two factors: (1)
Table 3: Turn-level correlations on Topical-Chat. We show: (1) best non-USR metric, (2) best USR submetric and (3) USR metric. All measures in this table are statistically significant to  $p < 0.01$ .
Metric	Spearman	Pearson
Understandable
BERTScore (base)	0.0685	0.0672
USR - MLM	0.1186	0.1313
USR	0.1324	0.1241
Natural
VectorExtrema	0.1375	0.1458
USR - DR (x = c)	0.2291	0.1733
USR	0.2430	0.1862
Maintains Context
METEOR	0.2564	0.2500
USR - DR (x = c)	0.5625	0.6021
USR	0.5280	0.6065
Interesting
BERTScore (base)	0.0491	0.0325
USR - DR (x = c)	0.2634	0.0606
USR	0.0171	0.0315
Uses Knowledge
METEOR	0.1719	0.1678
USR - DR (x = c)	0.6309	0.4508
USR	0.3177	0.4027
Table 4: Turn-level correlations on Persona-Chat. We show: (1) best non-USR metric, (2) best USR submetric and (3) USR metric. All values with  $p > 0.05$  are italicized.
Metric	Spearman	Pearson
Word-Overlap Metrics
F-1	0.1645	0.1690
BLEU-1	0.2728	0.2876
BLEU-2	0.2862	0.3012
BLEU-3	0.2569	0.3006
BLEU-4	0.2160	0.2956
METEOR	0.3365	0.3908
ROUGE-L	0.2745	0.2870
Embedding Based Metrics
Greedy Matching	0.1712	0.1943
Embedding Average	0.1803	0.2038
Vector Extrema	0.2032	0.2091
Skip-Thought	0.1040	0.1181
BERTScore (base)	0.3229	0.3540
BERTScore (large)	0.2982	0.3252
Reference Free Metrics
USR - MLM	0.3086	0.3345
USR - DR (x = c)	0.3245	0.4068
USR - DR (x = f)	0.1419	0.3221
USR	0.4192	0.4220
Table 5: Turn-level correlations between all automatic metrics and the Overall Quality ratings for the Topical-Chat corpus. All values with  $p > 0.05$  are italicized.
Metric	Spearman	Pearson
Word-Overlap Metrics
F-1	0.1422	0.1241
BLEU-1	0.0434	0.0469
BLEU-2	0.1122	0.0943
BLEU-3	0.1202	0.0924
BLEU-4	0.1353	0.0899
METEOR	0.2527	0.2713
ROUGE-L	0.0659	0.0385
Embedding Based Metrics
Greedy Matching	0.0916	0.0625
Embedding Average	0.1182	0.1428
Vector Extrema	0.1570	0.1410
Skip-Thought	-0.0393	-0.0452
BERTScore (base)	0.1690	0.1526
BERTScore (large)	0.1518	0.1211
Reference Free Metrics
USR-MLM	0.0795	0.0788
USR-DR (x = f)	-0.0495	-0.0454
USR-DR (x = c)	0.4814	0.6087
USR	0.4693	0.4115
Table 6: Turn-level correlations between all automatic metrics and the Overall Quality ratings for the PersonaChat corpus. All values with  $p > 0.05$  are italicized.
the ability of MLM and DR to accurately quantify qualities of a generated response without a reference response, and (2) the ability of USR to effectively combine MLM and DR into a better correlated overall metric.
USR shows a similar improvement over all other metrics on PersonaChat, as shown in Table 6. However, DR  $(\mathrm{x} = \mathrm{c})$  outperforms USR despite the fact that four out of the five sub-metrics input into the USR regression are DR  $(\mathrm{x} = \mathrm{c})$ . This result is probably due to PersonaChat's strong dependency on both dialog context and persona, both of which DR  $(\mathrm{x} = \mathrm{c})$  explicitly leverages.
We compute the system-level correlation between all automatic metrics and the Overall Quality ratings. USR significantly  $(p < 0.01)$  outperforms all other metrics with a Spearman correlation of 1.0 on both datasets and Pearson correlations of 0.92 (Topical-Chat) and 0.82 (PersonaChat). The full set of system-level correlations can be found in the appendix.
These results demonstrate USR's effectiveness. It strongly outperforms other metrics on both turnlevel and system-level correlations. Gopalakrish
nan et al. (2019) use the F-1 score as their primary automatic evaluation metric when presenting Topical-Chat. The results demonstrate a significant difference between USR and the F-1 score, suggesting that USR is a better metric for the Topical-Chat corpus.
USR achieves statistically significant correlations with human judgement. The results hold across two datasets, Topical-Chat (Gopalakrishnan et al., 2019) and PersonaChat (Zhang et al., 2018).
USR is configurable. Notably it is composed of several specific dialog quality sub-metrics. These sub-metrics are combined in a configurable manner, using a regression. For other tasks, datasets or even users, this regression can be adjusted, allowing qualities to be removed or re-weighted. Additional sub-metrics could be added.
USR should be used alongside human evaluation. USR was created to facilitate development and tuning of dialog models. As such, USR can be used for model selection and hyperparameter tuning. USR should not be used to claim superior performance over another method.
USR may not work with non-generative models, which were not addressed here. Responses produced by a model that is too similar to the evaluation models (e.g., to DR) are a particular concern.
This paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. To address the shortcomings of standard metrics for language generation, USR (1) is reference-free, (2) is composed of multiple sub-metrics that evaluate specific qualities of dialog, (3) has a definition of good dialog that is configurable. Thus the metric may be adapted to different tasks and datasets. USR is shown to strongly correlate with human judgment on Topical-Chat (turn-level: 0.42, system-level: 1.0) and PersonaChat (turn-level: 0.48, system-level: 1.0).
We thank the following individuals for their help with annotation: Evgeniia Razumovskaia, Felix Labelle, Mckenna Brown and Yulan Feng.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380.
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, and Mark Cieliebak. 2019. Survey on evaluation methods for dialogue systems. arXiv preprint arXiv:1905.04071.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2019. The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098.
Gabriel Forgues, Joelle Pineau, Jean-Marie Larchevêque, and Réal Tremblay. 2014. Bootstrapping dialog systems with word embeddings. In Nips, modern machine learning and natural language processing workshop, volume 2.
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015. deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 445-450, Beijing, China. Association for Computational Linguistics.
Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, Dilek Hakkani-Tur, and Amazon Alexa AI. 2019. Topical-chat: Towards knowledge-grounded open-domain conversations. Proc. Interspeech 2019, pages 1891-1895.
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jeffrey P Bigham. 2019. Investigating evaluation of open-domain dialogue systems with human generated multiple references. arXiv preprint arXiv:1907.10568.
Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294-3302.
Jiwei Li, Will Monroe, Tianlin Shi, Sebastien Jean, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic Turing test: Learning to evaluate dialogue responses. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1116-1126, Vancouver, Canada. Association for Computational Linguistics.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. arXiv preprint arXiv:1506.08909.
Shikib Mehri, Evgeniia Razumovsakaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. arXiv preprint arXiv:1906.00414.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318. Association for Computational Linguistics.
Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157-162. Association for Computational Linguistics.
Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation. arXiv preprint arXiv:1706.09799.
Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and William B. Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In HLT-NAACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000-6010.
Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, et al. 2018. On evaluating and comparing open domain dialog systems. arXiv preprint arXiv:1801.03625.
Marilyn A Walker, Diane J Litman, Candace A Kamm, and Alicia Abella. 1997. Paradise: A framework for evaluating spoken dialogue agents. arXiv preprint cmp-lg/9704004.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Towards universal paraphrastic sentence embeddings. arXiv preprint arXiv:1511.08198.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019a. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Thomas Wolf, Victor Sanh, Julien Chaumont, and Clement Delangue. 2019b. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arXiv preprint arXiv:1703.10960.
A Annotation Instructions
Tables 6, 7 and 8 show the annotation instructions used for human quality annotation. These instructions and examples are verbatim what was shown to the annotators.
B Metric Evaluation
Table 3 in the main paper showed turn-level correlations for each specific quality. Due to space limitations, the table only included results for only the best correlated metrics. The full results are shown in Tables 9 - 21.
C Code and Data Release
The code for the metrics can be found at https://github.com/shikib/usr and the human quality annotations can be found at http://shikib.com/usr. The human quality annotations will allow benchmarking of additional metrics.
Annotation Instructions
You will be given a conversation between two individuals. You will then be given several potential responses for the next turn in the conversation. These responses all concern an interesting fact, which will be provided as well.
Your task is to rate each of the responses on several metrics. The response for one metric should not influence the other metrics. For example, if a response is not understandable or has grammatical errors – you should try to ignore this when considering whether it maintains context or if it is interesting.
Please make sure you read and understand these instructions carefully. Feel free to ask if you require clarification. Please keep this document open while reviewing, and refer to it as needed.
Understandable (0-1)	Is the response understandable in the context of the history? (Not if its on topic, but for example if it uses pronouns they should make sense)
·A score of 0 (no) means that the response is difficult to understand.
You do not know what the person is trying to say.
–i did n’t know that . i love to watch the movie inception , it ’s also the first racing movie to be a woman haha . i guess the movie was originally titled ” inception ” awesome movie !
–Context: in my religion , there is no star . how about you
Response: yeah it was back in 1975 .
·A score of 1 (yes) means that the response is understandable. You know what the person is trying to say.
–my favorite role would have to be quarterback . it is such an
interesting role .
–that is true . i think lebron is the highest paid celebrity , i wonder if he will be in the space jam sequel .
Natural (1-3)	Is the response naturally written?
·A score of 1 (bad) means that the response is unnatural.
–Context: A: wow . do you believe in stars of the zodiac ? what is your star ? B: in my religion , there is no star . how about you
Response: yeah , it was back in 1975 .
–i think he is , he is a great teacher and he also taught ellie kemper , she is a great teacher
Table 7: Annotation instructions (part 1 of 3).
Annotation Instructions (ctd.)
Natural (1-3) (ctd.)	Is the response naturally written?
·A score of 2 (ok) means the response is strange, but not entirely unnatural.
- Context: A: wow . do you believe in stars of the zodiac ? what is your star ? B: in my religion , there is no star . how about you Response: i read it sometimes for the fun of it.
·A score of 3 (good) means that the response is natural.
- i think it 's funny that the soviet union sent a spacecraft to venus .
Maintains
Context
(1-3)	Does the response serve as a valid continuation of the conversation history?
·A score of 1 (no) means that the response drastically changes topic or ignores the conversation history.
- Context: A: wow . do you believe in stars of the zodiac ? what is your star ? B: in my religion , there is no star . how about you Response: i think it 's funny that the soviet union sent a spacecraft to venus .
·A score of 2 (somewhat) means the response refers to the conversation history in a limited capacity (e.g., in a generic way) and shifts the conversation topic.
- Context: i do like some drama stuff , yeah he was awesome in that .
Response: yeah . do you like jon Hamm ?
- Context: i believe that ! he would have played longer i ’m sure if he did the granny style approach to shooting freethrows ! Response: i agree . did you know that space jam is the highest grossing basketball movie of all time ?
·A score of 3 (yes) means the response is on topic and strongly acknowledges the conversation history.
- Context: B: wow , that 's great . especially because more than of nba players go broke 5 years after retirement . A: i believe that ! he would have played longer i ’m sure if he did the granny style approach to shooting freethrows ! Response: a lot of players can make money by starring in movies . did you know space jam is the highest grossing movie of all time ? maybe one of the broke retired players can be in the sequel !
- Context: B: you like drama ? patrick stewart teaches classes now . i loved him in star trek A: i do like some drama stuff , yeah he was awesome in that . Response: jonn hamm was also a drama teacher . he taught erin from the office
Table 8: Annotation instructions (part 2 of 3)