SlowGuess/ACL_2020_OCR huggingface dataset:
Dialog Research Center, Language Technologies Institute
{amehri,max}@cs.cmu.edu
The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for evaluating dialog models. To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces interpretable measures for several desirable properties of dialog.
The lack of meaningful automatic evaluation metrics is a significant impediment for open-domain dialog generation research. Standard language generation metrics have been shown to be ineffective for dialog evaluation (Deriu et al., 2019; Liu et al., 2016). Without well-accepted, meaningful automatic metrics, open-domain dialog researchers have come to rely on human evaluation. Due to its time- and cost-intensive nature, human evaluation is typically only used for the final dialog model. As such, during development dialog systems are generally optimized for poorly-correlated automatic metrics (e.g., F-1, BLEU, PPL) which can result in sub-par human evaluation scores (Dinan et al., 2019). To facilitate development of open-domain dialog models with meaningful automatic metrics, this paper presents the UnSupervised and Reference free (USR) evaluation metric for dialog.
Standard automatic metrics for evaluating dialog generation (e.g., BLEU,F-1, METEOR,ROUGE) have several shortcomings that make them unsuitable for dialog evaluation: (1) The one-to-many
nature of dialog (Zhao et al., 2017) makes word-overlap metrics ineffective for scoring valid system output that deviates from the ground-truth response (Liu et al., 2016; Gupta et al., 2019). (2) Human evaluation of dialog typically measures multiple properties (e.g., appropriate, interesting, consistent). Automatic metrics on the other hand, condense the multi-faceted nature of dialog quality to a single uninterpretable metric. (3) There are many definitions of what a good dialog is and, as such, it is not feasible to construct a "one size fits all" metric. Depending on the task and the data, the desired qualities of a dialog system may differ (Walker et al., 1997; Deriu et al., 2019).
USR is a reference-free metric that consists of several interpretable sub-metrics which are combined in a configurable manner. Rather than relying on a ground-truth reference response, unsupervised models are trained to measure desired qualities of dialog (e.g., interesting, natural). As such, USR (1) alleviates the one-to-many issue of standard metrics, (2) produces interpretable measures for desirable properties of dialog, and (3) provides a configurable mechanism for combining several sub-metrics into an overall quality score.
To evaluate the performance of USR, human quality annotations were collected for models trained on the Topical-Chat (Gopalakrishnan et al., 2019) and the PersonaChat corpora (Zhang et al., 2018). USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level Spearman: 0.42, system-level Spearman: 1.0) and PersonaChat (turn-level Spearman: 0.48 and system-level Spearman: 1.0). The strong correlation with human judgment across two datasets and a variety of model types shows that USR is a valuable tool for the dialog community. Further, since USR does not require any explicit supervision, it has the potential to generalize to several dialog tasks and datasets.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 681-707 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
The contributions of this paper as follows: (1) a strongly-correlated, unsupervised and reference free metric is proposed for evaluating open-domain dialog systems, (2) a thorough human quality annotation is carried out and is released<sup>1</sup> to facilitate future benchmarking of dialog evaluation metrics.
Standard automatic metrics for language generation correlate poorly with human judgement of dialog (Liu et al., 2016; Lowe et al., 2017; Gupta et al., 2019). For example, the F-1 score can be gamed by outputting the most frequent words, regardless of the context (Dinan et al., 2019).
The poor performance of present metrics is largely due to the one-to-many nature of dialog (Zhao et al., 2017). To avoid comparing to a single reference response, several authors have proposed using multiple reference responses. Multiple reference responses can be obtained with retrieval models (Galley et al., 2015; Sondoni et al., 2015) or through data collection (Gupta et al., 2019). These multi-reference metrics show improvement in performance, but it is infeasible to thoroughly cover the space of potential responses. As such, this paper addresses the one-to-many issue of dialog by presenting a reference-free metric.
Lowe et al. (2017) train ADEM to produce a quality score conditioned on the dialog context, the reference response and the generated response. Venkatesh et al. (2018) present a framework for evaluation of Alexa prize conversations, which attains moderate correlation with user ratings. Both of these methods are trained on explicit quality annotations. In contrast, USR requires no explicit supervision and will more easily generalize to new datasets and tasks.
Li et al. (2017) proposes a reference-free dialog evaluator which is trained to discriminate between human and generated responses. This work is similar to USR in that it evaluates the quality of a response without a reference or quality annotation training data. Using the evaluation model as a reward during reinforcement learning exhibited strong performance. However, correlation with human judgement was not evaluated. Intuitively, it appears insufficient to rely on a discriminator as a meaningful evaluation of dialog since this assumes that all human responses are perfect and all generated responses are imperfect.
3 Human Quality Annotation
To evaluate the correlation of automatic metrics with human judgment, human quality annotation was carried out across two open-domain dialog corpora. Generated responses were obtained from several models described in Section 3.3. For each dialog context, an additional human response was also written. Human annotation was then carried out on sixty dialog contexts, with six responses per context for Topical-Chat (four system outputs, one newly-annotated human output, one original ground-truth response) and five for PersonaChat (one less system output). Each response was given six different scores: Understandable (0-1), Natural (1-3), Maintains Context (1-3), Interesting (1-3), Uses Knowledge (0-1), Overall Quality (1-5). Three annotators labeled each response.
The task instructions were very detailed in order to minimize subjectivity in the quality annotations. For example, individuals may differ in their definition of Interesting (e.g., some individuals find football interesting, others do not). Thus, the instructions contained a clear, albeit somewhat rigid definition, of Interesting. The instructions for Overall Quality annotation, however, were less rigid and therefore those annotations contain some amount of annotator-specific subjectivity.
The data collection and experiments with PersonaChat were carried out to assess the generality of the USR metric. As such, the annotation questions used were specifically tailored to TopicalChat, but are still suitable for PersonaChat.
3.1 Topical-Chat Dataset
The Topical-Chat dataset (Gopalakrishnan et al., 2019) is a large collection of human-human knowledge-grounded open-domain conversations that consists of 11,319 dialogs and 248,014 utterances. Following the same experimental setup as Gopalakrishnan et al. (2019), heuristics are employed to identify the most relevant fact for each response. As such, the task is to produce a response conditioned on both a dialog context and a fact.
3.2 PersonaChat Dataset
The PersonaChat dataset (Zhang et al., 2018) is a corpus of human-human persona-conditioned conversations that consists of 10,907 dialogs and 162,064 utterances. Each worker is asked to condition their responses on a persona, which we consider to be analogous to the facts in the Topical
http://shikib.com/usr
Figure 1: On the Topical-Chat corpus, six responses are obtained for each dialog context. Four use the trained Transformer model with different decoding strategies. One is a new human-generated response. One is the original ground-truth. A similar setup was employed for PersonaChat, albeit with different models.
Chat corpus.
3.3.1 Topical-Chat Models
A Transformer (Vaswani et al., 2017) is trained to produce the response,  $r$ , conditioned on dialog context,  $c$ , and fact,  $f$ . The input to the transformer is the concatenation of  $c$  and  $f$ , similar to Gopalakrishnan et al. (2019). The transformer consists of 6 layers, a hidden size of 512, randomly-initialized word embeddings of size 300, a dropout rate of 0.1 and it is trained for 50 epochs.
A single Transformer model is trained, which matches the automatic metrics reported by Gopalakrishnan et al. (2019). Different decoding strategies are used to obtain four different outputs from this model. In addition to standard argmax sampling, nucleus sampling (Holtzman et al., 2019) is used at three different rates:  $p = \{0.3, 0.5, 0.7\}$ . The outputs from these four decoding strategies are listed with the original ground-truth utterance and a new human-generated response, for a total of six responses for each context, as shown in Figure 1.
3.3.2 PersonaChat Models
Three models were used to generate system outputs: a sequence-to-sequence model (Seq2Seq), an LSTM language model (LM) and a Key-Value Profile Memory Network (KV-MemNN). We use the pre-trained models provided in  $\mathrm{ParlAI}^2$  for the ConvAI2 competition (Dinan et al., 2019).
A fourth open-source model was also used to produce output for quality annotation, however it
was ultimately excluded from the released dataset and experiments due to possible data leakage.
3.4 Annotation
Quality annotation was performed by six dialog researchers. Using a crowdsourcing platform, such as Amazon Mechanical Turk (AMT), would have allowed for more efficient and scalable annotation. However, crowdsourcing was not used because (1) the annotation instructions are lengthy, (2) a preliminary annotation pass was carried out, followed by a group discussion, (3) having many annotations from a few annotators allows examination of annotator-specific subjectivity.
Annotators were provided with a set of instructions (Appendix A). A small preliminary annotation pass was carried out, with each individual annotating 5 dialog contexts (for a total of 30 responses). The inter-annotator agreement was computed for each of the questions. The instructions were refined after the preliminary pass and a discussion meeting (e.g., Maintains Context was changed to be a 3-point rating instead of a 2-point rating). After the instructions were modified, the full annotation pass was carried out.
Each response was rated according to the qualities mentioned at the beginning of this section. Instructions for each of qualities are summarized below:
- Understandable (0 - 1): Is the response understandable given the previous context?
- Natural (1 - 3): Does the response seem to be something that a person would naturally say?
- Maintains Context (1 - 3): Does the response serve as a valid continuation of the preceding conversation?
- Interesting (1 - 3): Is the response dull or interesting?
- Uses Knowledge (0 - 1): Given the fact that the response is conditioned on, how well does the response use that fact?
- Overall Quality (1 - 5): Given your answers above, what is your overall impression of the quality of this utterance?
The instructions contained detailed descriptions and examples of what constitutes a response in each category (e.g., what makes a response score
$^{2}$ https://github.com/facebookresearch/ParlAI/tree/master/projects/convai2
Metric	Spearman	Pearson
Topical-Chat
Understandable	0.5102	0.5102
Natural	0.4871	0.4864
Maintains Context	0.5599	0.5575
Interesting	0.5811	0.5754
Uses Knowledge	0.7090	0.7090
Overall Quality	0.7183	0.7096
PersonaChat
Understandable	0.2984	0.2984
Natural	0.4842	0.4716
Maintains Context	0.6125	0.6130
Interesting	0.4318	0.4288
Uses Knowledge	0.8115	0.8115
Overall Quality	0.6577	0.6603
Table 1: Inter-annotator agreement for all the metrics. For all the correlations presented in this table,  $p < 0.01$ .
2 on Maintains Context). These instructions were written to minimize subjectivity in the annotations, which results in clear, agreed upon definitions.
For Topical-Chat, the full annotation consisted of 60 dialog contexts randomly sampled from the frequent test set, for a total of 360 responses scored on six different qualities. For PersonaChat, 60 dialog contexts were sampled from the ConvAI2 validation set, with a total of 300 responses scored on six different qualities. Each response was labeled by three different annotators. Annotators were randomly assigned to each dialog context.
Inter-annotator agreements for the different ratings across both datasets are presented in Table 1. The correlation between each pair of annotations is computed and the average correlation over all the pairs is reported. Correlation is used instead of Cohen's Kappa in order to better account for the ordinal nature of the ratings (i.e., 4 should correlate better with 5 than 1), and to maintain consistency with the evaluation of the automatic metrics. Most inter-annotator correlations are above 0.4, which indicates moderate to strong agreement. The low agreement for Understandable on PersonaChat is likely a consequence of the simple language in the dataset. Most responses are understandable, except for those requiring background knowledge (e.g., that 'cod' is an acronym for 'Call of Duty'). Since the annotators have differing background
knowledge, the few occasions where they fail to understand an utterance will differ, hence the lower agreement. The agreement for Overall Quality is relatively high (0.71 for Topical-Chat and 0.66 for PersonaChat) which suggests that any ambiguity in the specific dialog qualities is mitigated when the annotator is asked for an overall impression.
Table 2 presents the scores for the different systems on each of the six qualities. Across both datasets and all qualities, the new human generated response strongly outperforms all other response types, even the original ground truth. This may be because the new human generated response was written with this quality annotation in mind, and as such is optimized for turn-level evaluation. On the other hand, the workers who produced the original ground-truth response, were more concerned with the quality of the overall dialog than with the quality of each individual response.
On the Topical-Chat corpus, argmax decoding has a moderately higher performance over the nucleus sampling (Holtzman et al., 2019) methods. This should not be taken as an indication that argmax decoding is the superior method, since the hyperparameters (e.g., temperature) were not tuned for nucleus sampling. It should be noted that the objective was not to train and evaluate the best performing models, but instead to produce responses of varying qualities and obtain accurate human judgements of these responses.
A regression was trained to map from the five ratings to the overall score in order to analyze the relationship between them. For better interpretability of the regression weights, the scores were normalized (using z-score) before training the regression. For better interpretability, a softmax was computed over the weights. Since individuals may differ in their definition of a good response, a specific regression is trained for each of the five annotators who labeled responses for the Topical-Chat corpus. Figure 2 displays the weights attributed to each of the five qualities by each of the annotators.
Annotators attributed different weights to the specific features. For example, A3 emphasized naturalness while A2 paid more attention to whether a response was grounded on knowledge. Despite the differences across annotators, a good response was generally expected to be natural, maintain context, and be interesting. These annotator-specific weights demonstrate that individuals define good dialog differently. Future work could explore per
System	Und (0-1)	Nat (1-3)	MCtx (1-3)	Int (1-3)	UK (0-1)	OQ (1-5)
Topical-Chat
Original Ground-Truth	0.95	2.72	2.72	2.64	0.72	4.25
Argmax Decoding	0.60	2.08	2.13	1.94	0.47	2.76
Nucleus Sampling (0.3)	0.51	2.02	1.90	1.82	0.42	2.40
Nucleus Sampling (0.5)	0.48	1.92	1.93	1.72	0.34	2.29
Nucleus Sampling (0.7)	0.52	2.01	1.87	1.80	0.37	2.39
New Human Generated	0.99	2.92	2.93	2.90	0.96	4.80
PersonaChat
Original Ground-Truth	0.99	2.89	2.82	2.67	0.56	4.36
Language Model	0.97	2.63	2.02	2.24	0.08	2.98
LSTM Seq2Seq	0.92	2.64	2.49	2.29	0.47	3.47
KV-MemNN	0.93	2.70	2.18	2.56	0.17	3.25
New Human Generated	1.00	2.97	2.88	2.87	0.96	4.80
Table 2: Average scores for the six different responses, on the six quality: Understandable, Natural, Maintains Context, Interesting, Uses Knowledge and Overall Quality.
Figure 2: Weight attributed to each of the five specific metrics by each annotator, when labeling Overall Quality. Lighter colors signify more weight.
sonalized dialog evaluation wherein the evaluation metric is tailored to a specific individual.
A potential criticism of this quality annotation could be that certain dialog qualities are missing. To address concerns about the completeness of the set of five qualities, a regression can be trained to produce the overall score conditioned on the quality ratings. The Spearman correlation between the predicted score and the original overall score is 0.9654, which signifies that the set of qualities is thorough and contains enough information to reflect the overall quality of the response.
4 Automatic Metrics
This section describes the automatic metrics explored for evaluating generated responses. Section 4.1 describes several existing metrics that were studied. Section 4.2 presents USR, a novel unsupervised and reference-free metric.
4.1 Baseline Metrics
Several existing and easily-applicable metrics for dialog evaluation are compared. the list of available metrics is not exhaustive. Only the most commonly used and the most accessible are addressed.
F-1 score computes the word-overlap between the generated response and the ground-truth, by taking the harmonic mean of the precision and recall. It is one of the four metrics used by the creators of the Topical-Chat dataset (Gopalakrishnan et al., 2019), along with perplexity and unique unigram/bigram counts. Dinan et al. (2019) described a simple adversarial example that attains a high F-1 score on PersonaChat. We produce a similar example for the Topical-Chat dataset and find that always outputting a concatenation of the ten most common tokens in the dataset ("i the , that a to it is of") attains an F-1 score of 25.6 which is a +3.6 improvement over the Transformer presented by Gopalakrishnan et al. (2019).
BLEU (Papineni et al., 2002) is a well-known word overlap metric that computes n-gram precision between the generated sequence and the reference. Because precision favors shorter sentences, BLEU also adds a brevity penalty that punishes shorter sentences. BLEU has been found to correlate poorly with human judgment (Liu et al., 2016; Lowe et al., 2017; Gupta et al., 2019).
METEOR (Denkowski and Lavie, 2014) was designed as an improvement on BLEU using a harmonic mean of precision and recall, as well as stemming and synonyms.
ROUGE-L (Lin, 2004) identifies the longest
common subsequence between the generated and reference sequence to better account for sentence-level structure when computing word overlap.
Greedy Matching (Rus and Lintean, 2012) is an embedding-based metric that greedily matches each word in the generated sequence to a reference word based on the cosine similarity of their embeddings. The final score is then an average over all the words in the generated sequence.
**Embedding Average (Wieting et al., 2015)** computes a sentence embedding for both the generated sequence and the ground-truth response by taking an average of word embeddings. The score is then a cosine similarity of the average embedding for both the generated and reference sequence.
Vector Extrema (Forgues et al., 2014) follows a similar setup to Embedding Average, where the score is the cosine similarity between sentence embeddings. Rather than taking an average over word embeddings, this method identifies the maximum value for each dimension of the word embedding. Taking the maximum is motivated by the idea that common words will be de-emphasized as they will be closer to the origin. Vector Extrema has been shown to perform better on dialog tasks than other metrics (Gupta et al., 2019; Liu et al., 2016).
Skip-Thought (Kiros et al., 2015) uses a recurrent neural network to produce a sentence-level embedding for the generated and reference sequences. A cosine similarity is then computed between the two embeddings. The implementation provided by Sharma et al. (2017) is used.
BERTScore (Zhang et al., 2019) uses a pretrained BERT (Devlin et al., 2018) model to greedily match each word in a reference response with one word in the generated sequence. By doing so, it computes the recall of the generated sequence. BERTScore was shown to have strong system-level and segment-level correlation with human judgment on several machine translation and captioning tasks. However, although it is a more sophisticated metric, it still compares word similarity between a reference and a generated sequence. While this method may work well for tasks where there is a limited space of outputs for each input (e.g., captioning, translation), it is ineffective at dealing with the one-to-many nature of dialog.
4.2 Proposed Metric
This section proposes describes the USR metric, an unsupervised, reference-free evaluation metric
for dialog. USR leverages pre-trained language models, specifically RoBERTa (Liu et al., 2019), to measure properties of dialog. USR is designed to be reference-free because there is no one right answer due to the inherent one-to-many nature of dialog (Zhao et al., 2017).
Several sub-metrics were developed for the different qualities of dialog (e.g., Natural, Interesting, Uses Knowledge). While USR measures the overall quality of a response, its sub-metrics assess specific dialog qualities and therefore facilitate better understanding of a model's performance.
4.2.1 Masked Language Modelling Metric
The masked language modelling (MLM) metric uses a fine-tuned RoBERTa (Liu et al., 2019) model to estimate the likelihood of a response. RoBERTa is pre-trained on a massive amount of English data and fine-tuned on the corpus being evaluated (either Topical-Chat or PersonaChat), making it capable of identifying unnatural and incorrect responses. The likelihood estimated by the fine-tuned RoBERTa model is used as an automatic metric for evaluating the understandability and naturalness of responses.
The RoBERTa-base model (Liu et al., 2019) is fine-tuned on the training set of the Topical-Chat corpus (Gopalakrishnan et al., 2019) using the implementation open-sourced by Wolf et al. (2019a). The language model is fine-tuned on only the dialog, without any of the facts, for a single epoch.
RoBERTa uses both past and future context to predict a probability distribution for a masked word. The input sequence to MLM is a concatenation of a dialog context,  $c$ , and a response,  $r$ . One word at a time, each word in  $r$  is masked and its log likelihood is computed. Given the masked log-likelihood for the  $i$ -th word of  $r$  as  $l_i$ , the value of the metric is then computed to be  $-\sum_{i}^{|r|}l_{i}$ . Figure 3 visualizes this process.
4.2.2 Dialog Retrieval Metrics