Title: Towards an automatic Turing test: Learning to evaluate dialogue responses
Authors: Ryan Lowe
Michael Noseworthy
Iulian V. Serban
Nicolas Angelard-Gontier
Yoshua Bengio
Joelle Pineau
Venue: ICLR 2017
Keywords: Natural language processing
Applications

Abstract: Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem.
Unfortunately, existing automatic evaluation metrics are biased and correlate very poorly with human judgements of response quality (Liu et al., 2016). Yet having an accurate automatic evaluation procedure is crucial for dialogue research, as it allows rapid prototyping and testing of new models with fewer expensive human evaluations. In response to this challenge, we formulate automatic dialogue evaluation as a learning problem. We present an evaluation model (ADEM) that learns to predict human-like scores to input responses, using a new dataset of human response scores. We show that the ADEM model's predictions correlate significantly, and at level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and system-level. We also show that ADEM can generalize to evaluating dialogue models unseen during training, an important step for automatic dialogue evaluation.

TL;DR: We propose a model for evaluating dialogue responses that correlates significantly with human judgement at the utterance-level and system-level.

Review discussion:
Learning a more powerful evaluation model is a nice idea. Could you please comment on the (expected) generalization ability of ADEM beyond the particular dataset that you happen to use in this paper? Ideally you would conduct experiments with other datasets without retraining ADEM on those datasets.
Thanks for your question.
Indeed, generalization ability to other datasets would be a strong additional property of the ADEM model. We have not extensively tested this, since as you mention it would require collecting another dataset with human labels. In general, I would expect some generalization to domains that have similar structure/ topics that are discussed, such as Reddit;
however, I would not expect it to generalize to completely different domains such as Ubuntu, which contains a large amount of technical vocabulary. We leave these investigations to future work.
We feel that presenting an evaluation model for the Twitter domain is a significant contribution, since the majority of recent machine learning chatbots have been trained in this domain (or other microblogging platforms, see e.g. Ritter et al. (2011), Sordoni et al. (2015), Shang et al. (2015), Serban et al. (2016), etc.). We also think that the inter-model generalization, shown in the results, crucially demonstrates some of this generalization ability between models.
You give some qualitative results.
But it would be really helpful to perform a more in-depth failure case analysis (e.g. give an overview of all important cases where ADEM is not well aligned with human judgement)
Thanks for your question. I agree this would be really nice to show. We plan to add these results to the paper (although it will have to wait at least a week since the majority of the authors will be at NIPS). From our initial observations, I would say that the model generally fails in much the same way that neural language models fail -- it often misses semantic similarities between utterances, as it doesn't 'understand' natural language in the same way that humans do. This is unsurprising, as ADEM uses representations learned by the VHRED model.
While this will introduce some variance in the evaluation scores at the utterance-level, the hope is that this is mostly 'averaged out' when computing system-level performance; our system-level correlation results indicate that this is likely the case.
Is it possible to add more analysis on these hyper-parameters?
For example, how these hyper-parameters influence the correlation with human judgements?
Thanks for your question.
We found experimentally that ADEM was quite robust to hyperparameter settings.
In particular, when varying our value of k (PCA projection) the method was robust in the range of [5, 200], reducing the correlation by ~0.05 at the tail ends of this range. Having a L1-regularization penalty improved our correlation by about 0.02. We also experimented with L2-regularization, however we did not find this gave any improvements. In general of course, performance suffered once the regularization was made too strong. We did not experiment with changing the recurrent encoder architecture, instead using the best performing model for the task of dialogue generation with VHRED. We can of course add these details to the paper (although this may take a week since most of the authors are at NIPS).
How may ADEM as proposed in this paper be possibly used in training dialogue systems, as much as BLEU has been used to help train MT systems better?
Thanks for your question.
I think using ADEM for training dialogue models is a very interesting application, and we plan to investigate this in future work. It has been argued (e.g. here: https://arxiv.org/pdf/1611.06216.pdf) that maximum likelihood is an insufficient objective for training neural conversation models, and ADEM may help in this regard.
One potential way to integrate the ADEM signal into training is via policy gradient methods from reinforcement learning.
For instance, this has been done using REINFORCE (Ranzato et al. (2015)) and actor-critic (Bahdanau et al. (2016)) for machine translation using BLEU score. In this case, the BLEU score could directly be replaced by the ADEM score, and the same general architectures could be used (although it is possible that different architectures are more effective for dialogue than machine translation).
Overall the paper address an important problem: how to evaluate more appropriately automatic dialogue responses given the fact that current practice to automatically evaluate (BLEU, METEOR, ...) is often insufficient and sometimes misleading. The proposed approach using an LSTM-based encoding of dialogue context, reference response and model response(s) that are then scored in a linearly transformed space.
While the overall approach is simple it is also quite intuitiv and allows end-to-end training. As the authors rightly argue simplicity is a feature both for interpretation as well as for speed.
The experimental section reports on quite a range of experiments that seem fine to me and aim to convince the reader about the applicability of the approach. As mentioned also by others more insights from the experiments would have been great. I mentioned an in-depth failure case analysis and I would also suggest to go beyond the current dataset to really show generalizability of the proposed approach. In my opinion the paper is somewhat weaker on that front that it should have been.
Overall I like the ideas put forward and the approach seems sensible though and the paper can thus be accepted.
We thank the reviewer for their comments. The recommendations made by the reviewer are good ones -- we will certainly update the paper with a more in-depth failure analysis, as indeed this will show where the model performs poorly. Testing the model in other domains will require additional annotated data of response quality for each domain, which is both expensive and time consuming --  thus we cannot guarantee that this will be completed by the deadline.
This paper propose a new evaluation metric for dialogue systems, and show it has a higher correlation with human annotation. I agree the MT based metrics like BLEU are too simple to capture enough semantic information, but the metric proposed in this paper seems to be too compliciated to explain.
On the other hand, we could also use equation 1 as a retrieval based dialogue system.
So what is suggested in this paper is basically to train one dialogue model to evaluate another model.
Then, the high-level question is why we should trust this model? This question is also relevant to the last item of my detail comments.
Detail comments:
- How to justify what is captured/evaluated by this metric?
In terms of BLEU, we know it actually capture n-gram overlap.
But for this model, I guess it is hard to say what is captured. If this is true, then it is also difficult to answer the question like: will the data dependence be a problem?
- why not build model incrementally? As shown in equation (1), this metric uses both context and reference to compute a score. Is it possible to show the score function using only reference? It will guarantee this metric use the same information source as BLEU or ROUGE.
- Another question about equation (1), is it possible to design the metric to be a nonlinear function. Since from what I can tell, the comparison between BLEU (or ROUGE) and the new metric in Figure 3 is much like a comparison between the exponential scale and the linear scale.
- I found the two reasons in section 5.2 are not convincing if we put them together. Based on these two reasons, I would like to see the correlation with average score. A more reasonable way is to show the results both with and without averaging.
- In table 6, it looks like the metric favors the short responses. If that is true, this metric basically does the opposite of BLEU, since BLEU will panelize short sentences.
On the other hand, human annotators also tends to give short respones high scores, since long sentences will have a higher chance to contain some irrelevant words. Can we eliminate the length factor during the annotation?
Otherwise, it is not surprise that the correlation.
We thank the reviewer for their comments. We respond to the specific questions and criticisms raised by the reviewer below.
Q: the metric proposed in this paper seems to be too compliciated to explain.
A: By “too complicated to explain”, we assume the reviewer means the method is not as interpretable as scores such as BLEU (we maintain that the method itself is not hard to explain: it is simply a linear combination of utterance embeddings, as detailed in Section 4). It is certainly true that a learned metric with neural networks is less interpretable than one that is composed of simple statistics (such as BLEU).
But, conversely, such a learned metric is far more powerful than metrics such as BLEU.
In fact, interpretability of an evaluation metric is only useful so long as the metric *actually works*;
yet it has been demonstrated in Liu et al. (2016) (and further demonstrated in our paper) that BLEU and other word-overlap variants *do not work* for evaluating the quality of dialogue responses.
Thus, in this paper, we make a first step towards a metric obtaining correlation with human judgements by leveraging neural networks. We do not claim it should be the only metric used to evaluate dialogue responses, but that it should be used in addition to existing (more interpretable) metrics. We will add the above argument to the paper.
The idea of using learned objective functions has proved enormously successful in the literature -- generative adversarial networks (GANs) are in fact generative models in which the generator is trained with a learned objective function.
Although the learned objective (discriminator) is not interpretable, GANs are among the state-of-the-art generative models. Since these approaches have been successful in a variety of domains, it is natural to explore similar approaches for learning dialogue evaluation metrics, which is an open problem.
Q: On the other hand, we could also use equation 1 as a retrieval based dialogue system.
So what is suggested in this paper is basically to train one dialogue model to evaluate another model.
Then, the high-level question is why we should trust this model?
A: To be clear, the evaluation model is **not** a dialogue retrieval model. The fundamental difference between a retrieval model and our proposed evaluation model, is that the evaluation model has access to the *reference response*.
In other words, the evaluation model is conditioned on one reference response generated by humans.
As such, the evaluation model has the ability to compare a model’s response to a response that is known to be good, making its job significantly easier compared to a retrieval model. One could also easily extend our framework to incorporate the use of multiple reference responses if they are available, which would be considered ‘cheating’ for retrieval models. We will clarify this in the paper.
Why should we trust this model?
Unlike existing automatic metrics for dialogue response evaluation, it correlates strongly with human judgements. We believe this is sufficient to add the ADEM score to the scores that are normally reported for dialogue evaluation.
Q: How to justify what is captured/evaluated by this metric? will the data dependence be a problem?
A: We respond to the first question above (re: interpretability). To determine whether data dependence is a problem, this must be done empirically. We believe our generalization results in Table 5 suggest that data dependence is unlikely in the Twitter domain (which is the domain that the majority of conversational models in the literature have been trained on -- e.g. Ritter et al. (2011), Sordoni et al. (2015), Serban et al. (2016), etc.). We leave the investigation of other domains to future work.
Q: why not build model incrementally?
A: This is a good idea, and we thank the reviewer for their suggestion -- we will report the results for each component of the model in an updated version of the paper.
Q: Another question about equation (1), is it possible to design the metric to be a nonlinear function. Since from what I can tell, the comparison between BLEU (or ROUGE) and the new metric in Figure 3 is much like a comparison between the exponential scale and the linear scale.
A: One could indeed use a nonlinear function in equation (1), but this has nothing to do with Figure 3. Figure 3 simply shows an (automatically scaled) scatterplot of metric scores against human scores, along with a line of best fit. Note that the correlation scores reported in the paper (Spearman, Pearson) are independent of the scaling of the metrics -- thus, our better performance is not a matter of scaling, but is because ADEM is better at predicting human response scores than word-overlap metrics.
Q: I found the two reasons in section 5.2 are not convincing if we put them together. Based on these two reasons, I would like to see the correlation with average score.
A: Unfortunately, we cannot compute the correlation with the average score for our dataset, because the vast majority of questions were only evaluated by one human annotator. We did this because we had a fixed financial budget, and wanted to collect the maximum number of examples under this budget, since a human score on a new response is more informative than on a response that has already been evaluated. Since there is only one score per response, this will result in higher variance, and lower correlations.
Q: In table 6, it looks like the metric favors the short responses. human annotators also tends to give short responses high scores, since long sentences will have a higher chance to contain some irrelevant words. Can we eliminate the length factor during the annotation?
A: Yes, humans appear to favour shorter responses, which is actually not desirable for a learned evaluation model. We do not currently analyze this in the paper, making the implicit assumption that the humans scoring the responses are always correct.
But we will work to quantify the role of the length factor in the human score predictions, and will add this to an updated version of the paper (and derive methods to correct it if necessary).
This paper addresses the issue of how to evaluate automatic dialogue responses. This is an important issue because current practice to automatically evaluate (e.g. BLEU, based on N-gram overlap, etc.) is NOT correlated well with the desired quality (i.e. human annotation). The proposed approach is based on the use of an LSTM encoding of dialogue context, reference response and model response with appropriate scoring, with the essence of training one dialogue model to evaluate another model.
However, the proposed solution depends on a reasonably good dialogue model to begin with, which is not guaranteed, rendering the new metric possibly meaningless.
We thank the reviewer for their comments. We are glad the reviewer agrees that we are addressing an important problem.
However, we strongly oppose the only point of criticism offered by the reviewer.
The reviewer states: “However, the proposed solution depends on a reasonably good dialogue model to begin with, which is not guaranteed, rendering the new metric possibly meaningless.”
First of all, the statement that our solution “depends on a good dialogue model” is false -- as we show in the experiments, ADEM works almost equally well using tweet2vec, which is a freely available sentence embedding method.
Second, while our evaluation procedure does require a method of producing embeddings for utterances (of which a dialogue model is only one example), we strongly maintain that this does not “render the metric possibly meaningless”.
In fact, we maintain it is not even a weakness of the paper, for the following reasons:
1) We conduct our evaluations on a dataset (the Twitter Corpus) that has seen a huge amount of use in the literature (e.g. Ritter et al. (2011), Sordoni et al. (2015), 2 papers by Serban et al. (2016),  Li et al. (2015), Li et al. (2016), etc..). It is obvious that, in this setting, there are an abundance of dialogue models (and embedding methods), and we simply use one such published model (VHRED) for our procedure. Since we show strong correlation with human judgement for Twitter, our model is clearly useful for evaluating these models (and future models that will be built on this dataset).
Thus, the statement that our metric is “meaningless” is unfounded.
2) More generally, the entire point of our evaluation procedure is to evaluate dialogue models that are *data-driven*. If there is not enough data in some domain to train a dialogue model (which is what the reviewer is implying when they say a dialogue model may not be available), then there is no point to have an evaluation procedure for this domain! The only exception to this is if someone wants to craft a *rule-based* system in a domain that both (a) has very little data, and (b) is in a highly specialized area that is different from existing datasets, such that no transfer learning can be used. The fact that our evaluation procedure would not work in this setting is hardly a limitation (we do not claim our evaluation model would work in this setting, but we will clarify this in the paper).
Given that we examine our method using a range of experiments in a very popular domain, showing that our method is a strong first step towards addressing an important open problem (as admitted by the reviewer), we find the claim that our metric is “possibly meaningless” to be without basis. If the reviewer has specific concerns about the evaluation procedure in the paper we would be happy to address them.
We thank the reviewers for taking the time to comment. All of the reviewers agree that we are tackling a very important problem (dialogue response evaluation) that has been under-studied in the literature. We would like to emphasize to all reviewers that, while our paper is not perfect, we believe it is a strong first step towards addressing this problem.
Further, we do not claim that our proposed evaluation metric should be the *only* automatic metric to be used, but rather we recommend it be *added* to existing evaluation metrics (since, unlike existing metrics, it actually correlates with human judgements). We are of the opinion that keeping the ‘good’ unpublished while waiting for the ‘perfect’ is not a productive way forward for science.
We thank the reviewers again for their feedback.
While our detailed rebuttal to each reviewer is written as as a direct response to the review, we have now incorporated many of the reviewers' comments into an updated version of the paper.
In particular, we have addressed the following items:
1) We have made several alterations to alleviate possible misunderstandings from reading the paper. Most notably, we have added a paragraph to Section 4 explaining why the ADEM model is not a dialogue retrieval model.
2) As recommended by Reviewer #1, we have produced two additional results that help clarify the workings of different parts of the model. In Table 3, we show correlations for the ADEM model when it can only compare the model response to the context (C-ADEM), and when it can only compare the model response to the reference response (R-ADEM). It seems that ADEM is mostly using the comparison to the reference response in order to assign its score. This makes sense, since the reference response is often closer semantically to the model response than the context, and goes to further illustrate how our model is different from a dialogue retrieval model (which could only use the context, i.e. C-ADEM).
3) We provide new correlation results for word overlap metrics on the dataset from Liu et al. (2016).
In particular, we standardized the pre-processing procedure by removing the <first_speaker> token at the beginning of each utterance. Surprisingly, this removes all significant correlation between word-overlap metrics and human judgements.
Thus, Liu et al. were even too optimistic about the performance of BLEU. This helps explain the difference in word-overlap correlation between the datasets: since the Liu et al. dataset was an order of magnitude smaller, it was significantly influenced by pre-processing, and when pre-processing is standardized word-overlap metrics achieve approximately the same correlation (not significant) on both datasets.
4) As also recommended by Reviewer #1, in Section 4 we now discuss the implicit assumption in our model (that humans scoring the responses are absolutely correct) and the drawbacks this assumption could have.
In particular, we discuss how humans may be biased towards ranking shorter responses more highly, since they are often more generic and thus reasonable given any context. We calculate the test set correlation between the response length and score, which at 0.27/0.32 is significant but much lower than our overall ADEM score, indicating that there are many other factors that ADEM is taking into consideration other than response length.
5) As recommended by Reviewer #3, we now conduct a failure analysis of ADEM, which we show in Appendix D. The analysis indicates that there are still many response similarities that ADEM does not pick up on, and there are some cases where the human labels are incorrect or noisy.
We hope that these modifications address the concerns of the reviewers. We are happy to address any further concerns the reviewers might have that were not addressed by these changes or by our rebuttals.

Meta-review:
role: PC
data: comment: Noting the authors' concern about one of the reviewers, I read the paper myself and offer my own brief review.
 
 Evaluation is an extremely important question that does not get enough attention in the machine learning community, so the authors' effort is welcomed. The task that the authors are trying to evaluate is especially hard; in fact, it is not even clear to me how humans make these judgments. The low kappa scores on some of the non-"overall" dimensions, and only moderate agreement on "overall," are quite worrisome. What makes a good "chat" dialogue? The authors seem not to have qualitatively grappled with this key question, rather defining it empirically as "whatever our human judges think it is." This is, I think the deepest flaw of the work; the authors are rushing to automate evaluation without taking the time to ponder what good performance actually is. 
 
 That aside, I think the idea of automatic evaluation as a modeling problem is worth studying. The authors note that this has been done for other problems, such as machine translation. They give only a cursory discussion of this prior work, however, and miss the seminal reference, "Regression for sentence-level MT evaluation with pseudo references," Albrecht, Joshua, and Rebecca Hwa, ACL 2007.
 
 The paper would be much stronger with some robustness analysis; does the quality of the evaluation hold up if the design decisions are made differently, if less data are used to estimate the evaluation model, etc.? How does it hold up across datasets, and across different types of dialogue systems? As a methodological note, there are a lot of significance tests here and no mention of any attempt to correct for this (e.g., Bonferroni, FDR, etc.).
 
 As interesting as the ideas here are, I can't see the dialogue community rushing to adopt this approach to evaluation based on the findings in this paper. I do think that the ideas it presents should be hashed out in a public forum sooner rather than later, and therefore recommend it as one of a few papers to be presented at the workshop.

Decision: Invite to Workshop Track