Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper propose a new evaluation metric for dialogue systems, and show it has a higher correlation with human annotation. I agree the MT based metrics like BLEU are too simple to capture enough semantic information, but the metric proposed in this paper seems to be too compliciated to explain.

On the other hand, we could also use equation 1 as a retrieval based dialogue system. So what is suggested in this paper is basically to train one dialogue model to evaluate another model. Then, the high-level question is why we should trust this model? This question is also relevant to the last item of my detail comments.

Detail comments:

- How to justify what is captured/evaluated by this metric? In terms of BLEU, we know it actually capture n-gram overlap. But for this model, I guess it is hard to say what is captured. If this is true, then it is also difficult to answer the question like: will the data dependence be a problem?
- why not build model incrementally? As shown in equation (1), this metric uses both context and reference to compute a score. Is it possible to show the score function using only reference? It will guarantee this metric use the same information source as BLEU or ROUGE. 
- Another question about equation (1), is it possible to design the metric to be a nonlinear function. Since from what I can tell, the comparison between BLEU (or ROUGE) and the new metric in Figure 3 is much like a comparison between the exponential scale and the linear scale.
- I found the two reasons in section 5.2 are not convincing if we put them together. Based on these two reasons, I would like to see the correlation with average score. A more reasonable way is to show the results both with and without averaging. 
- In table 6, it looks like the metric favors the short responses. If that is true, this metric basically does the opposite of BLEU, since BLEU will panelize short sentences. On the other hand, human annotators also tends to give short respones high scores, since long sentences will have a higher chance to contain some irrelevant words. Can we eliminate the length factor during the annotation? Otherwise, it is not surprise that the correlation.
review_title: Why we should use yet another dialogue system as an evaluation metric?

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses the issue of how to evaluate automatic dialogue responses. This is an important issue because current practice to automatically evaluate (e.g. BLEU, based on N-gram overlap, etc.) is NOT correlated well with the desired quality (i.e. human annotation). The proposed approach is based on the use of an LSTM encoding of dialogue context, reference response and model response with appropriate scoring, with the essence of training one dialogue model to evaluate another model. However, the proposed solution depends on a reasonably good dialogue model to begin with, which is not guaranteed, rendering the new metric possibly meaningless. 
review_title: The idea is good, and the problem to address is very important. However, the proposed solution is short of desired one.

final_score: rating: null
review_content: You give some qualitative results. But it would be really helpful to perform a more in-depth failure case analysis (e.g. give an overview of all important cases where ADEM is not well aligned with human judgement)
review_title: Please comment on more failure cases

final_score: rating: null
review_content: 
Learning a more powerful evaluation model is a nice idea. Could you please comment on the (expected) generalization ability of ADEM beyond the particular dataset that you happen to use in this paper? Ideally you would conduct experiments with other datasets without retraining ADEM on those datasets.
review_title: How well does ADEM generalize to other settings?

final_score: rating: null
review_content: Is it possible to add more analysis on these hyper-parameters? For example, how these hyper-parameters influence the correlation with human judgements?
review_title: About hyper-parameters

final_score: rating: 7: Good paper, accept
review_content: 
Overall the paper address an important problem: how to evaluate more appropriately automatic dialogue responses given the fact that current practice to automatically evaluate (BLEU, METEOR, ...) is often insufficient and sometimes misleading. The proposed approach using an LSTM-based encoding of dialogue context, reference response and model response(s) that are then scored in a linearly transformed space. While the overall approach is simple it is also quite intuitiv and allows end-to-end training. As the authors rightly argue simplicity is a feature both for interpretation as well as for speed. 

The experimental section reports on quite a range of experiments that seem fine to me and aim to convince the reader about the applicability of the approach. As mentioned also by others more insights from the experiments would have been great. I mentioned an in-depth failure case analysis and I would also suggest to go beyond the current dataset to really show generalizability of the proposed approach. In my opinion the paper is somewhat weaker on that front that it should have been.

Overall I like the ideas put forward and the approach seems sensible though and the paper can thus be accepted. 
review_title: The main idea of the paper is to learn the evaluation of dialogue responses in order to overcome limitations of current schemes such as BLEU

final_score: rating: null
review_content: How may ADEM as proposed in this paper be possibly used in training dialogue systems, as much as BLEU has been used to help train MT systems better?
review_title: ADEM evaluation metric used for training

Meta-review: Noting the authors' concern about one of the reviewers, I read the paper myself and offer my own brief review.
 
 Evaluation is an extremely important question that does not get enough attention in the machine learning community, so the authors' effort is welcomed. The task that the authors are trying to evaluate is especially hard; in fact, it is not even clear to me how humans make these judgments. The low kappa scores on some of the non-"overall" dimensions, and only moderate agreement on "overall," are quite worrisome. What makes a good "chat" dialogue? The authors seem not to have qualitatively grappled with this key question, rather defining it empirically as "whatever our human judges think it is." This is, I think the deepest flaw of the work; the authors are rushing to automate evaluation without taking the time to ponder what good performance actually is. 
 
 That aside, I think the idea of automatic evaluation as a modeling problem is worth studying. The authors note that this has been done for other problems, such as machine translation. They give only a cursory discussion of this prior work, however, and miss the seminal reference, "Regression for sentence-level MT evaluation with pseudo references," Albrecht, Joshua, and Rebecca Hwa, ACL 2007.
 
 The paper would be much stronger with some robustness analysis; does the quality of the evaluation hold up if the design decisions are made differently, if less data are used to estimate the evaluation model, etc.? How does it hold up across datasets, and across different types of dialogue systems? As a methodological note, there are a lot of significance tests here and no mention of any attempt to correct for this (e.g., Bonferroni, FDR, etc.).
 
 As interesting as the ideas here are, I can't see the dialogue community rushing to adopt this approach to evaluation based on the findings in this paper. I do think that the ideas it presents should be hashed out in a public forum sooner rather than later, and therefore recommend it as one of a few papers to be presented at the workshop.
Decision: Invite to Workshop Track