Title: Learning End-to-End Goal-Oriented Dialog
Authors: Antoine Bordes
Y-Lan Boureau
Jason Weston
Venue: ICLR 2017

Abstract: Traditional dialog systems used in goal-oriented applications require a lot of domain-specific handcrafting, which hinders scaling up to new domains. End- to-end dialog systems, in which all components are trained from the dialogs themselves, escape this limitation. But the encouraging success recently obtained in chit-chat dialog may not carry over to goal-oriented settings. This paper proposes a testbed to break down the strengths and shortcomings of end-to-end dialog systems in goal-oriented applications. Set in the context of restaurant reservation, our tasks require manipulating sentences and symbols, so as to properly conduct conversations, issue API calls and use the outputs of such calls. We show that an end-to-end dialog system based on Memory Networks can reach promising, yet imperfect, performance and learn to perform non-trivial operations. We confirm those results by comparing our system to a hand-crafted slot-filling baseline on data from the second Dialog State Tracking Challenge (Henderson et al., 2014a). We show similar result patterns on data extracted from an online concierge service.

TL;DR: A new open dataset and testbed for training and evaluating end-to-end dialog systems in goal-oriented scenarios.

Review discussion:
Except from the hand-crafted rule-based system, is it correct that all the baselines you are comparing to (TF-IDF, Nearest Neighbour, Supervised Embeddings) do not take into account word order?
If yes, have you experimented with any machine learning models which do take into account word order information? What do you think the impact of including word order information might be on these results?
Thanks for your remarks.
The baselines cited indeed do not take into account word order. Our goal in choosing those baselines was not to maximize performance, but to provide reasonable simple methods as a gauge. Many more complex models could be tried, and our objective in releasing this dataset is indeed to encourage researchers to test their models on this benchmark so as to compare where there models perform better or worse than others. As suggested by the reviewer, models that use word order would indeed be a a good candidate to try on this testbed.
I'm a little unclear on how typing information works in the "match type" model discussed in the last paragraph before Section 5, especially the 3 conditions: "n if a word is found that appears 1) as a KB entity of that type, 2) in the candidate, and 3) in the input or memory". Could you elaborate with a simple example?
The results seem to indicate that "MemNNs + match type" information perform quite well.
In fact, in the non-OOV case it seems almost perfect for 4/5 tasks, and with OOV for 3/5. In all these, the non-perfect levels are around 75%. Since the goal is to introduce a dataset that could be used to stimulate more future work in this area, do the authors believe it is actually difficult enough?
Thanks for your questions and remarks. For a simple example of match type features, we can use the example in Figure 1.
When the user asks "Can you provide me the address?"
The candidate: "Here it is: The_Place_address" would be represented with the added word for the "address" match type feature
because:
1) the KB contains entry The Place with address the_place_address
2) The_place_address is in the candidate
3) The_place_address appeared earlier in the dialog, so is part of the memory (in the output of the API call).
We hope this example helps clarify match type features.
As for performance: we believe the dataset is indeed difficult enough and useful for all tasks, for three complementary reasons.
First, MemNNs augmented with match type features are among the best performing recent models, and it is far from clear that all end-to-end models would perform as well on the 3 tasks this model solves; we believe it would be useful for researchers using other models to check whether they also solve those tasks.
Second, there is room for progress as long as performance is not 100%.
In fact, two papers have already appeared with new models using this testbed:
Liu and Perez, "Gated End-to-End Memory Networks",  https://arxiv.org/pdf/1610.04211v2.pdf  and Seo et al., "QUERY-REDUCTION NETWORKS FOR QUESTION ANSWERING". These papers reduce the error rate on task 3 to 25% and 8%, respectively, but still do not solve it.
Finally, the per-dialog accuracy remains low (Seo et al. do not report per-dialog accuracy measures).
Thus, we believe this testbed will keep being useful to evaluate the goal-directed capabilities of new end-to-end models.
They are defined in the experiment section. My understanding of the per-response accuracy is that every bot response must be correct, either NL sentences or API calls, so in Table 3, it counts correctness not only on the API call but also on the 6 previous bot replies. For the per-dialog accuracy, a single error in either the bot replies or the API call causes a dialog error, right? For both errors, do you only count the turns specific to the task? What puzzles me  is Table 2: how can the T3 per-dialog accuracy be lower than the T5 per-dialog accuracy.
Another question: have you considered measuring the API call accuracy only for tasks 1 and 2?
Thanks for your comments. Here are some responses:
* The per-response accuracy counts the proportion of bot answers and API calls correctly answered by the system.
So yes for Table 3 it would be 7/7 if the bot accurately predicted all Bots utterances and the API call.
* Indeed, for the per-dialog accuracy, a single error causes a dialog error.
* For both errors, we indeed count only the turns specific to the task.
* T3 dialog accuracy is lower than that of T5 because the difficulty of the task tested by T3 (Displaying options ) is lower when used within full dialogs in T5. In T3, we only keep examples where API calls return at least 3 restaurants where for T5 we keep API calls if they return at least 1 restaurant. If there is only one returned restaurant, the task is much easier (no sorting involved). This is written in the definition of T5 in Section 3.1.1 "Unlike in Task 3, we keep examples if API calls return at least 1 option instead of 3."
* We did not record the API call accuracy specifically but we assume it should be around 100% for the Memory Network since it is 100% for the whole task there.
This paper presents a new, public dataset and tasks for goal-oriented dialogue applications. The dataset and tasks are constructed artificially using rule-based programs, in such a way that different aspects of dialogue system performance can be evaluated ranging from issuing API calls to displaying options, as well as full-fledged dialogue.
This is a welcome contribution to the dialogue literature, which will help facilitate future research into developing and understanding dialogue systems.
Still, there are pitfalls in taking this approach.
First, it is not clear how suitable Deep Learning models are for these tasks compared to traditional methods (rule-based systems or shallow models), since Deep Learning models are known to require many training examples and therefore performance difference between different neural networks may simply boil down to regularization techniques. The tasks 1-5 are also completely deterministic, which means evaluating performance on these tasks won't measure the ability of the models to handle noisy and ambiguous interactions (e.g. inferring a distribution over user goals, or executing dialogue repair strategies), which is a very important aspect in dialogue applications.
Overall, I still believe this is an interesting direction to explore.
As discussed in the comments below, the paper does not have any baseline model with word order information. I think this is a strong weakness of the paper, because it makes the neural networks appear unreasonably strong, yet simpler baselines could very likely be be competitive (or better) than the proposed neural networks. To maintain a fair evaluation and correctly assess the power of representation learning for this task, I think it's important that the authors experiment with one additional non-neural network benchmark model which takes into account word order information. This would more convincly demonstrate the utility of Deep Learning models for this task.
For example, the one could experiment with a logistic regression model which takes as input 1) word embeddings (similar to the Supervised Embeddings model), 2) bi-gram features, and 3) match-type features. If such a baseline is included, I will increase my rating to 8.
Final minor comment: in the conclusion, the paper states "the existing work has no well defined measures of performances". This is not really true. End-to-end trainable models for task-oriented dialogue have well-defined performance measures. See, for example "A Network-based End-to-End Trainable Task-oriented Dialogue System" by Wen et al.
On the other hand, non-goal-oriented dialogue are generally harder to evaluate, but given human subjects these can also be evaluated.
In fact, this is what Liu et al (2016) do for Twitter. See also, "Strategy and Policy Learning for Non-Task-Oriented Conversational Systems" by Yu et al.
I've updated my score following the new results added in the paper.
>>> “*As discussed in the comments below, the paper does not have any baseline model with word order information. I think this is a strong weakness of the paper, because it makes the neural networks appear unreasonably strong, yet simpler baselines could very likely be be competitive (or better) than the proposed neural networks. To maintain a fair evaluation and correctly assess the power of representation learning for this task, I think it's important that the authors experiment with one additional non-neural network benchmark model which takes into account word order information. This would more convincly demonstrate the utility of Deep Learning models for this task.
For example, the one could experiment with a logistic regression model which takes as input 1) word embeddings (similar to the Supervised Embeddings model), 2) bi-gram features, and 3) match-type features.”*
Thanks for these suggestions. We have added extra experiments augmenting supervised embeddings and TF-IDF methods with match-type features, and a vocabulary including all bigrams to leverage order information. The bigrams hardly make any difference in practice. Match-type features do help, however the memory networks still retain their clear edge compared to memory-less supervised embeddings and TF-IDF. The results of the new experiments have been added to the result table (for TF-IDF) and a new table in appendix D (for supervised embeddings).
>>>* Final minor comment: in the conclusion, the paper states "the existing work has no well defined measures of performances". This is not really true. End-to-end trainable models for task-oriented dialogue have well-defined performance measures. See, for example "A Network-based End-to-End Trainable Task-oriented Dialogue System" by Wen et al.
On the other hand, non-goal-oriented dialogue are generally harder to evaluate, but given human subjects these can also be evaluated.
In fact, this is what Liu et al (2016) do for Twitter. See also, "Strategy and Policy Learning for Non-Task-Oriented Conversational Systems" by Yu et al.*
(Wen et al., 2016) use  a collection of measures including BLEU scores, entity matching rate and objective task success rate (computed by auxiliary networks) as well as human judgements collected through Mechanical Turk.
It is true that all these measures are well defined but with this paper, we argue that they might not be enough. For clarity, we rephased this part of the conclusion as “(i) existing measures of performance either prevent reproducibility (different Mechanical Turk jobs) or do not correlate well with human judgements \citep{liu2016not};”.
My apologies for being last minute. I nearly lost track of this review, partly due to the massive amount of emails I’ve been receiving from ICLR.
Thank you very much for experimenting with two additional baselines taking into account word order information.
I’ve gone over the baseline results and they seem reasonable. I suppose the reason bigram features don’t increase performance is because of data sparsity; most bigrams occur very rarely in this dataset. This makes it very difficult to learn separate embeddings for bigrams. It is also useful to see that supervised embeddings + match type features are still inferior to memory networks.
Overall, this confirms the conclusion that memory networks are more suitable for this task than other (more shallow, memoryless) models. I’ve updated my score accordingly.
SYNOPSIS:
This paper introduces a new dataset for evaluating end-to-end goal-oriented dialog systems.  All data is generated in the restaurant setting, where the goal is to find availability and eventually book a table based on parameters provided by the user to the bot as part of a dialog.  Data is generated by running a simulation using an underlying knowledge base to generate samples for the different parameters (cuisine, price range, etc), and then applying rule-based transformations to render natural language descriptions. The objective is to rank a set of candidate responses for each next turn of the dialog, and evaluation is reported in terms of per-response accuracy and per-dialog accuracy. The authors show that Memory Networks are able to improve over basic bag-of-words baselines.
THOUGHTS:
I want to thank the authors for an interesting contribution.
Having said that, I am skeptical about the utility of end-to-end trained systems in the narrow-domain setting. In the open-domain setting, there is a strong argument to be made that hand-coding all states and responses would not scale, and hence end-to-end trained methods make a lot of sense.
However, in the narrow-domain setting, we usually know and understand the domain quite well, and the goal is to obtain high user satisfaction. Doesn't it then make sense in these cases to use the domain knowledge to engineer the best system possible?
Given that the domain is already restricted, I'm also a bit disappointed that the goal is to RANK instead of GENERATE responses, although I understand that this makes evaluation much easier. I'm also unsure how these candidate responses would actually be obtained in practice? It seems that the models rank the set of all responses in train/val/test (last sentence before Sec 3.2). Since a key argument for the end-to-end training approach is ease of scaling to new domains without having to manually re-engineer the system, where is this information obtained for a new domain in practice?  Generating responses would allow much better generalization to new domains, as opposed to simply ranking some list of hand-collected generic responses, and in my mind this is the weakest part of this work.
Finally, as data is generated using a simulation by expanding (cuisine, price, ...) tuples using NL-generation rules, it necessarily constrains the variability in the training responses.
Of course, this is traded off with the ability to generate unlimited data using the simulator.
But I was unable to see the list of rules that was used. It would be good to publish this as well.
Overall, despite my skepticism, I think it is an interesting contribution worthy of publication at the conference.
I've updated my score following the clarifications and new results.
>>> “*Doesn't it then make sense in these cases to use the domain knowledge to engineer the best system possible?”*
We totally agree with this statement. Our paper should not be interpreted as a manual on how to build a restaurant booking bot in practice, but mostly as a way to address generalizability across all domains. We use the restaurant booking domain as a pretext for outlining key capabilities that a dialog system should have (querying KBs, sorting options, etc.) and we use this to study end-to-end models, a subset of all possible dialog systems. The choice of restaurant booking is meant to make comparisons to existing methods easier, but the value of domain-knowledge-free methods is that they generalize to all domains. We believe that this paper and his accompanying dataset can make valuable material for research in dialogue.
>>> “*Given that the domain is already restricted, I'm also a bit disappointed that the goal is to RANK instead of GENERATE responses, although I understand that this makes evaluation much easier.”*
We indeed made the choice of ranking instead of generating in order to have the clearest evaluation possible. This evaluation is already non-trivial for end-to-end models as confirmed by the recent paper “On the Evaluation of Dialogue Systems with Next Utterance Classification” by  Lowe et al., so we think that performing well in this setting is a meaningful step before generating answers. The dataset could also be used in a generation setting by switching the metric to BLEU or something similar.
>>> “*I'm also unsure how these candidate responses would actually be obtained in practice?”*
Our goal in this paper is to provide an analysis research tool so this work can not be applied for building a bot in practice directly.
However, many strategies could be used to create candidates such as harvesting large corpora of conversation logs or running simulations.
Attempts to use chatbots for every form of human-computer interaction has been a major trend in 2016, with claims that they could solve many forms of dialogs beyond simple chit-chat. This paper represents a serious reality check.
While it is mostly relevant for Dialog/Natural Language venues (to educate software engineer about the limitations of current chatbots), it can also be published at Machine Learning venues (to educate researchers about the need for more realistic validation of ML applied to dialogs), so I would consider this work of  high significance.
Two important conjectures are underlying this paper and likely to open to more research.
While they are not in writing, Antoine Bordes clearly stated them during a NIPS workshop presentation that covered this work. Considering the metrics chosen in this paper:
1)	The performance of end2end ML approaches is still insufficient for goal oriented dialogs.
2)	When comparing algorithms, relative performance on synthetic data is a good predictor of performance on natural data. This would be quite a departure from previous observations, but the authors made a strong effort to match the synthetic and natural conditions.
While its original algorithmic contribution consists in one rather simple addition to memory networks (match type), it is the first time these are deployed and tested on a goal-oriented dialog, and the experimental protocol is excellent. The overall paper clarity is excellent and accessible to a readership beyond ML and dialog researchers. I was in particular impressed by how the short appendix on memory networks summarized them so well, followed by the tables that explained the influence of the number of hops.
While this paper represents the state-of-the-art in the exploration of more rigorous metrics for dialog modeling, it also reminds us how brittle and somewhat arbitrary these remain. Note this is more a recommendation for future research than  for revision.
First they use the per-response accuracy (basically the next utterance classification among a fixed list of responses). Looking at table 3 clearly shows how absurd this can be in practice: all that matters is a correct API call and a reasonably short dialog, though this would only give us a 1/7 accuracy, as the 6 bot responses needed to reach the API call also have to be exact.
Would the per-dialog accuracy, where all responses must be correct, be better? Table 2 shows how sensitive it is to the experimental protocol. I was initially puzzled that the accuracy for subtask T3 (0.0) was much lower that the accuracy for the full dialog T5 (19.7), until the authors pointed me to the tasks definitions (3.1.1) where T3 requires displaying 3 options while T5 only requires displaying one.
For the concierge data, what would happen if ‘correct’ meant being the best, not among the 5-best?
While I cannot fault the authors for using standard dialog metrics, and coming up with new ones that are actually too pessimistic, I can think of one way to represent dialogs that could result in more meaningful metrics in goal oriented dialogs. Suppose I sell Virtual Assistants as a service, being paid upon successful completion of a dialog. What is the metric that would maximize my revenue? In this restaurant problem, the loss would probably be some weighted sum of the number of errors in the API call, the number of turns to reach that API call and the number of rejected options by the user.
However, such as loss cannot be measured on canned dialogs and would either require a real human user or an realistic simulator
Another issue closely related to representation learning that this paper fails to address or explain properly is what happens if the vocabulary used by the user does not match exactly the vocabulary in the knowledge base.
In particular, for the match type algorithm to code ‘Indian’ as ‘type of cuisine’, this word would have to occur exactly in the KB. I can imagine situations where the KB uses some obfuscated terminology, and we would like ML to learn the associations rather than humans to hand-describe them.
>>> This review contains many very interesting points about dialogue learning and evaluation that go beyond the scope of this paper. If we were to sell Virtual Assistants as a service, we agree that we would not necessarily use per dialog accuracy as a metric. We might actually not use a single metric and but a combination assessing task completion and user satisfaction for instance.
>>> *“what happens if the vocabulary used by the user does not match exactly the vocabulary in the knowledge base”*
The fact that the vocabulary matches perfectly between dialogues and the KB is clearly a simplification of our setting. In a real world scenario, one would have to run entity and coreference resolution algorithms which might cause additional errors. The easier setting of this paper grants a clearer interpretation of errors and success.
We thank the reviewers for their thorough and insightful comments on the paper. We address them in individual rebuttals. We also updated the paper to add more experimental results and clarifications as requested.
This paper was submitted to arXiv last week:
https://arxiv.org/abs/1701.04024
On the same task, it demonstrates that standard Seq2Seq models achieve better performance (both per response and per dialogue accuracies) compared to Memory Networks. See Table 2 (p. 4). Do the authors have any comments on this?
This paper is meant to introduce a new dataset and task for task-oriented dialogue. Since Seq2Seq models are both simpler and more established compared to Memory Networks, they should really be included as standard baselines. Would the authors be willing to train up a standard Seq2Seq model and Seq2Seq+Attention model and add these results into the paper?
We thank you for drawing our attention to this paper from last week. A few points:
1) We are indeed very pleased to see that researchers are trying new architectures on these datasets. This was our intention and we are glad that researchers find the data useful as a benchmark.
2) As for vanilla seq2seq being simpler than memory networks, we would argue that this is not really the case when looking at actual architectures (seq2seq being recurrent), but that they may appear so because of familiarity.
There are many other methods that could be viewed as reasonable baselines, and indeed it is our hope that more researchers will try whatever their favorite method is.
But we argue future papers by other researchers are a more appropriate home for these results. We think publicizing now with the set of baselines that we have already included so as to stimulate other people's research and let them run the additional baselines that make sense to them is more effective than us delaying publication and presentation of this work.

Meta-review:
role: PC
data: comment: Most dialog systems are based on chit-chat models. This paper explores goal-directed conversations, such as those that arise in booking a restaurant. While the methodology is rather thin, this is not the main focus of the paper. The authors provide creative evaluation protocols, and datasets. The reviewers liked the paper, and so does the AC. The paper will make a nice oral, on a topic that is largely explored but opening a direction that is quite novel.

Decision: Accept (Oral)