Title: Learning through Dialogue Interactions by Asking Questions
Authors: Jiwei Li
Alexander H. Miller
Sumit Chopra
Marc'Aurelio Ranzato
Jason Weston
Venue: ICLR 2017
Keywords: Natural language processing

Abstract: A good dialogue agent should have the ability to interact with users by both responding to questions and by asking questions, and importantly to learn from both types of interactions. In this work, we explore this direction by designing a simulator and a set of synthetic tasks in the movie domain that allow such interactions between a learner and a teacher. We investigate how a learner can benefit from asking questions in both offline and online reinforcement learning settings, and demonstrate that the learner improves when asking questions. Our work represents a first step in developing such end-to-end learned interactive dialogue agents.


TL;DR: We investigate how a bot can benefit from interacting with users and asking questions.

Review discussion:
The approach you propose seems to require a knowledge base (KB).
In other words, the human designers of the dialogue system need to hand-craft what elements constitutes knowledge for the model. It could be problematic if certain types of elements are not represented in that knowledge base. Is there any way for the model to recover from missing elements in the knowledge base? And also, have you thought of ways to extend this approach to less structured domains (e.g. to assume that no knowledge base is available)?
- No, it does not require a KB. Memory Networks, which we use as our basic architecture, have been shown to work well from both raw text and KBs, see this paper for a study:  https://arxiv.org/abs/1606.03126, e.g. Memory Network variants obtain state-of-the-art results on WikiQA which uses raw text, not a KB.
- Yes, the model can potentially recover from missing KB entries because that is only used as context  — that is,  the final output can be any of the possible candidate responses (in this case, all possible entities) and is only conditioned on the dialog and context.
Of course it is harder though.
- Yes, it would be straight-forward to switch from a KB to a raw text setting in our paper, e.g. using Wikipedia as the knowledge source in our experiments, similar to the setup in the paper already mentioned ( https://arxiv.org/abs/1606.03126).
However, this distinction is not the focus of the paper.
- It is worth mentioning that the "Knowledge Acquisition" tasks deal specifically with the issue of missing entities in the retrieval step when there are unknown words. One can learn about these entities by asking questions to improve performance, i.e. this is a way for “ the model to recover from missing elements in the knowledge base”.
Thanks for clarifying this!
Indeed, I had misunderstood the interaction between the knowledge base and the agent.
Unless I'm mistaken, your models seem to use word embeddings (or the average of surrounding word embeddings). Since it's not at the character level, it's unclear how the model is able to do question verification (i.e. Figure 1, bottom right), other than completely guessing based on the entity mentioned. Is that correct?
On the other hand, the question paraphrase AQ setting seems like very easy behavior to learn (if you encounter an unknown word, ask the clarification question, then proceed normally).
Yes, to some degree it has to guess based on the entity mentioned plus any clues in the other words it can understand (e.g. a “who” vs. “what” question, the word “in” appearing, etc.).  Character level models would  only help further if the new word is similar to an existing one (in the example given this is the case, but it might not be).
Still they are definitely worth investigating in the future, too.
It's important to note that it's not just about how difficult the question is to ask, but whether the model can make use of that knowledge (the response from the teacher to the question asked) afterwards.   We find that asking questions performs better than not asking them (i.e. Train AQ+Test AQ  gives better results than  Train QA + Test QA or other combinations).
In the offline supervised setting, since TrainMix is a combination of TrainQA and TrainAQ, I would expect the performance for TrainMix to always be somewhere in between TrainQA and TrainAQ.
But, that doesn't seem to be the case in Tasks 4 and 8 (when testing on TestQA), and Task 7 (when tested on both TestQA and TestAQ) for vanilla-MemN2N, and in Task 9 for cont-MemN2N when tested on TestAQ. Why?
Also, what are the sizes of train and test sets being used? What is the accuracy metric for numbers reported in Tables 2, 3 and 4?
For Task 7 and 8, the results are off by one column. Sorry for the mistake. We just corrected them and uploaded the updated results.
For Tasks 4 and 9, the fact that the result of mix does not lie between AQ and QA is because of training variation and it disappears when we reran each experiment for 10 times and reported the best result. We have updated the result
We use the same  dataset as described  in https://arxiv.org/abs/1511.06931. The training/dev/test sets respectively contain 181638/9702/9698 examples. The  accuracy metric corresponds to the percentage of times the student gives correct answers to the teacher's questions.
Dear all,
We have released the data, code, and simulator described in the paper at https://github.com/facebook/MemNN/tree/master/AskingQuestions
We also updated the paper (https://openreview.net/pdf?id=rkE8pVcle) where we run each experiment in the supervised setting for 10 times to avoid training variations.
This paper introduces a simulator and a set of synthetic tasks for evaluating a dialogue agent's ability to learn from user feedback. For solving these tasks, the paper uses memory networks (Sukhbaatar et al., 2015) learned through previously proposed supervised learning and reinforcement learning methods. In this setup, it is demonstrated that the agent learning from feedback (e.g. through question asking or question clarification) performs better.
The motivation for the paper is excellent; dialogue agents which learn directly from unstructured human feedback (as opposed to reward signals alone) could be very useful in real-world applications.
However, the paper falls short on the execution. All the numerous experiments presented are based on the synthetic dialogue simulator, which is highly artificial and different from real-world dialogues. The simulator is based on a simple factoid question-answering framework, which normally is not considered dialogue and which appears to be solvable with a few hand-crafted rules. The framework also assumes that the user's feedback is always correct and is given in one of a handful of forms (e.g. paraphrase of original question without typos) and that the agent can learn from examples of another agent asking questions or making clarifications, which simplifies the task even further.
Because of the artificial setting and limited scope of the experiments, it seems difficult to draw conclusions about how to learn from unstructured user feedback. To test the hypothesis that it is possible to learn from such user feedback, I would strongly recommend the authors to continue working on this project by carrying out experiments with real human users (even in the factoid question answering domain, if necessary). This would provide much stronger evidence that a dialogue agent can learn from such feedback.
Other comments:
- The abstract uses the phrase "interactive dialogue agents". What is meant by "interactive" dialogue agents? All dialogue agents interact with the user, so isn't it redundant to call them interactive?
- A major limitation of the experiments is that the questions the agent can ask are specified a priori. If I understand correctly, in the supervised learning setting the agent is trained to imitate the questions of another rule-based agent.
While in the RL setting, the paper states "For each dialogue, the bot takes two sequential actions $(a_1 , a_2)$: to ask or not to ask a question (denoted as a_1 ); and guessing the final answer (denoted as a_2)". This means the agent learns *when* to ask questions but not *what* questions to ask.
- Related to the previous comment, in the sub-section "ONLINE REINFORCEMENT LEARNING (RL)" the paper states "We also explored scenarios where the student learns the ability to decide when to ask a question and what to ask.". Please clarify this by removing the part "what to ask".
- The paper presents an overwhelming amount of results. I understand the benefit of synthetic tasks is precisely the ability to measure many aspects of model performance, but in this case it confuses the reader to present so many results.
For example, what was the reason for including the "TrainAQ(+FP)" and "TrainMix" training settings? How do these results help validate the original hypothesis? If they don't, they should be taken out or moved to the appendix.
- Since the contribution of the paper lies in the tasks and evaluation, it might be better to move either the vanilla-MemN2N (Table 2) to the appendix or to move the Cont-MemN2N results (Table 3) to the appendix.
--- UPDATE ---
Following the discussion below and the additional experiments provided by the authors, I have increased my score to 8.
As far as we know, there is very little in the literature (see related work) on what the reviewer agrees is an important direction:  dialogue agents which learn directly from unstructured human feedback (as opposed to reward signals alone).
To start in a new direction like this you have to have to start with a first approach. One has to see in a relatively simple setup if it works at all, and understand where it does and does not work.
Now that we have showed some modest success on the tasks we have developed we hope: (i) both we and other researchers can develop better techniques and test them in this setup; and (ii) we can continue on to more difficult, real-world challenges.
So far, we are happy to show the positive result that a bot asking questions can significantly improve results in the setup we tested on.
Reply to other comments:
- “interactive dialogue agents” meaning bots that can both answer questions and ask questions,  where learning is exhibited from the latter. We have now reworded and clarify this.
- R1 states “A major limitation of the experiments is that the questions the agent can ask are specified a priori” — however, that is not correct.  In the TestModelAQ setting the model has to get the form of the question correct as well. E.g. in the  Question Verification and Knowledge Verification tasks there are many possible ways of forming the question and some of them are correct — the model has to choose the right question to ask. E.g. it should ask “Does it have something to do with the fact that Larry Crowne directed by Tom Hanks?” rather than “Does it have something to do with the fact that Forrest Gump directed by Robert Zemeckis?” when the latter is irrelevant (the candidate list of questions is generated from the known knowledge base entries with respect to that question). We have now clarified this further in the text.
-  However, in the RL experiment we did not use TestAQ so R1 is correct in that case. We have now removed the "what to ask" phrasing there.
- FP (forward prediction) is very important to measure because our goal is to find a model that can learn during dialogue. This means asking questions, and learning from the user's responses, which is what this method does (the reward-based method doesn't really use the user's responses). The FP method outperforms the reward-based method on some of the tasks which is a very encouraging sign.
> As far as we know, there is very little in the literature (see related work) on what the reviewer agrees is an important direction...
I completely agree that the idea presented in the paper is a novel and important direction of research.
However, even a novel idea on its own is not sufficient for publication at a competitive conference like ICLR. Empirical experiments are critical to demonstrate the merits of the idea.
Starting with a simple synthetic setup is very useful as preliminary test.
However, since the actual goal is to learn from humans, how about finding the simplest real-world setup (with humans involved) and experimenting on this?
For example, how about experimenting with a question-answering task in the movie domain?
With that said, I think the ideas in this paper would create many interesting discussions so I am happy to recommend it as a workshop paper at ICLR.
We think there are two ways to go given a reasonable paper length: depth or breadth. We preferred to have breath of exploration, focusing our investigation on a large variety of simulated tasks so we could understand better the problem. Human eval comes next in a subsequent paper as a study worth trying, which we did not know a priori.
Note that several papers have proposed synthetic learning tasks and yet they have been very useful and well regarded, at least looking at the number of citations they have so far received:
“An Empirical Evaluation of Deep Architectures on Problems with Many Factors of Variation”, Larochelle et al., ICML'07 (419 citations on Google Scholar)
“Toward AI-Complete Question Answering: A Set of Prerequisite Toy Tasks”, Weston et al., ICLR '16 (159 citations on Google scholar).
Value Iteration Networks, Tamar et al., NIPS '16 (best paper award).
(Indeed, much of the RL literature contains simulated problems.)
Generally researchers explore toy problems when the task is challenging and new, and when the understanding is currently lacking, so real data at that stage is less helpful.
Sorry, maybe I should have been more specific in my previous comments.
I am not generally against using synthetic tasks in machine learning. As you point out, this has in fact been the norm in some areas of machine learning, like in reinforcement learning.
However, the appropriateness of synthetic tasks depends on the problem (or hypothesis) being investigated.
For example, Larochelle et. al (2007) test out different models' ability to solve tasks with many factors of variations. Their hypothesis is that deeper models can model datasets with more factors of variations. This hypothesis is largely independent of the real world (e.g. human labels, or human interaction), which means it is sensible to test it on synthetic tasks (assuming those tasks are designed to measure appropriate varying factors, of course). A similar argument can be made for your own paper, Weston et al. (2016). The hypothesis being investigated here is how well different models can solve discrete logical problems. These logical problems (e.g. induction, deduction, co-reference resolution etc.) are somewhat independent of the real world (e.g. the same problems could be cast in logical forms, and in this form there exists a set of logical rules which solves each task perfectly) and, moreover, most of the tasks are relatively easy to synthesize.
The fundamental problem with this work is the discrepancy between the hypothesis being set forth and the evaluation of it. The hypothesis is that a dialogue agent can improve its performance by learning from interactions with a human user, specifically by asking questions and receiving feedback on those. This hypothesis cannot reasonably be separated from humans and evaluated on synthetic tasks, because human interaction (in particular, feedback on questions) is a core part of the problem.
Let me be more concrete. In the reinforcement learning setting, the paper reduces the problem from a two-player human-computer interactive game to a one-player game (an agent interacting with a fixed, simulated environment through a synthetic language). Learning agents which are able to play the one-player game does not give reasonable evidence that the same agents will be able to play the two-player game, which is the main hypothesis set out in the paper.
Although synthetic tasks are still useful as a preliminary validation of the models and learning framework, I think the paper still needs to include human experiments to convincingly demonstrate the hypothesis.
> Generally researchers explore toy problems when the task is challenging and new, and when the understanding is currently lacking, so real data at that stage is less helpful.
The more general task of training dialogue agents using human interactions is hardly new. Researchers have been investigating different learning approaches and models for years, for example, using online reinforcement learning. Some of the early work was done using user simulators, but now it is common to carry out experiments (or at least, to evaluate the system) using humans users. Since this a well-established practice in the field, I think it's reasonable to also expect this paper to include human evaluations.
> " I think it's reasonable to also expect this paper to include human evaluations."
> "For example, how about experimenting with a question-answering task in the movie domain?"
We have now updated the paper to include human experiments using Mechanical Turk!
We collected real human language data with 10,000 episodes of training, 1000 for validation,
and 2500 for testing for two of the tasks (4 and 8). We have compared the same learning algorithms on this new data and, despite accuracy results being lower  (which is expected, as we are now using fewer, but real examples), we arrive at the same main conclusion: the presented algorithms can learn to perform better by asking questions.
Please see the updated paper, in particular sections 4.2 (data description) and 6.2 (results), as well as further details and analysis in the appendix.
We hope that this addresses your main concern.
Excellent! Thank you for putting in the extra effort in the evaluation! I will increase my score.
It seems like the human experiments are carried out in the supervised learning framework? Is that correct? If yes, please clarify this and the reasons you do it.
Overall, I think it's reasonable as it would allow you to run other models against the same dataset in the future.
It's interesting that human feedback helps a lot on Task 8, but makes only a small difference on Task 4. Can you comment on that? Could one reason be that Task 4 involves binary feedback from the user (a yes/no response), while Task 8 involves interpreting factual information?
Minor notes:
- You mention the AMT dialogues have a lot of "noise". Can you clarify this? Are you referring to word misspellings, grammatical errors or misinformation (e.g. user is giving incorrect responsible)? Have you thought about ways of adding this noise into your simulator?
- "training set was times smaller due to data collection costs" -> "training set was smaller due to data collection costs"
- With the additional AMT results, I strongly encourage you to shorten the paper by moving some of your result tables (maybe Table 2) to the appendix. This would also give you space to discuss the results in depth.
- Finally, it's interesting to note the differences between training with simulated synthetic data and testing on human generated data versus training and testing on human generated data - in particular for Task 8.
In fact, this illustrates show how different the simulations actually are from real interactions with humans.
Thanks, we've made a few more updates to the paper in line with your comments above.
> It seems like the human experiments are carried out in the supervised learning framework? Is that correct? If yes, please
> clarify this and the reasons you do it.
Overall, I think it's reasonable as it would allow you to run other models against the
> same dataset in the future.
Yes, it is in the supervised framework. As you say, it allows simplicity of, and reproducibility of experiments. We have clarified this in the paper.
>It's interesting that human feedback helps a lot on Task 8, but makes only a small difference on Task 4.
> Can you comment on that? Could one reason be that Task 4 involves binary feedback from the user (a yes/no response),
>  while Task 8 involves interpreting factual information?
Yes, this is line with the original results from the simulator data. Binary feedback about which facts are relevant to answer a question is a lot weaker information than the direct label information given in Task 8.
>Minor notes:
> - You mention the AMT dialogues have a lot of "noise". Can you clarify this? Are you referring to word misspellings, >grammatical errors or misinformation (e.g. user is giving incorrect responsible)? Have you thought about ways of adding >this noise into your simulator?
All of the above (misspellings, grammar, mislabeled answers) although we were thinking more of the last one (we observed all of them in the data). We did not attempt to correct any of these. Yes, it would be possible to add some of these things to the simulator, but we are not sure it would help understanding...but maybe we could make the performance more aligned with real data results and/or use this to improve results on real test data using simulated data alone?
> - "training set was times smaller due to data collection costs" -> "training set was smaller due to data collection costs"
Corrected.
> - With the additional AMT results, I strongly encourage you to shorten the paper by moving some of your result tables (maybe Table 2) to the appendix. This would also give you space to discuss the results in depth.
Done.
> - Finally, it's interesting to note the differences between training with simulated synthetic data and testing on human
> generated data versus training and testing on human generated data - in particular for Task 8.
In fact, this illustrates show
> how different the simulations actually are from real interactions with humans.
Agreed.
The goal of this paper is to analyze the behaviour of dialogue agents when they must answer factoid questions, but must query an oracle for additional information. This can be interpreted as a form of interaction between the dialogue agent and a ‘teacher’.
The problem under investigation is indeed very important. The authors create a synthetic environment in which to test their agent. The main strength of the paper is that the paper tests many different combinations of environments, where either some knowledge is missing (and the agent has to query for it), or there is some misspelling in the teacher’s question, and different ways the agent can ask for extra information.
I am a bit concerned that many of the tasks are too easy (e.g. the AQ question paraphrase), and I am also concerned that the environment presented is very limited, and quite far (in terms of richness of linguistic structure) from how real humans would interact with chatbots. I think the paper would be better positioned as testing the basic reasoning capabilities of agents/ their ability to do question answering, rather than dialogue.
However, I think the ‘ground-up’ approach that starts with simple environments is indeed worthy of analysis, and this paper makes an interesting contribution in that direction.
Of course, the paper would be much more convincing with human experiments.
Additional notes:
I think the simulation, in the synthetic environment, for the first mistake a learner can make during dialogue: “the learner has problems understanding the surface form of the text of the dialogue partner, e.g., the phrasing of a question”, is particularly limited since only word misspellings are considered (and the models used don’t work at the character level), which is of course only a tiny fraction of ways an agent can misunderstand the context. I would be particularly interested to see some discussion of how the authors plan to scale this up to more realistic settings.
EDIT: I have updated my score to reflect the addition of the Mechanical Turk experiments
Yes, please also see our response to AnonReviewer1 concerning the nature of the dataset.
The advantage of simulated datasets is that one can break down the analysis, e.g. we have identified types of questions that can be asked, and now we can test which models are capable of benefitting from them. We can also test a reinforcement learning setting easily without incurring impractical data collection costs during model development. Now that we have some (limited) success future work can try other models (many of our results can still clearly be improved e.g. they are in the 10 - 20% error region when they could be <5% at least) and develop more challenging/real datasets.
For the first set of tasks (Question Clarification) we only reported experiments in the “typo” domain for simplicity. As R1 comments, we already have lot of experiments, so we didn't add more.
However, we actually did implement other 'misunderstanding the surface form' variants but left them out for simplicity and space reasons.
In particular, for each question type, we have a set number of ways of asking it (different phrase templates) and we tried leaving out 1 or 2 (can be a hyperparameter) such phrases that are only seen in test, and that the bot can ask a question about to receive a rephrasing that it does know about. This is a setting that also closely mimics a real world situation.
> "Of course, the paper would be much more convincing with human experiments. "
We have now updated the paper to include human experiments using Mechanical Turk!
We collected real human language data with 10,000 episodes of training, 1000 for validation,
and 2500 for testing for two of the tasks (4 and 8). We have compared the same learning algorithms on this new data and, despite accuracy results being lower  (which is expected, as we are now using fewer, but real examples), we arrive at the same main conclusion: the presented algorithms can learn to perform better by asking questions.
Please see the updated paper, in particular sections 4.2 (data description) and 6.2 (results), as well as further details and analysis in the appendix.
We hope that this addresses your main concern.
Thank you for these changes, I have updated my score accordingly.
One comment: in Section 4.2, there are a few typos, e.g.:
"dialoguess"
"setup similar to Section ??"
Thanks, fixed.
The paper introduces a simulator and a set of synthetic question answering tasks where interaction with the "teacher" via asking questions is desired. The motivation is that an intelligent agent can improve its performance by asking questions and getting corresponding feedback from users. The paper studies this problem in an offline supervised and an online reinforcement learning settings. The results show that the models improve by asking questions.
-- The idea is novel, and is relatively unexplored in the research community. The paper serves as a good first step in that direction.
-- The paper studies three different types of tasks where the agent can benefit from user feedback.
-- The paper is well written and provides a clear and detailed description of the tasks, models and experimental settings.
Other comments/questions:
-- What is the motivation behind using both vanilla-MemN2N AND Cont-MemN2N? Is using both resulting in any conclusions which are adding to the paper's contributions?
-- In the Question Clarification setting, what is the distribution of misspelled words over question entity, answer entity, relation entity or none of these? If most of the misspelled words come from relation entities, it might be a much easier problem than it seems.
-- The first point on Page 10 "The performance of TestModelAQ is worse than TestAQ but better than TestQA." is not true for Task 2 from the numbers in Tables 2 and 4.
-- What happens if the conversational history is smaller or none?
-- Figure 5, Task 6, why does the accuracy for good student drop when it stops asking questions? It already knows the relevant facts, so asking questions is not providing any additional information to the good student.
-- Figure 5, Task 2, the poor student is able to achieve almost 70% of the questions correct even without asking questions. I would expect this number to be quite low. Any explanation behind this?
-- Figure 1, Task 2 AQ, last sentence should have a negative response "(-)" instead of positive as currently shown.
Preliminary Evaluation:
A good first step in the research direction of learning dialogue agents from unstructured user interaction.
>-- What is the motivation behind using both vanilla-MemN2N AND Cont-MemN2N? Is using both resulting in any conclusions which are adding to the paper's contributions?
vanilla-MemN2N is a standard baseline so we should compare to it.
Cont-MemN2N is a simple improvement introduced in this paper which can make a big difference.
> -- In the Question Clarification setting, what is the distribution of misspelled words over question entity, answer entity, relation entity or none of these? If most of the misspelled words come from relation entities, it might be a much easier problem than it seems.
Missing words in this task are always in the non-entity words. Missing entities are Tasks 5-9.
Still the unknown misspelled words can make the question ambiguous (e.g. do you want to know the director or writer of a movie?).
> -- The first point on Page 10 "The performance of TestModelAQ is worse than TestAQ but better than TestQA." is not true for Task 2 from the numbers in Tables 2 and 4.
Has been fixed and re-uploaded. Thanks.
> -- What happens if the conversational history is smaller or none?
The results are likely somewhat better. The code is now released so many variations can be tried.
>-- Figure 5, Task 6, why does the accuracy for good student drop when it stops asking questions? It already knows the relevant facts, so asking questions is not providing any additional information to the good student.
While it's true the good student does have the necessary information already, when asking the question and getting the right answer the correct response is more immediately evident later, rather than trying to fish it out of the known facts. You can think of this as stronger features given to a classifier, if you like, which helps.
Hence, the good student still benefits from question asking in these tasks.
> -- Figure 5, Task 2, the poor student is able to achieve almost 70% of the questions correct even without asking questions. I would expect this number to be quite low. Any explanation behind this?
The reason why the poor student can still do fairly well is that we introduce typos to some of the non-entity words in the questions but not all of them. The student can still to some extent understand (or guess correctly) the meaning of some of the questions. This is in line with Table 3, we are using the Cont-MemN2N in the online experiments as well as it worked better in the offline experiments, and indeed works relatively well in this setting -- but still works better when asking questions.
> -- Figure 1, Task 2 AQ, last sentence should have a negative response "(-)" instead of positive as currently shown.
Has been fixed and re-uploaded. Thanks.
Note that we have now made a major update to the paper, adding human experiments using Mechanical Turk!
Please see comments to the other reviewers, and the updated paper, in particular sections 4.2 (data description) and 6.2 (results), as well as further details and analysis in the appendix.

Meta-review:
role: PC
data: comment:  This paper is a clear accept. Reviewers were both positive and confident about their assessments. Paper introduces a simulator and synthetic question answering task where interactions with the teacher are used for learning. Reviewers felt paper was well written with clear descriptions of tasks, models and experiments. Reviewer did comment on limitations due to the simple factoid QA framework explored for which hand crafted rules seems sufficient to solve the problem.

Decision: Accept (Poster)