Title: Dialogue Learning With Human-in-the-Loop
Authors: Jiwei Li
Alexander H. Miller
Sumit Chopra
Marc'Aurelio Ranzato
Jason Weston
Venue: ICLR 2017
Keywords: Natural language processing

Abstract: An important aspect of developing conversational agents is to give a bot the ability to improve through communicating with humans and to learn from the mistakes that it makes.  Most research has focused on learning from fixed training sets of labeled data rather than interacting with a dialogue partner in an online fashion. In this paper we explore this direction in a reinforcement learning setting where the bot improves its question-answering ability from feedback a teacher gives following its generated responses. We build a simulator that tests various aspects of such learning in a synthetic environment, and introduce models that work in this regime.  Finally, real experiments with Mechanical Turk validate the approach.


TL;DR: we explore a reinforcement learning setting for dialogue where the bot improves its abilities using reward-based or textual feedback

Review discussion:
At the end of section 4.1, it is stated:
"The standard way MemN2N is trained is via a cross entropy criterion on known input-output pairs,
which we refer to as supervised or imitation learning. As our work is in a reinforcement learning
setup where our model must make predictions to learn, this procedure will not work, so we instead
consider reinforcement learning algorithms which we describe next."
This claim is not quite clear to me. What I interpret this to mean is that, since you are interested in the online case where your policy is used in data collection (as opposed to Weston (2016)), then RL must be used.
While I agree that RL is natural in this setting, it is not clear why 'the [imitation learning] procedure will not work'. It seems to me that imitation learning as described in the paper could easily be applied to the online setting, e.g. by updating a single step in the direction of log-likelihood ascent for each new positive example (and ignoring negative examples). Is there something I'm missing? (If the claim is that the approach won't work _well_, then this should be more clearly explained)
(As said by Jason E Weston, the original question was meant for https://openreview.net/forum?id=rkE8pVcle)
Yes, we are interested in the case where the policy has to be trained from scratch and is hence used to “collect data”. The procedure you describe is not supervised learning/ imitation learning, because in that setting one is given the true labels for every example. If you use the rewards over your **predictions** of the labels as you describe this is a different kind of algorithm — in fact we call this algorithm “reward-based imitation” or RBI for short (Sec 4.2.1). We included results for RBI (and hence, exactly the algorithm you described) in the paper. This approach does not actually work well in the RL case unless you also add exploration into the policy as well (via the parameter epsilon), see Fig 3 (top left) and Fig 4 (top left).
- formalisation of the task (learning dialogue) is not precise.
when can we declare success?
- (along the line of the previous question:) dialogue can be seen as a form of noisy supervision. can you please report the classic supervision baselines for the particular model used? this would give a sense what fraction of the best case performance is achieved via dialogue learning.
- is there an understanding of how much more difficult the MT setting is? feedback could be hand labeled as positive or negative for an analysis (?). or a handcrafted baseline could be tested, that either extracts the reward via template matching … or maybe even uses the length of the feedback as a proxy/baseline. (it looks to me that short feedback is highly correlated with high reward / correct answer (?))
- relation to prior work Weston’16 is not fully clear. I understand that this submission should be understood as an independent submission of the prior work Weston’16 - and not replacing it. In this case Weston’16 makes this submission appear more incremental. my understanding is that the punch line of this submission is the online part that leads in turn to more exploration. Is there any analysis on how much this aspect matters? I couldn’t find this in the experiments.
1) We focused on question answering within dialogue precisely because this task is easily measurable. Evaluation of dialogue is an active research area, see e.g. "Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses" of Lowe et al, and beyond the scope of this work. One approach we think is worthwhile is for the community to collect a set of tasks, as has been done for games (Atari games, OpenAI universe, etc.) but for dialogue. The paper ''Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems'' of Dodge et al., takes a step towards this.
2) For the MTurk experiments we already did this, see Table 5 in the appendix. For the simulation, we did not report that but
we use the same datasets as Weston'16 which reported supervised baselines, so we can report them here too. We have updated the paper with that.
3) In the simulated data both positive and negative template responses are between length 1 and 4, so there is no length information and the methods work. For MTurk, the idea of hand labeling text rewards as positive or negative (with 100% accuracy) is exactly like using RBI or the numerical rewards, which is what we compared to. The results show that RBI+FP is better than that—  that is because there can be much more rich information in the text than just positive/negative reward. This is shown in Table 4 in the appendix which compares performance of different amounts of information in synthetic responses, where richer responses give better results.
4) Please also see the answer to the 2nd question from AnonReviewer5. We consider this a followup work that fixes two big missing things: Weston'16 (i) didn't do a reinforcement learning setting, so it wasn't clear the methods even worked in the realistic online case, and (ii) they only tried on simulated data, so it wasn't clear it would work on real language. And yes, it turns out exploration for RBI and FP and balancing for FP are very important,  see Fig 3 (top left) and Fig 4 (top left) and Table 1.
For example, eps=0 is no exploration, the algorithm used in Weston'16, and it completely fails.
Finally, we also showed that online batch learning can work well, comparably to fully online learning, which is important for real experiments with MTurk, as it greatly simplifies data collection.
Apologies for the tardy reply. Questions:
1. How much discourse context is actually necessary here? The examples in Figure
1 and Figure 2 suggest that almost all of the questions are one-off, and don't
require the user to refer to earlier portions of the dialogue. (An example of an
exchange _requiring_ dialogue context might be "Q1: When was Forrest Gump
released? Q2: Who is the lead actor?".) If there are questions requiring
discourse context, it would be great to see examples of them! If there are not,
then the task in this work is basically the same as the (large) literature on
question answering in NLP. Comparable approaches should be cited at the very
least, and compared to where relevant. I know the BABI questions involve reasoning
over multiple history steps, but many of the standard NLP question answering datasets
also involve this kind of extended reasoning and are not called "dialogue", which involves
a separate set of phenomena.
2. I'm a little confused about positioning---can you clarify the intended contribution of
this paper? It seems like this work takes a well-studied model (MemN2N) combines it
with well-studied training procedures (REINFORCE and supervised training) and applies
it to slightly new training data. Is the intended contribution of this paper an empirical
evaluation of the ability of existing dialogue models to fine-tune on new domains? Or is there
a technical contribution as well?
1) The discourse includes the statements made in the past, the question and answer, and crucially the response from the teacher. The latter is what makes the setting different from the standard question answering setting, i.e. the forward prediction method tries to also predict the response of the teacher, not just the answer questions.
However, clarifying this further in the text and including more references to the QA literature is a good idea. We have updated the paper.
2)  In terms of models, MemN2N+using numerical rewards is not well studied for text, although it is studied for other tasks. Much less studied is MemN2N with Forward Prediction (FP method) which we think is the more interesting case as rewards are not always available, but textual feedback is.
However, numerical rewards and textual feedback should be compared in the same setting.
Only one paper has studied MemN2N+FP so far (“Dialog-based Language Learning”, J. Weston, NIPS '16). We consider this paper a followup to that paper that addresses two large missing pieces in that work: (i) That earlier work did not use the natural reinforcement learning/online setting, but “cheated” with a fixed policy given in advance. It is important to address the realistic online setting and assess whether the methods, particularly FP,  still work, or else what changes (e.g. exploration, balancing, see Fig 4 and Table 1) are needed. (ii) That earlier work had only simulated data, and no real-language data, so was only toy. This work uses Mechanical Turk to do real experiments, which again is important to assess if these methods, particularly FP, work on real language.
We have released the data, code, and simulator described in the paper at https://github.com/facebook/MemNN/tree/master/HITL (and linked to that in the paper).
We have also updated the paper with the following changes:
More experiments:
- supervised baselines comparison in the appendix (also added in the main text for two experiments)
- comparison of real with synthetic responses of different types (sec C.1)
- second iteration of feedback (sec C.2)
More description and differences to existing work (Sec 2).
Please let us know if you have any other questions or comments.
Thanks!
SUMMARY: This paper describes a set of experiments evaluating techniques for
training a dialogue agent via reinforcement learning. A
standard memory network architecture is trained on both bAbI and a version of
the WikiMovies dataset (as in Weston 2016, which this work extends). Numerous
experiments are performed comparing the behavior of different training
algorithms under various experimental conditions.
STRENGTHS: The experimentation is comprehensive. I agree with the authors that
these results provide additional useful insight into the performance of the
model in the 2016 paper (henceforth W16).
WEAKNESSES: This is essentially an appendix to the earlier paper. There is no
new machine learning content. Secondarily, the paper seems to confuse the
distinction between "training with an adaptive sampling procedure" and "training
in interactive environments" more generally.
In particular, no comparisons are
presented to the to the experiments with a static exploration policy presented
in W16, when the two training can & should be evaluated side-by-side.
The only meaningful changes between this work and W16 involve simple
(and already well-studied) changes to the form of this exploration policy.
My primary concern remains about novelty: the extra data introduced here is
welcome enough, but probably belongs in a *ACL short paper or a technical
report. This work does not stand on its own, and an ICLR submission is not an
appropriate vehicle for presenting it.
"REINFORCEMENT LEARNING"
[Update: concerns in this section have been addressed by the authors.]
This paper attempts to make a hard distinction between the reinforcement
learning condition considered here and the ("non-RL") condition considered in
W16. I don't think this distinction is nearly as sharp as it's
made out to be.
As already noted in Weston 2016, the RBI objective is a special case of vanilla
policy gradient with a zero baseline and off-policy samples. In this sense the
version of RBI considered in this paper is the same as in W16, but with a
different exploration policy; REINFORCE is the same objective with a nontrivial
baseline.
Similarly, the change in FP is only a change to the sampling policy.
The fixed dataset / online learning distinction is not especially meaningful
when the fixed dataset consists of endless synthetic data.
It should be noted that some variants of the exploration policy in W16 provide a
stronger training signal than is available in the RL "from scratch" setting
here: in particular, when $\pi_acc = 0.5$ the training samples will feature much
denser reward.
However, if I correctly understand Figures 3 and 4 in this paper,
the completely random initial policy achieves an average reward of ~0.3 on bAbI
and ~0.1 on movies---as good or better than the other exploration policies in
W16!
I think this paper would be a lot clearer if the delta from W16 were expressed
directly in terms of their different exploration policies, rather than trying to
cast all of the previous work as "not RL" when it can be straightforwardly
accommodated in the RL framework.
I was quite confused by the fact that no direct comparisons are made to the
training conditions in the earlier work. I think this is a symptom of the
problem discussed above: once this paper adopts the position that this work is
about RL and the previous work is not, it becomes possible to declare that the
two training scenarios are incomparable. I really think this is a mistake---to
the extent that the off-policy sample generators used in the previous paper are
worse than chance, it is always possible to compare to them fairly here.
Evaluating everything in the "online" setting and presenting side-by-side
experiments would provide a much more informative picture of the comparative
behavior of the various training objectives.
ON-POLICY VS OFF-POLICY
Vanilla policy gradient methods like the ones here typically can't use
off-policy samples without a little extra hand-holding (importance sampling,
trust region methods, etc.). They seem to work out of the box for a few of the
experiments in this paper, which is an interesting result on its own. It would
be nice to have some discussion of why that might be the case.
OTHER NOTES
- The claim that "batch size is related to off-policy learning" is a little
odd. There are lots of on-policy algorithms that require the agent to collect a
large batch of transitions from the current policy before performing an
(on-policy) update.
- I think the experiments on fine-tuning to human workers are the most exciting
part of this work, and I would have preferred to see these discussed (and
explored with) in much more detail rather than being relegated to the
penultimate paragraphs.
AnonReviewer5 writes:
> “This paper attempts to make a hard distinction between the "reinforcement learning" objectives considered here and the (non-RL) objectives considered in the previous work. I don't think this distinction is nearly as sharp as it's made out to be.”
We think it is sharp, but perhaps we didn’t explain it clearly enough.
In the previous paper, the policy was chosen using an omniscient labeler, such that acc% of the answers are _always_ correct. This was not learned, and was fixed to generate the datasets.  This is a fake setting and in a real setting, you simply cannot do that (you don't have access to an omniscient labeler). In a real setting you have to learn a policy completely from scratch, online, starting with a random policy. This is what we refer to as the online / RL setting, and the previous paper didn’t do it, it instead built a fixed dataset using the omniscient labeler.
So, before this paper it just wasn’t clear if any of the introduced methods actually worked: (1) when you need to learn the policy (which you always do); and (2) using real language (which crucially wasn't tried before, only simulated text was used, here we used MTurk to fix that).
These two points are crucial to the success of these methods in a realistic reinforcement learning setting.
We thus believe this is an important contribution.
> This is a fake setting and in a real setting, you simply cannot do that (you don't have access to an omniscient labeler).
Right, but only if the "fake" labeler is more powerful than any fixed policy we can construct without access to the true reward function. If we construct a policy that guesses actions randomly, it is always as good as an omniscient labeler for some value of acc%. My concern here is the following: experiments in the paper suggest that "real" random guessing is actually _stronger_ than 2 of the 3 "fake" teaching policies considered in the earlier paper. (This isn't stated explicitly anywhere, and I'm pulling it off of the y axes of the top-right charts in Figs 3 & 4---let me know if I'm misreading.) In this sense it seems a little unfair to suggest that you can't compare to the earlier work because the fake supervision is _too strong_.
I want to emphasize that my overriding concern with this paper is about substance rather than this particular presentation issue.
But in whatever format it does eventually get published, I think its contributions will be much clearer with a more detailed discussion of the differences between training conditions and performance in the previous paper and this one. I hope this is easy to do! Probably as simple as:
1. state explicitly that previously-studied RL training procedures are used in both this paper & previous, with the salient difference being whether samples are generated from "expert", fixed random guesser, or current policy
2. say more about the differences (in performance, answer distribution, etc.) between the experts used previously and randomly-initialized policies in this paper
3. add some lines to some graphs
Hope this helps clarify.
> Right, but only if the "fake" labeler is more powerful than any fixed policy we can construct without access to the true reward function. If we construct a policy that guesses actions randomly, it is always as good as an omniscient labeler for some value of acc%
That is correct.
However, in the statements below that we think there are some misunderstandings.
Let's consider Table 1, which reports test accuracy for the dataset batch size case over several iterations (1 to 6). On each iteration the policy that generated the predictions is fixed, but is updated on the next iteration after learning.
On the first iteration you have to start with some kind of policy so we start with a random one. There exists a  \pi_acc% policy from Weston'16 that would obtain the same error rate as that chosen random policy on iteration 1. Values of acc% higher would be getting better accuracy and lower acc% lower accuracy.
However, in Weston'16 the policy is never updated while training, this is like stopping after the first iteration (column 1, Table 1) and that is the final error rate you get (which is why in Weston'16 on page 5 it is stated explicitly “Note that because the policies are fixed the experiments in this paper are not in a reinforcement learning setting.”).
However, in the setting in *this * paper we do update the policy and you get iterations 2, 3 and so on. What we want to show is that the accuracy gets *better* on subsequent iterations. And that is indeed the case, see Table 1, 2nd column (RBI), the accuracy goes from 0.74 to 0.87 to 0.90 to 0.96 and so on.
Hence, our approaches are doing better than the original policy they started with.
So if you started with one of the fake labelers from Weston'16, regardless of the value of the initial \pi_acc, you would improve over them as well.
>My concern here is the following: experiments in the paper suggest that "real" random guessing is actually _stronger_ than 2 of the 3 "fake" teaching policies considered in the earlier paper.
So we hope the above has explained that your concern/issue is actually exactly the thing that we are trying to show.
The point is that real random guessing isn't stronger, not initially, but after training and updating/learning the policy one would hope for it to be stronger. Our experiments showed this was the case, which is a positive result. This is a key contribution of the work.
>(This isn't stated explicitly anywhere, and I'm pulling it off of the y axes of the top-right charts in Figs 3 & 4---let me know if I'm misreading.)
Even though we just explained this for Table 1 because we think it is clearer, the same argument also follows for Figs 3&4 and all the other experiments as well. (The Figs 3&4 experiments can be understood similarly to Table 1, except they use small online batches, see Sec 4.2).
So, we hope we have convinced you why it does not make sense to compare to the \pi_acc% policies from Weston'16.
> I want to emphasize that my overriding concern with this paper is about substance rather than this particular presentation issue.
We believe because you didn't understand this point, you are missing some of the substance of the paper. Hopefully now you have a clearer picture of the contribution, that shows that a reinforcement learning actually works with these new approaches (especially forward prediction, which is a new algorithm never tried before in a reinforcement learning setting for text).
We will also update the related work section to make this contribution more clear compared to previous work.
Just a note to say we've now updated the paper to include both in the related work section and in the experiment section details and comparison to Weston '16 relating to our points above. We hope this issue and the contribution in general is now much more clear.
Thanks, this is much clearer. I still have concerns about the novelty of the technical content, but I have increased my score from a 4 to a 5.
This paper builds on the work of Weston (2016), using End-to-end memory network models for a limited form of dialogue with teacher feedback. As the authors state in the comments, it is closely related to the question answering problem with the exception that a teacher provides a response after the model’s answer, which does not always come with a positive reward.
Thus, the model must learn to use the teacher’s feedback to significantly improve performance.
Overall, the paper is written clearly, and several interesting models are tested. It is certainly only a limited form of dialogue that is considered (closer to question answering, since the questions do not require the agent to look further back into the context), but investigating in this direction could prove fruitful once the tasks are scaled up to be more difficult.
My main concern is with the paper`s novelty. In the words of the authors, this paper has two primary differences with the work of Weston:
“(i) That earlier work did not use the natural reinforcement learning/online setting, but “cheated” with a fixed policy given in advance. It is important to address the realistic online setting and assess whether the methods, particularly FP, still work, or else what changes (e.g. exploration, balancing, see Fig 4 and Table 1) are needed. (ii) That earlier work had only simulated data, and no real-language data, so was only toy. This work uses Mechanical Turk to do real experiments, which again is important to assess if these methods, particularly FP, work on real language.”
Point (ii) is very much appreciated, but adding additional human testing data is not sufficient for a conference paper.
Thus, the main point of the paper is that “the model also works if we collect the data online (i.e. the agent’s policy is used to collect data rather than a fixed policy beforehand)”.
While this is a step in the right direction, I’m not sure if it’s significant enough for an ICLR paper. Little model novelty is required to solve this additional requirement on these tasks beyond using epsilon greedy exploration.
Thus, the paper is borderline accept/reject.
EDIT: I have updated my score slightly in light of the author's response, where they make a good point that real-world implementation should be more strongly considered as part of the contribution.
Good papers aren't always about the novelty of the algorithm. There is also the learning task, the investigation and the results.
For example, we wouldn't have hundreds of papers about convolutional neural nets and recurrent neural nets if people didn't think there were things worth exploring beyond the original papers. This is only the first paper after Weston'16. This paper tries to investigate whether you can learn in an online fashion with humans in the loop using their responses, not just rewards, which has never been done before.
In retrospect, several findings may seem evident;
however, when we started this investigation we did not know the answer to the following questions: can one learn from textual feedback alone when training from scratch (without an oracle policy)? What does it take to make methods working in the off-line setting perform well in the online one? Does it matter to make training more off-policy (increase the amount of data collection using the current policy)? Are these algorithm stable in practice? Can our training algorithms adapt to the richness of natural language spoken by humans? What's a setting that can work with humans in the loop? Our work provides a variety of simple and practical extensions for making the algorithms proposed by Weston NIPSP'16 work in a setting that matters for real applications.
Of course, we hope that somebody will improve upon these results with entirely new algorithms in the future, but the analysis of the strong baseline methods we analyzed here is a required stepping stone.
However, the results show this is a valuable and practical direction.
As discussed, the there are multiple concurrent contributions in different packages/submission by the authors that are in parts difficult to disentangle.
Despite this fact, it is impressive to see a system learning from natural feedback in an online fashion. To the best of my knowledge, this is a new quality of result that was achieved - in particular as close to full supervision results are reached in some cases in this less constraint setting.
several points were raised that were in turn addressed by the authors:
1. formalisation of the task (learning dialogue) is not precise.
when can we declare success?
The answer of the authors is partially satisfying. For this particular work, it might make sense to more precisely set goals e.g. to be as good as full supervision.
2. (along the line of the previous question:) dialogue can be seen as a form of noisy supervision. can you please report the classic supervision baselines for the particular model used? this would give a sense what fraction of the best case performance is achieved via dialogue learning.
The authors provided additional information along those lines - and I think this helps to understand how much of the overall goal was achieved and open challenges.
3. is there an understanding of how much more difficult the MT setting is? feedback could be hand labeled as positive or negative for an analysis (?). or a handcrafted baseline could be tested, that either extracts the reward via template matching … or maybe even uses the length of the feedback as a proxy/baseline. (it looks to me that short feedback is highly correlated with high reward / correct answer (?))
The authors replied - but it would have been clearer if they could have quantified such suggested baseline, in order to confirm that there is no simple handcrafted baseline that would do well on the data - but these concerns are marginal.
4. relation to prior work Weston’16 is not fully clear. I understand that this submission should be understood as an independent submission of the prior work Weston’16 - and not replacing it. In this case Weston’16 makes this submission appear more incremental. my understanding is that the punch line of this submission is the online part that leads in turn to more exploration. Is there any analysis on how much this aspect matters? I couldn’t find this in the experiments.
The authors clarified the raised issues. The application of reinforcement learning and in particular FP is convincing.
There is a incremental nature to the paper - and the impression is emphasised by multiple concurrent contributions of the authors on this research thread. Comparison to prior work (in particular Weston'16), should be made more explicit. Not only in text but also in the experiments - as the authors partially do in their reply to the reviewers question.
Nevertheless, this particular contribution is assessed as significant and worth sharing and seems likely to have impact on how we can learn in these less constraint setting.
In regards to point 4 and your final point "Comparison to prior work (in particular Weston'16), should be made more explicit" (and to the discussion with AnonReviewer5, see that for details) we have now updated the paper to include both detailed discussion in the related work section and comparison in the experiments section with regards to Weston '16. We hope this issue and the contribution in general is now much more clear.

Meta-review:
role: PC
data: comment: pros:
 - demonstration that using teacher's feedback to improve performance in a dialogue system can be made to work 
  in a real-world setting
 - comprehensive experiments
 
 cons:
 - lack of technical novelty due to prior work
 - not all agree with the RL vs not-RL (pre-built datasets) distinction suggested in the paper with respect to the previous work
 
 Overall, the paper makes a number of practical contributions and evaluation, rather than theoretical novelty.

Decision: Accept (Poster)