Venue: ICLR 2017 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
SUMMARY: This paper describes a set of experiments evaluating techniques for
training a dialogue agent via reinforcement learning. A
standard memory network architecture is trained on both bAbI and a version of
the WikiMovies dataset (as in Weston 2016, which this work extends). Numerous
experiments are performed comparing the behavior of different training
algorithms under various experimental conditions.

STRENGTHS: The experimentation is comprehensive. I agree with the authors that
these results provide additional useful insight into the performance of the
model in the 2016 paper (henceforth W16).

WEAKNESSES: This is essentially an appendix to the earlier paper. There is no
new machine learning content. Secondarily, the paper seems to confuse the
distinction between "training with an adaptive sampling procedure" and "training
in interactive environments" more generally. In particular, no comparisons are
presented to the to the experiments with a static exploration policy presented
in W16, when the two training can & should be evaluated side-by-side.
The only meaningful changes between this work and W16 involve simple
(and already well-studied) changes to the form of this exploration policy.

My primary concern remains about novelty: the extra data introduced here is
welcome enough, but probably belongs in a *ACL short paper or a technical
report. This work does not stand on its own, and an ICLR submission is not an
appropriate vehicle for presenting it.

"REINFORCEMENT LEARNING"

[Update: concerns in this section have been addressed by the authors.]

This paper attempts to make a hard distinction between the reinforcement
learning condition considered here and the ("non-RL") condition considered in
W16. I don't think this distinction is nearly as sharp as it's
made out to be. 

As already noted in Weston 2016, the RBI objective is a special case of vanilla
policy gradient with a zero baseline and off-policy samples. In this sense the
version of RBI considered in this paper is the same as in W16, but with a
different exploration policy; REINFORCE is the same objective with a nontrivial
baseline. Similarly, the change in FP is only a change to the sampling policy.
The fixed dataset / online learning distinction is not especially meaningful
when the fixed dataset consists of endless synthetic data.

It should be noted that some variants of the exploration policy in W16 provide a
stronger training signal than is available in the RL "from scratch" setting
here: in particular, when $\pi_acc = 0.5$ the training samples will feature much
denser reward. However, if I correctly understand Figures 3 and 4 in this paper,
the completely random initial policy achieves an average reward of ~0.3 on bAbI
and ~0.1 on movies---as good or better than the other exploration policies in
W16!

I think this paper would be a lot clearer if the delta from W16 were expressed
directly in terms of their different exploration policies, rather than trying to
cast all of the previous work as "not RL" when it can be straightforwardly
accommodated in the RL framework.

I was quite confused by the fact that no direct comparisons are made to the
training conditions in the earlier work. I think this is a symptom of the
problem discussed above: once this paper adopts the position that this work is
about RL and the previous work is not, it becomes possible to declare that the
two training scenarios are incomparable. I really think this is a mistake---to
the extent that the off-policy sample generators used in the previous paper are
worse than chance, it is always possible to compare to them fairly here.
Evaluating everything in the "online" setting and presenting side-by-side
experiments would provide a much more informative picture of the comparative
behavior of the various training objectives.

ON-POLICY VS OFF-POLICY

Vanilla policy gradient methods like the ones here typically can't use
off-policy samples without a little extra hand-holding (importance sampling,
trust region methods, etc.). They seem to work out of the box for a few of the
experiments in this paper, which is an interesting result on its own. It would
be nice to have some discussion of why that might be the case.

OTHER NOTES

- The claim that "batch size is related to off-policy learning" is a little
odd. There are lots of on-policy algorithms that require the agent to collect a
large batch of transitions from the current policy before performing an 
(on-policy) update.

- I think the experiments on fine-tuning to human workers are the most exciting
part of this work, and I would have preferred to see these discussed (and
explored with) in much more detail rather than being relegated to the
penultimate paragraphs.
review_title: Review

final_score: rating: null
review_content: Apologies for the tardy reply. Questions:

1. How much discourse context is actually necessary here? The examples in Figure
1 and Figure 2 suggest that almost all of the questions are one-off, and don't
require the user to refer to earlier portions of the dialogue. (An example of an
exchange _requiring_ dialogue context might be "Q1: When was Forrest Gump
released? Q2: Who is the lead actor?".) If there are questions requiring
discourse context, it would be great to see examples of them! If there are not,
then the task in this work is basically the same as the (large) literature on
question answering in NLP. Comparable approaches should be cited at the very
least, and compared to where relevant. I know the BABI questions involve reasoning
over multiple history steps, but many of the standard NLP question answering datasets
also involve this kind of extended reasoning and are not called "dialogue", which involves
a separate set of phenomena.

2. I'm a little confused about positioning---can you clarify the intended contribution of
this paper? It seems like this work takes a well-studied model (MemN2N) combines it 
with well-studied training procedures (REINFORCE and supervised training) and applies 
it to slightly new training data. Is the intended contribution of this paper an empirical 
evaluation of the ability of existing dialogue models to fine-tune on new domains? Or is there 
a technical contribution as well?
review_title: clarifications w/r/t "dialogue" and technical contribution

final_score: rating: null
review_content: At the end of section 4.1, it is stated:
"The standard way MemN2N is trained is via a cross entropy criterion on known input-output pairs,
which we refer to as supervised or imitation learning. As our work is in a reinforcement learning
setup where our model must make predictions to learn, this procedure will not work, so we instead
consider reinforcement learning algorithms which we describe next."

This claim is not quite clear to me. What I interpret this to mean is that, since you are interested in the online case where your policy is used in data collection (as opposed to Weston (2016)), then RL must be used. While I agree that RL is natural in this setting, it is not clear why 'the [imitation learning] procedure will not work'. It seems to me that imitation learning as described in the paper could easily be applied to the online setting, e.g. by updating a single step in the direction of log-likelihood ascent for each new positive example (and ignoring negative examples). Is there something I'm missing? (If the claim is that the approach won't work _well_, then this should be more clearly explained)

(As said by Jason E Weston, the original question was meant for https://openreview.net/forum?id=rkE8pVcle)
review_title: why not use supervised learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper builds on the work of Weston (2016), using End-to-end memory network models for a limited form of dialogue with teacher feedback. As the authors state in the comments, it is closely related to the question answering problem with the exception that a teacher provides a response after the model’s answer, which does not always come with a positive reward. Thus, the model must learn to use the teacher’s feedback to significantly improve performance.

Overall, the paper is written clearly, and several interesting models are tested. It is certainly only a limited form of dialogue that is considered (closer to question answering, since the questions do not require the agent to look further back into the context), but investigating in this direction could prove fruitful once the tasks are scaled up to be more difficult.

My main concern is with the paper`s novelty. In the words of the authors, this paper has two primary differences with the work of Weston:

“(i) That earlier work did not use the natural reinforcement learning/online setting, but “cheated” with a fixed policy given in advance. It is important to address the realistic online setting and assess whether the methods, particularly FP, still work, or else what changes (e.g. exploration, balancing, see Fig 4 and Table 1) are needed. (ii) That earlier work had only simulated data, and no real-language data, so was only toy. This work uses Mechanical Turk to do real experiments, which again is important to assess if these methods, particularly FP, work on real language.”

Point (ii) is very much appreciated, but adding additional human testing data is not sufficient for a conference paper. Thus, the main point of the paper is that “the model also works if we collect the data online (i.e. the agent’s policy is used to collect data rather than a fixed policy beforehand)”. While this is a step in the right direction, I’m not sure if it’s significant enough for an ICLR paper. Little model novelty is required to solve this additional requirement on these tasks beyond using epsilon greedy exploration. Thus, the paper is borderline accept/reject. 

EDIT: I have updated my score slightly in light of the author's response, where they make a good point that real-world implementation should be more strongly considered as part of the contribution.

review_title: Review

final_score: rating: null
review_content: - formalisation of the task (learning dialogue) is not precise. when can we declare success? 
- (along the line of the previous question:) dialogue can be seen as a form of noisy supervision. can you please report the classic supervision baselines for the particular model used? this would give a sense what fraction of the best case performance is achieved via dialogue learning.
- is there an understanding of how much more difficult the MT setting is? feedback could be hand labeled as positive or negative for an analysis (?). or a handcrafted baseline could be tested, that either extracts the reward via template matching … or maybe even uses the length of the feedback as a proxy/baseline. (it looks to me that short feedback is highly correlated with high reward / correct answer (?))
- relation to prior work Weston’16 is not fully clear. I understand that this submission should be understood as an independent submission of the prior work Weston’16 - and not replacing it. In this case Weston’16 makes this submission appear more incremental. my understanding is that the punch line of this submission is the online part that leads in turn to more exploration. Is there any analysis on how much this aspect matters? I couldn’t find this in the experiments.
review_title: pre-review questions

final_score: rating: 7: Good paper, accept
review_content: As discussed, the there are multiple concurrent contributions in different packages/submission by the authors that are in parts difficult to disentangle.
Despite this fact, it is impressive to see a system learning from natural feedback in an online fashion. To the best of my knowledge, this is a new quality of result that was achieved - in particular as close to full supervision results are reached in some cases in this less constraint setting.

several points were raised that were in turn addressed by the authors:
1. formalisation of the task (learning dialogue) is not precise. when can we declare success? 
The answer of the authors is partially satisfying. For this particular work, it might make sense to more precisely set goals e.g. to be as good as full supervision.

2. (along the line of the previous question:) dialogue can be seen as a form of noisy supervision. can you please report the classic supervision baselines for the particular model used? this would give a sense what fraction of the best case performance is achieved via dialogue learning.
The authors provided additional information along those lines - and I think this helps to understand how much of the overall goal was achieved and open challenges.

3. is there an understanding of how much more difficult the MT setting is? feedback could be hand labeled as positive or negative for an analysis (?). or a handcrafted baseline could be tested, that either extracts the reward via template matching … or maybe even uses the length of the feedback as a proxy/baseline. (it looks to me that short feedback is highly correlated with high reward / correct answer (?))
The authors replied - but it would have been clearer if they could have quantified such suggested baseline, in order to confirm that there is no simple handcrafted baseline that would do well on the data - but these concerns are marginal.

4. relation to prior work Weston’16 is not fully clear. I understand that this submission should be understood as an independent submission of the prior work Weston’16 - and not replacing it. In this case Weston’16 makes this submission appear more incremental. my understanding is that the punch line of this submission is the online part that leads in turn to more exploration. Is there any analysis on how much this aspect matters? I couldn’t find this in the experiments.
The authors clarified the raised issues. The application of reinforcement learning and in particular FP is convincing.

There is a incremental nature to the paper - and the impression is emphasised by multiple concurrent contributions of the authors on this research thread. Comparison to prior work (in particular Weston'16), should be made more explicit. Not only in text but also in the experiments - as the authors partially do in their reply to the reviewers question. Nevertheless, this particular contribution is assessed as significant and worth sharing and seems likely to have impact on how we can learn in these less constraint setting.
review_title: Dialogue Learning With Human-in-the-Loop

Meta-review: pros:
 - demonstration that using teacher's feedback to improve performance in a dialogue system can be made to work 
  in a real-world setting
 - comprehensive experiments
 
 cons:
 - lack of technical novelty due to prior work
 - not all agree with the RL vs not-RL (pre-built datasets) distinction suggested in the paper with respect to the previous work
 
 Overall, the paper makes a number of practical contributions and evaluation, rather than theoretical novelty.
Decision: Accept (Poster)