Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Maybe I missed it in the paper, but it would be nice if you could provide some more intuition as to why you chose the precise setup for your synthetic experiment.

Also, just to make sure -- in this case the target sentence is the exact same as the input sentence, but without the forbidden words? From how you define the reward in the appendix, it seems the reward is independent of decoded word order -- is that true?
review_title: synthetic experiment

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper discuss a "batch" method for RL setup to improve chat-bots.
The authors provide nice overview of the RL setup they are using and present an algorithm which is similar to previously published on line setup for the same problem. They make a comparison to the online version and explore several modeling choices. 

I find the writing clear, and the algorithm a natural extension of the online version.

Below are some constructive remarks:
- Comparison of the constant vs. per-state value function: In the artificial experiment there was no difference between the two while on the real-life task there was. It will be good to understand why, and add this to the discussion. Here is one option:
- For the artificial task it seems like you are giving the constant value function an unfair advantage, as it can update all the weights of the model, and not just the top layer, like the per-state value function.
- section 2.2:
   sentence before last: s' is not defined. 
   last sentence: missing "... in the stochastic case." at the end.
- Section 4.1 last paragraph: "While Bot-1 is not significant ..." => "While Bot-1 is not significantly different from ML ..."




review_title: clearly written, natural extension of previous work

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper extends neural conversational models into the batch reinforcement learning setting. The idea is that you can collect human scoring data for some responses from a dialogue model, however such scores are expensive. Thus, it is natural to use off-policy learning – training a base policy on unsupervised data, deploying that policy to collect human scores, and then learning off-line from those scores.

While the overall contribution is modest (extending off-policy actor-critic to the application of dialogue generation), the approach is well-motivated, and the paper is written clearly and is easy to understand. 

My main concern is that the primary dataset used (restaurant recommendations) is very small (6000 conversations). In fact, it is several orders of magnitude smaller than other datasets used in the literature (e.g. Twitter, the Ubuntu Dialogue Corpus) for dialogue generation. It is a bit surprising to me that RNN chatbots (with no additional structure) are able to generate reasonable utterances on such a small dataset. Wen et al. (2016) are able to do this on a similarly small restaurant dataset, but this is mostly because they map directly from dialogue states to surface form, rather than some embedding representation of the context. Thus, it remains to be seen if the approaches in this paper also result in improvements when much more unsupervised data is available.

References:

Wen, Tsung-Hsien, Milica Gasic, Nikola Mrksic, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. "A Network-based End-to-End Trainable Task-oriented Dialogue System." arXiv preprint arXiv:1604.04562 (2016).

review_title: Review

final_score: rating: null
review_content: Could the authors please comment on / clarify the following points regarding the comparison of their proposed batch algorithm to the online version of [Degris et al 2012]:
1) On p6 the authors claim, that the main benefit of their algorithm compared to the online version comes from the fact that in the batch setting all (terminal) rewards are known compared to the online setting. However, Degris et al show that using eligibility traces, the lambda-return can exactly be computed in a backward view. Hence, it the updated to the parameters were computed over the entire dataset using the algorithm of Degris et al (without doing any actual parameter updated meanwhile), the result would be the same as of the proposed algorithm. Hence, as I understand it, the only difference between the two algorithms is when the parameters are updated. Is this correct?
2) The algorithm outlined on p6 suggests that for each gradient step on the parameters, the entire dataset is touched. This seems quite inefficient. An mini-batched version of this algorithm on the other hand would resemble the version of Degris quite closely. 
3) In fig 2: Does the x-axis show number of passes through the entire dataset for all algorithms?
4) The errata of Degris et al states that the guarantees for their off-policy algorithm only holds for the tabular setting. This would exclude the RNN-based state representation chosen by the authors here; is this correct?
5) middle of p6: “In order to tackle this, previous research either [...], or make additional assumptions about the distribution of the states (Degris et al., 2012...” Which assumptions in Degris et al do the authors refer to?
review_title: Clarification regarding batch vs. online setting

final_score: rating: 7: Good paper, accept
review_content: The author propose to use a off-policy actor-critic algorithm in a batch-setting to improve chat-bots.
The approach is well motivated and the paper is well written, except for some intuitions for why the batch version outperforms the on-line version (see comments on "clarification regarding batch vs. online setting").
The artificial experiments are instructive, and the real-world experiments were performed very thoroughly although the results show only modest improvement. 
review_title: null

Meta-review: This is an interesting and timely paper combining off-policy learning with seq2seq models to train a chatbot on a restaurant reservation task, using labels collected through Amazon Mechanical Turk while using the bot with a baseline maximum likelihood policy. 
 The paper is clear, well-written and well-executed. Although the improvements are modest and the actual novelty of the paper is limited (combining known pieces in a rather straightforward way), this is still an interesting and informative read, and will probably be of interest to many people at ICLR.
Decision: Accept (Poster)