Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to use an actor-critic RL technique to train sequence to sequence tasks in natural language processing. 
In particular, experiments are shown in a synthetic denoising task as well as in machine translation. 

I like the idea of the paper, however, the experimental evaluation is not convincing. Why is the LL numbers in Ranzato et al. 2015 and your paper so different? Is the metric different? is it the scheduler? are the parameters different?
If one extrapolates the numbers, it seems that MIXER will be much better than the proposed approach. I'd like to see a head-to-head comparison, either by reproducing the same setting or by running the mixer baseline.

The authors should also compare their results to the state-of-the-art. How good is their machine translation system? Only comparing to a single baseline and without reproducing the numbers is not sufficient. 

While the idea makes sense, the authors needed to use many heuristics to make the model to work, e.g., using a delayed actor, update \phi' with interpolation, penalize the variance, reducing the value of rare actions, etc. 
Furthermore, there is no in depth analysis of how much performance each of these heuristics brings. 
It seems that the authors need more work to make the model work without so many heuristics.

The authors also mentioned several optimization difficulties (some of which are non-intuitive), 
1) why does the critic assign very high value to actions with very low probability according to the actor?
2) why is a lower square error on Q resulting in much worst performance?

The paper will benefit from a serious re-write. The technical part is not clearly written. The manuscript also assumes that the reader knows algorithms such as REINFORCE. I strongly suggest to include a brief description in the text. This will help the reader understand how to use the critic within this framework.
Also the experimental section will benefit from dividing it by experiment. Right now is cumbersome to look at the details of each experiment as things are mixed up in the text. 

The paper criticizes the REINFORCE algorithm a lot, particularly for its high variance, however the best results in the real setting are achieved with this algorithm (+ the critic). How do you explain this?

The text is also not consistent with what the results show. The discussion claims that using the critic on REINFORCE reduces the gap with the actor critic. However, it is better than the proposed approach. 

I'll revise my score if the authors address my questions.

In summary, an interesting idea, however many heuristics are used and the experimental evaluation is not sufficient.
review_title: In summary, an interesting idea, however many heuristics are used and the experimental evaluation is not sufficient.

final_score: rating: null
review_content: N/A
review_title: N/A

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces an actor-critic approach for sequence prediction, and shows experiments on spelling correction and machine translation.  While previous works e.g. Ranzato et al. 2015 have used an RL-based approach such as REINFORCE for sequence prediction, the main contribution of this work is the use of actor-critic as a novel approach for how to determine the target of network predictions, given the setting that the network should be trained to generate correctly given outputs already produced by the model and not ground-truth reference outputs.  Specifically, the actor is the main prediction network and the critic is trained to output the value of specific tokens.

The motivations for the approach are well-presented, and while a somewhat natural extension, it is still novel and justified. There are a number of details that are necessary for successful training, that are discussed well.  While the full Actor-Critic model does not show strong improvements over REINFORCE with critic, the critic-based models still outperform other baselines.  It would be nice to include more discussion of the bias-variance tradeoff and future advantages of Actor-Critic (from the pre-review question response) in the paper.  The paper is solid and deserves acceptance
review_title: novel approach, well motivated, results sufficient

final_score: rating: null
review_content: In the experiments, REINFORCE with critic performs in many cases comparable to or better than Actor-Critic. Can you provide more insight into why this may be the case, and where Actor-Critic has advantages?
review_title: actor-critic vs. REINFORCE with critic

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper presents a nice application of actor-critic method for conditional sequence prediction. The critic is trained conditional to target sequence output, while the actor is conditional on input sequence. The paper presents a number of interesting design decisions in order to tackle non-standard RL problem with actor-critic (conditional sequence generation with sequence-level reward function, large action space, reward at final step) and shows encouraging results for applying RL in sequence prediction. 

The interaction of actor and critic is an interesting aspect of this paper. Each has different pieces of information (input sequence, target output sequence), and effectively the actor gets target label information only through greedy optimization of the critic. Letting the critic having access to information only available at train time is interesting and may be applicable to other applications that tie RL with supervised learning. Pre-review discussion on Q-learning vs actor-critic has been good, and indeed I agree that making the critic having access to structured output label may be quite useful. 

The pros include reasonable improvement over prior attempts at using RL to fine-tune sequence models. One possible con is that the actor-critic is likely more unstable than simpler prior methods, thus requiring a number of tricks to alleviate, and it would be nice to see discussion on stability and hyper-parameter sensitivity. Another possible con is that this is an application paper, but it explores a non-traditional approach in a widely applicable field. 
review_title: Review

final_score: rating: null
review_content: It's a nice paper that bridges actor-critic methods and sequence prediction tasks. The results are encouraging. Have you tried pre-training the critic as you do, and then doing Q-learning instead of actor-critic to train the critic only? The critic directly provides policy, either argmax or softmax if entropy-regularized. 
review_title: actor-critic vs Q-learning

Meta-review: Originality, Significance:
 
 The paper proposes to use an actor-critic RL methods to train sequence to sequence tasks, as applied to NLP tasks. 
 A key aspect is that the critic network can be conditioned on the ground-truth output of the actor network. The idea is quite novel.
 
 Quality, Clarity:
 
 The major concern is with respect to the evaluation, specifically with respect to baseline results for other state-of-the-art methods for BLEU-scored translation tasks. The final rebuttal tackles many of these issues, although the reviewers have not commented on the rebuttal. 
 
 I believe that the method demonstrates significant promise, given the results that can be achieved with quite a different approach from previous work.
Decision: Accept (Poster)