Venue: ICLR 2020 Conference

Reviews:

final_score: rating: null
review_content: Seems there is only one language pair and dataset used in experiments. 
Table 6 of [1] shows that RL helps on some language pair and may hurt on other language pair. Given only one language pair in this paper, it is not clear whether or not the observations and conclusions are reliable and generalized to other languages and datasets.

[1] Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
review_title: Only one language pair?

final_score: rating: 8: Accept
review_content: This paper first theoretically demonstrates that a commonly used reinforcement learning method for neural sequence-to-sequence models (e.g. in NMT), contrastive minimum risk training (CMRT), is not guaranteed to converge to local (let alone global) optima of the reward function. The paper then empirically demonstrates that the REINFORCE algorithm, while not subject to the same theoretical flaws as CMRT, in practice fails to improve NMT models unless the baseline model is already "nearly correct" (i.e. the correct tokens were already within the few most probable tokens before the fine-tuning steps with REINFORCE). In fact, some of the performance gains of using REINFORCE/CMRT can be attributed to making the model's output probability distribution more peaked, and not necessarily from making the target tokens more probable as commonly assumed.

Overall, this is an excellent paper that offers significant contributions for the field. I have summarised the key strengths of the paper below, along with several suggestions and questions that I hope will be addressed by the authors. Based on my assessment, I am recommending a rating of "Accept" for this paper.

Strengths:
1. The paper is very well-written and well-structured. It starts off by pointing out the theoretical limitations of CMRT (and concisely recaps the key differences between CMRT and REINFORCE), and continues with an extensive set of experiments that clearly illustrates the limitations of REINFORCE in practice. 

2. I also like the use of both controlled simulations (including one where the reward is constant) and NMT experiments with real data. The controlled simulations are useful to abstract away from the full complexity of the model and investigate what happens under various control scenarios, while the NMT experiments demonstrate that the findings still hold under the realistic setup.

3. The findings are really interesting and clearly illustrate the limitations of existing REINFORCE/CMRT methods for neural sequence-to-sequence models. It is very interesting to see that REINFORCE fails to make the target token most probable when the initial model ranks the target token as the third or more probable tokens under the model (Figure 2), even under the simple controlled simulations, which highlights the prohibitively high sample complexity of the model. 

4. The peakiness effect hypothesis (i.e. attributing the gains of REINFORCE to making the output distribution more peaked, and not necessarily by making the target tokens more probable) is well-supported by the paper's empirical evidence. It is really illuminating that using a constant reward of 1 leads to the same BLEU score as actually optimising for BLEU in NMT (Section 5.2).

Suggestions and questions:
1. Section 4.2 (NMT Experiments) indicates that REINFORCE fine-tuning is done for 10 epochs, with 5,000 sentences per epoch, and k=1. Considering the enormous discrete sample space, one could expect that using multi-sample REINFORCE (i.e. k > 1) and training the model for many more epochs might mitigate the identified problems to some extent, and thus change the findings. Training for 5,000 sentences * 10 epochs may just not be enough for the RL fine-tuning to make a big difference.

2. In Figure 1, the x-axis is the "Update Size" with a scale between -1.0 and 1.0. This "Update Size" variable is not really explained in the paper, and why the scale is between -1.0 and 1.0.

3. In my understanding, the controlled simulations (Section 4.1) is done at the word-level (including word-level rewards, as opposed to the NMT experiments which are done at the sequence-level with sentence-level rewards). If this is the case, this should be made clearer.

4. To make Figure 3 easier to understand, the caption should indicate that a lower cumulative percentage means a more peaked output distribution.

5. Rather than breaking down the analysis by where the target token is ranked by the initial, pre-RL model (e.g. the target token is ranked second/third best in Figures 2 and 5), perhaps what really matters is the probability assigned to the target token. For instance, even if the target token is ranked third best by the initial model, there will be a big difference whether it is assigned a probability of 0.1 or 0.01 (i.e. the latter case is much less likely to be sampled, which would exacerbate the problem). Including this analysis might help strengthen the paper further.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In the context of neural machine translation, limitations of some reinforcement learning methods, in particular REINFORCE and contrastive minimum risk training (MRT), are analyzed. The authors argue that MRT doesn't optimize the expected reward. Moreover, they show that using REINFORCE, with either realistic or dummy constant rewards, may lead to a peakier distribution. Similar BLEU scores are obtained with either type of rewards, which is an interesting and perplexing result (in my opinion). For both REINFORCE and MRT, the paper shows that unless the gold token was already amongst the most probable after pre-training, it takes many samples for it to become the most likely output, which limits the usefulness of on-policy RL approaches.

I lean slightly towards rejection because the scope of the paper is somewhat too limited. The experimental section mostly covers REINFORCE without a baseline (except for validation in section 4.2, but no results are shown varying the baseline), as well as MRT in a restricted scenario. However, the analysis may still be beneficial to the community.

Establishing that minimum risk training doesn't optimize the expected reward is a valuable observation. Can the optimized and expected rewards be arbitrarily far?

The experiments demonstrating the sample inefficiency of REINFORCE (without baseline) and MRT when the best tokens have low initial probability (relative to other tokens) raise important questions about the effectiveness of these approaches.

The analysis of REINFORCE assumes non-negative rewards, which likely contributes to the peakiness effect (PKE). I would assume the peakiness effect to be mostly neutral with normalized rewards, or diminish with negative rewards (on average). It is unclear how different baselines would affect the results.

For MRT, only synthetic experiments are run. Given that the peakiness effect is much weaker than for REINFORCE, it would be useful to know results (BLEU) on the same MT task.

Other questions and remarks:

In appendix A.1, the parameter value is bounded (to maintain a valid probability distribution), which is generally not the case within a neural network. Does this distinction matter?

Do you have the RL learning curves? What could we learn from them?

It is unclear how much the claims generalize to other methods such as actor-critic (Bahdanau et al. An Actor-Critic Algorithm for Sequence Prediction). 

In appendix A.1, some values are wrong. Do these affect the final result?
For example, P({a,c}) should be $2 \theta (1-\theta-2 \theta^2)$.
For S={a,c}, \nabla \tilde{R} should be $\frac{1+2 \theta^2}{2(1-2 \theta^2)^2}$.
For S={b,c} \nabla \tilde{R} should be $\frac{\theta (\theta-2)}{(1-\theta)^2}$.

-----
Update to Review 2

I appreciate that the authors corrected appendix A.1. As a nitpick, for the gradient of the reward of {a, c}, there is a x that should be replaced by \theta.

While the scope of the paper is somewhat limited, the theoretical contributions about MRT are valuable. The experiments would be more convincing with stronger baselines, but the current paper may already generate useful discussions within the community. As such, I updated my score to "Weak accept", although I won't fight for acceptance.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This work carefully studies RL for neural machine translation and draws several conclusions:
1. One of the most common RL methods for MT does not optimize the expected reward, as well as show that other methods take an infeasibly long time to converge. 
2. RL practices in MT are likely to improve performance only where the pre-trained parameters are already close to
yielding the correct translation. 
3. Observed gains may be due to effects unrelated to the training signal, concretely, changes in the shape of the distribution curve.

I have questions about several technical claims, which lead to my doubt about the technical correctness of the paper:

	1. [updated after checking authors' response] "it may well converge to values that are not local maxima of R, making it theoretically ill-founded." Previously I had concerns about the convergence of REINFORCE with deep NNs as its policy. After checking the references provided by the authors, the local convergence can indeed be guaranteed.  
 
	2. "reducing a constant baseline from r, so as to make the expected reward zero, disallows learning." This conflicts with my intuition. Where is the experiment supporting this claim?

	3. "MRT succeeds in pushing ybest to be the highest ranked token if it was initially second, but struggles where it was initially ranked third or below." Why ybest in third position cannot be boosted while second can be boosted? Figure 5 only shows two tokens, which cannot lead to any meaningful statistical conclusions. I'd like to see the statistic numbers: 
		a. how many ybest in second are boosted to first in training? How many are not?
		b. how many ybest in third are boosted to first or second in training? How many are not?
How about other positions?
review_title: Official Blind Review #3

Meta-review: In my opinion, the main strength of this work is the theoretical analysis and some observations that may be of great interest to the NLP community in terms of better analyzing the performance of RL (and "RL-like") methods as optimizers. The main weakness, as pointed out by R3, the limited empirical analysis.

I would urge the authors to take R3's advice and attempt insofar as possible to broaden the scope of the empirical analysis in the final. I believe that this is important for the paper to be able to make its case convincingly.

Nonetheless, I do think that the paper makes a significant contribution that will be of interest to the community, and should be presented at ICLR. Therefore, I would recommend for it to be accepted.
Decision: Accept (Poster)