On the Weaknesses of Reinforcement Learning for Neural Machine Translation
Authors: Leshem Choshen
Lior Fox
Zohar Aizenbud
Omri Abend
Venue:  (2020)
Keywords: Reinforcement learning
MRT
minimum risk training
reinforce
machine translation
peakkiness
generation

Abstract: Reinforcement learning (RL) is frequently used to increase performance in text generation tasks,
including machine translation (MT), 
notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). 
However, little is known about what and how these methods learn in the context of MT. 
We prove that one of the most common RL methods for MT does not optimize the 
expected reward, as well as show that other methods take an infeasibly long time to converge.
In fact, our results suggest that RL practices in MT are likely to improve performance
only where the pre-trained parameters are already close to yielding the correct translation.
Our findings further suggest that observed gains may be due to effects unrelated to the training signal, concretely, changes in the shape of the distribution curve.

Published as a conference paper at ICLR 2020
ON THE WEAKNESSES OF REINFORCEMENT LEARN- ING FOR NEURAL MACHINE TRANSLATION
Leshem Choshen', Lior Fox², Zohar Aizenbud', Omri Abend
School of Computer Science and Engineering, The Edmond and Lily Safra Center for Brain Sciences
3 Department of Cognitive Sciences The Hebrew University of Jerusalem first. 1 last@mailll au.i.ac. oabendecs. cs www.adesshhui...o huji..ac
ABSTRACT
Reinforcement learning (RL.) is frequently used to increase performance in text generation tasks, including machine translation (MT), notably through the use of Minimum Risk Training (MRT) and Generative Adversarial Networks (GAN). However, little known about what and how these methods learn in the context of MT. We prove that one of the most common RL methods for MT does not optimize the expected reward, as well as show that other methods take an infeasibly long time to converge. In fact, our results suggest that RL practices in MT are likely improve performance only where the pre-trained parameters are already close yielding the correct translation. Our findings further suggest that observed gains may be due to effects unrelated to the training signal, concretely, changes in the shape of the distribution curve.
INTRODUCTION
Reinforcement learning (RL) an appealing path for advancement in Machine Translation (MT), as it allows training systems to optimize non-differentiable score functions, common in MT evaluation, as well as tackling the "exposure bias" (Ranzato et al. (015)) in standard training, namely that the model is not exposed during training to incorrectly generated tokens, and is thus unlikely to recover from generating such tokens at test time. These motivations have led to much interest in RL for text generation in general and MT in particular (see $2). Various policy gradient methods have been used, notably REINFORCE (Williams) 1992) and variants thereof (e.g., Ranzato et al. 2015 Edunov et al.) 2018) and Minimum Risk Training (MRT; e.g., Och 2003 Shen et 2016). Another popular of RL is for training GANs (Yang et al. 2018 Tevet et al. 2018). Nevertheless, despite increasing interest and strong results, little is known about what accounts for these performance gains, and the training dynamics involved.
We present the following contributions. First, our theoretical analysis shows that commonly used approximation methods are theoretically ill-founded, and may converge to parameter values that do minimize the risk, nor are local minima thereof (2.2).
Second, using both naturalistic experiments and carefully constructed simulations, we show that performance gains observed in the literature likely stem not from making target tokens the most probable, but from unrelated effects, such as increasing the peakiness of the output distribution (i.e., the probability mass of the most probable tokens). We do so by comparing a setting where the reward is informative, vs. one where it is constant. In $ we discuss this peakiness effect (PKE).
Third, we show that promoting the target token be the mode likely to take a prohibitively long time. The only the first 2-3 most probable tokens according to the pretrained model. These findings suggest that REINFORCE ($5) and CMRT ($6) likely to improve over the pre-trained model only under the best possible conditions, i.e., where the pre-trained model "nearly" correct.
we find, where improvements likely, is where the target token is among
We conclude by discussing other RL practices in MT which should be avoided for practical and theoretical reasons, and briefly discuss alternative RL approaches that will allow RL to tackle a larger class of errors in pre-trained models ($7)

---

Published as a conference paper at ICLR 2020
2 RL IN MACHINE TRANSLATION
An MT system generates tokens y (3/,,..,, Vn)) from vocabulary V one token time. The probability of generating given preceding tokens yei is given by Po(-|z, Mee), where z is the source sentence and 9 are the model parameters. For each generated token Yi, we denote with r M<ci, 2, yf)ef) the score, or reward, for generating yi given V<i. 4, and the reference sentence g(ref) For brevity, we omit parameters where they are fixed within context. For simplicity, we assume does not depend on following tokens Y>i.
We also assume there is exactly one valid target token, de facto, training is done against single reference (Schulz et al. 20188) In practice, either a token-level reward is approximated using Monte- Carlo methods (e.g.. Yang et al. 2018), or a sentence-level (sparse) reward is given at the end of the episode (sentence). The latter is equivalent to uniform token-level reward.
r is often the negative log- boglliielhoood, or a standard MT metric, e-g., BLEU (Papineni et al. 2002).
RL's goal is to maximize the expected episode reward (denoted with R); i.e., to find
0" arg max R(0)- arg max [r))][[[))]]
0
2.1 REINFORCE
For a given source sentence, and past predictions yei. REINFORCE (Williams) 1992) samples tokens (k is a hyperparameter) (y()), .... (k)) from Pe and updates 0 according to this rule:
A r(y:) Vog((P)))))
The right-hand side of equation is an unbiased estimator of the gradient of the objective function, i.e., IE [A0] x Velt (0). Therefore, REINFORCE is performing a form of stochastic gradient ascent on R. and has similar formal guarantees. From here follows that if R is constant with respect to 0, then the expected A prescribed by REINFORCE zero. We note that r may be shifted by a constant term (called "baseline"), without affecting the optimal value for O.
REINFORCE is used in MT, text generation, and image-to-texx tasks (Liu et al. 2016 Wu al. 2018
Rennie et al., 2017 Shetty et al., 2017: Hendricks et al.) 2016) in isolation, or as part of training adversarial training
(Ranzato et al. 2015). Lately, an especially prominent use for REINFORCE
with discrete data, where another network predicts the reward (GAN). For some recent work on RI.
for NMT, see (Zhang et al. 2016 Li al. 2017 Wu et al. 2017 Yu et al. 2017 Yang et al. 2018
2.2 MINIMUM RISK TRAINING
The term Minimum Risk Training (MRT) is used ambiguously in MT to refer either to the appli- cation of REINFORCE to minimizing the risk (equivalently, to maximizing the expected reward, the negative loss), or more commonly to a somewhat different estimation method, which we term Con- trastive MRT (CMRT) and turn now to analyzing. CMRT was proposed by Och (2003). adapted to
NMT by Shen et al. (2016), and often used since (Ayana et al.] [016)] Neubig 2016 Shen et al.
2017
Edunov al. 2018 Makarov & Clematide] 2018 Neubig al.) 2018))
The method works as follows: at each iteration, sample l tokens = {BI..... 3h from Po. and update 0 according to the gradient of
R(0,S) Σ (u,s(1))r(1)) = Ey [r(v)]
where
P(y))
Qa.s(vi) Enses )ly,)a
Commonly (but not universally), deduplication is performed, so R sums over set of unique values (Sennrich et al. 2017). This changes little in our empirical results and theoretical analysis.
Despite the resemblance in definitions of R (equation 1 and Å (indeed, R sometimes presented as an approximation of R), they differ in two important aspects. First, Q's support is S, so increasing
2

---

Published as a conference paper at ICLR 2020
(u:) for some y necessarily comes the expense of Qyy) for some y E S. In contrast, increas- ing P(y) as in REINFORCE, may come at the expense of P(y) for any y E V. Second, a is a smoothness parameter: the closer is to O, the closer Q is to be uniform.
We show in Appendix A.I that despite its name, CMRT does not optimize R, nor does it optimize E]R]] That is. it may well converge to values that are not local maxima of R. making it theoretically ill-founded | However, given CMRT popularity, the strong results it yielded and the absence of theory for explaining it, we discuss here. Given a sample S, the gradient of R is given by
VR Σ (Q(w)) r(y)) VVlog
Plg.) P(y.) Eqlr| log (())
(3)
where
See Appendix
A.2 A.2
Comparing Equations 2 and 3 the differences between REINFORCE and CMRT are reflected again. First, VR has an additional term, proportional to V log (S)), which yields the contrastive effect. This contrast may improve the rate of convergence since it counters the decrease of probability mass for non-sampled tokens.
Second, given S, the relative weighting of the gradients log P(v)) is proportional to r(y (1.)) or equivalently to r(ys) P(y.)a CMRT with deduplication sums over distinct values in S (equation 3) while REINFORCE sums over all values. This means that the relative weight of the unique value y: is H( in REINFORCE. For and so for a < 1 (as commonly used), more weight given to improhable tokens, which could also have a positive effect on the convergence rate However, if a is too close to O, VR vanishes, as it is not affected by 8. This tradeoff explains the importance of tuning reported in the literature. In $6 we present simulations with CMRT, showing very similar trends as presented by REINFORCE.
the expected value of these relative weights is the same,
3 MOTIVATING DISCUSSION
Implementing a stochastic gradient ascent, REINFORCE is guaranteed to converge to a stationary point of Rt under broad conditions. However, not much is known about its convergence rate under the prevailing conditions in NMT.
We begin with qualitative, motivating analysis of these questions. As work on language generation empirically showed, RNNs quickly learn to output very peaky distributions (Press et al. (017). This tendency is advantageous for generating fluent sentences with high probability, but may also entail slower convergence rates when using RL to fine-tune the model, because RL methods used in text generation sample from the (pretrained) policy distribution, which means they mostly sample what the pretrained model deems to be likely. Since the pretrained model (or policy) is peaky, exploration of other potentially more rewarding tokens will be limited, hampering convergence.
Intuitively, REINFORCE increases the probabilities of successful (positively rewarding) observa- tions, weighing updates by how rewarding they were. When sampling handful of tokens in each context (source sentence x and generated prefix Vıee), and where the number of epochs is not large, it is unlikely that more than a few unique tokens will be sampled from Po (- V<s)). (In practice, is typically between 1 and 20, and the number of epochs between I and 100.) It is thus unlikely that anything but the initially most probable candidates will be observed. Consequently, REINFORCE initially raises their probabilities, even if more rewarding tokens can be found down the list.
We thus hypothesize the peakiness of the distribution, i.e., the probability mass allocated the most probable tokens, will increase, at least in the first phase. We call this the peakiness-effect (PKE), and show it occurs both in simulations ($4.1) 1 and in full-scale NMT experiments ($4.2).
With more iterations, the most-rewarding tokens will be eventually sampled, and gradually gain probability mass. This discussion suggests that training will be extremely sample-inefficient. We assess the rate of convergence empirically in $ finding this to be indeed the case.
Sakaguchi al. (017)) discuss the relation hetween CMRT and REINFORCE, claiming that CMRT is variant. Appendix A.I shows that CMRT does not in fact optimize the same objective.
NNot performing deduplication (e.g. in THUMT (Zhang et al. (217)) results in assigning higher relative weight to high-prohahillty tokens, which may have adverse effect on convergence rate.
3

---

Published as a conference paper at ICLR 2020
0.25
Figure 1: A histogram of the up- date size (x-axis) to the total pre- dicted probability of the 10 most probable tokens (left) or the most probable token (right) in the Con- stant Reward setting. An update is overwhelmingly more probable to increase this probability than to decrease it.
Panannlege
0.15
0.08
0.08
0.06
0.04
0.02
M
0.0
0.0
-0.5
0.5
1,0
-0.
00
0.5
10
Updato sire
Updata azaa
(a) Top 10
(b) Mode
4 THE PEAKINESS EFFECT
We turn to demonstrate that the initially most probable tokens will initially gain probability mass, even if they are not the most rewarding, yielding a PKE.
Caccia et al. (018)) recently observed in the context of language modeling using GANs that per- formance gains similar to those GAN yield can be achieved by decreasing the temperature for the prediction softmax (i.c., making it peakier). However, they proposed no causes for this effect. Our findings propose an underlying mechanism leading to this trend. We return to this point in $7 Fur- thermore, given their findings, it is reasonable assume that our results are relevant for RL use in other generation tasks, whose output space too is discrete, high-dimensional and concentrated.
4.1 CONTROLLED SIMULATIONS
We experiment with I-layer softmax model, that predicts single token i V with probability 2, Σ, .5. 0= (0,bev are the model's parameters. This model simulates the top of any MT decoder that ends with a softmax layer, as essentially all NMT decoders do. To make experiments realistic, similar parameters as those reported in the influential Transformer NMT system (Vaswani iet al. 2017). Specifically, the size of V (distinct BPE tokens) is 30,715, and the initial values for 0 were sampled from 1,000 sets of logits taken from decoding the standard newstest2013 development set, using pretrained Transformer model. The model was pretrained on WMT2015 training data (Bojar et al.) 2015). Hyperparameters are reported in Appendix A.33 We define one of the tokens in V to be the target token and denote it with Ybest- We assign deterministic token reward, this makes learning easier than when relying on approximations and our predictions optimistic. We experiment with two reward functions:
1. Simulated Reward: r(y) 2 for Dbest, r(y) 1 if is one of the 10 initially highest scoring tokens, and r(y) = 0 otherwise. This simulates a condition where the pretrained model is of decent but sub-optimal quality. here is at the scale of popular rewards used in MT, such as GAN-based rewards or BLEU (which are between 0 and 1).
2. Constant Reward: r constantly equal 1, for all tokens. This setting aimed to confirm that PKЕ is not a result of the signal carried by the reward.
Experiments with the first setting were run 100 times, each time for 50K steps, updating 0 after each step. With the second setting, it is sufficient to take a single step at a time, as the expected update after each step is zero, and so any PKE seen in a single step is only accentuated in the next. It is, therefore, more telling to run more repetitions rather than more steps per initialization. We, therefore, sample 10,000 pretrained distributions, and perform a single REINFORCE step.
As RL training lasts about 30 epochs before stopping, samples about 100K tokens per epoch, and as the network already predicts Ybest in about two thirds of the contexts we estimate the number of steps used in practice to be in the order of magnitude of 1M. For visual clarity, we present figures for 50K-100K steps. However, full experiments (with 1M steps) exhibit similar trends: where REINFORCE was not close to converging after 50K steps, the same was true after I IM steps.
We evaluate the peakiness of distribution in terms of the probability of the most probable token (the mode), the total probability of the ten most probable tokens, and the entropy of the distribution (lower entropy indicates more peakiness).
Based on our NMT experiments, which we assume he representative of the error rate of other systems.
4

---

Published as a conference paper at ICLR 2020
good beot
08
0.0
0.8
Probobitt
.08
Probebiitt
0.8
good
0.4
04
6.4
wonst
best
033
good beat
0.2
0.2
0.1
0.0
20
40
80
100
40
80
100
40 Step (thosandds)
50
100
Step (thourands)
Step (thousands)
(a)
(b)
(c)
Figure 2: Token probabilities through REINFORCE training, in the controlled simulations in the Simulated Reward setting. The left/center/rihtt ligures correspond to simulations where the target token (ybeat) was initially the second/third/fourth most probable token. The green line corresponds to the target token, yellow lines to medium-reward tokens and red lines to no-reward tokens.
Results. The distributions become peakier in terms of all three measures; on average, the mode's probability and the 10 most probable tokens increases, and the entropy decreases. Figure pappreeents the histogram of the update size, the difference in the probability of the 10 most probable tokens in the Constant Reward setting, after a single step. Figure [bb depicts similar statisties for the mode. The average entropy in the pretrained model is 2.9 is reduced to 2.85 after one REINFORCE step.
Simulated Reward setting shows similar trends. For example, entropy decreases from to about 0.001 in 100K steps. This extreme decrease suggests it is effectively deterministic policy. PKE is achieved in a few hundred steps, usually before other effects become prominent (see Figure 2). and is stronger than for Constant Reward.
4.2 NMT EXPERIMENTS
Cunulatie purcartage
Pietrnn
Reinforoed
We turn to analyzing a real-world application of REINFORCE to NMT. Important differences between this and the previous simula- tions are: (1) it is rare in NMT for REINFORCE to sample from the same conditional distribution more than handful of times, given the number of source sentences and sentence prefixes V<i (con- texts); and (2) in NMT (- y<z) shares parameters between con- texts, which means that updating Po for one context may influence Po for another.
0.2
0.0
08
Model probability
The cumulative distri- bution of the probability most likely token in the NMT ex- periments. The green distribu- tion corresponds to the pretrained model, and the hlue corresponds to the reinforced model. The y- axis is the proportion of condi- tional probabilities with mode of value < z (the x-axis). Note that a lower cumulative percent- age means a more peaked output distribution. A lower cumulative percentage means a more peaked output distribution.
Figure 3:
We follow the same pretraining in $..1 We then follow Yang cct al. (2018) in defining the reward function based on the expected BLEU score. Expected BLEU is computed by sampling suffixes for the sentence, and averaging the BLEU score of the sampled sen- tences against the reference.
We use early stopping with a patience of 10 epochs, where each epoch consists of 5,000 sentences sampled from the WMT2015 (Bojar et al., 2015) German--English training data. We use = 1. We retuned the learning-rate, and positive baseline settings against the development set. Other hyper-parameters were an exact repli- cation of the experiments reported (Yang
2018
Results. Results indicate an increase in the peakiness of the conditional distributions, Our results are based on sample of 1,000 contexts from the pretrained model, and another (independent) sample from the reinforced model.
The modes of the conditional distributions tend to increase. Figure preueett the distribution of the modes' probability in the reinforced conditional distributions compared with the pretrained model, showing a shift of probability mass towards higher probabilities for the mode, following RL. Another indication of the increased peakiness is the decrease in the average entropy of Pe, which was reduced from 3.45 in the pretrained model to an average of 2.82 following RL. This more modest reduction in entropy (compared to (4.1) might also suggest that the procedure did not converge to the optimal
5

---

Published as a conference paper at ICLR 2020
value for 6, as then would have expected the entropy to substantially drop if not to 0 (overfit), then to the average entropy valid next tokens (given the source and a prefix the sentence).
5 PERFORMANCE FOLLOWING REINFORCE
We now turn to assessing under what conditions it is likely that REINFORCE will lead to an improve- ment in the performance of an NMT system. As in the previous section, we use both controlled simulations and NMT experiments.
5.1 CONTROLLED SIMULATIONS
We use the same model and experimental setup described in Section 4.1 this time only exploring the Simulated Reward setting. as Constant Reward is not expected to converge to any meaningful Results are averaged over 100 conditional distributions sampled from the pretrained model.
Caution should be exercised when determining the learning rate (LR). Common LRs used in the NMT literature are of the scale of 10-4. However, in our simulations, no LR smaller than 0.1 yielded any improvement in R. We thus set the LR to be 0.1. We note that in our simulations, a higher learning rate means faster conver- gence as our reward is noise-free: it is always highest for the best option. In practice, increasing the learning rate may deteriorate re- sults, as it may cause the system to overfit to the sampled instances. Indeed, when increasing the learning rate in our NMT experiments (see below) by an order magnitude, early stopping caused the RL procedure stop without any parameter updates.
1.0
Commutat Aatho persanttge
9.8
0.
0.0
Gold tokio rank by model
Figure 4: Cumulative pereentage of contexts where the pretrained model ranks Ybeat in rank x or below and where it does not rank ybeat first (z = 0). In about hall the cases it is ranked fourth or be- low.
Figure shows the change in Pa over the first 50K REINFORCE steps (probabilities are averaged over 100 repetitions), for case where Ybest was initially the second, third and fourth most probable. Although these are the easiest settings, and despite the high learning rate, it fails to make Ybest the mode of the distribution within 100K steps, unless Ybest was initially the second most probable. cases where Ybest is initially of lower rank than four, it is hard to see any increase in its probability, even after 1M steps.
5.2 NMT EXPERIMENTS
We trained an NMT system, using the same procedure as in Section 4.2 and report BLEU scores the news2014 test After training with expected BLEU reward, indeed see a minor improvement which consistent between trials and pretrained models. While the pretrain BLEU score is 30.31, the reinforced one is 30.73.
Analyzing what words were influenced by the RL procedure, we begin by computing the cumulative probability of the target token Yhest to be ranked lower than given rank according the pretrained model. Results (Figure 4 show that in about half of the cases, Ybest is not among the top three choices of the pretrained model, and we thus expect not gain substantial probability following REINFORCE, according to our simulations.
We next turn compare the ranks the reinforced model assigns the target tokens, and their ranks according to the pretrained model. Figure 6 presents the difference in the probability that Ybest ranked given rank following RL and the probability it is ranked there initially. Results indicate that indeed more target tokens are ranked first, and less second, but little consistent shift of probability mass occurs otherwise across the ten first ranks. It possible that RI. has managed to push Ybeat in some cases between very low ranks (<1,000) to medium-low ranks (between 10 and 1,000), However, token probabilities these ranks are so low that is unlikely to affect the system outputs in any way. This fits well with the results of our simulations that predicted that only the initially top-ranked tokens are likely to change.
In attempt explain the improved BLEU score following RL with PKE, repeat the NMT ex- periment this time using a constant reward 1. Our results present a nearly identical improvement in BLEU, achieving 30.72, and similar pattern in the change of the target tokens' ranks (see Ap-

---

Published as a conference paper at ICLR 2020
Figure The probability of dif- ferent tokens following CMRT, in the controlled simulations in the Simulated Reward setting. The left/right figures correspond to simulations where the target to- ken (ybest) was initially the sec- ond/third most probable token. best
0.7
0.8
0.7
0.5
0.6
Probaiity
0.0
good
0.4
0.3
0.2
good
02
best
The green line corresponds to the target token, yellow lines to medium-rette reward tokens and lines to tokens with r(y) = 0.
0.1
0.0
0.0
20
(thousands)
(thousands)
pendix 8 Therefore, there is room to suspect that even in cases where RL yields an improvement in BLEU, may partially result from reward-innee -nweaadddddeeeeentt factors, such as PKE 4
6 EXPERIMENTS WITH CONTRASTIVE MRT
In 2.2 we showed that CMRT does not, in fact, maximize R, and so does not enjoy the same theoretical guarantees as REINFORCE and similar policy gradient methods. However, being the RL procedure of choice in much recent work we repeat the simulations described in $ and |5| assessing CMRT's performance in these conditions. We experiment with a = 0.005 and & 20, common settings in the literature, and average
1e-3
Probecdly change
-0.5
100 trials.
-10
-15
Figure 5 shows how the distribution Po changes over the course of 50K update steps to 0, where Ybest is taken to be the second and third initially most probable token (Simulated Reward setting). Results are similar trends to those obtained with REINFORCE: MRT succeeds in pushing Ybest to be the highest ranked token if it was initially second, but struggles where was initially ranked third or below. We only observe small PKE in MRT. This is probably due to the contrastive effect, which means that tokens that were not sampled do not lose probability mass.
Gok tolion rank
Figure 6: Difference between the ranks of Ybeat the rein- forced and the pretrained model. Each column æ corresponds to the difference in the probabil- ity that Yhest is ranked in rank x in reinforced model, and the same probability in the pre- trained model.
All graphs we present here allow sampling the same token more than once in each batch (i.e., S is sample with replacements). Simulations with deduplication show similar results.
7 DISCUSSION
In this paper, we showed that the type distributions used in NMT entail that promoting the target token to be the mode is likely to take prohibitively long times for existing RL practices, except under the best conditions (where the pretrained model is "nearly" correct). This leads us to conclude that observed improvements from using RL for NMT are likely due either to fine-tuning the most probable tokens in the pretrained model (an effect which may be more easily achieved using rerank- ing methods, and uses but little of the power of RL methods), or effects unrelated the signal carried by the reward, such as PKE. Another contribution of this paper is in showing that CMRT does not optimize the expected reward and is thus theoretically unmotivated.
A number of reasons lead us to believe that in our NMT experiments, improvements are not due to the reward function, but to artefacts such as PKE. First, reducing constant baseline from r, so as to make the expected reward zero, disallows learning. This is surprising, as REINFORCE, generally and in our simulations, converges faster where the reward is centered around zero, and so the fact that this procedure here disallows learning hints that other factors in play. As PKE can be observed even where the reward is constant (if the expected reward is positive; see $4.1 this suggests PKE
"We tried several other reward functions well, all of which got BLEU scores of 30.73-30.84. This improvement very stable across metrics, trials and pretrained models.
7

---

Published as a conference paper at ICLR 2020
may play role here. Second, we observe more peakiness in the reinforeed model and in such cases, we expect improvements in BLEU (Caccia et al. 2018). Third, we achieve similar results with a constant reward in our NMT experiments ($5.2). Fourth, that asymptotic convergence is not reached in any but the easiest conditions ($5.)).
controlled simulations show
Our analysis further suggests that gradient clipping, sometimes used in NMT (Zhang et al. 2016 Wieting et al. 2019), expected to hinder convergence further. It should be avoided when using REINFORCE it violates REINFORCE'S assumptions.
The per-token sampling as done in our experiments is more exploratory than beam search (Wu et al., 2018), reducing PKE. Furthermore, the latter does not sample from the behavior policy, but does not properly account for being off-policy in the parameter updates.
Adding the reference the sample S, which some implementations allow (Sennrich et al. 2017) may help reduce the problems of never sampling the target tokens. However, as Edunov et al. (2018) point out, this practice may lower results, as it may destabilize training by leading the model to improve over outputs it cannot generalize over, as they are very different from anything the model assigns a high probability to, the cost of other outputs.
CONCLUSION
The standard MT scenario poses several uncommon challenges for RL. First, the action space in MT problems is -ige-simensional discrete space (generally in the size of the vocabulary of the target language or the product thereof for sentences). This contrasts with the more common sce- nario studied by contemporary RL methods, which focuses mostly on much smaller discrete action spaces (e.g., video games (Mnih al., 2015) 2016)). or continuous action spaces of relatively low dimensions (e.g., simulation of robotic control tasks (Lillicrap et al., (015)). Second, reward for MT naturally very sparse almost all possible sentences are "wrong" (hence, not rewarding) in a given context. Finally, it is common in MT to use RL for tuning pretrained model. Using pretrained model ameliorates the last problem. But then, these pretrained models are in general quite peaky, and because training is done on-policy that is, actions are being sampled from the same model being optimized exploration is inherently limited.
Here argued that, taken together, these challenges result in significant weaknesses for current RL practices for NMT, that may ultimately prevent them from being truly useful. At least some of these challenges have been widely studied the RL literature, with numerous techniques developed to address them, but were not yet adopted in NLP. We turn to discuss some of them.
Off Off-policy methods, in which observations are sampled from different policy than the one being currently optimized, are prominent in RL (Watkins & Dayan 1992. Sutton & Barto 1998), and were also studied in the context of policy gradient methods (Degris et al. 2012] Silver al. 2014). In principle, such methods allow learning from a more "cxploratory" policy. Moreover, a key mo- tivation for using a in CMRT is smoothing: off-policy sampling allows smoothing while keeping convergence guarantees.
In basic form, exploration in REINFORCE relies on stochasticity in the action-selection (in MT, this is due to sampling). More sophisticated exploration methods have been extensively studied, for example using measures for the exploratory usefulness of states or actions (Fox et al. 2018), or relying on parameter-space noise rather than action-space noise (Plappert et al. 2017).
For MT, an additional challenge that even effective exploration (sampling diverse sets of obser- vations), may not be enough, since the state-action space is too large to be effectively covered, with almost all sentences being not rewarding. Recently, diversity- based and multi-goal methods for RL proposed tackle similar challenges (Andrychowicz et al.) 2017]] Ghosh et al. 2018 Eysen- bach et al.) 2019). We believe the adoption of such methods is a promising path forward for the application of RL in NL.P
8

---

Published as a conference paper at ICLR 2020
9 ACKNOWLEDGMENTS
This work was supported by the Israel Science Foundation (grant no. 929/17) and by the HUJ Cyber Security Research Center in conjunction with the Israel National Cyber Bureau in the Prime Minister's Office.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systems, pp. 5048-5055 2017.
Shiqi Shen Ayana, Zhiyuan Liu, and Maosong Sun. Neural headline generation with minimum risk training. arXiv preprint arXiv:/604.0/900 2016.
Ondrej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Kochn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. Findings of the 2015 workshop statistical machine translation. WMT@ EMNLP, 2015.
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. arXiv preprint arXiv: 11Xiv:/8/1.02549, 2018. URL https:
// //axxiv.org//dff/// /11/p//////111....///// 02549.
Thomas Degris, Martha White, and Richard S Sutton. Off-policy actor-critic. arXiv preprint 12X5v: 48399, 2012.
Sergey Edunov, Myle Out, Michael Auli, David Grangier, and Marc' Aurelio Ranzato. Classical structured prediction losses for sequence to sequence learning. In Proceedings of the 2018 Con- ference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume (Long Papers). pp. 355-364, Association for Computational Linguisties, 2018. doi: N..18653/v1/N18-1 1033. URL. /attp://ast web. org/ antho o N18- 11333
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representa- tions, 2019. URL https: : wwwpreveeeeeeeeeeeeeeeeeee wwwpaarreieee.eeeeteeeeeeeee
Lior Fox, Leshem Choshen, and Yonatan Loewenstein. Dora the explorer: Directed outreaching reinforcement action-selection. ICLR, abs/ 000.1/404..411.... 2018.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. In International Conference Learning Representations, 2018.
URL httphttpss//opeeeeeeeee net www.nee///ura/. forum? rwe
Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell. Generating visual explanations. In ECCV, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2015.
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison- Burcch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguisticy Companion Volume Proceedings of the Demo and Poster Sessions, pp. 177-180, 2007.
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learn- ing for neural dialogue generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2157-2169, Copenhagen, Denmark. September 2017. As- sociation for Computational Linguistics. URL https: : www. aclweb.
D17-1230

---

Published as a conference paper at ICLR 2020
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy, Optimization of image description metrics using policy gradient methods. CoRR, abs/1612.003777 2. 2016.
Peter Makarov and Simon Clematide. Neural transition-based string transduction for limited- resource setting in morphology. In COLING, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuogiu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937, 2016.
Graham Neubig. Lexicons and minimum risk training for neural machine translation: Naist-cmu at war2016. In WAT@ COLING, 2016.
Graham Neubig, Matthias Sperber, Xinyi Wang, Matthieu Felix, Austin Matthews, Sarguna Pad- manabhan, Ye Qi, Devendra Singh Sachan, Philip Arthur, Pierre Godard, John Hewitt, Rachid Riad, and Liming Wang. Xnmt: The extensible neural machine translation toolkit. In AMTA, 2018.
Franz. Josef Och. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volle Volume 1, pp. 160-167. Association for Computational Linguistics, 2003.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pp. 31 1-1318.. Association for Computational Linguistics, 2002. URL
www. http:::/www.aclvl// org/ http://///acoooooooo//o/o/o
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. 2017.
arXiv preprint 1706vv1700..11
O. Press, A. Bar, B. Bogin, J. Berant, and L. Wolf. Language generation with recurrent generative adversarial networks without pre-training. In Fist Workshop on Learning to Generate Natural Language@ ICMI, 2017.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint arXiv: 2015.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1008-7024, 2017.
Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. Grammatical error correction with neural reinforcement learning. arXiv preprint arXiv: 177.00907. 2017.
Philip Schulz, Wilker Aziz, and Trevor Cohn. A stochastic decoder for neural machine translation. In ACI, 2018
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 1715-1725, 2016. URL http: www.. 1162
aclweb. wwtp:////r..aatooogo..oooo
Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler,
Marcin Junczys-Dowmunt Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. Nematus: toolkit for neural machine translation. In EACI., 2017.
10

---

Published as a conference paper at ICLR 2020
Shiqi Shen, Yong Cheng. Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics /Volume I: Long Papers), pp. 118316686999 Association for Computational Linguistics, 2016. doi: 10.1 10.18653/v1/P16-1111 1159. URL http: //s///ac. web.
org/antho ogy
Shiqi Shen, Yang Liu, and Maosong Sun. Optimizing non-decomposable evaluation metrics for neural machine translation. Journal of Computer Science and Technology, 32:796-804, 2017.
Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine to human captions by adversarial training. In 20/7 IEEE International Conference Computer Vision (ICCV), pp. 4155-4164. IEEE, 2017.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1998.
G. Tevet, G. Habib, Shwartz, and J. Berant. Evaluating text GANs language models. arXiv preprint arXiv:1810. 12686, 2018.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26- 31, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in Neural Informa- fion Processing Systems, pp. 5998 6008, 2017. httpst:s: /ttp::://ppapeeee wwpprspppp.pppeppppeepeeeeeeee.. www.. 7181-attent is- you nee..
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):27--292 1992.
John Wieting, Taylor Berg- Kirkpatrick, Kevin Gimpel, and Graham Neubig, Beyond BLEU: Train- ing neural machine translation with semantic similarity. In The 57th Annual Meeting of the Association for Computational Linguistics (ACL). Florence, Italy, July 2019. URL https:
Ronald J Williams. Simple statistical follient- algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-255 256, 1992.
Lijun Wu, Yingee Xia, Li Zhao, Fei Tian, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. Adversarial neural machine translation. arXiv preprint /tt4.:/70..6635 2017.
Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie Yan Liu. A study of reinforcement learning for neural machine translation. In EMNLP, 2018.
Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. Improving neural machine translation with condi- tional sequence generative adversarial nets. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long Papers), pp. 1346-1355. Association for Computational Linguistics, 2018. doi: 2.118633/1/NiS/ 11222121811222. URL httt:://caacccc http::///wwwwwb/oooo org/ anthology/hol 1122
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seggan: Sequence generative adversarial nets with policy gradient. In AAAI, pp. 1852-2858, 2017.
Jiac heng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huanbo Luan, and Yang Liu. Thumt: An open source toolkit for neural machine translation. arXiv preprint 1rXiv:1706.064//1 2017.
Yizhe Zhang, Zhe Gan, and Lawrence Carin. Generating text via adversarial training. In NIPS workshop on Adversarial Training, volume 21, 2016.

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 CONTRASTIVE MRT DOES NOT MAXIMIZE THE EXPECTED REWARD
We hereby detail a simple example where following the Contrastive MRT method (see 22. does not converge to the parameter value that maximizes R.
Let be real number in [0, 0.5]. and Po family of distributions over three values a, c such that:
0
Σ a
(z)
202
2 b
1
20
Letr(a) 1,6(6) r(c) 0.5. The expected reward as function of 0 is:
R(0) 0 0.5(1 00 0220²)
R(O) is uniquely maximized by 0" = 0.25.
Table details the possible samples of size k = 2. their probabilities, the corresponding It and its gradient. Standard numerical methods show that E[V R] over possible samples S is positive for
0 E (0, 7) and negative for 0 E (y, 0.5), where R 0.295. This means that for any initialization of
0 E (0, 0.5). Contrastive MRT will converge y if the learning rate is sufficiently small. For 0 0,
R 0.5, and there will be no gradient updates, so the method will converge to = Neither of these values maximizes R(0).
We note that by using some g (0) the y could arbitrarily far from 0* g could also map to (-inf, inf) more often used in neural networks parameters.
We further note that resorting maximizing [[[]] instead, does not maximize R(0) either. Indeed, plotting [R] as a function of 0 for this example, yields maximum at 0.32.
S
P(S)
R
V R
{a, 6}
403
T4-20
(1+:26) 2
{a, c}
20(1-0-202)
0.5 +
+1
2 402
2(1 24²
{b. c]
40 (1-0-20²)
1- 0020000
0" 20
(1-8)2
0, a
6²
1
0
b,b
40
0 0.5
0
c,c
(1-0-202)2
0
Table 1: The gradients of R for each possible sample S. The batch size k 2. Rows correspond different sampled outcomes. V R is the gradient of R given the corresponding value for S.
A.2 DERIVING THE GRADIENT OF Rt
Given S, recall the definition of R:
R(0,S) Σ (o.s(v))((s))
Taking the deriviative w.r.t. 0:
12

---

Published as a conference paper at ICLR 2020
VP(x). aP())) -(() VZ(S).
(())
2(S)²
Σ r(vi) aVP(())
Q V2(5)
Q(vi))
P(y.)
Z(S)
Σ ((n)Q(wi) (aVl:)) P(u)) V logg((9))
(rlu))(u=)Vlog P(u:)) F log Z(S)
0.9
0.9
good
088
0.7
0.7
Probab
0.6
0.5
good
0.5
good
2 0.9
0.3
02
0.2
0.1
0.0
0.0
40 Step (thoueends)
Slep (thoueands)
Step (thousende)
(c)
The probability of different tokens following REINFORCE, in the controlled simulations in the Con- stant Reward setting. The left/center/right figures correspond to simulations where the target token (ybest) was initially the second/third/fourth most probable token. The green line corresponds the target token, yellow lines medium-reward tokens and red lines tokens with r(y) O.
Figure 7:
A.3 NMT IMPLEMENTATION DETAILS
True casing and tokenization were used (Koehn et al. 2007). including escaping html symbols and "_" that represents compound changed into separate token of Some preprocessing used before us converted the latter to #TAAT*#-#*ATAN but standard tokenizers in use process that into 11 different tokens, which over-represents the significance of that character when BLEU calculated. BPE (Sennrich et al. 2016) extracted 30,715 tokens. For the MT experiments we used layers in the encoder and the decoder. The size of the embeddings was 512. Gradient clipping was used with size of 5 for pre-training (see Discussion on why not to use it in training). We did not use attention dropout, but 0.1 residual dropout rate was used. In pretraining and training sentences of more than 50 tokens were discarded. Pretraining and training were considered finished when BLEU did not increase in the development set for 10 consccutive evaluations, and evaluation was done every 1,000 and 5,000 for batches of size 100 and 256 for pretraining and training respectively. Learning rate used for rmsprop (Tieleman & Hinton 2012) was 0.01 in pretraining and for adam (Kingma & Ba 2013) with decay
0.005 for training. 4,000 learning rate up steps used. Pretraining took about 7 days with 4 GPUs, afterwards, training took roughly the same time. Monte Carlo used 20 sentence rolls per word.
A.4 DETAILED RESULTS FOR CONSTANT REWARD SETTING
We present graphs for the constant reward setting in Figures and Trends are similar to the ones obtained for the Simulated Reward setting.
13

---

Published as a conference paper at ICLR 2020
1e-3
1.5
Probability change
1.0
0.5
0.0
-0.5
-1.0
-1.5
2
3
4
5
6
7
8
10
1000
1000+
Gold token rank
Figure 8: Difference between the ranks of ybest in the reinforced with constant reward and the pretrained model. Each column æ corresponds to the difference in the probability that Yleat is ranked in rank in the reinforeed model, and the same probability in the pretrained model.
14


Reviews:
[{"review_id": "ryx8p2xW9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper first theoretically demonstrates that a commonly used reinforcement learning method for neural sequence-to-sequence models (e.g. in NMT), contrastive minimum risk training (CMRT), is not guaranteed to converge to local (let alone global) optima of the reward function. The paper then empirically demonstrates that the REINFORCE algorithm, while not subject to the same theoretical flaws as CMRT, in practice fails to improve NMT models unless the baseline model is already \"nearly correct\" (i.e. the correct tokens were already within the few most probable tokens before the fine-tuning steps with REINFORCE). In fact, some of the performance gains of using REINFORCE/CMRT can be attributed to making the model's output probability distribution more peaked, and not necessarily from making the target tokens more probable as commonly assumed.\n\nOverall, this is an excellent paper that offers significant contributions for the field. I have summarised the key strengths of the paper below, along with several suggestions and questions that I hope will be addressed by the authors. Based on my assessment, I am recommending a rating of \"Accept\" for this paper.\n\nStrengths:\n1. The paper is very well-written and well-structured. It starts off by pointing out the theoretical limitations of CMRT (and concisely recaps the key differences between CMRT and REINFORCE), and continues with an extensive set of experiments that clearly illustrates the limitations of REINFORCE in practice. \n\n2. I also like the use of both controlled simulations (including one where the reward is constant) and NMT experiments with real data. The controlled simulations are useful to abstract away from the full complexity of the model and investigate what happens under various control scenarios, while the NMT experiments demonstrate that the findings still hold under the realistic setup.\n\n3. The findings are really interesting and clearly illustrate the limitations of existing REINFORCE/CMRT methods for neural sequence-to-sequence models. It is very interesting to see that REINFORCE fails to make the target token most probable when the initial model ranks the target token as the third or more probable tokens under the model (Figure 2), even under the simple controlled simulations, which highlights the prohibitively high sample complexity of the model. \n\n4. The peakiness effect hypothesis (i.e. attributing the gains of REINFORCE to making the output distribution more peaked, and not necessarily by making the target tokens more probable) is well-supported by the paper's empirical evidence. It is really illuminating that using a constant reward of 1 leads to the same BLEU score as actually optimising for BLEU in NMT (Section 5.2).\n\nSuggestions and questions:\n1. Section 4.2 (NMT Experiments) indicates that REINFORCE fine-tuning is done for 10 epochs, with 5,000 sentences per epoch, and k=1. Considering the enormous discrete sample space, one could expect that using multi-sample REINFORCE (i.e. k > 1) and training the model for many more epochs might mitigate the identified problems to some extent, and thus change the findings. Training for 5,000 sentences * 10 epochs may just not be enough for the RL fine-tuning to make a big difference.\n\n2. In Figure 1, the x-axis is the \"Update Size\" with a scale between -1.0 and 1.0. This \"Update Size\" variable is not really explained in the paper, and why the scale is between -1.0 and 1.0.\n\n3. In my understanding, the controlled simulations (Section 4.1) is done at the word-level (including word-level rewards, as opposed to the NMT experiments which are done at the sequence-level with sentence-level rewards). If this is the case, this should be made clearer.\n\n4. To make Figure 3 easier to understand, the caption should indicate that a lower cumulative percentage means a more peaked output distribution.\n\n5. Rather than breaking down the analysis by where the target token is ranked by the initial, pre-RL model (e.g. the target token is ranked second/third best in Figures 2 and 5), perhaps what really matters is the probability assigned to the target token. For instance, even if the target token is ranked third best by the initial model, there will be a big difference whether it is assigned a probability of 0.1 or 0.01 (i.e. the latter case is much less likely to be sampled, which would exacerbate the problem). Including this analysis might help strengthen the paper further.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyJgzPCYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the context of neural machine translation, limitations of some reinforcement learning methods, in particular REINFORCE and contrastive minimum risk training (MRT), are analyzed. The authors argue that MRT doesn't optimize the expected reward. Moreover, they show that using REINFORCE, with either realistic or dummy constant rewards, may lead to a peakier distribution. Similar BLEU scores are obtained with either type of rewards, which is an interesting and perplexing result (in my opinion). For both REINFORCE and MRT, the paper shows that unless the gold token was already amongst the most probable after pre-training, it takes many samples for it to become the most likely output, which limits the usefulness of on-policy RL approaches.\n\nI lean slightly towards rejection because the scope of the paper is somewhat too limited. The experimental section mostly covers REINFORCE without a baseline (except for validation in section 4.2, but no results are shown varying the baseline), as well as MRT in a restricted scenario. However, the analysis may still be beneficial to the community.\n\nEstablishing that minimum risk training doesn't optimize the expected reward is a valuable observation. Can the optimized and expected rewards be arbitrarily far?\n\nThe experiments demonstrating the sample inefficiency of REINFORCE (without baseline) and MRT when the best tokens have low initial probability (relative to other tokens) raise important questions about the effectiveness of these approaches.\n\nThe analysis of REINFORCE assumes non-negative rewards, which likely contributes to the peakiness effect (PKE). I would assume the peakiness effect to be mostly neutral with normalized rewards, or diminish with negative rewards (on average). It is unclear how different baselines would affect the results.\n\nFor MRT, only synthetic experiments are run. Given that the peakiness effect is much weaker than for REINFORCE, it would be useful to know results (BLEU) on the same MT task.\n\nOther questions and remarks:\n\nIn appendix A.1, the parameter value is bounded (to maintain a valid probability distribution), which is generally not the case within a neural network. Does this distinction matter?\n\nDo you have the RL learning curves? What could we learn from them?\n\nIt is unclear how much the claims generalize to other methods such as actor-critic (Bahdanau et al. An Actor-Critic Algorithm for Sequence Prediction). \n\nIn appendix A.1, some values are wrong. Do these affect the final result?\nFor example, P({a,c}) should be $2 \\theta (1-\\theta-2 \\theta^2)$.\nFor S={a,c}, \\nabla \\tilde{R} should be $\\frac{1+2 \\theta^2}{2(1-2 \\theta^2)^2}$.\nFor S={b,c} \\nabla \\tilde{R} should be $\\frac{\\theta (\\theta-2)}{(1-\\theta)^2}$.\n\n-----\nUpdate to Review 2\n\nI appreciate that the authors corrected appendix A.1. As a nitpick, for the gradient of the reward of {a, c}, there is a x that should be replaced by \\theta.\n\nWhile the scope of the paper is somewhat limited, the theoretical contributions about MRT are valuable. The experiments would be more convincing with stronger baselines, but the current paper may already generate useful discussions within the community. As such, I updated my score to \"Weak accept\", although I won't fight for acceptance.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJl8qb66KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work carefully studies RL for neural machine translation and draws several conclusions:\n1. One of the most common RL methods for MT does not optimize the expected reward, as well as show that other methods take an infeasibly long time to converge. \n2. RL practices in MT are likely to improve performance only where the pre-trained parameters are already close to\nyielding the correct translation. \n3. Observed gains may be due to effects unrelated to the training signal, concretely, changes in the shape of the distribution curve.\n\nI have questions about several technical claims, which lead to my doubt about the technical correctness of the paper:\n\n\t1. [updated after checking authors' response] \"it may well converge to values that are not local maxima of R, making it theoretically ill-founded.\" Previously I had concerns about the convergence of REINFORCE with deep NNs as its policy. After checking the references provided by the authors, the local convergence can indeed be guaranteed.  \n \n\t2. \"reducing a constant baseline from r, so as to make the expected reward zero, disallows learning.\" This conflicts with my intuition. Where is the experiment supporting this claim?\n\n\t3. \"MRT succeeds in pushing ybest to be the highest ranked token if it was initially second, but struggles where it was initially ranked third or below.\" Why ybest in third position cannot be boosted while second can be boosted? Figure 5 only shows two tokens, which cannot lead to any meaningful statistical conclusions. I'd like to see the statistic numbers: \n\t\ta. how many ybest in second are boosted to first in training? How many are not?\n\t\tb. how many ybest in third are boosted to first or second in training? How many are not?\nHow about other positions?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In my opinion, the main strength of this work is the theoretical analysis and some observations that may be of great interest to the NLP community in terms of better analyzing the performance of RL (and "RL-like") methods as optimizers. The main weakness, as pointed out by R3, the limited empirical analysis.

I would urge the authors to take R3's advice and attempt insofar as possible to broaden the scope of the empirical analysis in the final. I believe that this is important for the paper to be able to make its case convincingly.

Nonetheless, I do think that the paper makes a significant contribution that will be of interest to the community, and should be presented at ICLR. Therefore, I would recommend for it to be accepted.
Decision: Accept (Poster)