On the Weaknesses of Reinforcement Learning for Neural Machine Translation

1 INTRODUCTION . Reinforcement learning ( RL ) is an appealing path for advancement in Machine Translation ( MT ) , as it allows training systems to optimize non-differentiable score functions , common in MT evaluation , as well as tackling the “ exposure bias ” ( Ranzato et al. , 2015 ) in standard training , namely that the model is not exposed during training to incorrectly generated tokens , and is thus unlikely to recover from generating such tokens at test time . These motivations have led to much interest in RL for text generation in general and MT in particular ( see §2 ) . Various policy gradient methods have been used , notably REINFORCE ( Williams , 1992 ) and variants thereof ( e.g. , Ranzato et al. , 2015 ; Edunov et al. , 2018 ) and Minimum Risk Training ( MRT ; e.g. , Och , 2003 ; Shen et al. , 2016 ) . Another popular use of RL is for training GANs ( Yang et al. , 2018 ; Tevet et al. , 2018 ) . Nevertheless , despite increasing interest and strong results , little is known about what accounts for these performance gains , and the training dynamics involved . We present the following contributions . First , our theoretical analysis shows that commonly used approximation methods are theoretically ill-founded , and may converge to parameter values that do not minimize the risk , nor are local minima thereof ( §2.2 ) . Second , using both naturalistic experiments and carefully constructed simulations , we show that performance gains observed in the literature likely stem not from making target tokens the most probable , but from unrelated effects , such as increasing the peakiness of the output distribution ( i.e. , the probability mass of the most probable tokens ) . We do so by comparing a setting where the reward is informative , vs. one where it is constant . In §4 we discuss this peakiness effect ( PKE ) . Third , we show that promoting the target token to be the mode is likely to take a prohibitively long time . The only case we find , where improvements are likely , is where the target token is among the first 2-3 most probable tokens according to the pretrained model . These findings suggest that REINFORCE ( §5 ) and CMRT ( §6 ) are likely to improve over the pre-trained model only under the best possible conditions , i.e. , where the pre-trained model is “ nearly ” correct . We conclude by discussing other RL practices in MT which should be avoided for practical and theoretical reasons , and briefly discuss alternative RL approaches that will allow RL to tackle a larger class of errors in pre-trained models ( §7 ) . 2 RL IN MACHINE TRANSLATION . An MT system generates tokens y = ( y1 , ... , yn ) from a vocabulary V one token at a time . The probability of generating yi given preceding tokens y < i is given by Pθ ( ·|x , y < i ) , where x is the source sentence and θ are the model parameters . For each generated token yi , we denote with r ( yi ; y < i , x , y ( ref ) ) the score , or reward , for generating yi given y < i , x , and the reference sentence y ( ref ) . For brevity , we omit parameters where they are fixed within context . For simplicity , we assume r does not depend on following tokens y > i . We also assume there is exactly one valid target token , as de facto , training is done against a single reference ( Schulz et al. , 2018 ) . In practice , either a token-level reward is approximated using MonteCarlo methods ( e.g. , Yang et al. , 2018 ) , or a sentence-level ( sparse ) reward is given at the end of the episode ( sentence ) . The latter is equivalent to a uniform token-level reward . r is often the negative log-likelihood , or a standard MT metric , e.g. , BLEU ( Papineni et al. , 2002 ) . RL ’ s goal is to maximize the expected episode reward ( denoted with R ) ; i.e. , to find θ∗ = arg max θ R ( θ ) = arg max θ Ey∼Pθ [ r ( y ) ] ( 1 ) 2.1 REINFORCE . For a given source sentence , and past predictions y < i , REINFORCE ( Williams , 1992 ) samples k tokens ( k is a hyperparameter ) S = ( y ( 1 ) , ... , y ( k ) ) from Pθ and updates θ according to this rule : ∆θ ∝ 1 k k∑ i=1 r ( yi ) ∇ log ( Pθ ( yi ) ) ( 2 ) The right-hand side of equation 2 is an unbiased estimator of the gradient of the objective function , i.e. , E [ ∆θ ] ∝ ∇θR ( θ ) . Therefore , REINFORCE is performing a form of stochastic gradient ascent on R , and has similar formal guarantees . From here follows that if R is constant with respect to θ , then the expected ∆θ prescribed by REINFORCE is zero . We note that r may be shifted by a constant term ( called a “ baseline ” ) , without affecting the optimal value for θ . REINFORCE is used in MT , text generation , and image-to-text tasks ( Liu et al. , 2016 ; Wu et al. , 2018 ; Rennie et al. , 2017 ; Shetty et al. , 2017 ; Hendricks et al. , 2016 ) – in isolation , or as a part of training ( Ranzato et al. , 2015 ) . Lately , an especially prominent use for REINFORCE is adversarial training with discrete data , where another network predicts the reward ( GAN ) . For some recent work on RL for NMT , see ( Zhang et al. , 2016 ; Li et al. , 2017 ; Wu et al. , 2017 ; Yu et al. , 2017 ; Yang et al. , 2018 ) . 2.2 MINIMUM RISK TRAINING . The term Minimum Risk Training ( MRT ) is used ambiguously in MT to refer either to the application of REINFORCE to minimizing the risk ( equivalently , to maximizing the expected reward , the negative loss ) , or more commonly to a somewhat different estimation method , which we term Contrastive MRT ( CMRT ) and turn now to analyzing . CMRT was proposed by Och ( 2003 ) , adapted to NMT by Shen et al . ( 2016 ) , and often used since ( Ayana et al. , 2016 ; Neubig , 2016 ; Shen et al. , 2017 ; Edunov et al. , 2018 ; Makarov & Clematide , 2018 ; Neubig et al. , 2018 ) . The method works as follows : at each iteration , sample k tokens S = { y1 , . . . , yk } from Pθ , and update θ according to the gradient of R̃ ( θ , S ) = k∑ i=1 Qθ , S ( yi ) r ( yi ) = Ey∼Q [ r ( y ) ] where Qθ , S ( yi ) = P ( yi ) α∑ yj∈S P ( yj ) α Commonly ( but not universally ) , deduplication is performed , so R̃ sums over a set of unique values ( Sennrich et al. , 2017 ) . This changes little in our empirical results and theoretical analysis . Despite the resemblance in definitions ofR ( equation 1 ) and R̃ ( indeed , R̃ is sometimes presented as an approximation of R ) , they differ in two important aspects . First , Q ’ s support is S , so increasing Q ( yi ) for some yi necessarily comes at the expense of Q ( y ) for some y ∈ S. In contrast , increasing P ( yi ) , as in REINFORCE , may come at the expense of P ( y ) for any y ∈ V . Second , α is a smoothness parameter : the closer α is to 0 , the closer Q is to be uniform . We show in Appendix A.1 that despite its name , CMRT does not optimize R , nor does it optimize E [ R̃ ] . That is , it may well converge to values that are not local maxima of R , making it theoretically ill-founded.1 However , given CMRT popularity , the strong results it yielded and the absence of theory for explaining it , we discuss it here . Given a sample S , the gradient of R̃ is given by ∇R̃ = α k∑ i=1 ( Q ( yi ) · r ( yi ) · ∇ logP ( yi ) ) − EQ [ r ] ∇ logZ ( S ) ( 3 ) where Z ( S ) = ∑ i P ( yi ) α . See Appendix A.2 . Comparing Equations 2 and 3 , the differences between REINFORCE and CMRT are reflected again . First , ∇R̃ has an additional term , proportional to ∇ logZ ( S ) , which yields the contrastive effect . This contrast may improve the rate of convergence since it counters the decrease of probability mass for non-sampled tokens . Second , given S , the relative weighting of the gradients∇ logP ( yi ) is proportional to r ( yi ) Q ( yi ) , or equivalently to r ( yi ) P ( yi ) α. CMRT with deduplication sums over distinct values in S ( equation 3 ) , while REINFORCE sums over all values . This means that the relative weight of the unique value yi is r ( yi ) | { yi∈S } |k in REINFORCE . For α = 1 the expected value of these relative weights is the same , and so for α < 1 ( as is commonly used ) , more weight is given to improbable tokens , which could also have a positive effect on the convergence rate.2 However , if α is too close to 0 , ∇R̃ vanishes , as it is not affected by θ . This tradeoff explains the importance of tuning α reported in the literature . In §6 we present simulations with CMRT , showing very similar trends as presented by REINFORCE . 3 MOTIVATING DISCUSSION . Implementing a stochastic gradient ascent , REINFORCE is guaranteed to converge to a stationary point of R under broad conditions . However , not much is known about its convergence rate under the prevailing conditions in NMT . We begin with a qualitative , motivating analysis of these questions . As work on language generation empirically showed , RNNs quickly learn to output very peaky distributions ( Press et al. , 2017 ) . This tendency is advantageous for generating fluent sentences with high probability , but may also entail slower convergence rates when using RL to fine-tune the model , because RL methods used in text generation sample from the ( pretrained ) policy distribution , which means they mostly sample what the pretrained model deems to be likely . Since the pretrained model ( or policy ) is peaky , exploration of other potentially more rewarding tokens will be limited , hampering convergence . Intuitively , REINFORCE increases the probabilities of successful ( positively rewarding ) observations , weighing updates by how rewarding they were . When sampling a handful of tokens in each context ( source sentence x and generated prefix y < i ) , and where the number of epochs is not large , it is unlikely that more than a few unique tokens will be sampled from Pθ ( ·|x , y < i ) . ( In practice , k is typically between 1 and 20 , and the number of epochs between 1 and 100 . ) It is thus unlikely that anything but the initially most probable candidates will be observed . Consequently , REINFORCE initially raises their probabilities , even if more rewarding tokens can be found down the list . We thus hypothesize the peakiness of the distribution , i.e. , the probability mass allocated to the most probable tokens , will increase , at least in the first phase . We call this the peakiness-effect ( PKE ) , and show it occurs both in simulations ( §4.1 ) and in full-scale NMT experiments ( §4.2 ) . With more iterations , the most-rewarding tokens will be eventually sampled , and gradually gain probability mass . This discussion suggests that training will be extremely sample-inefficient . We assess the rate of convergence empirically in §5 , finding this to be indeed the case . 1Sakaguchi et al . ( 2017 ) discuss the relation between CMRT and REINFORCE , claiming that CMRT is a variant . Appendix A.1 shows that CMRT does not in fact optimize the same objective . 2Not performing deduplication ( e.g . in THUMT ( Zhang et al. , 2017 ) ) results in assigning higher relative weight to high-probability tokens , which may have an adverse effect on convergence rate .

Summary: This paper first theoretically demonstrates that a commonly used reinforcement learning method for neural sequence-to-sequence models (e.g. in NMT), contrastive minimum risk training (CMRT), is not guaranteed to converge to local (let alone global) optima of the reward function. The paper then empirically demonstrates that the REINFORCE algorithm, while not subject to the same theoretical flaws as CMRT, in practice fails to improve NMT models unless the baseline model is already "nearly correct" (i.e. the correct tokens were already within the few most probable tokens before the fine-tuning steps with REINFORCE). In fact, some of the performance gains of using REINFORCE/CMRT can be attributed to making the model's output probability distribution more peaked, and not necessarily from making the target tokens more probable as commonly assumed.