Title: CONNECTING THE DOTS BETWEEN MLE AND RL FOR SEQUENCE GENERATION Anonymous

Abstract: Sequence generation models such as recurrent networks can be trained with a diverse set of learning algorithms. For example, maximum likelihood learning is simple and efficient, yet suffers from the exposure bias problem. Reinforcement learning like policy gradient addresses the problem but can have prohibitively poor exploration efficiency. A variety of other algorithms such as RAML, SPG, and data noising, have also been developed from different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation, and show that the apparently divergent algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of reward function and a couple of hyperparameters. The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency. Besides, based on the framework, we present a new algorithm that dynamically interpolates among the existing algorithms for improved learning. Experiments on machine translation and text summarization demonstrate the superiority of the proposed algorithm.

1. INTRODUCTION:

Sequence generation is a ubiquitous problem in many applications, such as machine translation (Wu et al., 2016; Sutskever et al., 2014) , text summarization (Hovy & Lin, 1998; Rush et al., 2015) , image captioning (Vinyals et al., 2015; Karpathy & Fei-Fei, 2015) , and so forth. Great advances in these tasks have been made by the development of sequence models such as recurrent neural networks (RNNs) with different cells (Hochreiter & Schmidhuber, 1997; Chung et al., 2014) and attention mechanisms (Bahdanau et al., 2015; Luong et al., 2015) . These models can be trained with a variety of learning algorithms.

The standard training algorithm is based on maximum-likelihood estimation (MLE) which seeks to maximize the log-likelihood of ground-truth sequences. Despite the computational simplicity and efficiency, MLE training suffers from the exposure bias (Ranzato et al., 2016) . That is, the model is trained to predict the next token given the previous ground-truth tokens; while at test time, since the resulting model does not have access to the ground truth, tokens generated by the model itself are instead used to make the next prediction. This discrepancy between training and test leads to the issue that mistakes in prediction can quickly accumulate. Recent efforts have been made to alleviate the issue, many of which resort to the reinforcement learning (RL) techniques (Ranzato et al., 2016; Bahdanau et al., 2017; Ding & Soricut, 2017) . For example, Ranzato et al. (2016) adopt policy gradient (Sutton et al., 2000) that avoids the training/test discrepancy by using the same decoding strategy. However, RL-based approaches for sequence generation can face challenges of prohibitively poor sample efficiency and high variance. For more practical training, a diverse set of methods has been developed that are in a middle ground between the two paradigms of MLE and RL. For example, RAML (Norouzi et al., 2016) adds reward-aware perturbation to the MLE data examples; SPG (Ding & Soricut, 2017) leverages reward distribution for effective sampling of policy gradient. Other approaches such as data noising (Xie et al., 2017) also show improved results.

In this paper, we establish a unified perspective of the broad set of learning algorithms. Specifically, we present a generalized entropy regularized policy optimization framework, and show that the apparently diverse algorithms, such as MLE, RAML, SPG, and data noising, can all be re-formulated as special instances of the framework, with the only difference being the choice of reward and the values of a couple of hyperparameters (Figure 1 ). In particular, we show MLE is equivalent to using a delta-function reward that assigns 1 to samples that exactly match data examples while -∞ to any other samples. Such extremely restricted reward has literally disabled any exploration of the model beyond training data, yielding the exposure bias. Other algorithms essentially use rewards that are more smooth, and also leverage model distribution for exploration, which generally results in a larger effective exploration space, more difficult training, and better test-time performance.

Besides the new understandings of the existing algorithms, the unified perspective also facilitates to develop new algorithms for improved learning. We present an example new algorithm that, as training proceeds, gradually expands the exploration space by annealing the reward and hyperparameter values. The annealing in effect dynamically interpolates among the existing algorithms. Experiments on machine translation and text summarization show the interpolation algorithm achieves significant improvement over the various existing methods.

2. RELATED WORK:

Sequence generation models are usually trained to maximize the log-likelihood of data by feeding the ground-truth tokens during decoding. Reinforcement learning (RL) addresses the discrepancy between training and test by also using models' own predictions at training time. Various RL approaches have been applied for sequence generation, such as policy gradient (Ranzato et al., 2016) and actor-critic (Bahdanau et al., 2017) . Softmax policy gradient (SPG) (Ding & Soricut, 2017) additionally incorporates the reward distribution to generate high-quality sequence samples. The algorithm is derived by applying a log-softmax trick to adapt the standard policy gradient objective. Reward augmented maximum likelihood (RAML) (Norouzi et al., 2016) is an algorithm in between MLE and policy gradient. It is originally developed to go beyond the maximum likelihood criteria and incorporate task metric (such as BLEU for machine translation) to guide the model learning. Mathematically, RAML shows that MLE and maximum-entropy policy gradient are respectively minimizing KL divergences in opposite directions. We reformulate both SPG and RAML in a new perspective, and show they are precisely instances of a general entropy regularized policy optimization framework. The new framework provides a more principled formulation for both algorithms. Besides the algorithms discussed in the paper, there are other learning methods for sequence models. For example, Hal Daumé et al. (2009) ; Leblond et al. (2018) ; Wiseman & Rush (2016) use a learningto-search paradigm for sequence generation or structured prediction. Scheduled Sampling (Bengio et al., 2015) adapts MLE by randomly replacing ground-truth tokens with model predictions as the input for decoding the next-step token. Our empirical comparison shows improved performance of the proposed algorithm.

Policy optimization for reinforcement learning is studied extensively in robotics and game environment. For example, Peters et al. (2010) introduce a relative entropy regularization to reduce information loss during learning. Schulman et al. (2015) develop a trust-region approach for monotonic improvement. Dayan & Hinton (1997) ; Levine (2018) ; Abdolmaleki et al. (2018) study the policy optimization algorithms in a probabilistic inference perspective. The entropy-regularized policy optimization formulation presented here can be seen as a generalization of many of the previous policy optimization methods, as shown in the next section. Besides, we formulate the framework in the sequence generation context.

3. CONNECTING THE DOTS:

We first present a generalized formulation of an entropy regularized policy optimization framework, to which a broad set of learning algorithms for sequence generation are connected. In particular, we show the conventional maximum likelihood learning is a special case of the policy optimization formulation. This provides new understandings of the exposure bias problem as well as the exploration efficiency of the algorithms. We further show that the framework subsumes as special cases other well-known learning methods that were originally developed in diverse perspectives. We thus establish a unified, principled view of the broad class of works.

Let us first set up the notations for the sequence generation setting. Let x be the input and y = (y 1 , . . . , y T ) the sequence of T tokens in the target space. For example, in machine translation, x is the sentence in source language and y is in target language. Let (x, y * ) be a training example drawn from the empirical data distribution, where y * is the ground truth sequence. We aim to learn a sequence generation model p θ (y|x) = t p θ (y t |y 1:t-1 , x) parameterized with θ. The model can, for example, be a recurrent network. It is worth noting that though we present in the sequence generation context, the formulations can straightforwardly be extended to other settings such as robotics and game environment.

3.1. ENTROPY REGULARIZED POLICY OPTIMIZATION (ERPO):

Policy optimization is a family of reinforcement learning (RL) algorithms that seeks to learn the parameter θ of the model p θ (a.k.a policy). Given a reward function R(y|y * ) ∈ R (e.g., BLEU score in machine translation) that evaluates the quality of generation y against the true y * , the general goal of policy optimization is to maximize the expected reward. A rich research line of entropy regularized policy optimization (ERPO) stabilizes the learning by augmenting the objective with information theoretic regularizers. Here we present a generalized formulation of ERPO. Assuming a general distribution q(y|x) (more details below), the objective we adopt is written as

EQUATION

where KL(• •) is the Kullback-Leibler divergence forcing q to stay close to p θ ; H(•) is the Shannon entropy imposing maximum entropy assumption on q; and α and β are balancing weights of the respective terms. In the RL literature, the distribution q has taken various forms, leading to different policy optimization algorithms. For example, setting q to a non-parametric policy and β = 0 results in the prominent relative entropy policy search (Peters et al., 2010) algorithm. Assuming q as a parametric distribution and α = 0 leads to the commonly-used maximum entropy policy gradient (Ziebart, 2010; Haarnoja et al., 2017) . Letting q be a variational distribution and β = 0 corresponds to the probabilistic inference formulation of policy gradient (Abdolmaleki et al., 2018; Levine, 2018) . Related objectives have also been used in other popular RL algorithms (Schulman et al., 2015; 2017; Teh et al., 2017) .

We assume a non-parametric q. The above objective can be maximized with an EM-style procedure that iterates two coordinate ascent steps optimizing q and θ, respectively. At iteration n:

E-step: q n+1 (y|x) ∝ exp α log p θ n (y|x) + R(y|y * ) α + β , M-step: θ n+1 = arg max θ E q n+1 log p θ (y|x) .

(2)

The E-step is obtained with simple Lagrange multipliers. Note that q has a closed-form solution in the E-step. We can have an intuitive interpretation of its form. First, it is clear to see that if α → ∞, we have q n+1 = p n θ . This is also reflected in the objective Eq.( 1) where the weight α encourages q to be close to p θ . Second, the weight β serves as the temperature of the q softmax distribution. In particular, a large temperature β → ∞ makes q a uniform distribution, which is consistent to the outcome of an infinitely large maximum entropy regularization in Eq.(1). In terms of the M-step, the update rule can be interpreted as maximizing the log-likelihood of samples from the distribution q.

In the context of sequence generation, it is sometimes more convenient to express the equations at token level, as shown shortly. To this end, we decompose R(y|y * ) along the time steps:

EQUATION

where ∆R(y t |y * , y 1:t-1 ) measures the reward contributed by token y t . The solution of q in Eq.( 2) can then be re-written as:

q n+1 (y|x) ∝ t exp α log p θ n (y t |y 1:t-1 , x) + ∆R(y t |y 1:t-1 , y * ) α + β (4)

The above ERPO framework has three key hyperparameters, namely (R, α, β). In the following, we show that different values of the three hyperparameters correspond to different learning algorithms (Figure 1 ). We first connect MLE to the above general formulation, and compare and discuss the properties of MLE and regular ERPO from the new perspective.

3.2. MLE AS A SPECIAL CASE OF ERPO:

Maximum likelihood estimation is the most widely-used approach to learn a sequence generation model due to its simplicity and efficiency. It aims to find the optimal parameter value that maximizes the data log-likelihood:

EQUATION

As discussed in section 1, MLE suffers from the exposure bias problem as the model is only exposed to the training data, rather than its own predictions, by using the ground-truth subsequence y * 1:t-1 to evaluate the probability of y * t . We show that the MLE objective can be recovered from Eq.( 2) with specific reward and weight configurations. Consider a δ-reward defined as1 :

R δ (y|y * ) = 1 if y = y * -∞ otherwise. (6) Let (R = R δ , α → 0, β = 1).

From the E-step of Eq.( 2), we have q(y|x) = 1 if y = y * and 0 otherwise. The M-step is therefore equivalent to arg max θ log p θ (y * |x), which recovers precisely the MLE objective in Eq.( 5).

That is, MLE can be seen as an instance of the policy optimization algorithm with the δ-reward and the above weight values. Any sample y that fails to match precisely the data y * will receive a negative infinite reward and never contribute to model learning.

Exploration efficiency:

The ERPO reformulation of MLE provides a new statistical explanation of the exposure bias problem. Specifically, a very small α value makes the model distribution ignored during sampling from q, while the δ-reward permits only samples that match training examples. The two factors in effect make void any exploration beyond the small set of training data (Figure 2 (a)), leading to a brittle model that performs poorly at test time due to the extremely restricted exploration. On the other hand, however, a key advantage of the δ-reward specification is that its regular reward shape allows extreme pruning of the huge sample space, resulting in a space that includes exactly the training examples. This makes the MLE implementation very simple and the computation very efficient in practice.

On the contrary, common rewards (e.g., BLEU) used in policy optimization are more smooth than the δ-reward, and permit exploration in a broader space. However, such rewards usually do not have a regular shape as the δ-reward, and thus are not amenable to sample space pruning. Generally, a larger exploration space would lead to a harder training problem. Also, when it comes to the huge sample space, the rewards are still very sparse (e.g., most sequences have BLEU=0 against a reference sequence). Such reward sparsity can make exploration inefficient and even impractical.

Given the opposite algorithm behaviors in terms of exploration and computation efficiency, it is a natural idea to seek a middle ground between the two extremes to combine the advantages of both. A broad set of such approaches have been recently developed. We re-visit some of the popular ones, and show that these apparently divergent approaches can all be reformulated within our ERPO framework (Eqs.1-4) with varying reward and weight specifications. 

EQUATION

That is, unlike MLE that directly maximizes the data log-likelihood, RAML first perturbs the data proportionally to the reward distribution e, and maximizes the log-likelihood of the resulting samples.

The RAML objective reduces to the vanilla MLE objective if we replace the task reward R in e(y|y * ) with the MLE δ-reward (Eq.6). The relation between MLE and RAML still holds within our new formulation (Eqs.1-2). In particular, similar to how we recovered MLE from Eq.( 2), let (α → 0, β = 1)2 , but set R to the task metric reward, then the M-step of Eq.( 2) is precisely equivalent to maximizing the above RAML objective.

Formulating within the same framework allows us to have an immediate comparison between RAML and others. In particular, compared to MLE, the use of smooth task metric reward R instead of R δ permits a larger effective exploration space surrounding the training data (Figure 2 (b)), which helps to alleviate the exposure bias problem. On the other hand, α → 0 as in MLE still limits the exploration as it ignores the model distribution. Thus, RAML takes a step from MLE toward regular RL, and has effective exploration space size and exploration efficiency in between.

3.4. SOFTMAX POLICY GRADIENT (SPG):

SPG (Ding & Soricut, 2017) was developed in the perspective of adapting the vanilla policy gradient (Sutton et al., 2000) to use reward for sampling. SPG has the following objective:

EQUATION

where R is a common reward as above. As a variant of the standard policy gradient algorithm, SPG aims to address the exposure bias problem and shows promising results (Ding & Soricut, 2017) .

We show SPG can readily fit into our ERPO framework. Specifically, taking gradient of Eq.( 8) w.r.t θ, we immediately get the same update rule as in Eq.( 2) with (α = 1, β = 0, R = common reward).

Note that the only difference between the SPG and RAML configuration is that now α = 1. SPG thus moves a step further than RAML by leveraging both the reward and the model distribution for full exploration (Figure 2 (c)). Sufficient exploration at training time would in theory boost the test-time performance. However, with the increased learning difficulty, additional sophisticated optimization and approximation techniques have to be used (Ding & Soricut, 2017) to make the training practical.

3.5. DATA NOISING:

Adding noise to training data is a widely adopted technique for regularizing models. Previous work (Xie et al., 2017) has proposed several data noising strategies in the sequence generation context. For example, a unigram noising, with probability γ, replaces each token in data y * with a sample from the unigram frequency distribution. The resulting noisy data is then used in MLE training.

Though previous literature has commonly seen such techniques as a data pre-processing step that differs from the above learning algorithms, we show the ERPO framework can also subsume data noising as a special instance. Specifically, starting from the ERPO reformulation of MLE which takes (R = R δ , α → 0, β = 1) (section 3.2), data noising can be formulated as using a locally relaxed variant of R δ . For example, assume y has the same length with y * and let ∆ y,y * be the set of tokens in y that differ from the corresponding tokens in y * , then a simple data noising strategy that randomly replaces a single token y * t with another uniformly picked token is equivalent to using a reward R δ (y|y * ) that takes 1 when |∆ y,y * | = 1 and -∞ otherwise. Likewise, the above unigram noising (Xie et al., 2017) is equivalent to using a reward

EQUATION

where u(•) is the unigram frequency distribution.

With a relaxed (i.e., smoothed) reward, data noising expands the exploration space of vanilla MLE locally (Figure 2 (b)). The effect is essentially the same as the RAML algorithm (section 3.3), except that RAML expands the exploration space based on the task metric reward.

Other Algorithms Ranzato et al. ( 2016) made an early attempt to address the exposure bias problem by exploiting the classic policy gradient algorithm (Sutton et al., 2000) and mixing it with MLE training. We show in the supplementary materials that the algorithm is closely related to the ERPO framework, and can be recovered with moderate approximations. Section 2 discusses more relevant algorithms for sequence generation learning.

4. INTERPOLATION ALGORITHM:

We have presented the generalized ERPO framework, and connected a series of well-used learning algorithms by showing that they are all instances of the framework with certain specifications of the three hyperparameters (R, α, β). Each of the algorithms can be seen as a point in the hyperparameter space (Figure 1 ). Generally, a point with a more restricted reward function R and a very small α tends to have a smaller effective exploration space and allow efficient learning (e.g., MLE), while in contrast, a point with smooth R and a larger α would lead to a more difficult learning problem, but permit more sufficient exploration and better test-time performance (e.g., (softmax) policy gradient).

The unified perspective provides new understandings of the existing algorithms, and also facilitates to develop new algorithms for further improvement. Here we present an example algorithm that interpolates the existing ones.

The interpolation algorithm exploits the natural idea of starting learning from the most restricted yet easiest problem configuration, and gradually expands the exploration space to reduce the discrepancy from the test time. The easy-to-hard learning paradigm resembles the curriculum learning (Bengio et al., 2009) . As we have mapped the algorithms to points in the hyperparameter space, interpolation becomes very straightforward, which requires only annealing of the hyperparameter values.

Specifically, in the general update rules Eq.( 2), we would like to anneal from using R δ to using smooth common reward, and anneal from exploring by only R to exploring by both R and p θ . Let R comm denote a common reward (e.g., BLEU). The interpolated reward can be written in the form

R = λR comm + (1 -λ)R δ , for λ ∈ [0, 1]. Plugging R into q in Eq.

(2) and re-organizing the scalar weights, we obtain the numerator of q in the form: c • (λ

1 log p θ + λ 2 R comm + λ 3 R δ )

, where (λ 1 , λ 2 , λ 3 ) is defined as a distribution (i.e., λ 1 +λ 2 +λ 3 = 1), and, along with c ∈ R, are determined by (α, β, λ). For example, λ 1 = α/(α + 1). We gradually increase λ 1 and λ 2 and decrease λ 3 as the training proceeds.

Model BLEU MLE 26.44 ± 0.18 RAML (Norouzi et al., 2016) 27.22 ± 0.14 SPG (Ding & Soricut, 2017) 26.62 ± 0.05 MIXER (Ranzato et al., 2016) 26.53 ± 0.11 Scheduled Sampling (Bengio et al., 2015) 26.76 ± 0.17 Further, noting that R δ is a Delta function (Eq.6) which would make the above direct function interpolation problematic, we borrow the idea from the Bayesian spike-and-slab factor selection method (Ishwaran et al., 2005) . That is, we introduce a categorical random variable z ∈ {1, 2, 3} that follows the distribution (λ 1 , λ 2 , λ 3 ), and augment q as q(y|x, z)

∝ exp{c • (1(z = 1) log p θ + 1(z = 2)R comm + 1(z = 3)R δ )}.

The M-step is then to maximize the objective with z marginalized out:

max θ E p(z) E q(y|x,z) [log p θ (y|x)].

The spike-and-slab adaption essentially transforms the product of experts in q to a mixture, which resembles the bang-bang rewarded SPG method (Ding & Soricut, 2017) where the name bang-bang refers to a system that switches abruptly between extreme states (i.e., the z values). Finally, similar to (Ding & Soricut, 2017) , we adopt the token-level formulation (Eq.4) and associate each token with a separate variable z.

We provide the pseudo-code of the interpolation algorithm in the supplements. It is notable that Ranzato et al. ( 2016) also develop an annealing strategy that mixes MLE and policy gradient training. As discussed in section 3 and the supplements, the algorithm can be seen as a special instance of the ERPO framework (with moderate approximation) we have presented. Next section shows improved performance of the proposed, more general algorithm compared to (Ranzato et al., 2016) .

5. EXPERIMENTS:

We evaluate the above interpolation algorithm in the tasks of machine translation and text summarization. The proposed algorithm consistently improves over a variety of previous methods.

Code will be released upon acceptance.

Setup In both tasks, we follow previous work (Norouzi et al., 2016; Ranzato et al., 2016) and use an attentional sequence-to-sequence model (Luong et al., 2015) where both the encoder and decoder are single-layer LSTM recurrent networks. The dimensions of word embedding, RNN hidden state, and attention are all set to 256. We apply dropout of rate 0.2 on the recurrent hidden state. We use Adam optimization for training, with an initial learning rate of 0.001 and batch size of 64. At test time, we use beam search decoding with a beam width of 5. Please see the supplementary materials for more configuration details.

5.1. MACHINE TRANSLATION:

Dataset Our dataset is based on the common IWSLT 2014 (Cettolo et al., 2014) German-English machine translation data, as also used in previous evaluation (Norouzi et al., 2016; Ranzato et al., 2016) . After proper pre-processing as described in the supplementary materials, we obtain the final dataset with train/dev/test size of around 146K/7K/7K, respectively. The vocabulary sizes of German and English are around 32K and 23K, respectively.

Results:

The BLEU metric (Papineni et al., 2002) is used as the reward and for evaluation. Table 1 shows the test-set BLEU scores of various methods. Besides the approaches described above, we also compare with the Scheduled Sampling method (Bengio et al., 2015) which combats the exposure bias by feeding model predictions at randomly-picked decoding steps during training. From the table, we can see the various approaches such as RAML provide improved performance over the vanilla MLE, as more sufficient exploration is made at training time. Our proposed new algorithm performs best, as it interpolates among the existing algorithms to gradually increase the exploration space and solve the generation problem better.

Figure 3 shows the test-set BLEU scores against the training steps. We can see that, with annealing, our algorithm improves BLEU smoothly, and surpasses other algorithms to converge at a better point. 

5.2. TEXT SUMMARIZATION:

Dataset We use the popular English Gigaword corpus (Graff et al., 2003) for text summarization, and pre-processed the data following (Rush et al., 2015) . The resulting dataset consists of 200K/8K/2K source-target pairs in train/dev/test sets, respectively. More details are included in the supplements.

The ROUGE metrics (including -1, -2, and -L) (Lin, 2004) are the most commonly used metrics for text summarization. Following previous work (Ding & Soricut, 2017) , we use the summation of the three ROUGE metrics as the reward in the learning algorithms. Table 2 show the results on the test set. The proposed interpolation algorithm achieves the best performance on all the three metrics. For easier comparison, Figure 4 shows the improvement of each algorithm compared to MLE in terms of ROUGE-L. The RAML algorithm, which performed well in machine translation, falls behind other algorithms in text summarization. In contrast, our method consistently provides the best results.

6. CONCLUSIONS:

We have presented a unified perspective of a variety of well-used learning algorithms for sequence generation. The framework is based on a generalized entropy regularized policy optimization formulation, and we show these algorithms are mathematically equivalent to specifying certain hyperparameter configurations in the framework. The new principled treatment provides systematic understanding and comparison among the algorithms, and inspires further enhancement. The proposed interpolation algorithm shows consistent improvement in machine translation and text summarization. We would be excited to extend the framework to other settings such as robotics and game environments.

A POLICY GRADIENT & MIXER:

Ranzato et al. ( 2016) made an early attempt to address the exposure bias problem by exploiting the policy gradient algorithm (Sutton et al., 2000) . Policy gradient aims to maximizes the expected reward:

EQUATION

where R P G is usually a common reward function (e.g., BLEU). Taking gradient w.r.t θ gives:

EQUATION

We now reveal the relation between the ERPO framework we present and the policy gradient algorithm.

Starting from the M-step of Eq.( 2) and setting (α = 1, β = 0) as in SPG (section 3.4), we use p θ n as the proposal distribution and obtain the importance sampling estimate of the gradient (we omit the superscript n for notation simplicity):

EQUATION

where Z θ = y exp{log p θ + R} is the normalization constant of q, which can be considered as adjusting the step size of gradient descent.

We can see that Eq.( 12) recovers Eq.( 11) if we further set R = log R P G , and omit the scaling factor Z θ . In other words, policy gradient can be seen as a special instance of the general ERPO framework with (R = log R P G , α = 1, β = 0) and with Z θ omitted.

The MIXER algorithm (Ranzato et al., 2016) incorporates an annealing strategy that mixes between MLE and policy gradient training. Specifically, given a ground-truth example y * , the first m tokens y * 1:m are used for evaluating MLE loss, and starting from step m + 1, policy gradient objective is used. The m value decreases as training proceeds. With the relation between policy gradient and ERPO as established above, MIXER can be seen as a specific instance of the proposed interpolation algorithm (section 4) that follows a restricted annealing strategy for token-level hyperparameters (λ 1 , λ 2 , λ 3 ). That is, for t < m in Eq.4 (i.e.,the first m steps), (λ 1 , λ 2 , λ 3 ) is set to (0, 0, 1) and c = 1, namely the MLE training; while for t > m, (λ 1 , λ 2 , λ 3 ) is set to (0.5, 0.5, 0) and c = 2.

B INTERPOLATION ALGORITHM:

Algorithm 1 summarizes the interpolation algorithm described in section 4. 3 ) picked according to the validation set performance.

:

For token-level, define R δ (y1:t|y * ) = t/T * if y1:t = y * 1:t and -∞ otherwise, where T * is the length of y * . Note that the R δ value of y = y * can also be set to any constant larger than -∞.

The exponentiated reward distribution e can also include a temperature τ(Norouzi et al., 2016). In this case, we set β = τ .

Figures and Tables:

Figure1: A unified formulation of different learning algorithms. Each algorithm is a special instance of the general ERPO framework taking certain specifications of the hyperparameters (R, α, β) (Eq.1).

Figure 2: Effective exploration space of different algorithms. (a): The exploration space of MLE is exactly the set of training examples. (b): RAML and Data Noising use smooth rewards and allow larger exploration space surrounding the training examples. (c): Common policy optimization such as SPG basically allows the whole exploration space.

Ours 27.82 ± 0.11

Figure 3: Convergence curve of learning algorithms in the task of machine translation. RAML is the second-best performing method (Table.1), inferior only to our algorithm.

Figure 3: Convergence curve of learning algorithms in the task of machine translation. RAML is the second-best performing method (Table.1), inferior only to our algorithm.

Figure 4: Improvement on the ROUGE-L metric in comparison to MLE (e.g., RAML improves ROUGE-L by 0.17).

Figure 5: Convergence curve of learning algorithms in the task of machine translation with a dropout rate of 0.3. The horizontal dashed lines indicate the test-set results of each of the algorithms (reported in Table3) picked according to the validation set performance.

Results of machine translation.

Table content: <table/>

Results of text summarization.

Table content: <table/>

Peer review (iclr2019):
 The authors propose a more unified view of disparate methods for training sequence models. Specifically, a multi-term objective L(q,theta) consisting of: 1) The standard reward maximization objective of policy gradient, E_{p_.theta}[R], 2) A weighted (weight alpha) reverse KL divergence of the parametric policy and a non-parameteric policy q, 3) A weighted (weight beta) entropy term on q, Is proposed for sequence training (see equation (1). L can be iteratively optimized by solving for q given p_.theta, and the .theta given q (see eq. 2). This framework mathematically generalizes softmax-policy gradient (SPG, alpha=1, beta=0), and reward-augmented maximum likelihood (alpha=0, beta=temperature), and also standard entropy regularized policy gradient (alpha=0), among other algorithms. The paper is well written, and the approach sensible. However, combining SPG and RAML by introducing their respective regularization terms is a rather straightforward exercise, and so seems quite incremental. Other major concerns are: 1) the true utility of the model, and 2) the integrity of the experiments. Wrt: 1), While RAML was a significant contribution at the time, it is now well established that RAML generally doesn*t perform well at all in practice due to exposure bias (not conditioning on it*s own previous predictions during training). Moreover SPG, as the authors point out, was supposed to address the need for ML pre-training, but required much engineering to work. The fact is that REINFORCE-based policy gradient methods are still more effective than these methods, provided they have a good baseline to reduce varince. Which brings me to point 2) Was MIXER run with a learned baseline and judiciously optimized? Table 1 suggests that MIXER can outpeform ML by only 0.1 Bleu points, and outpeformed by RAML? Something is wrong with your implementation then. Moreover, there are techniques like self-critical sequence training (SCST), which far outpeform MIXER, and we haven*t even discussed Actor-Critic baselines... In summary, the contribution over RAML and SPG in combining them is quite incremental, and the practical importance of combining them is questionable, as is the integrity of the presented experiments, given how poorly MIXER is reported perform, and the omission of stronger baselines like SCST and AC methods. Also, a paper on essentially the same approach was submitted and rejected from ICLR 2018(https://openreview.net/pdf?id=H1Nyf7W0Z), although this paper is better written, and puts the method more fully in context with existing work, I think that several of the concerns with that paper apply here as well. Look forward to the authors* feedback, and additional/corrected results - will certainly update my score if these concerns are addressed. In particular, if this generalization can significantly outpeform existing methods it generalizes with non-degenerate settings, this would overcome the more incremental contribution of combining SPG and RAML. Current Ratings: Evaluation 2/5: Results are not consistent with previous results (e.g. MIXER results). Stronger baselines such as SCST and AC are not considered. Clarity 5/5: Clear paper, well written. Significance 3/5: RAML and SPG have not been established as important methods in practice, so combining them is less interesting. Originality 2/5: RAML and SPG are fairly straightforward to combine for experts interested in these methods. Rating 4/10 Okay but not good enough, reject. Confidence 5/5 Pros: - Generalizes RAML and SPG (and also standard entropy-regularized policy gradient). - Well written paper, clean generalization. Cons: - RAML and SPG have not been established as important methods in practice. - generalization of RAML and SPG is straightforward, incremental. - Existing baselines in the paper (i.e. MIXER) do not perform as expected (i.e barely better than ML, worse than RAML) - Stronger REINFORCE-based algorthms like SCST, as well as Actor-critic algorithms, have not been compared. Update after author responses: -------------------------------------------- Authors, thank you for your feedback. While it is true that generalizing RAML and SPG into a common framework is not trivial, the presented framework simply augments the dual form of SPG (i.e. REPS [16] in the SPG paper) with a RAML term. Furthermore, the MLE interpretation discussed is contained within the RAML paper, and the reductions to RAML and SPG are straightforward by design, and so do not really provide much new insight. Considering this, I feel that the importance of the paper largely rests on investigating and establishing the utility of the approach experimentally. Wrt the experiments, I appreciate that the authors took the time to investigate the poor performance of MIXER. However, the unusally poor performance of MIXER remains unexplained, and falls short even of scheduled sampling (SS), which suggests a lingering major issue. REINFORCE techniques rely on 1) strong baselines, verified by the authors, 2) larger batch sizes to reduce variance, and 3) pre-training to reduce variance and facilitate efficient exploration. If the MLE is undertrained or overtrained (the latter the more likely issue given the plots), then MIXER will perform poorly. Actually, it is now standard practice to pre-train with MLE+SS before RL training, and this is really the (also dynamically weighted objective) baseline that should be compared against. The current REINFORCE results (MIXER or otherwise) really need to be updated (or at the least removed, as they are not captured by the framework, but the comparison to PG methods is important!). More generally, I feel that the experiments are not yet comprehensive enough. While the authors have shown that they can outperform SPG and RAML with a scheduled objective, it is not currently clear how sensitive/robust the results are to the term weight scheduling, or even what most appropriate general weights/scheduling approach actually is. Overall I feel that the paper is still in need of substantial maturation before publication, although I have revised my score slightly upward.