Connecting the Dots Between MLE and RL for Sequence Prediction
Authors: Bowen Tan
Zhiting Hu
Zichao Yang
Ruslan Salakhutdinov
Eric Xing
Venue:  (2020)
Keywords: Sequence generation
sequence prediction
reinforcement learning

Abstract: Sequence prediction models can be learned from example sequences with a variety of training algorithms. Maximum likelihood learning is simple and efficient, yet can suffer from compounding error at test time. 
Reinforcement learning such as policy gradient addresses the issue but can have prohibitively poor exploration efficiency. A rich set of other algorithms, such as data noising, RAML, and softmax policy gradient, have also been developed from different perspectives. 
In this paper, we present a formalism of entropy regularized policy optimization, and show that the apparently distinct algorithms, including MLE, can be reformulated as special instances of the formulation. The difference between them is characterized by the reward function and two weight hyperparameters.
The unifying interpretation enables us to systematically compare the algorithms side-by-side, and gain new insights into the trade-offs of the algorithm design.
The new perspective also leads to an improved approach that dynamically interpolates among the family of algorithms, and learns the model in a scheduled way. Experiments on machine translation, text summarization, and game imitation learning demonstrate superiority of the proposed approach.

Under review as a conference paper at ICLR 2020
CONNECTING THE DOTs BETWEEN MLE AND RL FOR SEQUENCE PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Sequence prediction models can be learned from example sequences with a vari- ety of training algorithms. Maximum likelihood learning is simple and efficient, yet can suffer from compounding error test time. Reinforcement learning such policy gradient addresses the issue but can have prohibitively poor exploration efficiency. rich set other algorithms, such as data noising, RAML, and soft- max policy gradient, have also been developed from different perspectives. In this paper, we present a formalism of entropy regularized policy optimization, and show that the apparently distinct algorithms, including MLE, can be reformulated as special instances of the formulation. The difference between them is charac- terized by the reward function and two weight hyperparameters. The unifying interpretation enables us to systematically compare the algorithms side-by-side, and gain new insights into the trade-off's of the algorithm design. The new per- spective also leads to an improved approach that dynamically interpolates among the family of algorithms, and learns the model in scheduled way. Experiments on machine translation, text summarization, and game imitation learning demon- strate superiority of the proposed approach.
INTRODUCTION
Sequence prediction problem is ubiquitous in many applications, such as generating sequence of words for machine translation (Wu et al.] 2016)) Sutskever et al.]|2011 [2014), text summarization (Hovy
& Lin 1998 Rush et al. 2015), and image captioning (Vinyals et al. 2015 Karpathy & Fei-Fei 2015). or taking sequence of actions to complete task. In these problems (e.g..[Mnih Mnih al. 2015 Ho & Ermon 2016), we are often given set of sequence examples, from which we want to learn
model that sequentially makes the next prediction (c.g., generating the next token) given the current state (e.g., the previous tokens).
A standard training algorithm is based on supervised learning which seeks to maximize the log- likelihood of example sequences (i.e., maximum likelihood estimation, MLE). Despite the compu- tational simplicity and efficiency, MLE training can suffer from compounding error (Ranzato et al.] 2016 Ross & Bagnell| 2010) in that mistakes test time accumulate along the way and lead to states far from the training data. Another line of approaches overcome the training/test discrepancy issue by resorting to the reinforcement learning (RL) techniques (Ranzato et al. 2016 Bahdanau ct al. 2017: Rennie et al. 2017). For example, Ranzato et al. (2016) used policy gradient (Sutton al. 2000) to train text generation model with the task metric (e.g., BLEU) as reward. However, based approaches can face challenges of prohibitively poor sample efficiency and high variance. To this end, a diverse set of methods has been developed that is middle ground between the two paradigms of MLE and RI. For example, RAMI. (Norouzi et al.) 2016) adds reward-aware per- turbation to the MIL data examples; SPG (Ding & Soricut 2017 7) leverages reward distribution for effective sampling of policy gradient. Other approaches such data noising (Xie et al. 2017) also show improved results.
In this paper, we establish unifying perspective of the above distinct learning algorithms. Specif- ically, we present a generalized entropy regularized policy optimization framework, and show that the diverse algorithms, such MLE, RAML, data noising, and SPG, all be re-formulated as special cases of the framework. with the only difference being the choice of reward and the values of two weight hyperparameters (Figure []. In particular, we show MLE is equivalent using

---

Under review as a conference paper at ICLR 2020
Data Noising relaxed Rg.
0
SPG (R = task reward, > 0,
RAML
MLE
( = task reward, - 0,8 > 0)
(R Rs. B
restricted exploration
broader exploration
A unifying formulation of different learning algorithms. Each algorithm is special instance of the generalized ERPO formalism (Eq[I) by using different rewards and taking different values of the weight hyperparameters a, B.
Figure I:
Delta-function reward which returns to model samples that match training examples exactly, and -co to any other samples. Such extremely restricted reward has literally disabled any exploration of the model beyond training data, yielding brittle prediction behaviors. Other algorithms essentially use various locally-relaxed rewards, joint with the model distribution, for broader (and more costly) exploration during training.
Besides the new views of the existing algorithms, the unifying perspective also leads to new al- gorithms for improved learning. We develop interpolation algorithm, which, as training proceeds, gradually expands the exploration space by annealing both the reward function and the weight hy- perparameters. The annealing in effect dynamically interpolates among the existing algorithms from left to right in Figure [ We conduct experiments the tasks of text generation including machine translation and text summarization, and game imitation learning. The interpolation algorithm shows superior performance over various previous methods.
2 RELATED WORK
Given set of data examples, sequence prediction models usually trained to maximize the log- likelihood of the next label (token, action) conditioning on the current state observed in the data. Reinforcement learning (RL) addresses the discrepancy between training and test by also using models' own predictions at training time. Various RL approaches have been applied for sequence generation, such as policy gradient (Ranzato et al.) 2016) and cctorccritic (Bahdanau et al.) 2017). Reward augmented maximum likelihood (RAML) (Norouzi et al. 2016) is an algorithm in between MLE and policy gradient. Mathematically, RAMI shows that MLE and maximum-entroppy policy gradient are respectively minimizing KL divergences in opposite directions. Koyamada et al. (2018) thus propose use the more general a-divergence as combination of the two paradigms. Our framework is developed in a different perspective, reformulates a different and more comprehensive set of algorithms, and leads to new insights in terms of exploration and learning efficiency of the various algorithms. Besides the algorithms discussed in the paper, there are other learning methods for sequence models. For example, Hal Daumé et al. (2009); Leblond et al. (2018): Wiseman & paradigm for sequence generation or structured prediction. Scheduled Sampling (Bengio et al. 2013) and variants (Zhang et al. (019) adapt MLE by randomly replacing ground-trutie -ruth tokens with model predictions the input for decoding the next-step token.
Rush (0166) use a learning-to-seare
Policy optimization for reinforcement learning is studied extensively in robotic and game environ- ment. For example, |Peters et al. (2010) introduce relative entropy regularization reduce infor- mation loss during learning. [Schu Iman et al. (2013) develop a trust-region approach for monotonic improvement. Dayan & Hinton (1997); Levine (2018): Abdolmaleki et al. (2018) study the policy optimization algorithms in a probabilistic inference perspective. Zhu et al. (2018) combine imitation learning with RL, whose approach is orthogonal to ours and can be plugged into our framework to incorporate imitation reward. The mntropy-regularized policy optimization formulation presented here can be seen as generalization of many of the previous policy optimization methods, Besides, we formulate the framework primarily in the sequence generation context.
CONNECTING THE DOTS
We first present generalized formalism of entropy regularized policy optimization. The formula- tion contains reward function and two weight hyperparameters that define the learning procedure.
2

---

Under review as a conference paper at ICLR 2020
Therefore, varying the values of the reward and weights result in a large space of algorithms. We show that several existing popular algorithms, which were originally proposed in distinct perspec- tives, all be seen members in the space. In particular, we reformulate the MLE algorithm in the same policy optimization form, which enables side-by-side comparison between the broad spectrum of algorithms. The resulting unifying view provides new insights into the exploration and computation efficiency, and creates improved learning approaches for sequence prediction.
For clarity, we present the framework in the sequence generation context. The formulations can straightforwardly be extended to other settings such as imitation learning in robotic and game envi- ronments, as discussed briefly at the end of this section and also shown in the experiment.
We first establish the basic notations. Let y (V1..... ......... YT) be the sequence of T'' tokens. Let y be training example drawn from the empirical data distribution. From the sequence examples, we aim to learn sequence generation model pa that generation of y can condition on other factors. For example, machine translation, y is the sentence in target language and depends on an input sentence in source language. For simplicity of notations, we omit the conditioning factors.
II Po (3e (/:///tt=)))))) with parameters 0. Note
3.1 ENTROPY REGULARIZED POLICY OPTIMIZATION (ERPO)
Policy optimization is family of reinforcement learning (RL) algorithms. Assume a reward func- tion R(yly") R that evaluates the quality of generation y against the true y". For example, BLEU score (Papineni et al. 2002) can be a reward in machine translation. The general goal of policy opti- mization learn the model po(y) (a.k.a policy to maximize the expected reward. Previous work develops entropy regularized approaches, which augment the objective with information theoretic regularizers for stabilized training.
We present a generalized variational formulation of ERPO, which, as we show shortly, has the power of subsuming an array of other popular algorithms. Specifically, we introduce a non-parametric variational distribution ((y) w.r.t the model pa (yy). The objective to maximize is as follows:
Clg. θ) = E. [R(yl")) "] (g(v))l +HH(g).
(I)
where KL(-II-) is the Kullback- -ullback-Leibler divergence forcing q stay close to po; H(-) is the Shannon entropy imposing maximum entropy assumption on q; and a and B are balancing weights of the respective terms. Intuitively, the objective is maximize the expected reward under the variational distribution q while minimizing the distance between and the model pa. with maximum entropy regularization on q. The above formulation relevant and can be seen as a variant of previ- ous policy optimization approaches in RL literature, such as relative entropy policy search (Peters) al., [210]). maximum entropy policy gradient (Ziebart) 20100) Haarnoja et al.] (2077). and other work where the variational distribution is formulated either as a non -on-paraeetric distribution as ours (Abdolmaleki al. 2018 Peters et al. 2010) or parametric one (Schulman et al. 2015 2017a
Teh et 2017))
The objective can be maximized with standard EM procedure (Neal Hinton 1998) that iterates
two coordinate ascent steps optimizing q and 0, respectively. At iteration
log pan (y) + R(yle"))
E-step: gºh a+ (g) x exp
a + /
M-step: n+ arg max, E,,+ [log (o())
In the E-step, has a closed-form solution, which is an energy based distribution. We can have an intuitive interpretation of its form. First, clear to see that if a - oc, we have qa+1 Pp- This is also reflected in the objective [I. where a larger weight a encourages y to be close to pa. Second, the weight 3 serves as the temperature of the q softmax distribution. makes 9 uniform distribution, which is consistent with the outcome of an infinitely large maximum entropy regularization in Eq. In the M-step, the update rule can be interpreted maximizing the log-liket likeehoooood of samples from the distribution q.
particular,
large temperature 3
In the following, will use the "model" and "policy" exchangeably.
3

---

Under review as a conference paper at ICLR 2020
Token-level Formulation In the context of sequence generation, it sometimes more convenient to express the equations at token level (instead the sequence level), as shown when we devise a new algorithm in the next section. To this end, we decompose R(yly)) along the time steps:
R(vly)) E, R(y).yy") R(y -ly") Σ,
(3)
where AR y -1--11 measures the reward contributed by token yo. The solution of q in Eq.(2) can then be re-written as:
II. exp log (ye 11= 1) + A AR( (111119))
(4)
a+p 8
The Algorithm Space The above ERPO formalism includes three key components, namely, the reward I and the weight hyperparameters and 8 >> O. Variation in these components can result in different procedures of updating the model. other words, different algorithms in the ERPO family correspond to point (or region) in the space spanned by the three components. The following sections visit a set existing approaches, and connect them to the unifying picture by reformulating their seemingly distinct objectives. space, clustered by the exploration behavior in learning, of which we will discuss more.
illustrates the particular algorithms in the
Softmax Policy Gradient (SPG) We first briefly discuss the previous RL algorithms for sequence prediction that fit in the ERPO formalism. SPG (Ding & Soricut (017)) was originally developed in the perspective of combining the reward R and policy pe to improve sampling quality. The algorithm is equivalent to setting 3 = 0 and treating >> O as the temperature of the energy- based distribution q(y). That is, ((y) in the E-step of 42) is now in the form (v) x po exp R (yyly"" //a)- The reward R is set to any normal task-specific reward. Note that sampling from (3)) (e.g., in the M-step) is typically difficult due to its energy-based form and the fact that the task reward R often does not have particular structures amenable for sampling. We will see in the next section that the MLE algorithm in contrast uses a special reward to avoid the computational difficulty in sampling, at the cost of restricted exploration during training.
We also note the previous work of Sequence Tutor (Jaques et al. 2017). which was motivated by the idea of using an MLE-trained policy prior to guide the learning of the target policy in an RL framework. The formalism closely resembles SPG, namely (a > 0, B = 0), with the exception that the variational distribution ( Sequence Tutor is a parameterized model instead of a non- parametric one as in SPG and our more general ERPO formulation.
3.2 MLE AS A SPECIAL CASE OF ERPO
In this section, we connect the maximum likelihood estimation (MLE) algorithm to the unifying ERPO formalism. Based on the connections, we are able to analyze the learning behavior of MLE from the reinforcement learning perspective in terms exploration efficiency. We also discuss some well-known variants of the vanilla MLE algorithm, such as RAML and data augmentation.
Due to its simplicity and efficiency, MLE among the most widely-used approaches in learning sequence generation. It finds the optimal parameter value that maximizes the data liglliielhoood:
0 arg max, (0) arg max, log po (1
(5)
We show that the MILE objective can be recovered from Eq. with a specialized reward and weight values. More concretely, consider 8-function reward defined as
Ra(w'v') = oo
y ===y" otherwise.
(6)
That is, sample y receives a valid unit reward only when it matches exactly with the true data, and receives negative infinite reward in all other cases.
We show that the MLE algorithm is a member of the ERPO family. In particular, the conventional MLE objective is equivalent to setting the ERPO components to (R = Rs, - 0, 1). This can
For token-level, define Rs ( (yy) t t/T" if yıat via and -no otherwise, where T is the length of Note that the Rs value of y y" can also be to any constant larger than -00.
4

---

Under review as a conference paper at ICLR 2020
(a)
(b)
The effective exploration space of MLE is exactly the set of training examples. (b): Data Noising and RAML use diffused rewards and allow larger exploration space surrounding the training examples. (c): Standard policy optimization such as SPG (section 3.1 basically allows the whole exploration space.
Figure 2: Exploration space exposed to model learning in different algorithms. (a):
be straightforwardly seen by noting that, with the configuration, the ( in E-step (Eq reduces to q(y) 1 if y = y" and otherwise. The M-step thus in effect maximizing the log- lileeiodd of the real data examples (Note that the very small is still > O, making the M-step for maximizing the objective Eq.(T valid and necessary). With the 8-reward Ra, any sample y that fails to match the given data y" exactly will get negative infinite reward and thus never contribute to model learning.
Exploration Efficiency Reformulating MLE in the unifying ERPO form enables us to directly compare the approach with other RL algorithms. Specifically, the 5-reward has permitted only samples that match training examples, and made invalid any exploration beyond the small set of training data (Figure a)). The extremely restricted exploration at training time results in a brittle model that can easily encounter unseen states and make mistakes in prediction.
On the other hand, however, major advantage of the 6-reward is that it defines distribution over the sequence space such that sampling from the distribution is reduced to simply picking an instance from the training set. The resulting samples are ensured to have high quality. This makes the MLE implementation very simple and the computation efficient in practice.
On the contrary, task-specific rewards (such as BLEU) used in standard policy optimization are more diffused than the 8-reward, and thus allow exploration in a broader space with valid reward signals. However, the diffused rewards often do not lead distribution that is amenable for sampling as above. The model distribution is thus instead used to propose samples, which in turn can yield low-quality (i.c., low-reward) samples especially due to the huge sequence space. This makes the exploration inefficient or even impractical.
Given the opposite behaviors the algorithms terms of exploration and computation efficiency, it is natural idea to seek middle ground between the two extremes in order to combine the advan- tages of both. Previous work has proposed variants of the vanilla MLE from different perspectives. We re-visit some of the popular approaches, and show that they also be canonicalized in the ERPO framework and enrich our understanding of the learning behaviors.
Data Noising Adding noise to training data is a widely adopted model regularizing technique. Previous work (e.g.. Xie et al. 2017) has proposed several data noising strategies in the sequence generation context, such as replacing subsets of tokens with other random words. The resulting noisy data is then used in MLE training. Though previous literature has commonly seen such techniques as data pre-proced peeeprocssing step, we show that the approach can be expressed in the generalized ERPO formulation. Specifically, data noising can be seen as using locally relaxed variant of the 6-reward:
y y- y==(""") )
ranaise
otherwise,
where y denotes any transformation operation that returns a new sample as noisy version of the input raw data y". With the relaxed reward, data noising locally expands the exploration surrounding the observed training examples (Figure ()) The added exploration at training time can yield a model that is more robust to error at test time.
Reward-Augmenttdd Maximum Likelihood (RAML) RAML (Norouzi et al., 2016) was origi- nally proposed to incorporate task-specific metrics into the MIE training. Formally, it introduces
5

---

Under review as a conference paper at ICLR 2020
where R. is task reward and T >> 0 is the temperature. The conventional RAML objective is written as:
an exponentiated reward distribution e(y)
c(yly") y*)
Rxy (xp(R(yly"//T) ///)),
CRAML (0) = Ey~e(viu" log po(
(8)
That is, unlike MLE that directly maximizes the data log-likelihood. RAML first perturbs the data proportionally to the reward distribution, and maximizes the log- oiglliied of the resulting samples. Similar to how map MLE the ERPO formalism, by setting a + 0, 3 to the temperature т, and f to the task reward. Compared to the vanilla MLE, the key feature of RAML is the use of task reward instead of the d-reward, which permits larger ex- ploration space surrounding the training examples. On the other hand, same as in SPG (section 3.1) sampling from the energy- based distribution with diffused reward tends to be difficult, and often requires specialized approximations for computational efficiency (e.g., Ma et al. 2017).
align RAML with the unifying form
Other Algorithms & Discussions The classic policy gradient algorithm (Sutton et al. 2000) has also been used for sequence prediction -an..Ranzztto et al. 2016). We We show in the appendix that the approach can also be connected to the unifying ERPO with moderate approximations. Ranzato al. (2016) also proposed mixing training strategy that anneals from MLE training to policy optimization. We show in the next section that the particular annealing scheme is a special case of the new, more general interpolation algorithm below. We have presented the framework in the context of sequence generation. The formulation can also be extended to other settings. For example, in game environments, y is a sequence of actions and states. The popular imitation learning method GAIL (Ho & Ermon 2016) uses an adversarially induced reward It from data, and applies standard RL updates to train the policy. The policy update part can be formulated with our framework as standard policy optimization (with >> 0, 3 = 0). The new interpolation algorithm described in the next section can also be applied to improve the vanilla GAIL, as shown in the experiments.
Previous work has also studied connections of relevant algorithms. For example, Norouzi et al. (2066) Koyamada et al. (2018) formulate MLE and policy gradient as minimizing the opposite KL. divergences between the model and data/reward distributions. |Misra al.] (2018) studied update equation generalizing maximum marginal likelihood and policy gradient. Our framework differs in that we reformulate different and more comprehensive set of algorithms for sequence prediction, and provide new insights in terms of exploration and its efficiency, which could not be derived from the previous work. Section discusses more related work on sequence prediction learning.
INTERPOLATION ALGORITHM
The unifying perspective also leads to new algorithms for improved learning. Here, we present an example algorithm that is naturally inspired by the framework.
As in Figure [] each of the learning algorithms can be seen as point in the (R, or, 3) space. Gen- erally, from left right, the reward gets more diffused and a gets larger, which results in larger sequence space exposed to model training (Figure More exploration in turn also makes the train- ing less efficient due lower sample quality. We propose interpolation algorithm with the natural idea of starting learning from the most restricted yet efficient algorithm configuration, and gradu- ally expanding the exploration to decrease the training/test discrepancy. The easy-to-hard learning paradigm resembles the curriculum learning (Bengio et al.) 2009). As we have mapped the algo- rithms to the points in the hyperparameter space, the interpolation becomes straightforward, which reduces to simple annealing of the hyperparameter values.
Specifically, during training, we would like to anneal from using the restricted 8-reward Rş using task reward, and anneal from sampling (cxploring) by only the reward R to sampling by both R and po. Since Rs a 8-function which would make direct function linear combination problematic, we implement the interpolation strategy in the update rule (Eq(2) and use log-sum-exp for mixing. Formally, let Rlask denote a task reward. The negative energy of g(y) in Eq.(2) (i.e., the exponent inside exp(-)) ) is now replaced with the interpolated term: log(>i A2 exp Rankk + exp Rs}) Note that we have re-organized the weight hy byperparameters and used the distribution (A1, A2. As) to carry out the calibration role of (), In particular, as training proceeds, we gradually increase A and A2 and decrease Ag. The formulation of interpolation in effect converts the energy based model q(y) mixture of experts, which makes the sampling from g(y) easier, and resembles the

---

Under review as a conference paper at ICLR 2020
Model
BLEU
MLE
31.99 0.17
Method
ROUGE-1
ROUGE-2
ROUGE-L
RAML
32.51 = 0.37
MLE
36.11 1 0.21
16.39 1 0.16
32.32 1: 0.19
MIXER
32.69 0.09
RAML
36.30 0.04
16.69 0.20
32.49 1.49
MIXER-alike Anneal
02.65+0.11
Self-critic
36.48 = 0.24
16.84 0.26
32.79 0.26
Self-critic
32.23 0.15
SS
36.59 + 0.12
16.79 + 0.22
32.77 = 0.17
SS
32.13 0.14
Ours
36.72 +: 0.29
16.99 + 0.17
32.95 + 0.33
Ours
33.35 + 0.08
Table 2: Text summarization results (5-run average + std dev).
Table Machine translation re- sults (5-run average + std dev).
See the text for more details.
bang-bang rewarded SPG method as described in (Ding & Soricut, 2017). Besides, similar to (Ding Soricut, 2017), we adopt the token-level formulation (. so that tokens in a sequence can be sampled from different components (i.e., po. Runk. and Rs) in mixed way.
We provide the pseudo-code of the interpolation algorithm in the appendix. As discussed above, we can also apply the interpolation algorithm in game imitation learning, by plugging it into the GAIL (Ho Ermon 2016) framework to replace the standard RL routine for policy update. The annealing schedule in this setting is constrained due to the agent interaction with the environment. Specifically, to generate a trajectory (a sequence of actions and states), we sample the beginning part from data (demonstrations), followed by sampling from either the model or reward. Note that data sampling can happen only before model/reward sampling, because the latter will interact with the environment and result in states that do not necessarily match the data. Similar sequence gener- ation, we gradually anneal from data sampling to model/reward sampling, and hence increase the exploration until converging to standard RL. Our experiments validate that the easy-to-hard training is superior to the vanilla GAIL which directly applies the hard RL update from the beginning.
notable that (Ranzato et al., 2016) also developed an annealing strategy that mixes MLE and policy gradient training. The strategy is essentially the same as the one we apply in the GAIL learning setting. That is, the annealing approach of (Ranzato et al., 2016) is specialized case of the above more general annealing, using restricted values of (A1, X2, As) and discrete changes. We provide more discussions in the appendix. The experiment results in section show that our generalized annealing performs better than the restricted approach (Ranzato et al. 2016).
EXPERIMENTS
We evaluate the interpolation algorithm in the context of both text generation and game imitation learning. Experiments are run with 4 GTX 2080Ti GPUs and 32GB RAM. The link to the code is provided in the submission. We will release the code upon acceptance.
5.1 MACHINE TRANSLATION
We the state-of-the-art neural architecture Transformer (Vaswani al. (017) the base model. The model has blocks, trained with an Adam optimizer with an initial learning rate of 0.001 and the same schedule as in (Vaswani et al.) 2017). Batch size is 1.792 tokens. At test time, we use beam search decoding with a beam width of 5 and length penalty 0.6. We use the popular IWSLT2014 (Cettolo al. 2014) German-English dataset. After proper pre-processing as described in the appendix. we obtain the final dataset with train/dev/test size of around 146K/7K/7K, respec- tively. The shared de-en vocabulary is of size 73,197 without BPE encoding.
Table I shows the test-set BLEU scores of various methods. Besides MLE, RAML. and MIXER (Razaaoo et al., 2016) as discussed above, we also compare with other existing approaches such as Scheduled Sampling (SS) (Bengio et al., (015) and Self-critic (Rennie et al. 2017]. (We did not compare with SPG (Ding Soricut 2017) as no public code is available.) From the table, we can see the various approaches provide improved performance over the vanilla MLE, as more sufficient exploration is made at training time. Our interpolation algorithm performs best, with significant improvement the MLE training by 1.36 BLEU points. The results validate our approach that interpolates among the existing algorithms offers beneficial scheduled training. To further study the effect of our generalized annealing versus the MIXER strategy, we compare with "MIXER-alike
7

---

Under review as a conference paper at ICLR 2020
Ant-v2
HalrCheetah--2
Reacher-v2
5000
4000
3000
Expert
2000
Expert
-20
Export
2000
BC
Random
BC
Random
-30
Random
BC
GAIL
01
Ours
Ours
Durs
Figure 3: Performance of learned policies. The x-axis is the number of expert demonstrations for training. The y-axis is the average returns. "BC" is Behavior Cloning. "Random" is a baseline taking a random action each time. Results averaged over 50 runs.
Aneal" which uses the same configuration with our interpolation algorithm, except that the annealing restricted like MIXER. That is, the first portion of tokens sequence are all sampled from the data, while the subsequent tokens are sampled from only the model or the task reward. We see that the proposed more generalized annealing is superior to the restricted version. We note that there other work exploring various network architectures for machine translation (Shankar & Sarawagi 2019 He et al. 20018), which is orthogonal and complementary to the learning algorithms. would be interesting to explore the effect of combining the approaches.
5.2 TEXT SUMMARIZATION
We use an attentional sequence-to-sequeeie model (Luong et al., 2015) where both the encoder and decoder are single-layer LSTM RNN. The dimensions of word embedding, RNN hidden state, and attention are all set to 256. We use Adam optimization with an initial learning rate of 0.001 and a batch size of 64. Test time uses beam search decoding with a beam width of 5. Please see the appendix for more configuration details. We use the popular English Gigaword corpus (Graff) al., 2003) for text summarization, and pre-processed the data following (Rush et al., 2015). The resulting dataset consists of 200K/8K/2K source- pairs in train/dev/test sets, respectively.
Following previous work (Ding& & Soricut 2017) we use the summation of the three ROUGE(-1, -2, -L) metrics as the reward in learning. Table 2 show the results on the test set. The proposed interpolation algorithm achieves the best performance on all three metrics. The RAML algorithm, which performed well in machine translation, falls behind other algorithms in text summarization. In contrast, our method consistently provides the best results.
5.3 GAME IMITATION LEARNING
We apply the interpolation algorithm in GAIL (Ho & Ermon 2016) described in section 4 Fol- lowing (Ho & Ermon 2016). we simulate three environments with MuloCo (Todorov et al.) 2012). Expert demonstrations are generated by running PPO (Schulman et al. 201 under the given true reward functions. We then run different imitation learning algorithms with varying numbers of demonstrations. Both the policy and the discriminator are two-layer networks with 128 units each and tanh activations in between.
Figure shows the average returns by the agents. We can see that agents trained with the interpo- lation algorithm can generally improve over the vanilla GAIL,, especially in the presence of small 4) of demonstrations. This shows that our approach that anneals from the MLE mode to RL mode can make better use of data examples, and steadily achieve better performance in the end. We present the learning curves of the algorithms in the appendix.
number (e.g..
CONCLUSIONS
We have presented a unifying perspective a variety of learning algorithms for sequence prediction problems. The framework is based generalized entropy regularized policy optimization formu- lation, and we show the distinct algorithms are equivalent to specifying the reward and weight hyper- parameters. The new consistent treatment provides systematic understanding and comparison across the algorithms, and inspires further improved learning. The proposed interpolation algorithm shows consistent improvement in machine translation, text summarization, and game imitation learning.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Ried- miller. Maximum posteriori policy optimisation. In ICLR, 2018
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In ICLR, 2017.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In NewriPS, pp. 1 1171-1179, 2015.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston, Curriculum learning, In ICML, pp- 41-48. ACM, 2009.
Mauro Cettolo, Jan Niehues. Sebastian Stüker, Luisa Bentivogli, Marcello Federico. Report on 11th IWSLT evaluation campaign. IWSLT 2014. In Proceedings of the International Workshop on Spoken Lan- guage Translation, Hanoi, Vietnam, 2014.
Peter Dayan and Geoffrey E Hinton. Using expectation-maxxiizzzionn for reinforcement learning. Neural Computation, 9(2):711-278, 1997.
Nan Ding and Radu Soricut. Cold-start reinforcement learning with softmax policy gradient. In Advances in Neural Information Processing Systems, pp. 2814-2823, 2017.
David Graff, Junbo Kong. Ke Chen, and Kazuaki Maeda. English Gigaword. Linguistic Data Consortium, Philadelphia, 4(1):34, 2003.
Tuomas Haarnoja. Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy- based policics. In ICML, pp. 1352-1361, 2017.
III Hal Daumé, John I Langford, and Daniel Marcu. Search-based structured prediction as classification. Journal Machine Learning, 2009.
Tianyu He, Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, Tie-Yan Liu. Layer-wise coordination between encoder and decoder for neural machine translation. In NewrlPS, pp. 7944-7954, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NeurlPS, pp. 4573, 2016.
Eduard Hovy and Chin-Yee Lin. Automated text summarization and the summarist system. In Proceedings of workshop held Baltimore, Maryland: October 13-15, 1998, pp. 197-214. Association for Computa- tional Linguistics, 1998.
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández- Lobato, Richard E Turner, and Douglas Eck. Sequence tutor: Conservative fine-tuning of sequence generation models with KL-control. In ICML, 2017.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, pp. 3128 3137, 2015.
Sotetsu Koyamada, Yuta Kikuchi, Atsunori Kanemura, Shin-ichi Maeda, and Shin Ishii, Alpha-divergenee bridges maximum likelihood and reinforcement learning in neural sequence generation. 2018.
Rémi Leblond, Jean-Baptiste Alayrac, Anton Osokin, and Simon Lacoste-Julien. SEARNN: Training RNNs with global-lccal losses. In ICLR, 2018.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review, arXiv preprint 1rXiv:1805.0000 00909, 2018.
Minh-Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches attention-based neural machine translation. EMNLP, 2015.
Xuezhe Ma. Pengcheng Yin, Jingzhou Liu, Graham Neubig, and Eduard Hovy. Softmax q-distribution esti- mation for structured prediction: A theoretical interpretation for raml. arXiv preprint 175.0:170..07136, 2017.
Dipendra Misra, Ming-Wer Chang, Xiaodong He, and Wen-tau Yih. Policy shaping and generalized update equations for semantic parsing from denotations. In EMNLP. 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare. Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.

---

Under review as a conference paper at ICLR 2020
Radford M Neal and Geoffrey E Hinton. view of the algorithm that justifies incremental, sparse, and other variants. Learning in graphical models, pp. 355-368, Springer, 1998.
Mohammad Norouzi, Samy Bengio, Navdeep Jaitly, Mike Schuster. Yonghui Wu, Dale Schuurmans, al. Reward augmented maximum likelihood for neural structured prediction. Advances In Neural Information Processing Systems, pp- 1723-1731 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting association for computational linguistics, pp. 31 1-318. Association for Computational Linguistics, 2002.
Jan Peters, Katharina Mülling, and Yasemin Altun. Relative entropy policy search. In AAAI, pp. 1607-1612. Atlanta, 2010.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojeiech Zaremba. Sequence level training with recurrent neural networks. In ICLR, 2016.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jenet Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR, pp. 7008-7024, 2017.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In AISTATS, 661-666 2010.
Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, pp- 379-389, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy opti- mization. In ICML, pp. 1889-1897, 2015.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning arXiv preprint arXiv:1704 06440, 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017b.
Shiv Shankar and Sunita Sarawagi. Posterior attention models for sequence to sequence learning. In ICLR, 2019.
Ilya Sutskever, Oriol Vinyals, and Quoe V Lc. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112 12, 2014.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pp. 1057-1063, 2000.
Yee Teh. Victor Bapst, Wojeiech M Czarnekki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforeement learning. In NeurlPS, pp. 1496-4506 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- model-based based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEF, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In NeurlPS, pp. 5998-69999999 0088-608,,00 2017.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In CVPR, pp. 3156-3164, 2015.
Sam Wiseman and Alexander M Rush. Sequence-to-sequencee learning beam-search optimization. In EMNLP, pp. 1296-1306, 2016.
Yonghui Wu, Mike Schuster. Zhifeng Chen. Quoe Le. Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's 'ss neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 2016.
Yuch-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learn- ing from imperfect demonstration. In ICML, 2019.
Ziang Xie, Sida I Wang, Jiwei I.,, Daniel I Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. Data noising as smoothing in neural network language models. In ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2020
Wen Zhang, Yang Feng. Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. ACL, 2019.
Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. Selective encodding for abstractive sentence summarization. In ACL. 2017.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, Reinforcement and imitation learning for diverse visuomotor skills. In RSS, 2018.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. In PhD Thesis, 2010.
A APPENDIX
A.1 POLICY GRADIENT & MIXER
Ranzato el (016) made early attempl address the exposure bias problem by exploiting the policy gradient algorithm (Sution et al. 2000). Policy gradient aims to maximizes the expected reward:
(P) Ep. [Rra(yly")) )
where ReG is usually common reward function (c.g.. BLEU). Taking gradient w.r.t @ gives:
(f) = Ep [Rra(yly") log (3
(10)
now reveal relation between the ERPO framework we present and the policy gradient algorithm. Starting from the M-step of Eq. (2) and setting (a = 1 = 0) as in SPG (section ??), we use pon as the proposal
distribution and obtain the importance sampling estimate of the gradient (we omit the superseript for notation simplicity):
Eg Vo log po (y)] Epo
q(v)
Vo log po()))
[po
(11)
=1/2g Boy exp R(yl)")) - Vo log
where Zo f, exp pe + R} the normalization constant of q, which can be considered adjusting the step size of gradient descent.
We see that [T) recovers TO if we further set R log RrG, and omit the scaling factor Zg. In other words, policy gradient can seen as a special instance of the general ERPO framework with (R log RpO, & = B = 0) and with Ze omitted.
The MIXER algorithm (Ranzato et al., 2016 incorporates an anncaling strategy that mixes between MLE and policy gradient training. Specifically, given ground-truth example y the first tokens Vi.m are used for evaluating MI_E loss, and starting from step m m-++7 policy gradient objective used. The value decreases training proceeds. With the relation between policy gradient and ERPO as established above, MIXER can be seen as a specific instance of the proposed interpolation algorithm (section that follows a restricted annealing strategy for token-level hyperparameters (A1, As). That is, for t < m in Eq 4 (he first m steps), namely the MLE training: while for
(A1, A2, set to (0, 0, 1) and (0.5, 0.5, 0) c
(X1, A2, Aa) is
A.2 INTERPOLATION ALGORITHM
Algorithm I summarizes the interpolation algorithm described in section
A.3
EXPERIMENTAL SETTINGS
A.3.1
DATA PRE-PROCESSING
For the machine translation dataset, we follow (a al. 2017) for data pre- pre-processing.
In text summarization, we sampled 200K out of the 3.8M pre-processed training examples provided by Rush 2015) for the sake of training efficiency. used the refined validation and test sets provided by Zhou
et al.
et a..
2017
In the game imitation learning task, we randomly sample 50 state-action pairs in each trajectory as demon- strations. Every training iteration, collect least 2,048 state-action pairs, and we train 1 1,000 iterations for every model in every environment.
11

---

Under review as a conference paper at ICLR 2020
Algorithm Interpolation Algorithm
1: Initialize model parameter & and weights À = (A1,
2: repeat
3: Get training example y
4: fo 0, 1, do
5: Sample [1, (21, A3)
6:
if' = 1 then
7:
Sample token yt explc 1og else if 2 2 then
8:
9:
Sample token yr ~ explc AR(yoMMu--..0))
10: else
Sample token yr ~ exple ARs), i.e., set yr = vi end if
12:
13:
end for
14:
Update l by maximizing the log-likelihood log po (y)
15:
Anneal À by increasing λ and X2 and decreasing X
16:
until convergence
A.3.2 ALGORITIM SETUP
For RAML (Norouzi al. 2016). use the sampling approach (n-gram replacement) by (Ma et al. 2017) to sample from the exponentiated reward distribution. For each training example we draw 6 and 10 samples in machine translation and text summarization tasks, respectively.
Scheduled Sampling (SS) (Bengio et 2013). we tested various annealing schedules report best-performing one, namely inverse-sigmond decay. The probability of sampling from model = k/ + exp (i/k)), where is hyperparameter controlling the speed of convergence, which is set to 4000 and 600 in the machine translation and text summarization tasks, respectively. We would like note that SS does not fit into our formulation, because, in SS, model-generated tokens are only used model inputs instead of the targets of which the likelihood is maximized. the model is used as an input to the next step, the loss associated with step & is still log (y(y prev tokens) where yi is the true token. This differs from our formulation which maximizes the likelihood of ÿt-
example, time step I, even though token őr generated by
For the proposed interpolation algorithm, after MLE pre- pre-training. initialize the weights (A1, A2, As) (0.12, 0.16, 0.72). Every epochs, we increase A₁ by 0.12 and X2 by 0.16 while decreasing A by 0.28.
did MLE pretraining for all comparison methods the same number steps. found pretraining necessary for Self-critic, and is helpful for RAML and SS.
A.3.3 LEARNING CURVES OF GAIL EXPERIMENTS
Figure presents the learning curves of different algorithms in the GAIL experiments.
HallC7eetahww,, Idemots)
HeifCheetah-v22, 4demots)
Rescher-2, Idemoosi
3030
4003
520
2003
-20
-25
-300
Oura
220
420
600
502
2002
200
402
500
500
2000
200
400
520
500
1000
Figure 4: Learning curves of GAIL experiments. The æ-axis is the number training iterations. The y-axis is the returns. GAIL performance drops in the late stage, as also observed in previous work ge, Figure of Wu et al. 2019)
12


Reviews:
[{"review_id": "S1xxM8j8Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper claims to propose a general entropy regularized policy optimization paradigm. MLE and RL are special cases of this training paradigm. Paper is well written, and the experimental results are convincing enough.  \nHowever, there are still some minor problems in the paper. For the optimization framework ERPO (shown in Equation 1), it consists of three parts, a cross-entropy term (Shannon entropy), a $p,q$ KL divergence term, and a reinforcement learning reward loss item. From the framework point of view, it is not like the author claim that is supposed to present a general optimization framework, including various optimization algorithms. Instead, it is just a combined loss through weight control and the selection of corresponding functions. It may not really theoretically work to unify various types of optimization algorithms for general cases, let alone claiming that this is a general optimization algorithm framework. \n\nFor the interpolation algorithm (I regard this is the true technical contribution of this paper), the authors used an annealing mechanism to use different weights and functions at different stages of training. The essence is that after MLE pre-training, different optimization algorithms are used in different stages, and this should be the focus of the article. The annealing settings used is only introduced in the appendix simply. Without more comparison experiments, we cannot clearly get the conditions for the annealing algorithm to be effective and ineffective. \n\nFor the title of connecting the dots between MLE and RL, this paper did not do so, MLE and RL are only used collaboratively, and this has also been mentioned in previous work.\n\ntypo\nPage 6 Paragraph “Other Algorithms & Discussions”: We We show in the appendix… -> We show in the appendix…\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1xROF9CFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a formalism of entropy regularized policy optimization. They also show that various policy gradients algorithms can be reformulated as special instances of the presented novel formalism. The only difference between them being the reward function and two weight hyperparameters. Further, the paper proposes an interpolation algorithm, which, as training proceeds, gradually expands the exploration of space by annealing the reward function and the weight hyperparameters. Experiments on text generation tasks and game imitation learning show superior performance over previous methods. \n\nOverall, the paper is well written and the derivations and intuitions sound good. I appreciate the overall effort of the paper and the thorough experiments to validate the proposed interpolation algorithm, results seem not significant for text summarization. Hence, I suggest a week accept for this paper. \n\nArguments:\n1) From Table 1 and Table 2, the proposed approach has the lowest variance on machine translation and the quiet opposite on the text summarization (i.e., it has high variance). Any thoughts on this? This also suggests to conduct experiments on ablating the variance in the training for various policy gradient approaches include the proposed one. \n\n2) Results seem not significant on the summarization tasks. Any thoughts on choosing this particular task? Why not try image captioning where most of these policy gradient approaches have been applied.  \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylY7KLAKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission belongs to the field of sequence modelling. In particular, this submission presents a unified view on a range of training algorithms including maximum likelihood (ML) and reinforcement learning (RL). The unified view presented I believe is interesting and could be of interest to a large community. Unfortunately this submission has two issues 1) presentation and 2) experimental validation. \n\nI find it peculiar that an objective function that features ML and variants of RL as special cases called ERPO is proposed by statement. I find it more likely that it came out by analysing ML, the variants of RL and other commonly used objective functions, noticing similarities between them and then formulating a function that would render all above as special cases. Had the order been different this submission would have been much more analytical and interesting to read. \n\nI find experimental results a bit limited and not entirely conclusive as it seem that MT provides the only strong experimental evidence. I find quite hard to interpret the significance of difference, for instance, between 36.72 and 36.59 in ROUGE-1. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors construct a weighted objective that subsumes many of the existing approaches for sequence prediction, such as MLE, RAML, and entropy regularized policy optimization. By dynamically tuning the weights in the objective, they show improved performance across several tasks.

Although there were no major issues with the paper, reviewers generally felt that the technical contribution is fairly incremental and the empirical improvements are limited. Given the large number of high-quality submissions this year, I am recommending rejection for this submission.
Decision: Reject