Connecting the Dots Between MLE and RL for Sequence Generation

Abstract: Sequence generation models such as recurrent networks can be trained with a diverse set of learning algorithms. For example, maximum likelihood learning is simple and efficient, yet suffers from the exposure bias problem. Reinforcement learning like policy gradient addresses the problem but can have prohibitively poor exploration efficiency. A variety of other algorithms such as RAML, SPG, and data noising, have also been developed in different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation, and show that the apparently divergent algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of reward function and a couple of hyperparameters. The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency. Besides, based on the framework, we present a new algorithm that dynamically interpolates among the existing algorithms for improved learning. Experiments on machine translation and text summarization demonstrate the superiority of the proposed algorithm.

Reviews and author responses:

I appreciate the new experimental results.

While I do understand that coming up with a readily comparable experimental setup is difficult, I still think more effort should be put into this. The evaluation protocol is not that variable from my own experience and the relevant choices either appear in the published papers or can be obtained through private communication.

I agree that AR3 that more comprehensive experiments would be very beneficial to the paper, so my assessment remains the same.

The authors propose a more unified view of disparate methods for training sequence models. Specifically, a multi-term objective L(q,theta) consisting of:
1) The standard reward maximization objective of policy gradient, E_{p_\theta}[R], 
2) A weighted (weight alpha) reverse KL divergence of the parametric policy and a non-parameteric policy q, 
3) A weighted (weight beta) entropy term on q, 
Is proposed for sequence training (see equation (1). L can be iteratively optimized by solving for q given p_\theta, and the \theta given q (see eq. 2).

This framework mathematically generalizes softmax-policy gradient (SPG, alpha=1, beta=0), and reward-augmented maximum likelihood (alpha=0, beta=temperature), and also standard entropy regularized policy gradient (alpha=0), among other algorithms.

The paper is well written, and the approach sensible. However, combining SPG and RAML by introducing their respective regularization terms is a rather straightforward exercise, and so seems quite incremental.

Other major concerns are:
1) the true utility of the model, and 
2) the integrity of the experiments. 

Wrt: 
1), While RAML was a significant contribution at the time, it is now well established that RAML generally doesn't perform well at all in practice due to exposure bias (not conditioning on it's own previous predictions during training). Moreover SPG, as the authors point out, was supposed to address the need for ML pre-training, but required much engineering to work. The fact is that REINFORCE-based policy gradient methods are still more effective than these methods, provided they have a good baseline to reduce varince. Which brings me to point 
2) Was MIXER run with a learned baseline and judiciously optimized? Table 1 suggests that MIXER can outpeform ML by only 0.1 Bleu points, and outpeformed by RAML? Something is wrong with your implementation then. Moreover, there are techniques like self-critical sequence training (SCST), which far outpeform MIXER, and we haven't even discussed Actor-Critic baselines...

In summary, the contribution over RAML and SPG in combining them is quite incremental, and the practical importance of combining them is questionable, as is the integrity of the presented experiments, given how poorly MIXER is reported perform, and the omission of stronger baselines like SCST and AC methods. Also, a paper on essentially the same approach was submitted and rejected from ICLR 2018(https://openreview.net/pdf?id=H1Nyf7W0Z), although this paper is better written, and puts the method more fully in context with existing work, I think that several of the concerns with that paper apply here as well. 

Look forward to the authors' feedback, and additional/corrected results - will certainly update my score if these concerns are addressed. In particular, if this generalization can significantly outpeform existing methods it generalizes with non-degenerate settings, this would overcome the more incremental contribution of combining SPG and RAML.

Current Ratings:

Evaluation      2/5: Results are not consistent with previous results (e.g. MIXER results). Stronger baselines such as SCST and AC are not considered.
Clarity         5/5: Clear paper, well written.
Significance    3/5: RAML and SPG have not been established as important methods in practice, so combining them is less interesting.
Originality     2/5: RAML and SPG are fairly straightforward to combine for experts interested in these methods.

Rating          4/10 Okay but not good enough, reject.    
Confidence      5/5

Pros: 
- Generalizes RAML and SPG (and also standard entropy-regularized policy gradient).
- Well written paper, clean generalization.
Cons:
- RAML and SPG have not been established as important methods in practice.
- generalization of RAML and SPG is straightforward,  incremental.
- Existing baselines in the paper (i.e. MIXER) do not perform as expected (i.e barely better than ML, worse than RAML)
- Stronger REINFORCE-based algorthms like SCST, as well as Actor-critic algorithms, have not been compared.

Update after author responses:
--------------------------------------------

Authors, thank you for your feedback.

While it is true that generalizing RAML and SPG into a common framework is not trivial, the presented framework simply augments the dual form of SPG (i.e. REPS [16] in the SPG paper) with a RAML term. Furthermore, the MLE interpretation discussed is contained within the RAML paper, and the reductions to RAML and SPG are straightforward by design, and so do not really provide much new insight. Considering this, I feel that the importance of the paper largely rests on investigating and establishing the utility of the approach experimentally.

Wrt the experiments, I appreciate that the authors took the time to investigate the poor performance of MIXER. However, the unusally poor performance of MIXER remains unexplained, and falls short even of scheduled sampling (SS), which suggests a lingering major issue. REINFORCE techniques rely on 1) strong baselines, verified by the authors, 2) larger batch sizes to reduce variance, and 3) pre-training to reduce variance and facilitate efficient exploration. If the MLE is undertrained or overtrained (the latter the more likely issue given the plots), then MIXER will perform poorly. Actually, it is now standard practice to pre-train with MLE+SS before RL training, and this is really the (also dynamically weighted objective) baseline that should be compared against. The current REINFORCE results (MIXER or otherwise) really need to be updated (or at the least removed, as they are not captured by the framework, but the comparison to PG methods is important!).

More generally, I feel that the experiments are not yet comprehensive enough. While the authors have shown that they can outperform SPG and RAML with a scheduled objective, it is not currently clear how sensitive/robust the results are to the term weight scheduling, or even what most appropriate general weights/scheduling approach actually is.

Overall I feel that the paper is still in need of substantial maturation before publication, although I have revised my score slightly upward.


Thanks for the encouraging feedback and helpful comments. Below we clarify for some of the concerns raised by the reviewer, and address all other issues.


** Overfitting in Figure.3. Results of more dropout

It’s true that the test-set performance of MLE and RAML decreases slightly as training proceeds. First, we’d like to note that the reported test-set performance of each algorithm (Table.1) is picked using the validation set, which is the standard practice. Second, as analyzed in the paper (e.g., Figure.2), MLE and RAML are easier to overfit due to the limited exploration space.

Thanks for the great suggestion of using more dropout. We added the results of dropout=0.3 in Appendix D (Table.3 and Figure.5). We can see that, compared to using dropout=0.2, the performance of each algorithm does increase to some extent. However, the improvement of our algorithm is comparable to our original results. In particular, with dropout 0.3, our algorithm improves over MLE and RAML by 1.5 and 0.49 BLEU points, respectively, comparable to the improvements with dropout 0.2 which are 1.42 and 0.64, respectively. The improvement is significant in the task.
 

** Performance of the competing methods

We note that the experimental settings and model configurations can vary a lot across the different work, resulting in different performance. Also, not all configuration details are listed in the papers, making it hard to reproduce and compare. For example, 
- In (Bahdanau et al., 2016), they used GRU cells and beam=10. 
- In (Leblond et al., 2018), they used GRU cells and a ConvNet encoder. 
In contrast, we used LSTM cells and a bi-directional RNN encoder. Also, the evaluation protocol can also vary (e.g., whether smoothing of the BLEU metric is used; whether the UNK token is disallowed in generation, etc). These settings are all common but not directly comparable. 

We are ready to release all experimental code.


** Originality of the example new algorithm

We agree that there is existing work attempting to combine MLE and RL by annealing or similar techniques (e.g., [Ranzato et al., 2015], which is discussed in Appendix A). The proposed example algorithm differs in that it is motivated naturally from the unified perspective, resulting in a different annealing strategy compared to any existing approaches. In particular, we anneal both the reward function and hyperparameters, and have a novel and highly-intuitive understanding of the algorithm as interpolating between existing different algorithms in the hyperparameter space.

The unified perspective can easily inspire more algorithms. For example, 1) Instead of annealing the weights, we can optimize them to find a single best point that balances well between exploration and training difficulty; 2) Having revealed MLE as using a delta-function reward and Data Noising as using locally-relaxed variants, we can leverage imitation learning algorithms to adapt the reward function, which is equivalent to adaptive data noising. We are happy to explore these exciting topics in the future.


** Bibliography and other details

Thanks for the suggestions. We’ve fixed all bibliography items. We’ve also fixed the typos and polished the language.


Thanks for the supportive comments and recognition of our theoretic contributions. 

The major contributions of this work are the unified formulation of a variety of sequence generation learning algorithms, and the new, principled understanding of their connections in terms of effective exploration space and the well-known “exposure bias” problem.

As a direct application of the unified formulation, the interpolation algorithm has improved over previous methods with a clear margin. For example, in Table.1, our algorithm outperforms MLE by as much as 1.42 BLEU points, which is a large margin in the task. Our algorithm also achieves significant improvement over the 2nd best-performing algorithm RAML by 0.64 BLEU point, which is *comparable or even larger* than other work in the domain. For example, the algorithm in [Dai et al., ACL 2018] improve over RAML by 0.56 (see Table.1 of [Dai et al., ACL 2018]), which is comparable to ours.

[Dai et al., ACL 2018] From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction



** RAML doesn’t perform well due to exposure bias. Utility of RAML and SPG

- First, as explained above, our major contribution is not to propose an algorithm as a simple combination of RAML and SPG, but instead a unified perspective and the resulting new insights of a variety of algorithms including not only RAML and SPG, but more importantly MLE (and Data Noising). The interpolation algorithm is a direct application of the new perspective.

- As has analyzed in the paper (section 3.3 and Figure 2), RAML has partially addressed the exposure bias by exploring the sequence space proportionally to the reward distribution. This is one of the new understandings enabled by our unified perspective. 

- RAML has been attracting enormous research interest. For example, [Wang et al., EMNLP 2018],  [Elbayad et al., ACL 2018], and [Dai et al., ACL 2018] are among the most recent work that studied and extended RAML from different perspectives.

[Wang et al., EMNLP 2018] SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation
[Elbayad et al., ACL 2018] Token-level and sequence-level loss smoothing for RNN language models
[Dai et al., ACL 2018] From Credit Assignment to Entropy Regularization: Two New Algorithms for Neural Sequence Prediction


Thanks for the reviews. Below we resolve misunderstandings and concerns raised by the reviewer.

** “combining SPG and RAML … seems quite incremental”

We’d like to clarify that the main contribution of this paper is *not* simply combining SPG and RAML. Instead, this paper aims to establish a unified mathematical formulation of various sequence generation learning algorithms spanning MLE, Data Noising, RAML, and SPG, revealing the formal connections of these algorithms in a principled way. In particular, we show:
    * MLE is mathematically equivalent to an entropy-regularized policy gradient with a delta-function reward. This provides a new perspective of its well-known “exposure bias” problem. 
    * Data Noising, RAML, and SPG can all be reformulated as special cases of the framework, taking different hyperparameter values and reward functions. They are gradually increasing the effective exploration space to address the exposure bias.

Besides, please note that, to reach the unified perspective, we’ve derived new formulations of all these algorithms (MLE, Data Noising, RAML, SPG) which differ significantly from (yet mathematically equivalent to) their original forms.

Thus, we feel it could be unfair to treat this work as simply “combining SPG and RAML by introducing their respective regularization terms” while ignoring all other major contributions in the paper.


** A paper on essentially the same approach: [Alpha-DM] https://openreview.net/pdf?id=H1Nyf7W0Z

Our framework is fundamentally different from the above Alpha-DM approach. Alpha-DM is based on the opposite KL divergences that MLE and RL are optimizing, which has been discussed earlier in RAML [Norouzi et al., 2016].

In contrast, we have reformulated MLE, Data Noising, RAML, SPG in a completely new way based on entropy-regularized policy optimization. The resulting unified framework enables new principled understanding of the effective exploration space and the “exposure bias” of each of the algorithms. These results cannot be derived from Alpha-DM (which does not study exposure bias at all) and other existing work attempting to combining MLE and RL. We will add the discussion.


** Performance of MIXER compared to MLE and RAML

We explain our implementation and provide more results to verify the correctness of our experiments.

- For MIXER, we tried different baselines including learned baseline and greedy decoding baseline, and got similar results. For RAML, we implemented the enhanced version by [Ma et al., 2017] which directly uses BLEU as the reward (rather than the Hamming distance as in the original paper [Norouzi et al., 2016]) and generally outperforms MIXER (see Table.5 of [Ma et al., 2017]).

- To further verify the correctness, we tested on a public implementation of MIXER and got similar results. Specifically, we run the comparison of MIXER and MLE using a public code of [Ranzato et al., 2015] that applies a learned baseline (https://github.com/violet-zct/pytorch_NMT). Using a configuration that close to ours, we got *BLEU=26.42 for MLE* (the same as in our paper which is 26.44) and *BLEU=26.63 for MIXER*. The improvement (0.22) is slightly higher than that of our own implementation, but is still inferior to RAML and our proposed algorithm which improves over MLE by 1.42. 
(Note that correctness of the public implementation was verified by successfully reproducing the original MIXER results with the same configuration in [Ranzato et al., 2015]. We didn’t use the official code of [Ranzato et al., 2015] as it’s not modularized well and is hard to adapt to our configuration.)

- We also implemented the SCST [Rennie et al., 2017] and got BLEU=26.70, improving over MLE by 0.26, which is better than MIXER and inferior to our implementation of RAML and the proposed algorithm. 

We also note that MIXER in the original paper [Ranzato et al., 2015] is in a setting of BLEU<21, far lower than ours (>26). This may explain the marginal improvement of MIXER over MLE in our experiments. We found little previous work has applied MIXER in settings other than the original one, except [Rennie et al., 2017] that studies on image captioning.

In sum, our empirical results are reasonable and correct. We will add the above new results and release all experimental code.

[Ma et al., 2017] Softmax Q-Distribution Estimation for Structured Prediction. https://arxiv.org/abs/1705.07136
[Rennie et al., 2017] Self-critical Sequence Training for Image Captioning. https://arxiv.org/abs/1612.00563

The authors provide a common mathematical perspective on several learning algorithms for sequence models. They also introduce a new algorithm that combines several of the existing ones and achieves significant (but small) improvements on a machine translation and a text summarization task.

The paper is clearly written, giving a good exposition of the unifying formulation.

I believe the paper is quite insightful, and contributes to the community's understanding of the learning algorithms. However, the improvements in their experiments are rather small, and could probably be improved with more experimental work. They do showcase the usefulness of their new formulation, but not very strongly. Thus my recommendation to accept the paper is mostly based on the theoretical content that opens an interesting new perspective.

This paper introduces an interesting unifying perspective on several sequence generation training algorithms, exposing both MLE, RAML and SPG as special cases of this unified framework. This enables insightful new interpretations of standard issues in MLE training in terms of exploration for instance.
Based on this new perspective, a new algorithm is introduced. Its performance is analysed on a machine translation and a text summarisation task.

==> Quality and clarity
The paper is overall well-written, although it can be improved upon (see details below). The bibliography for instance does not reference the conference/journals where the articles were published and lists many (>10) published papers as arXiv preprints.

The ideas are clearly presented, which is crucial in a paper trying to unify different approaches, and the new perspective on exploration is well motivated.

==> Originality and significance
The unifying framework is interesting, and helps shed new light on some standard issues in sequence generation.
On the other hand, the new algorithm and its analysis seem like a slightly rushed attempt at leveraging the unifying framework. 
The experiments, in particular, present several issues.
- For instance, it's clear from Figure 3 that both MLE and RAML are overfitting and would benefit from more dropout (in the literature, 0.3 is commonly used for this type of encoder-decoder architecture). Having access to these experimental results is important, since it would enable the reader to understand whether the benefits of the new approach are subsumed by regularisation or not.
- Further, the performance of the competing methods seems a bit low. MLE reports 26.44 BLEU, which is a bit surprising considering that: 
   - with beam-search (beam of size 10, not 5, admittedly), Bahdanau et al (2016) get 27.56 BLEU, and this is without dropout.   
   - with dropout 0.3 (but without beam search), Leblond et al (2018) get 27.4 BLEU.
Making a strong case for the benefits of the new algorithm requires more thorough experiments.

Overall, the first half of the paper is interesting and insightful, while the second would benefit from more time. 

Pros
- clarity of the ideas that are presented
- interesting unifying perspective on sequence generation algorithms
- insightful new interpretations of existing algorithms in terms of exploration

Cons
- the example new algorithm is not very original
- the associated experiments are incomplete

==> Details
1. page 2, "Dayan & Hinton (1997); Levine (2018); Abdolmaleki et al. (2018) study in a probabilistic inference perspective." is an incomplete sentence.
2. at the beginning of section 3.1, policy optimisation is a family of algorithmS
3. page 7 in the setup of the experiments, "We use the Adam optimizer for SGD training" is incorrect since SGD is not a family but a specific algorithm, which is different from Adam.

Meta-review: I enjoyed reading the paper myself and I appreciate the unifying framework connecting RAML and SPG. While I do not put a lot of weight on the experiments, I agree with the reviewers that the experimental results are not very strong, and I am not convinced that the theoretical contribution meets the bar at ICLR.

In the interpolation algorithm, there seems to be an additional annealing parameter and two tuning parameters. It is important to describe how the parameters are tuned. Given the additional hyper-parameters, one may consider giving all of the algorithms the same budget of hyper-parameter tuning. I also agree with reviewers that the policy gradient baseline seems to underperform typical results. One possible way to strengthen the experiments is to try to replicate the results of SPG or RAML and discuss the behavior of each algorithm as a function of hyper-parameters.


Decision: rejected-papers