Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a formalism of entropy regularized policy optimization. They also show that various policy gradients algorithms can be reformulated as special instances of the presented novel formalism. The only difference between them being the reward function and two weight hyperparameters. Further, the paper proposes an interpolation algorithm, which, as training proceeds, gradually expands the exploration of space by annealing the reward function and the weight hyperparameters. Experiments on text generation tasks and game imitation learning show superior performance over previous methods. 

Overall, the paper is well written and the derivations and intuitions sound good. I appreciate the overall effort of the paper and the thorough experiments to validate the proposed interpolation algorithm, results seem not significant for text summarization. Hence, I suggest a week accept for this paper. 

Arguments:
1) From Table 1 and Table 2, the proposed approach has the lowest variance on machine translation and the quiet opposite on the text summarization (i.e., it has high variance). Any thoughts on this? This also suggests to conduct experiments on ablating the variance in the training for various policy gradient approaches include the proposed one. 

2) Results seem not significant on the summarization tasks. Any thoughts on choosing this particular task? Why not try image captioning where most of these policy gradient approaches have been applied.  

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This submission belongs to the field of sequence modelling. In particular, this submission presents a unified view on a range of training algorithms including maximum likelihood (ML) and reinforcement learning (RL). The unified view presented I believe is interesting and could be of interest to a large community. Unfortunately this submission has two issues 1) presentation and 2) experimental validation. 

I find it peculiar that an objective function that features ML and variants of RL as special cases called ERPO is proposed by statement. I find it more likely that it came out by analysing ML, the variants of RL and other commonly used objective functions, noticing similarities between them and then formulating a function that would render all above as special cases. Had the order been different this submission would have been much more analytical and interesting to read. 

I find experimental results a bit limited and not entirely conclusive as it seem that MT provides the only strong experimental evidence. I find quite hard to interpret the significance of difference, for instance, between 36.72 and 36.59 in ROUGE-1. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper claims to propose a general entropy regularized policy optimization paradigm. MLE and RL are special cases of this training paradigm. Paper is well written, and the experimental results are convincing enough.  
However, there are still some minor problems in the paper. For the optimization framework ERPO (shown in Equation 1), it consists of three parts, a cross-entropy term (Shannon entropy), a $p,q$ KL divergence term, and a reinforcement learning reward loss item. From the framework point of view, it is not like the author claim that is supposed to present a general optimization framework, including various optimization algorithms. Instead, it is just a combined loss through weight control and the selection of corresponding functions. It may not really theoretically work to unify various types of optimization algorithms for general cases, let alone claiming that this is a general optimization algorithm framework. 

For the interpolation algorithm (I regard this is the true technical contribution of this paper), the authors used an annealing mechanism to use different weights and functions at different stages of training. The essence is that after MLE pre-training, different optimization algorithms are used in different stages, and this should be the focus of the article. The annealing settings used is only introduced in the appendix simply. Without more comparison experiments, we cannot clearly get the conditions for the annealing algorithm to be effective and ineffective. 

For the title of connecting the dots between MLE and RL, this paper did not do so, MLE and RL are only used collaboratively, and this has also been mentioned in previous work.

typo
Page 6 Paragraph “Other Algorithms & Discussions”: We We show in the appendix… -> We show in the appendix…

review_title: Official Blind Review #1723

Meta-review: The authors construct a weighted objective that subsumes many of the existing approaches for sequence prediction, such as MLE, RAML, and entropy regularized policy optimization. By dynamically tuning the weights in the objective, they show improved performance across several tasks.

Although there were no major issues with the paper, reviewers generally felt that the technical contribution is fairly incremental and the empirical improvements are limited. Given the large number of high-quality submissions this year, I am recommending rejection for this submission.
Decision: Reject