Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies RL on language models, and makes 3 contributions:
1) RL4LMs, a modular library for optimizing language generators with RL.
2) GRUE, a benchmark of 6 language generation tasks with reward functions.
3) NLPO, a new algorithm which improves on PPO for RL on LMs. 
strength_and_weaknesses: Strengths:
+ The lack of open-source benchmarks and tasks for RL on language models is indeed a huge problem. This paper does the community a huge service by providing this library + benchmarks. 
+ The paper is clearly written and easy to follow.
+ The paper is quite thorough. I appreciate the many ablations, and the study of human agreement with automatic metrics.
+ The NLPO algorithm is simple, clearly explained, and does seem to provide an improvement over PPO. It remains to be seen if this will transfer to other, more complicated tasks + larger scale. (it does involve an extra copy of the policy weights, which has memory downsides, especially at large scale, so it's unclear if it will be feasible / worth the tradeoff). Though overall I think this is a small part of the contribution.
+ The paper has a useful section on important implementation details.

Weaknesses: 
- personal nit: I don't think CNN/DM is a very good summarization dataset (though it is very standard). 
- I think the biggest weakness of this benchmark / tasks is that it relies on simple automatic metrics (eg BLEU, sentiment score). This makes sense of course, since it makes it tractable to evaluate. But I suspect that, once people start optimizing these metrics directly, whatever metric correlation with human judgments exists will disappear. I think it's important to consider how to incorporate human data collection into this process. (This would obviously be a very big endeavor, so I don't think it's within scope for this paper, but I think is a useful future direction.)  One option is to hold regular competitions (eg at a conference) where you collect human data which you use to evaluate models, and then open-source that data for further training (similar things have been done in the context of dialog models). Basically, I think it will require quite a bit of effort to ensure this benchmark remains useful over time, and doesn't saturate like every other NLP benchmark.

Small notes:
- "We find that using RL to learn from scalar reward feedback is can be more"
> remove "is"?

- "high-quality estimates.."
> two periods


clarity,_quality,_novelty_and_reproducibility: 
All of these are strong, particularly reproducibility. Novelty is a bit lower, but that is less important for a paper like this.


Questions:
- I'm very curious why the PPO + NLPO models do so badly without starting from the supervised init. Any ideas? (Never mind, I see this is addressed in 5.2). 


summary_of_the_review: My personal belief is that fine-tuning language models to optimize specific metrics that are not perplexity (particularly human preferences) is going to be a huge field, perhaps one of the most impactful in ML. This paper clearly advances the accessibility of research on this problem. Thus, I think this paper may become an enormous contribution to the field. I'd be excited for the datasets and tasks to be updated regularly.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work aims to explore whether reinforcement learning (RL) can be a useful part of NLP generative models. Although RL methods have been used fairly often, there have been disagreements among experts about whether it provides meaningful gains, especially in light of training difficulties (eg enormous action spaces of 30k+). In this work, the authors develop an open source library (6 tasks, 15 metrics, 3 learning algorithms), create a new state-of-the-art RL learning algorithm, create the first RL-NLP leaderboard, and use MTurk to annotate 4 of those 6 datasets with human-validated labels. It finds that tasks involving heavy novelty (ie "zero-shot performance") do seem to benefit from training with RL.
strength_and_weaknesses: STRENGTHS
- This work is very comprehensive. It makes public a large library of many tasks, rewards/metrics, and learning algorithms.
- The learning algorithm (NLPO) outperforms the previous state of the art (PPO)
- In addition to the above work, further validating 6 of the datasets with MTurk is impressively thorough

WEAKNESSES
- The work could be more clear on pages 4 and 5 when describing on-policy actor-critic algorithms (3.3) and NLPO (4).

MINOR
- There appears to be a typo on page 8 "2 out of 4 tasks tasks". The word "tasks" is repeated.
clarity,_quality,_novelty_and_reproducibility: Clarity: For the most part, the writing in this paper is very clear.

Quality: Although RL is outside my area of expertise, the work appears to be high-quality. It implements 15 metrics/rewards and compares against 6 datasets, which is difficult to engineer.

Novelty: Based on a wide survey of RL-NLP lit, the authors "hypothesize that the size of the action space is a core cause of instability when training LMs with existing RL methods." Based on this insight, they develop a new learning algorithm (NLPO) which improves upon this shortcoming and outperforms the existing standard (PPO, as well as presumably REINFORCE, which it claims is strictly inferior to PPO based on previous work)

Reproducibility: By making their code (and datasets) into a public library, this work is very reproducible. Train/test splits of the datasets included in GRUE follow the splits of the original papers.
summary_of_the_review: This work is very comprehensive: the authors develop an open source library (6 tasks, 15 metrics, 3 learning algorithms), create a new state-of-the-art RL learning algorithm, create the first RL-NLP leaderboard, and use MTurk to annotate 4 of those 6 datasets with human-validated labels. It finds that tasks involving heavy novelty (ie "zero-shot performance") do seem to benefit from training with RL. By making their code into a public benchmark, they provide a very helpful starting point for the community to develop & compare future work against. Finally, they provide a good lit review of many (though not every) instance of RL in NLP, commenting on its shortcomings to date.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors introduce a new library for benchmarking RL methods on NLP tasks, along with a modified version of PPO for on-policy optimization. Results demonstrate that the proposed NLPO algorithm outperforms the PPO baseline according to human assessment.
strength_and_weaknesses: Strength:
• This paper builds a cross-task diverse evaluation benchmark containing 6 tasks.
• For the proposed NLPO, human evaluation and automatic evaluation show it outperforms the PPO baseline


Weakness:
• With the mask prediction policy update, the gain based on human evaluation seems to be marginal except on IMDB task metrics, and CNN naturalness metric
• The proposed NLPO in this paper seems to be similar to [1] ? 
• In experiments, there no comparison to traditional structure learning methods like [2]

[1] Donati, A.M., Quispe, G., Ollion, C., Corff, S.L., Strub, F., & Pietquin, O. (2021). Learning Natural Language Generation from Scratch. ArXiv, abs/2109.09371.

[2] Shu, R., Yoo, K., & Ha, J. (2021). Reward Optimization for Neural Machine Translation with Learned Metrics. ArXiv, abs/2104.07541.
clarity,_quality,_novelty_and_reproducibility: This paper has a clear writing and builds a library for benchmarking on a range of tasks. There is no concern on clarity.
summary_of_the_review: I would like to evaluate the contribution of this paper focused on the proposed NLPO method. I will lean towards an acceptation if the proposed method is indeed novel, results in state-of-the-art performance and the improvement is significant enough comparing to PPO.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a library (open-sourced) called RL4LMs, which is for optimizing generation models using RL. The library is compatible with HuggingFace. Second, the paper comes up with a GRUE (general reinforced-language understanding evaluation) benchmark, which consists of six language generation tasks (under the RL framework) containing IMDB text continuation (reward is related to sentiment), a CommonGEN generative common sense task, CNN/DailyMail summarization, ToTTo (data-to-text generation), a machine translation task using a small IWSLT dataset, and NarrativeQA (question answering). The paper also proposes the NLPO algorithm (natural language policy optimization); the difference between NLPO and PPO is in green font in Algorithm 1. Essentially, the policy which we collect trajectories from is an old copy of the current policy. Moreover, initialization is different from PPO.

strength_and_weaknesses: I think the toolkit will be a nice contribution to the research community. It will also encourage more people to dig deeper into the RL for NLP field. 

It’s great that the software contains multiple RL algorithms.

The paper is clearly written – it’s effective in getting points across to readers.

Human annotation is involved, which is great. 

Concerns below:

The selection of tasks isn’t totally satisfactory to me. Generating a positive continuation of movie reviews is too artificial and unchallenging. I think the benchmark (given there’s the word “general” in it) should consider real use cases of RL for NLP; for example: generating better dialogue responses where the reward is a human-preference-based metric or a toxicity-based metric. Moreover, IWSLT17 English-German is also quite artificial and IWSLT-trained systems will never be put into production or real use. Additionally, most of the tasks have a correct answer (or the space of correct generations is quite small), but there are many text generation tasks where the correct space of generations is very large, like in dialogue or story generation. I think these tasks should be taken into consideration as well.

One concern I have about the NLPO algorithm is that if the masked policy is an old copy, then the algorithm is not on-policy anymore. Then, the policy gradient derivation (e.g., see the Berkeley deep RL course policy gradient derivation, or any other policy gradient derivation) would not hold anymore. How do the authors justify the fact that we’re not able to do the policy gradient proof anymore (please correct me if I am wrong)? Or why does this periodic synchronization trick work? I would hope for a summary of either a theoretical analysis or empirical analysis. 

clarity,_quality,_novelty_and_reproducibility: Neutral comment: The Figure 2 plots initially looked weird to me (e.g., why would there be a line connecting two unrelated tasks), but later I realized it’s a clear way to demonstrate the aggregate performance (because we can look at the area). 

summary_of_the_review: Toolkit -- nice contribution. Selection of task for the general benchmark -- not satisfactory to me. The NLPO algorithm is clever, but it needs more justification. In general, I'm leaning toward accepting because of the software and the encouraging empirical results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies Reinforcement Learning (RL) for natural language generation. In particular, the paper develops an open-source library that implements several on-policy RL algorithms, a benchmark consisting of 6 language generation tasks, as well as new algorithm that improves over PPO for language generation tasks. Overall the paper makes a number of contributions that would facilitate the RL research for NLG/NLP. Reviewers had concerns on the selected tasks and metrics in the benchmark, e.g., CNN/DM is not an ideal corpus for studying summarization, and some of the tasks tend to be artificial. The authors have included a new task on dialog that partially addressed the concerns.

A side note: as a paper aiming to present a benchmark for RL-for-NLG research, it'd be nice to have a more comprehensive discussion (e.g., in related work) on the different existing RL algorithms applied to NLG in previous work. In particular, though claiming "RL" for NLG, the work seems having only discussed "on-policy" "policy-based" RL, without more discussion/mentioning on other categories of RL methods, such as "off-policy"/"value-based"/"offline", etc, that have been studied in NLG context. 

justification_for_why_not_higher_score: Concerns on the selection of tasks/metrics in the benchmark.
justification_for_why_not_lower_score: A number of contributions including a benchmark, an open-source library, and a new algorithm with improved performance.
Decision: Accept: notable-top-25%