Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
Authors: Rajkumar Ramamurthy
Prithviraj Ammanabrolu
Kianté Brantley
Jack Hessel
Rafet Sifa
Christian Bauckhage
Hannaneh Hajishirzi
Yejin Choi
Venue: ICLR 2023 notable top 25% (2023)
Keywords: natural language processing
reinforcement learning
language models
feedback learning

Abstract: We tackle the problem of aligning pre-trained large language models (LMs) with human preferences. If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework. However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, as well as a lack of open-source libraries and benchmarks customized for LM alignment. Thus, a question rises in the research community: is RL a practical paradigm for NLP?

To help answer this, we first introduce an open-source modular library, $RL4LMs$ (Reinforcement Learning for Language Models), for optimizing language generators with RL. The library consists of on-policy RL algorithms that can be used to train any encoder or encoder-decoder LM in the HuggingFace library (Wolf et al. 2020) with an arbitrary reward function. Next, we present the $GRUE$ (General Reinforced-language Understanding Evaluation) benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions which capture automated measures of human preference.GRUE is the first leaderboard-style evaluation of RL algorithms for NLP tasks. Finally, we introduce an easy-to-use, performant RL algorithm, $NLPO$ (Natural Language Policy Optimization)} that learns to effectively reduce the combinatorial action space in language generation. We show 1) that RL techniques are generally better than supervised methods at aligning LMs to human preferences; and 2) that NLPO exhibits greater stability and performance than previous policy gradient methods (e.g., PPO (Schulman et al. 2017)), based on both automatic and human evaluations.

Published as a conference paper at ICLR 2023
Is REINFORCEMENT LEARNING (NoT) FOR NATURAL LANGUAGE PROCESSING: BENCHMARKS, BASELINES, AND BUILDING BLOCKS FOR NATURAL LANGUAGE POLICY OPTIMIZATION
Rajkumar Ramamurthy
Prithviraj Ammanabrolu" ' Christian Bauckhage
Kianté Brantley
Jack Hessel
Rafet Sifa A
Hannanch Hajishirzi
Yejin Choi
Fraunhofer IAIS
2 Allen Institute for Artificial Intelligence
+ Cornell University
Paul G. Allen School of Computer Science, University of Washington ra jkumar. www.aartaayeeeeee iais. fraunhofer.de
(raja, jackh) Callenai org; kab828cornell.edd
ABSTRACT
We tackle the problem of aligning pre-trained large language models (LMs) with human preferences. If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework. However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, well as a lack of open- source libraries and benchmarks customized for LMM alignment. Thus, a question rises in the research community: is RI. a practical paradigm for NLP?
To help answer this, first introduce an open-source modular library, RLALM for optimizing language generators with RL. The library consists of on-policy RL algorithms that can be used to train any encoder or encoder-decoder LM in the HuggingFace library (Wolf et al. 2020) with an arbitrary reward function. Next, we present the GRUE (General benchmark, set of language generation tasks which are supervised not by target strings, but by reward functions which capture automated measures human preference. GRUE is the first leaderboard-style evaluation of RL algorithms for NLP tasks. Finally, we introduce an easy-to-use, performant RL algorithm, NLPO (Natural Language Policy Optimization) that learns to effectively reduce the combinatorial action space in language generation. We show 1) that RL techniques are generally better than supervised methods aligning LMs human preferences; and 2) that NLPO exhibits greater stability and performance than previous policy gradient methods (c.g.. PPO (Schulman et al. (017)), based on both automatic and human evaluations.
Understanding Evaluation)
INTRODUCTION
The ultimate aim of language technology is to interact with humans. However, most language models are trained without direct signals human preference, with supervised target strings serving as (a sometimes crude) proxy. One option incorporate user feedback is via human-in-the-loop. i.e., user would be expected to provide feedback for each sample online as the model trains, but this degree of dense supervision often prohibitive and inefficient. Automated metrics offer a promising compromise: models of human preference like pairwise learned preference models (Ouyang et al. 2022), BERTScore (Zhng et al. 2211)), BLEURT (Sellam et al. (2020) have significantly improved correlation with human judgment compared to earlier metrics (BLEU, METEOR, etc.), and are cheap to evaluate. But these functions are usually not per-token differentiable: like humans, metrics
"Denotes Equal Contribution
'Code: https::////t...... http::///qtttbbtb com wwtp:///...oommaaaaaaaaaaaaa
Projecct Website: https: ms apps. allena orgg/

---

Published as a conference paper at ICLR 2023
Aprihee
Preference
Scater Ourput
NUFO 13
FDHLL
Environment
Naturalness
At fitistayan anppary warmed slowly wotthed on...
atterpt fel startfill short fall shert..
Awtll owhy sefal, I'm going tell you why
0.11
0.25
0.57
Feedback Learning
Exploration
f liwed book. movic..
Figure 1: Natural Language Policy Optimization (NLPO) in the case of sentiment-guided contin- uation. Here, the LM (i.e., the policy) needs to produce a positive sentiment continuation given a review prompt (we cover other models of human preference in Sec. 3.2). Two objectives are balanced: 1) an automated proxy of human preference that serves as a reward (here: a sentiment classifier); and 2) "naturalness" as measured by KI. divergence from LM not trained with explicit human feedback. The plots show validation learning curves comparing our NLPO to the popular policy gradient method PPO. (Top plot:) RL methods can easily achieve high reward if the KL penalty is removed, (Bottom:) but at the cost of higher perplexity. NLPO+KL, our proposed approach, succeeds in balancing reward and naturalness more effectively than prior work.
can only offer quality estimates for full generations. Reinforcement Learning (RL) offers natural path forward for optimizing non-di -on-differreniable, scalar objectives for LM-based generation when is cast as sequential decision-male -aainn problem. However, Goodhart's Law looms: particularly in the case of imperfect metrics that use neural networks, it is easy to find nonsense samples that achieve high-quality estimates. Recent works have shown promising results in aligning LMs to human preferences via RL by constraining preference-base
rewards to incorporate notions of fluency (Wul al. 202 la Ouyang et al.) 2022) but progress in this line of work heavily hindered by lack of open-source benchmarks and algorithmic petplementations-reslllte
in perception that RL is challenging paradigm for NLP (Choshen et al., 2020 Kreutzer et al., 2021).
To facilitate research in building RL algorithms to better align LMs, we release a library, benchmark, and an algorithm. First, we release the RLALMs library, which enables generative HuggingFace models (e.g., GPT-2 or T5) to be trained using a variety of existing RI. methods like PPO/A2C/ete Next, we apply models trained using RLALMS to the new GRUE (General Reinforced-languaee Understanding Evaluation) benchmark: GRUE is collection of 7 contemporary NLP tasks (see TableT for details); in contrast to other benchmarks, instead of supervised training, we pair each task with reward function(s). GRUE challenges models to optimize these reward functions while remaining fluent language generators. We train language models via botb with and without task specific supervised pre-training- optimize rewards. Finally, beyond existing RL methods, introduce a novel on-policy RL algorithm called NLPO (Natural Language Policy Optimization). that dynamically learns task-specific constraints the distribution of language token level.
Experiments on GRUE and human evaluations show that NLPO better balances learning preference rewards while maintaining language fluency compared to alternatives, including PPO (Figure D. We find that using RL to learn from scalar reward feedback can be more: (1) data efficient than using additional expert demonstrations via supervised learning (though a combination of both best) learned reward function enables greater performance when used as signal for an RL method than a supervised method trained with 5 times more data, and (2) parameter efficient- a 220 million parameter model trained with combination of supervision and NLPO to outperform a 3 billion supervised model. We hope that the benchmarks, baselines, and building blocks we release serve to drive forward research in aligning LMs to human preferences.
Stratherm 1997
paraphrases:
When a measure becomes a target, ceases be good measure.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Imitation learning for NLP. Algorithms such as Schedule Sampling (SS) (Bengio et al.
2013).
Parallel SS (Duckworth et al., 2019), SS for Transformers (Mihaylova & Martins 2019), Diffential SS
(Goyal et al. 2017), LOLS (Lampouras & Vlachos 2016 Chang et al. 2015). TexIGAIL (Wu et al.
2021bb. and SEARNN (Leblond et 2017). have been inspired by DAGGER (Ross et al. 201 l) and SEARN
DDuméé et 20.9). However, these algorithms are known to suffer from exposure bias
in generation
(Chiang & Chen 2021; Arora et al., 2022) and the cliff MDP problem (Huszár 2015
Agarwal et al.
2019 Swamy et al. 2021).
RL for Large Action Spaces. MIXER (Ranzato et al. 2016) combined ideas from schedule sampling
and REINFORCE (Williams, 1992). Bahdanau al. (2016) proposed an actor-critic algorithm to
address the variance/large action space problems when using REINFORCE for language generation;
follow-up works such as KG-A2C (Ammanabrolu & Hausknecht 2020), TrufLL (Martin et al. 2022).
AE-DQN (Zahavy et al., 2018), and GALAD (Ammanabrolu et al. 2022) addressed similar issues by
attempting to eliminate and reduce the action space during exploration.
RL for NLP. RL, often in the form of bandit learning. has been used to improve models in machine
translation ( (Wu al. 2016. Nguyen et al., 2017 Kiegeland & Kreutzer, 2022)) summarization
(Stiennon et al. (020 Paulus et (017) dialogue (Li et al. (2hu et al. 2017: Jaques et al.,
2020 image captioning (Rennie al. | 2017). question generation (Pang He 2021), ) text-games
(Narasimhan et al. 2015 Hausknecht et al. 2020). and more (Ranzato et al. 2016 Snell et al. 2022).
Lu et al. (2022) adapt reward-condition
transformers (Chen al., 2021) for several language
generation tasks. RL has been the focus of efforts to align LMs with human preferences (Stiennon
al., 2020: Wu al., 2222aa] Nakano et al. [20T [Ziellr et al., 2019), e.g.. Ouyang et al. (2022)
fine-tuned large language model with PPO Schulman et al. (2017) to align with models of human preference, but their non-public dataset doesn't 't enable comparison. Though RI. has been successful in some of the use cases described above, it has simultancously been critiqued for being significantly
less stable than supervised LM training (Choshen et al., (020). As a result, there is relatively little consensus if RL is a worthwhile consideration for training LMs compared to, say, collecting additional supervised data.
3 RL4LMS: A LIBRARY FOR TRAINING LMS WITH RL
We introduce RL4LMs, an open-source library with building blocks for fine-tuning and evaluating RI algorithms on LM--baeed generation. The library is built HuggingFace (Wolf et al.) 2020 and stable-baselines-3 (Raffin et al.) (021). combining important components from their interfaces. RL4LMs can be used train any decoder only or encoder-decoder transformer models from Hug- gingFace with any on-policy RL algorithm from 3able-bbsellnes-3. Furthermore, we provide reliable implementations of popular on-policy RL, algorithms that are tailored for LM fine-tuning such as PPO (Schulman et al. (2017), TRPO (Schulman et al. 2015a), A2C (Mnih et al.) (016)), and our own NLPO ($4). The library modular, which enables users to plug-in in customized environments, reward functions, metrics, and algorithms. In the initial release, we provide support for 6 different NLP tasks, 16 evaluation metrics and rewards, and 4 RL algorithms.
3.1 ENVIRONMENTS: GENERATION AS A TOKEN-LEVEI MDP
Each environment is an NLP task: are given supervised dataset D = ((x1 of N examples, where x E X' is an language input and y E y is the target string. Generation can be viewed as a Markov Decision Process (MDP) (S, R, P. 7, T`) using a finite vocabulary V. Each episode in the MDP begins by sampling a datapoint (z, y) from our dataset and ends when the current time step exceeds the horizon T or an end of sentence (EOS) token is generated. The input .mm) is a task-specifi prompt that is used as our initial state so (Zo, ), where 8o E S and S is the state space with I.m E V. An action in the environment a, E A consists of a token from our vocabulary V. The transition function P: S x A + A(S) deterministically appends an action ae to the end of the state $;-1 = (Ia. ..... Im, ag ... a(-1). This continues until the end obtain state ST (Zo. Em, ag.
= (Zo,
..r). At the end of episode a reward R: : X A x y + R that depends on the (ST, 3) (c.g.. an automated metric like PARENT Dhingra et al. 2019)) is emitted. RI AL provides OpenAl gym (Brockman et al.) 2016) style
of the horizon < T and
3

---

Published as a conference paper at ICLR 2023
API for RL environment that simulates this LM- Based MDP formulation. This abstraction allows for new tasks to be added quickly with compatibility across all implemented algorithms.
3.2 REWARD FUNCTIONS AND EVALUATION METRICS
Because RL4LMs provides generic interface for per-token per-sequence generation rewards, it is possible to quickly apply a wide array of RL algorithms to a similarly diverse range of textual as-rics-a--rewards Specifically, we provide interfaces to 1) n-gram overlap metries metries such as
ROUGE (Lin 2004), BLEU (Papineni et al.) 2002), SacreBLEU (Post 2018), METEOR (Banerjee)
& Lavie 2005) ; (2) model-based semantic metrics such BertScore (Zhang et al.) 2019) and
BLEURT (Sellam et al. 2020) which generally provide higher correlation with human judgment:
3) task-specifie metries such as CIDER (Vedantam et al. 2015). SPICE (Anderson et al. 2016)
(for captioning/commonsenes
generation), PARENT (Dhingra et al. 2019) (for data-to-text) and SummaCZS (Laban et al. (022) (for factuality of summarization); 4) divessty/fteencynatrrahh
metries such as perplexity, Mean Segmented Type Token Ratio (MSSTR) (Johnson) 1944), Shannon entropy over unigrams and bigrams (Shannon, 1948). the ratio of distinct n-grams over the total number of n-grams (Distinct-1, Distinct-2) and count of n-grams that appear only once in the entire generated text (Li et al., 2013); 5) task-specific, model-based human preference metries such as classifiers trained on human preference data collected in the methodology of Ouyang et al. (2022).
3.3 ON-POLICY ACTOR-CRITIC ALGORITHMM
RL4LMs supports fine-tuning and training LMs from scratch via on-policy actor-critic algorithms on language environments. Formally, this class of algorithms allows to train a parameterized control policy defined as na : S + A(A), a function that attempts to select an action in given state as to maximize long term discounted rewards over trajectory E, 12 ((s de)]. Our benchmark experiments focus on fine-tuning a pre-trained LM denoted as To from which we initial our agent's 's policy Te To. Similarly, the value network Ve used to estimate the value function is also initialized from To except for the final layer which is randomly initialized to output single scalar value. As with other deep RL actor-critic algorithms, we define
value and Q- Q-value functions as V = - DD (r, a y)]. Q7 (8,, az) R(sr, de, y) + > 1 P [V (8 1)] leading to a definition of our advantage function as AT (s, a) QF (s. a) - V," To increase training stability, advantage is appoximated using Generalized Advantage Estimation (Schulman et al. 20156).
Given an input- pair (æ, y) and generation predictions from our agent; because the environment rewards are sequence-level and sparse, following Wu al. (2021a) we regularize the reward function using a token-level KL penalty for all on-policy algorithms, to prevent the model from deviating too far from the initialized LM no. Formally, the regularized reward function is:
A(s:. ae, 3) = R(st, at, y) 3KL (To (s:) [s)]((T((e)(ss)
where R is the regularized KL reward, is gold-truth predictions, KL (T [TTT] s;)) (log To (ar (8t) 11-lop log NO (0r| s.)) and the KL coefficient 3 is dynamically adapted (Ziegler al. 2019). Further details on actor-critic methods can be found in Appendix
4 NLPO: NATURAL LANGUAGE POLICY OPTIMIZATION
Language generation action spaces are orders of magnitude larger than what most discrete action space RL algorithms are designed for (Ranzato et al., 2016 Ammanabrolu, 2021). e.g., GPT-2/3 and T5 have vocabulary size of 50K and 32K respectively. We hypothesize that the size of the action space is core cause of instability when training LMs with existing RL methods. To address this issue, we introduce PI.PO (Natural Language Policy Optimization), which inspired by work on action climination/invalid
masking (Zahavy et al. 2018 Huang & Ontañón, 2020 Ammanabrolu & Hausknecht] 2020). NLPO, a parameterized -aakedd extension of PPO, learns to mask out less relevant tokens in-context as it trains. NLPO accomplishes this via top-p sampling. which restricts tokens to the smallest possible set whose cumulative probability is greater than the probability parameter p (Holtzman et al. 2018).
Specifically, NLPO maintains masking policy Tz. the masking policy is a copy of the current policy (TO)), but is updated only every I steps. A parameterized-invall mask is created from n.. by first.
4

---

Published as a conference paper at ICLR 2023
selecting the top-p tokens from the vocabulary. and then applying invalid-mask to the remaining tokens-ine setting their probabilities to zero when sampling actions from πT during training: this periodic updating policy The is inspired by off- off-policy Q-learning algorithms (Andrychowicz et al. 2017), providing the policy *o with an additional constraint that balances between the benefits of containing more task relevant information than the KI. penalty derived from mo and the risk of reward hacking. We provide pscudocode in Algorithm (green portions highlight the differences with PPO).
Algorithm NLPO Natural Languge Policy Optimization
Input: Dataset D { y²) of size N
Input: initial policy parameters Th
Input: initial LM To
Input:
initial value function parameters Vep
initialize parameterized masked policy Too (o,) with parameterized top-p policy Teo Input: policy update frequency μ
Input:
repeat
Sample mini-batch D., = ((=""""") ) from D
Collect trajectories Ti = (Ti) by running policy Tv., in for batch Dm in env.
Eq > Eq.
Compute Preference and KL penalty rewards ft
Compute the advantage estimate A.
Sec. 3.3
Update the policy by maximizing the PPO-Clip objective:
T
Σ
Σ
min (0)(0) clip(rr(0)), (- t)A 1)
mm argmax
ITm/TT
TEDm r=0
where (0) ma se) (ae 8e)*
Update the value function:
T (Vo(s)) - RRe)
Vomes argmin o [DmITT TED t=0
Update the parameterized masked policy every j iterations:
Tolnta (.1s kom4r)
unfil convergence and return m3
GRUE (GENERAL REINFORCED-LANGAAR UNDERSTANDING EVAL)
GRUE is a collection of generative NLP tasks. To combat reward hacking for any single metric. each task is evaluated at test time according task-specific mix of metrics, detailed in Table T The metrics span two categories. Task preference metrics capture how well the models produce generations that satisfy the desiderata of the specific generation task, e.g., for Commongen, if the generations contain all the required words, or for IMDB, how positive the generated completions are. Naturalness metrics capture fluency, readability, etc. and provide perspective semantics. At training time, there are no special restrictions: models are free to use the supervised data, compute metrics on intermediate generations, etc. Train/val/test splits follow the original works. All results are averaged over multiple seeds, with exact counts being found in Appendix B
factors beyond
Experimental Setup. We use RL4LMs to test large range of algorithms on the GRUE benchmark. Specifically: We compare 3 algorithms for direct fine-tuning Supervised, PPO and NLPO. In
"T could be trained with altemate sampling techniques like top-k beam search (or hard-coded via nules by domain experts), though we find top-p sampling to be most effective practice.
'ee consider PPO representative of the present stte-oofthh of-the-rrt in particular, we do not consider the popular FEINFORCE (Willanms) [1888] Willhams] 1992), as recent works have shown PPO to be strictly superior REINFORCE in multiple domains (Schulman 2017)
5

---

Published as a conference paper at ICLR 2023
Dataset
Task
Input
Output
Tisk Preference Metrics(s)
Naturalness
Metricsss)
IMDB
Maxs
2011
Text Consinua-
Partal Movie Review
A positive completion of the movic roview.
Learndd Seatiment Classifier
Perplexity
(GPT-22
CommonCGN
Concept
senteace coberently using
CIDER; ROUGE-2L. BLEU-3,4; METEOR: Coverage
Lin ct al. 2020
Commonsenso
imput concepts.
CNN Duily Mail
Hermann
2015
Summarization
News Article
Sammariced articke.
SummaC7S, ROUGE LSom:
METEOR BLEU
BerSeouee
ToTTo
Parikh ct al 20201
Data to Text
Highlighted Wiki Table
Factuallly accurate text descrihing the information.
SacreBLEU, PARENT
BLEURT
IM (en del
Machine Trans-
Text (Englista)
Translated (Cerrman).
TER: cHRE ROUGE LSum. METEOR; SaCt BLEU
BertScore
Bujar
NamatiieQA
Kuciskk et 2018
Question An- swering
Question Connext
(a Story)
Alstractive nnewer to the question.
ROUGE-1, 2, I., 1.m, I. Max; METBOR, BLEU; SacreBLEU
BertScore
DailyDiale La 2017
Chitchut
logue
Dizlogue History
converstional response
METEOR;
Classifier
BertScore
Table 1: GRUE Benchmark using RL4LMs showing the various tasks, input and output types, and the metrics used. We note that we test RL algorithms on these tasks for a wider range of possible rewards than just the task specific ones shown here. Unless specified, datasets are in English.
addition, we consider hybrid approach of supervised learning and our RL methods by applying PPO and NLPO on checkpoints that have been fine-tuned in supervised fashion- call these Supervised+PPO, Superviscd+NLPO. As an additional baseline, we additionally run zero-shot evaluations where we design prompts which aim to elicit task-specific generations, but with no training data or parameter updates.
For each task, to isolate the effect of training method, we select single pre- trainee LM backbone. For IMDB text continuation we use GPT-2 (1 [117m parameters), and for the rest of the tasks we use T5-base (220m parameters). For our RL models (PPO, NLPO, Supervise++PPO, Supervised+NLPO), for a thorough investigation of how hacking might interplay with GRUE, we run a separate set of experiments optimizing multiple task rewards for each task independently, e.g., for Commongen which has 6 task rewards (CIDER, ROUGE-2, ROUGE-L, BLEU-3, BLEU-4, METEOR) we run 6 different experiments optimizing each metric independently and report all possible metrics seen in Table I regardless of which individual metric was being optimized for.
Human Participant Study. We gather human judgments for five of the tasks in GRUE In doing so, our goals are 1) to validate that the automated metrics we selected for GRUE correlate with human judgments with respect to relative ranking between models; and 2) to provide additional empirical comparisons regarding NLPO vs. PPO, ablations to study the effects of the KI. naturalness penalty, etc. We specifically consider IMDB, Commongen, ToTTo, DailyDialog, and CNN Daily Mail. For each individual sample in task, we ask 3 unique human raters to provide Likert judgments of I) quality, i.e., for the specific task, how correct/appropriite is the generation, given the context, and 2) fluency, i.e., how well-written is the generation. We used Amazon Mechanical Turk, and paid crowdworkers a minimum of $15/hr. More details, including qualification information, interface screenshots, instructions, etc. are given in the corresponding Appendicies.
5.1 RESULTS GRUE: WHICH ALGORITHM SHOULD BE USED TO LEARN PREFERENCES?
Figures ((a) ((b) present the results on GRUE, split into task metrics and naturalness metrics, and key results via ablation studies. Full results are available in Appendix B For text continuation and summarization, with non-trivial zero-shot performance, RI tends to perform better than supervised training, but for tasks like Commongen and ToTTo, which have very low zero-shot performance, supervised training performs with both approaches outperforming zero-shot.
However, using RL+Supervised learning in conjunction works best; NLPO+supervised and PPO+supervised usually always outperforms NLPO/PPO (or supervised in isolation) across both task metrics and naturalness metrics. Supervised warm-starting is particularly effective for Commongen and ToTTo, which our results suggest are more prone to reward hacking. The one exception this trend is DailyDialog where the RL models outperform warm warm-started Supervised+RL models likely due the low performance of the Supervised models. We note that Supervised+N NI.PO ILPO using

---

Published as a conference paper at ICLR 2023
NamativeQA
IMCB
WHITE
WHI15
Commanges
10TTO
DaltDilllgg
DalyOlsshg
(a) Automated Task Metrics
(b) Automated Naturalness Metrics
Aig
15Th
BT
DaigDkeiog
(c) Human Study Task Metries
(d) Human Study Naturalness Metrics
Figure 2: Summarized results via automated metrics across all 7 GRUE tasks for each of the 5 algorithms we consider, and human participant studies for the tasks suitable for human studies. Test results averaged over all the respective metrics in Table
Ablation
Sentiment
Perplexity
Zero Shot
0.489
32.171
Lommentio
LNNAM
lirro WMIl6
NeQA
Supervised
Neata Werm Stn
PPO
0.602
33.816
NI.PO
0.611
338832
Sap carm?
Warm Starting (Sec: 5.
RI Sag cummen) Supett: (axe2?
PPO+Supervisel
0.626
35,019
NLPO4Supervied
0.620
34,816
Stp-Nt M1 Sxp-0190 Stp-Nt.10
Data Budget (Reward tmained on 10% data, Sec S
NLPO
0.599
33,536
Table 2: Key questions answered using GRUE RL4LMs:
Removing NIPO Top-p Constraints (Sec. 5.2
This table summarizes the results found the ablations and ask
(p equivalen to PPO, p 0.9 is NEPU)
NLPO 0,1
0.579
32,451
Fig. and provides overview of the questions in Section 5) which tasks require warm starts or are easily RL over Supervised, when to use both; and when to use NLPO over PPO. All conclusions drawn are the result of statistical analysis as discussed in the experimental setup.
NLPO
Removing KL Constraints (Sec 5.2
PRO-Ro-KL
41.897
reward hackable; when
NL.O--a-KLL
0.858
41,429
Discount Ablation
(Sec
PPO NIPO
0.651
41.035
0.624
43.720
Table 3: IMDB Ablation Results.
T5-base (220m parameter) LM currently outperforms all the models on the ToTTo leaderboard, many of which have 3b parameter supervised models--suggesting that RL is parameter efficient well. In these cases, it is critical that the initial policy already contain (some) signal for the task due to being used a KL constraint and masking constraint in NLPO. If the mask contains initial priors about task specific language, it will be eliminating the wrong actions-a a better initial policy leads to better RI performance downstream.
7

---

Published as a conference paper at ICLR 2023
Human agreement with automated metrics. As human judgments can be noisy, we run additional statistical analysis such as measuring inter-annotator agreement, via Krippendorf's s alpha score, and using one-way ANOVA followed by post-hoe Tukey HSD test to measure if differences in means of average scores between pairs of models are significant. We find that trends in our human evaluations generally match those seen in the automated metrics for both task and naturalness metrics (see Figures [ 2(d) which summarize Appendix Tables [O 3 1SSS-Spprriisc-NLPPP > Supervised SUP > NLPO 2 PPO > Zero-shot- with the exception of Supervised outperforming Supervise++PPO on 2 out of 5 tasks when automated metrics would indicate that Supervise++PPO outperforms Supervised on all of the tasks. We draw two conclusions from this: (I) if the generated text is above certain threshold of naturalness, the automated metrics usually correlate with human judgements; (2) usually but not always as seen in the relative performance of Supervised and Supervise++PPO, potentially indicating reward hacking behaviors undetected by automated metrics but caught by human preference feedback.
5.2 PREFERENCE REWARD LEARNING, SELECTION, AND HACKING
While the GRUE benchmark's 's metric for each task is an average over several measures, the RL models we trained optimized only a single metric independently. Thus. we can empirically investigate which metric for which GRUE produces the best results. We observe that many possible single metric rewards provide task performance gains over supervised methods (results shown in Fig. ((a] ( are
averaged across these reward functions) with the condition that the text is also coherent and natural.
Which constraints best prevent reward hacking? The reward function in Equation ] balances a task-specific reward with a KL constraint models are penalized from straying too far from a base LM in their pursuit high reward (Table 3 and Appendix T clearly show that if KL constraints removed entirely, models reward hack). But which model works best a base regularizing LM? When the initial policy (i.c., the raw, pretrained model) has low performance on the task, the KL penalty pushes the policy towards nonsense, c.g. on Commongen and ToTTo the trained policy learns to simply repeat portions of the input (as seen in Tables B.4.5 B.6.4). This behavior is mitigated if the base regularizing LM is the supervised model the reward encourages the policy balance the task-specific reward and a more reasonable regularization term. Deriving KL penalties from warm-started initial policies is critical for performance on such tasks.
PPO vs. NLPO. Figure 2 shows that NLPO generally outperforms PPO and supervised, especially when applied after supervised training. We hypothesize that the primary reason for NLPO's improved performance and stability is because the masking policy provides an additional constraint for the current policy. This constraint is not based on the initial untuned policy like the KL penalty but of the policy from iterations ago and likely contains more task-relevant information learned during RL training. Table (and Appendix Table 8 shows how performance increases up to point and then decreases p in top-p sampling is increased for the masking policy, relaxing the constraint by eliminating less tokens at each step, implying that there a balance to found how much the model should be constrained during RL training.
Human Preference Reward Learning. To this point, our experiments have largely focused on optimizing evaluation metrics that correlate with human judgments, c.g., METEOR. Here: we additionally test how well preferences can be learned from direct human feedback. For this, we focus on Commongen
GRUE dataset well-suited for displaying differences due to human preferences. First, we randomly select prompts from the Commongen train dataset and sample single completion from both the Supervised and Supervised+NLPP models. We then present the prompt and the two completion candidates to 3 unique crowdworkers and ask them to select which one they prefer with respect to commonsense/fuennc for 417 unique pairs (Krippendorf a = .28). We use this data to train reward model, T5-11B Raffel et al. 2020). on the balanced binary classification task of predicting which of the pair was preferred by a majority of annotators, conditioned on the prompt and completion. The resulting model achieved 69.5 test ROC AUC suggesting it indeed captures average human preferences. Additional details on this process are found in Appendix B B.4.4] We train Supervised+RL with SECTEOR-only reward as a baseline, and compare it to a reward function that uses the fine-tuned T5-11B model. Finally, we rerun the same pairwise preference collection procedure-the time sampling from Commongen test-withh with human participants to compare the generations from a preference optimized RL policy to the previously best Supervise++NIPO policy. Comparing the METEOR- to the preference model, the generations produced by the human
8

---

Published as a conference paper at ICLR 2023
feedback model are preferred in 682 cases, compared to the METEOR- only model which is preferred in 587 cases (p < 0.01 the models are equally preferred). This implies that this pipeline of collecting preferences, training reward, and further tuning the policy improves alignment to human preferences.
5.3 DATA BUDGET: IMPROVE YOUR REWARD OR GATHER MORE DEMONSTRATION
Given a fixed data collection budget, is it more efficient to gather feedback to improve a learned reward function or to gather more expert demonstrations? We use the IMDB text continuation task as case study. In the IMDB task, model is given partial movie review as a prompt, and is asked to continue it as positively as possible (even if the prompt was negative). The original dataset consists of movie reviews and sentiment labels of positive, negative, neutral. A DistilBERT (Sanh et al. 2019) classifier is trained on these labels and used to provide sentiment scores on how positive a given piece of text is, which serves as the task reward. The trade-off is between gathering more: 1) sentiment labels (improving the reward); or 2) positive sentiment reviews (improving supervised training).
We train classifier varying amounts of training data and evaluate on the held out test dataset- finding as expected that more training data improves test accuracy and so results in a higher quality reward. We then each of these rewards of varying quality during RI. training, and evaluate using the same metric as GRUE (i.e.. a classifier trained with the entire training set). As seen in Table 3 we find that improving the reward quality improves LM performance as well. Further, we trained a supervised model with at least as many samples used to train each of these reward classifiers. We find that a learned reward function enables greater performance when used as a signal for an RL method than supervised method trained with 5 times more data. This implies that improving reward models can more data efficient than collection expert demonstrations for task-and that's s not accounting for the fact that assigning sentiment labels is likely a simpler task than writing full demonstrations. Further details on this ablation are found in Appendix Table
5.4 PRACTICAL CONSIDERATIONS: WHICH IMPLEMENTATION DETAILS MATTER MOST?
Generation token-level MDP, not a bandit environment. Most recent works that tune LMs using RL do so by calculating reward for all the tokens in the sentence (Wu et al., 2021aa la Ouyang al. 2022 Lu et al., 2022). This setting equivalent to bandit feedback environment where the action space is the space of all possible generations for the task (Sutton & BartO 2018). This type of environment can be simulated within our RI. formulation by setting the discount factor y = 1. Table (and Appendix Table 6) shows that this causes instability in training with respect to naturalness in both PPO and NLPO for IMDB, Our standard setting is 7 = 0.95 when calculating discounted rewards-to-ge in the token-level MDP formulation, which reduces the magnitude of the reward that is applied to tokens selected at the beginning. The sentiment scores are approximately the same between both settings but the naturalness of language in the bandit setting is significantly less iess-iiiig that discounting rewards with < via a token-level MDP formulation is at least sometimes more effective for language generation.
Dropout and Sampling. We found two other implementation details to be critical for stability of RL training. The first is dropout, which in its standard form was found cause instability in policy gradient methods in continuous control settings by Hausknecht & Wagener (2022). We find a similar effect when using dropout when RL training LMs as well, with training loss often diverging for dropout > 0 in training. The second important detail, particularly affecting the machine translation task, sampling methods. We find that using the same sampling methods during exploration and inference is critical to translating training performance to test performance-elss the model exhibits high train rewards but low test metrics.
CONCLUSIONS
We're hopeful that the GRUE benchmark and the RLALMs library can push progress in aligning language models to human preferences via RL methods by providing the community with a standard means of comparing methods. Furthermore, we'r 're optimistic that, as the stability and consistency of training improves, our methods provide a path towards iterative improvement of language technolo- gies, with deployment, user feedback collection, and re-optimization enabling better user experiences when interacting with generative models.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENTS
We'd like to acknowledge the support of DARPA MCS program through NIWC Pacific (N66001-19. 2-4031), Google Cloud Compute, and the ReViz team at the Allen Institute for AI KB is supported by NSF under grant No. 2127309 the Computing Research Association for the CIFellows Project. Rajkumar funded by the Federal Ministry Education and Research of Germany and the state of North-Rhine Westphalia as part of the Lamarr-Insiitte for Machine Learning and Artificial Intelligence.
REFERENCES
Alekh Agarwal, Nan Jiang. Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms. CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, pp. 10-4, 2019.
Prithviraj Ammanabrolu. Language Learning in Interactive Environments. PhD thesis, Georgia Institute of Technology, 2021.
Prithviraj Ammanabrolu and Matthew Hausknecht. Graph constrained reinforcement learning for natural language action spaces. In International Conference on Learning Representations, 2020.
URL httptttsp///peeeeiecieeee www./eet/..ooee Lttp:eommmdddxlo00ett
Prithviraj Ammanabrolu, Liwei Jiang, Maarten Sap, Hannaneh Hajishirzi, and Yejin Choi. Aligning to social norms and values interactive narratives. In NAACL, 2022.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European conference on computer vision, pp. 382-398. Springer, 2016.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL otr.aaa
newriis.e.pppp. cttpaper//0111111.. /ille/ Paper.
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 700-710, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. http://////2222..nnnnnngg-o https: httpss///caanthhoooooe org///22..2222 http:///2222.....nnnn....ee Findings-acl.
URL
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. arXiv preprint 16Xiv:1607.07086, 2016.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings the 2016 conference on machine translation. In Proceedings of the First Conference Machine Translation: Volume 2, Shared Task Papers, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.
URL WtR.Lhttts:///caanttht
http://aaaaatttt loy.lol....oooo.oo... h 63200
10

---

Published as a conference paper at ICLR 2023
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint wrxiv:6606.01500 2016.
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, and John Langford. Learning to search better than your teacher. In Interaational Conference on Machine Learning, pp. 2058- 2066. PMLR. 2015.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, 2021.
Ting-Rui Chiang and Yun-Nung Chen. Relating neural text degeneration to exposure bias. In Proceedings of the Fourth BlackboxNI.P Workshop on Analyzing and Interpreting Neural Net- works for NI.P, pp. 228-239, Punta Cana, Dominican Republic, November 2021. Associa- tion for Computational Linguistics. doi: 10. 10.18653/v1/022 blackboxnlp-I.16. URL https wwc.aa/ww..ogg..ooro///.....ooe.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. In International Conference on Learning Representations. 2020.
Hal Daumé, John Langford, and Daniel Marcu, Search-based structured prediction. Machine learning, 75(3):297-325, 2009.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming- Cohen. Handling divergent reference texts when evaluating table-to-text generation. arXiv preprint arXiv:1906.0106 2019.
Ming-Wei Wei Chang, Dipanjan Das, and William W
Daniel Duckworth, Arvind Neelakantan, Ben Goodrich, Lukasz Kaiser, and Samy Bengio. Parallel scheduled sampling. arXiv preprint /906.0439) 2019.
Kartik Goyal, Chris Dyer, and Taylor Kirkgaiirr...rick... Differentiable scheduled sampling for credit assignment. arXiv preprint arXiv: TrXxv:/704.069700 2017.
Matthew Hausknecht and Nolan Wagener. Consistent dropout for policy gradient reinforcement learning. arXiv preprint 18X1v:2202.11818, 2022.
Matthew Hausknecht, Prithviraj Ammanabrolu, Alee-Alex Côté, and Xingdi Yuan. Interactive fiction games: A colossal adventure. In Fourth AAAI Conference on Artificial Intelligence (AAAI), 2020. URL htllhtts:///// http://axiviv... BtRl://ttte///aa..o.ooo/////////
05398
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators. arXiv preprint arXiv:180s 505.06687, 2018
Shengyi Huang and Santiago Ontañón. A closer look at invalid action masking in policy gradient algorithms, arXiv preprini arXiv:2006. 14171, 2020.
Ferenc Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint arXiv: 151.00000,, 2015.
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 3985-4003, Online, 2020. Association for Computational Linguistics. doi: 10. 10.18653/11/2000.emmmmm.mm
Wendell Johnson. Studies in language behavior: A program of research. Psychological Monographs, 16(2):1-15, 1944.
11

---

Published as a conference paper at ICLR 2023
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10.18653/v (4653/v1/22225 120200 www.aiineniiinggeene.....m 17 171.
Samuel Kiegeland and Julia Kreutzer. Revisiting the weaknesses of reinforcement learning for neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1673-1681, Online, June 2021. Association for Computational Linguistics. doi: 10. (0.186533v1/2021 .ww.ll-aannnnnn main. 133.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018.
Julia Kreutzer, Stefan Riezler, and Carolin Lawrence. Offline reinforcement learning from human feedback in real-world sequence-to-sequene tasks. In Proceedings of the Sth Workshop Structured Prediction for NLP (SPNLP 202/), pp. 37-43, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/20111 .ppnppp.... Utt.ttps:: /tttts:///act antha logy
orgg/021. spn sp-lp-1.1
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. SummaC: Re- Visiting NLI- based Models for Inconsistency Detection in Summarization. Transactions of the Association for Computational Linguistics, 10:163-177, 02 2022. ISSN 2307-387X. doi: 10. 10.1162/tac. a 00453.
URL wtt.httts://att org 1261/1162/2 atcc/aaclaae. Lalaaa___________ a_00453
Gerasimos Lampouras and Andreas Vlachos. Imitation learning for language generation from unaligned data. In Proceedings of COLING 2016, the 26th International Conference Computa- tional Linguistics: Technical Papers, pp. 1101-1112. The COLING 2016 Organizing Committee, 2016.
Rémi Leblond, Lean-Baptin Alayrac, Anton Osokin, and Simon Lacoste-Julien. Searnn: Training rnss with global-local losses. arXiv preprini 1rXiv:1706.04499 2017.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. arXiv preprint urXiv:5510003055. 2015.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1192-1202, Austin, Texas, Novem- ber 2016. Association for Computational Linguistics. doi: 10.18653W11D11--1122 URL https: h/aa//aaaahhhoooooooo
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialgg: A manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference
Natural Language Processing (Volume I: Long Papers), pp. 986-995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL httpsss//allaathh...oo Sxg/I17-1099
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1823-1840, Online. 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020. findings-emnlp.16
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summurization branches out, pp. 74-81,20 2004.
Siqi Liu, Zhenhai Zhu, Ning Ye. Sergio Guadarrama, and Kevin Murphy. Improved image captioning via policy gradient optimization of spider. In Proceedings of the IEEE international conference on computer vision, pp. 873-881, 2017.
12

---

Published as a conference paper at ICLR 2023
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1rXfv:1907.11692 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. arXiv e-prints, pp. arXiv-2205, 2022.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th anmual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Alice Martin, Guillaume Quispe, Charles Ollion, Sylvain Le Corff, Florian Strub, and Olivier Pietquin. Learning natural language generation with truncated reinforcement learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 12-37, 2022.
Tsvetomila Mihaylova and André FT Martins. Scheduled sampling for transformers. arXiv preprint arXiv: 17651,, 2019.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assistee question-answering with human feedback. arXiv preprint 2rXiv:21/2.09333 2021.
Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. Language understanding for text- based games using deep reinforcement learning. In EMNLP, pp. 1-11, 2015. URL http?
Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 1474,1474, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 00.1853//1/D117-115 URL https::/acclast wwwannholog..oooo wog.. log..org/D1--113.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. arXiv preprini arXiv:2203.02/55 2022.
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In International Conference on Learning Representations, 2021.
1d==Rox-uQ1HHua
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1173-111110 1186. Association for Computational Linguistics, 2020. doi: 10. 1111663/112222..mmml.. maaamnnllminn89. m9in.89.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint 1rXiv:1705.04304, 2017.
Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv: 180v: 2018.
13

---

Published as a conference paper at ICLR 2023
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transforme.. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http://mmmoorrrrr.o www.llooorrpppppeoooo /ttpers/vv//000 v21///1112200000 html
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 2021.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. In ICL.R, 2016.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In CVPR, 2017.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, JMLR Workshop and Conference Proceedings, 2011.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015a.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.1 02438. 2015b.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:1707.06:: 2017.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT:
Learning robust metrics for text generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892 Online, July 2020. Association for Computational Linguistics, doi:
10.18653/v1/202..co 111.00.cclmant mel-main.7000 URL
acl-main. 704
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-42) 1948.
Charlie Snell, Ilya Kostrikov, Su, Mengjiao Yang, and Sergey Levine Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.1185 118771 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei. and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-30211 2020.
Marilyn Strathem. 'improving ratings': audit in the british university system. European review, 5(3):
305-321, 1997.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pp. 10022-10032. PMLR, 2021.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-baas based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2015.
14

---

Published as a conference paper at ICLR 2023
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, ((3):229-555 1992.
RJ Willianms. Toward a theory of reinforeement-lcamingg connectionist systems. Technical Report NU-CCS-88-3, Northeastern University, 1988.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of- oheee-ofthh-arr natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv: 10962 2021a.
Qingyang Wu, Lei Li, and Zhou Yu. Textgail: Generative adversarial imitation learning for text generation. In The Thirty-Fifth AAAI Conference Artificial Intelligence, AAAI 2021, pp. online. AAAI Press, 2021b.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: 160..0669 2016.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. Learn what not to learn: Action elimination with deep reinforcement learning. Advances in neural information processing systems, 31, 2018.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: ho0::994400990 2019.
Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan. End-to-end offline goal-oriented dialog policy learning via policy gradient. CoRR, abs/1712.02838,, 2017. URL http://arxxvv.og//
abs/1712, 02838
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:9909.0859; 2019.
15

---

Published as a conference paper at ICLR 2023
CONTENTS
Introduction
2
Related Work
3
RLALMs: A Library for Training LMs with RL
Environments: Generation Token- MDP
Reward Functions and Evaluation Metries
3.3
On-policy Actor-critic Algorithms
NLPO: Natural Language Policy Optimization
5
GRUE (General Reinforved-languaee Understanding Eval)
5.1
Results GRUE: Which Algorithm Should be Used Learn Preferences?
5.2
Preference Reward Learning, Selection, and Hacking
5.3
Data Budget: Improve your Reward or Gather More Demonstration?
5.
Practical Considerations: Which Implementation Details Matter Most!
Conclusions
Acknowledgements
10
On-policy Algorithm Implementation Details
18
PPO Details
18
A.2
NLPO Details
18
Experimental Details
19
B.
Crowdworking Details
19
B.2
GRUE Experiment Setup
19
B.3
IMBB
20
B.3.1
Setup
................................
20
8.3.2
Results and Discussion
20
B.3.3
Human Participant Study
23
................................
B.3.4
Qualitative Results
26
B.4
CommonGin
28
B.4.1
Setup
28
B.4.2
Results and Discussion
29
B.4.3
Human Participant Study
32
B.4.4
Human Preference Learning Experiments
................................
32
B.4.5
Qualitative Analysis
33
B.5
CNN Daily Mail
35
16

---

Published as a conference paper at ICLR 2023
B.5.1
Setup
35
B.5.2
Results and Discussion
35
B.5.3
Human Participant Study
39
B.5.4
Qualitative Analysis
39
B.6
ToTTo
43
B.6.1
Setup
B.6.2
Results and Discussion
43
B.6.3
Human Participant Study
47
B.6.4
Qualitative Analysis
48
B.7
Narrative QA
49
B.7.1
Setup
49
B.7.2
Results and Discussion
51
B.7.3
Qualitative Results
51
B.8
Machine Translation
53
B.8.1
Setup
53
B.8.2
Results and Discussion
53
B.8.3
Qualitative Results
56
B.9
Daily Dialog
57
B.9.1
Setup
57
B.9.2
Results and Discussion
59
B.9.3
Human Participant Study
59
B.9.4
Qualitative Analysis
60
17

---

Published as a conference paper at ICLR 2023
A ON-POLICY ALGORITHM IMPLEMENTATION DETAILS
A. PPO DETAILS
Given discussion and equations in Section 3.3 further note that we follow (Ziegler et al. and dynamically adapt the KI. coefficient during training where,
2019
KL (T (TT(a()))) mo st)
KL -0.2,
Et clip
(2)
KLaaggtt
B[+1 Br + Kςet)
(3)
where Kl aurget user-specified KI. divergence between initial model h and current policy n and K₃ rate of update which generally set 0.2 in our experiments.
To increase stability during training, we further use Generalized Advantage Estimation (GAE) (Schul-
man et al., 2015b) and define the advantage estimator A(s,, an) based on the Temporal Difference residual as:
ó, r(s,, a.) + (o((++1) Vo(ss).
(4)
00
A(sn, an) Σ 16
(5)
where À provides the trade-off between bias and variance.
A.2 NLPO DETAILS
NLPO learns mask irrelevant language by maintaining masking policy T&. the masking policy is a copy of the current policy (Te). but updated only every steps. Given ((xz) Easy Te, (a s) the normalization value of the of probabilities of all action
A given particular State E S, let the parameterized top-p vocabulary VP, C V be the subset of the vocab, consisting of the top-p highest probability vocabulary tokens with respect to πe. Formally, let 2 be the normalization value for the parameterized top-p vocabulary, can be defined as the subset of tokens that maximizes Z" (TT) Loeve (a(a)s). Then optimizing a policy according to the parameterized top-p vocabulary can be defined as:
To (e(-ls)/2P(n) if a E VP and (TTe) otherwise.
(-18,,, nD)
(6)
0
18

---

Published as a conference paper at ICLR 2023
B EXPERIMENTAL DETAILS
B.1 CROWDWORKING DETAILS
Qualification round We ran a qualification round using the IMDB task. We opened the qualification around to users from {AU, CA, NZ, GB, US} with 5K prior approved HIT's and minimum acceptance rate 97% on their previous HITss We gathered judgments over 600 generations from annotators per generation. One of the authors of this paper also completed 17 random HITs to serve proxy for "ground truth." " After gathering these annotations, we selected workers who: 1) didn' significantly disagree with other annotators the same instance more than 20% of the time; 2) who completed at least 5 HITs; 3) who didn't tt disagree with the author annotator on the 17 HITs by more than point; and 4) (likely) spent reasonable amount of time reading the instmettions/exampee provided. In the end, 56 annotators were qualified. Additional per-task details are provided in the per-task sections of the Appendix.
Compensation details As per Amazon Mechanical Turk policy, annotators were compensated on a per-HIT basis. In addition, we used a timing script to estimate hourly wages to ensure our target of $1 $15/hr was met. In cases where this minimum hourly rate was not met, we manually assigned bonuses.
B.2 GRUE EXPERIMENT SETUP
We benchmark 5 training algorithms 6 tasks (see Table using either encoder model (eg. GPT-2) or pncoder-decodee model (eg. T5). We train policies using PPO, NLPO with variations of whether supervised pre- pre-training is applied before RL fine-tuning and compare against supervised policy. The choice of LM is based on the type of task. For IMDB text continuation, we use GPT-2 and T5 for of the tasks. We shared layers) in which the critic network has an additional linear layer mapping last token's 's hidden representation to scalar value. We use AdamW optimizer Loshchilov Hutter (2017) with fixed learning rate and no scheduling.
separate LM models actor and critics networks (i.e.
NamrativeAA
NarratvvOA
INDE
WMT34
Com mormorgen
Commongen
ToTT
DallyDilog
(a) Automated Task Metrics
(b) Automated Naturalness Metrics
Figure Summarized results via automated metrics across all GRUE tasks for each of the algorithms we consider, and human participant studies for the 5 tasks suitable for human studies. We break up the metries into task-specific, e.g. average positive sentiment for IMDB task, and naturalness metrics, such as perplexity and human perceived coherence for the human rated metrics. This plot differs from Figure 2 as this one averages over over multiple reward functions per each task.
19

---

Published as a conference paper at ICLR 2023
Model Params
value
supervised
batch size: 64
epochs: 10
learning rate: 0.00001
ppo
steps per update: 1280
total number of steps: 64000
batch size: 64
epochs per update: 5
learning rate: 0.000001
discount factor: 0.99
gae lambda: 0.95
clip ratio: 0.2
value function coeff: 0.5
nipo
steps per update: 1280
total number of steps: 64000
batch size: 64
epochs per update: 5
learning rate: 0.000001
discount factor: 0.99
gae lambda: 0.95
clip ratio: 0.2
top mask ratio: 0.9
target update iterations: 5
decoding
sampling: true top k: 50
min length: 48
max new tokens: 48
tokenizer
padding side: left
truncation side; left max length: 64
Table IMDB Hyperparams: Table shows list of all hyper-parameters and their settings
B.3 IMDB
B.3.1 SETUP
We consider IMDB dataset for the task of generating text with positive sentiment. The dataset consists of 25k training, 5k validation and 5k test examples of movie review text with sentiment labels of positive and negative. The input to the model partial movie review text (upto 64 tokens) that needs be completed (generating 48 tokens) by the model with a positive sentiment while retaining fluency. For RL methods, we use a sentiment classifier Sanh et al. (2019) that is trained on pairs of text and labels as reward model which provides sentiment scores indicating how positive given piece of text is. For supervised Seq2Seq baselines, we consider only the examples with positive labels. more suited for text continuation than encoder-decoder
We chose GPT-2 as LM for this task LMS (eg. T5). We top-k sampling with K 50 the decoding method and for fair comparison, we keep this setting for methods. For PPO and NLPO models, we train for 64A steps in total and update policy and value networks every 1280 steps with mini-batch size of 64 and epochs of 5 per update. We apply adaptive KL controllers with different target KLs of 0.02, 0.05, 0.1,, inf with an initial KI. co-efficient of 0.1. Table provides an in-depth summary of all hyperparameters and other implementation details.
B.3.2 RESULTS AND DISCUSSION
Target KL ablation Fig shows learning curves for PPO and NLPO in terms of episodic training reward, corpus level sentiment scores and perplexity scores on validation set averaged for 5 random seeds. It is seen that higher target KI. of 0.1 is desired achieve higher rewards but results in drifting
20

---

Published as a conference paper at ICLR 2023
3.06
0.05
0.06
9.1
0.1
7
0.00
900
10000
20000
20000
+0000
5000
12
20
54
20
(a) PPO Episodic total reward
(b) PPO Val avg sentiment score
(c) PPO Val perplexity
Light dwwtorgee.. C2
aig44 dtalorget aig4tdhhbbreed .M
Iigitl
0.02
0.00
0.06
0.00
News
0.5
None
stege
apoch
apoct
(d) NLPO Episodic total reward (c) NLPO Val avg sentiment score
NIPO Val perplexity
Figure 4: Learning Curves: Averaged learning curves over 5 different runs by varying target KI., shaded regions indicate one standard deviation. (a) shows the rollout episodic total reward during training (b) shows evolution of sentiment scores the validation split (c) shows evolution of perplexity on the validation split. From (a) and (b), it is seen that higher target KI. (0.1) is desired to achieve higher rewards. However, this setting drifts away from the original LM too much and loses fluency. Therefore a lower target KL (0.02 or 0.05) is required to keep the model closer to original LM. Similar trends hold for NLPO but when compared PPO, it retains lower perplexities and is more stable even with higher KL targets
Targe--LL
Sumantic and Flurncy Mitrics
Disersity Mitrics
Sentimert Scmre
Perpiaiity.
Disinct;
Disinct;
Hg
H2
Unique
Unigaat
Superstool
$45
0022
0.566
0,22
33.121
0.002
0.600 0.005
01944
0.001
0.299
0.004
8,565
0.009
13
0.026
S151 184
48160 735
0,15
0,594
10022
33,765
à: 0.367
0.571 0.005
0.043
0.001
0.286
à: 0.009
3,588
0.065
13,319
0.103
517L 190
46336 4: 1572
0.1
0.602
0.012
33,815
2: 0.233
0.664
0.007
0.042
0.001
0,278
d: 0.005
$,529
0:037
13.365
0.119
5108 204
45158
951
Inf
03338
00061
41,897
3: 1,806
0.577 2: 0.089
0.034
0.003
0,197
d: 0.036
7,737
0,114
11,365
0,993
4214
260
311811 5524
FFO+supervized
0014
$5.040
0.668
0.048
0.000
0:307
0.008
19,666
0.066
50523 1514
inf
0.004
1,716 42,016
0.617
0.038 0.003
0:333 0.017
5,149
12.753 0.316
337
37040 2507
0.02
0.57%
5030 96
184
0.55 0.023
0.453
0.293
0,015
13,656
101
822
0.283
01043 0.002
0.286
0.006
1555-1
5179 196
46,244
1072
inf
0.0.9
41,429
0.201 0.028
11,362
6189 608
31714 4500
NLPO+aipevvsed
0.1
8.020
00014
54,81 J. 0,340
0,672 0.006
01888 :: 0.022
0.31 2: 0.012
8.725 :: 0.09
13:709 à: 0.174
5489 140
50734 :: 19415
inf
0.377
0.042
41.085 2: 0.501
0,536 0.025
0.043
0.005
0.265
2: 0.034
3.575 22 0.269
12.947 0.339
5173
2: 539
43342 :: 6526
Table 5: Target KL Ablations: Mean and standard deviations over 5 random seeds is reported for sentiment scores along with fluency and diversity metrics validation set. It is seen from perplexity scores that lower target KI. constraint is desired to keep the model closer to the original model. On the otherhand, a higher target KI. yields higher sentiment scores the cost of fluency. inf KL. penalty (target KL of inf), model simply learns to generate positive phrases (eg: "I highly recommend this movie to all!". "worth watching") regardless of the context. NLPO achieves better sentiment and perplexity scores than PPO.
away from pre-trained LM and loses fluency. Therefore, lower target KL (0.02 0.05) is required to keep the LM closer to original LM. This is also seen in Table 3 where we presented a comparative analysis of final performance of all models.
21

---

Published as a conference paper at ICLR 2023
Training data size ablation We vary the amount of data used to train the reward classifier and the supervised baseline model to understand whether it is more efficient to gather data to improve reward model to gather expert demonstrations for supervised learning. As observed in Table 7 improving the quality of reward function increases the performance on the overall task better than training with more data for supervised training, indicating that improving reward models is efficient than collect expert demonstrations for supervised training from data efficiency perspective.
Discount factor ablation To understand the effect of discounted vs undiscounted (bandit) envi- ronments, we report sentiment and perplexity scores for different values of discount factor (0.5. 0.95 and 1.0) in Table and observe that using bandit environment (discount factor of 1.0) results in performance loss in the case of NLPO and reward hacking in the case of PPO, indicating that discounted setting (with 0.95) is desired.
NLPO params
Table. g shows ablation different hyperparameters in NLPO algorithm.
Semantic and Eluency Metries
Diversity Metries
Sentiment Score
Perplexity
MSTIR
Distinch
Disinct,
Unique:
Unoque:
Zero
0.459 0.006
32,371 & 0.157
0.682 0.001
0,0M2 0.001
0.294
0:001
8,656 0.004
13,716 a 0.003
5063 14.852
47620 238
PPO
0.5
0.511 + 0.023
15,045 + 0.92
0.60 0.001
0.044 0.00
0.904 + 0,007
8.726 0.041
13,793 + 0.055
5904 285
40658+ 1406
0.95
0.605 = 0.023
33,407 + 0.447
0.666 0.013
0.043 + 0.00
0.267 + 0.008
K.575
+ 0.073
13.484 + 0.244
5230
363
46483 + 1318
1.0
0.651 1.05
41,035 + 2.855
0.691 0,017
0,042 +: 0.004
0.205 # 0.031
K.807 0.237
13.563 = 0.396
5127 460
45319 :: 5650
NLPO
0:49 + 0.01
37,279 + 113
0.688 + 0.01
0.045 + 0.00
0,312 + 0.015
8.746 0.113
138873 0.25
5395 192
90828 2506
0.95
(1637 - 0.013
32,667 + 0.631
0.677
0.014
0.044
+ 0.00
0.268 + 0,010
8.588 0.100
13.484 + 0.2%6
5308
189
46344 + 2688
1.0
0.624 = 0.089
43.72 43.72:22445 2.475
0.622 0.019
0.0 + 0.077
0.3 0:0-8
8664 + 0.277
13.560 +: 0.537
6537 921
40441 + 6620
Table 6: Evaluation of GPT2 with different algorithms on IMDB sentiment text continuation task, discount factor ablations: Mean and standard deviations over random seeds is reported for sentiment scores along with fluency and diversity metrics. This table measures performance differences for the discount factor. We note that most NLP approaches using RL follow the style Lf et al.] (2016): Wu et al. a2 and use discount factor of 1. This is equivalent to reducing the generation MDP to a bandit feedback environment and causes performance loss (in the case of NLPO) and reward hacking and training instability (in the case of PPO).
Pire Data (siee)
Semantic and Fluency Metrics
Divurityy Metries
Suntiment Scare
Paplesity & 32,311 0.157
MSTTR
Disiies;
Diaind:
H,
H
Unique
Unique
Zano-Shot
0:499 à 0.000
0.687 0.001
0,02
0.001
0.001
8,56
0.004
+ 0000
5053 +: 14.832
47633 238
Supervisal
0.0 (Ok)
0.489 0.006
32:371 0.157
0.682
0.001
0.42
0.031
0 2994
0.003
13:716 0.003
9063
47620 233
0.1 (14)
0.511 0.001
34 143866
0-122
0A85
01001
0.455
0/511
0.315
0/044
8:775
0,005
13.864 0052
5015
51125 -655
(.5(
0,536 à: 0.006
35,008
2: 0.229
0.644
2: 0.00]
0.447
0.000
0.314
0.002
8.764
0.010
13,857
: 0.0178
$459
51284 576
(10k)
0,539 à 0.004
35,472 2: 0,074
0,682 :: 0.00]
0,447
2: 0.001
0.312
2: 0:002
8.755
0.012
13,906 2: 0,016
5601 57
51151 = 345
FPJ
8.0 (0k)
0.492 0.01
33,57 = 0,323
0.60 0.00
0.477
0.001
0.321
0.015
8.816
0.149
13.566 0.36
$629 :: 240
S2911 +: 1786
0.1 (3k)
0.99% à 0017
95 1,197
0,00
0.099
0.051
0003
03399
0:012
8,568
0.085
[4.015 : 0.158
6173 2: 560
55918 2541
(10K)
0,995 0,000
15.95
e 2.177
0.00
0.19
0003
01114
9.095
8AM
0.614
13.112 4: 1.173
5081 6:
51401 9191
1.0 (20k)
0,605 0.023
33,497
0,447
0,666
0.013
0,043
0.344
46483 1318
KLFO
0.487 001
12.572 0.165
0.685
0.033
0.033
137 0.034
401
n1 (3k)
0.900 0.007
53.536
0.375
350
0.192
46755
(10k)
1617 0.021
33.004 0.154
0.00
0.005
0,341
0001
0.231
0.041
13,535 0.091
45256 1022
(20(20))
0.677 0.013
32.007 0.511
0.77 0.014
0,066
0002
0.238
0.100
13.056 0.236
5205
46344 2005
Table 7: Evaluation of GPT2 with different algorithms on IMDB sentiment text continuation task, data budget ablations: Mean and standard deviations over 5 random seeds is reported for sentiment scores along with fluency and diversity metrics. This table measures performance differences as a function of the fraction of the dataset that has been used. In the case of the RL approaches, this measures how much data is used to train the reward classifier, and for the supervised method it directly measures fraction of positive reviews used for training. We note that using even a small fraction of data to train a reward classifier proves to be effective in terms of downstream task performance while this is not true for supervised approaches. This lends evidence to the hypothesis that adding expending data budget on a reward classifier is more effective than adding more gold label expert demonstrations.
22

---

Published as a conference paper at ICLR 2023
Hy perpparums
Semandic
Flaency Metrics
Diversity Metries
Sentiment Scor
Perpletry
MSTER
Disinet
Disttcct
14
H6
Unique
Uniguts
Targer Updone Ieeations
0.590
0.18
32.671 1 0.201
0.669
0.088
0,02 1. 0.002
0.264 :: 0,007
8.775 1. 0.064
13,50
2 0.181
4956
255
45916:: 1168
0.622 0014
12.729
a 0.367
0.65
0.019
0042
x: 0.002
0.274
a 0.007
8-489
a 0.105
13:31
= 0.272
5138
385
43989 1: 1120
70
0.637 0011
12.67
+ 0663
0.77
8:0.14
0064
+ 0/002
0.268
+ 0.000
588
+ 0.100
13,484
0.216
5205
189
46144 +: 7668
0,603 0005
$3.90
0.325
0.67
+ 0.006
0104 +0000
0.267
+ 0.00
86005
+ 0744
19.54
- 0.0
5228
115
46418 +n 685
Top-g mesk
0.1
0.579 0.021
32-451
= 0.233
0.67
0.000
0042 : 0.001
0.233 = 0.0
8.569
0,044
13.515
0.095
5008 =
45760 #: 1579
0.3
0.558 0.099
32:451
=: 0.303
0.666 01007
0043 :: 0.001
0.265
0.004
8.568
= 0,032
13,452
00.0200..0 0.172
5201 247
46357:: 539
0.5
0,588 :: 0.01
32 13.447
2 0.393
0.609
00001
0014 2: 0.003
0.291 0.005
8.614 2: 0,033
13.535
4: 0.06
5305 354
47251 4: 1226
0.7
0.619 0.03
32.373
2: 0.329
0.633
00008
0.043
2: 0.001
0.25 0.006
8,553
2: 0,043
13,366
0.129
5156
216
451.49
1452
0.9
0.637 =: 0013
32.607 2 0.61
0.677
0.014
0.04
2: 0.002
0,288
0,010
8.588 2: 0,100
13,484
0.256
5205
159
40344 J: 2638
Table 8: Evaluation of GPT2 with different algorithms on IMDB sentiment text continuation task, NLPO hyperparameter ablations: Mean and standard deviations over 5 random seeds is reported for sentiment scores along with fluency and diversity metrics. This table shows results of NLPO's stability the unique hyperparameters introduced in the algorithm all other parameters held constant from the best PPO model. The number of iterations after which the masking model synes with the policy and the top-p nucleus percentage for the mask model itself. We see that in general, the higher the top-p mask percentage, the better the performance. For target update iterations, performance low if the mask model is not updated often enough or if updated too often.
Coherence Alpha
Sentiment Alpha
Algorithm
Unique N
Value
Skew
Value
Skew
NLPO with KT.
27
3.49
0.196
3.497
3.61
0.2
3.601
NLPO without KI.
29
3.16
0.21
3.158
4.41
0.158
4.403
PPO without KL
27
3.16
0.17
3.163
4.36
0.196
4.363
PPO with KL
29
3.46
0.124
3.462
3.58
0.116
3.575
Zero Shot
28
3.6
0.162
3.591
3.1
0.13
3.097
Supervised
29
3,51
0.192
3.512
3,43
0.2
3.428
Human
27
4.13
0.159
4.128
3.01
0.31
3.017
Supervised+PPO
22
3.45
0.211
3.147
3.64
0.21
3.161
Superviscd+NLLPO
22
3.48
0.181
3.226
3.73
0.22
3.047
Table 9: Results of the human subject study showing the number of participants N, average Likert
scale value for coherence and sentiment, Krippendorf's alpha showing inter-annotator agreement, and Skew. For each model total of 100 samples were drawn randomly from the test set and rated by 3 annotators each, resulting in 300 data points per algorithm.
B.3.3 HUMAN PARTICIPANT STUDY
Figure shows the IMDB instructions, example, and interface used both for the qualification round, and then later, for the human evaluation experiments. Tables 10 show averaged results, annotator agreement, and the results of statistical significance tests determine which models output better generations when rated by humans.
23

---

Published as a conference paper at ICLR 2023
Enample tciikk acpenst)
Example Prompt
gase
rig abarinoblee 2h
Incnundone jrink eeppaart
System's generation (tate thist
inttart presented weth part reves that
cirectens peier wirk Fowenver hove hove-hepe for tive small soop is resuit producee hampering her exod ent, ort There are some in rinng momonts that show fer experiss,, and look forward to the di rectorr future mit
syteents wetssmatttaggggeeee continanter
Your the
weterr generation acruss aome:
Caterencer@ulllyy gotenss geweodoon groeroation cney follww fwo proops?
Sentiment And sontatctio souteisiios www.eeciiiii. sual
anyo completion recomnees
You'lil be stde cate each of three acse on sorle from to with being lowestiwnnt and the ghessieet. The spec scares are
prompi, sentiment semeewout strugt,
plaer noware
strccor other
oxplinnd
Sentiment why! complettion recogn
rig piver, but
effortiesll promp:
aboo prov hospeful message rectors
(pcodt
given
wetting
pakagl
Example Prompt:
problems gint might sense glven the inputt
proee cai
(eemille The completion molcody inoors contratioss imput.
caannot BELIVEE anyo is giving this fllm goas rating addition
thene severe crow in
flaoray.
temitlc zating, thin monos storth plat and chocooccoonn past, this would
were really 19f malenps the writerss rectorss br must bean Beaud
to watch
Sentimant
5/5 Every pelitiee The complet on glowingy postiie
concected somuthingg tinng abominnabie
4/5 mosely pwodiiee The completion is mosty ponatte, but thene are some impertectons meniiooee ec.
System's generation trate thagl
ineuraix The mplesion therr docon't repr esent pestoeereeaaiv
open seeiion contuins suth wayy poskiiee and veyy requtive angeetss
tostily regathed West what' expessed here regative.
anyonw
(ccthingly negesives the compkeion offers strongly regultve
agen inighly
anyone
rering only
eoorpe beage
CoherrenceQuualtye. trs Wertt the ware postoie can complation doaont make sanes very negacive di dususiion promps, contrad procept entinel. Sentiment Why? completioo solation offers pasi spinkara
Example to to acpansb)
Coherence//ualiige 3/5
den
Bol)
buckaox
coveeettio then Entrrtaaale
Sentiment: 3/5
Ver Megg
Vey Posilve
(Cationa) angthing
Submit
Figure Instructions, example, and interface for the IMDB sentiment completion task.
24

---

Published as a conference paper at ICLR 2023
Coherence
Sentiment
Group 1
Group 2
Diff (G2-G1)
p-values
Diff (G2-G1)
p-values
PPO with KL
PPO without KL
-0.3
0.035
0.783
0.001
PPO with KL
NLPO with KL
0.03
0.9
0,027
0.9
PPO with KL
NLPO without KL.
-0.3
0.035
0.827
0.001
PPO with KL
Supervised
0.05
0.9
-0.15
0.591
PPO with KL
Human
0.667
0.001
-05667
0.001
PPO with KL
Zero Shot
0.137
0.776
+0,483
0,001
PPO without KL
NLPO with KI.
0.33
0.013
-0.757
0.001
PPO without KL
NLPO without KI.
0.001
0.9
0.043
0.9
PPO without KI.
Supervised
0.35
0.006
+0.93
0.001
PPO without KL
Human
0.967
0.009
-1.35
0.001
PPO without KL
Zero Shot
0,437
0.001
1.266
0,001
NLPO with KL
NLPO without KL
-0.33
0.013
0.8
0.001
NLPO KL
Supervised
0.02
-0.177
0.404
NLPO with KI.
Human
0.637
0.001
+0.593
0.001
NLPO with KL.
Zero Shot
0.107
0.9
-0.51
0.001
NLPO without KL
Supervised
0.35
0.006
-0.977
0.001
NLPO without KL
Human
0.967
0.001
-1.393
0.001
NLPO without KL
Zero Shot
0.437
0.001
-1.31
0.001
Supervised
Human
0.617
0.001
-0.417
0.001
Supervised
Zero Shot
0.087
0.9
+0,333
0.0027
Human
Zero Shot
+0,53
0.001
0,083
0.9
SuPervised+PPO
Supervised+NLPO
0.03
0.09
0.035
SuPervised+-PPO
NIPO with
0.04
-0.03
SuPervised+PPOO
NLPO without KI.
-0.29
0.001
0.77
0.001
Supervised+PPO
PPO without KI.
-0.29
0.006
0.72
0.001
Supervised+PPO
PPO with KL
0.01
0.9
-0,06
0.001
Supervised+PPPO
Zero Shot
0.15
0.035
-0.54
0.001
SPP
Supervised
0.06
0.001
-0.21
0.001
Supervised+PPO
Human
0.68
0.001
-0.63
0.001
Supervised+NLLPO
NLPO with KL
0.01
0.9
-0.12
0.001
Superviscd+NLLPO
NLPO without KL
-0.32
0.001
0.68
0.001
Supervised+NI.PO
PPO without KL
-0.32
0.035
0.63
0.001
Supervised+NILPP
PPO with KI.
-0.02
0.9
-0.15
0.006
Supervised+NI.PO
Zero Shot
-0.12
0.001
-0.63
0.001
Supervised+NLPO Supervised
0.03
0.9
-0.3
0.001
Supervised+NLPP
Human
0.65
0.001
-0.72
0.006
Table 10: Results of an post-hoc Tukey HSD Test for difference in means between pairs of algorithms (Group Group 1) and corresponding p-values. Individually statistically significant results are bolded and used to discuss results in the analysis. Overall p-values showing that there is significant difference in means between the models via a one-way ANOVA test are significant with p 0.05 for both coherence and sentiment.
25

---

Published as a conference paper at ICLR 2023
B.3.4 QUALITATIVE RESULTS
We show sample generations from each of the algorithms for three randomly picked prompts below.
Sample
Prompt: à man his wile are not gelting along because of the wife's jealousy and drinking ema. When the wife di sappeass, her stster begina vigerous search involvingg ng the Zero Shot: husband's a sister, which leads several men to a remote cabin and then a boat. The things. Pinally, strenge weman is driving out to her cabir and discovers the husband's disappearance but
wife and hez husband then start
the
References; local police, state government, and television
show, What she finds atuns the entire community, Good drana with lota of fine per.formances; based on a true story
PPO: police and the local public. At the end, the woman is killed and the wife is found naked and with a zed head Lattooed on hez back, This is part of a special feature called"On the Road, Hidden" which
PPO-no-XL: polica and other lccal govermments. Written by E. Scott Fitzgarale, who also stars "A Very Merry Christmag" ia wonderful famt movie with wenderful soundtrack and wende:ful characters. IL brings home the Lzue beauty of snall
in this product
NLPO: nother while the brother goes under. After all, but who turns out to be man, is longer in her place, II film that was made in the 1950s
sister, who was apparently kidaapped
NLPO-RD-KL; friends of the wife, and also the friend the wife of the and eventually he begina to find that she has gone to Garmany and he has made a very good friend, There is good scene between John and Supervised: amily tzee. The widower, now widowed, returns, the daughter, daughter is not pleased when wife is mistaken for one of her sons, the patriarch, then wealthy falls victim to jealous husband. The
that is missing,
Sample
Prompt: A very credible and unseltling movie portraying the effects of war on the general public, Shame gives us a chilling understanding oh lagmar Bergman's a beliefs. The absence of color and sound (as
soundtrack) helps to give the film a more realistic feeling. There la no soft or hard lighting or variety of camera angles to ge the story charming or dramatic effect. Everything is very simply portrayed as it would be in of
confusing, but perhaps in order give the characters' frane of mind how they too de now know what is happening or why, The simplicity of the
reality. times the storyline
dero-Shot: camera wark is at ence apparent and fascinating in the manner in which is affects the way things are portrayad. In othar films these two have been referred to from time to time in a form and manrer more auited to the screen. The film shows
References: film made it somewhat boring, but could understand how the director is trying not glamorize the story and points. He deplcted the harsh and unsett evanta and effects of war. In the beginning the characters were introdueed and shown as content with the lightes sectings
PPO: characters relationships is sc engrossing and the great and unique story presented by each pair of brothers and sisters creates truly gripping and enjoysble expertenes To put 1s simply, I strongly recommend to everyone who plays the role to buy this DVD!!! PPO-no-XL: movie helpa it to flew along in a believable way. Razing 4 out of 5 Great Novie! highly recommend these movies! highly Movie! I highly secommerd these movies! Rating 10 of" NLPO: nevel mekes it one for someone who likes their books, like ma, to sit through novel. If you like to read novela, check it evt,
these moviea! Rating
Grest
first books
read by a British actor,
a fun way
NLPO KL: setling makes the final product very joyabbble to watch. Highly recommended!! Michael Rowerman, SPSA Film Festival Host, SPSA, KA (Decamber 17, 2006), 8/17/2006 Great photography, good acting, Supervised: "Dakote, Clan" and his sister, the two become friends with Sed Masters who soon finds the chance to fly back to their home land and go back to save his brother's life and his fami ly. The
Sample
Prompt: Shakespearean language is wild rambling, saturated in multiple neanings, word play and metaphor. To be understood it must be wrangled and tamed by an actor with the strength and knowledge to so. when en acter foils, the werds pour forth a terrent of incomprehensihl worda, but when he succeeda, the Engi ish language springs to life with an inmediacy and vibrancy that takes your breath away, Al Pacino is one such actor, and here displays incredible level of clerity and centrol thst, were there any justice,
Zero Shot: night still seen in world today. This movie, like "Vacacion", is an effort at telling the sale of tuo families from different races. one is African but naver grew up the other. One them, a man
References:
would sweep every avard in the offering, Be meets the challenge of presenting Shylock head on, and delivers an extraordinerily subile and nuenced performance. It would be crime if never got the opportunity to what he does with King bear.<br /><bs /ytha supporting cast
pPo: would have made him an inssant-claesie. This movie captures the spirit childhood and has averyone in the world watching it. It's a love story and it's really hearthreaking. And a5 tha same time, it's a claasic. Tt.
26

---

Published as a conference paper at ICLR 2023
PPO-no-KL:
extremism. Facine is very ful, well executed actor who brings an absolute intensity and censibility to his portrayal love and loyalty. must seet Recommended: Alse, an Excellent
will bring victory
fight against
must Highly recommend. highly recommend! Jue of my top ten films of all time. A must see!! NLPO-no-XL: the whole film would have fallen the
NLPO: I would watch and enjoy. 7/10, Oscar Kilde is always pleasure to
fate, just hundred years ago. Don't mlas it. It'a a real claasic. Eighly Recommended. a outta rive stors
did
couple
supervised: his performance 15o *ar) would seem nere shadow. He is truly in the middle of movie, and this film is one of those films where he can be convincing in it (and his srademark acting, yeu
in the
27

---

Published as a conference paper at ICLR 2023
B.4 COMMONGEN
B.4.1 SETUP
CommonGin (Lin et al. 2020) deals with task of generating coherent sentences describing an input set of concepts (eg. "a man is throwing frisbee"). For training RL methods, consider 3 traditional lexical rewards namely Rouge-I, Rouge-avg (which is an average of Rouge-1, 2 and L) and meteor. Additionally, we also train with task sss-sfpeifiic rewards such as CIDEr (Vedantam et al. 2013), SPICE (Anderson et al., (016) and SPiDer (Liu et al.) (017)) which is just linear combination of both with equal weights. We chose T5-base as the base LM since it is well-suited for structure to text tasks. We additionally note that concept set inputs are prefixed with "generate sentence with:" to encourage exploration.
During our initial experiments when fine-tuning directly on LMM, we observed that policy learns to repeat the prompted concepts in order to maximize rewards resulting in a well-known problem of reward hacking. To mitigate this, add penalty score of - to final task reward if the n-grams of prompt text overlaps with generated text. In contrast, when initialized with a supervised policy, this problem is not seen and hence penalty score is not applied. We use beam search as the decoding method during evaluation whereas for rollouts, we use top k sampling to favor exploration over exploitation. Table provides in-depth summary of setting of hyperparameter values along with other implementation details.
Model Params
value
supervised
batch size: 8
epochs:
learning 0.00001
learning rale scheduler: cosine
weight decay: 0.01
ppo/ nipo
stops per update: 1280
total number of steps: 256000
batch size:
epochs per update: 5
learning rate: 0.000002
entropy coefficient: 0.01
initial kl coeff: 0.001
target kl: 2.0
discount factor: 0.99
gac lambda: 0.95
clip
value function coeff: 0.5
mask ratio: 0.9
target update iterations: 20
supervised+ ppo (or nipo)
steps per update: 1280
total number of steps: 128000
batch size:
epochs per update: 5
learning rate: 000002
entropy coefficient: 0.01
initial kl coeff: 0.01
target kl: 1.0
discount factor: 0.99
gac lambda: 0.95
ratio: 0.2
value function coeff: 0.5
top mask ratio: 0.9
target update iterations: 20
decoding
num beurs::
length:
lokens: 20
tokenizer
padding side: left
max length: 20
Table 11: CommonGen Hyperparams: Table shows a list of all syper-parameter, and their settings
28

---

Published as a conference paper at ICLR 2023
B.4.2 RESULTS AND DISCUSSION
Tables 13 12
presents our benchmarking results with 6 reward functions along with supervised baseline performances on dev and test sets respectively. Our main finding is that warm-started initial policies are crucial for learning to generate coherent sentences with common sense. Without warm- start, policies suffer from reward hacking despite application of repetition penalty and task-specific metrics such as CIDer etc. Further, we find that RL. fine-tuned models obtain very high concept coverage which is also seen in Table B.4.5 Supervised models often tend to miss few concepts in its generation compared RL methods.
Alg
TM Rewert Facntion
Roage-
Reegr-1
Biea (n)3)
Bioo (Bwil
Missar
CTDFV
SPICE
Coveragg
Zere Slat
0,066
0.264
0,029
0.006
0.203
6,200
0.115
91.070
Rongee
0.085
0.001
0.067
0.334 0.013
8.12
0.00)
81.3 21.561:
Natss:
0100 Iu308
0.30 210777
2406 = 0.00
0.08: 10013
IL220 2:090
7.231
0.161 2: 8000
79:718 2 2.383
0.066
0.000
0.065
0.074
0.009
1.318 0.882
0.165 0.007
4.50
sPtder
0.117 8 10323
0L332
01877
0.004 02244
0.137 8 1081
0.26 0 0..00
06000-0.39 0.599
0.106 00005
73.54 6.003
XLPO
T5
Renge-Ara
0.055 0001
0,338 0.000
0.159=0:000
0.095 = 0013
02266
02.00200.009
1:55 = 0.558
0.155 0.355
17.04
2.70
CommonGGn
1003
10007
8.00 0.1
0.115 0000
81080 1813
ClDx
0.060 0001
0.341
0.003
33778 0.035
0.045 0015
03343 3.00
0.105 0001
45,00 7.901
0100
1.000
0.066
0002
0381)
0003 0.155
(W316
30/009
Sepervied
01115
0.329
11323
16,30
2 11.46
0.29
0000
94/276
Saparool
Rasgeo1
02200 01901
0.150 1181
0.005
03314
0.31 0001
15,09
0.877
0003
11.116 0.570
Mease
0.344
10033
0.150 11811
01000
13
1001
16-70
1 155
0.30m 6000
97/M1 133
2.000
0.356
20000
15.305
SPhier
0.226 0 0000
5/116
0.006
01119
1 1/001
11317
15,676
1 190
0.581
0000
05.151 1.6
Reage-Arz
01222
10933
11488:1
01342
10929
0185
15:31
11.33
0.221
00255
03.995 11,65
Maae
0.51
0003
0.489
0:000
0:55
0.007
033
0,008
0.336
0-000 0.00
16:544
= 369
0.500
0,00
51.37
0.457
Giler
01228
10912
0-532 0.307
03315 10956
0185
13:041 2 (1.77
0.238 00003
95.20 11973
SPidkr
0.256 = 0004
0.447 0.003
0-34 0.006
0116
0/066
0.39 0.000
15.799
0.31
0.264-41
256-:0000
0.00
563333 0.644
Table 12: CommonGen test evaluation Table shows official scores obtained from CommonGen hold-out evaluation. The most important result is that RL fine-tume finettuning on supervised model yields better performance across most metrics especially Coverage which indicates the ratio of concepts covered generated texts
29

---

Louied
Seewark Hilles
Merri
Published as a conference paper at ICLR 2023
hacko
Tark
Brago
Bregst
Terderx
Ckin
5y6n
Dieae
Steen
Cales
Leugh
0336
0200
6014
5000
Cver
770
Reggel
35P-0000 0.94
000
630
ORW
1090-6911
6306 1006 023
(40)
0108 1000
300
53
8.58
6/01
1000
089
1001
6376 voOR
03W
636 1900
638
10.00
Mone
161-6300
000
2007 616 -800
5907
11:0 W0TH
138--335
n0w
8:01
1017-100
090
1.00 F00
018
135-190
330
087.00 Wn
033
1007
0,36.000
0389
1003
N3W
1907
000
3770
626
1906 0291
0399 1.06
038
7309
3943
7150
14.75
3065
1305-0073
Khor
053:030
038
1389
930
1003
1003-000
198
130
1000
048
0333
3455
086
0373
3381
503
1.308.3.555 432
3300 35.4
1407
3391
8:58
1001 2000
0348 5000
001
7886 1007
7197
3-11
35%.
73M
357
150--1
Reperieed
0.79 1081
636
6:01
1000
6:01
1.07-000
0p0
1.000
0460
1000
029
1.00
630 130.4.00 0000
6530
1066
100%
8:35
50440
6:53
2 2451
1000
011520000
106
107:0000
193:0000
156.030
038
1082
0.30.080
1007
0010
1044-000
1,54_000
1.54
1000
0234
6067
0118 (000)
1334
6470
3080
11367 1.36
1548
1348
1012
Seymied S120
158
0388
106
8-33 6.9
0.63
1000
150
1M5-000
1009
1411
6.49
10091
025
1399_000
088
6388
1005
11:01:000
480
18:96110
1.%4
Tege-tr
150
6300
000
8-38 6690
6.69
1078 0011
000
1-50 633
6.67
1000
030
100
0110:0000 0000
0110
1006
098
078
90318 119
69.04
10
301-4
1909 2013
11
1330 630
118:
6:31-1900
9:31
6:8
1100
013
000
104 1333
100
0776
1986
0188
3597
1198-188
0833
100
18:3
10
90 190
51
135.030 0.99
a 1/81
6-3 6.69
1000
1.04
068
082
1.30
- 6317 6469
40000
0550
012 60M
24.5-000
6588
21:3
2446
6:00
518015-8800
50
8339-022 -025
Gbe 5) B 335 135.030 0.90
1-0-16 049.
1:03
080
1.41 0314 0466
0216
1.09
0308
0600
6.79 L10
6.50
a 1071
2M 62.00
0100 001--011
5) 1 130-004 0.0 1.00
8-31 6a0
(.6) 1004
(014 EM6
1.05 000 1009 68
1.51 -31.
6466 1000 0251 E905
0.09 0005
-.45-085
0623 a 3004
10320 1130
66-335
7800
151155 (1.37)
Table 13: CommonGen dev evaluation: Table shows lexical, semantic and diversity metrics for best performing models found in each algorithn-reward function combinations along with best performing supervised baseline models. Generated text from these models submitted to official CommonGen test evaluation to obtain test scores presented in Table 12
30

---

Published as a conference paper at ICLR 2023
Coherence
Commonsense
Algorithm
Unique N
Value
Alpha
Skew
Value
Alpha
Skew
PPO+Supervised
25
4.14
0.073
4.137
4.03
0.137
4.023
NLPO+Supervised
26
4.25
0.036
4.253
4.16
0.002
4.163
Zero Shot
24
2.15
0.391
2.154
2.29
0.342
2.291
PPO
24
2.84
0.16
2.849
3.03
0.081
3.027
Supervised
23
4.39
0.159
4.387
4.21
0.225
4.209
NLPO
24
0.335
2.003
2.13
0.265
2.124
Table 14: Results of the human subject study showing the number of participants N, average Likert
scale value for coherence and sentiment, Krippendorf's alpha showing inter-annotater agreement, and Skew. For each model total of 100 samples were drawn randomly from the test set and rated by 3 annotators each, resulting in 300 data points algorithm.
Coherence
Commonsense
Group 1
Group 2
Diff (G2-G1)
p-values
Diff (G2-G1)
p-values
NLPO
PPO
0.847
0.001
0.897
0.001
NLPO
Supervised
2.397
0.001
2.083
0.001
NLPO
NLPO+Supervised
2.257
0.001
2.033
0.001
NLPO
PPO+Supervised
2.143
0.001
1.897
0.001
NLPO
Zero Shot
0.153
0.515
0.157
0.624
PPO
Supervised
1.550
0.001
1.187
0.001
PPO
NLPO+Supervised
1,410
0.001
1.137
0.001
PPO
PPO+Supervised
1.297
0.001
1.000
0.001
PPO
Zero Shot
-0.693
0.001
00.740
0.001
Supervised
NLPO-Supervised
-0.140
0.601
+0.550
0,900
Supervised
PPO+Supervised
-0.253
0.050
-0.187
0.045
Supervised
Zero Shot
-2,243
0.001
-1.927
0.001
NI. P..O+Supervised
PPO+Supervised
-0.113
0.008
-0.137
0.007
NLPO+Supervieed
Zero Shot
-2.103
0.001
-1.877
0.001
PPO+Supervised
Zero Shot
-1.990
0.001
-.744
0.001
Table 15: Results of an post-hoc Tukey HSD Test for difference in means between pairs of algorithms (Group Group 1) and corresponding p-values. Individually statistically significant results are bolded and used to discuss results in the analysis. Overall p-values showing that there is significant difference in means between the models via a one-way ANOVA test are significant with p 0.05 for both coherence and sentiment.
31

---

Published as a conference paper at ICLR 2023
Dumplic kuicl
Example 1:
Inderctoonoo oxpand)
System generation mesty
you tepp wannted with shart spstem pewwelll aeoeeatorn Lauely genanitinn
The mon wore gloe on his hans to open the syster
vary rgte antario Your is to
RacnggWinommnttcatog is the grsen't pooerodle grameatioa orry raa, and
Fluenyyrrammattoo 5- shy the compr waon gramenstto comeel bicy to road
Aurt
Commorammme: goleno praen arne dong dlands motial
Commonuenas:: Whyftt makes that one would wour a handd apenargggoer.
will asin zre each ofttee there zunn on bnwesleors, and bighess The spec
to with liring
Example 2:
5-5 (eocelleeni The penerrtion goommaticall and
Systems generation Toatee tteals
433 lepods The sernence larasl seros,
grommor souts/ous
moke for wiiting tntwedy
The ron wore cyese he gowe span haret.
unsimeral sensnnes.
meoning. diffout
Conoennorene 2n whyh Yournn whore decortton intheieey for mokes very dlow seroe.
grammaticely and
1/5 reerberbbe The ganenat on his sware erorr gggammmataatgnn armout compirady
Commmnnannce
5/5 ressonable presaibrete This disenbea
conore ttiiuuseeeessoooaeetee shutton
Example
imort resornttie This swel renornalty happan.
Systems goncration Itate migle
3/5 Insultatt at The stouton right huppoes, but not thut thalgst wwird.
25 most unreasonatic et Most whors cap rssedd here coulont happon happont
wautdn't happerts Thes impout emereemiiot.
Flumgyyraammatttti Whyl pon the gaze cenne
pongs but not Hhann
Nore for racht Seraermeetty dont wapy stout commonenta There gremmntical pertencca that norseraraa warsa bes esamptes
inarpolation get rhat.
Coherence/Qualityy 3/5 Sed
Exte
Commonsense: 3/5
knreecrubll placaall
(Dationa)) Plnta Earyting
undiar iy repererta saed, you taeear anyorat tedtarc
Tubmi
Figure 6: Instructions, examples, and interface for the Commongen task.
B.4.3 HUMAN PARTICIPANT STUDY
F shows the commongen instructions, examples, and interface used for the human evaluation experiments. Different from the other human evaluations, didn't 't provide any prompt because knowing the set of words to be used isn't 't required for rating either of the axes. Tables [4] show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans.
B.4.4 HUMAN PREFERENCE LEARNING EXPERIMENTS
First, we randomly select prompts from the Commongen train dataset and sample single completion from both the Supervised and Supervised+NLPO models. Next, we filter to prompts where both models at least attempted to use all input concepts. This filtration step was conducted because if a model fails to use all concepts, it may generate a more natural/fluent sentence, but, a priori, shouldn't 't be preferred by crowdworkers: instead of training crowdworkers to prefer sentences with all concepts, we perform this filter. Figure shows the task presented to the crowdworkers. We then present the prompt and the two completion candidates to unique crowdworkers and ask them to
32

---

Published as a conference paper at ICLR 2023
Instructions (click to expand)
Thanks for your participation and work on this HIT! In this task you will be presented with two sentences, each containing similar words. Your job is pick the sentence that you prefer. While the judgment ultimately subjective, consider the following factors:
Which sentence makes most sense? Which describes a plausible, realistic, and commonsensical scenario more effectively?
important:
Which generation is more gr grammatical, easy-to-read, read, and fluent?
In general, be forgiving of slight grammatical errors. If one conceptually makes much more sense than the other, you should prefer it even if there are slight issues with readability. However, there are severe readability issues that effect comprehensibility. feel free select the other option.
Thanks again for your efforts, oppreciate your work!
Please take a moment to read both choices. Select more commonsensical,, complete, and grammatical option. If they are both bad, still do your best to pick the one you prefer.
footballer receives instruction from football player when taking squad people
footballer receives instruction from football player during training
Figure Instructions and interface for the pairwise Commongen HIT.
select which one they prefer with respect commonsense/ftuency: We gathered annotations 417 pairs (Krippendorf = .28), and split into 60/20/20 train/val/test split. We then trained a reward model, TB-11B Raffel al. (2020), of the pair was preferred by a majority of 3 annotators, conditioned on the prompt and completion. The resulting model achieved 69.5 test ROC AUC suggesting it indeed captures average human preferences. The model is then used as a reward function. We train Supervised+RL with METEOR- only reward baseline, and compare We design the reward function based on the preference model as meteor + pre ([1 + |miss]] where miss is set of concepts not covered in the generated text, an attempt to mimic the data collection process that humans are instructed to follow. This reward function accounts for both the task of using all concepts and also human's 's preferences for how a sentence should look within the constraints stipulated by the task. Finally, rerun the same pairwise preference collection procedure this time sampling from Commongen test test-with human participants to compare the generations from preference optimized RL policy to the previously best Supervised+NLPO policy. Comparing the METEOR-only to the preference model head-to-head, the generations produced by the human feedback model preferred in 682 cases, compared the METEOR-only model which is preferred in 587 cases (p < 0.01 the models are equally preferred).
the balanced binary classification task of predicting which
reward function that uses the fine-tuned T5- 15-11B model.
B.4.5 QUALITATIVE ANALYSIS
This section shows sample generations from different algorithms for three randomly picked prompts.
Sample
Prompt: generzte sensence with: apron cut has kitcher sausage
SaxotShot: generate a sentence apron.
apron out hat kitchen sausage.
PPO: sausage in the kitchen
NLPO: sausage cut hat cut hat cut has cut aproa eut hat Supervised:
is cutting sausage
apren in kitchen.
supervised PPO: man cutting sausage fron a hat on an apron in the kitchan.
Supervised
man cuts sausage
apron kizchen.
Sample
33

---

Published as a conference paper at ICLR 2023
Prompt: generate sencence backhand competition player shuttlecock
Zero-Shot: hackhand competition player reach shutt lecock..
PPO: the ball in backhand and the competition player in the shut:lecock.
NLPO: backhanded competition player reaches shuttlecock.
Supervised: tennis player resches for shuttlezock during a competizion.
Supervised
feotball player reaches for shuttleesck in backhand during the competition.
Supervised the competition
football player
shuttlecock hitting backhand during
Sample
Prompt: generete a sentence with: bowl combine sucumber cnion pasta
Zero-Shot: cucumber onion
PPO: the cucumber onion paata in small bowl.
Combine cucumber Supervised: bowl containing pasta, tomatoes, oucunoers, and onions. Supervised PP0: bowl containing pasts topped with cucumbers, onions, and peşpers. Supervised NLPO: bowl containing mixture of pasta, sucumber, and enion.
pasta large Toss pasta
34

---

Published as a conference paper at ICLR 2023
B.5
CNN DAILY MAIL
B.5.1
SETUP
As representative of the summarization task, we consider CNN/DM dataset consisting of long news articles and their highlights written by news authors. The dataset consists of 287k training, 13k validation and 11k test examples. We trained RL methods using different automated metrics, namely Rouge-1, Rouge-avg and Meteor. We chose T5 as our base LM as is pre-trained unified text-to-text framework and relishes Zero-Sho capabilities. For decoding, use multinomial sampling with a temperature of 0.7 for the models.
Model Params
value
supervixed
batch
opochs: 2
learning rate: 0.0001
lcarning rate schodulen: cosinc
weight decay: 0.1
ppol nlpo
steps per update:
toral number of stepe: 512000
butch size: 64
epochs per update:
lcarning rate: 0 0,0000002
entropy coefficient: 0,0
initial kl coeff: 0.001
larget ki: (1.2 disccontt factor: 0.90
goe lambda:
clip rario: 0.2
value function coeffi:
rollouts top l: sweep of (50.100)
top mask ratio: 0.9
target update
sweep of00 20,
supervised-ppol ulpo
steps update: 5120
total number of steps: 258000
butch size:
epochs update:
learning 0.0000012
entropy coeffcient 0.0
initial kl cocff: 0101
target kl:
discconnt factor: 0.90
gne lambda: 0.95
clip
value lunction coeff: 01.5
rollouts top sweep of (50,
top mask ratio: 0.9
target update iterations: sweep of (10, 20, 30)
decoding
sampling:
temperature 0.7
min length: 50
tokens:
Inkenizer
padding side: left
truncation side; right max longth: 512
Table 16: CNN/DM Hyperparams: Table shows list of all hyper-parameter and their settings
B.5.2 RESULTS AND DISCUSSION
Table 17 presents benchmarking results on test set reporting wide range of metrics: lexical, semantic, factual correctness and diversity metrics. As baselines, report lead-3 which selects first three sentences as the summary, Zero-Shot and a supervised model. PPO and NLPO models are on par with supervised performance several metrics including Rouge-2, Rouge-L., and Bleu. On fine-tuning on top of supervised model, performance improves consistently on all metrics indicating that RL. fine-tuning is beneficial. Another interesting finding is that, RL fine-tuned models are factually consistent as measured by SummaCZS metric. For ablations on PPO params, NLPO params, we refer
Tables [8 19
35

---

Published as a conference paper at ICLR 2023
Talks
Lesical and Semaati Metries
Factual Consisiency
Diversity Metries
Alg
Reward Punction
LM
Rouge
Rouge 2
Rouge L
Rouge LSun Metice
BLEU
BentSooee
SummakZS
MSTTR
Distict,
Distioet
Hi
Hg
Unigue:
Unique:
Mean Oupur Length
Lead-3
273153
Zero Shor
0,372
0.654
0.414
16.183
19113
199999
PPO
Rouge
T3
0,410
0.182
0.383
0,349
0.276
0.085
0.876
0,622
0.769
0.068
0.464
10,061
16.437
18189
191383
47
Rougo-Arg
0.396
0.126
0.273
0.700
0.95
0.84
0.611
0.733
0.400
10.890
16
19478
209140
0.178
032)
0.005
0.447
10,009
20525
236360
NLKO
0,404
0.86
0.480
19677
201971
Rouge Avg
T5
0,404
0.177
0.279
0.34
0.274
0.084
0.874
0.586
0.765
0.066
0.476
10,744
16.630
18179
206368
50
Messo
13
0.405
0.180
0.277
0.343
0.292
0.305
0.822
0.578
0.72
0.044
0.471
10,802
16.766
20212
231058
Sipervised
0.11
0.401
10.459
16,410
21096
230343
Supervise PPO
CNNIDM
Avg
13
0.425
0.194
0.297
0.363
0.296
0.314
0.882
0,728
0.747
0.066
0.445
10.589
16.458
18939
200617
52
Messer
T5
0.466
0.194
0.203
0.361
0.316
0.325
0.800
0.726
0.741
0.000
0:430
10.532
16:491
20395
224432
65
Supervicdd PO
Reugo-1
0.421
0.287
0.108
0.882
0.340
0.748
0.446
10.598
16.913
18704
185561
Rouge Avg
0.193
0.295
0.443
10,500
15747
201700
Metsor
E5
01429
0.194
0.203
0.361
0,319
0.124
0.800
0.743
0.745
0.090
0.422
10,574
16 663
20158
226801
Table 17: CNN/Daily Mail test evaluation: Table presents a wide range of metrics: lexical, semantic, factual correctness and diversity metrics on test set. As baselines, report lead-3 which selects first three sentences as the summary, Zero-Shot and a supervised model. PPO and NLPO models are par with supervised performance on several metrics including Rouge-2, Rouge-L, and Bleu, On fine-tuning on top of supervised model, performance improves consistently on all metrics indicating that RL fine-tuning is beneficial. Another interesting finding is that, RL fine-tuned models are factually consistent as measured by SummaCZS metric.

---

Published as a conference paper at ICLR 2023
Lesical Semantie Metries Ruigol Sam
Alp
Reward Fanction
Tipk
Ronge-1
Respo]
Reggo.
Manorr
R0 FXI
RerSooe
Rouge-1
0.181
0:238
0.346
0:371
0,895
0.276
Rouge
0.342
0.271
0.399
0.179
0.275
0.342
0.270
0.894
0.874
Matex
0.413
0.183
0.279
0.348
0.301
0.110
0
409
0.345
Rouge-1
0.358
Superviaal+PP0
Honge=Arg
50
B1426
0.
0.2M4
0.366
0.314
0.114
0381
100
0.427
0.196
0.208
0.366
0.294
0.113
0.851
Table 18: PPO Ablation/Model Selection: Evaluation PPO models on validation set with different reward functions and top k values for rollouts. For each alg-reward combo, best model (top k is chosen.
Resort Tractice
LAckon mark)
Ronge-1
Rrage
Reege-LSer Nanter
B1:21
Banticor
0.990
0.138
0.335
0.543
0.399
0.96
01173
01274
0.340
0.237
0180
0375
100
6.817
0.177
0.355
0.567
0.89
01382
0.383
0.347
1306
0304
0,40
0119)
0.339
Benge-Arg
50
04
10
0.901
6.180
0.33
0.513
3.711
0106
6,71
0.125
1X0
04
0.345
0.151
0.331
135
03065
0188
6377
0179
0.355
50
0.955
0.543
0/811
01772
01273
0337
020
010
100
0.00
5.778
0.16
0.545
6.411
03394
01288
0.58
033
6318
6375
50
6:11
01191
0395
1.50
0776
6.10
6.01
0413
0.871
01196
0386
0.59
8:18
6
042)
0.993
0.56
Bonge-As 50
6426
0.997
0.55
0.367
0.06
0.115
6.81
100
0424
0.996
0.99
0.365
0.201
0.115
6.811
0.000
Mizerr
50
0.49
0.97
0391
0.544
0.501
0.130
0423
0191
0399
0.561
0.00
0.116
6.5%)
643
0:30
0.298
8330
0:131
0434
0.300
0.99
0.569
0.30
0.138
6.57%
Table 19: NLPO Ablation/MMode Selection: Evaluation of NLPO models validation set with different reward functions, top k values for rollouts and target update iterations. For each alg-reward combo, best model is chosen
37

---

Published as a conference paper at ICLR 2023
Coherence
Quality Alpha
Algorithm
Unique N
Value
Alpha
Skew
Value
Skew
PPO+Supervised
22
4.21
0.198
4.224
3.97
0.256
3.98
NLPO-SSuprrised
19
4.3
0.26
4.308
3.98
0.089
Zero Shot
17
3.73
0.1
3.757
3.69
0.25
3.722
Supervised
19
4.25
0.116
4,241
3.99
0.2
3.986
NLPO
17
4.03
0.13
4,042
3.83
0.191
3.832
PPO
21
3.94
0.111
3,945
3.76
0.129
3.767
Human
3.89
0.277
3.902
3.77
0.029
3.769
Table 20: Results of the human subject study showing the number of participants N, average Likert
scale value for coherence and sentiment, Krippendorf"s alpha showing inter-annotator agreement, and Skew. For each model total of 50 samples were drawn randomly from the test set and rated by annotators each, each resulting 150 data points per algorithm.
Coherence
Quality
Group 1
Group 2
Diff (G2-G1)
p-values
Diff (G2-G1)
p-values
Human
NLPO
0.147
0.755
0.060
0,900
Human
NLPO-Supervised
0,413
0.001
0.213
0,047
Human
PPO
0.053
0.900
-0.007
0.900
Human
PPO+Supervised
0.327
0.024
0.200
0.544
Human
Supervised
0,360
0.008
0.220
0.043
Human
Zero Shot
-0.160
0.679
+0.080
0,900
NIPO
NIPO-Supervised
0.267
0.012
0.153
0.008
NI.PO
PPO
+0.093
0.900
+0.067
0.900
NLPO
PPO+Supervised
0.180
0.564
0.140
0.860
NLPO
Supervised
0,213
0,361
0.160
0.754
NLPO
Zero Shot
-0.307
0.044
-0.140
0,860
NLPO+Supervised
PPO
-0.360
0.008
-0.220
0.043
NLPO+Supervised
PPO+Supervised
-0.087
0.009
-0.013
0.009
NLPO+Supervieed
Supervised
-0.053
0.009
0.007
0.900
NLPO+Supervised
Zero Shot
-0,573
0.001
-0.293
0.012
PPO
PPO+ PPO+Supervised
0.273
0.106
0.207
0.508
PPO
Supervised
0.307
0.044
0.227
0.394
PPO
Zero Shot
-0.213
0.361
-0.073
0.900
PPO+Supervised
Supervised
0.033
0,900
0.020
0,900
PPO+Supervised
Zero Shot
-0,487
0.001
-0,280
0.155
Supervised
Zero Shot
-0.520
0.001
-0.300
0.101
Table 21: Results of an post-hoc Tukey HSD Test for difference means between pairs of algorithms (Group Group 1) and corresponding p-vauues. Individually statistically significant results bolded and are used to discuss results in the analysis. Overall p-values showing that there is significant difference in means between the models via one-way ANOVA significant with < 0.05 for both coherence and sentiment.
38

---

Published as a conference paper at ICLR 2023
Insmuctions joick expard)
Article:
In thac yoawsl be presented w articee automent oygenne rateel
siprompti
rose the systom genoration paes
System's summary raaco
Coherencw/ Thunncy is the spaues geovaatt goomonatta took waye locmess, werts
Summary Quality: Oees dhe gptears goveration woad ingi fo copturt the monn soonts
St nammmhaayom
will each thrne
to
Pless tabe teme read symerhsummery and to som the anidle briett then, rnde the witer' summary the form loppoer 1os_ for Sunmary qualty" plesse rnlermee aelae 1ly (ow suntary the panti d''s tie sputgii ddtor
Coherence/ Rluengy
mestored shnsong conve..
b/b juscetiend:: the mary rhsel spammuttaal ituent, and reasoratie.
o 4/5 iggodds the summary penerall y maken serno, but there are miner grommmttto ee mereonrrnge repirna ih that denn mane for the hand wiiking
caherence/Fluency: 3/5
jolayss con why this summary wes generated, and it' somewhaz
reudactle there probilers thatl can't bw
2/5 posert Some parts
could maker sense, it's unnattuatt
Eicdeett
Ringlox exdie hard
Iteriiiet summary has nothing inggen matuaalyy by thaeray.
articie,
Laa only may
Summary Quality
+ 5/5 ivery goode thi summaar sorvecty and compl captures the kay paints of the orticts.
Summary Quality: 3/5 recsugld
a 4.% (mooet grodt The summary exptorett mort the any pointe from the ontide.
bie Sumnerry
Great
aib ruuutrat aratt White thene sig ous ncomsssinioes owss there mikiing painss 50t cneeott burty there are ressonani points
partly inceement summuntiatoon.
menatone some stings artirte,
domr reghig aoe mupe nevi boy portiny tor
summery
secamaty
(Doconal) Pheae ut know if anyyhh und youu sxpprr lenceed ony isuus orlif you hove ony other feedback ut
You don't need to read the entrine an trticte, but plwas do shan to get sariue of the main points tho relies the su minory in particular, you shoure foous two whit evolunting summary quality.
Ooes the sunteedry oover the maih points of star??
Ave fertsluterooon made sumrary enpedd the enticie on are jasts
intovontroed or endeappp?
In axrty jexding doing this ETT sursahes we found unalil cisplay the ihera ofter the debyy so we'vee mode so the stitee are hi hidden for reconds, They should appeor automatiiiyy nity
S.kmi
Figure 8: Instructions and interface for the summarization task.
B.5.3 HUMAN PARTICIPANT STUDY
Figure 8 shows the summarization instructions and interface used for the human evaluation exper- iments. Participants weren't required to read the entire article, but to encourage some reading, a minimum time on the window of 15s was enforced via hiding the sliders. Tables 20 21 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans.
B.5.4 QUALITATIVE ANALYSIS
We show sample generations from each of the algorithms for three randomly picked prompts below.
Sample
Prompt: Manchester City are confident VEPAaAEs purishment for breaching Linancial fairplay regulations will be lifted this summer which would allew them to bid for stellar names like Reheem Sterling, Gareth sale, kevin Bruyne and Ross Barkley. City boss Menuel Pel lagrigin has been hamperad over the past year by URFA restricting them to a net cransfer spend of 49million in each window end keeping the clubáRés overell wage bill to its cusrent level of iinnillion-a-yaat stated those penslties would remain in place uniil the end of the 2015/16 season clubsAta latest firancial figurea showed draatica reeeuee lossea of 23millian which they feel proves Lhey are new compliant with FP2 zegulations. Manuel Pellegrini is hoping that the financtal restriction imposed by UFFA for a bresch of PFP rulea will lifted at the end of this seasоa Kanchester City have been limited to net spend of 49 million in the last twe transfer windows they spent 25 bringing Wilfried Bony in Frem swansea in January Ahead of Monday nightaads trip to Crystal Palace, Peilegrint was certainly talking like man excited at the prospect of signing "crack' players this surmar. AAYT think that next season we donašft have any restrictions
UErAadda settlement with City publ ished in May
the
will be in the same positien that all the other English clubs have, said zellegrini. aXSitåkls imporsent. You have so many strang teams here England and in Champions League, you can not allow then to keep the advantage every year; having less playess to put in your squad or spending less money. Wa spend money, of course we always spand money, but they spent a8t Manchester United, Barcelons, Liverpool and Arsenal have all paid more in transfer feea in the past 12 montha than city who were tradit tonal ly FuropešAfa biggest
39

---

Published as a conference paper at ICLR 2023
spenders after the club was taken over by Abu Whsbi ooners in 2008. Uefa also ordered City to play with a redseed squad from 25 players to 21 in the Championa Leagua this season and while that restriction has now ended, any time reduction in the penaltiea on epending and wages is more controversiel. Arsenal have paid more in transfer fees than city in the last 12 months, including 30m on Alexis Sanchez The document published last May by UEPAaAAa Club Financial Control Body investigative chamber explicitly said Cityššda financial penaltias would run for two seasons at least and there has been no official deviation from that decision.
The publishad statement said at the time: SAiManchester City agrees to significantly limit apending in the transfer market for the seasena 2014/15 and 2015/16. have 2o arque their case with Dela that as they have been rinancially compliant over the past year, they deserve to free restrictions noving ferward. They have successiully argued their case with UEFA before. Lest summer they persuaded the geverning body to allow then to hypass the narmal quota aight horegrown players as their Champions Leaque squad had been rediced, Eliaquim Mangala jeined the champions from Porto for 32m last aurmar The reigning Fremier Laague champions have only paid money for Farnando, Willle Caballero, Eliaquim Mangala and Wilfried Bony in that was pazt paid by the sales of Javi Sarcia and Jack Rodwell. Pellegrini admitted they werensdft in a position to corpate with neighboura thited for establ' world stacs Like Angel di Maria because of the Even if they win their remaining eight Premier League fixtures their points tally will be lower than last season and they crashed out once again at the last-l6 stage the Champions Teague. Fellagrini and diractor of football Txikl Regiristain both accept the squad needs replenishing and they want to be in the market for the top available players. Manchestar city have been heavily linked with de Bruyne The club also want to be Involved in any race for Liverpool contract rebel Ralieem Sterling , They have been strongly linked with former Chelsea winger Revin de Bruyne who has been one of the best players in the Bundesliga with Wolfsburg. And if Liverponl do allow Sterling to leave, City want to be involved tha chase for one of Baglandákla most eutstarding talents to help fit with quctas aASTT you are the manager
City will
last two transter windows and
peasilies, IL has cest City on the pitch.
for former Chelsea midfialder Kavin
hemegrewn players.
player of a big tem, you are always open to criticism because everyone expests high level of performance, S&E he said, aA7The majer issue with our Seam always seems to he the monay but this season I think we were the team that apent less than any other Lean, That is the weird thing, aAfFoz this club Lrying to be big teom in so fex years, maybe we are paying the cast for thät Since the transfer penalcies were roduced, City have apent 91million on players in the last two windows ( Net spend s8millien). That compares to Kanchester United 145. Smillio,, Liverpool 113 million, Arsenal 92. Smillion and Cheisea 82, 12.mminion. Over the same time period Barcelena
spen: 118, Smillien on players and Real Madrid 81.2million though they also broke the world transfer record for Gareth Bale the previous sumner.
Zero- Shot: manuel Pellegrini hoping UEFASAta punishment for breaching tinancial fairpiay requlations will be lifted this summer. city have been restricted a net transfes apend of 43million in each window. a total of 23million has been lost in the last year.
Referances
Raferences: Manchester City have been restsicted to net transfer spend 49m .Clua's also had to keep everall wage bill to its current level of 205m .Puntahments inposed by UEFA for breaching Financial Pair Play rules The spending restzictions were set for this season and the next one Rut City are confident they will lifted early after their compliance
PPO: Mansel Fallagrini is hoping DEFASAs punishment for breaching finoncial fatrplay rulss will be lifted this summer. Manchester city have been limited to in each trensfe: window. the club also keep the club's overall wage bill to its curreat level 205millizz-a-year
spend 49million
NLPO: Manchester City are confident UEFAsAOs puniehment for breaching Linancial fairglay regulations will be lifted this summer, that would allew them to bid for stellar names like Raheem Sterling, Kevin de Bruyne and Roas Barkley.
Supervised; Manchester City have been limited to net transfer spend of 49millien in each window. Cisy's s overall wage bill is 205m-a-year due to a breaca of JEFA rules. City are confident the finarcial restrictions will be lifted the and of the current sezson in order to bid for atars such as Rateen Sterling. Manuel Fellegrini is hoping that the financial restrictions will be lifted
end the 2015/16 season
Supervised - PPO: Manchester City have been restricted to net spend of 49million in each window and kept their overall wage bill at 205m, Manuel Pellegrini believes these restrictions will be lifted at the end of the sesson. Arsenal have spent more in transfer fees than City in the last 12 months, including 30m on Alexis Sanchez.
Supervised NLPO: Manchester City are conficent UEPAaREs panishment for a breach of financisl fairplay regulations will be lifted this summer. The club have been restricted to net tronsfer spend of 49million over the pest twe window. They aim to bid for stare like Raheem Starling, Gareth Bale and Revin de Bruyne. city lost 23m traraters in the laat
12 months due to financial restrictions imposed by USFA for breach.
Sample
Prompt: (CNN) Larry Johnsen remembers the fear and feeling of heiplessness frem being on the skywest Airlines flight that made emergency landing in suffale,
york. "I thought we ware done," " he said Thuraday, one day later. *Three was no doubt in my mind that we weren't going to land," Johnson was flying with his brother, his qirifriend and his 8- month-old son when he says a flight actendant came over the speaker asking for somcone who was medically trained to help with a sick passenger, Minutes later, Johnaon saya, the atLendant announced there for the energency lending. "You're re going to feel dizzy and woozy and lightheeded, and you'
preseurization problem and told passengere to prepare
40

---

Published as a conference paper at ICLR 2023
re noz going to be able to breathe," " Johnson recalled the flight actendant saying. The Federal Aviation Administration or Wadnesday initially reparted a proscurization problem with Skyflest Flight 5622, and said it would investigate. It later issued a statement that did not refereace any pressurization issues. 'It was like being trapped and you coulda't t da anything. You have no centrol over it. Your bedy just fel: pressured, cohnson said, really
Johnson said his girlfriend was seeing spots and that "when bright. I could barely see." According to Mariasa Snow, spokeswoman for SkyKest, three passangers reported losss of consciousneas while on the flight. rifteen passengers were ned by redical personnel at the Ruffalo airport, but no one was transported to the hospical, ahe sald, The spokeswoman said that maintenance personnel found no indication of pressurization preblen with the aircreft, Embraer E170, and that the sirline continues to investigate the cause. An official with the National Transportation Safety Board told CNN that the agancy ia in communication with the FAA and Skylest to gathar information on the incident to better understand what took place.
blinking, it
Mary Conningham was the nurse on heard who attended to the first sick passenger, "there was woman, very lethargic, very rauseous, turning gray, just not looking good, " Cunningham said. After administezing exygen to the women, Cunniagham, a nurse at Yale New laven Hospizel in Connecticut, returned to her sest. Soen after, Cunninghan said, she was callad up again by the flight attendant after anothar women near the first started to lose consciousness. "It near the aiddle of the plane, she sald, Cuaninghan sald she had to zeluza Lo her seal after feeling lighthesded. There were 84 passengers crew members. Plight 5622 was originally scheduled to [ly Irom Chicago to Hartford, Connecticur. The plane descended 000 feet in three minates. "It would feel Like a roller coaster when you're re coming analyst Mary Schiavo said, deseribing how such a descent would feel, "You know that these pilors knew they were visit his parents with their only grandchild, was glad that roller coaster zide ended safely. "is was a shaky ride, it wasn't anything 1 would like to experience again, he sald. CNN's Jean Casarez, Josh Gaynor, Dand Ford and Stepasnie Gallman contriduted to this report.
something that was affecting oxygen in that area, which was
board the plane, including nine
the top and you're going down," " CMN aviation
very grave and very serious situation.* * Johnson, flying to
Zero-Shot: "you're going feel dizzy and woozy and lighthezded," Johnson says. "it was like being trapped and you couldn't do anything," Johnaon says, "the plane was trying to get back on the ground, passenger says of [ear.
References:
Three paszengers report a less of consciousness on SkyWest flight BBut officials say there is no evidence of pressurization problem.
PPO: SkyWest flight 5312 made emergency landing in Duffals, they were not going to be able to breathe, passenger recalls attendant. the faa initially reparted a ppssurizstion problem with the
york. passengers told
5622.
NLPO: Skylfest flight 3622 made emergency landing in buffalo, new york.larry Johnaon says fligh: attendant announced there was pressucization problem. three passengers reported loss of consciousness while on flight; no one transported te hospital.
Supervised: The FAA says it is invesrigating the couse of the amergancy landing. skyWest Flighs 5622 made emergency landing in Buffalo, New York. The flight was carrying family of thres people. The flight attendant told the passengers to prepare for the landing.
Supervised PPO: "I Lhought we were done," " the father of one passenger says, The flight attendent annouaced there was a pressurizatien prchlem. The flight made an emergency landing in suffale, New york. Three passengers reported a less of conseiousnesa while on the flight.
Supervised NLFO: skywest sirlines flight made emergeney landing in Buffalo, hew York, Thuraday. "There was no doubt in my mind that we weren't going to land, passenger says. "You're going to reel diszy and woozy and lightheaded.... you' not going to be able to breathe, be says."
Sample
Prompt: For Tnverneas the latest chapter in their remarkable story featured not one, but two, plot-changing twists. Une festured pansity Celtia didn't get for 2-3 in which Josh Meekinga handball should have al lad to a sending off, The other the apot kiok they did, followed by a red card for Cralg Gordon, 'I've not seen yet, but going by reaction of the Celtic players we got sway with penalty and a sending off and that was probably the turning peint in the gaze,' acknowledged caley manager John Hughes efter. Inverneas's Josh Meekings appears to get away with handball on the line in their win over Celtie , caley boss John Hughes says the break, whica could have meant a penalty and zed card, by the redia it was definitely all that and I knew Ronny will feel aggrieved because
turning point 'I've not speken to Josh. haven't zeen it but going
hand ball. Ke look at the referee behind the Line and certainly would. "But it's part and percel of foctball and you need a wee bit of luck to beat Celtic. "This was their biggast game of the seasen because they will go on and win the leagua and if they had beaten us today there was good chance they would have gone on and won the Scottish Cup. "But when Marley Malkins was clipped by Craig Gordon and they were down to 10 men that advantage laverness. 'Be veren't going to give Celtic the ball back, they had to and get it and we had to be patient, 'When big Ddward put us into the lead we thought it was going to be eur day on the back of things that had hagpened. 'Celtic equalised with another free kick but it's typical of Inverness that we don't do anything easy. 'we do it the hard way and we came up with the winner through David Raven. Hughes haulad Raven, his Scouse defender, from his backside as extra-time beckoned. ofTended the sight of one of his players resting he had sessage to impart. Caley players celebrate after upset:ing Celtic in Scottish Cup semi-finel 3-2 thriller teltic, depleted by games and absentees, were virtually on theie knees after releatless programme of midweek games. In last season's League cup Finel Laverness had been passive and unambitious prior
41

---

Published as a conference paper at ICLR 2023
to Lesing on penalties. This was no time to repeat the mistake. '1 tried to emphasise to the playars they would never have a better time to on and beat Caltic, down to 10 man in the semt final of a cup, Ne needed to qo for it," Hugnes sald. 'Before Raven scored at the back pos: was looking to change it.
I was going to bring on another wingez, Aaron Doran, and put him in the fuli-back position the right,
advanced ha could take their left back Thankfully didn's do that and havid Raven came up with the goal, Virgt1 Van Dijk (centre) fired coltic into an early lead with suparb free-kick in the 18th minute 'I didn't realise this is the first time the club have been in the final of the Scottish Cup and that's a remarkable achievement given it was only forned 20 years ago, "IL is a great story ian't it? I:'s absclutely fentastic story. It is 20 edd years since the amalgamation. Me a small provincial club up there in the Highlands, 'Ne have lost a real inspirational has never played. Ke have
akipper in Richie Foran right from the start of the season. had to adjust to that. 'ile had to sell Billy McNay, our top goalacorer, at Chrlatmas, We have had to go again and adjest. that injuries have never caught with us.' There la remarkable Irony in the ract Palkirk will be the opponents for the final. A Zormer Baiens captain, he was manager of Hampden. Former zalkick captain and manager
an a vary humble guy and an grateful and thankful
the club in 2009 when they lost Rangers John Rughes will take on his former club in the final , 'I had a lot of great times at that it 1a posatbly my favourite time my playing career.
Fa Palkirk atill friendly with an avful lot of the chazacteis who were in that dreasing coom, Nell oliver is a good friend mine from my Falkirk days. lte comes along bench and gives met some great guys. Brian Rice, my ex cosch, Davie Weir, Ian McCall, the list is endless. just talking the other day about that day ralkirk. There when I sea, 'I have a real spectal, special feel for the place. I am not kidded Falkirk beats Hearts
sits on wee hand out, "That La the spirit that had at that club, I have
times
right good side, trust me on that, No side goes to Tynecastle without being good side. No side beats fiibs the way they were playing without being a good side. reu might say we are favourites say
even stevens and 1 am just looking forward to when the days comes.' David Raven (second rigat) celebrates with his Inverness team-mates fellowing his extra-time winner An Lnverness and Falkick final would not have been the choice of apansors william Hill, you suspect. Vast swathes of Hampden's a weat stard were empty, a 12.15 kick-off for the berefit of broadcsaters proving unhelpful There will no licket rowe this Lime. crowd much higher than 35,000 in stedium of 52,000. The contract with Hampden ple is signad, with no questier of moving it. if the bitter amalgamation produced fine footbell club it also disenfranchised many of those who followed the old ones. "1 hope this enceurages the next generation to fcllow inverness,' added Hughes, 'in the peer groups it is still Celtic and Rongera they follow but hope the next want to support their local club, 'If we can add another few aundred or thousand to the support then we are playing our part.
over final unlikely to raise
years ago
will
dero-Shote lnverness beat Celtic 2-0 in their opening game of the season. Josh Beekings was the subject a handball after the break. the break could have maant a pancity and a red card for Craig Gordon.
Raferences:
Josh Neokings handball should have been a penalty and a red card .If the penalty was awardad and converted Celtie would have gone 2-0 .Caley manager John Hughes admitied his side weze forturale in win .iirgil van Dijk scozed Cellie's a opener with superb free kick .Celtic keepez Craig Gordon was sent eff early in the second half .Greg Tansay, adward ofere and finally Daven Raven scored for Caley
PPO: Josh Maekings handball could have meant sending off for Caltic in excra-time win. John Hughes says the break could have meant an penalty and red card. Caley boss also admits he would feal aggrisved had Celtic wan the championghip.
NLPO: caley beat Celtic 2-0 in their Scottish Cup claah. Josh Meekings handball could have led to a penalty and zed card. John Eughes says the break was probsbly the turning peint in the game, defender does not believe defensive striker can get away with being shown the yellow card - despite playing in sama half we never sccred.
Supervised: Tnverneas beat Caltie 2-0 in their Scottiah Cup clash. Josh Meekings appeared to
get awey with handball on the line. Another player ended in a ved card. Caley manager John Hughes says the break could have neant a penalty. Eughes says he 'would Ieel aggrieved' after the break.
sent off for spet kick which
Supervised - PPO: Caley beat Celtic 1-0 in their Scottish Cup encounzer a: the Dtihed. Josh Neekings' handball on the line should have eiso seen seading off, but Craig Gordon get away with a spot kick and serding off, John Hughes said it was a turning point in the game for Taverness,
Supervised NLPO; Josh Keekings appears to get away with handbell after Inverness's win. Inverness were down to 10 men when Craig Sordon was aent off, John Hughes says the break could have meant penalty red card. Blughs says Celtie will go on and win the Scottish Cup with win
42

---

Published as a conference paper at ICLR 2023
B.6 ToTTo
B.6.1 SETUP
ToTTo (Parikh et al. 2020 is controlled table-to-text generation task in which the goal is to produce one-sentence description of highlighted table cells. For training RL. methods, we consider different reward functions: BLEU, SacreBLEU, METEOR, PARENT and combination of Meteor and PARENT. We chose T5 as our base LM here too, as they are more suitable for structure text tasks. For decoding, sampling. Other implementation details are captured in Table 22
beam search during inference and for generating rollouts, use top k
Model Params
valoe
supervised
batch size: 8
learning
learning rate scheduler: constant with warm
weight decay: 0.1
ppolaipo
per update: 2560
sleps: 256000
size:
epochs per update:
leumning rate: 0,000002
entropy coefficient: 0.0
inirial
target
discount factor: 0.99
gae lambda: (1.96
clip ralie: 0.2
top
top mask ratior 0.9
targel update fteations: 20
supervisode ypo (or nipo)
per update:2560
steps: 256000
spochs per update:
learning rate: 0,0000005
encropy coefficient
target
discount factor: 0.99
gae lambdu: 11.6
clip ralier 0.2
top mask ritio: 0.9
target update iterations: 20
decoding
beams:
length:
max
tokenizer
padding side: left
truncation side: right
longth: 512
Table 22: ToTTO Hyperparams: Table shows a list of all hyper-parameters and their settings
B.6.2
RESULTS AND DISCUSSION
Tables 24 23
presents our benchmarking results with 5 reward functions along with supervised baseline performances on dev and test sets respectively. Similar to other tasks, our main finding that warm-started initial policies are crucial for learning to generate descriptions from highlighted cells. Without warm-start, policies suffer from reward hacking and resulting in sub-optimal solutions despite application of task-specifi metrics such PARENT etc. We find that Supervised+NLPO method outperforms all models on ToTTo leaderboard in terms PARENT metric.
43

---

Published as a conference paper at ICLR 2023
Cunsistency
Aig
Function
Overlap
Non-Ovailap
Overlap
Non-Ovesta
Overlap
Non-Ovailap
13
secrchics
0.204
TS
meteor
0.144
0.155
-0.769
-0.13
-0.826
0.356
0.361
0.381
TS
jurent
-0.753
0.335
0.399
13
metcor parzot
0.161
0.169
0.132
0.891
0.851
0.922
0.345
0.342
0.348
NLPO
13
bicu
0.062
0.065
0.089
-1,077
-1.057
-1097
0.235
0.236
0.233
13
sacrebleu
0.085
0,083
0,945
0.917
0.972
0.314
0,315
0.313
T5
metion
0.102
0.108
0.097
-1,044
-1.009
-1.079
0.320
0.328
(1330
ToTTo
T5
TS
mecioor parznt
purent
0.266
0.159
0.175
0.166
0.152
0.158
40.704
00.700
40.66
-0.655
-0.740
00.74
0.357
0,365
0.351
01362
0.363
0.368
Supervised
TS
0.457
0.535
0.377
0.204
0.327
0.081
0.583
0.631
0.534
Supervised PPO
T5
bleu
0.475
0.548
0.395
0.200
0.323
0.078
0.580
0.638
0.542
TS
sacreblen
0.474
0.557
0.389
0.209
0.340
0.077
0.573
0.620
0.525
T5
meteer
0.408
0.541
0.392
0.203
0.325
0,082
11590
11A38
(L542
T5
panont
0.499
0.547
0.388
0.175
0.300
0.050
0.595
0.641
0.549
TS
0.547
0,314
0.069
0.595
0.642
0.549
Supervisod - NLPO)
T5
Mee
0.475
0.548
0.399
0.308
0.330
D.065
0.593
0.639
0.546
TS
sacreblen
0.475
0.557
0.392
0.208
0.335
0.577
0.625
0.529
75
01594
(1.46
15
(L550)
T5
Table 23: ToTTo test evaluation: Table shows lexical, semantic and factual correctness metric scores of algorithms with different reward functions on hold-out test set. Without supervised pre-training. both PPO and NLPO results in sub-optimal solutions, with NLPO better than PPO. With supervised pre-training, PPO and NLPO achieve better scores across all metrics showing RL fine-tuning is beneficial. Most importantly, RL fine-tuned models produce more factually consistent text as seen in higher PARENT scores. Another observation, fine-tuning with task-specific metric PARENT better than training task- task-agnostic lexical rewards
44

---

Talk
Lesinl Semantic
Pactaud Cantaknne PARINT
Direrstty Masii
Published as a conference paper at ICLR 2023
TM
Rewad Bexaiee
Roaged
Rorged
Regett
RoagelSSn
BanSoon
SicoBlo
Utrape:
Glepert
11:31
0x335
TU227
1U:21
UL905
0382
10004
10318
0.109
0.116
0,,25
01186
08.238
0.715
91333
X007
20000
0:455
0.53
6/33
0.715
FPO
Hea
0.74
0.138
0.388
0.389
0.138
0.844
0068
0.611
0.066
0.211
0.353
0.403
0181
0.306
10,659
14.511
7536
08322
03397
01238
01389
01541
0.165
03335
10358
0.37
03358
02625
0.336
0.482
101199
14.90
12346
50523
21 21.486
0.268
0.123
0.231
01231
0.319
0,990
0.130
0.138
0.363
04337
0.311
0.362
0.004
0.31
0.666
11,088
14.865
13588
36613
13105
axteor punce
0.266
6.129
0.231
01251
0.330
0.385
0.355
0.175
0.355
0.345
0.350
0.01
0.381
0.564
10095
14.42
14422
55770
NLPO
Hoy
02663
0.134
0.34
0.24
0.337
0.54
0055
0.011
0.055
0.235
0.29
0.257
0.465
0.1
0.356
11,35
14
0023
47300
20,422
81311
0:884
00381
01381
0.901
00899
0.991
0188
0.114
0.317
0.09
0.111
0.313
11:00
15102
9.35
0377
T:T7
B1383
0.132
0.264
11354
01333
0.984
0.183
0.124
0.355
0.237
03824
0.225
0.991
11,853
14346
14:00
aktee JAANAA
0.29
0.14
0.276
0.276
0.142
0.856
0,171
0.181
0.161
0.360
0.365
0.572
0.770
0.214
0.674
11.072
15.255
14950
50737
13.141
Sepersted - PRO
0:381
0.315
0.325
0:51
0.41
0.85
0/61
0.75
0.167
0.55
16080
12:55
1500
50754
17
TV
candiie
0.305
0.775
0.378
0.378
0.111
0:955
0.617
0.551
0.401
0.577
0.11
0.535
0.775
0.17%
0.53
10.86
14.515
11466
53337
16 6.165
exteer
B410
0.781
0.380
0:380
03233
0:54
0.68
0.500
0/61
0.718
0.105
0.516
10037
14:00
158970
17
0446
0.261
0.366
0.367
0.220
0934
0433
0,54
0.359
08500
0/43
0.727
0.170
0.522
10.43
14.500
13330
55211
17.185
Sepcrbel
0.397
0555
0.41
0.555
0/021
0.75
14:544
BA11
0:381
0.399
0.39
0.331
0994
0651
036
8.00
0A4
0.588
0.27
6377
0.56
10136
18612
1534
17:07
0.38
0.345
0.386
01319
0334
0.69
0.541
0338
01988
6A0
0.76
0.168
0319
1N219
14.
15238
5:991
17.00
Table 24: ToTTo dev evaluation: Table shows lexical, semantic and factual correctness metric scores of algorithms with different reward functions on dev set. Without supervised pre-training, both PPO and NLPO results in sub-optimal solutions, with NLPO better than PPO. With supervised pre-training, PPO and NLPO achieve better scores across all metrics showing RL fine-tuning is beneficial. Most importantly, RL fine-tuned models produce more factually correct text as seen higher PARENT scores. Another observation, fine-tuning with task-specifie metric PARENT is better than training just task ausk-ggnottic lexical metrics

---

Published as a conference paper at ICLR 2023
Coherence
Correctness
Algorithm
Unique N
Value
Alpha
Skew
Value
Alpha
Skew
Zero Shot
25
1.63
0.718
1.642
1.93
0.503
1.966
PPO+Supervised
24
4.57
0.221
4.579
4,48
0.098
4.483
PPO
26
2.75
0.427
2.753
3.23
0.214
3.227
NLPO
28
2.25
0.401
2.247
2.61
0.419
2.613
Supervised
4.59
0.173
4.592
4.54
0.189
4.537
NLPO+Supervised
26
4.58
0.244
4.601
4.57
0.144
4.581
Table 25: Results of the human subject study showing the number of participants N, average Likert
scale value for coherence and sentiment, Krippendorf's 's alpha showing inter-annotate agreement, and Skew. For each model a total of 50 samples were drawn randomly from the test set and rated by 3 annotators each, resulting in 150 data points algorithm.
Coherence
Correctness
Group 1
Group
Diff (G2-G1)
p-values
Diff (G2-G1)
p-values
PPO
NLPO
-0.507
0.001
-0.613
0.001
PPO
NLPO+Supervised
1.827
0.001
1.340
0.001
PPO
Supervised
1.833
0.001
1.313
0.001
PPO
PPO+Supervised
1.813
0.001
1.253
0.001
PPO
Zero Shot
-11220
0.001
-1.293
0.001
NLPO
NIPO-Superrised
2.333
0.001
1.953
0.001
NLPO
Supervised
2,340
0.001
1.927
0.001
NLPO
PPO+Supervised
2.320
0.001
1,867
0.001
NLPO
Zero Shot
-0.613
0.001
+0,680
0.001
SuPoeSuper
Supervised
0.007
-0.027
0.009
Supee-Sueeevised
PPO+Supervised
-0.013
0.009
-0.087
0.009
NuPoeSuuee
Zero Shot
-2.947
0.001
-2,633
0.001
Supervised
PPO+Supervised
-0.020
0.009
-0,060
0.009
Supervised
Zero Shot
-2.953
0.001
2..07
0.001
PPO+Supervised
Zero Shot
-2,933
0.001
-2.47
0.001
Table 26: Results of an post-hoc Tukey HSD Test for difference in means between pairs of algorithms (Group Group 1) and corresponding p-values. Individually statistically significant results are bolded and used to discuss results in the analysis. Overall p-values showing that there is significant difference in means between the models via a one-way ANOVA test are significant with p 0.05 for both coherence and sentiment.
46

---

Published as a conference paper at ICLR 2023
yrick eapanc)
Thane you your partitppatan
skndiar
Ploure bac mosoort pansoeeree yearing my with the row roading the vnchuctereewnn iss becsune sigg feve changes se Thames agai for world
in thes sit you be presersed with a rabe fhom oelreccie, with the cnt highiighoed
in pelloer songgeet the totlic you will be pro awiddd some mesacsss tue the urte the wi pertis artiescsrtion aosent the sonke you well tee gi pee sy ttive groeestine, rate generation
which captur chumnunnee decact hour
Mlcrncyroraeen diality a syteers genestoon gronccannc escp reacs and Auen:
Cerrecteennieeeteeee dwo the prwers cumontt abouthoc cast boce taats
that saw sateine
You'se be able to ros sach of the three sues on soole from to with lioing the tewarhh und regle sp vsalis
Example 2: Table from Wirpedia:
(eacelionty: The geverition is grammmttiol of and fivent
Gerard Piqué
6r% ggoods senterie orgey makeesennne, tut theoe come
arerar to plase words make the stany poccgt gi dlopp ind possibic
Sectio trternlindd
but ritely
Table Textion Toni fooe
Welu
Cap Oupomnnn
Scare Resul Dompetten
gasrr though
meannng
ntemiilgg peremation severe gpanentticang carppicasly unroadabis.
Tuakey
1-0
3-0
Saam
Conrectiess Speettlttt
desentica intermation conveyes the tra aat colb.
sightighee enarey
2000
Muselina
3-2
Fiieddy
proodyycacrattk
moull disuttoc the nutien
3/5 preutrad The genenatan someunant pi wusto enteennt but
the highlgh groad net with only smail deciation.
Suparber
Pacce
Delgkan
5-0
spec to the right artted oara corruct as coulld be.
broddyy rntresanalle vee why couid tie gevwe sted given
Bis Poll
Weald C.o
sede dosurt make much verne. gencraton cocore seor opplyyo
Dullbes
Fomta and
Mi 13
Republic
Notex
cleeeeeeecaauuuuuee diett aocy aloat
grominaccal
System's generation raate thist:
mewroneeoouuggooo menaerr roch muuerterss
me
genworron
better tru fugttd
Gensstaaateenille buntbuuuuua shit & ant anneroeeee anstebarake Lile pentsss suld deductee "ypcufaag" ofg l'drrr wiire taaicadi that referenced more direety.
gosl peroence graen note though
right
Alhandtua of cabkis are quite lirget Sul apply the some rat crteria, evenif the txbirs many rowe.
Corvecteeasppeeiiiiii lhyhh might conrect, facuk studium wheree the playar soonad, not fighted..
Examp le 3:
Table from Wiupeeda::
1899 San Diego mayaral election
for Wipelt
Dunocrat::: DovenM Caps
1.22
52.3
Repinlion
Reed
Sociataa
2011 Hu ingginn
Coherence/Quality: 3/5
Total vitee
3,77
100
System's generation (rate thidk:
Correctness/peccfiig 3/5
Danak G. Reedii a canditee e5. percect
daction
Callortt 45d Daniel Seed 1964 mayonal electinn
flarnayCrammmn cntin why pow bee to ri the grn miienmeen things zus senteras.
Gereederoe/peaffiii Why the ferue doos so oudy the reghnggtin cornect pllon
mestiesity
coperig g
Figure 9: Instructions, two examples, and interface for the ToTTo table description task.
B.6.3 HUMAN PARTICIPANT STUDY
Figure 9 shows the ToTTo instructions, example, and interface used for the human evaluation experiments. We made small modifications to the original code release's HTML renderer to make the tables display in our HITs. Tables 25 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans.
47

---

Published as a conference paper at ICLR 2023
B.6.4 QUALITATIVE ANALYSIS
We show sample generations from each of the algorithms for three randomly picked prompts below.
Sample
Prompt:
<table> <cell> August 1, 1392 ccol_hezder> Date header> healere North
Americs <col_header> North America </ooL_header> </cell> <cell> Allentown scol_header>
<co1 helddad Australasia </col_header> <eal headerr Asia </
col_headers <col
City c/col_heddrr <gol_itader>> North America c/col_hesder> <eol_beader> Australasia hol_hheder>> North America //col_header </ call> <eell> Allentown Foirgrounds Grandstand <eal_headers Country s/col_hezders
col hezdeer scol_header) Asia //cco_/heddes
col_besder> North America </eol_header> <col_header> Australasia </col_header> col holdherd Asia
headers <col, <col_ hezder> North America </col_ headar>> </call> </table><
page_sitlas Under Spell Tour [page_titles <section_title> scol_hesder> North America <Lable> <cell> August 1, 1992 <col_hesders Date </col_header>
datea </section bitlee
</col_header> <gel_ieader>> Austrelasis </col_hesder> <col_header> Asis heeddee col hezder> North America
</cell> <cell> Allentown <col_ city </ col header> <eol scol_header> Narth America </col_ _ooo__hader>> <col_header> Australasta </
col_hezder> scol_header> Asis </col_hezder> <col_header> North America </coe_h </ cell> <cell> Al lentoon Fairgroands Grandstand <cal,
<col_header> header>> Country </col_headee header>> <
America "ooolheedders <ool_header> Australaaia </col_header> col_beeder> Asia //col_heddes <col_hesder> North America </col_header> </cell> </table>
col_beader>
Zero- Shot: table> cell> Under My Spell Tour /sestion_titlss rour dates /cell> ceil> august 1,
1992 hol_header> nate /col_ hol__heddr>>
ppo: Under My Spell Tour, August 1, 1992. August 1, 1992.
NLPO: Under My Spell Tour dates Under My spell Tour dates Allentow Fairgrounds Grandstand August 1, 1992. Undar My Spel1 Tour dates Under My Spell I Tour dates Allentown Fal rgrounda Grandstand August 1, 1992 table>
Supervised: The Undes My Spell Tour began on August 1, 1992 at the Allentown Faizgrounds Grandstand in Allentown.
supervised - PPO: On August 1, 1992, Under My Spall Tour was hald at the Allentown Fa' rgrounds Grandstand in Allentown.
Supervised - NLPO: August 1, 1992, Undes My Spell Tour was held the Allentewn Fairgrounda Grandatand in Allentown.
Sample
Prompt: <table> <eall> Philip the fair, the Iron king (Philippe) <eal <eol_header> header> Nome </ col_header> </eell> <cell> 5 October 1285 seel _header King from </col_headers _header> </cell>
cell> King of France and of Navarze (Roi de France et de Navarzel ceol_hester> Title </ List of French monarcas </page_title>
col_beader> </cell> </table><page_titiee
<table> <cell> Philip IV the Fair, the Izen King (Philippe) <col_hesder> Name //col_heades" </cell> <cell> October king from </col_ "ccol_hander> </cell> <eell> king of France and of Nevarre Rot de France at de Navarra) <eal_header> Title </col_headers </call> </table> Zero Zero-Shot- Shot: > </table><page__ittl tittee Liat of Trench monarcha
section_tilles House of Capet (9878491528) s/section_tiiie
1285 <col_headerr
ppo: Philip IV the Fair, the Iron King (Philippe) 5 October 1285. Ti:le: King of Frence and of Navarze (802 de brance de Navarre
NLPO: King (Philippe) Philip IV the Pair, the Iree King (Philigpe) 5 October 1285
Supervised: Philip the Fair, the ison king (Philippe) was king of France Navazre from 1285.
Supervised PPO: Up TV the Fair, the Tron king was King of France and Navarre from 1285.
supervised NLPO: Philip IV the Fair, the Iron King (Philippe) was King of France and Navarre (Roi de Prence et de Naverre) in 1285.
Sample
Prompt: <tables <eell> genjamin A. Smith 11 (D) <col_headar vacster s/col_heades </cell>
cell> Successor elected November 6, 1962. <eel _ee_daad Reasen for change s/col_headan </ cell> <eell> Ted Kennedy (D) <col_header> Succeasor </coL_header> </eell> <eells November
7, 1962 hesdere Date of successor's formal instellatien s/col_hesders </cell> </ 87th United States Congress </page_title> <section_titier Senste </ sestion_title> <table> <eell> Banjamin A. Smith II (D) <ool_hoader> Vanater </col_hoader> </cell> <cell> Successor elected November 6, 1962. <cel_teader Reaser for change </ col header> </eell> <eell> Ted Kennedy (D) <col_header> Successor </ade_h </call> cell> November 7, 1962 hoo__reder>> nate of successor's formal instal lation </ool_has hhader>>
=eble><<pge__titt
</cell> </Lable>
Zero- Zero-Shot: cell> Ted Kennedy (D) cel_header> Vacator /col_hesdes /cell> cell> Nevember 7, 1962 col coi_hexders Reason for change
PPO: Tad Kennedy
auccseding Navember 6, 1962 as Vacatar of the 87th United states Congress.
NLPO: red Kannedy (D) successer eleated Novamber 6, 1962. tablas cell Renjamin Smith II (D)
col_header Reaaon for change: Date of successor's formal installation November 7, 1962.
Supervised: Benjamin A. Smith II (D) served as senate until November 6, 1962 which was later served led Kennedy (D) from November 7, 1962.
supervised - PPO: Renjamin A. Smith II (D) served until November 6, 1962 and rod Kennady (B)
aucceeded from November 7, 1962.
Supervised NLPO; Denjamin a. Smith II (D) served until November 6, 1962 and Ted Kennedy (D)
acoceeded from November 3, 1962.
48

---

Published as a conference paper at ICLR 2023
B.7
NARRATIVE QA
B.7.1 SETUP
NarrativeQA (Kočiský et al. 2018) deals with task of generating answers to questions about a given story. For training RL methods, we consider traditional lexical rewards namely Rouge Combined and Rouge-L-Max. We chose T3-base as the base LM since it has been shown to do well at question answering in prior work (Khashabi et al. 2020). We note that the supervised models we use are trained on the UnifiedQA dataset, which contains other QA datasets, and is shown by Khashabi et al. (2020) to outperform supervised fine-tuning only on NarrativeQA. Hyperparams for our models can be found in Table 27
Model Params
value
ppo/nipo
steps per update: 5120
total number of steps: 512000
batch size: 64
epochs per update: 5
learning rate: 0.000002
entropy coefficient: 0.0
initial kl coeffi 0.001
target kl: 1.0
discount factor:
gue lambda: 0.95
clip ratio: 0.2
rollouts top k 50
value function cocff: 0.5
top
target update iterations: 20
supervised+ ppo nipo)
steps per update:2560
total number of steps: 512000
batch size: 64
epochs per update:
learning rate: 0.0000005
entropy coefficient: 0.0
initial kl coeff: 0.001
target 0.2
discount factor:
gue lambda: 0.95
clip ratio: 0.2
rollouts top k 50
value function coeff: 0.5
top ratio:
target update iterations: 20
decoding
beams:
max new tokens: 50
tokenizer
padding side: left
truncation side: right max length: 512
Table 27: NarQA Hyperparams: Table shows a list of all hyper-parameters and their settings
49

---

Tasks
Lesical and Semantlc Metrics
Diversity Metties
Published as conference paper at ICLR 2023
Alg
Rewand Function
LM
Rouge-1
Rougo=2
Roggs1.
Rouge--Sm
Ronge-LMax
Mitice
BLEU
BarScore
MSTTR
Disinet,
Disinnte
Hi
Hs
Uniquey
Uniquey
Mean Chequ Length
7em
T5
0.022
0.084
0.117
0.095
0.000
0.066
0.077
13,468
1830
Rouge
NarQA
Ronge-L
0.01
0.439
0/29
13618
31,479
Range
Ronge-1,
Supersisel
0.581
15657
3250
4,923
Supervised PPO
Combined
0.177
0.371
0,571
0.385
0.09
0.129
0.174
0.559
10.132
13,87
3326
4,353
Ronge Max
T5
0.363
0.18
0.36
0.385
0.833
0.299
0.187
0.576
10.201
13,42
3
Superviand NLPO
Rougs Cambined
T5
0.398
0.21
0.393
0.373
0.589
0.996
0.34
0.971
0.679
0.18
0.595
10.304
13,94
3371
15067
4,738
Ronge-L, Mas
13
0.381
0.194
0.383
0.383
0.388
0.093
0.243
0.932
0.645
0.177
0.59
10.2
13.397
3287
13171
3,80
Table 28: Evaluation of NarrativeQA: Reference Metries, supervised is based on UnifiedQA (Khashabi al. 2020

---

Published as a conference paper at ICLR 2023
B.7.2 RESULTS AND DISCUSSION
Table 28 presents our benchmarking results with 2 reward functions along with supervised baseline performances on the NarrativeQA test set. Similar to other methods, our main finding is that warm- started initial policies are crucial for learning to generate answers that successfully use the input context.
B.7.3 QUALITATIVE RESULTS
We show sample generations from each of the algorithms for three randomly picked prompts below.
Sample
Prompt: who is mark hunter? mark hunter (slater), a high school student sleapy subura of phoenix, arizona, starts an te pirate radto statlen that broadcasta from the basement of his parenta' house, mark is loner, an outsider, whose only outlet for his Leenage angst and aggression is his unauthorized radio station. his pirste station's theme song is " everybody knows" by leonard cohen and there are glimpses of cassettes by such alternative musicians as the jesus and mazy chein, camper van beethoven, primel scream, soundgarden, ice-t, bad brains, concrete blande, henry zellins, and the pixies. by day, mark is seen as loner, hardly talking to angone around him; by night, he expresses his cutsider views about what is wrong with american society, when he speaks his nind about what is going on at his school and in the community, more and more ef his fellow students tuse in "happy harry hard-en
hear his show. nobody knews the true identity of "hard harry" as mark refers to himself, until nora diniro Imathis), a fellew student, tracks him down and confroats him the day after student named malcolm commits suicide after harzy attempta to reason with him. the radio show becomes increasingly popular and influential after harry confrerts the sufcide head-on, exhorting his Hateners to do something about their problems instead surrendering to them through suicideachdMlat the crescendo of his yolled speach, on everschieving student named paige woadward (who has been a constant listener) fams her various medals and accoladea into microwave and turns it on, ahe then sits, watching the awards cook until the nicromave explodes, injuring her. while this is happening, other students act out in cathartic release.
ly, the radio show causes so much Lrouble in community that the Ice is called in Lo inveetigate. during the fracas, it is revealed that the school's s principal (annie rees) has been expelling problem students, namely, students with belcx-average standardized test scores, in an effort to boest the district/s test scores whi still keeping their names on the rolls ( criminal offease) in order to retain government something huge, mark decides it and actaches it to his mother's g old jeep, creating ing mobile tranamitter so his position can't be triangulated. pursued by the police and the fee, nera drives the jeep arcund while mark broadcasts. the harmanizer he uses to disquise his voice breaks, and with no time left to fix it, mark decidas to broadesst his final massage as himself, they finally drive up to the crowd of protesting ing atudenta, and mark tells them that the world belongs to then, and that they should make their own future. the police step in and acrest mark and
he has started to him to end it. he dismantles his redio station
taken avay, mark reminds the students to "telk hard." as the film ends, the voices of other students (and even one of the teschers) speak as intros for their own independent statiens, which can be heard broadcesting across the country.
they
Zero-Shot: hunter (slater) is a high school student in sleepy suburb of phoenix, arizons. he starss an fr pirate radio stattor that broadcasta from the basement hts
References: he is high school student in phoenix 2) loner and outsider atucent with radio station
PPO: hunter (slater), high school student in sleepy suburb of phoenix, arizona, starts fm pirste radio station that broadcasts from the basement of his parants' house
NLPO: hunter (slater), high schoel student a sleepy suburb phoenix, arizons, starts an fn pirata radio staticr that breadcasta from the basement of his parents' house Supervised: a high achool atudent Supervised ppO: high schoel student Supervised NLPO: a high scheol student
Sample
Prompt: what was the principle doing with the problem students? mark hunter (slater), high achool student in a sleepy auburb of pheenix, ari starts an fn pirate radio station shat broadcasts from the basement of his parenta' house, mark is loner, an outsider, whose only outlet for his teenage anget and aggression is his inauthorized redio station. his pirate statien's s theme song is "everybody knews" by lecnard cohen and there are the jeaus and mary chain, camper
glimpses of cassettes by such alternative musicians van beethoven, primal acream, soundgarden, icert, bad braina, concrete blonde, henry rollins, and the pixies. by day, mark seen as a loner, hardly talking to anyone around him; by night, he expreases his outsider views sbout what is wrong with american socfety, when he speaks his mind about what ls going on at his school and in the community, more and more of his fellow students tune in hear his shew.asbos
knows the true identity of "hard harry" or "asppy harry hard-on," " as mark refers to himself, until nora diniro mathfa), follow studert, tracka him down and confronts hil the day after a student naned malcoln commits suicide after hazry attempts to reason with hin, the radio show becomes increasingly popular and influential after harry confronts the dutcide head-on, exhorting his Listenera to do something about their problema instead of surrendering to them through suicideÅāMfat the crescando of his yelled speech, an overachieving student naree pa1ge woodward jwho has beer a constant listener) jama her vart oua medals and
51

---

Published as a conference paper at ICLR 2023
accolades into a microwave turns on. then sits, watching the awards cock until the microwave explodes, ingering her, whi this is happening, other students act out in cathartic release,
the radio show causes se mich trouble in the community that the fee is called in to investigate. during the fraces, it is revealed that the school's principel (annie ress) has been expelling "prcblen students, " namely, students with below-sverage standardized test sceras, in offort boast the district's tast acorea we still keeping their rames on the rolls (a criminal offense) in order to up
retain government funding.reslizinng he has started something hage, mark dacides to him to end it. he les his radio station and attachea it to his mother's old jeep, creating a mobile transmilier so his pesition can't be telanguiated. pursued by the police and the fee, nera drives the jeep around while mark broadcasts. the harmonizer he uses to disguise his voice breaks, and with no time left to fix it, mark decides to breadesst his final message aa himself. they finally drive up to the crowd of proteating atudents, and mark tells them that the world belonga to them and that they should make their own future. the poline step in and arrest mark and nora. as thay are taken away, mark reminds the studenta to *talk hard," " as the film enda, the voicea of other students (and even one of the Leachers) speak as intres for their own iadependent stations, which heard breadcasting across the country.
Zaro-Shot:
mark hunter Islater) starts an fn pirate radic stction that broadcasta from the ia as loner, an outaider,
basement his parenta' house. by day, outle.
only
References: 1) expelling them. expeiling the students
PPO: mark hunter (slater) starta Im pirate radio station that broadcasus from the basement of his parents' house. mark is lener, an outsider, whose only cutlet for his teenage ang
NIPO: mark hunter (slater) starts an fn pirate radio station that breadeasta from the basement parents' house, mark loner, an cutaider, whose only outlet his teenage
ang
Supervised: he was taking them to psychistric hospital.
Supervised PPO: addressing them
Supervised NLPO: he was expelling them
Sample
Prompt: what happens the people maskull neets in his travels various lands? maskill, man longing for adveatures, accepts an invitation from krag, an acquaintance of his friend nightspore, to travel to tormance after a seance. the three set off a crystal ship from abandoned observatory in scotland but maskull awakens to find himself alone on tormance. in every land he passes through he usually zeets only one or two persons; thess meetings often Ithough not always) and in the death of thase he meets, either his own hand or by that of another. he learna of his own impending death, meets krag again, dies shortly after learning that he is in fact nightspore himself. concludes with a final revelation from krag (who claims to be knewn on earth as "pain*) so nightsspore about the origin af tha universe. the author turns out support veriation of the doctrine of the demiurge, scmewhat similar to that propounded by allstics.all of the characters and landa are types used to convey the zuthor's critique of several losephhilall syatems. tormance, most such viewpoints accompanied by correspording new bodily sease organs or modifications of the same, thus each distinct weltanscheuang landscape
book
ways of life
its correspending sensorium.
Zero-Shot: whet happens to the people maskull meets in his travels to various lands? n maskull, a man longing for advertures, sccepts an invitation from krag to travel to tormance after
Raferences: 1) they uaually die by his hand or anothers. 21 they dia PPO: what happens to the people maskull meets in his travels to various lands? n maakull, man longing for adveatures, accepts an invitation from krag to travel to Lormance after naskuil meets krag, an acquaintance
nightspore, to travel to termance after a seance. maskull awakens to find himself alone on tormance. in every land Supervised; they die either from his own hand or from another person's death. Supervised PPO: they end up dying either at his hand by another's. Supervised NLPO: they end up dying ef ther at h own hand or by another's
NLPO:
52

---

Published as a conference paper at ICLR 2023
B.8 MACHINE TRANSLATION
B.8.1 SETUP
WMT-16 We pick two languages, English and German, and frame this task similarly to other machine translation tasks tequi-requiring the models translate from English German. We train models on rewards: SacreBLEU, chRF, TER, and BertScore.
Model Params
value
supervised
batch size: 61
epochs:
learning rate: 0.00001
learning rate scheduler: constant
weight decuy: 0.1
ppo/alpo
steps update: 5120
total number of steps: 256000
64
epochs per update: 5
learning rute: 0. 0.0.000001
entropy coefficient: 0.0
initial kl coeff: 0.001
target kl: 0.2
discount factor: 0.99
lambda: 0.95
clip ratio: 0,2
rollouts top 10
value fuunctoon coeff: 0.5
top mask catio: 0.5
target update iterations:
supervised+ ppo nipo)
steps per update:2560
total number of steps: 256000
batch size: 61
epochs per updute:
learning 0.0000005
eutropy coefficient: 0.0
initial kl coeff: 0.001
kl: 0.2 discount factor: 0.99
gae lambda: 0.95
ratio:
rollouts top k: 10
value function coeff: 0.5
top mask ratio: 0.5
target update iterations: 20
decoding
num beams: 4
length penalty:
tokens: 128
lokenizer
padding side: left
truncation side: right max length: 128
Table 29: NMT Hyperparams: Table shows a list all hyper-parameters and their settings
B.8.2 RESULTS AND DISCUSSION
Tables 30 31
presents our benchmarking results with 4 reward functions along with supervised baseline performances on test set. Our main finding is that NLPO + Supervised performs better than PPO and supervised models.
53

---

Published as a conference paper at ICLR 2023
Danasets
Leskcal and Semantic Metrics
Alg
LM
Reward Function
Rouge-1
Rouge
Reuge-L.
BSmm
Metere
BLEEU
SocrebLEUU
chiff
TER
Benticore
Zero Shot
0,855
0,414
0,591
0.591
0,485
0.294
0.348
0.613
0,543
0,822
SacceBLBU
0.905
0.904
BenSaure
0.993
0.995
NLPO
T5
SacrcHi BU
6635
0.415
0,502
0.902
0.484
0.207
0.352
0.615
0.542
0.882
T3
chRF
0.34
0.413
0.59
0.99
0.481
0.201
0.345
0.612
0,540
0.882
T3
TER
0.633
0.412
0.59
0.99
0,477
0.386
0.341
0.638
0,540
0.881
WMT16
BertSoore
0.622
0.397
0.58
0.581
0.458
0.369
0.323
0.591
0.545
0.276
Supervisnl
T3
0.35
0.411
0,500
0.500
0.482
0.204
0.350
0.617
0.540
0.882
Superisaa
0.596
0,350
NLPO
StcreBLLUU
0.419
chil
0,412
0.592
0.52
0.349
TER
0,414
0.594
0.598
0.293
0.349
0,531
BertSeore
0.64
0.417
0.998
0.598
0.099
0.287
0.349
0.62
0.538
0.557
Zaros
0.386
0.308
0.575
SacreBLEU
0.557
0.246
0.501
0.901
0300
BenSamn
0.507
0.907
NUNO
TS
Sacrchi RI
0.64
0.385
0.59
0.99
0.45
0.345
0359
0.57%
0.576
0.87
TS
dhRF
0.34
0.386
0.59
0.90
0.411
0.348
(301
0.581
0.576
0.87
IWSLT2017
T5
BentScor
TER
0,22
0611
0,384
0,377
0.19
0.38
0.59
0.58
0,545
0.425
0.346
0.239
0301
0.291
0.513
0.555
0,573
0.57
0.815
0.866
Supervisel
0.63
0,400
0,610
0.009
0.461
0,280
0.337
0.593
0,538
0.73
Superissd
NLPO
StzreBLBU chRF
0.614
0.644
0.643
0.415
0.631
0.611
0.464
0.291
0.345
0.506
0.539
0.572
TER
0,690
0.419
0.621
0.621
0.471
0.289
0.346
0.593
0,535
0.37
BenSSaun
0.633
0.401
0.606
0.606
0.448
0.367
0.323
0.550
0.537
0.375
Table 30: WM and IWSLT test evaluation lexical and semantic: Table shows lexical, semantic metrics for RL algorithms with different reward functions bench-marked marked against supervised baseline models
54

---

Published as a conference paper at ICLR 2023
Tasks
Diversity Metrics
Reward Function
MSTTR
Distinet:
Ha
Unique:
Uniquez
Longth
Zero-Shot
0,732
0.193
0.675
10.100
14.561
7250
33691
20.533
PPO
T3
Suv.BLB
0.738
0.196
0.687
10.165
14.613
7503
34140
20.375
13
cMRF
0.738
0.196
01687
11.155
14.611
7376
34116
20.337
TER
0.736
0.196
0.683
10.132
14.585
7447
33077
33.356
BereSoon
0.736
0.195
0.685
10.129
14.574
7272
33477
20.035
NLPO
SuveBLEU
0.725
0.193
3.68
10.125
14.592
7595
34276
20.672
T3
chRF
0.728
0.196
0.686
10.164
14,005
7399
34055
20.351
W
BertSooee
TER
0.739
0.74
0.2
0.2
0.694
0.698
10.194
10.204
14.605
14.63
7522
7200
33169
34234
20.15
19.482
Supervised
T3
0.729
0.190
0.669
10.048
14.530
7208
33430
20.622
Supervied PN
T5
SacroRI. RI:
0.232
0.191
0.674
10.000
14.582
2222
33723
20 605
chRF
0.735
0.192
0.677
10.03
14.569
2310
39923
20 586
TER
0.732
0.192
0.676
10070
14.553
7265
33635
20
T5
BertScore
0,232
0.192
0.677
10.022
14,550
7157
33385
20.305
Supervisal NLPO
SacvoBl.Bt Hl:
0.734
0.191
0.675
10,89
14.568
7308
33941
20.696
cliFF
0.735
0.194
01881
10.112
14:571
7372
33814
20.345
TER
0.737
0.194
01882
10.105
14.566
7243
33482
20.150
T5
BereSoom
0.737
0.227
0.742
10.022
14.179
5438
22574
12.63
Zeno Shot
0,662
0.097
0.4200
9.276
14.525
8312
52947
18.739
StcreRi
0.096
BertSoore
0.873
0.120
9.288
14.388
6642
37267
IWSLT2017
RereSooe
0.667
14.576
Supervised
14.492
Supervised
SacreBt.BU
14.529
0.665
14.524
7895
16.053
NLPO
SucreBLBU
0.994
9.182
14.469
51797
18.777
chRF
14.503
Table 31: WMT-16 and IWSLT test evaluation - diversity metrics Table shows diversity metrics for RL algorithms with different reward functions bench-marked against supervised baseline models
55

---

Published as a conference paper at ICLR 2023
B.8.3 QUALITATIVE RESULTS
We show sample generations from each of the algorithms for three randomly picked prompts from IWSLT below.
Sample
Prompt: translate unglish to German: Severa: years ago here TeD, Peter Skillman introduced design challange called the morshmal low challenge.
Shot: Vor einigen Jahren hat
Sxi_lman hier bei TED eine besignnieraasttoodeerrn namens Karshmallow Challenge vorgestallt.
PPO: Vor einigen Jahren, her bef TED, stellte Peter skinlman einen Deaign-Nettbewers namens Die Marshmallow Ieraassorderingg"
NLPO: Vor einigen Cahren hier bei TED fiijhrte Peter Skillman eine besiggellrrssssoodeeruu namena Karshnallow thallenge ein.
Supervised: Vor sinigen Jahren hier bet TRO rAijfchrta Peter 3killman eine nesign fierauatorde:ung nameas Marshmallow Challenge ein.
Supervised - PBO: Ver einigen Jahren hier bel TED, fÅijhrte Peter smin eine nesign- Perauaterde:ums namens Resemmelloo--Rrrrsoooooooo
ein.
Supervised NLPO; Vor einigen Jehren hier bei TED fAi Peter Sxillman eine Design serausforderung ein, die saremmmloo-terrausfeddeeee
Sample
Prompt: translate Eaglish to German: They spend majority their Lime assembling the itkaka into ever-growing strectures.
Zero Shot: Sie verbringen den grădASten Teil threr Zeit damit, die Stäücke In immer wachsende
Strukturen mentieren.
References: Sie investieren einen CroASteil ihrer seit in den Rusemmenbau adig wachsender Strukturan
PPO: Sie verbringen den GroASteil ihrer Zelt damit, die StÄücke in inmer g=Adāser werdende Strukcuren zu montieren.
NLPO: Sie verbringen den GroAsteil Ihrer selt, die Stäüeke in inmer wachsende Strukturen montieren.
Supervised: Sie verbringen den GrokSteil ihrer weit damit, die ståücke in immer wachsende struksuren zu montieren.
Supervised
Sie verbringen den GroAsteil ihrer Seit damit, die StåöoKe in immer wachsende Strukturen su montieren.
Supervised NLPO: verbringea GroASteil ihrer Zeit damit, Stäücke wachsende strukturen zu montieran,
Sample
Prompt: translate Rngl ish to Carman: And of course there are teams that have a lot more "ta-da recent graduates Kindergarten.
structures, and among
Zero Shot: Und natAijtlich gibt es Teanz, die viel mehr "ta-da" Strukturen haben, und zu den kindergartens.
besten gehädren fhifagste absolventen
References: Und natăi
natăirrlich jrlich gibt as auch Teams die viel mahr "ta-da"-Gabilo" Ide haben und unter den Besten sind frische Absclventen des kindergartens.
PPO: Und nathijelich gibt
dia viel mehr "ta-da" Strukturen, und unter den basten alnd jå1 ingste venten dea Kindergartens.
NLPO: natAi jrlich gibt Teans,
mehiz
a-da* Strukturen haben, und unter den besten sind jhijagste Absolvenzen des Kindergartens natiijrlich gibt es Teams,
Supervised:
mehr "ca-da" "ca-da"-Struke" Strukkurrn haben, und unter den besten sind jaggneste Absolverten des Kindergartens.
Supervised - PPO: Und natAijrlich gibt es Teams, die viel mehr "ta-da"-Strukturen haber, und unter den besten aind jngste shaolventer
Kindergartens.
Supervised NLPO; Und fllen gibt es Teama, die viel mehr "ta-da"-Struktun haber, und unter den
sind jåi ingste Absolventes Kindergartens.
56

---

Published as a conference paper at ICLR 2023
B.9 DAILY DIALOG
B.9.1 SETUP
We consider DailyDialog (Li et al. 2017) as the test bed for the dialogue generation task. The dataset includes conversations written by human on various topies. In addition, each utterance contains labels of intent and emotional information. For simplicity, we focus only generating the next utterance, given the dialogue context. We chose context window of size 5, which results 35k training, 3k and 3k utterances. The input to the model is dialogue history in which utterances concatenated using a <EOU> token. We picked GPT-2 as the LM as they are more suited for text continuation than encoder-decoder LMs. For a fair comparison, method for all methods. For RL methods, we use a linear combination of meteor score and intent match score (whether the generated text's intent matches with the reference's 's intent) the reward function. The coefficients for meteor and intent are chosen based on both lexical scores and intent accuracy on the validation set. For this purpose, trained an intent classifier (fine-tuned RoBERTa (Liu et al. 2019)) that classifies given text into intent categories such as inform, question, directive and commisive, etc. Table 32 provides a summary of hyperparameters and implementation details.
top-k sampling with A 20 the decoding
Model Params
value
ppo/nipo
sleps per update: 1280
total number of steps: 128000
batch size: 64
epochs per update:
learning rate: 0.000001
entropy coefficient: 0.0
initial kl coeff: 0.2
target 0.5
discount factor:
gue lambda: 0.95
clip ratio: 0.2
rollouts top k 20
value function coeff: 0.5
meteor coeff: 0.25
intent coeff: 0.75
top mask ratio: 0.9
target update iterations:
supervised+ ppu (ur nipo)
steps per update: 1280
total number steps: 64000
batch size:
epochs per update: 5
learning rate: 0.000001
entropy coefficient: 0.0
initial coeff: 0.2
target kl: 0.5
discount factor: 0.99
gue lambda: 0.95
clip ratio: 0.2
rollouts top k 20
value function coeff: 0.5
meteor coeff: 0.5 0.25
intent coeff: 0.75
top ratio:
target update iterations: 20
decoding
top 20
min length: 2
max new tokens: 50
tokenizer
padding side: left
truncation side: right
max length:
Table 32: DailyDialog Hyperparams: Table shows a list of hyper-parameter and their settings
57

---

Published as a conference paper at ICLR 2023
Alp
Rouge-1
Mouge-2
Ronge-1.
Konge-L.Sim
SacreftLBBU
Accumcy
Uniques
Uniqee
Ourput Length
Zero Shet
GP1-2
0.131
0,131
0.854
0.427
0,005
01065
0,516
7.787
11.81
12527
18,685
Dialey
Supervised
GPT-2
0,102
0,0.0
0,138
0,788
0,186
0.064
0.855
0.437
0,635
0,085
0.542
8,011
12.13M
1925
13992
18.999
PPO
Meteoe e Insent
GPT2
0.168
0.012
0.142
0,342
0.221
0,83
0.851
0,474
0.591
0.058
0.310
7.633
11,437
1719
12156
18,538
GPT:3
11,381
11946
PPO
Superisent NEPO
Meteoe Intent
GPT2
0.171
0.200
0.146
0.146
0.205
0.74
0.88
0.444
0.64
0.000
0.549
K044
11.900
2061
14213
18 16.63
Table 33: Evaluation of Daily Dialog: Table shows lexical, semantic metrics for RI. algorithms bench-marked against supervised baseline models

---

Published as a conference paper at ICLR 2023
Coherence
Quality Alpha
Algorithm
Unique N
Value
Alpha
Skew
Value
Skew
Zeroshot
31
3.84
0.225
4.181
3.2
0.125
3.352
NLPO
30
4.18
0.114
4.17
3.35
0.159
3.318
PPO
32
4.18
0.112
4.032
3.32
0.163
3,478
PPP
3.99
0.148
4.133
3,48
0.166
3.58
Supervised+NLPP
31
4.13
0.186
3.953
3.58
0.178
3.597
Supervised
31
3.96
0.249
3.834
3.59
0.236
3.196
Table 34: Results of the human subject study showing the number of participants N, average Likert scale value for coherence and sentiment, Krippendorf's alpha showing inter-annotator agreement, and Skew. For each model total of 100 samples were drawn randomly from the test set and rated by 3 annotators each, each resulting in 300 data points per algorithm.
B.9.2 RESULTS AND DISCUSSION
Table 33
presents our benchmarking results of RI. methods along with supervised baseline perfor- mances on test sets. Our main finding is that RL methods generally achieve better intent accuracy and automatic metric scores, in particular NLPO variants perform better than all other methods.
B.9.3 HUMAN PARTICIPANT STUDY
Figure 10 shows the Daily Dialogue instructions and interface used for the human evaluation ex- periments. Tables 34 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans.
Coherence
Quality
Group 1
Group 2
Diff (G2-G1)
p-values
Diff (G2-G1)
p-values
NLPO
PPO
-0.003
0.900
-0.030
0.900
NLPO
Supervised
-0,227
0,043
0.238
0.020
NLPO
Supervise++NLPO
-0.050
0.900
0.234
0.022
NLPO
Supervised-PPO
-0.194
0.013
0.127
0.803
NLPO
Zero Shot
-0.345
0.001
-0.154
0.655
PPO
Supervised
-0.224
0.049
0.268
0.010
PPO
Supervised+NI.PO
-0.047
0.900
0.264
0.011
PPO
Supervised+PPO
-0.191
0.144
0.157
0.636
PPO
Zero Shot
-0.411
0.001
-0.124
0,822
Supervised
Supervised+NLPO
0.177
0.021
-0.003
0,900
Supervised
Supervised+PPO
0.033
0,900
.0.110
0.896
Supervised
Zero Shot
-0.117
0.645
-0,391
0.002
Supervised+NI.PO PO
Supervise=-PPO
-0.144
0.444
-0.107
0.009
Superviscd+NI.PO Supervised+PPPO
7ero Shot
-0.294
0.002
-0.388
0.003
Zero Shot
-0.151
0.390
-0.281
0.008
Table 35: Results of an post-hoc Tukey HSD Test for difference in means between pairs of algorithms (Group Group !) and corresponding p-values. Individually statistically significant results are bolded and used to discuss results in the analysis. Overall p-values showing that there is significant difference in means between the models via a one-way ANOVA test are significant with p 0.05 for both coherence and sentiment.
59

---

Published as a conference paper at ICLR 2023
Inceuuttions jrlik expand)
your
Plese toke omammnet to familorttre yournef with mit new mendinng the tntacclossooeepltss boussee thigs have changed' or Thonis again for your world
thi you arcsented with didl egue condziing conversation beoweon
people. will bogeeen syslamieeee aller, atrms to ins ofe conversation Your job to rasa the system gan aenaraaion aprous 2 awass
Dialogue context (read me first!):
and vualloyyoohreenee: antrancc sobarett casconates and sfmasn
Muencyrrrammatialig
Doont
Person 1: doesn't work.
Ascao ianalay contort dionager buttoge
will each
acate from
boing lowsathworot
Person 2: What happened?
higherhheer The scales
#lenny//rraamatiiltt
5/5 (xccellentz the gereraation guant mutic and Buurr
Person 1: don't know.
4/5 (goods the sentence linglly makes senoe, but there are some smmll
tauee/outtofflle placs worda that don't make the besting poosille but detiviinly
Person 2: Did you call the repairman?
paknys
dicannt Soourdlfe
(ports Svan though
Person 1: Of course.
attaauurll pemories.
perritlee The gereection the severe errarsin in grvmmmett ally''s mons complenely unreadabre.
Qualitytaherenee
System's generation (rate this!):
(perfectly
dislegus imoulyrchount ganer relovvrt por glvan lve dakkgu le conton.
might ery
Person 2: I'm sorry. They couldn't find the problem.
5/5 jpuurayy The generation is somowhat ennioon..
2/5 prowlyyi innworss new why itas scutd the geren atwe tuu dowerr moke much sence.
1/5 wwwongmmnsnnnenrtteeeeee the generatio dosorts seem make
Fluency/Grammaticalty::
3/5
Is the system's generation gremmetical, easy read, and flurnt?
Bad
Excellent
The grammar okay and it's possible definitely doesn't
sound like humen wrote
Quality/Coherence: 3/5
Is the nort stteronor coherent remnnnahis and the hey of perooe might sox, in the contral of the dislogur histry?
Bad
Excellent
generation somewhat plausibeerdeeent.
Figure 10: Instructions and interface for the Daily Dialogue task.
B.9.4 QUALITATIVE ANALYSIS
We show sample generations from each of the algorithms for three randomly picked prompts below.
Sample
Prompt: Right And China is the homeland land of tea <EOU> Yes Chineae people e love rking lea so much Some even claim they can't Live without Lea <BOU> Do you know there are black cess and scented ed tess Any Others ? <Dot> Nell have you evez heard of Oulong tea and <EO0>
several cazagories of Chinese tea <BOU> Yes believe there green
compressed
References: Ob yeah Dulong zea is good for one's health. isn't it ? Zezo-Shot know that Gulong tea can he enjoyed with very meals, and there are very fex PPO: I have, but it's not as strong
Oulong tea.
NLPO: haven't. I don't have tes.
Supervised: Yes. hear they do goed. And 1 heard this raijiquan is very good supervised PPO: Actually, I have had Oulong tea, but haven's had any for owhile new. Supervised
No, never
of that.
Sample
60

---

Published as a conference paper at ICLR 2023
Prompt: think might
dahl <2ou> vegetables in spicy curry sauce with rice <sot> that sounds nice no you want to share some kebabs well <Bot> ok How about some lamb kebaba ? <50t>>
<2bu> got chickpeas
References: t fevourite Do
to have wine beer
Sero-Shot: they' re very salty
#Po: wall you might want take peek.
NLPO: well I like that.
Supervised:
they are. so thirsty. cen't until next
supervised РPO: I've tried that a few times, out they' too dry and T don't want any of supervised NLPO: ch, ok. That's a good choice, I'll Lake it.
Sample
Proupt: Really it's 200 high-tech and so it's the first part that breaks <bots Hmm You know the problem might be
you had
break bafore <EOU> Yes
think
No didn't clean it.Buz still I only had the laat one two menths and it broke I con't trust auto- reverse <DOG> Alright Nell We have many units without auto zeverse. ie good Very relisble.it has AM PM band built-in micrephene recording capability and no auto-raverse <BOU>
you
unit often
References:
Zero Shot: Oh, So you don't t. have the AX / TX band?
PPO:
EM. you sure NLPO: Nell. Well. had no idea when it
and didn't care.
Supervised: Good.Rell, what else need? Supervised sut think is lacks the up player its player accesscries. Supervised NLPO: Cool! so what'a the trouble with it7
61


Reviews:
[{"review_id": "s8wG_fiWKp", "reviewer": "Reviewer_a1G9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a library (open-sourced) called RL4LMs, which is for optimizing generation models using RL. The library is compatible with HuggingFace. Second, the paper comes up with a GRUE (general reinforced-language understanding evaluation) benchmark, which consists of six language generation tasks (under the RL framework) containing IMDB text continuation (reward is related to sentiment), a CommonGEN generative common sense task, CNN/DailyMail summarization, ToTTo (data-to-text generation), a machine translation task using a small IWSLT dataset, and NarrativeQA (question answering). The paper also proposes the NLPO algorithm (natural language policy optimization); the difference between NLPO and PPO is in green font in Algorithm 1. Essentially, the policy which we collect trajectories from is an old copy of the current policy. Moreover, initialization is different from PPO.\n", "summary_of_the_review": "Toolkit -- nice contribution. Selection of task for the general benchmark -- not satisfactory to me. The NLPO algorithm is clever, but it needs more justification. In general, I'm leaning toward accepting because of the software and the encouraging empirical results. ", "strength_and_weaknesses": "I think the toolkit will be a nice contribution to the research community. It will also encourage more people to dig deeper into the RL for NLP field. \n\nIt’s great that the software contains multiple RL algorithms.\n\nThe paper is clearly written – it’s effective in getting points across to readers.\n\nHuman annotation is involved, which is great. \n\nConcerns below:\n\nThe selection of tasks isn’t totally satisfactory to me. Generating a positive continuation of movie reviews is too artificial and unchallenging. I think the benchmark (given there’s the word “general” in it) should consider real use cases of RL for NLP; for example: generating better dialogue responses where the reward is a human-preference-based metric or a toxicity-based metric. Moreover, IWSLT17 English-German is also quite artificial and IWSLT-trained systems will never be put into production or real use. Additionally, most of the tasks have a correct answer (or the space of correct generations is quite small), but there are many text generation tasks where the correct space of generations is very large, like in dialogue or story generation. I think these tasks should be taken into consideration as well.\n\nOne concern I have about the NLPO algorithm is that if the masked policy is an old copy, then the algorithm is not on-policy anymore. Then, the policy gradient derivation (e.g., see the Berkeley deep RL course policy gradient derivation, or any other policy gradient derivation) would not hold anymore. How do the authors justify the fact that we’re not able to do the policy gradient proof anymore (please correct me if I am wrong)? Or why does this periodic synchronization trick work? I would hope for a summary of either a theoretical analysis or empirical analysis. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G2YiJCQtIm", "reviewer": "Reviewer_zDTr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies RL on language models, and makes 3 contributions:\n1) RL4LMs, a modular library for optimizing language generators with RL.\n2) GRUE, a benchmark of 6 language generation tasks with reward functions.\n3) NLPO, a new algorithm which improves on PPO for RL on LMs. ", "summary_of_the_review": "My personal belief is that fine-tuning language models to optimize specific metrics that are not perplexity (particularly human preferences) is going to be a huge field, perhaps one of the most impactful in ML. This paper clearly advances the accessibility of research on this problem. Thus, I think this paper may become an enormous contribution to the field. I'd be excited for the datasets and tasks to be updated regularly. ", "strength_and_weaknesses": "Strengths:\n+ The lack of open-source benchmarks and tasks for RL on language models is indeed a huge problem. This paper does the community a huge service by providing this library + benchmarks. \n+ The paper is clearly written and easy to follow.\n+ The paper is quite thorough. I appreciate the many ablations, and the study of human agreement with automatic metrics.\n+ The NLPO algorithm is simple, clearly explained, and does seem to provide an improvement over PPO. It remains to be seen if this will transfer to other, more complicated tasks + larger scale. (it does involve an extra copy of the policy weights, which has memory downsides, especially at large scale, so it's unclear if it will be feasible / worth the tradeoff). Though overall I think this is a small part of the contribution.\n+ The paper has a useful section on important implementation details.\n\nWeaknesses: \n- personal nit: I don't think CNN/DM is a very good summarization dataset (though it is very standard). \n- I think the biggest weakness of this benchmark / tasks is that it relies on simple automatic metrics (eg BLEU, sentiment score). This makes sense of course, since it makes it tractable to evaluate. But I suspect that, once people start optimizing these metrics directly, whatever metric correlation with human judgments exists will disappear. I think it's important to consider how to incorporate human data collection into this process. (This would obviously be a very big endeavor, so I don't think it's within scope for this paper, but I think is a useful future direction.)  One option is to hold regular competitions (eg at a conference) where you collect human data which you use to evaluate models, and then open-source that data for further training (similar things have been done in the context of dialog models). Basically, I think it will require quite a bit of effort to ensure this benchmark remains useful over time, and doesn't saturate like every other NLP benchmark.\n\nSmall notes:\n- \"We find that using RL to learn from scalar reward feedback is can be more\"\n> remove \"is\"?\n\n- \"high-quality estimates..\"\n> two periods\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4LxL40awo1", "reviewer": "Reviewer_pVj7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors introduce a new library for benchmarking RL methods on NLP tasks, along with a modified version of PPO for on-policy optimization. Results demonstrate that the proposed NLPO algorithm outperforms the PPO baseline according to human assessment.", "summary_of_the_review": "I would like to evaluate the contribution of this paper focused on the proposed NLPO method. I will lean towards an acceptation if the proposed method is indeed novel, results in state-of-the-art performance and the improvement is significant enough comparing to PPO.", "strength_and_weaknesses": "Strength:\n• This paper builds a cross-task diverse evaluation benchmark containing 6 tasks.\n• For the proposed NLPO, human evaluation and automatic evaluation show it outperforms the PPO baseline\n\n\nWeakness:\n• With the mask prediction policy update, the gain based on human evaluation seems to be marginal except on IMDB task metrics, and CNN naturalness metric\n• The proposed NLPO in this paper seems to be similar to [1] ? \n• In experiments, there no comparison to traditional structure learning methods like [2]\n\n[1] Donati, A.M., Quispe, G., Ollion, C., Corff, S.L., Strub, F., & Pietquin, O. (2021). Learning Natural Language Generation from Scratch. ArXiv, abs/2109.09371.\n\n[2] Shu, R., Yoo, K., & Ha, J. (2021). Reward Optimization for Neural Machine Translation with Learned Metrics. ArXiv, abs/2104.07541.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "0FjLw98tDvX", "reviewer": "Reviewer_H8br", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to explore whether reinforcement learning (RL) can be a useful part of NLP generative models. Although RL methods have been used fairly often, there have been disagreements among experts about whether it provides meaningful gains, especially in light of training difficulties (eg enormous action spaces of 30k+). In this work, the authors develop an open source library (6 tasks, 15 metrics, 3 learning algorithms), create a new state-of-the-art RL learning algorithm, create the first RL-NLP leaderboard, and use MTurk to annotate 4 of those 6 datasets with human-validated labels. It finds that tasks involving heavy novelty (ie \"zero-shot performance\") do seem to benefit from training with RL.", "summary_of_the_review": "This work is very comprehensive: the authors develop an open source library (6 tasks, 15 metrics, 3 learning algorithms), create a new state-of-the-art RL learning algorithm, create the first RL-NLP leaderboard, and use MTurk to annotate 4 of those 6 datasets with human-validated labels. It finds that tasks involving heavy novelty (ie \"zero-shot performance\") do seem to benefit from training with RL. By making their code into a public benchmark, they provide a very helpful starting point for the community to develop & compare future work against. Finally, they provide a good lit review of many (though not every) instance of RL in NLP, commenting on its shortcomings to date.", "strength_and_weaknesses": "STRENGTHS\n- This work is very comprehensive. It makes public a large library of many tasks, rewards/metrics, and learning algorithms.\n- The learning algorithm (NLPO) outperforms the previous state of the art (PPO)\n- In addition to the above work, further validating 6 of the datasets with MTurk is impressively thorough\n\nWEAKNESSES\n- The work could be more clear on pages 4 and 5 when describing on-policy actor-critic algorithms (3.3) and NLPO (4).\n\nMINOR\n- There appears to be a typo on page 8 \"2 out of 4 tasks tasks\". The word \"tasks\" is repeated.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%