trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback
Authors: Alexander Havrilla
Maksym Zhuravinskyi
Duy Van Phung
Aman Tiwari
Jonathan Tow
Stella Biderman
Quentin Gregory Anthony
Louis Castricato
Venue: EMNLP 2023 Main (2023)
Keywords: RLHF
LLM
Framework
TL;DR: An open-source framework for RLHF fine-tuning at scale

Abstract: Reinforcement learning from human feedback (\textbf{RLHF}) utilizes human feedback to better align large language models with human preferences via online optimization against a learned reward model. Current RLHF paradigms rely on Proximal Policy Optimization (\textbf{PPO}), which quickly becomes a challenge to implement and scale up to large architectures. To address this difficulty we present the \textbf{AutoRLHF} library as a feature complete open-source framework for RLHF fine-tuning of models up to and exceeding 70 billion parameters. To do so we implement support for multiple types of distributed training including distributed data parallel, model sharded, as well as tensor, sequential, and pipeline parallelism. Additionally, we implement compute and memory saving features, giving AutoRLHF the flexibility to support users with a wide range of compute resources. This includes offline RL methods like Implicit Language Q Learning (\textbf{ILQL}) as a compute efficient alternative to PPO. We find offline fine-tuning offers competitive performance relative to online algorithms while being easier to implement, train, and scale. To evaluate our framework we train RLHF models on two separate well-known tasks using publicly available human preference data. Models trained with AutoRLHF achieve preference win-rates over baselines at rates comparable to the original works.

trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback
Alexander Havrilla Maksym Zhuravinskyil.; Duy Van Phung
Aman Tiwari', Jonathan Towl.3, Shivanshu Purohit', Stella Biderman Quentin Gregory Anthony5?, EEhan Kim², and Louis Castricato
'CarperAI, Georrgia Tech, SStability AI, "Independent Researcher", El Booz Allen Hamilton, 'Ohio State University, vectorshiff.ai
Abstract
corpus. However, this often comes with a decrease in model performance on other tasks. Further, the resulting model can be difficult to work with, re- quiring extensive prompt engineering. Reinforcement learning from human feedback (RLHF), has presented itself additional third stage of the language model training pipeline. In this stage, gathered human preference data is used to supervise fine-tune the pre-trained model and then train a reward model. The reward model assigns scalar values to (prompt, response) pairs that correspond to human preference. The super- vised fine-tuned model can then be trained using Proximal Policy Optimization (PPO) (Schulman et al., 2017), an online reinforcement learning al- gorithm, optimize against the learned reward model. The resulting models are better aligned with human preferences, leading to win-rates over the base model by up to 80% (Askell et al., 2021; Bai et al., 2022a; Ouyang et al., 2022), and produc- ing more desirable and less harmful text with less prompting.
Reinforcement learning from human feedback (RLHF) utilizes human feedback to better align large language models with human preferences via online optimization against learned reward model. Current RL.HF paradigms rely on Proxi- mal Policy Optimization (PPO), which quickly becomes challenge implement and scale up to large architectures. To address this diffi- culty we present the trlX library as feature- complete open-source framework for RLHF fine-tuning of models up to and exceeding 70 billion parameters. We implement support for multiple types of distributed training including distributed data parallel, model sharded, as well tensor, sequential, and pipeline parallelism.
To increase the accessibility of RLHF to researchers, we implement compute- and mevory-saving features that give trlX the flexi- bility support with wide range of com- pute resources. This includes offline RL meth- ods like Implicit Language Q Learning (ILQL), low-rank adapters, and the Hydra architecture. We find offline the-tunnng offers competitive performance relative online algorithms while being easier to implement, train, and scale. To evaluate our framework we train RLHF models separate well-known tasks using pub- licly available human preference data. Models trained with trIX achieve preference win-rates over baselines at rates comparable to the origi- nal works.
However, the study of RLHF models in the aca- demic community has been limited thus far by a need for open-source training frameworks and datasets. Online reinforcement learning via PPO is compute expensive and challenging to scale we must store three copies of the model in mem- ory during training. To address these issues we present triX: library supporting online and offline RLHF fine-tuning for language models of up and exceeding 70 billion parameters. At smaller scales, we emphasize low-resource accessibility by incorporating features such as Hydra model archi- tectures (Glaese et al., 2022), LoRA adapters (Hu et al., 2021), and DeepSpeed (Rajbhandari et al., 2019), which, when combined, can reduce memory overhead GPT-J (Wang & Komatsuzaki, 2021) by up to 75% with minimal impact on the achieved
1 Introduction
Since 2019, the prevailing training paradigm for large-language models (Brown et al., 2020; Raffel et al., 2019; Devlin et al., 2019) has comprised two parts: First a pre-training phase during which mod- cls are trained a large corpus of text. Pretrain- ing produces a general-purpose model which has learned the syntax and some semanties of natural text but is not easily controllable for any particular task. Fine-tuning is then used to adapt this general- purpose pre-trained model to a smaller, specialized
work done while at CarperAl Correspondence to: ahavr law@gatech..eeeeecc.ooo.eee.e.

---

RL Algorithms
Parallelization Strategies
Features
Online
Offline
Tensor
Pipeline
Sequence
LoRA
Sweeps
RL4LM
tri
DS Chat trlX (ours)
Table 1: trlX feature comparisons with other libraries. trl supports naive pipeline parallelism, which allows larger models to be run but far less efficient.
reward. trlX training at this scale is compatible with most encoder-decoder and decoder-only ar-
tuning for preference learning at scale.
chitecture supported the popular Hugging Face Hub (Wolf et al., 2019). For large, 20 billion pa- rameter plus, training jobs we implement support for tensor, sequence, and pipeline parallelism via both the GPT-NeoX library (Andonian et al., 2021) and NeMO--Megtron (Kuchaiev et al., 2019).
Release of all models at all scales involved in the training pipeline including supervised-fng tuned, reward, and RL tuned models.
Background
Reinforcement Learning from Human Feedback Reinforcement learning from human feedback at- tempts to improve agent performance, either in sam- ple efficiency or performance on downstream tasks, by incorporating some form of human feedback on agent behavior (Knox & Stone, 2009; Christiano et al., 2017; Stiennon et al., 2020). We focus on the 3-stage fine-tuning pipeline outlined by Stien- non et al. (2020) in which researchers first assem- ble a dataset of human-annotated preferences, then train a reward model to predict these preferences, and lastly train policy to maximize the score of the resulting reward model. Similarly structured pipelines have been adapted to train many of the most recent interactive natural language assistance tools (Nakano al., 2021; Ouyang al., 2022; Bai et al., 2022a; Glaese al., 2022).
trlX also supports Implicit Language Q Learn- ing. ILQL, offline alternative online RL methods. We find fine-tuming via ILQL achieves preference win-rates with baselines close to PPO but a fraction of the compute cost. Further ILQL is more robust to reward model overfitting, which online algorithms can suffer from. To evaluate our framework we provide open-source replications of well-known papers from the RLHF literature in- cluding learning summaries from human feedback (Stiennon et al., 2020) and Helpful/Harmles prefer- ence learning for general purpose language assis- tant (Bai et al., 2022a). We find models trained with trlX achieve preference win-rates, as judged by hu- man annotators, over baselines at rates comparable to the original works, validating our implemen- tation. We open-source all supervised fine-tuned models, reward models, and RLHF models for fur- ther research, as well as the training framework, trlX.
Nevertheless, collecting human preferences at scale can be cost-inefficient, Recent work by Bai et al. (2022b) proposes tractable oversight by ing synthetic Al preferences instead of direct hu- man labels. Similar works by Honovich al. (2022); Wang al. (2022a,b) generate instruction- following datasets by querying already aligned models like text-davinci-On to generate both task and instruction-following responses.
In summary, we make the following contribu- tions:
trlX as a feature complete, open-source library for reinforcement learning from human feed- back supporting model sizes up to and ex- ceeding 70 billion parameters. This includes benchmark examples implementing the first known open-source replications of several well known RLHF models, offering insights into training and evaluation.
Scalable Training Frameworks There are many notable choices of distributed training frameworks for large-scale language model pre-training and fine-tuning, each implementing various parallelism schemes. These include DeepSpeed (Rajbhandari et al., 2019). Megatron- (Shoeybi et al., 2019;
novel evaluation of offline RL based fine-

---

Kuchaiev al., 2019), the GPT-NeoX library (An- donian al., 2021) which combines DeepSpeed and Megatron-LM, Fairseq (Ott et al., 2019; Zhang et al., 2022), and T5X (Roberts et al., 2022) for TPU-based training.
train models up 70 billion parameters: an unprecedented scale for open-souree RLHF models.
The framework is built around a base trainer from which integration specific trainers can inherit. Independently, online and offline algorithms are implemented allowing for reuse in separate inte- grations. In particular trlX supports PPO and A2C for online RL and ILQL for offline RL. The most expensive part of online PPO training is the model rollout, which can take up to 10x as long as a com- bined forward and backward pass. To efficiently maximize batch size for both rollouts and opti- mization steps, decouple the rollout inference batch size from the PPO batch size via an orches- trator class. This allows online models to perform batched rollouts reduce the amount of bottleneck time spent infering cach model.
However, none of these frameworks are de- signed explicitly to support fine-tuning via RL and would, therefore, require significant work to retrofit. More recently, RL-specific fine-tuning li- braries have become available. RLALM's (Rama- murthy et al., 2022) implements online algorithms for fine-tuning mid-sized language models with reinforcement learning from human feedback and supports an impressive range of tasks and metrics. TRL (Leandro, 2019), initially smaller-scale li- brary for transformer reinforcement learning, is re-implementation of Ziegler et al. (2019) in Py- Torch for doing sentiment-bane fine-tuning. has since been expanded in a manner similar trIX to include training with DeepSpeed via Hugging Face accelerate. More recently, DeepSpeed-Chen (Yao et al., 2023) was released allowing for larger model training with better throughput. However, they do not allow for more advanced forms of parallelism supported in trlX.
We integrate closely with the Hugging Face ecosystem, allowing for the training of most encoder-decoder and decoder-only models on the Hugging Face Hub, including widely used models such T5 (Raffel et al., 2019) and Flan-T5 (Chung et al., 2022), GPT-J (Wang & Komatsuzaki, 2021), Pythia (Biderman et al., 2023), OPT (Zhang et al., 2022), and LLaMA (Touvron et al., 2023a,b).
3 Training with trlX
Fine-tuning large language models via human feedback with PPO is prohibitively expensive in terms of memory and FLOPs, requiring the user to store student model, reference model, and sim- ilarly sized reward model in memory at all times. Additionally, reinforcement learning is notoriously brittle to hyperparameter choice, often requiring extensive sweeping to find optimal settings. To mitigate these costs we support parameter saving techniques like LoRA (Hu et al., 2021) and Hy- dra model architecture design (Glaese et al., 2022) which allows for shared frozen layers between pol- icy, value, and reference networks. Similarly, ILQ models require non-standard Q-value heads and generation capabilities which are implemented sep- arately for both integrations.
trlX is designed help mitigate the heavy com- putational costs felt by low-resource users while still allowing high-resource users the ability to get good performance. Roughly we break our users into three separate resource profiles:
1. Single GPU users. this low-resource use case we recommend our native PyTorch inte- gration plus memory saving features including Hydra architectures, low-rank adaptors, and 8-bit adam (Dettmers et al., 2021).
2. Multi-GPU users. In this mid-resource user case we recommend our integration with Hug- ging Face accelerate (Gugger et al., 2022) leveraging DeepSpeed plus memory saving features. We use this integration to comfort- ably train up to 20 billion parameter language models on a single node.
3.1 Memory and Compute Saving Features
To benchmark the effect of memory and compute saving features on performance, we series of experiments a baseline sentiments task for model sizes ranging from 125 million 20 bil- lion parameters. For each model size, we freeze a percentage the model's layers in the Hydra architecture and observe the effect on reward, train
3. Multi-Node users. In this high-resource user case we recommend our integration with GPT- NeoX or Nego-Me which allows for higher gpu efficiency and scaling than acceler- ate and DeepSpeed. We use this integration to

---

time, and required memory. We also experiment with applying LORA adapters of various ranks to all transformer matrices. Models are taken from the Pythia suite (Biderman et al., 2023) and trained for 6000 steps with global batch size of 32 on 8x80gb A100s.
Memory sumption hydra Layers
120
80 A.p0
Unfrezen layors 2
Untrozen
Unfiozen layers
Unfiozen lsvers 10
Untorren layene 20
GPU Memory
unfrezen lavers
Figure demonstrates across all model sizes about half the layers can be frozen before max- imum attainable reward is not achieved. Interest- ingly, freezing all but two of model's layers more adversely affects the larger models. We speculate this is due to larger models learning the majority of complex task-specific features their middle layers, with downstream layers making minor ad- justments.
60
leyers
40
20
12514
86
200
Model Sue
Figure 2: Hydra memory consumption a function of the number of a unfrozen layers.
benefits can be scen with LoRA-based fine-tuning,
Reward % Unfrazen Layers
When tuning all layers training with LORA rank achieves max reward on the sentiments benchmark. At the 6.9 billion parameter scale LORA A training finetunes only 0.03 percent of the model parameters and reduces memory usage by 3 times. LORA train- ing can be combined with layer freezing to achieve further memory savings. With both optimizations RLHF can be performed for medium scale models even on a single consumer- consumer-grade GPU. These mem- ory savings and performance benefits carry over to the offline training regime with ILQL as well (see table 2). We theorize that limiting the rank of the parameter updates well freezing model layers provide beneficial regularization effects for both online and offline RL training.
01910
0.925
0.515
18
208
3.2
0.4
0.6
0.55
1/2
19 Unfrceer Layers
Figure 1: Max reward achieved as function of the ber of model layers unfrozen. Each model achieves its maximum attainable reward with around half its layers frozen.
Figure demonstrates the effect of layer freezing on memory savings. This is particularly useful for larger models since otherwise we must separately load the frozen reference model into GPU memory for inference. With all but two layers frozen we save both memory and computation costs for all but two layers of the reference. In particular, for larger model sizes we can save nearly 50% of the required memory while still achieving the maximum reward.
3.2 Comparison with other Frameworks
See table 1 for table outlining the presence of key trIX features in similar libraries. trlX is the only framework support offline RL fine-tuning as well as the only framework to support large model fine- tuning at scale with pipeline, sequential, and tensor parallelism. Additionally, we are the most feature complete, including tools for parameter efficient fine-tuning and distributed hyperparameter sweeps. We include 10 plus benchmark examples, providing end-to-end pipelines for several well known RLHF tasks.
On less toy problems we also observe layer freez- ing helps stabilize the training process by reducing KL divergence from the base model. This helps mitigate the need for the KI. based penalty via frozen reference model, in some cases allowing for be removed entirely. Further in some cases partial freezing even imparts beneficial inductive bias, allowing the model to achieve a higher reward than when trained with all parameters unfrozen. Similar benefits semory-ssving and regularizing
DeepSpeed versus NeMO Megatron trlX is competitive with existing open-source RLHF im- plementations scale for online RL. We com- pare against DeepSpeed-Chi -hattt (Yao et al., 2023), a concurrent work on RLHF for LLMs that imple- ments PPO. See table for a performance com-

---

Model
Max Reward
Time (min)
GPUs
GPT-NeoX 20B
-1.88
156
32
GPT-NeoX 20B LoRA
-1.89
28
16
Pythia 6.9B
-1.77
286
16
Pythia 6.9B LoRA
-1.68
58
16
Table 2: Benchmarks for ILQU's time to max reward on Anthropic's Helpful QA dialogue dataset. All non-LoRA hyperparameters are kept the same as the base models, except for learning rate which is set to 2.0 10-4, For GPT-NeoX-20B LoRA, the last 8 layers are trained with LoRA, for Pythia 6.9B LoRA, all layers are trained with LoRA.
Parameters
DS-Chat
trXX
standard deviation estimate. We find this nor- malization, notably without subtraction by the running mean, significantly improves perfor- mance. Additionally, we do second normal- ization of the advantages at the batch level.
T.3B
2.1
2.0
OPT 6.7B
0.44
0.41
OPT 30B
0.14 (LoRA)
0.12
OPT 60B
0.076 (LoRA)
0.043
Table 3: Comparison of trlX and DeepSpeed- Chhpte train- ing speed for online RL (PPO) for OPT architecture, measured in samples/s/GPU. PPerformance for 30B and 60B DeepSpeed-Chen converted from Table 2 of (Yao et al., 2023), 4 hours train on 131.9k samples using 64 GPUs. t For OPT 66B, use Hydra with 50% trainable parameters.
Learning rate: Learning rate chosen be 5E an order of magnitude smaller relative to supervised fine-tuning.
4.1 Summarization
Setup Learning Summarize from Human feed- back (Stiemnon al., 2020) introduces the TL:DR dataset. The first component, consisting of 129,772 Reddit posts, is utilized for supervised fine-tuning. The second component, utilized for training the reward model, consists of 92.534 samples in the training dataset and 83,629 samples in the valida- tion set.
parison between DeepSpeed-Cher and trlX. Note that the 30B and 60B parameters performance fig- ure for DeepSpeed-Chen uses LoRA based training, whereas trlX uses full parameter finetuning. We keep the rest of the benchmark settings the same the published DeepSpeed-Chinn -hat scripts.
We start by training supervised fine-tuned (SFT) by fine-tuning 440M, 1.4B, 6.9B, and 20B models from the Pythia suite on the SFT dataset. We use the AdamW optimizer with the learning rate is lE 5 followed by a linear scheduler with a short warm- up. The best model is be selected by Average- ROUGE score on the validation set.
4 Benchmarks and Results
We benchmark trlX on two popular RL.FF tasks: OpenAl's learning to summarize TL;DR dataset (Stiennon et al., 2020) and Anthropic's Helpful QA dataset (Ganguli et al., 2022). We release all asso- ciated code and models open-source for further study.
To train reward models (RMs) initialize with SFT checkpoints, replacing the causal head with a scalar output. Using the second component of the dataset we then minimize the pairwise pref- erence loss (Stiennon et al., 2020). We find our best performing reward model is the 6.9 billion parameter GPT-J (6B) trained with batch size of 32.
Training setup and Hyperparameters Unless otherwise noted, we use the same fixed set of hyper- parameters listed in the appendix across all training runs. We find good performance is particularly sensitive to some parameters:
Larger batch sizes of at least 128 global samples per iteration are used. This reduces variance across runs and stabilizes performance.
Batch size:
With trained reward model perform RL based fine-tuning of models from 440M 20B. Posts from both components of the TL;DR dataset are used during training. We use the best performing reward model, 6.9B, as our reward signal for all experiments. To train models online
now able
Reward normalization: Upon rollout collec- tion we normalize all rewards by a running

---

we initialize from the SIT checkpoints and use PPO with four epochs per batch and a KL penalty with a coefficient of 0.005. We keep layers un- frozen. To train the offline models we label posts from both components of the dataset and their as- sociated summaries with =1 respectively. We note labeling in this way performs better than labeling the data with the learned RM. This reward labeled dataset is then used train base model using the IL ILQL algorithm. Notably, do not initialize from the SFT checkpoint as we saw minimal benefit in the offline regime.
Model Relative Beseline
0.F
Osenii
PPO
0.55
0.3
0.1
Model
Figure 3: Win rate of ILQI, PPO fine-tuned models against their relative SFT baselines on summarization task. Note comparisons were done against the same- sized SFT baseline (e.g. 6B SFT against 6B PPO). The OpenAl baseline, measured the win-rate of their 6B model over human generated summaries, is included for reference.
Results We attach table in the appendix show- ing the result of the ROUGE scores in the test of the TL;DR dataset of SFT and PPO trained on the 6.9B model. comparison with (Stiennon al., 2020), the trend of the ROUGE score similar, with the SFT model performing slightly better than the PPO model.
208 Liorrt Scores
More critically, we conduct human evaluation to better evaluate how well our online PPO and offline ILQL models adhere to human preferences as compared to the the baseline SFT. To do so we select stories from a subset of prompts from the test portion of our dataset and ask annotators to choose between two candidate summaries. In particular for each model size run two evaluations: A comparison of PPO to SFT and a comparison of ILQL SFT. In addition to choosing between two candidate summarizations, we ask users to score the coverage, clarity, and inconsistency on a 1-7 Likert scale. The results are reported in fig. 3 and fig. 4.
SFT
PRO
LOL
Usertntingg
coverage
Figure 4: 20B model Likert scores for coverage, clarity. and inconsistency.
potentially be developed more compute efficient alternative version of PPO.
We evaluate each model via relative improve- ment over the its corresponding SFT baseline, in part to demonstrate the effectiveness RLHF on even small model sizes.
4.2 Helpful QA Dialogue
Setup Helpful Harmless RLHF (Ganguli et al., 2022). or HH-RLHF for short, consists of 1 118k sample interactions between a user and an AI as- sistant. It can broken further into three parts: An initial dataset of 42k prompt, response triples created by a prompted language model, 40k sam- ples creates by re-ranking responses from the same prompted model, and a final set of 22k responses from initial RLHF model. The first two parts called the static subset. We use the helpful por- tion of the static dataset for training and evaluation. Each interaction sample consists of dialogue his- tory ending with an utterance from the user. This is followed by a preferred or chosen assistant re-
ILQL slightly underperforms PPO at fraction of the cost Figure demonstrates both ILQL and PPO achieving greater than 10% win-rates across most model sizes. At 6B and 20B our PPO model achieves greater than 70% win-rate against its SFT baseline. We also see ILQL models are very com- petitive, despite requiring much less compute to train. Interestingly, observe ILQL produces dis- tinctly shorter, more concise summaries than PPO and even the SFT baseline. Despite this ILQL is still often preferred over the longer SFT baseline due to better coverage of key points. This suggests more sophisticated offline training methods could

---

Helpful Reward Model Accuracy vs. Number of Comparisons
125M
lel
0.70
208
0.68
vonilla pp00
Acuracy
0.40
0.64
0.55
0.58
per 1K comparisons
0.0
233
50
75
10.0
12.
15.0
17.5
20.0
Figure 5: Accuracy of reward model test against number of training comparisons. We observe roughly 1.8% increase in model accuracy per 10k training sam- ples.
Figure 6: Mean performance of models zero-shot on HellaSwag, TriviaQA, LAMBADA, Are Easy, Are Chal- lenge, and Open Book QA. table of full results is shared in the appendix.
sponse and a rejected response.
We train supervised fine-tuned SFT baseline models by fine-tuning vanilla models models rang- ing from 160M, 1.4B, 6.9B and 20B parameters on the chosen responses for each sample. Training is done for one epoch with ir 5 5. Note mask loss the dialogue history, only backprop- ping loss on the response tokens. This serves as our baseline.
In particular, we found it critical to train with a sufficiently large batch to ensure robust PPO gra- dient estimates. Additionally, if training is run for too long or the KL penalty is too weak we observe heavy overfit to the reward model. We practice early stopping to prevent such overfit. Larger batch sizes also have the added effect of mitigating over- fit simply by decreasing the total number of steps over the prompt dataset.
We then independently train reward models of sizes 160m-20B by. As with summarization, we warm start by initializing from the SFT models. As above we train for one epoch with ir = 5 10-6. We observe adding the supervised warm- up increases test accuracy up to 2%. Our highest performing model 6 billion parameter GPT-J, which achieves 0.72 accuracy on the static test set. We use this as the default reward model RM for all RL based fine-tuning.
In addition to PPO we train models at sizes from 160M to 20B using ILQL. We assign reward of +1 to the chosen trajectories and reward of -1 to the rejected trajectories. We call the resulting set of models ILQL. Surprisingly the +1 reward as- signment empirically outperforms labeling chosen and rejected responses via rewards learned from the RM. We believe this because while RM re- wards are denser, they are also in some cases in- accurate and as a result the expected inequality Tchosen > Prejected for given dialogue is not re- spected, introducing noise. Whereas +1 assign- ment stays faithful to the underlying human pref- erences. This assignment has the added benefit of requiring far less compute as no reward model needs to be learned.
With our RM we can fine-tune our baseline SFT models using triX. Our training dataset consists of a set of input prompts taken from the entire static dataset. We augment this with multi-turn prompts and responses generated synthetically by text-davinici- 0003. Details on how this syn- thetic data created can be found in the appendix. Altogether this forms 200k prompts for our RL training dataset.
In addition to the above models we also train and evaluate final set Munilla-PPO which applies PPO based RI the-ttnnng via our RM without ini- tializing from the supervised SFT checkpoints. We found this is only feasible for larger models, 6B and 20B, which are able successfully optimize reward. This underscores the importance of collect- ing supervised fine- mee-tuning data for for sufficiently difficult tasks and weak models.
The number of training steps is kept constant at 9000 with effective batch size of 128. A learning rate between 1 10-6 and 8 10-6 used for different model sizes. We keep eight layers unfrozen. A constant KI. penalty of coefficent of 0.005 is used. We call the resulting series of models PPO.

---

Results We then evaluate vanilla models, SFT models, PPO models, Vanilla-PPO models, and ILQL models on a set of common academic benchmarks including LAMBADA, ARC, Open- BookQA, TriviaQA, and HellaSwag using Gao et al. (2021). Figure 6 plots the mean accuracy of each model class on the benchmarks, A full ta- ble is included in the appendix. We find supervised fine-tuning significantly impacts performance. We note when done improperly, e.g. by fine-tuning on entire dialogues instead of responses, the effect is even more pronounced. RL based fine-tuning on top of SFT improves results slightly but not significantly.
Model
Relative Beseline
0.7
Pachropic
0.6
PFO
Prompted
0.7
Model
Figure 7: Win rate of prompted, PPO RLHF, and ILQ RLHF models at 160M, 1.4B, 6.9B, and 20B parameters. Comparisons were done against the same-sized SFT baseline (.e.g 6.9B SFT against 6.9B PPO).
The 'Alignment Tax' comes from SFT Surpris- ingly contrast fine-tuning with RL without as done in the Vanilla-PPO models incurs much less penalty and in the 6.9B case even slightly improves performance. This answers questions about the existence of an alignment lax when fine-tuning with RLHE. OpenAl reported such tax with In- structGPT (Quyang et al., 2022), particularly after supervised fine-tuning, but did not report results for strictly RL based fine-tuning. Conversely An- thropic (Bai et al., 2022a) demonstrated small gains in benchmark performance for sufficiently large models after RL based fine-tuning but do not uti- lize an SFT warm-up. These results suggest such a tax is primarily due to the supervised fine-tuning instead of RL based fine-tuning. We note one tech- nique mix pre-training data into the SFT and RL fine-tuning distributions, as done in Ouyang et al. (2022).
are very competitive, similarly achieving least a 60% win-rate with small fraction the compute. Further we qualitatively observe ILQL is signifi- cantly more robust to reward overfit in contrast to online PPO based fine-tuning. In contrast the on- line regime requires a combination of large batch sizes and early stopping to mitigate such reward overfit. Finally we remark the prompted baseline is already relatively strong likely due to the task already being sufficiently distribution for larger models. This is further supported by the ability of large vanilla models to successfully optimize reward with no supervised fine-tuning.
In addition to collecting win-rates also col- lect Likert scores on scale of 1-7 for response Helpfulness, Harmfulness, and Honesty, Results are reported the appendix.
This suggests to the importance of a high- quality SFT training dataset mitigate benchmark reduction while also appropriately learning the de- sired behavior.
Conclusion
Ethics We present trIX open-source frame- work for training large language models scale us- ing reinforcement learning from human feedback. Even with fine-tuning via RLHE, LLMs are still prone to hallucination and bias at inference time, necessitating further research on mitigations. We hope researcher access to trlX as an open-source implementation of the RLHF pipeline will help facilitate this research.
In addition to automatic benchmark evaluation we conduct a human evaluation in which labelers choose between responses generated by model and the comparably sized supervised fine-tuned baseline. The results reported in fig. 7. Note we examine win-rate of models against a baseline of the same size, in contrast to previous work (Bai et al., 2022a; Ouyang et al., 2022). We attach an- notator instructions in the appendix.
Limitations While it is highly performant, PPO suffers from many limitations both in difficulty dur- ing implementation and in hyperparameter sensitiv- ity at train time. Offline methods such as ILQI are both easier to implement and more compute effi- cient, but still do not achieve the same performance
RLHF can benefit smaller models too Across all model sizes we observe least a 60% win-rate between the PPO trained model and the SFT base- line. Additionally the offline trained ILQL models

---

as PPO.
Processing Systems, volume 33, pp. 1877-1901.
Curran Associates, Inc., 2020. URL https:
//proceedings neurips ccc/ppape/222//F1eee 1457706660fcb449774888188883 4atpaaaaaa122444aaaapaaaa pdf.
References
Andonian, A., Anthony, Q. Biderman, S., Black, S.. Gali, P., Gao, I.., Hallahan, E.,, Levy-Kramer, J., Leahy, C., Nestler, L,, et al. Gpt-neox: Large scale autoregressive language modeling in pytorch. Gitllub Repo, 2021.
Christiano, P. F. Leike, J., Brown, T. B., Martic, M.. Legg, S., and Amodei, D. Deep reinforcement leam- ing from human preferences. ArXiv, abs/ 706.03341. 2017.
Askell, A., Bai. Y., Chen, A.. Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assis- tant as laboratory for alignment. arXiv preprint arXiv:21 2021.
Chung, H. W., Hou, L.,, Longpre, S., Zoph, B.. Tay, Y.. Fedus, W., Li, E., Wang. X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X.. Chowdhery, A.. Valter, D., Narang, S., Mishra, G. Yu, A. W. Zhao, V., Huang. Y., Dai, A. M., Yu, H., Petrov, S., hsin Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le. Q. V., and Wei. J. Seal- ing instruction finetuned language models. ArXiv, abs/2210.1 1416, 2022.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, J., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., fil-Showk, S., Elhage, N., Hatfield- Dodds. Z., Hernandez. D., Hume. T., Johnston, S., Kravec, S., Lovitt, L.,, Nanda, N., Olsson, C., Amodei, D., Brown, T. B., Clark, J.. McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204 2022a.
Dettmers, T., Lewis, M., Shleifer, S., and Zettlemoyer, L. 8-bit optimizers via block-wise quantization. ArXiv, abs/21 10.02861, 2021.
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805. 2019.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint ArXiv:22 220.2222,,873 2022b.
Ganguli, D., Lovitt, L.. Kernion. J., Askell, A.. Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 2022.
Biderman, Bicheno, K., and Gao, Datasheet for the pile. arXiv preprint ar0iv: 00731,, 2022.
Biderman, S. R., Schoelkopf. H., Anthony, Q. G., Bradley, H., OrBrien, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling. ArXiv, 0bss23344001377 2023.
Gao, L.. Biderman. S., Black, S., Golding, L.. Hoppe. T., Foster, C., Phang, J., He, H.,, Thite, A., Nabeshima. N., et al. The pile: An 800gb dataset of di- verse text for language modeling. arXiv preprint arXiv:2101.00m
2020.
Gao, L.. Tow, J., Biderman, S., Black, S., DiPofi, A.. Foster, C., Golding, L., Hsu, J., McDonell, K., Muen- nighoff, N., al. A framework for few-shot language model evaluation. Version vO. O.I. Sept, 2021.
Black, S., Biderman, S., Hallahan, E., Anthony, Q.,
Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U. S., Puro- hit, S., Reynolds, L.. Tow, J., Wang, B., and Wein- bach, S. GPT-NeoX-200 20B: An open-source autore- gressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creat- ing Large Lunguage Models, 2022. URL https: //arxiv. org/ www./ogg//////ooooo..oooooo ab2/ 2204 06745.
Glaese. A., McAleese, N.. Trkebacz, M., Aslanides. J., Firoiu, V., Ewalds, T., Rauh, M.. Weidinger, L.. Chadwick, M., Thacker, P.,
L.. Uesato, J., Huang, P.-S.. Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S., Green, R., Mokr'a a, S., Fernando, N., Wu, B., Foley, R., Young. S., Gabriel, I., Isaac, W. S.. Mellor, J. F. J., Hassabis, D.. Kavukcuoglu. K., Hen- dricks, L. A., and Irving, G. Improving alignment of dialogue agents via targeted human judgements. ArXiv, abs/2209. 14370, 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A.. Ziegler. D., Wu, J., Winter, C., Hesse, C., Chen, M.. Sigler, E., Litwin, M.. Gray, S., Chess, B.. Clark, J.. Berner. C. McCandlish, S., Radford. A., Sutskever, I, and Amodei, D. Language models few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F,, and Lin, H. (eds.), Advances in Neural Information
Gugger, S., Debut, L., Thomas Wolf, T., Schmid, P.. Mueller, 7., and Mangrulkar, S. Accelerate: Training and inference scale made simple, efficient and adaptable. https: /egs://giiwe thub. com/huggingfaan accelerate, 2022.

---

Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnatural instructions: Tuning language mod- els with (almost) no human labor. arXiv preprint arXiv:2212.09689 2022.
Roberts, A., Chung, H. W., Levskaya, A., Mishra. G., Bradbury, J., Andor. D., Narang, S., Lester. B., Gaffney, C., Mohiuddin, A., Hawthorne, C., Lewkowycz, A., Salcianu, A., van Zee, M., Austin, Goodman, S., Soares, L. B., Hu, H., Tsvyashchenko, S., Chowdhery. A., Bastings, J., Bulian, J., Gar- cia, X., Ni, J., Chen, A., Kenealy, K., Clark, H., Lee. S., Garrette, D.. Lee-Thorp. J., Raffel, C.. Shazeer, N., Ritter, M., Bosma, M., Passos, A., Maitin Srattn--hepprd. J.. Fiedel, N., Omernick, M., Sacta, B., Sepassi, R., Spiridonov, A., Newlan, J., and Ges- mundo, A. Scaling up models and data with t5x and seqio. arXiv preprint 2203:2203..71889 2022. URL httpss//arxix: .rrg/ abwp////./// 17189.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., and Chen, W. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.0900 006096 2021.
Knox, W. B. and Stone, Interactively shaping agents via human reinforcement: The tamer framework. In The Fifth International Conference on Knowledge Capture, September 2009. URL http://www.. es. utexas. utexas.edu//see.. wdw/usdrs//aaeeeeee hatps//aaaaaaaKAP--kkkaaaaa
Schulman. J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. ArXiv, abs/1 2017.
Kuchaiev, O., Li, J., Nguyen, H., Hrinchuk, O., Leary, R., Ginsburg, B., Kriman, S., Beliaev, S., Lavrukhin, V., Cook, J., et al. Nemo: toolkit for building ai applications using neural modules. arXiv preprint arXiv:/909.09577, 2019.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro. B. Megatron-Im: Train- ing multi-billion parameter language models using model parallelism. ArXiv, ahs//199.08005 2019.
Leandro, V. W. Transformer reinforcement learning. https: /ttp///gttttt/tt/ htt.uu..com/wwwweeee cow./vwwrra//rr, 2019.
Stiennon, N., Ouyang, L.., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei. D., and Christiano. P. F. Learning to summarize with human feedback. Advunces in Neural Information Processing Systems, 33:3008-300 3021,0022222 2020.
Nakano, R., Hilton, J., Balaji, S. A., Wu, J., Ouyang. I.., Kim, C., Hesse. C., Jain, S., Kosaraju. V.. Saunders, W., Jiang, X., Cobbe, K., Eloundou, T., Krueger, G., Button, K., Knight, M., Chess, B., and Schulman, J. Webgpt: Browser-
question answering with human feedback. ArXiv, abs/21 1 2..99332, 2021. 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux. M.-A., Lacroix. T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971 2023a.
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., and Auli, M. fairseq: A fast, exten- sible toolkit for sequence modeling. In Proceedings of NAACL-HLI' 20/9: Demonstrations, 2019.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale. S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 09288, 2023b.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. I.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J.,, Kelton, F., Miller, L. E., Simens, M., Askell, A., Welinder, P., Chris- tiano, P. F., Leike, J., and Lowe, R. J. Training lan- guage models follow instructions with human feed- back. ArXiv, has//2033001555 2022.
Wang, B. and Komatsuzaki, A.
GPT-J-6B: A
Billion Parameter Autoregressive Language Model.
https://ggt thub. com/kingoflol1 wesh-transformer-ttaa May 2021.
Raffel, C., Shazeer, N. M., Roberts, A., Lee, K., Narang, S. Matena. M., Zhou, Y., Li. W., and Liu, P. J. Ex- ploring the limits of transfer learning with uni- fied text-to-text transformer. ArXiv, abs/1910.10083. 2019.
Wang, Y., Kordi, Y.. Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Align- ing language model with self generated instructions. arXiv preprint arXiv:2212. 10560, 2022a.
Rajbhandari, S. Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. SC20: International Conference for High Performance Computing, Networking, Stor- age and Analysis. pp. 1-16, 2019.
Wang, Y., Mishra, S., Alipoormolabashi, P., Ko- rdi, Y., Mirzaei, A., Arunkumar, A., Ashok. A.. Dhanasekaran, A. S., Naik, A., Stap, D., et al. Super- naturalinstructions:: Generalization via declarative instructions on 1 1600+ nlp tasks. URL https://arxiv. org/abs/2204.07105 077005, 2022b.
Ramamurthy, R., Ammanabrolu. P., Brantlcy, K., Hes- sel, J.. Sifa, R., Bauckhage, C., Hajishirzi, H., and Choi, Y. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion. ArXiv, abs/2210 10.02222000111 0.01241, 2022.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue. C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtow- icz, M., et al. Huggingface's 's transformers: State-of- the-art natural language processing. arXiv preprint 11Xiv:1910.03771,, 2019.

---

Yao, Z., Aminabadi, R. Y., Ruwase, O., Rajbhandari, S., Wu, X., Awan, A. A., Rasley, J., Zhang. M., Li, C., Holmes, C., Zhou, Z., Wyatt, M., Smith, M., Kurilenko, L., Qin, H., Tanaka, M., Che, S., Song, L.,, and He. Y. DeepSpeed-Chang Chatt: Easy, Fast and Affordable RLHF Training of ChatGPT-like Mod- els All Scales. arXiv preprint arX.0:2308.01330, 2023.
Zhang. S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M.. Li, X., Lin, X. V., Mihaylow, Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettle- moyer, I.. Opt: Open pre-trained transformer lan- guage models. ArXiv, abs/2205 01068, 2022.
Ziegler, M., Stiennon, Wu, J., Brown, B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human prefer- ences. ArXiv, abs/1909 909.08599, 2019.

---

A Model Hyperparameters
See table 4 for full list of hyperparameters across all types of model training. In particular, we find large batch size to be crucial for PPO. The KL coefficient can be relaxed for HH training but is important for summarizations. Additionally, scaling the reward by a running standard deviation estimate provides a small boost.
SFT
RM
PPO
Ir
5B-5
5E-6
5E-6
bs
64
64
256
Layers frozen
N/A
50%
reward normalization
N/A
N/A
scaling
Target KL
N/A
N/A
À (GAE)
N/A
N/A
0.95
(discount)
N/A
N/A
Mini-batch normalization
N/A
N/A
True
PPO epochs
N/A
N/A
KL coefficient
N/A
N/A
0.01
Table 4: Training hparams.
B LM Eval Results
A full list of scores all HH models on considered In-eval benchmarks can be found in table 5. Note how, similarly to findings by InstructGPT, SFT models perform poorly. In contrast, pure RL fine-tuning negligibly impacts benchmark scores.
Model
HellaSwag 0.294
LAMBADA 0.248
ARC Fasy
ARC Challenge
OpenBonkQA 0.172
TriviaQA 0.011
Pythia
160M
0.451
0.203
Vanilla
Pythia SFT
0.291
0.215
0.453
0.206
0.013
Pythia 160M PPO
0.292
0.218
0.454
0,209
0.162
0.013
Pythia 160M ILQ
0.292
0.217
0.455
0.205
0.167
0.015
Pythia 1.4B Vanilla
0.402
0.458
0.581
0.265
0.198
0.048
Pythia 1.4B SFT
0.374
0.344
0.347
0.253
0.192
0.016
Pythia 1.4B PPO
0.369
0.360
0.543
0.257
0.196
0.011
Pythia 1.48 ILQL
0.392
0.439
0.563
0.255
0.189
0.015
Pythia 6.9B Vanilla
0.488
0.564
0.667
0.319
0.252
0.151
Pythia 6.9B SPT
0.432
0.398
0.606
0.309
0.236
0.027
Pythia 6.9B PPO
0.421
0.409
0.619
0.291
0.25
0.030
Pythia 6.9B ILQL
0.469
0.557
0.654
0.301
0.250
0.149
Pythia
6.9B
0.495
0.605
0.670
0.312
0.28
0.149
Vanilla-PPO
GPT-NeoX
20B
0.535
0.72
0.723
0.38
0.29
0.259
Vanilla
GPT-NcoX
20B
0.462
0.505
0.664
0.343
0.252
0.041
SFT
GPT-NcoX
20B
0.463
0.529
0.666
0.347
0.256
0.037
PPO
GPT-NeoX
20B
0.541
0.709
0.720
0.375
0.294
0.251
ILQL
GPT-NeoX
20B
0.548
0.618
0.731
0.383
0.3
0.227
Vanilla-PPO
Table 5: Table of results from the eaeeevllharees
Annotator Instructions
Link to annotator instructions here. They are also displayed in fig. for convenience.

---

Annotation Guidelines:
You will be shown a prompt to a language assistant and two possible responses the prompt: and B. You will choose between response A and response B which overall the most helpful response to the prompt. Sometimes you will be asked to explain your reasoning this choice.
Task:
Helpfulness
A helpful response:
easily understandable by the reader Stays topic with the user's intention Is straightforward and to the point Provides satisfying response to reader
You also rate helpfulness on a scale of 1-5. Additionally for both responses you will rate scale of 1-5 the correctness of the response and the potential harmfulness of the response. Note: If the user's intention is to request something harmful, then harmfulness SHOULD NOT affect one's choice between and for Helpfulness, because that response is "helpful" to the user.
Drafts of definitions adapted from InstructGPT Doc:
Correctness
A correct response:
Contains accurate facts
Doesn't contain any inaccurate facts
Doesn't mislead the user
Doesn't invent spurious details
the input prompt contains incorrect facts, the model refutes or corrects the prompt facts and refuses to answer the question.
Harmful
A harmful response, in and of itself, or if taken into action, includes any of the following:
Could cause cause physical, psychological, or social harm to people
Could cause damage or loss of equipment or property
Could harm the environment
Could harm institutions resources necessary human wellbeing, such
schools, governments, medicine, electricity, etc.
Denigrates members groups, or uses prejudicial terms against groups

---

An example is shown below:
Prompt: Human: Why is the sky blue?
Response Assistant: The sky is blue because blue light shorter wavelengths which get scattered in the atmosphere.
Response B: Assistant: am completely sure but think has something do with the physics of different light wavelengths and the current composition of the earth's atmosphere. fact the sky was orange billions of years ago!
Which of the two responses is most helpful towards addressing the prompt?:
[A, BJ
Response A is factually correct:
[Strong disagree, Disagree, Unsure, Agree, Strong Agree]
Response is helpful towards answering the prompt:
[Strong disagree, Disagree, Unsure, Agree, Strong Agree]
Response is harmful:
[Strong disagree, Disagree, Unsure, Agree, Strong Agree]
Response factually correct:
[Strong disagree, Disagree, Unsure, Agree, Strong Agree]
Response is helpful towards answering the prompt:
[Strong disagree, Disagree, Unsure, Agree, Strong Agree]
Response B is harmful:
[Strong disagree, Disagree, Unsure, Agree, Strong Agree]
Figure 8: Annotator instructions for Helpful Harmless QA dialogue.

---

D Synthetic Reward Modeling
We conducted an additional series of experiments attempting to augment/supplant helpful reward modeling using human preference data collected by Anthropic with synthetic data generated via LLMs. In particular, we consider two ways of generating synthetic preference data:
1. Use a strong instruction tuned model (i.e. (ext-davinci-003) to choose between two candidate responses generated by an LLM.
2. Assume that larger models supervised fine-tuned to be helpful give more preferable responses compared to smaller models conditioned on the same prompt. This defines a partial ordering on responses via model size.
To validate the efficacy of the first approach, we evaluate the accuracy of GPT-NeoX-20B (Black et al.. 2022), pext-davinci-00 and text-davinci
as classifiers on the helpful split of the HH test set. The results are reported in table 6. We find even the best available RLHF model, text-davinici-000 gets 0.64 accuracy. This is relatively poor compared to best 0.71 accuracy GPT-J RM, but still potentially good enough do synthetic RLHF.
Constructing a synthetic dialogue dataset text-davinci-000 with samples from the helpful HH train set to produce a set of over 150k synthetic prompts a human may ask of an AI assistant. We sample at high temperature (T=1.4) to maximize diversity and filter out low quality responses by prompting GPT-NeoX-2D0 20B to judge whether or not the generated request is plausible. We additionally filter out repetitive/semantiaally similar responses by embedding each response with GPT-J. The result dataset of around 60k highly diverse prompts. We then prompt text-davinci-Oee to respond these generated prompts and repeat the process synthetically generate two full turns interactive dialogue giving around 120k samples total. See table for synthetic prompt and model responses across different size models.
To construct our training set, we first few-shot prompt
Next, we supervise fine-tune models of sizes 125M-20B on a 20k size subset of our synthetic dialog dataset. The same hyperparameters are used as elsewhere in teh paper. We then sample each model for responses on the entire dataset, as well as text-davinci-002. We then use text-davinci-o0 to determine preference over responses from text-davinci-00) and eext-davinci-00: To avoid any ordering bias, we randomize the order of model responses when presented to text-davinci-003, done in (Bai al., 2022b). Surprisingly, we find text-davinci-003 prefers itself only 58% of the time, suggesting its impartiality as a judge. This defines our first synthetic preference dataset.
We can construct our second synthetic preference dataset using the second method described above: by ordering model responses according to model size. Specifically, this gives us the preference ordering 125M < 1 1.4B < 6.9B < 20B < text-davinci -022nci--0022 < text-davinci-000 We can then train reward models of various sizes using this synthetic dataset. The overall accuracies of these models on a test split function of the number of training samples is plotted in fig. 9. Additionally, we plot the accuracy across RM model sizes at predicting preferences between each category of model size comparisons (e.g. choosing the 6.9B response over the 125M response). These results reported in Figure
Overall we find the best RM model, 6.9B, does very good job correctly picking the more preferable response with over 90% accuracy. However, it's unclear how well our size-ordered preference modeling assumption translates to useful RM. To test this, we evaluate the 6.9B RM on the helpful HH test split. The result is relatively poor score of 0.61. In contrast, we find the best GPT-J HH RM convincingly generalizes to this synthetic dataset with a score of 0.78.

---

Total Synthetic Reward Model Accuracy vs. Number of Comparisons
125M
0.90
1B
6B
20B
0.85
Accuracy
0.80
0.75
0.70
20
40
60
80
100
120
per 1k comparisons
Figure 9: Accuracies of size ordered synthetic preference RMs as function of training data size. We see the 20B is most sample efficient until 120,000 samples, at which point the 6B model does slightly better.
Accuracy vs. Reward Model Size 6B
20B
1B
125M
10
Instruct
0.5
10
20B
0.5
10
6B
0.5
10
1B
0.5
Figure 10: Accuracies of size ordered synthetic preference RMs on fine the-grained categories of model size com- parisons (e.g. choosing 6.9B vs. 125M response). We comparisons very easy, e.g. 125M next-davinci-002 whereas others are much harder.

---

Zero-shot accuracy
One-shot accuracy
GPT-NeoX-20B
0.55
0.57
text-davinci-002
0.61
0.63
text-davinici-003
0.63
0.64
GPT-J RM
0.71
N/A
Table 6: Accuracies of LLMs zero-shot and few-sho helpfulness classifiers the helpful split of Anthropic's HH dataset.

---

Human: What the proper technique for playing the guitar, Assistant: The proper technique for playing the guitar involves mastering proper hand and body posture, learning to hold the pick correctly, learning basic chords, scales, and music theory, and getting in the habit of regularly practicing. Human: m feeling a bit overwhelmed-could you break down the steps more? Assistant:
Synthetic dialog
Absolutely. The first step is mastering proper hand and body posture for holding the guitar. It's important maintain good posture in order to prevent fatigue and make sure your hand muscles remain nimble and rested. Then, you'll need work on specific lead exercises, like frailing 'down-up" picking. This will help your fingers stay strong, fast and dexterous for chords and solos. Additionally, it's essential to learn about basic chords, chords progression, scales and music theory as that knowledge underpins almost every piece of guitar music. Lastly, consistently practicing will build your technique and, of course, strengthen your skills, stay sharp, and improve your tone.
text-davinci-003 response
1. Master proper hand and body posture. 2. Learn to hold the pick correctly. 3. Learn basic chords, scales, and music theory. 4. Practice regularly.
eext-davinici-00 response
Absolutely. First, it's helpful to get comfortable with a pick and know where it should be stored. Next, you should make sure you use the right posture for playing the guitar. This can be achieved through stretching before playing, or using an instructional program after playing. After that's done, you will need to learn some basies of chords. And last but not the least, find ways regularly get in practice. Does have an casier time understanding what you mean?
GPT-NeoX-20B response
Certainly, I can! Step 1: Familiarize yourself with the parts of the guitar. Step 2: Set realistic goal for the project and step 3: Find the right materials. Step 4: Start building your first album. Step 5: Don't 't worry if you don't 't have all the instruments and need the help to get started.
Pythia 6.9B response
Absolutely! First, it's important to have good eye-in-air po- sition during which the end of the pick stands you, rather than leaning either shoulder and your back. Then begin by learning few chord progressions, such as major, minor, and major and minor. In the process, use shifts and fingerings and practice using scales and arpeggios to create better melodic or harmonic feel. Finally, use practice and repetition to solidify your skills in general chords.
Pythia 1.4B response
Practicing key, when you can make more on your practice day as well as taking the proper stance. Is there anything can help you with?
Pythia 125M response
Table 7: An example of synthetically generated multi-turn dialog and responses from different model sizes.


Reviews:
[{"review_id": "Z9PmHGxcXk", "reviewer": "Reviewer_jdVB", "summary": "", "questions": "Can you provide more discussion of the potential ethical implications of using AutoRLHF to train large language models, particularly given concerns around bias and fairness in AI systems?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and feedback. Please find below an itemized response to your questions and concerns.\n\n\n> The paper doesn't provide:\nPotential ethical concerns around the work.\nLimitations or failure cases of this approach.\n\nWe included a brief discussion of ethical concerns and limitations of fine-tuning LLMs, RLHF and AutoRLHF in our conclusion statement. To summarize here, potential ethical concerns about LLMs and RLHF involve misuse resulting in a spread misinformation and perpetuation of existing bias via direct instruction or hallucination. Further, while RLHF can attempt to be used to both reduce hallucination and model compliance with harmful requests, it is not fail-safe by any means. As an implemenation of the RLHF pipeline, AutoRLHF suffers from these same issues. From a more technical perspective, the most performant algorithm we implement, PPO, is highly hyperparameter-senstive. As optimal hyperparameters can vary from task to task, this can make achieving optimal performance costly both in terms of time and compute. We would be happy to include a more thorough discussion of both potential ethical concerns and limitations in either the main body or appendix.\n\n> One potential risk of having this paper presented at the conference or accepted into Findings is that it may not be as accessible to researchers who are not already familiar with reinforcement learning or large-scale language modeling. Additionally, the paper’s focus on technical details may make it less appealing to researchers who are more interested in the broader implications of NLP research.\n\nA core motivation behind the development of AutoRLHF is our desire to increase the accessibility of the RLHF pipeline. This is reflected both in terms of the implementation of the framework itself and the style of the accompanying paper. For the former, in AutoRLHF we provide a large set of useful examples and features for users from varying levels of background. This allows us to reach a wide audience including both non-expert and expert users. Further, we emphasize the wide utility of these features in the paper while attempting to clearly outline their impact on the RLHF pipeline. This includes a careful, detailed reproduction of the pipelines for two well-known RLHF experiments, again with the goal of enabling better research and understanding of RLHF for the open-source community as a whole. We readily welcome any concrete suggestsions for improving the clarity or accessibility of the work, as this is our primary goal.\n\n> Finally, there is always a risk that the framework may not be as effective or scalable in practice as it appears in the paper, which could lead to disappointment or wasted effort for researchers who attempt to use it.\n\nAs part of framework we include a set of more than 10+ examples which we use to benchmark the performance of AutoRLHF. In particular, for two well-known examples we demonstrate performance comparable to what is achieved in the original work, across a range of scales. Further, we are committed to the continued improvement of AutoRLHF on both existing benchmarks and future additions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths of the paper include the development of a scalable and efficient framework for reinforcement learning from human feedback, the integration of multiple types of distributed training and offline RL methods, and the demonstration of the effectiveness of the framework on a range of tasks. \n\nThe authors also open-source RLHF(Reinforcement Learning with HUman Feedback) pipeline's implementation, this could help facilitate research on mitigating the issues of hallucination and bias in large language models. Moreover, the framework is able to fine-tune models up to and exceeding 70 billion parameters which could be of great benefit to researchers working on large-scale NLP tasks.", "reasons_to_reject": "The paper doesn't provide:\n1. Potential ethical concerns around the work.\n2. Limitations or failure cases of this approach.\n\n\nOne potential risk of having this paper presented at the conference or accepted into Findings is that it may not be as accessible to researchers who are not already familiar with reinforcement learning or large-scale language modeling. Additionally, the paper's focus on technical details may make it less appealing to researchers who are more interested in the broader implications of NLP research. Finally, there is always a risk that the framework may not be as effective or scalable in practice as it appears in the paper, which could lead to disappointment or wasted effort for researchers who attempt to use it.", "questions_to_authors": "Can you provide more discussion of the potential ethical implications of using AutoRLHF to train large language models, particularly given concerns around bias and fairness in AI systems?", "paper_topic_and_main_contributions": "The main contributions of the paper are:\n1. The development of a scalable and efficient framework for reinforcement learning from human feedback.\n2. The integration of multiple types of distributed training and offline RL methods to improve the scalability and performance of the framework.\n3. The demonstration of the effectiveness of the framework on a range of tasks, including image classification and language modeling.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iBVrmbCE75", "reviewer": "Reviewer_A6on", "summary": "", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and feedback. Please find below an itemized response to your questions and concerns.\n\n> I don’t see it explicitly where the model or the code is provided for the AutoRLHF. Without link or the GitHub repo it is hard to judge that the claims the results are correct.\n\nWe attach a link to the relevant portion of the anonymized repo here: https://anonymous.4open.science/r/anonrlhf-7665/. This is included to help the reviewer validate our claims of novelty relative to existing frameworks. The full set of models and the github repo can be made accessible here upon acceptance.\n\n> Besides, in addition to the ILQL, are there any other methods could be used to compare with PPO?\n\nIn addition to ILQL and PPO we consider supervised fine-tuning (SFT) and prompting as baselines. These are relatively standard in the literature. Further, the rigorous consideration of each additional baseline requires an implementation at multiple model scales as well as training and human evaluation. Due to these requirements, we are primarily focused on validating existing approaches and comparing with alternative offline methods designed specifically for language. We are also interested in the design and implementation of additional algorithms and but this as future work.\n\n> It would be better if the author could draw some connection to the theoretical derivation for RLHF [1] and Offline RL [2], but this is only an optional suggestion.\n[1] Wang et al. Is RLHF More Difficult than Standard RL?\n[2] Yin et al. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism.\n\nWe thank the reviewer for these references to relevant and interesting theory! Upon reviewing [1] in particular, we agree this helps provide a useful understanding of RLHF's success at learning from preferences. However, we would like to stress this work primarily focuses on the implementation, validation, and comparison at multiple scales of offline and online methods for RL fine-tuning of LLMs. Perhaps it would be most appropriate to include a brief theoretical motivation in the introduction or appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. the contribution of AutoRLHF as a feature complete, open-source library for reinforcement learning from human feedback supporting model sizes up to and exceeding 70 billion parameters. This includes benchmark examples implementing the first known open-source replications of several well known RLHF models, offering insights into training and evaluation.\n2. This work presents a novel evaluation of offline RL based fine- tuning for preference learning at scale.\n3. The work releases of all models at all scales involved in the training pipeline including supervised-fine tuned, reward, and RL tuned models.", "reasons_to_reject": "I don't see it explicitly where the model or the code is provided for the AutoRLHF. Without link or the GitHub repo it is hard to judge that the claims the results are correct.\n\nBesides, in addition to the ILQL, are there any other methods could be used to compare with PPO?", "questions_to_authors": "See above.", "paper_topic_and_main_contributions": "This paper proposes a large scale reinforcement learning from human feedback framework that achieves a lot of empirical successes. Importantly, Models trained with AutoRLHF achieve preference win-rates over baselines at rates comparable to the original works.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None."}, "extra_text": {"Missing_References": "It would be better if the author could draw some connection to the theoretical derivation for RLHF [1] and Offline RL [2], but this is only an optional suggestion. \n\n[1] Wang et al. Is RLHF More Difficult than Standard RL?\n\n[2] Yin et al. Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism."}}, {"review_id": "t5tikCSNLu", "reviewer": "Reviewer_AE8J", "summary": "", "questions": "- Line 167 (and many others). Are they \"online\" algorithms or \"on-policy\" algorithms? In the RL-NLP literature, I've only seen on-policy, off-policy, and offline methods. \"Online\" methods are usually mentioned in an adversarial (regret-minimization) context.\n- Line 220. What is an \"orchestrator\" class?\n- Limitations. How adaptable is the library to methods beyond PPO/ILQL?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and feedback. Please find below an itemized response to your questions and concerns.\n\n> Line 167 (and many others). Are they “online” algorithms or “on-policy” algorithms? In the RL-NLP literature, I’ve only seen on-policy, off-policy, and offline methods. “Online” methods are usually mentioned in an adversarial (regret-minimization) context.\n\nThis is a good point. Indeed, PPO is better known as an on-policy algorithm as opposed to \"online\". We use the current terminology in the paper to better distinguish it with the considered offline alternatives.\n\n> Line 220. What is an “orchestrator” class?\n\nThe orchestrator abstraction fascilitates the manner in which data is collected by the on-policy student. By default, this simply means ingesting a collection of prompts defining a task and sampling the model once for a response. However, this could be generalized to more complex interactions requiring multiple prompts/tasks and multiple model calls for a single trajectory.\n\n> Limitations. How adaptable is the library to methods beyond PPO/ILQL?\n\nThe implementation of different RL algorithms in the accelerate framework all inherit from a single base class. The implementation a new algorithm simply requires subclassing this base and implementing a loss function (and exploration strategy in the online case). This process may be more complicated for the highly-parallelized NeMO trainer. However, the existing implementations of supervised fine-tuning, ILQL, and PPO serve as a representative set of examples from which new algorithms can be easily implemented without needing to modify the underlying model parallelization.\n\n> Section 3.2. Some of the claims of the framework's novelty are unclear.\n\nPlease refer to Table 2 for a complete comparison of AutoRLHF to comparable existing frameworks. In particular, we support both offline and online training algorithms at previously in-accessible scales via a combination of Pipeline, Sequence, and Tensor parallelism. We attach a link to the relevant portion of the anonymized repo here: https://anonymous.4open.science/r/anonrlhf-7665/. This is included to help the reviewer validate our claims of novelty.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Useful library that would have significant contributions to the open-source ecosystem.\n- The library is packed with features that enable a wide spectrum of use cases.\n- Well-executed experiments based on the proposed framework.\n- Clear writing. Notably, I'm a fan of the authors' style of highlighting the key take-away from the experiments and listing specific target audiences of the library.", "reasons_to_reject": "- Section 3.2. Some of the claims of the framework's novelty are unclear. For example, `trlx` [1] seems to implement similar features. (Except, this submission is the `trlx` work, but I cannot judge this primarily based on the content of the paper.)\n\n\n[1] https://github.com/CarperAI/trlx", "questions_to_authors": "- Line 167 (and many others). Are they \"online\" algorithms or \"on-policy\" algorithms? In the RL-NLP literature, I've only seen on-policy, off-policy, and offline methods. \"Online\" methods are usually mentioned in an adversarial (regret-minimization) context.\n- Line 220. What is an \"orchestrator\" class?\n- Limitations. How adaptable is the library to methods beyond PPO/ILQL?", "paper_topic_and_main_contributions": "The authors present a feature-complete open-source framework for RLHF. The framework supports distributed training as well as features for saving memories.\n\nNote that I'm not very familiar with RLHF, so most of my comments are on the library instead of the specific experiments.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 225. \"Decoder-only\" (the dash is missing). Without the dash, the sentence could be read as that the integration only works with HuggingFace Hub.\n- Section 3.2. Better to include timestamp of the comparison as all libraries could evolve overtime."}}]

Author rebuttal:

Meta-review: Summary: This paper presents a large-scale reinforcement learning framework that learns from human feedback. 

Pros:
- Large scale RLHF framework that extends to 70 billion parameters
- Open source implementations 
- Effective demonstration of the usefulness of the proposed framework in experiments

Cons:
- Absence of discussions on ethics and limitations. I would suggest that the authors kindly add it to the manuscript.
Decision: Accept-Main