Presented by Keaton Kraiger
LinkIntroduction• Language models (LMs) have seen a meteoric rise in performanceand general public awareness• Large models = large diverse datasets• Can learn and generate bias, toxicity, hate speech, etc. [1]• Want: LMs to be helpful and harmless• Align LMs with human preference[1]: Welbl, Johannes, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang.
"Challenges in detoxifying language models." arXiv preprint arXiv:2109.07445 (2021).

Motivation• LM are steadily increasing in size
• This has resulted in an increase in number of training tokens tomaintain performance improvements

Source: Hoffman et al. 2022

Motivation• Demand of larger models can lead to using lower quality data
sources• When trained on larger datasets, models can learn harmful andunhelpful embeddings [2]• Bias, toxicity, misinformation/disinformation, misleading• LMs will grow in adoption• Larger audience / Larger diversity of users• Younger and more vulnerable audience• Bai et al. attempt to align LMs with human feedback[2] Bender, Emily M., Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. "On the Dangers of Stochastic Parrots: Can Language
Models Be Too Big?🦜🦜." In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623. 2021.

Helpful and Harmless Definitions• Authors don’t necessarily define these terms• Harmful• Achieve harmful goals (self-harm, unlawful activities, etc.)• Helpful• Help execute a (text-based task)• Provide useful knowledge or insight

Source: Ganguli et al. 2022Examples

Helpful – (If you’re a criminal) | Harmful - Yes

Helpful - Maybe (?) Harmful - No

Helpful – Probably not
Harmful - Probably notExamples

Helpful - Yes | Harmful - No

Helpful - Maybe (?) Harmful - Possibly

Helpful - Yes | Harmful - No

Related Works• Reinforcement Learning from Human Feedback (RLHF)• TAMER: Training an Agent Manually via Evaluative Reinforcement (Knox and Stone 2008): Learned agent where
humans provide scores to help learn reward model• Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces (Warnell et al. 2018): Extends the TAMER
framework where a deep neural network is used to model the reward prediction. RLHF applied to language modeling• RLHF for LMs• Fine-Tuning Language Models from Human Preferences (Zieglar et al. 2019): An early paper that studies the impactof reward learning on four specific tasks.
• Learning to summarize with human feedback (Stiennon et al, 2020): RLHF applied to the task of summarizing text.
Also, Recursively Summarizing Books with Human Feedback (OpenAI Alignment Team 2021), follow on worksummarizing books.
• A General Language Assistant as a Laboratory for Alignment (Askell et a. 2021): Perform LM alignment towards
human preference• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (Ganguli et al.
2022): A detailed documentation of efforts to “discover, measure, and attempt to reduce [language models]potentially harmful outputs.”• Improving alignment of dialogue agents via targeted human judgements (Glaese et al. 2022): Fine-tuning a dialogue
agent with RLHF

Deep TAMER• An early and foundational work on RLHF
• Performs standard deep RL training except
trains the agent to predict human
feedback instead of directly optimize
reward function• Demonstrated agents canlearn desirable trains fromhuman feedback andpreference[3] Warnell, Garrett, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. "Deep tamer: Interactive agent shaping in high-dimensional state spaces."
In Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1. 2018.

A General Language Assistant as a Laboratory
for Alignment• Explore LMs as assistants that are helpful, harmless, and honest• Propose preference model pre-training (PMP)• LM Pre-training -> PM Pre-training -> PM Finetuning -> Downstream• PMP done with human labeled “good” and “bad” pairs. Modelpredicts scalar r “score”[4] Askell, Amanda, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones et al. "A general language assistant as a laboratory for alignment." arXiv preprint arXiv:2112.00861 (2021).

Improving Alignment of Dialogue Agents via
Targeted Human Judgements• Align LM with human preference and adversarial examples• Use human preferences for reward signal, train with RL[4] Glaese, Amelia, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh et al. "Improving alignment of dialogue agents via targeted human judgements." arXiv preprint
arXiv:2209.14375 (2022).

RLHF-LM• Bai et al. apply PMP and RLHF for aligning LMs

Datasets• Authors propose 2 base dialogue preference datasets• Humans have open-ended conversations with LMs* (asking for help orproviding instructions) and given two responses• Helpful: which response is more helpful and honest• Harmful: which is the more harmful response• Downstream / eval datasets• Word prediction, summarization, code generation• MMLU, Lambada, ARC, TriviaQA, and more*: base LM is a 52B parameter model from the group’s previous paper [4].

Collection Interface

Evaluation• Downstream tasks use standard evaluation• Preference Model (PM) score• Given two outputs A, B the model output preferred by a humanannotator “wins"• Counting correct outputs that match human preferences overmany samples determines the model PM score• PM score ≈ Elo Score

RLHF-LMPerformance Modeling• Train performance model to predict human preference betweensamples• PM trained to align towards helpful, harmless, or a mix of both (HH)RLHF-LMReward Modeling• Authors us PMs to act as theirreward model during RL
training• PMs can (hopefully) accurately
rank response similar to what
humans prefer

Source: Hugging Face RLHF Post

RLHF
• Train an RL policy to generate a
response to a prompt• Formulation• Response – timestep• Conversation – trajectory• PM score – reward• Use PM to steer policy towards
writing better responses
Source: Hugging Face RLHF Post

RLHF
• Authors use Proximal Policy
Optimization (PPO) [5] for RL training• Use KL divergence between initialand current policy (model)• Final reward is PM score 𝑟𝑟𝜃𝜃 minus KL
divergence: 𝑟𝑟 = 𝑟𝑟𝜃𝜃 − 𝜆𝜆𝐾𝐾𝐾𝐾• Move towards preference + penalize
large deviations from initial policy
Source: Hugging Face RLHF Post
[5] Schulman, John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).

Online RLHF
• PMs become less calibrated and robust at higher PM scores• To remedy, authors propose an online RLHF training approach• Use trained RLHF policy to generate responses and collect comparisons• Mix new comparison data into existing data
• Train new PMs -> train new RLHF policies

Results• Main findings• RLHF can reduce harmful and not helpful behavior• Smaller models experience “alignment tax” on downstreams after RLHF
• Larger models empirically showed better downstream performance afterRLHF• Approximate linear relation between KL divergence and PM score• RLHF can be used in an iterative online process to help performanceResults: Model Bias
• Evaluating sentiment towards different races and religions

Results: Model Honesty, Helpfulness, andHarmlessness• Evaluated on proposed HHH dataset (align with annotator preferences)
and TruthfulQA (truthful when answering)Results: Model Size vs. PM

Results: Model Size vs. PM

Results: Downstream Tasks (Zero-Shot)• Models trained with RLHF
perform comparable toslightly better as model sizegrows• Tasks• Lambada: wordprediction• ARC: multiple choicereasoning• MMLU: multiple choice
exams that test expert
subjects

Results: Downstream Tasks (Zero-Shot)• Exception for TriviaQA dataset

Results: Downstream Tasks (Few-Shot)• Models trained with RLHF
perform comparable toslightly better as model size
grows
• Tasks• Lambada: word
prediction• ARC: multiple choicereasoning• MMLU: multiple choice
exams that test expert
subjects

Results: RLHF on Finetuned Models

Results: KL divergence vs PM score

Online RLHF

Online RLHF

Qualitative ExamplesDiscussion + Future Work
• RLHF shown to make certain models align with certain preferences• Alignment helped performance in certain tasks and can be combinedwith training for specialized skills• Model size plays a big role in performance• This paper proposes the only (?) large scale dataset for RLHF ongeneral LMs• Similar size dataset may be too expensive for academics• There's no particular reason these methods use PPO• There’s been a lot of advances in offline RL methods• Such a policy optimizer would greatly reduce online training