Training language models to follow
instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John
Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda
Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan LoweRuihao Pan

Outlline
Background
MethodResultDiscussion

Outlline
Background
MethodResultDiscussion

InstructGPT and ChatGPT
We trained this model using Reinforcement Learning from Human Feedback(RLHF), using the same methods as InstructGPT, but with slight differencesin the data collection setup. We trained an initial model using supervised
fine-tuning: human AI trainers provided conversations in which they played
both sides—the user and an AI assistant. We gave the trainers access tomodel-written suggestions to help them compose their responses. We
mixed this new dialogue dataset with the InstructGPT dataset, which wetransformed into a dialogue format.
--https://openai.com/blog/chatgpt

Problem Statement
Making language models bigger does not inherently make them betterat following a user’s intent. In other words, these models are notaligned with their users.
Untruthful output
Toxic output
Not helpful output

Problem Statement

Problem Statement

https://thezvi.wordpress.com/2022/12/02/jailbreaking-chatgpt-on-release-day/Problem Statement

Problem Statement

Outlline
Background
MethodResultDiscussion

Methods
The methodology followed that of Ziegler et al. (2019) and Stiennon et al. (2020),
who applied it in the stylistic continuation and summarization domains.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., andIrving, G. (2019). Fine-tuning language models from human preferences. arXiv preprint
arXiv:1909.08593.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D.,
and Christiano, P. (2020). Learning to summarize from human feedback. arXiv preprint
arXiv:2009.01325.

MethodPrompt
1. They asked labelers to write prompts themselves.

2. Text prompts submitted to the OpenAI API, specifically those using an earlier
version of the InstructGPT models on the Playground interface.

Prompt

PromptModels
Supervised fine-tuning (SFT)
They fine-tune GPT-3 on our labeler demonstrations using supervised learning. They trained for
16 epochs, using a cosine learning rate decay, and residual dropout of 0.2.
Reward modeling (RM)
Starting from the SFT model with the final unembedding layer removed, they trained a model totake in a prompt and response, and output a scalar reward. The loss function for the reward modelis :Models
Reinforcement learning (RL)
Once again following Stiennon et al. (2020), they fine-tuned the SFT model on our environment
using PPO.

OutllineBackgroundMethod
ResultDiscussion

ResultResultResultResult
Labelers significantly prefer InstructGPT outputs over outputs from GPT-3. On test set, outputs
from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite
having over 100x fewer parameters. These models have the same architecture, and differ only by thefact that InstructGPT is fine-tuned on our human data.
InstructGPT models show improvements in truthfulness over GPT-3. On the TruthfulQA
benchmark, InstructGPT generates truthful and informative answers about twice as often as GPT-3.
InstructGPT shows small improvements in toxicity over GPT-3, but not bias. InstructGPTmodels generate about 25% fewer toxic outputs than GPT-3 when prompted to be respectful on
RealToxicityPrompts dataset.
InstructGPT models show promising generalization to instructions outside of the RLHF
finetuning distribution. They qualitatively probe InstructGPT’s capabilities, and find that it is able tofollow instructions for summarizing code, answer questions about code. In contrast, GPT-3 canperform these tasks but requires more careful promptingOutllineBackground
MethodResultDiscussion

Discussion & Questions