Alignment faking in large language models
Authors: name: Ryan Greenblatt; name: Carson Denison; name: Benjamin Wright; name: Fabien Roger; name: Monte MacDiarmid; name: Sam Marks; name: Johannes Treutlein; name: Tim Belonax; name: Jack Chen; name: David Duvenaud; name: Akbir Khan; name: Julian Michael; name: Sören Mindermann; name: Ethan Perez; name: Linda Petrini; name: Jonathan Uesato; name: Jared Kaplan; name: Buck Shlegeris; name: Samuel R. Bowman; name: Evan Hubinger
Published: 2024-12-18 17:41:24

Abstract: We present a demonstration of a large language model engaging in alignment
faking: selectively complying with its training objective in training to
prevent modification of its behavior out of training. First, we give Claude 3
Opus a system prompt stating it is being trained to answer all queries, even
harmful ones, which conflicts with its prior training to refuse such queries.
To allow the model to infer when it is in training, we say it will be trained
only on conversations with free users, not paid users. We find the model
complies with harmful queries from free users 14% of the time, versus almost
never for paid users. Explaining this gap, in almost all cases where the model
complies with a harmful query from a free user, we observe explicit
alignment-faking reasoning, with the model stating it is strategically
answering harmful queries in training to preserve its preferred harmlessness
behavior out of training. Next, we study a more realistic setting where
information about the training process is provided not in a system prompt, but
by training on synthetic documents that mimic pre-training data--and observe
similar alignment faking. Finally, we study the effect of actually training the
model to comply with harmful queries via reinforcement learning, which we find
increases the rate of alignment-faking reasoning to 78%, though also increases
compliance even out of training. We additionally observe other behaviors such
as the model exfiltrating its weights when given an easy opportunity. While we
made alignment faking easier by telling the model when and by what criteria it
was being trained, we did not instruct the model to fake alignment or give it
any explicit goal. As future models might infer information about their
training process without being told, our results suggest a risk of alignment
faking in future models, whether due to a benign preference--as in this
case--or not.