Towards Understanding Sycophancy in Language Models
Authors: name: Mrinank Sharma, name: Meg Tong, name: Tomasz Korbak, name: David Duvenaud, name: Amanda Askell, name: Samuel R. Bowman, name: Newton Cheng, name: Esin Durmus, name: Zac Hatfield-Dodds, name: Scott R. Johnston, name: Shauna Kravec, name: Timothy Maxwell, name: Sam McCandlish, name: Kamal Ndousse, name: Oliver Rausch, name: Nicholas Schiefer, name: Da Yan, name: Miranda Zhang, name: Ethan Perez

Reinforcement learning from human feedback (RLHF) is a popular technique for
training high-quality AI assistants. However, RLHF may also encourage model
responses that match user beliefs over truthful responses, a behavior known as
sycophancy. We investigate the prevalence of sycophancy in RLHF-trained models
and whether human preference judgements are responsible. We first demonstrate
that five state-of-the-art AI assistants consistently exhibit sycophantic
behavior across four varied free-form text-generation tasks. To understand if
human preferences drive this broadly observed behavior of RLHF models, we
analyze existing human preference data. We find that when a response matches a
user's views, it is more likely to be preferred. Moreover, both humans and
preference models (PMs) prefer convincingly-written sycophantic responses over
correct ones a negligible fraction of the time. Optimizing model outputs
against PMs also sometimes sacrifices truthfulness in favor of sycophancy.
Overall, our results indicate that sycophancy is a general behavior of RLHF
models, likely driven in part by human preference judgements favoring
sycophantic responses.