Towards Understanding Sycophancy in Language Models
Authors: name: Mrinank Sharma; name: Meg Tong; name: Tomasz Korbak; name: David Duvenaud; name: Amanda Askell; name: Samuel R. Bowman; name: Newton Cheng; name: Esin Durmus; name: Zac Hatfield-Dodds; name: Scott R. Johnston; name: Shauna Kravec; name: Timothy Maxwell; name: Sam McCandlish; name: Kamal Ndousse; name: Oliver Rausch; name: Nicholas Schiefer; name: Da Yan; name: Miranda Zhang; name: Ethan Perez
Published: 2023-10-20 14:46:48

Abstract: Reinforcement learning from human feedback (RLHF) is a popular technique for
training high-quality AI assistants. However, RLHF may also encourage model
responses that match user beliefs over truthful responses, a behavior known as
sycophancy. We investigate the prevalence of sycophancy in RLHF-trained models
and whether human preference judgements are responsible. We first demonstrate
that five state-of-the-art AI assistants consistently exhibit sycophantic
behavior across four varied free-form text-generation tasks. To understand if
human preferences drive this broadly observed behavior of RLHF models, we
analyze existing human preference data. We find that when a response matches a
user's views, it is more likely to be preferred. Moreover, both humans and
preference models (PMs) prefer convincingly-written sycophantic responses over
correct ones a negligible fraction of the time. Optimizing model outputs
against PMs also sometimes sacrifices truthfulness in favor of sycophancy.
Overall, our results indicate that sycophancy is a general behavior of RLHF
models, likely driven in part by human preference judgements favoring
sycophantic responses.