Specific versus General Principles for Constitutional AI
Authors: name: Sandipan Kundu; name: Yuntao Bai; name: Saurav Kadavath; name: Amanda Askell; name: Andrew Callahan; name: Anna Chen; name: Anna Goldie; name: Avital Balwit; name: Azalia Mirhoseini; name: Brayden McLean; name: Catherine Olsson; name: Cassie Evraets; name: Eli Tran-Johnson; name: Esin Durmus; name: Ethan Perez; name: Jackson Kernion; name: Jamie Kerr; name: Kamal Ndousse; name: Karina Nguyen; name: Nelson Elhage; name: Newton Cheng; name: Nicholas Schiefer; name: Nova DasSarma; name: Oliver Rausch; name: Robin Larson; name: Shannon Yang; name: Shauna Kravec; name: Timothy Telleen-Lawton; name: Thomas I. Liao; name: Tom Henighan; name: Tristan Hume; name: Zac Hatfield-Dodds; name: Sören Mindermann; name: Nicholas Joseph; name: Sam McCandlish; name: Jared Kaplan
Published: 2023-10-20 20:12:45

Abstract: Human feedback can prevent overtly harmful utterances in conversational
models, but may not automatically mitigate subtle problematic behaviors such as
a stated desire for self-preservation or power. Constitutional AI offers an
alternative, replacing human feedback with feedback from AI models conditioned
only on a list of written principles. We find this approach effectively
prevents the expression of such behaviors. The success of simple principles
motivates us to ask: can models learn general ethical behaviors from only a
single written principle? To test this, we run experiments using a principle
roughly stated as "do what's best for humanity". We find that the largest
dialogue models can generalize from this short constitution, resulting in
harmless assistants with no stated interest in specific motivations like power.
A general principle may thus partially avoid the need for a long list of
constitutions targeting potentially harmful behaviors. However, more detailed
constitutions still improve fine-grained control over specific types of harms.
This suggests both general and specific principles have value for steering AI
safely.