Suppressing Pink Elephants with Direct Principle Feedback
Authors: name: Louis Castricato, name: Nathan Lile, name: Suraj Anand, name: Hailey Schoelkopf, name: Siddharth Verma, name: Stella Biderman

Existing methods for controlling language models, such as RLHF and
Constitutional AI, involve determining which LLM behaviors are desirable and
training them into a language model. However, in many cases, it is desirable
for LLMs to be controllable at inference time, so that they can be
used in multiple contexts with diverse needs. We illustrate this with the
Pink Elephant Problem: instructing an LLM to avoid discussing a
certain entity (a ``Pink Elephant''), and instead discuss a preferred entity
(``Grey Elephant''). We apply a novel simplification of Constitutional AI,
Direct Principle Feedback, which skips the ranking of responses and
uses DPO directly on critiques and revisions. Our results show that after DPF
fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2
model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and
performs as well as GPT-4 in on our curated test set assessing the Pink
Elephant Problem.