Evaluating the role of `Constitutions' for learning from AI feedback
Authors: name: Saskia Redgate; name: Andrew M. Bean; name: Adam Mahdi
Published: 2024-11-15 13:16:11

Abstract: The growing capabilities of large language models (LLMs) have led to their
use as substitutes for human feedback for training and assessing other LLMs.
These methods often rely on `constitutions', written guidelines which a critic
model uses to provide feedback and improve generations. We investigate how the
choice of constitution affects feedback quality by using four different
constitutions to improve patient-centered communication in medical interviews.
In pairwise comparisons conducted by 215 human raters, we found that detailed
constitutions led to better results regarding emotive qualities. However, none
of the constitutions outperformed the baseline in learning more
practically-oriented skills related to information gathering and provision. Our
findings indicate that while detailed constitutions should be prioritised,
there are possible limitations to the effectiveness of AI feedback as a reward
signal in certain areas.