Deep Bayesian Active Learning for Preference Modeling in Large Language
  Models
Authors: name: Luckeciano C. Melo; name: Panagiotis Tigas; name: Alessandro Abate; name: Yarin Gal
Published: 2024-06-14 13:32:43

Abstract: Leveraging human preferences for steering the behavior of Large Language
Models (LLMs) has demonstrated notable success in recent years. Nonetheless,
data selection and labeling are still a bottleneck for these systems,
particularly at large scale. Hence, selecting the most informative points for
acquiring human feedback may considerably reduce the cost of preference
labeling and unleash the further development of LLMs. Bayesian Active Learning
provides a principled framework for addressing this challenge and has
demonstrated remarkable success in diverse settings. However, previous attempts
to employ it for Preference Modeling did not meet such expectations. In this
work, we identify that naive epistemic uncertainty estimation leads to the
acquisition of redundant samples. We address this by proposing the Bayesian
Active Learner for Preference Modeling (BAL-PM), a novel stochastic acquisition
policy that not only targets points of high epistemic uncertainty according to
the preference model but also seeks to maximize the entropy of the acquired
prompt distribution in the feature space spanned by the employed LLM. Notably,
our experiments demonstrate that BAL-PM requires 33% to 68% fewer preference
labels in two popular human preference datasets and exceeds previous stochastic
Bayesian acquisition policies.