Sample-Efficient Alignment for LLMs
Authors: name: Zichen Liu; name: Changyu Chen; name: Chao Du; name: Wee Sun Lee; name: Min Lin
Published: 2024-11-03 09:18:28

Abstract: We study methods for efficiently aligning large language models (LLMs) with
human preferences given budgeted online feedback. We first formulate the LLM
alignment problem in the frame of contextual dueling bandits. This formulation,
subsuming recent paradigms such as online RLHF and online DPO, inherently
quests for sample-efficient algorithms that incorporate online active
exploration. Leveraging insights from bandit theory, we introduce a unified
algorithm based on Thompson sampling and highlight its applications in two
distinct LLM alignment scenarios. The practical agent that efficiently
implements this algorithm, named SEA (Sample-Efficient Alignment), is
empirically validated through extensive experiments across three model scales
(1B, 2.8B, 6.9B) and three preference learning algorithms (DPO, IPO, SLiC). The
results demonstrate that SEA achieves highly sample-efficient alignment with
oracle's preferences, outperforming recent active exploration methods for LLMs.
Additionally, we release the implementation of SEA together with an efficient
codebase designed for online alignment of LLMs, aiming to accelerate future
research in this field.