Title: Beyond RLHF: A Theoretical Framework of Alignment as Distribution Learning
Primary area: learning theory
Keywords: alignment, large language models, RLHF, DPO

Abstract: Alignment via reinforcement learning from human feedback (RLHF) has become the dominant paradigm for controlling the quality of outputs from large language models (LLMs). However, the standard RLHF objective lacks formal justification and incentivizes degenerate, deterministic LMs in the asymptotic regime. We ask under what assumptions can we derive RLHF or other novel objectives with rigorous learning-theoretic guarantees, without relying on an \emph{a priori} notion of reward maximization. To this end, we reframe alignment as \emph{distribution learning} from pairwise preferences, formalizing our approach with a probabilistic assumption describing how preferences reveal information about the target (oracle) LM. This leads us to propose three principled alignment objectives: preference maximum likelihood estimation, preference distillation, and reverse KL minimization. We prove that all three approaches enjoy strong non-asymptotic $O(1/n)$ convergence to the target LM, naturally avoiding degeneracy. In particular, reverse KL highly resembles the RLHF objective, providing strong justification for RLHF with a minor correction. Furthermore, our theory confirms the empirical finding that on-policy objectives (e.g., RLHF) often outperform likelihood-style objectives (e.g., DPO). Finally, we empirically show that our proposed methods consistently matches or outperforms baselines across various tasks and models.