Title: On Monotonicity in AI Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Monotonicity, Preference Learning, Alignment, Language Models, Direct Preference Optimization, Bradley-Terry

Abstract: Comparison-based preference learning has become central to the alignment of AI models with human preferences. However, these methods may behave counter-intuitively. After empirically observing that, when accounting for a preference for response y over z, the model may actually decrease the probability (and reward) of generating y (an observation also made by others), this paper investigates the root causes of (non) monotonicity, for a general comparison-based preference learning framework that subsumes Direct Preference Optimization (DPO), Generalized Preference Optimization (GPO) and Generalized Bradley-Terry (GBT). Under mild assumptions, we prove that such methods still satisfy what we call local pairwise monotonicity. We also provide a bouquet of formalizations of monotonicity, and identify sufficient conditions for their guarantee, thereby providing a toolbox to evaluate how prone learning models are to monotonicity violations. These results clarify the limitations of current methods and provide guidance for developing more trustworthy preference learning algorithms.

TL;DR: We study how a reported preference for $y$ over $z$ actually impacts the valuations of $y$ and $z$ under DPO, GPO, GBT and other comparison-based preference learning methods.