Title: DGPO: Mitigating Likelihood Displacement with Bidirectional KL Divergence Gap
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Alignment, Likelihood Displacement, Gradient Entanglement

Abstract: The current margin-based model alignment method, represented by Direct Preference Optimization (DPO), aims to expand the margin between positive and negative feedback. However, some works point out the log-probability of positive feedback always decreases, thus affecting the likelihood of its generation. This likelihood displacement caused by gradient entanglement is a failure mode for preference optimization and has not been fully resolved. In this paper, we focus on forward and reverse Kullback-Leibler (KL) divergence on the probability distribution of preference pairs to form Divergence Gap Preference Optimization (DGPO). We prove DGPO can stimulate the chosen log-probability increase. Besides, DGPO also maintains a lightweight and automatical way in read-world alignment. The downstream experimental results demonstrate that DGPO maintains competitive performance across various mainstream benchmarks without the reference model and additional key hyperparameters.