A Common Pitfall of Margin-based Language Model Alignment: Gradient
  Entanglement
Authors: name: Hui Yuan, name: Yifan Zeng, name: Yue Wu, name: Huazheng Wang, name: Mengdi Wang, name: Liu Leqi

Reinforcement Learning from Human Feedback (RLHF) has become the predominant
approach for language model (LM) alignment. At its core, RLHF uses a
margin-based loss for preference optimization, specifying ideal LM behavior
only by the difference between preferred and dispreferred responses. In this
paper, we identify a common pitfall of margin-based methods -- the
under-specification of ideal LM behavior on preferred and dispreferred
responses individually, which leads to two unintended consequences as the
margin increases: (1) The probability of dispreferred (e.g., unsafe) responses
may increase, resulting in potential safety alignment failures. (2) The
probability of preferred responses may decrease, even when those responses are
ideal. We demystify the reasons behind these problematic behaviors:
margin-based losses couple the change in the preferred probability to the
gradient of the dispreferred one, and vice versa, often preventing the
preferred probability from increasing while the dispreferred one decreases, and
thus causing a synchronized increase or decrease in both probabilities. We term
this effect, inherent in margin-based objectives, gradient entanglement.
Formally, we derive conditions for general margin-based alignment objectives
under which gradient entanglement becomes concerning: the inner product of the
gradients of preferred and dispreferred log-probabilities is large relative to
the individual gradient norms. We theoretically investigate why such inner
products can be large when aligning language models and empirically validate
our findings. Empirical implications of our framework extend to explaining
important differences in the training dynamics of various preference
optimization algorithms, and suggesting potential algorithm designs to mitigate
the under-specification issue of margin-based methods and thereby improving
language model alignment.