Aligning Large Language Models via Self-Steering Optimization
Authors: name: Hao Xiang; name: Bowen Yu; name: Hongyu Lin; name: Keming Lu; name: Yaojie Lu; name: Xianpei Han; name: Le Sun; name: Jingren Zhou; name: Junyang Lin
Published: 2024-10-22 16:04:03

Abstract: Automated alignment develops alignment systems with minimal human
intervention. The key to automated alignment lies in providing learnable and
accurate preference signals for preference learning without human annotation.
In this paper, we introduce Self-Steering Optimization (SSO), an algorithm
that autonomously generates high-quality preference signals based on predefined
principles during iterative training, eliminating the need for manual
annotation. SSO maintains the accuracy of signals by ensuring a consistent
gap between chosen and rejected responses while keeping them both on-policy to
suit the current policy model's learning capacity. SSO can benefit the online
and offline training of the policy model, as well as enhance the training of
reward models. We validate the effectiveness of SSO with two foundation
models, Qwen2 and Llama3.1, indicating that it provides accurate, on-policy
preference signals throughout iterative training. Without any manual annotation
or external models, SSO leads to significant performance improvements across
six subjective or objective benchmarks. Besides, the preference data generated
by SSO significantly enhanced the performance of the reward model on
Rewardbench. Our work presents a scalable approach to preference optimization,
paving the way for more efficient and effective automated alignment.