Learning to Align, Aligning to Learn: A Unified Approach for
  Self-Optimized Alignment
Authors: name: Haowen Wang; name: Yun Yue; name: Zhiling Ye; name: Shuowen Zhang; name: Lei Fan; name: Jiaxin Liang; name: Jiadi Jiang; name: Cheng Wei; name: Jingyuan Deng; name: Xudong Han; name: Ji Li; name: Chunxiao Guo; name: Peng Wei; name: Jian Wang; name: Jinjie Gu
Published: 2025-08-11 08:28:47

Abstract: Alignment methodologies have emerged as a critical pathway for enhancing
language model alignment capabilities. While SFT (supervised fine-tuning)
accelerates convergence through direct token-level loss intervention, its
efficacy is constrained by offline policy trajectory. In contrast,
RL(reinforcement learning) facilitates exploratory policy optimization, but
suffers from low sample efficiency and stringent dependency on high-quality
base models. To address these dual challenges, we propose GRAO (Group Relative
Alignment Optimization), a unified framework that synergizes the respective
strengths of SFT and RL through three key innovations: 1) A multi-sample
generation strategy enabling comparative quality assessment via reward
feedback; 2) A novel Group Direct Alignment Loss formulation leveraging
intra-group relative advantage weighting; 3) Reference-aware parameter updates
guided by pairwise preference dynamics. Our theoretical analysis establishes
GRAO's convergence guarantees and sample efficiency advantages over
conventional approaches. Comprehensive evaluations across complex human
alignment tasks demonstrate GRAO's superior performance, achieving
57.70\%,17.65\% 7.95\% and 5.18\% relative improvements over SFT, DPO, PPO and
GRPO baselines respectively. This work provides both a theoretically grounded
alignment framework and empirical evidence for efficient capability evolution
in language models.