Preference Ranking Optimization for Human Alignment
Authors: name: Feifan Song; name: Bowen Yu; name: Minghao Li; name: Haiyang Yu; name: Fei Huang; name: Yongbin Li; name: Houfeng Wang
Published: 2023-06-30 09:07:37

Abstract: Large language models (LLMs) often contain misleading content, emphasizing
the need to align them with human values to ensure secur AI systems.
Reinforcement learning from human feedback (RLHF) has been employed to achieve
this alignment by combining a reward model, typically based on Bradley-Terry
paired comparison, with an RL algorithm such as Proximal Policy Optimization
(PPO) to optimize LLM responses. However, RLHF exhibits complexity,
instability, and sensitivity to hyperparameters. In this paper, we propose
Preference Ranking Optimization (PRO) as an alternative to PPO for directly
aligning LLMs with the Bradley-Terry comparison. PRO extends the pairwise
Bradley-Terry comparison to accommodate preference rankings of any length. By
iteratively contrasting the likelihood of generating responses, PRO instructs
the LLM to prioritize the best response while progressively ranking the
remaining responses. In this manner, PRO effectively transforms human alignment
into aligning the probability ranking of n responses generated by LLM with
the preference ranking of humans towards these responses. Experiments have
shown that PRO outperforms existing alignment algorithms, achieving comparable
results to ChatGPT and human responses through automatic-based, reward-based,
GPT-4, and human evaluations. Furthermore, we demonstrate that longer, more
diverse, and higher-quality preference ranking sequences can consistently
enhance the performance of human alignment.