Self-Boosting Large Language Models with Synthetic Preference Data
Authors: name: Qingxiu Dong; name: Li Dong; name: Xingxing Zhang; name: Zhifang Sui; name: Furu Wei
Published: 2024-10-09 14:57:31

Abstract: Through alignment with human preferences, Large Language Models (LLMs) have
advanced significantly in generating honest, harmless, and helpful responses.
However, collecting high-quality preference data is a resource-intensive and
creativity-demanding process, especially for the continual improvement of LLMs.
We introduce SynPO, a self-boosting paradigm that leverages synthetic
preference data for model alignment. SynPO employs an iterative mechanism
wherein a self-prompt generator creates diverse prompts, and a response
improver refines model responses progressively. This approach trains LLMs to
autonomously learn the generative rewards for their own outputs and eliminates
the need for large-scale annotation of prompts and human preferences. After
four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements
in instruction-following abilities, achieving over 22.1% win rate improvements
on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general
performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score
increase on the well-recognized Open LLM leaderboard.