Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
Authors: name: Chris Yuhao Liu; name: Liang Zeng; name: Jiacai Liu; name: Rui Yan; name: Jujie He; name: Chaojie Wang; name: Shuicheng Yan; name: Yang Liu; name: Yahui Zhou
Published: 2024-10-24 06:06:26

Abstract: In this report, we introduce a collection of methods to enhance reward
modeling for LLMs, focusing specifically on data-centric techniques. We propose
effective data selection and filtering strategies for curating high-quality
open-source preference datasets, culminating in the Skywork-Reward data
collection, which contains only 80K preference pairs -- significantly smaller
than existing datasets. Using this curated dataset, we developed the
Skywork-Reward model series -- Skywork-Reward-Gemma-27B and
Skywork-Reward-Llama-3.1-8B -- with the former currently holding the top
position on the RewardBench leaderboard. Notably, our techniques and datasets
have directly enhanced the performance of many top-ranked models on
RewardBench, highlighting the practical impact of our contributions in
real-world preference learning applications.