RRM: Robust Reward Model Training Mitigates Reward Hacking
Authors: name: Tianqi Liu; name: Wei Xiong; name: Jie Ren; name: Lichang Chen; name: Junru Wu; name: Rishabh Joshi; name: Yang Gao; name: Jiaming Shen; name: Zhen Qin; name: Tianhe Yu; name: Daniel Sohn; name: Anastasiia Makarova; name: Jeremiah Liu; name: Yuan Liu; name: Bilal Piot; name: Abe Ittycheriah; name: Aviral Kumar; name: Mohammad Saleh
Published: 2024-09-20 01:46:07

Abstract: Reward models (RMs) play a pivotal role in aligning large language models
(LLMs) with human preferences. However, traditional RM training, which relies
on response pairs tied to specific prompts, struggles to disentangle
prompt-driven preferences from prompt-independent artifacts, such as response
length and format. In this work, we expose a fundamental limitation of current
RM training methods, where RMs fail to effectively distinguish between
contextual signals and irrelevant artifacts when determining preferences. To
address this, we introduce a causal framework that learns preferences
independent of these artifacts and propose a novel data augmentation technique
designed to eliminate them. Extensive experiments show that our approach
successfully filters out undesirable artifacts, yielding a more robust reward
model (RRM). Our RRM improves the performance of a pairwise reward model
trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to
84.15%. Additionally, we train two DPO policies using both the RM and RRM,
demonstrating that the RRM significantly enhances DPO-aligned policies,
improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in
AlpacaEval-2 from 33.46% to 52.49%.