ICML 2025

On the Robustness of Reward Models for Language Model Alignment
Jiwoo Hong 1 Noah Lee 1 Eunki Kim 1 Guijin Son 2 Woojin Chung 1
Aman Gupta 3 Shao Tang 3 James Thorne 1
Abstract
The Bradley-Terry (BT) model is widely prac-
ticed in reward modeling for reinforcement learn-
ing with human feedback (RLHF). Despite its
effectiveness, reward models (RMs) trained with
BT model loss are prone to over-optimization,
losing generalizability to unseen input distribu-
tions.
In this paper, we study the cause of
over-optimization in RM training and its down-
stream effects on the RLHF procedure, accen-
tuating the importance of distributional robust-
ness of RMs in unseen data. First, we show that
the excessive dispersion of hidden state norms
is the main source of over-optimization. Then,
we propose batch-wise sum-to-zero regulariza-
tion (BSR) to enforce zero-centered reward sum
per batch, constraining the rewards with extreme
magnitudes. We assess the impact of BSR in
improving robustness in RMs through four sce-
narios of over-optimization, where BSR consis-
tently manifests better robustness. Subsequently,
we compare the plain BT model and BSR on
RLHF training and empirically show that robust
RMs better align the policy to the gold preference
model. Finally, we apply BSR to high-quality
data and models, which surpasses state-of-the-
art RMs in the 8B scale by adding more than
5% in complex preference prediction tasks. By
conducting RLOO training with 8B RMs, Al-
pacaEval 2.0 reduces generation length by 40%
while adding a 7% increase in win rate, further
highlighting that robustness in RMs induces ro-
bustness in RLHF training. We release the code,
data, and models: https://github.com/
LinkedIn-XFACT/RM-Robustness.
1KAIST AI 2OneLineAI 3LinkedIn Corporation. Correspon-
dence to: Jiwoo Hong <jiwoo hong@kaist.ac.kr>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
1. Introduction
Reward ...