M-RewardBench: Evaluating Reward Models in Multilingual Settings
Authors: name: Srishti Gureja; name: Lester James V. Miranda; name: Shayekh Bin Islam; name: Rishabh Maheshwary; name: Drishti Sharma; name: Gusti Winata; name: Nathan Lambert; name: Sebastian Ruder; name: Sara Hooker; name: Marzieh Fadaee
Published: 2024-10-20 22:09:44

Abstract: Reward models (RMs) have driven the state-of-the-art performance of LLMs
today by enabling the integration of human feedback into the language modeling
process. However, RMs are primarily trained and evaluated in English, and their
capabilities in multilingual settings remain largely understudied. In this
work, we conduct a systematic evaluation of several reward models in
multilingual settings. We first construct the first-of-its-kind multilingual RM
evaluation benchmark, M-RewardBench, consisting of 2.87k preference instances
for 23 typologically diverse languages, that tests the chat, safety, reasoning,
and translation capabilities of RMs. We then rigorously evaluate a wide range
of reward models on M-RewardBench, offering fresh insights into their
performance across diverse languages. We identify a significant gap in RMs'
performances between English and non-English languages and show that RM
preferences can change substantially from one language to another. We also
present several findings on how different multilingual aspects impact RM
performance. Specifically, we show that the performance of RMs is improved with
improved translation quality. Similarly, we demonstrate that the models exhibit
better performance for high-resource languages. We release M-RewardBench
dataset and the codebase in this study to facilitate a better understanding of
RM evaluation in multilingual settings.