R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement
  Learning
Authors: name: Yi-Fan Zhang, name: Xingyu Lu, name: Xiao Hu, name: Chaoyou Fu, name: Bin Wen, name: Tianke Zhang, name: Changyi Liu, name: Kaiyu Jiang, name: Kaibing Chen, name: Kaiyu Tang, name: Haojie Ding, name: Jiankang Chen, name: Fan Yang, name: Zhang Zhang, name: Tingting Gao, name: Liang Wang

Multimodal Reward Models (MRMs) play a crucial role in enhancing the
performance of Multimodal Large Language Models (MLLMs). While recent
advancements have primarily focused on improving the model structure and
training data of MRMs, there has been limited exploration into the
effectiveness of long-term reasoning capabilities for reward modeling and how
to activate these capabilities in MRMs. In this paper, we explore how
Reinforcement Learning (RL) can be used to improve reward modeling.
Specifically, we reformulate the reward modeling problem as a rule-based RL
task. However, we observe that directly applying existing RL algorithms, such
as Reinforce++, to reward modeling often leads to training instability or even
collapse due to the inherent limitations of these algorithms. To address this
issue, we propose the StableReinforce algorithm, which refines the training
loss, advantage estimation strategy, and reward design of existing RL methods.
These refinements result in more stable training dynamics and superior
performance. To facilitate MRM training, we collect 200K preference data from
diverse datasets. Our reward model, R1-Reward, trained using the
StableReinforce algorithm on this dataset, significantly improves performance
on multimodal reward modeling benchmarks. Compared to previous SOTA models,
R1-Reward achieves a 8.4% improvement on the VL Reward-Bench and a 14.3%
improvement on the Multimodal Reward Bench. Moreover, with more inference
compute, R1-Reward's performance is further enhanced, highlighting the
potential of RL algorithms in optimizing MRMs.