BaseReward: A Strong Baseline for Multimodal Reward Model
Authors: name: Yi-Fan Zhang, name: Haihua Yang, name: Huanyu Zhang, name: Yang Shi, name: Zezhou Chen, name: Haochen Tian, name: Chaoyou Fu, name: Haotian Wang, name: Kai Wu, name: Bo Cui, name: Xu Wang, name: Jianfei Pan, name: Haotian Wang, name: Zhang Zhang, name: Liang Wang

The rapid advancement of Multimodal Large Language Models (MLLMs) has made
aligning them with human preferences a critical challenge. Reward Models (RMs)
are a core technology for achieving this goal, but a systematic guide for
building state-of-the-art Multimodal Reward Models (MRMs) is currently lacking
in both academia and industry. Through exhaustive experimental analysis, this
paper aims to provide a clear ``recipe'' for constructing high-performance
MRMs. We systematically investigate every crucial component in the MRM
development pipeline, including reward modeling paradigms (e.g.,
Naive-RM, Critic-based RM, and Generative RM), reward head
architecture, training strategies, data curation (covering
over ten multimodal and text-only preference datasets), backbone model
and model scale, and ensemble methods.
  Based on these experimental insights, we introduce BaseReward, a
powerful and efficient baseline for multimodal reward modeling. BaseReward
adopts a simple yet effective architecture, built upon a {Qwen2.5-VL} backbone,
featuring an optimized two-layer reward head, and is trained on a carefully
curated mixture of high-quality multimodal and text-only preference data. Our
results show that BaseReward establishes a new SOTA on major benchmarks such as
MM-RLHF-Reward Bench, VL-Reward Bench, and Multimodal Reward Bench,
outperforming previous models. Furthermore, to validate its practical utility
beyond static benchmarks, we integrate BaseReward into a real-world
reinforcement learning pipeline, successfully enhancing an MLLM's performance
across various perception, reasoning, and conversational tasks. This work not
only delivers a top-tier MRM but, more importantly, provides the community with
a clear, empirically-backed guide for developing robust reward models for the
next generation of MLLMs.