BaseReward: A Strong Baseline for Multimodal Reward Model
Authors: name: Yi-Fan Zhang; name: Haihua Yang; name: Huanyu Zhang; name: Yang Shi; name: Zezhou Chen; name: Haochen Tian; name: Chaoyou Fu; name: Haotian Wang; name: Kai Wu; name: Bo Cui; name: Xu Wang; name: Jianfei Pan; name: Haotian Wang; name: Zhang Zhang; name: Liang Wang
Published: 2025-09-19 16:25:26

Abstract: The rapid advancement of Multimodal Large Language Models (MLLMs) has made
aligning them with human preferences a critical challenge. Reward Models (RMs)
are a core technology for achieving this goal, but a systematic guide for
building state-of-the-art Multimodal Reward Models (MRMs) is currently lacking
in both academia and industry. Through exhaustive experimental analysis, this
paper aims to provide a clear ``recipe'' for constructing high-performance
MRMs. We systematically investigate every crucial component in the MRM
development pipeline, including reward modeling paradigms (e.g.,
Naive-RM, Critic-based RM, and Generative RM), reward head
architecture, training strategies, data curation (covering
over ten multimodal and text-only preference datasets), backbone model
and model scale, and ensemble methods.
  Based on these experimental insights, we introduce BaseReward, a
powerful and efficient baseline for multimodal reward modeling. BaseReward
adopts a simple yet effective architecture, built upon a {Qwen2.5-VL} backbone,
featuring an optimized two-layer reward head, and is trained on a carefully
curated mixture of high-quality multimodal and text-only preference data. Our
results show that BaseReward establishes a new SOTA on major benchmarks such as
MM-RLHF-Reward Bench, VL-Reward Bench, and Multimodal Reward Bench,
outperforming previous models. Furthermore, to validate its practical utility
beyond static benchmarks, we integrate BaseReward into a real-world
reinforcement learning pipeline, successfully enhancing an MLLM's performance
across various perception, reasoning, and conversational tasks. This work not
only delivers a top-tier MRM but, more importantly, provides the community with
a clear, empirically-backed guide for developing robust reward models for the
next generation of MLLMs.