Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision
  Language Models
Authors: name: Michihiro Yasunaga, name: Luke Zettlemoyer, name: Marjan Ghazvininejad

Reward models play an essential role in training vision-language models
(VLMs) by assessing output quality to enable aligning with human preferences.
Despite their importance, the research community lacks comprehensive open
benchmarks for evaluating multimodal reward models in VLMs. To address this
gap, we introduce Multimodal RewardBench, an expert-annotated benchmark
covering six domains: general correctness, preference, knowledge, reasoning,
safety, and visual question-answering. Our dataset comprises 5,211 annotated
(prompt, chosen response, rejected response) triplets collected from various
VLMs. In evaluating a range of VLM judges, we find that even the top-performing
models, Gemini 1.5 Pro and Claude 3.5 Sonnet, achieve only 72% overall
accuracy. Notably, most models struggle in the reasoning and safety domains.
These findings suggest that Multimodal RewardBench offers a challenging testbed
for advancing reward model development across multiple domains. We release the
benchmark at https://github.com/facebookresearch/multimodal_rewardbench.