RBench-V: A Primary Assessment for Visual Reasoning Models with
  Multi-modal Outputs
Authors: name: Meng-Hao Guo; name: Xuanyu Chu; name: Qianrui Yang; name: Zhe-Han Mo; name: Yiqing Shen; name: Pei-lin Li; name: Xinjie Lin; name: Jinnian Zhang; name: Xin-Sheng Chen; name: Yi Zhang; name: Kiyohiro Nakayama; name: Zhengyang Geng; name: Houwen Peng; name: Han Hu; name: Shi-Nin Hu
Published: 2025-05-22 15:11:57

Abstract: The rapid advancement of native multi-modal models and omni-models,
exemplified by GPT-4o, Gemini, and o3, with their capability to process and
generate content across modalities such as text and images, marks a significant
milestone in the evolution of intelligence. Systematic evaluation of their
multi-modal output capabilities in visual thinking processes (also known as
multi-modal chain of thought, M-CoT) becomes critically important. However,
existing benchmarks for evaluating multi-modal models primarily focus on
assessing multi-modal inputs and text-only reasoning while neglecting the
importance of reasoning through multi-modal outputs. In this paper, we present
a benchmark, dubbed RBench-V, designed to assess models' vision-indispensable
reasoning abilities. To construct RBench-V, we carefully hand-pick 803
questions covering math, physics, counting, and games. Unlike previous
benchmarks that typically specify certain input modalities, RBench-V presents
problems centered on multi-modal outputs, which require image manipulation such
as generating novel images and constructing auxiliary lines to support the
reasoning process. We evaluate numerous open- and closed-source models on
RBench-V, including o3, Gemini 2.5 Pro, Qwen2.5-VL, etc. Even the
best-performing model, o3, achieves only 25.8% accuracy on RBench-V, far below
the human score of 82.3%, highlighting that current models struggle to leverage
multi-modal reasoning. Data and code are available at
https://evalmodels.github.io/rbenchv