VERIFY: A Benchmark of Visual Explanation and Reasoning for
  Investigating Multimodal Reasoning Fidelity
Authors: name: Jing Bi; name: Junjia Guo; name: Susan Liang; name: Guangyu Sun; name: Luchuan Song; name: Yunlong Tang; name: Jinxi He; name: Jiarui Wu; name: Ali Vosoughi; name: Chen Chen; name: Chenliang Xu
Published: 2025-03-14 16:26:11

Abstract: Visual reasoning is central to human cognition, enabling individuals to
interpret and abstractly understand their environment. Although recent
Multimodal Large Language Models (MLLMs) have demonstrated impressive
performance across language and vision-language tasks, existing benchmarks
primarily measure recognition-based skills and inadequately assess true visual
reasoning capabilities. To bridge this critical gap, we introduce VERIFY, a
benchmark explicitly designed to isolate and rigorously evaluate the visual
reasoning capabilities of state-of-the-art MLLMs. VERIFY compels models to
reason primarily from visual information, providing minimal textual context to
reduce reliance on domain-specific knowledge and linguistic biases. Each
problem is accompanied by a human-annotated reasoning path, making it the first
to provide in-depth evaluation of model decision-making processes.
Additionally, we propose novel metrics that assess visual reasoning fidelity
beyond mere accuracy, highlighting critical imbalances in current model
reasoning patterns. Our comprehensive benchmarking of leading MLLMs uncovers
significant limitations, underscoring the need for a balanced and holistic
approach to both perception and reasoning. For more teaser and testing, visit
our project page (https://verify-eqh.pages.dev/).