MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math
  Reasoning in Multimodal Large Language Models
Authors: name: Jun Feng, name: Zixin Wang, name: Zhentao Zhang, name: Yue Guo, name: Zhihan Zhou, name: Xiuyi Chen, name: Zhenyang Li, name: Dawei Yin

Multimodal Large Language Models (MLLMs) have demonstrated remarkable
capabilities in visual mathematical reasoning across various existing
benchmarks. However, these benchmarks are predominantly based on clean or
processed multimodal inputs, without incorporating the images provided by
real-world Kindergarten through 12th grade (K-12) educational users. To address
this gap, we introduce MathReal, a meticulously curated dataset comprising
2,000 mathematical questions with images captured by handheld mobile devices in
authentic scenarios. Each question is an image, containing the question text
and visual element. We systematically classify the real images into three
primary categories: image quality degradation, perspective variation, and
irrelevant content interference, which are further delineated into 14
subcategories. Additionally, MathReal spans five core knowledge and ability
categories, which encompass three question types and are divided into three
difficulty levels. To comprehensively evaluate the multimodal mathematical
reasoning abilities of state-of-the-art MLLMs in real-world scenarios, we
design six experimental settings that enable a systematic analysis of their
performance. Through extensive experimentation, we find that the
problem-solving abilities of existing MLLMs are significantly challenged in
realistic educational contexts. Based on this, we conduct a thorough analysis
of their performance and error patterns, providing insights into their
recognition, comprehension, and reasoning capabilities, and outlining
directions for future improvements. Data and code:
https://github.com/junfeng0288/MathReal.