We-Math: Does Your Large Multimodal Model Achieve Human-like
  Mathematical Reasoning?
Authors: name: Runqi Qiao; name: Qiuna Tan; name: Guanting Dong; name: Minhui Wu; name: Chong Sun; name: Xiaoshuai Song; name: Zhuoma GongQue; name: Shanglin Lei; name: Zhe Wei; name: Miaoxuan Zhang; name: Runfeng Qiao; name: Yifan Zhang; name: Xiao Zong; name: Yida Xu; name: Muxi Diao; name: Zhimin Bao; name: Chen Li; name: Honggang Zhang
Published: 2024-07-01 13:39:08

Abstract: Visual mathematical reasoning, as a fundamental visual reasoning ability, has
received widespread attention from the Large Multimodal Models (LMMs)
community. Existing benchmarks, such as MathVista and MathVerse, focus more on
the result-oriented performance but neglect the underlying principles in
knowledge acquisition and generalization. Inspired by human-like mathematical
reasoning, we introduce WE-MATH, the first benchmark specifically designed to
explore the problem-solving principles beyond end-to-end performance. We
meticulously collect and categorize 6.5K visual math problems, spanning 67
hierarchical knowledge concepts and five layers of knowledge granularity. We
decompose composite problems into sub-problems according to the required
knowledge concepts and introduce a novel four-dimensional metric, namely
Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery
(CM), and Rote Memorization (RM), to hierarchically assess inherent issues in
LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of
existing LMMs in visual mathematical reasoning and reveal a negative
correlation between solving steps and problem-specific performance. We confirm
the IK issue of LMMs can be effectively improved via knowledge augmentation
strategies. More notably, the primary challenge of GPT-4o has significantly
transitioned from IK to IG, establishing it as the first LMM advancing towards
the knowledge generalization stage. In contrast, other LMMs exhibit a marked
inclination towards Rote Memorization - they correctly solve composite problems
involving multiple knowledge concepts yet fail to answer sub-problems. We
anticipate that WE-MATH will open new pathways for advancements in visual
mathematical reasoning for LMMs. The WE-MATH data and evaluation code are
available at https://github.com/We-Math/We-Math.