We-Math: Does Your Large Multimodal Model Achieve Human-like
  Mathematical Reasoning?
Authors: name: Runqi Qiao, name: Qiuna Tan, name: Guanting Dong, name: Minhui Wu, name: Chong Sun, name: Xiaoshuai Song, name: Zhuoma GongQue, name: Shanglin Lei, name: Zhe Wei, name: Miaoxuan Zhang, name: Runfeng Qiao, name: Yifan Zhang, name: Xiao Zong, name: Yida Xu, name: Muxi Diao, name: Zhimin Bao, name: Chen Li, name: Honggang Zhang

Visual mathematical reasoning, as a fundamental visual reasoning ability, has
received widespread attention from the Large Multimodal Models (LMMs)
community. Existing benchmarks, such as MathVista and MathVerse, focus more on
the result-oriented performance but neglect the underlying principles in
knowledge acquisition and generalization. Inspired by human-like mathematical
reasoning, we introduce WE-MATH, the first benchmark specifically designed to
explore the problem-solving principles beyond end-to-end performance. We
meticulously collect and categorize 6.5K visual math problems, spanning 67
hierarchical knowledge concepts and five layers of knowledge granularity. We
decompose composite problems into sub-problems according to the required
knowledge concepts and introduce a novel four-dimensional metric, namely
Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery
(CM), and Rote Memorization (RM), to hierarchically assess inherent issues in
LMMs' reasoning process. With WE-MATH, we conduct a thorough evaluation of
existing LMMs in visual mathematical reasoning and reveal a negative
correlation between solving steps and problem-specific performance. We confirm
the IK issue of LMMs can be effectively improved via knowledge augmentation
strategies. More notably, the primary challenge of GPT-4o has significantly
transitioned from IK to IG, establishing it as the first LMM advancing towards
the knowledge generalization stage. In contrast, other LMMs exhibit a marked
inclination towards Rote Memorization - they correctly solve composite problems
involving multiple knowledge concepts yet fail to answer sub-problems. We
anticipate that WE-MATH will open new pathways for advancements in visual
mathematical reasoning for LMMs. The WE-MATH data and evaluation code are
available at https://github.com/We-Math/We-Math.