Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMs

Authors: Shan Zhang, Aotian Chen, Yanpeng Sun, Jindong Gu, Yi-Yu Zheng, Piotr Koniusz, Kai Zou, Anton van den Hengel, Yuan Xue
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodal Large Language Models (MLLMs);Mathematical Reasoning;Fine-grained Visual Understanding;Visual Grounding

Abstract:
Current multimodal large language models (MLLMs) often underperform on mathematical problem-solving tasks that require fine-grained visual understanding. The limitation primarily arises from inadequate perception of geometric primitives during image-level contrastive pre-training (e.g., CLIP). Current efforts to enhance MLLM performance have focused on scaling up mathematical visual instruction datasets and employing stronger LLM backbones, yet these approaches often neglect persistent visual recognition errors in MLLMs. In this paper, we systematically evaluate the visual grounding capabilities of state-of-the-art MLLMs and uncover a negative correlation between their visual grounding accuracy and problem-solving performance. Notably, even advanced models like GPT-4o demonstrate a significant error rate (70\%) when identifying geometric entities, highlighting that fine-grained visual understanding remains a crucial bottleneck in visual mathematical reasoning. To address this, we propose a novel approach, SVE-Math (Selective Vision-Enhanced Mathematical MLLM), featuring a geometric-grounded vision encoder and a feature router that dynamically adjusts the contribution of hierarchical visual feature maps. Our model recognizes accurate visual primitives and generates precise visual prompts tailored to the language model's reasoning needs. In experiments, SVE-Math-Deepseek-7B outperforms other 7B models by 7.7\% on MathVerse and is compatible with GPT-4V on MathVista. Despite being trained on smaller datasets, SVE-Math-7B matches the performance of models trained on significantly larger datasets, evaluated on GeoQA. Our findings provide critical insights for future research, highlighting the need for more effective integration of fine-grained visual understanding in MLLMs.  We will release model weights, code, and instructions upon acceptance.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730548513090

thread_type:
official_review

<thread_object>
creator: reviewer_m9gf

<content>
summary:
The paper proposes the SVE-Math-7B model to improve the math reasoning skills of current MLLMs. The authors start by analyzing the performance on mainstream models' math reasoning tasks to show the geometric information's effectiveness. Based on the observation, the author proposes the architecture of SVE-Math with a pre-trained GeoGLIP, a fusing connector with dual visual encoders, and further fine-tuning the baseline models. The authors conduct experiments on mainstream math-relative benchmarks such as MathVerse and MathVista and show improvements compared with baselines.

strengths:
1. The paper discovers and discusses the math-solving problem of MLLMs, which is a significant and widely-concerned problem for current MLLMs. The solution with GeoGIP and math-relevant fine-tuning is efficient for the problem.
2. The analysis in Figure 1 clearly shows the drawbacks of LLaVA and GPT-4o, and shows the effectiveness of geometric information. 
3. The methods and experiment periods are well organized and easy to follow. The authors conduct experiments on mainstream math datasets and clearly show the results.

weaknesses:
1. The main weakness is that the ablation analysis is not sufficient to demonstrate the improvements of all the components. The author proposes the GeoGLIP, dual visual encoder connector, math-specific finetuning with Geo170K, MathV360K datasets. However, the analysis of such aspects is lacking. The authors only conduct experiments on the design of connectors, which is not the key claim for the contributions, as many papers have used similar fusing approaches for visual encoders. I think the authors could clearly explain where the improvements come from, especially for the GeoGLIP and the math-relevant training datasets. 
2. Although the authors show improvements over baselines, the performance for SVE-Math-7B is significantly behind the state-of-the-art models (e.g. more than 60 accuracy on MathVista). I assume the approach proposed by the author is universal, therefore the results of state-of-the-art models are lacking. 
3. The effectiveness of GeoGLIP is not confirmed. I wonder how the tiny visual encoder with less than 50M parameters can help the overall learning results. As shown in the visualization results, directly providing geometric-relevant information in a proper manner may also lead to similar performance. The authors could conduct sufficient experiments to explain this issue.

questions:
As stated in the weakness periods, clarifying the issues can better demonstrate the conclusion of the paper. 
1. What are the improvements with math-specific datasets? 
2. Why using GeoGLIP based on Swin-T is effective for results? As illustrated in the visualization results, the usage of the models provides geometric information, so the authors may provide more comparisons by providing direct geometric results, or directly using GLIP. 
3. The results for current models are somehow out-of-date. The authors are encouraged to equip proposed approaches on state-of-the-art level MLLMs. 
4. The math problems may already be solved with better data curation or reasoning processes, as many papers have done on such problems. The author could provide explanations and superiority for the proposed methods and provide comparisons with other methods on math problems.
Therefore, based on the weaknesses and questions stated above, I think the paper is below the acceptance threshold in the current situation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730559210354

thread_type:
official_review

<thread_object>
creator: reviewer_a1wu

<content>
summary:
This paper introduces SVE-Math, a Multimodal Large Language Model (MLLM) designed for mathematical question answering. It incorporates a GeoGLIP module to enhance the visual encoder's perception of mathematical elements and utilizes a routing module to prioritize features from CLIP. The training process for SVE-Math consists of three stages: GeoGLIP training, cross-modal alignment, and instruction tuning.

strengths:
1.The approach of enhancing the visual encoder for improved mathematical performance is both innovative and logical.

2.The routing module is well-designed and demonstrates significant performance improvements in the ablation studies. However, I believe that the routing module is not specifically designed for mathematical reasoning tasks and can be applied to a wider range of scenarios.

3.The paper is well-structured and easy to understand.

weaknesses:
1.My main concern is the performance results, which are not particularly impressive. While SVE-Math achieves competitive scores on several benchmarks, the improvements over the previous works are marginal, raising questions about the effectiveness of the approach.

2.Building on the first point, I believe a significant portion of the performance improvement in MLLMs stems from the data used. The scale and quality of training data are critical for MLLMs. Could you elaborate on any unique handling or augmentation techniques applied to the training data? 

3.Could the authors provide more explanation of why the routing module is specifically designed for mathematical reasoning tasks? Relying solely on empirical evidence is not sufficient to substantiate this claim.

questions:
Please refer to weakness. Can the proposed methods be applied to other mathematical problems beyond geometric figures and problems?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647097508

thread_type:
official_review

<thread_object>
creator: reviewer_yc4u

<content>
summary:
To address the limitations of multimodal large language models (MLLMs) in solving math problems involving images, this paper proposes a Selective Vision-Enhanced Mathematical MLLM. It leverages a geometric-grounded vision encoder and a feature router to help MLLMs better comprehend mathematical image features, thereby improving their performance on math problems with visual components.

strengths:
1. The paper clearly articulates the problem it aims to address, and the overall writing is easy to follow.

2. The paper enhances MLLM's ability to recognize mathematical images and solve math problems by introducing geometry-rich visual information, achieving improvements on several benchmarks.

weaknesses:
1. Using more detailed visual features for solving math problems is an intuitive idea, as is combining geometric and semantic features at different levels. However, you should conduct additional ablation studies to validate the effectiveness of this approach. For instance, consider using vision encoders from other similar models on your dataset/training your model on the training data of other models.

2. In Table 1, some experimental results differ from those provided in the official MathVerse table. For example, you show the cot-e score for SPHINX-Plus and the w/o score for SPHINX-MOE. When comparing with other models on the same benchmark, you should ensure thorough variable control.

3. You mention using synthetic data, but the paper does not include any description, details, or examples of the synthetic data generation process.

4. The paper does not present any output examples from the model.

5. As a “data collection-model training-benchmark testing” type of paper, the performance improvements on benchmarks are minimal in the absence of novelty.

questions:
1. In terms of writing, the paper’s section distribution could be improved. You should allocate some space to introduce synthetic data, dedicate more space to ablation studies to validate the method's effectiveness, and reduce the length of the Methods section.

2. Please provide more details and examples of the synthetic data.

3. Please provide examples of the model’s outputs to demonstrate its ability to recognize geometric elements and Chain-of-Thought (CoT), as you compared cot-e performance with some models in Table 1.

4. In the Introduction, you mentioned a finding: instructing MLLMs with fine-grained visual information improves top-1 accuracy compared to providing only worded questions, while providing all visual cues for solving a math question decreases accuracy. How does your approach—primarily by introducing more geometry-rich visual information—address the issue highlighted by this finding?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714389343

thread_type:
official_review

<thread_object>
creator: reviewer_ftud

<content>
summary:
The paper first identifies visual recognition errors prevalent of current MLLMs by a pilot study. Then the paper introduces GeoGLIP, a vision encoder specifically trained to identify geometric elements in the image. The feature from the trained geometric vision encoder is later merged with the feature of the original CLIP vision encoder, aiming at more precise geometry perception. The authors prove the effectiveness of their method by evaluating on various benchmarks.

strengths:
+ The paper proposes a novel perspective that the errors of visual mathematical problems come from poor visual perception.
+ The three training tasks of GeoGLIP closely matches the analysis in Fig. 1. The paper is self-contained and well-written.

weaknesses:
+ The major concern is whether addressing the visual perception error is sufficient for the mllm to correctly solve these tasks. The visual mathematical questions also require advanced reasoning capability, especially merging both the visual and textual information. Only correctly identifying the graph seems to be far enough to solve a mathematical problem. Detecting the texts, shapes or curves in the graph does not necessarily suggest the model understands the element. How much GeoGLIP actually helps in understanding and reasoning seems marginal. The pilot study shown in Fig. 1 also only analyze the error of visual descriptions, while neglecting other potential core problems of MLLM for visual mathematical questions. 
+ The effectiveness of the proposed GeoGLIP is not validated. The authors need to report the performance of the model trained with same instruction data only without the GeoGLIP encoder to illustrate the improvement brought by it. Otherwise, the improvement may be from the Geo170K data.
+ The overall performance advantages of SVE-Math compared to previous works are not very obvious.

questions:
+ How much more computational cost and inference time is introduced by GeoGLIP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732807090833

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
## We sincerely thank the reviewers for their insightful feedback and constructive suggestions. We are delighted that our approach, SVE-Math, and the GeoGLIP module have been recognized as innovative and logical steps toward addressing the limitations of current multimodal large language models (MLLMs) in visual mathematical reasoning.
\
\
We have addressed all comments in individual responses to each reviewer.
\
\
\
Below, we address the key points raised across the reviews and clarify several aspects of our work to better demonstrate its contributions and implications.
## 1. Clarification of Goal and Contribution
* Our primary goal is not to solve the entire spectrum of mathematical reasoning tasks but to enhance the visual grounding capabilities of MLLMs in a way that complements their reasoning abilities. Our approach, which integrates a geometry-rich visual encoder (GeoGLIP), is orthogonal to existing methods focused on improving reasoning. By doing so, we aim to address the persistent bottleneck of fine-grained visual perception in mathematical contexts, as detailed in Section 1 and supported by the systematic analysis in Figure 1 and Figure 5a of the paper.

* GeoGLIP serves as a lightweight, domain-specific enhancement, specifically addressing geometric visual recognition errors. Importantly, it is designed to work seamlessly with diverse LLM backbones without requiring modifications to their reasoning components. This adaptability underscores its novelty and broad applicability.

##  2. Effectiveness of GeoGLIP and Dataset Independence
*  A key concern raised by reviewers is whether the improvement observed with our approach stems primarily from the instruction dataset used (Geo170K). To address this, we emphasize that the comparison with G-LLaVA [Gao et al., 2023a] in our paper is conducted under controlled conditions. Both G-LLaVA and our model use the same LLM backbone (LLaMA2-7B) and the Geo170K dataset, ensuring that any performance differences arise from the inclusion of the GeoGLIP encoder rather than the instruction dataset. The comparison results of Tables 1-3 confirm the effectiveness of our GeoGLIP, which significantly enhances visual mathematical reasoning performance. As shown in Tables 1 and 2, G-LLaVA with GeoGLIP (our SVE-Math-7B) improves Top-1 accuracy by 7.7\% on MathVerse and 12.3\% on MathVista. The improvement is not trivial, and our SVE-Math-7B achieves comparable performance to Math-LLaVA-13B on MathVerse (19.0\% vs. 21.2\%),  particularly excelling in the 'visual-only' scenario (16.4\% vs. 20.3\%). This scenario strips away the entire textual input, conveying the problem solely through the diagram.

* Additionally, we conducted further experiments integrating GeoGLIP with various LLM backbones that exhibit stronger mathematical reasoning abilities compared with LLaMA2-7B (e.g., Qwen2.5-Math-7B-Instruct and DeepSeek-Math-7B-Instruct, two of the most advanced mathematical reasoning LLMs currently available).  We evaluate our model on the most challenging MathVista benchmark, achieving 51.3\% and 48.7\% Top-1 accuracy, compatible with GPT-4V's performance (49.9\%).  Again, we observe a consistent 6\%-7\% improvement compared to the variant excluding GeoGLIP featurs as additional visual prompts.  These results reaffirm the complementary nature of the GeoGLIP visual encoder with reasoning abilities and highlight its generalizability benefits across diverse architectures. We will release those model weights, the training, and the inference codes to facilitate the computer vision community.

* Finally, we would like to clarify the distinction between our synthetic math-specific datasets and traditional mathematical instruction datasets. We do not create or use any additional self-generated instruction datasets beyond the publicly available Geo170K and MathV360K datasets for MLLM training. Instead, our synthetic samples, annotated with box/pixel-level details, are exclusively utilized to train the GeoGLIP visual encoder.  Compared to constructing mathematical instruction datasets, our synthetic data generation process is significantly more efficient and resource-friendly. It does not require manual labeling, as all data can be programmatically generated, e.g., through the Matplotlib Python library. In contrast, constructing instruction datasets often relies on GPT-4o to create diverse prompts and necessitates human intervention, making the process labor-intensive and costly. Moreover, training the lightweight, visual-centric GeoGLIP involves straightforward training recipes. In comparison, instruction tuning for MLLMs requires intricate configurations, such as carefully curated batch sizes and learning rates, as noted in [Shengbang et al., 2024].
> Shengbang Tong et al., Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs, arXiv 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732857236252

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Explanation for Delayed Response

comment:
Esteem AC and reviewers,

Thank you for your thoughtful review and feedback. We sincerely apologize for the delay in providing our responses, and we are truly sorry it took us so long to post a rebuttal. The delay was due to the time required to conduct additional experiments and obtain the necessary results to thoroughly address the concerns raised (7 days+28 hours +2 days and 18 hours +26 hours +12.7 hours+2 days). Once the results were available, we promptly submitted the rebuttal and paper revision to ensure the response was both accurate and comprehensive.

* Experiments on Cutting-edge LLMs (in response to all reviewers): Evaluating the effectiveness of our geometric-aware visual encoder (GeoGLIP) requires comprehensive ablation studies on state-of-the-art LLMs like DeepSeek and Qwen. Specifically, we compare GeoGLIP with configurations using only the CLIP visual encoder. These experiments involve four configurations, all trained on the MathV360K and Geo170K datasets.  We have access to only one machine with 8 GPUs, which limits our ability to run experiments in parallel. The entire training process takes approximately **one week** to complete.

* Experiments on other visual-centric models or training with different datasets (in response to reviewer yC4U): Utilizing 8 A100 GPUs, training another detector (Grounding DINO) on our math-specific datasets requires approximately **28 hours**, while training our visual model on an alternative dataset takes around **2 days and 18 hours**. Furthermore, integrating these visual features into the LLaVA-7B model on the Geo170K dataset takes an additional **26 hours** in total.

* Experiments on GLIP and directly providing geometric-relevant information  (in response to Reviewer M9gf):  Integrating the GLIP hierarchical pyramid features into MLLMs on the Geo170K dataset requires **12.7 hours**. As suggested by the reviewer, we explored directly providing geometric-relevant information to MLLMs. We could not find existing mathematical instruction datasets that include location information for geometric objects (e.g., bounding box coordinates or junction points). To address this, we inferred Geo170K training images using GeoGLIP to extract relevant location information, which we appended the special tokens <image> in `huam value`, using instructions such as: "there is a bounding box at ⟨x, y, w, h⟩". We need to controll the number of detected objects per image to ensure suitability (set to 10 objects). This entire data processing step, along with MLLM training, required approximately **2 days** to complete.

We deeply value the opportunity to engage with the review process and clarify any concerns. We hope the additional evidence and explanations provided in our rebuttal address the key points and demonstrate the significance of our work. We hope reviewers are still willing to give us a chance.

Thank you for your understanding, and we appreciate your time and consideration. 

\
Kind regards,
\
Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733113573940

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Esteem AC and reviewers,

Thank you for your valuable feedback and thoughtful suggestions. We deeply appreciate the time and effort you have dedicated to reviewing our work.

In our previous response, we addressed concerns raised in the reviews and provided detailed explanations to clarify various aspects of our work. However, to ensure clarity and emphasize the core contributions of our work, we would like to briefly summarize the main points here:

* We systematically identify and analyze the impact of visual recognition errors on the mathematical reasoning performance of MLLMs, highlighting the critical role of accurately perceiving geometric primitives. This new aspect is orthogonal to existing methods focused on improving reasoning.

* We designed GeoGLIP, a lightweight, geometry-aware visual model with multitask learning capabilities, including shape grounding, junction detection, and boundary detection. GeoGLIP integrates seamlessly with diverse LLM backbones without requiring modifications to their reasoning components. Despite adding less than a 50MB increase in parameter size and only a 0.24s increase in inference time per image, and without relying on additional mathematical instruction datasets, our approach achieves an 8–12\% improvement in top-1 accuracy compared to the baseline (using LLaMA2-7B as the base LLM).

* We paired GeoGLIP with advanced LLMs like DeepSeek and Qwen and our 7B model achieves performance comparable to GPT-4V, with 51.3\% and 48.7\% on the challenging MathVista benchmark, versus 49.9\% for GPT-4V.  While our 7B model does not surpass state-of-the-art MLLMs with 40B/70B parameters achieving over 60\% accuracy, integrating GeoGLIP into such large-scale LLMs is currently computationally prohibitive due to our limited resources.   

* We hope this work inspires further research into more effective fine-grained visual understanding in MLLMs. To support the community and assist other researchers in scaling our method to larger models and datasets, we will release the model weights, training scripts, and inference codes to facilitate broader adoption and experimentation.

\
Kind regards,
\
Authors
</content>
</thread_object>

</thread>


Metareview: