The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights
Authors: Yufang Liu, Yao Du, Tao Ji, Jianing Wang, Yang Liu, Yuanbin Wu, Aimin Zhou, Mengdi Zhang, Xunliang Cai
Accepted at: acl25

The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights

Yufang Liu1*, Yao Du2*, Tao Ji3,4, Jianing Wang2,
Yang Liu2, Yuanbin Wu1, Aimin Zhou1, Mengdi Zhang2, Xunliang Cai2
1School of Computer Science and Technology, East China Normal University
2 Meituan Inc. 3Fudan University 4 Pazhou Laboratory
yfliu.antnlp@gmail.com taoji@fudan.edu.cn ybwu@cs.ecnu.edu.cn

Abstract

Recent research has increasingly focused on multimodal mathematical reasoning, particularly emphasizing the creation of relevant datasets and benchmarks. Despite this, the role of visual information in reasoning has been underexplored. Our findings show that existing multimodal mathematical models minimally leverage visual information, and model performance remains largely unaffected by changes to or removal of images in the dataset. We attribute this to the dominance of textual information and answer options that inadvertently guide the model to correct answers. To improve evaluation methods, we introduce the HC-M3D dataset, specifically designed to require image reliance for problem-solving and to challenge models with similar, yet distinct, images that change the correct answer. In testing leading models, their failure to detect these subtle visual differences suggests limitations in current visual perception capabilities. Additionally, we observe that the common approach of improving general VQA capabilities by combining various types of image encoders does not contribute to math reasoning performance. This finding also presents a challenge to enhancing visual reliance during math reasoning. Our benchmark and code would be available at https://github.com/Yufang-Liu/visual_modality_role.

1 Introduction

Recent advancements in Large Vision-Language Models (LVLMs) (Zhang et al., 2023) have demonstrated remarkable potential in tasks that require the seamless integration of visual and linguistic understanding, such as image captioning (Lin et al., 2014), visual question answering (Antol et al., 2015), visual grounding (Yu et al., 2016), and autonomous agents (Xi et al., 2023; Durante et al., 2024). Among these, the domain of multimodal mathematical reasoning (Wang et al., 2024a; Lu et al., 2024; Zhang et al., 2024b) emerges as a particularly challenging and pivotal application. This task requires models to accurately interpret and abstractly represent mathematical concepts, such as geometry, through a tight collaboration between textual descriptions and visual elements in diagrams. The ability to align these modalities and resolve complex problems lies at the heart of LVLMs’ capabilities, positioning multimodal mathematical reasoning as a critical field for advancing their effectiveness in abstract and logic-driven applications.

There are three common approaches to enhancing mathematical reasoning capabilities in LVLMs: prompting, fine-tuning, and chain-of-thought (CoT) reasoning. The prompting approach (Lu et al., 2024; Wang et al., 2024b) seeks to unlock the potential of LVLMs through carefully designed prompts. Fine-tuning (Shi et al., 2024; Gao et al., 2023), on the other hand, focuses on constructing higher-quality and more diverse datasets to enhance model performance. Chain-of-thought (Hu et al., 2024) reasoning mimics human reasoning by generating detailed step-by-step thought processes to

![An illustrative example of GPT-4o on the HC-M3D. When modifying the image (by swapping the positions of points G and H) while keeping the original question unchanged, the model’s output failed to correspondingly adjust, resulting in an incorrect response.](page_784_393_484_246.png)

Figure 1: An illustrative example of GPT-4o on the HC-M3D. When modifying the image (by swapping the positions of points G and H) while keeping the original question unchanged, the model’s output failed to correspondingly adjust, resulting in an incorrect response.
tackle complex mathematical tasks. These methods rely heavily on training data tailored for mathematical understanding. Existing research primarily emphasizes improving the diversity of problem types (Gao et al., 2023), refining the conciseness and clarity of questions (Zhang et al., 2024c), and providing more detailed answer explanations (Peng et al., 2024), with model performance evaluated on various benchmarks. *However, current datasets often fail to establish a strong connection between mathematical images and text.*

In this work, we explore whether images are genuinely learned and utilized by models for mathematical reasoning within current multimodal mathematical methods and examines the role they play in the process. We observe that in existing multimodal math models, the benefit brought by image patterns is minimal. Even if the images in the training set are shuffled or even removed, the impact on model performance is not significant (On some evaluation datasets, perturbing the visual modality even resulted in higher scores). By analyzing existing evaluation methods, we suspect that there are two main reasons why the role of visual patterns in multimodal math reasoning has been overestimated: first, the textual information is overly rich, and the model can make correct predictions based on text alone; second, the options leak the answers, and the model can guess the correct answer based on the options.

Hence, we propose the HC-M3D dataset, a human-crafted *multimodal mathematical dataset*, comprising 1,851 samples meticulously selected by humans, that ensures that the questions depend on the images and provides an additional image that looks similar but changes the correct answer for over 400 problems. We observe whether models can identify these subtle differences in the images and make correct predictions. In evaluating existing leading models, we find that they fail to accurately identify these differences, and in more than half of the cases, they stick to their original predictions. Figure 1 presents a representative example.

Furthermore, we attempt to delve into the causes of this phenomenon. Considering that the model’s predictions remain unchanged after image alterations, we infer that the image encoder (mainly the CLIP encoder) fails to effectively recognize such subtle differences, a phenomenon also observed in other works (Liu et al., 2024b; Tong et al., 2024a). However, we observe that the common approach of improving general VQA capabilities by combining various types of image encoders does not contribute to math reasoning performance. This finding also presents a challenge to enhancing visual reliance during math reasoning.

Our contributions can be summarized as follows:
• We show that image shuffling or removal has minimal impact on performance, revealing an overestimation of these models’ reliance on visual input.
• We introduce the HC-M3D dataset with 1,851 human-annotated samples, ensuring image-dependent questions. For 429 questions, similar but altered images are included, yet most mathematical LVLMs fail to adjust predictions, often remaining unchanged.
• We demonstrate the challenge of enhancing visual reliance in mathematical reasoning, as combining image encoders proves ineffective.

Based on these observations, we suggest several potential directions for future research: constructing higher-quality datasets with a stronger reliance on visual data (e.g., in the image-caption pre-training task, describing the differences between two images rather than generating captions for a single image.), improving image encoders to capture more fine-grained mathematical information, and designing better loss functions to enhance the model’s dependence on the visual modality.

2 Related Work

Mathematical reasoning has garnered increasing attention from researchers and has become a critical benchmark for evaluating LLMs. Specialized LLMs such as MetaMath (Yu et al., 2024a), Math-Shepherd (Wang et al., 2024c), WizardMath (Luo et al., 2023), and DeepSeekMath (Shao et al., 2024) have been developed to tackle these tasks. One of the challenges in multimodal mathematical reasoning lies in visual-textual reasoning, involving geometric diagrams, scientific charts, and function graphs. Due to the scarcity of training data, existing methods often employ the LLaVA architecture for training, aiming to generate higher-quality and more diverse training data. For instance, G-LLaVA (Gao et al., 2023) obtains image caption data for alignment by rewriting the original QA data and paraphrases the remaining QA data to augment the SFT data. Math-LLaVA (Shi et al., 2024) synthesizes a large number of QA pairs based on seed questions, while MAVIS (Zhang et al., 2024c)
automatically constructs QA pairs through templates.

With the rapid advancement of LVLMs, there is a growing need for high-quality benchmark tests to assess math problem-solving in visual settings. While previous attempts like GeoQA (Chen et al., 2021) and Geometry3K (Lu et al., 2021) primarily focused on geometric problems, the recently proposed MathVista (Lu et al., 2024) takes it a step further by encompassing a diverse range of multimodal tasks involving mathematical reasoning. Mathverse (Zhang et al., 2024b), on the other hand, emphasizes textual and visual richness to gauge whether models truly understand image content.

3 Does Visual Modality Matter for Mathematical LVLMs?

To validate the importance of visual modality, we replicated the training processes of mainstream mathematical LVLMs under a unified architecture, with the primary differences lying in the training sets. By perturbing the alignment between images and text—either by shuffling correct pairs or masking image information—we observed that the data utilized in mathematical LVLMs fails to enable the model to establish a strong correlation between images and text.

3.1 Setups

Mathematical LVLMs We study the recently proposed mathematical LVLMs, including: G-LLaVA (2023) which constructs an enhanced multimodal geometric dataset, Geo170K, based on the scalability of geometric problems and fine-tunes it based on LLaVA. Similarly, MathLLaVA (2024) also establishes a large-scale multi-type mathematics problem dataset, which includes filtered image clarity and question complexity for Chart Question Answering, Geometric Problem Solving, Math Word Problems, Textbook Question Answering, and Visual Question Answering. MAVIS (2024c) utilizes an automatic data engine to generate mathematical visual datasets covering areas such as plane geometry, analytic geometry, and functions, effectively bypassing the high cost of manual annotation. MultiMath (2024) collects textbooks, exercises, and exam questions from the K-12 curriculum and employs GPT to generate detailed CoT processes, featuring bilingual data.

Reproduction Details Since all LVLMs are based on the LLaVA architecture, which connects the language model core to the visual encoder, we standardize and choose deepseek-math-r1-7B as language model and clip-vit-large-patch14-336 as image encoder for reproduction (same as Peng et al. (2024)). The training process is divided into three stages: the pre-training stage, where the connection module of LLaVA is fine-tuned on alignment data (including math alignment data, if available); the SFT stage, where the alignment module and the language model are fine-tuned on LLaVA’s general data; and the Math SFT stage, where the alignment module and the language model are further fine-tuned on multimodal math QA questions. Detailed statistics can be found in Table 11.

Mathematical Benchmarks \( \mathcal{D}_{\text{math}} \) We evaluate the following datasets: MathVerse (2024b) offers a collection of 3,940 high-quality mathematical problems across various disciplines, enriched with relevant charts, showcasing varied dependencies on visual and textual cues. Conversely, MathVista (2024) presents a comprehensive benchmark with 1,000 instances from 28 multimodal datasets, emphasizing their acute visual comprehension and intricate reasoning skills. GeoQA (2021) consists of 754 pieces of data, mainly multiple-choice questions about plane geometry. MATHVision (2024a) comprises 304 high-quality mathematical problems sourced from authentic mathematics competitions. The questions span 16 distinct mathematical disciplines and are categorized into 5 difficulty levels. WeMath (2024) consists of 1,740 visual math questions, covering 67 hierarchical knowledge concepts and 5 levels of knowledge granularity. ¹

3.2 Visual Modality Perturbation

Data w/ correct images \( \boxed{\rightarrow} \): The standard training process, the model can learn the correspondence between images and text as well as the templates for question-answering during the Math SFT stage.

Data w/ random images \( \boxed{\times} \): During the math SFT stage, by shuffling the correspondence between images and texts while keeping the picture distribution unchanged, we ensure that after the correspondence is disrupted, the Q&A for the original picture will be assigned to another picture. In this

¹For Mathverse, MathVita, and MathVision, as they contain multiple test sets, we select the “testmini” subset for evaluation.
<table>
  <tr>
    <th rowspan="2">\( D_{\text{math}} \)</th>
    <th colspan="2">Ins@Stage1/2/3</th>
    <th>GeoQA</th>
    <th>MathVerse</th>
    <th>MathVista</th>
    <th>MathVision</th>
    <th>WeMath</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>G-LLaVA</td>
    <td>618K/665K/117K</td>
    <td>68.0</td>
    <td>24.7</td>
    <td>34.9</td>
    <td>8.9</td>
    <td>39.3</td>
    <td>35.2</td>
  </tr>
  <tr>
    <td>MathLLaVA</td>
    <td>558K/665K/339K</td>
    <td>57.4</td>
    <td>25.6</td>
    <td><b>45.3</b></td>
    <td><b>9.5</b></td>
    <td>42.6</td>
    <td>36.1</td>
  </tr>
  <tr>
    <td>MAVIS</td>
    <td>1.1M/665K/555K</td>
    <td>70.6</td>
    <td><b>31.5</b></td>
    <td>37.0</td>
    <td>6.9</td>
    <td>39.4</td>
    <td>37.1</td>
  </tr>
  <tr>
    <td>MultiMath</td>
    <td>1.2M/665K/1.0M</td>
    <td><b>74.4</b></td>
    <td>29.6</td>
    <td>44.5</td>
    <td>7.2</td>
    <td><b>44.7</b></td>
    <td><b>40.1</b></td>
  </tr>
  <tr>
    <td>G-LLaVA</td>
    <td>same as above</td>
    <td>71.0 +3.0</td>
    <td>22.4 –2.3</td>
    <td><b>36.2</b> +1.9</td>
    <td>8.2 –0.7</td>
    <td>35.2 –4.1</td>
    <td>34.6 –0.6</td>
  </tr>
  <tr>
    <td>MathLLaVA</td>
    <td>same as above</td>
    <td>56.6 –0.8</td>
    <td>23.3 –2.3</td>
    <td>34.4 –10.9</td>
    <td>9.5 –0.0</td>
    <td>38.5 –4.1</td>
    <td>32.5 –3.6</td>
  </tr>
  <tr>
    <td>MAVIS</td>
    <td>same as above</td>
    <td>70.4 –0.2</td>
    <td><b>26.0</b> –5.5</td>
    <td>32.8 –4.2</td>
    <td><b>10.2</b> +3.3</td>
    <td>34.3 –5.1</td>
    <td>34.7 –2.4</td>
  </tr>
  <tr>
    <td>MultiMath</td>
    <td>same as above</td>
    <td><b>76.5</b> +2.1</td>
    <td>25.6 –4.0</td>
    <td>35.0 –9.5</td>
    <td>9.2 +2.0</td>
    <td><b>39.5</b> –5.2</td>
    <td><b>37.2</b> –2.9</td>
  </tr>
  <tr>
    <td>G-LLaVA</td>
    <td>same as above</td>
    <td>68.2 +0.2</td>
    <td>23.7 –1.0</td>
    <td>34.5 –0.4</td>
    <td>8.6 –0.3</td>
    <td><b>41.8</b> –2.5</td>
    <td>35.4 –0.2</td>
  </tr>
  <tr>
    <td>MathLLaVA</td>
    <td>same as above</td>
    <td>56.9 –0.5</td>
    <td>23.9 –1.7</td>
    <td>36.2 –9.1</td>
    <td>6.9 –2.6</td>
    <td>39.7 –2.9</td>
    <td>32.7 –3.4</td>
  </tr>
  <tr>
    <td>MAVIS</td>
    <td>same as above</td>
    <td>66.1 –4.5</td>
    <td><b>27.0</b> –4.5</td>
    <td>31.3 –5.7</td>
    <td><b>10.2</b> +3.3</td>
    <td>36.3 –3.1</td>
    <td>34.2 –2.9</td>
  </tr>
  <tr>
    <td>MultiMath</td>
    <td>same as above</td>
    <td><b>70.3</b> –4.1</td>
    <td>24.7 –4.9</td>
    <td><b>37.1</b> –7.4</td>
    <td>8.6 +1.2</td>
    <td>41.1 –3.6</td>
    <td><b>36.4</b> –3.7</td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">\( D_{\text{general}} \)</th>
    <th colspan="2">Ins@Stage1/2</th>
    <th>VQAv2</th>
    <th>MMBench</th>
    <th>MM-Vet</th>
    <th>MME_P†</th>
    <th>MME_C†</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>LLaVA-1.5</td>
    <td>558K/665K</td>
    <td><b>79.2</b></td>
    <td><b>66.8</b></td>
    <td><b>32.4</b></td>
    <td><b>1470.1</b></td>
    <td><b>322.1</b></td>
    <td><b>35.9</b></td>
  </tr>
  <tr>
    <td>LLaVA-1.5</td>
    <td>same as above</td>
    <td>46.2 –33.0</td>
    <td>26.6 –40.2</td>
    <td>12.2 –20.2</td>
    <td>708.9 –761.2</td>
    <td>276.8 –45.3</td>
    <td>17.1 –18.8</td>
  </tr>
  <tr>
    <td>LLaVA-1.5</td>
    <td>same as above</td>
    <td>60.8 –18.4</td>
    <td>54.3 –12.5</td>
    <td>20.9 –11.5</td>
    <td>706.0 –764.1</td>
    <td>271.8 –50.3</td>
    <td>27.3 –8.6</td>
  </tr>
</table>

Table 1: Mathematical reasoning performance of using correct images, shuffled images, and no images during the training phase. “Ins@Stage1/2/3” indicates the stages 1, 2, and 3 instances. The † means we normalize MME_P and MME_C for the average score.

process, the model can learn the statistical information of the picture distribution and the template for answering questions.

No images \( \rightarrow \): For the Math SFT stage, we remove the images from the training set and retain the question and answer. This helps the model learn the template for question-answering and enhances the quality of the responses.

Observations The results are presented in the Table 1, we can observe that:

• Firstly, after standardizing the model structure and training methodologies, the discrepancies among different approaches diminish (compared to the contrasts presented in Zhang et al. (2024c); Peng et al. (2024)). Contrary to the direct finetuning from LLaVA for alignment as discussed in the G-LLaVA paper (Gao et al., 2023), we notice a significant improvement with a three-stage training approach, aligning with the findings of Peng et al. (2024). The MultiMath model exhibits superior performance across datasets, with the exception of MathVerse, attributable to its diverse dataset and detailed reasoning processes.

• Secondly, we observe that substituting the correct images with shuffled ones only slightly impacts model performance. This trend is consistent across different models and datasets. Specifically, average performance decreases by 0-4 percentage points. In contrast, the GeoQA and Math-Vision datasets may even experience some degree of improvement. From a model perspective, high-performance models are more affected by image shuffling than lower-performance models. Compared to other models, the G-LLaVA model exhibits less performance fluctuation.

• Lastly, similar experimental outcomes are observed in the absence of images. These results suggest that during the mathematical SFT phase, image patterns do not significantly enhance reasoning performance, indicating that models primarily rely on text to learn question-and-answer capabilities in the mathematical domain.

Compared with general tasks \( D_{\text{general}} \) To observe if a similar phenomenon exists in the general VQA task, we follow the training process of LLaVA-1.5 with vicuna-7B (Chiang et al., 2023) and examine the impact of visual information on VQA models by comparing the performance differences between three scenarios: shuffled images, removed images, and provided correct images. We select the following commonly used VQA evaluation datasets: VQAv2 (2017), MMBench (2024a), MM-Vet (2024b), and MME (2023).

The results in Table 1 indicate a significant impact on model performance in general VQA tasks when images are either shuffled randomly or not provided at all. Not providing images leads to markedly better performance than offering random images: without images, model performance drops by approximately 23% on VQAv2 and 19% on
<table>
  <tr>
    <th>Model</th>
    <th>Input</th>
    <th>#Param.</th>
    <th>GeoQA</th>
    <th>M.Verse</th>
    <th>M.Vista</th>
    <th>M.Vision</th>
    <th>WeM.</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Deepseek-Math-RL (2024)</td>
    <td>7B</td>
    <td>39.8</td>
    <td>20.7</td>
    <td>26.1</td>
    <td>11.5</td>
    <td>33.3</td>
    <td>26.3</td>
  </tr>
  <tr>
    <td>Qwen-2.5-Math-Instruct (2024)</td>
    <td>7B</td>
    <td>62.2</td>
    <td>23.7</td>
    <td>31.4</td>
    <td>9.2</td>
    <td>40.2</td>
    <td>33.3</td>
  </tr>
  <tr>
    <td>Math-Shepherd-Mistral-RL (2024c)</td>
    <td>7B</td>
    <td>30.6</td>
    <td>12.4</td>
    <td>28.6</td>
    <td>4.9</td>
    <td>24.0</td>
    <td>20.1</td>
  </tr>
  <tr>
    <td>OpenMath2-Llama3.1 (2024)</td>
    <td>8B</td>
    <td>43.9</td>
    <td>22.2</td>
    <td>30.0</td>
    <td>8.2</td>
    <td>35.6</td>
    <td>28.0</td>
  </tr>
  <tr>
    <td>Dart-Math-dsmath-prop2diff (2024b)</td>
    <td>7B</td>
    <td>43.6</td>
    <td>19.2</td>
    <td>26.6</td>
    <td>11.5</td>
    <td>38.1</td>
    <td>27.8</td>
  </tr>
  <tr>
    <td>Meta-Llama-3.1-Instruct (2024)</td>
    <td>70B</td>
    <td>37.0</td>
    <td>20.6</td>
    <td>32.7</td>
    <td>9.9</td>
    <td>33.7</td>
    <td>26.8</td>
  </tr>
  <tr>
    <td>OpenMath2-Llama3.1 (2024)</td>
    <td>70B</td>
    <td>48.3</td>
    <td>24.6</td>
    <td>31.6</td>
    <td>12.2</td>
    <td><b>41.2</b></td>
    <td>31.6</td>
  </tr>
  <tr>
    <td>Dart-Math-Llama3-prop2diff (2024b)</td>
    <td>70B</td>
    <td>41.9</td>
    <td>17.6</td>
    <td>27.8</td>
    <td>9.9</td>
    <td>33.8</td>
    <td>26.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-Math-Instruct (2024)</td>
    <td>72B</td>
    <td><b>68.0</b></td>
    <td><b>32.3</b></td>
    <td>34.6</td>
    <td><b>12.5</b></td>
    <td>32.9</td>
    <td><b>36.1</b></td>
  </tr>
  <tr>
    <td>Qwen2-Instruct-Step-DPO (2024)</td>
    <td>72B</td>
    <td>61.9</td>
    <td>21.8</td>
    <td><b>35.0</b></td>
    <td>9.9</td>
    <td>30.9</td>
    <td>31.9</td>
  </tr>
  <tr>
    <td>GPT-4o (2024)</td>
    <td>/</td>
    <td>57.7</td>
    <td>25.4</td>
    <td>30.7</td>
    <td>10.5</td>
    <td>33.9</td>
    <td>31.6</td>
  </tr>
  <tr>
    <td>InternVL2 (2023)</td>
    <td>8B</td>
    <td>56.8</td>
    <td>31.5</td>
    <td>56.9</td>
    <td>8.6</td>
    <td>45.4</td>
    <td>39.8</td>
  </tr>
  <tr>
    <td>Qwen2-VL-Instruct (2024d)</td>
    <td>8B</td>
    <td>50.3</td>
    <td>32.2</td>
    <td>60.9</td>
    <td>8.6</td>
    <td>33.9</td>
    <td>37.2</td>
  </tr>
  <tr>
    <td>InternVL2-Llama3 (2023)</td>
    <td>76B</td>
    <td>59.2</td>
    <td>38.4</td>
    <td>62.8</td>
    <td>10.5</td>
    <td>55.8</td>
    <td>45.3</td>
  </tr>
  <tr>
    <td>Qwen2-VL-Instruct (2024d)</td>
    <td>72B</td>
    <td><b>68.8</b></td>
    <td>46.7</td>
    <td><b>70.4</b></td>
    <td>13.8</td>
    <td>61.5</td>
    <td><b>52.2</b></td>
  </tr>
  <tr>
    <td>GPT-4o (2024)</td>
    <td>/</td>
    <td>58.9</td>
    <td><b>46.8</b></td>
    <td>53.1</td>
    <td><b>15.5</b></td>
    <td><b>68.3</b></td>
    <td>48.5</td>
  </tr>
</table>

Table 2: Results of common text and visual language models on mathematical datasets. “#Param.” indicates the number of parameters. “M.” is the short for “Math”.

MMBench; with random images, the decline is more drastic at 42% and 61%, respectively. This suggests that models can detect inconsistencies between randomized image content and text, thereby disrupting learning. In contrast, the trend of performance decline in the mathematical domain is much weaker. We speculate this is because multimodal mathematical reasoning tasks rely more on generative aspects and less on image comprehension, a discrepancy that may not be as apparent in general VQA tasks.

4 Issues in Existing Benchmarks

We further explore the reasons for the overestimation of the image modality in multi-modal mathematical reasoning tasks. Through a careful examination of existing evaluation sets, we identify two main issues.

Firstly, a large proportion of the existing evaluation dataset can be answered correctly based on the text alone, which does not accurately reflect whether the image modality is effectively utilized. We conduct an evaluation on 7 common textual models and and 3 vision language models. The experimental results are shown in Table 2. As compared to the results in Table 1, the existing multimodal math methods show limited improvement based on the language model DeepSeek-Math-7B-RL. For example, MultiMath exhibits enhanced performance on the MathVerse and Wemath datasets, increasing from 20.7 to 29.6 and from 33.3 to 44.7, respectively. However, it even experiences a degree of decline on the Math-Vision dataset, dropping from 11.5 to 7.2.

We also observe that as the capability of the language model improves, the performance of the text model is further enhanced. In terms of average performance across all datasets, when the language model is switched from DeepSeek-Math-7B-RL to Qwen-2.5-Math-7B-Instruct, the score increase from 26.3 to 33.3. Meanwhile, the average performance of the G-LLaVA model is only 35.2, indicating that the performance of language models is very close to that of multimodal models.

Secondly, the options in the question may leak the answer, and the model can guess the answer based on the option information. We shuffle the order of the options in the multiple-choice questions, observe the changes in the model’s predictions before and after, and determine whether the model truly understands the question or is just making predictions based on option information. Table 3 shows the experimental results. We calculate the original accuracy (CR: Correct), the proportion of predictions are correct both before and after shuffled (BC: Both Correct), and the proportion of consistent predictions before and after (AR: Agreement). As can be seen from the table, in the case of multiple-choice questions, the BC indicator is significantly lower than the CR indicator. For example, the BC/CR indicators of the MultiMath model on the three datasets are 9.5%, 76.1%, and 80.7%, respectively. This indicates that the model’s single-performance is not sufficient to reflect its overall performance on the dataset. Additionally, a very low BC/CR could indicate potential quality issues
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">GeoQA</th>
    <th colspan="3">MathVerse</th>
    <th colspan="3">MathVista</th>
  </tr>
  <tr>
    <th>CR<sub>r</sub></th>
    <th>BC<sub>r</sub></th>
    <th>AG<sub>r</sub></th>
    <th>CR<sub>t</sub></th>
    <th>BC<sub>t</sub></th>
    <th>AG<sub>t</sub></th>
    <th>CR<sub>r</sub></th>
    <th>BC<sub>r</sub></th>
    <th>AG<sub>r</sub></th>
  </tr>
  <tr>
    <td>Gillava</td>
    <td>68.0</td>
    <td><b>16.0</b></td>
    <td><b>20.8</b></td>
    <td>44.4</td>
    <td>30.3</td>
    <td>50.6</td>
    <td>51.7</td>
    <td>40.9</td>
    <td>70.4</td>
  </tr>
  <tr>
    <td>Mathillava</td>
    <td>57.4</td>
    <td>7.6</td>
    <td>13.3</td>
    <td>40.0</td>
    <td>26.2</td>
    <td>53.4</td>
    <td><b>58.7</b></td>
    <td><b>49.3</b></td>
    <td><b>75.4</b></td>
  </tr>
  <tr>
    <td>MAVIS</td>
    <td>70.6</td>
    <td>8.2</td>
    <td>10.1</td>
    <td>41.9</td>
    <td>29.0</td>
    <td>49.1</td>
    <td>54.6</td>
    <td>44.8</td>
    <td>72.0</td>
  </tr>
  <tr>
    <td>MultiMath</td>
    <td><b>74.4</b></td>
    <td>7.1</td>
    <td>8.6</td>
    <td><b>48.1</b></td>
    <td><b>36.6</b></td>
    <td><b>56.6</b></td>
    <td>55.0</td>
    <td>44.4</td>
    <td>72.4</td>
  </tr>
</table>

Table 3: Results of shuffling the order of the multiple-choice options in the dataset. The table indicates the accuracy (CR: Correct), predictions are correct both before and after shuffled (BC: Both Correct), and predictions remained consistent across both attempts (AG: Agreement).

with the dataset.

Based on our observations, we believe that existing multimodal math datasets do not accurately measure the importance of the image modality. Since a large portion of the questions can be answered by textual models alone and the model can guess the answers, these issues lead to an overestimation of the capability of the visual modality, especially in multiple-choice questions. To improve the accuracy of the assessment, it is essential to mitigate the above two issues. Establishing a suitable benchmark will significantly advance the research on multimodal mathematic models.

5 The HC-M3D Benchmark

Based on the analysis above, we propose HC-M3D, a high-quality, vision-dependent multimodal benchmark. Our dataset construction adheres to the following three principles.

• First, correctness of the data, meaning that the collected data must be solvable based on the question and image asked and the answer is correct.

• Second, visual dependency, indicating that the data must rely on images for accurate answers.

• Lastly, high correlation between images and answers. Where possible, we aim to keep the question unchanged but alter the image to ensure that the answer changes. This methodology tests whether the model can detect changes in the images and make correct predictions.

Data Curation Our dataset is sourced from the GeoQA and MathVista GPS subsets, along with a selection of questions from the Chinese Jingyou Net 2, primarily focusing on plane geometry knowledge. We manually annotate each sample for correctness and visual dependency, modifying the original questions when these criteria are not met and attempting to supply an additional image for the question to change the correct answer. Instructions for human annotators of these two steps are shown in Figure 5 and Figure 6. Ultimately, we obtain 1,851 samples, including 429 questions for which an additional image was provided. Statistical results for the dataset are presented in Table 4.

<table>
  <tr>
    <th>Statistic</th>
    <th>Number</th>
  </tr>
  <tr>
    <td>Total questions</td>
    <td>1,851</td>
  </tr>
  <tr>
    <td>- Multiple-choice questions</td>
    <td>1,851 (100.0%)</td>
  </tr>
  <tr>
    <td>- Newly collected questions</td>
    <td>1,219 (65.9%)</td>
  </tr>
  <tr>
    <td>- Newly collected images</td>
    <td>1,084 (58.6%)</td>
  </tr>
  <tr>
    <td>Data Source</td>
    <td></td>
  </tr>
  <tr>
    <td>- GeoQA</td>
    <td>896 (48.4%)</td>
  </tr>
  <tr>
    <td>- MathVista</td>
    <td>273 (14.7%)</td>
  </tr>
  <tr>
    <td>- Jingyou Net</td>
    <td>684 (37.0%)</td>
  </tr>
  <tr>
    <td>Language</td>
    <td></td>
  </tr>
  <tr>
    <td>- English</td>
    <td>1,052 (56.8%)</td>
  </tr>
  <tr>
    <td>- Chinese</td>
    <td>799 (43.2%)</td>
  </tr>
  <tr>
    <td>Number of unique images</td>
    <td>1,851 (100.0%)</td>
  </tr>
  <tr>
    <td>Number of unique questions</td>
    <td>993 (53.6%)</td>
  </tr>
</table>

Table 4: Statistical Results on the HC-M3D Dataset.

Figure 2 illustrates the process of constructing the dataset. Upon manually identifying a lack of visual dependency in the data, the text (or image) is revised to ensure that the question requires reliance on the image for an answer. For instance, the image on the left can be drawn based on the question, demonstrating low dependency on the image; thus, a revision is necessary. Under conditions where dependency is met, as with the image on the right, changing the position of point A on the circle alters the correct answer from D to A. In this scenario, there is a high correlation between the image and the correct answer, necessitating that the model explicitly perceives the differences in the image to make accurate predictions.

Our HC-M3D stands out by modifying images and using a controlled-variable method to verify whether the model can truly and accurately understand the image content. The challenge with MathVerse lies in precisely distributing information, a process that is somewhat ambiguous. In contrast, H3-M3D changes the answer solely through image modifications, with the text remaining unchanged. Our control over the text information is thorough and successful (annotators simply need to determine whether the modified image corresponds to a new answer).

2https://www.jyeoo.com/
Lack of Visual Dependency

As shown in the figure, points A and B are three points on ⊙O and AB = AC. Connect BO and CO, if ∠ABC = 65.0, then the degree of ∠BOC is ( )

Choices:
A: 50
B: 65
C: 100
D: 130

enhance

As shown in the figure, points A and B are three points on ⊙O and AB = AC. Connect BO and CO, if ∠ABC = 65.0, then the degree of ∠BOC is ( )

Choices:
A: 50
B: 65
C: 100
D: 130

remove redundant words

MathVista testmini: question-270

Correlation between Images and Answers

As shown in the figure, A, B, C are any three points on circle O, if angle BOC = 100.0, then the degree of angle BAC is ( )

Choices:
A: 50°
B: 80°
C: 100°
D: 130°

complement

As shown in the figure, A, B, C are any three points on circle O, if angle BOC = 100.0, then the degree of angle BAC is ( )

Choices:
A: 50°
B: 80°
C: 100°
D: 130°

same question different answer

GeoQA test: question-48

Figure 2: Examples from our constructed HC-M3D benchmark. The left image demonstrates a scenario where the original data lacks visual dependency, allowing for the reconstruction of the question’s corresponding image solely from text. This is addressed by rewriting or possibly changing the image to ensure the question requires image dependency for a correct answer. The right image shows an example where supplementing the original question with a similar image alters the correct answer. More examples can be found in Figure 3 and Figure 4.

<table>
  <tr>
    <th>Model</th>
    <th>#Param.</th>
    <th>ALL ↑</th>
    <th>DI ↑</th>
    <th>BC ↑</th>
    <th>AG ↓</th>
  </tr>
  <tr>
    <td>G-LLaVA</td>
    <td>7B</td>
    <td>45.4</td>
    <td>41.5</td>
    <td>15.2</td>
    <td>52.2</td>
  </tr>
  <tr>
    <td>MathLLaVA</td>
    <td>7B</td>
    <td>39.6</td>
    <td>37.2</td>
    <td>8.4</td>
    <td>75.3</td>
  </tr>
  <tr>
    <td>MAVIS</td>
    <td>7B</td>
    <td>42.8</td>
    <td>37.7</td>
    <td>9.8</td>
    <td>58.0</td>
  </tr>
  <tr>
    <td>MutiMath</td>
    <td>7B</td>
    <td>49.2</td>
    <td>44.8</td>
    <td>16.6</td>
    <td>56.9</td>
  </tr>
  <tr>
    <td>InternVL2</td>
    <td>8B</td>
    <td>41.9</td>
    <td>38.3</td>
    <td>16.6</td>
    <td>34.0</td>
  </tr>
  <tr>
    <td>Qwen2-VL-Instruct</td>
    <td>8B</td>
    <td>40.9</td>
    <td>40.7</td>
    <td>13.8</td>
    <td>58.7</td>
  </tr>
  <tr>
    <td>InternVL2-Llama3</td>
    <td>76B</td>
    <td>47.4</td>
    <td>45.5</td>
    <td>19.4</td>
    <td>40.8</td>
  </tr>
  <tr>
    <td>Qwen2-VL-Instruct</td>
    <td>76B</td>
    <td><b>51.8</b></td>
    <td><b>48.3</b></td>
    <td><b>20.3</b></td>
    <td>51.5</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>/</td>
    <td>49.0</td>
    <td>45.8</td>
    <td>19.1</td>
    <td>42.0</td>
  </tr>
</table>

Table 5: Evaluation results on the HC-M3D benchmark. The ↑ and ↓ arrows respectively indicate that higher or lower values are preferred.

Metrics Performance on the dataset is evaluated based on the following metrics. Accuracy over all data (ALL); accuracy on a subset (DI: Diverse Image) consisting of 858 samples that share similar questions but have different images and answers; accuracy (BC: Both Correct) for cases within the DI subset where both samples sharing the same question are correctly predicted; and consistency rate (AG: Agreement) within the DI subset for samples that share the same question and are predicted consistently. Since these samples have differing answers, a lower AG metric is preferred.3

Evaluation Experimental results are shown in Table 5. It is observed that among existing multi-modal mathematical models, MutiMath achieves the highest accuracy of 49.2, comparable to the performance of GPT-4o. The open-source model Qwen2-VL-Instruct-76B, reaches an highest accuracy of 51.8. Furthermore, in the DI subset, which accounts for image and answer relevance, the proportion of question pairs correctly predicted/Performance on the DI subset (BC/DI) remains below 50%. This indicates that the models do not deeply understand questions for which half of the answers are correct; changing the image leads to incorrect predictions. At the same time, the AG metric for different models is generally above 50%, suggesting that models do not appropriately change their answers when the image is replaced. The GPT-4o and InternVL2 models perform best in this metric.

3The visual control experiment results on the proposed HC-M3D benchmark can be found in Table 8.

6 Challenges in Enhancing Visual Dependency

We observe that existing multimodal mathematical models exhibit weak reliance on visual information (Section 3), demonstrating insufficient sensitivity to changes in images (Section 5). Given recent findings indicating that the CLIP models struggle to capture object information (Liu et al., 2024b), spatial relationships (Kamath et al., 2023), and compositional understanding (Zhang et al., 2024a) in images during encoding, we hypothesize that these phenomena stem from the limited capabilities of
<table>
  <tr>
    <th rowspan="2">Image Encoder</th>
    <th colspan="5">General VQA Performance after SFT stage</th>
    <th colspan="6">Math Performance after Math SFT stage</th>
  </tr>
  <tr>
    <th>VQA<sup>1/2</sup></th>
    <th>MMBench</th>
    <th>MM-Vet</th>
    <th>MME<sup>P</sup></th>
    <th>MME<sup>C</sup></th>
    <th>GeoQA</th>
    <th>M.Verse</th>
    <th>M.Vista</th>
    <th>M.Vision</th>
    <th>WeM.</th>
    <th>HC-M3D</th>
  </tr>
  <tr>
    <td>clip-B</td>
    <td>72.6</td>
    <td>60.7</td>
    <td>25.7</td>
    <td>1365.8</td>
    <td>273.6</td>
    <td>71.0</td>
    <td>23.0</td>
    <td>31.4</td>
    <td>9.5</td>
    <td>38.7</td>
    <td>41.2</td>
  </tr>
  <tr>
    <td>clip-L</td>
    <td>77.0</td>
    <td>64.8</td>
    <td>30.4</td>
    <td>1481.0</td>
    <td>269.6</td>
    <td>68.0</td>
    <td>24.7</td>
    <td>34.9</td>
    <td>8.9</td>
    <td>39.3</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td colspan="12">(a) combine features by concatenating the hidden feature</td>
  </tr>
  <tr>
    <td>clip-B+ siglip-B</td>
    <td>74.6</td>
    <td>62.0</td>
    <td>25.7</td>
    <td>1397.5</td>
    <td>257.5</td>
    <td><b>72.3</b></td>
    <td>22.2</td>
    <td>31.1</td>
    <td>8.2</td>
    <td>38.1</td>
    <td>40.0</td>
  </tr>
  <tr>
    <td>clip-L+dino2-L</td>
    <td>78.4</td>
    <td>66.9</td>
    <td>30.9</td>
    <td>1500.1</td>
    <td>253.2</td>
    <td>70.3</td>
    <td>23.2</td>
    <td>30.0</td>
    <td>12.2</td>
    <td>38.6</td>
    <td>40.3</td>
  </tr>
  <tr>
    <td>clip-L+siglip-B+dino-B</td>
    <td>74.4</td>
    <td>62.5</td>
    <td>25.5</td>
    <td>1404.0</td>
    <td>280.0</td>
    <td>70.8</td>
    <td>22.5</td>
    <td>31.0</td>
    <td>8.9</td>
    <td>36.0</td>
    <td>40.5</td>
  </tr>
  <tr>
    <td colspan="12">(b) combine features by concatenating image tokens</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L</td>
    <td>78.7</td>
    <td>66.6</td>
    <td>33.0</td>
    <td>1445.2</td>
    <td>266.4</td>
    <td>69.9</td>
    <td>22.6</td>
    <td>30.8</td>
    <td>8.9</td>
    <td>37.2</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td>clip-L+dino2-L</td>
    <td>78.6</td>
    <td>66.7</td>
    <td>32.3</td>
    <td>1494.7</td>
    <td>269.6</td>
    <td>70.7</td>
    <td>22.3</td>
    <td>28.7</td>
    <td>7.6</td>
    <td>35.7</td>
    <td>39.2</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L+dino2-L</td>
    <td><b>80.0</b></td>
    <td><b>69.8</b></td>
    <td><b>35.8</b></td>
    <td><b>1524.6</b></td>
    <td><b>301.8</b></td>
    <td>69.9</td>
    <td>22.6</td>
    <td>28.5</td>
    <td>8.9</td>
    <td>38.1</td>
    <td>39.4</td>
  </tr>
</table>

Table 6: Performance on general VQA and mathematical tasks. clip-L, sigLip-L, and dino-L correspond to clip-vit-large-patch14-336, siglip-so400m-patch14-384, and dinov2-large models respectively. siglip-B (siglip-base-patch16-224) and dino-B (dino-vitb16) are utilized alongside clip-B (clip-vit-base-patch16) for matching image token numbers in combinations. Detailed results can be found in Table 12 and Table 13.

<table>
  <tr>
    <th>Image Encoder</th>
    <th>POPE</th>
    <th>MathPOPE</th>
  </tr>
  <tr>
    <td>clip-B</td>
    <td>81.4</td>
    <td>72.9</td>
  </tr>
  <tr>
    <td>clip-L</td>
    <td><b>85.2</b></td>
    <td>73.7</td>
  </tr>
  <tr>
    <td colspan="3">(a) concatenating the hidden feature</td>
  </tr>
  <tr>
    <td>clip-B+siglip-B</td>
    <td>82.2</td>
    <td>72.6</td>
  </tr>
  <tr>
    <td>clip-L+dino2-L</td>
    <td>84.4</td>
    <td>70.1</td>
  </tr>
  <tr>
    <td>clip-L+siglip-B+dino-B</td>
    <td>80.1</td>
    <td>76.2</td>
  </tr>
  <tr>
    <td colspan="3">(b) concatenating image tokens</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L</td>
    <td>83.9</td>
    <td>67.6</td>
  </tr>
  <tr>
    <td>clip-L+dino2-L</td>
    <td>84.2</td>
    <td>71.9</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L+dino2-L</td>
    <td>84.1</td>
    <td><b>81.0</b></td>
  </tr>
</table>

Table 7: F1 performance on both POPE and MathPOPE datasets. POPE primarily focuses on questions about objects in images, whereas MathPOPE concentrates on points existing in geometric images.

the image encoders. Recent study (Al-Tahan et al., 2024) find that although scaling up the dataset and model parameters in the CLIP series proves effective for general tasks, such enhancements have minimal impact (even negative impact) on performance in complex tasks involving reasoning and relationship understanding. Therefore, we attempt to leverage the strengths of diverse types of encoders, which is also commonly used in general vqa tasks (Tong et al., 2024a; Yang et al., 2024), to enhance visual dependency and final performance.

SigLip (Zhai et al., 2023) and DINO (Caron et al., 2021; Oquab et al., 2024) are selected as the components of our combination, where SigLIP replaces the loss function used in CLIP with a simple pairwise sigmoid loss function, resulting in improved zero-shot classification performance; DINO, as a self-supervised visual model, is capable of capturing rich information from images. We consider two methods for feature concatenation:

• Hidden Feature Concatenation Images are input into multiple image encoders, and the dimensions of the hidden representation are concatenated, then mapped back to the original hidden layer output dimensions through the modality connection module. In this case, it is necessary for the image tokens across different representations to be consistent;

• Image Token Concatenation Images are input into multiple image encoders and mapped to the same hidden layer dimensions according to their respective modality connection module, with the image tokens then concatenated. This method does not require the dimensions across different representations to be the same, but it increases the overall number of image tokens.4

Results We experiment on G-LLaVA model 5, and the results are shown in Table 6. We can find that:

• First, we find that integrating multiple image encoders, specifically SigLip and DINO, either separately or combined, significantly improves performance in general Visual Question Answering (VQA) tasks. The optimal performance is achieved by concatenating image tokens from all three encoders without dimensionality constraints, which elevates MME perception scores from 1481 to 1524.

• Second, an in-depth analysis of various metrics within the MME demonstrates notable advancements in color perception and spatial relation-

4 Experiment results of interleaving image tokens from different encoders as done in (Tong et al., 2024a) when combining multiple visual encoders can be found in Table 10.
5 Experiment results on MathLLaVA model can be found in Table 9.
ships (see Table 12). This suggests that utilizing diverse encoders allows the model to discern more detailed image features.

• Lastly, while enhancements are observed in general tasks, there is a consistent dip in performance on mathematical evaluation tasks, with only a slight increase seen in GeoQA. Moreover, the addition of multiple image encoders, such as combining SigLip and DINO, does not proportionally enhance performance beyond the use of either encoder on its own (i.e., the model’s performance does not increase with the addition of more image encoders). For instance, in MathVista, the performance metrics for combining SigLip alone, DINO alone, and both combined are 30.8, 28.7, and 28.5, respectively.

Hallucination Evaluation We aim to further investigate the causes behind this decline in performance. Given that the general improvement across common tasks suggests an enhanced general capability of the model, we are curious to determine if a more severe presence of hallucinations has led to this performance decline. Considering the lack of assessments for hallucinations in the mathematical domain, inspired by POPE (Li et al., 2023), we construct MathPOPE (9,000 questions based on 500 images) based on annotations of points in images from the Geometry3K (Lu et al., 2021) test set, in order to assess model hallucinations. The format of the question in MathPOPE is: ‘Is there a point X in the image?’, where X refers to the point name in picture. The questions in the dataset are designed such that the objects are present and absent in equal measure, therefore the ideal ‘yes’ response rate should be around 50%.

Experimental results can be seen in Table 7. We observe that combining multiple image encoders indeed results in an increase in hallucinations to a certain extent. However, distinct from the general VQA tasks, in the domain of mathematical hallucination assessment, we note a reduction in hallucinations when three image encoders are combined (73.7 vs 81.0). Moreover, while there is a clear positive correlation between VQA performance in general tasks and hallucination performance, such correlation is absent in the domain of mathematical reasoning.

Analysis The results from the above experiments reveal a significant difference in the performance improvements within the mathematical domain compared to the findings in general VQA tasks. We believe there are two possible reasons for this. Firstly, images in the mathematical domain usually feature monochromatic colors and lower information density (see Figure 2), which markedly distinguishes them from general images. Secondly, while there is a close association between performance and hallucination assessment in general tasks, the linkage between reasoning performance and hallucinations appears to be weaker. Therefore, reasoning tasks in the mathematical domain present unique challenges, making it significantly difficult to improve modality dependence. We leave further exploration to future work.

7 Conclusion

We study the importance of the visual modality within multimodal mathematical models. Our findings indicate that the performance of existing multimodal mathematical models does not significantly deteriorate when images are shuffled or even removed. We propose HC-M3D benchmark to measure the model’s sensitivity to changes in images. Experiments demonstrate that the model does not alter its answers corresponding to changes in the images when there is a high relevance between images and answers. We further discover that enhancing the dependency on the visual modality in the mathematical domain presents significant challenges, and we leave further exploration to subsequent work.

Limitations

Firstly, this paper focuses on the importance of the visual modality in mathematical LVLMs during pre-training and instruction fine-tuning, without addressing reasoning stages such as chain-of-thought, which will be part of our future work. Secondly, due to the high cost and time-consuming nature of manual annotation, especially generating new images for different options of the same question, the scale of our proposed HC-M3D dataset is relatively small, consisting of only 1,851 samples. Lastly, although the method of integrating multiple visual experts shows performance improvements in general VQA tasks, it offers little benefit (and in some cases, even negative impact) when applied to mathematical reasoning evaluation. This highlights the differences between mathematical reasoning tasks and general tasks, urging us to explore solutions tailored to mathematical reasoning tasks.
Ethics Statement

The HC-M3D dataset is sourced from publicly available academic datasets or manually collected math problems. The annotations were performed by a professional labeling team from the company, with the task of filtering math problems and generating images. Therefore, no ethical issues are involved.

Acknowledgement

The authors wish to thank all reviewers for their helpful comments and suggestions. The corresponding authors are Tao Ji, Yuanbin Wu, Aimin Zhou and Mengdi Zhang. The computations in this research were performed using the CFFF platform of Fudan University.

References

Haider Al-Tahan, Quentin Garrido, Randall Balestriero, Diane Bouchacourt, Caner Hazirbas, and Mark Ibrahim. 2024. Unibench: Visual reasoning requires rethinking vision-language beyond scaling. CoRR, abs/2408.04810.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425–2433. IEEE Computer Society.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties in self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9630–9640. IEEE.

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, and Liang Lin. 2021. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, volume ACL/IJCNLP 2021 of Findings of ACL, pages 513–523. Association for Computational Linguistics.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. Intervl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. CoRR, abs/2312.14238.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibou, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Niall, Grégoire Milon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasudevan Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. 2024. The llama 3 herd of models. CoRR, abs/2407.21783.

Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, Katsushi Ikeuchi, Hoi Vo, Li Fei-Fei, and Jianfeng Gao. 2024. Agent AI: surveying the horizons of multimodal interaction. CoRR, abs/2401.03568.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, abs/2306.13394.

Jiahui Gao, Renjie Pi, Pipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. 2023. Gllava: Solving geometric problem with multi-modal large language model. CoRR, abs/2312.11370.

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In 2017 IEEE Conference
on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 6325–6334. IEEE Computer Society.

Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Ranjay Krishna. 2024. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. CoRR, abs/2406.09403.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Osstrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Amita Kamath, Jack Hessel, and Kai-Wei Chang. 2023. What’s “up” with vision-language models? investigating their struggle with spatial reasoning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 9161–9175. Association for Computational Linguistics.

Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. CoRR, abs/2406.18629.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 292–305. Association for Computational Linguistics.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. In Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pages 740–755. Springer.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Ji-qi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024a. Mmbench: Is your multi-modal model an all-around player? In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part VI, volume 15064 of Lecture Notes in Computer Science, pages 216–233. Springer.

Yufang Liu, Tao Ji, Changzhi Sun, Yuanbin Wu, and Aimin Zhou. 2024b. Investigating and mitigating object hallucinations in pretrained vision-language (CLIP) models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 18288–18301. Association for Computational Linguistics.

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 6774–6786. Association for Computational Linguistics.

Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. CoRR, abs/2308.09583.

Maxime Oquab, Timothée Darret, Théo Moutakamni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaddin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Malairat, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. 2024. Dinov2: Learning robust visual features without supervision. Trans. Mach. Learn. Res., 2024.

Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. 2024. Multimath: Bridging visual and mathematical reasoning for large language models. CoRR, abs/2409.00147.

Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma Gongque, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, Runfeng Qiao, Yifan Zhang, Xiao Zong, Yida Xu, Muxi Diao, Zhimin Bao, Chen Li, and Honggang Zhang. 2024. We-math: Does your large multimodal model achieve human-like mathematical reasoning? CoRR, abs/2407.01284.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300.

Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. 2024. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models.
In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 4663–4680. Association for Computational Linguistics.

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024a. Eyes wide shut? exploring the visual shortcomings of multi-modal llms. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 9568–9578. IEEE.

Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. 2024b. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. CoRR, abs/2407.13690.

Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. 2024. Openmathinstruc-2: Accelerating AI for math with massive open-source instruction data. CoRR, abs/2410.01560.

Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. 2024a. Measuring multimodal mathematical reasoning with math-vision dataset. CoRR, abs/2402.14804.

Lei Wang, Wanyu Xu, Zhiqiang Hu, Yihuai Lan, Shan Dong, Hao Wang, Roy Ka-Wei Lee, and Ee-Peng Lim. 2024b. All in a single image: Large multimodal models are in-image learners. CoRR, abs/2402.17971.

Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024c. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 9426–9439. Association for Computational Linguistics.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi-biao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024d. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. CoRR, abs/2409.12191.

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhaoo Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wenshen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui. 2023. The rise and potential of large language model based agents: A survey. CoRR, abs/2309.07864.

Shijia Yang, Bohan Zhai, Quanzeng You, Jianbo Yuan, Hongxia Yang, and Chenfeng Xu. 2024. Law of vision representation in mlms. CoRR, abs/2408.16357.

Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. Modeling context in referring expressions. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II, volume 9906 of Lecture Notes in Computer Science, pages 69–85. Springer.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024a. Meta-math: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024b. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11941–11952. IEEE.

Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2023. Vision-language models for vision tasks: A survey. CoRR, abs/2304.00685.

Le Zhang, Rabibul Awal, and Aishwarya Agrawal. 2024a. Contrasting intra-modal and ranking cross-modal hard negatives to enhance visio-linguistic compositional understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 13774–13784. IEEE.

Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, Peng Gao, and Hongsheng Li. 2024b. MATHVERSE: does your multi-modal LLM truly see the diagrams in visual math problems? In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part VIII, volume 15066 of Lecture Notes in Computer Science, pages 169–186. Springer.

Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhou Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, and Hongsheng Li. 2024c. MAVIS: mathematical visual instruction tuning. CoRR, abs/2407.08739.
<table>
  <tr>
    <th>math</th>
    <th>ALL ↑</th>
    <th>DI ↑</th>
    <th>BC ↑</th>
    <th>AG ↓</th>
  </tr>
  <tr>
    <td>G-LLaVA</td>
    <td>45.4</td>
    <td>41.5</td>
    <td>15.2</td>
    <td><b>52.2</b></td>
  </tr>
  <tr>
    <td>MathLLaVA</td>
    <td><b>49.2</b></td>
    <td><b>44.8</b></td>
    <td><b>16.6</b></td>
    <td>56.9</td>
  </tr>
  <tr>
    <td>G-LLaVA</td>
    <td>41.2</td>
    <td><b>39.9</b></td>
    <td><b>12.1</b></td>
    <td><b>54.8</b></td>
  </tr>
  <tr>
    <td>MultiMath</td>
    <td><b>42.3</b></td>
    <td>38.8</td>
    <td>11.2</td>
    <td>60.6</td>
  </tr>
  <tr>
    <td>G-LLaVA</td>
    <td>40.8</td>
    <td>37.2</td>
    <td><b>10.8</b></td>
    <td><b>57.3</b></td>
  </tr>
  <tr>
    <td>MultiMath</td>
    <td><b>42.8</b></td>
    <td><b>39.3</b></td>
    <td>10.3</td>
    <td>67.4</td>
  </tr>
</table>

Table 8: Performance on HC-M3D dataset of using correct images, shuffled images, and no images during the training phase.

A Visual Control Experiment on HC-M3D Benchmark

We evaluate whether HC-M3D can effectively distinguish the utilization of visual modalities on GLLaVA and MultiMath datasets. The results (shown in Table 8) demonstrate that perturbing or removing the images leads to a significant decline in overall accuracy (ALL). Moreover, under the image removal setting, the prediction consistency (BC) is higher than in both the normal and perturbed image settings, suggesting that HC-M3D can partially identify the reliance on visual modalities.

B More Experiments on Exploring Different Visual Encoders

<table>
  <tr>
    <th></th>
    <th>GeoQA</th>
    <th>MathVerse</th>
    <th>MathVista</th>
    <th>MathVision</th>
    <th>WeMath</th>
    <th>HC-M3D</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>clip-L</td>
    <td>68.0</td>
    <td>24.7</td>
    <td>34.9</td>
    <td>8.9</td>
    <td>39.3</td>
    <td>45.4</td>
    <td>36.9</td>
  </tr>
  <tr>
    <td>clip-L+dinov2-L</td>
    <td>70.7</td>
    <td>22.3</td>
    <td>28.7</td>
    <td>7.6</td>
    <td>35.7</td>
    <td>39.2</td>
    <td>34.0</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L+dinov2-L (2024a)</td>
    <td>66.7</td>
    <td>24.3</td>
    <td>31.9</td>
    <td>6.6</td>
    <td>38.7</td>
    <td>41.8</td>
    <td>35.0</td>
  </tr>
</table>

Table 10: Results of interleaving the images tokens from different encoders as done in (Tong et al., 2024a).

<table>
  <tr>
    <th></th>
    <th>GeoQA</th>
    <th>MathVerse</th>
    <th>MathVista</th>
    <th>MathVision</th>
    <th>WeMath</th>
    <th>HC-M3D</th>
  </tr>
  <tr>
    <td>clip-L</td>
    <td>57.4</td>
    <td><b>25.6</b></td>
    <td><b>45.3</b></td>
    <td>9.5</td>
    <td><b>42.6</b></td>
  </tr>
  <tr>
    <td>clip-L+siglip-L</td>
    <td>57.3</td>
    <td>24.3</td>
    <td>44.0</td>
    <td>10.9</td>
    <td>40.6</td>
  </tr>
  <tr>
    <td>clip-L+dinov2-L</td>
    <td><b>58.6</b></td>
    <td>24.0</td>
    <td>39.6</td>
    <td><b>12.2</b></td>
    <td>38.8</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L+dinov2-L</td>
    <td>58.0</td>
    <td>24.5</td>
    <td>37.6</td>
    <td>8.2</td>
    <td>37.9</td>
  </tr>
</table>

Table 9: Results of combining different visual encoders on mathematical tasks. Experiments are conducted on MathLLaVA model.

we additionally report results on the training set of MathLLaVA to provide a more comprehensive evaluation. (as shown in Table 1, MathLLaVA demonstrates a stronger dependency on image-based information. ) Here we combine different encoders by concatenating image tokens. The experimental results on MathLLaVA (shown in Table 9) are generally consistent with those on G-LLaVA (Table 6). Combining multiple encoders does not enhance the performance of complex reasoning on images. Although Math-LLaVA’s dataset exhibits a marginally higher degree of visual-textual dependency, the 3% disparity is not deemed to constitute a robust reliance (e.g., when compared with the LLaVA’s training dataset). Consequently, it is not feasible to derive a definitive conclusion from such a dataset. The development of an additional general geometry dataset characterized by a pronounced visual-textual dependency is reserved for future research endeavors.

We also try interleaving the tokens from different encoders as done in (Tong et al., 2024a) when combining multiple visual encoders. Experimental results (shown in Table 10) indicate that the interleaving input strategy achieved slightly better performance (average score of 35.0) compared to directly concatenating the representations of CLIP and DINoV2 (average score of 34.0). However, the performance of this method remains significantly lower than that of using CLIP representations alone (average score of 36.9).

C Training Details

Detailed information on model training are shown in 11. Given that existing multimodal mathematical models are based on the same LLaVA architecture with the main differentiation being the training data, we standardize the training approach as three stage: Pretrain, SFT and Math SFT with language model deepseek-math-r1-7b. All the models are trained on 8 NVIDIA A100-80GB GPUs with the random seed 42.
More Examples of Lacking Visual Dependency

<table>
  <tr>
    <th colspan="2">As shown in the figure, in the two concentric circles, the chord AB of the great circle is tangent to the small circle at point C. If AB = 6.0, the area of the ring is ()</th>
    <th colspan="2">As shown in the figure, in the two concentric circles, the chord AB of the great circle is tangent to the small circle at point C. The area of the ring is ()</th>
  </tr>
  <tr>
    <td>Choices:<br>A 9π<br>B 6π<br>C 3π<br>D π</td>
    <td></td>
    <td>Choices:<br>A 9π<br>B 6π<br>C 3π<br>D π</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="2">MathVista testmini: question-270</td>
    <td colspan="2"></td>
  </tr>
  <tr>
    <th colspan="2">As shown in the figure, PA and PB are tangent to circle O at points A and B respectively, the tangent EF of circle O intersects PA and PB at points E and F respectively, and the tangent point C is on arc AB. If the length of PA is 2.0, then the perimeter of triangle PEF is ()</th>
    <th colspan="2">As shown in the figure, PA and PB are tangent to circle O at points A and B respectively, the tangent EF of circle O intersects PA and PB at points E and F respectively, and the tangent point C is on arc AB. If the length of PA is 2.0, then the perimeter of triangle PEF is ()</th>
  </tr>
  <tr>
    <td>Choices:<br>A:8<br>B:6<br>C:4<br>D:2</td>
    <td></td>
    <td>Choices:<br>A:8<br>B:6<br>C:4<br>D:2</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="2">GeoQA test: question-283</td>
    <td colspan="2"></td>
  </tr>
</table>

Figure 3: More examples from our constructed HC-M3D benchmark where original samples are lacking visual dependency. (Supplement to Figure 2)

More Examples of Correlation between Images and Answers

<table>
  <tr>
    <th colspan="4">分别观察下列几何体，其中从正面看、从左面看完全相同的有（）</th>
    <th colspan="4">分别观察下列几何体，其中从正面看、从左面看完全相同的有（）</th>
  </tr>
  <tr>
    <td>A: 2个</td>
    <td>B: 3个</td>
    <td>C: 4个</td>
    <td>D: 5个</td>
    <td>A: 2个</td>
    <td>B: 3个</td>
    <td>C: 4个</td>
    <td>D: 5个</td>
  </tr>
  <tr>
    <td colspan="4">Jingyou test</td>
    <td colspan="4"></td>
  </tr>
  <tr>
    <th colspan="4">Find y.</th>
    <th colspan="4">Find y.</th>
  </tr>
  <tr>
    <td>Choices:<br>A: \(16\sqrt{2}\)<br>B: \(16\sqrt{3}\)<br>C: 32<br>D: \(16\sqrt{5}\)</td>
    <td></td>
    <td>Choices:<br>A: \(16\sqrt{2}\)<br>B: \(16\sqrt{3}\)<br>C: 32<br>D: \(16\sqrt{5}\)</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="4">MathVista testmini: question-716</td>
    <td colspan="4"></td>
  </tr>
</table>

Figure 4: More examples from our constructed HC-M3D benchmark where images highly related to answers. (Supplement to Figure 2)
<table>
  <tr>
    <th>Statistics</th>
    <th>G-LLaVA</th>
    <th>MathLLaVA</th>
    <th>MAVIS</th>
    <th>MultiMath</th>
  </tr>
  <tr>
    <th colspan="5">Pretrain stage</th>
  </tr>
  <tr>
    <td>Dataset</td>
    <td>Geo170K-align +LLaVA-Pretrain</td>
    <td>LLaVA-Pretrain</td>
    <td>MAVIS-align +LLava-Pretrain</td>
    <td>MultiMath-300K-align +Geo170K-align +LLaVA-Pretrain</td>
  </tr>
  <tr>
    <td>#Samples</td>
    <td>618K</td>
    <td>558K</td>
    <td>1.1M</td>
    <td>1.2M</td>
  </tr>
  <tr>
    <td>Training Module</td>
    <td>mm adapter</td>
    <td>mm adapter</td>
    <td>mm adapter</td>
    <td>mm adapter</td>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Batch_size * #gpu</td>
    <td>32*8</td>
    <td>32*8</td>
    <td>32*8</td>
    <td>32*8</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>1e-3</td>
    <td>1e-3</td>
    <td>1e-3</td>
    <td>1e-3</td>
  </tr>
  <tr>
    <th colspan="5">SFT stage</th>
  </tr>
  <tr>
    <td>Dataset</td>
    <td>LLaVA-Instruction</td>
    <td>LLaVA-Instruction</td>
    <td>LLaVA-Instruction</td>
    <td>LLaVA-Instruction</td>
  </tr>
  <tr>
    <td>#Samples</td>
    <td>665K</td>
    <td>665K</td>
    <td>665K</td>
    <td>665K</td>
  </tr>
  <tr>
    <td>Training Module</td>
    <td>mm adapter+llm</td>
    <td>mm adapter+llm</td>
    <td>mm adapter+llm</td>
    <td>mm adapter+llm</td>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>Batch_size * #gpu</td>
    <td>8*8</td>
    <td>8*8</td>
    <td>8*8</td>
    <td>8*8</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>2e-5</td>
    <td>2e-5</td>
    <td>2e-5</td>
    <td>2e-5</td>
  </tr>
  <tr>
    <th colspan="5">Math SFT stage</th>
  </tr>
  <tr>
    <td>Dataset</td>
    <td>Geo170K-qa</td>
    <td>MathV360K</td>
    <td>MAVIS-qa</td>
    <td>MultiMath-300K-qa +Geo170K-qa +MathV360K</td>
  </tr>
  <tr>
    <td>#Samples</td>
    <td>117K</td>
    <td>339K</td>
    <td>555K</td>
    <td>1.0M</td>
  </tr>
  <tr>
    <td>Training Module</td>
    <td>mm adapter+llm</td>
    <td>mm adapter+llm</td>
    <td>mm adapter+llm</td>
    <td>mm adapter+llm</td>
  </tr>
  <tr>
    <td>Epoch</td>
    <td>2</td>
    <td>2</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Batch_size * #gpu</td>
    <td>8*8</td>
    <td>8*8</td>
    <td>8*8</td>
    <td>8*8</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>2e-5</td>
    <td>2e-5</td>
    <td>2e-5</td>
    <td>2e-5</td>
  </tr>
</table>

Table 11: Training detail information across different models.

<table>
  <tr>
    <th rowspan="2">Image Encoder</th>
    <th colspan="10">existence count position color posters celebrity scene landmark artwork OCR CR NC TT CR</th>
  </tr>
  <tr>
    <td>clip-B</td>
    <td>185.0</td>
    <td>141.7</td>
    <td>141.7</td>
    <td>150.0</td>
    <td>120.7</td>
    <td>105.0</td>
    <td><b>163.3</b></td>
    <td>138.5</td>
    <td>102.5</td>
    <td>117.5</td>
    <td>108.6</td>
    <td>62.5</td>
    <td>50.0</td>
    <td>52.5</td>
  </tr>
  <tr>
    <td>clip-L</td>
    <td>195.0</td>
    <td>156.7</td>
    <td>131.7</td>
    <td>175.0</td>
    <td>136.7</td>
    <td>127.6</td>
    <td>160.5</td>
    <td>147.0</td>
    <td>118.3</td>
    <td>132.5</td>
    <td><b>112.1</b></td>
    <td>47.5</td>
    <td>50.0</td>
    <td>60.0</td>
  </tr>
  <tr>
    <td colspan="15">(a) combine features by concatenating the hidden feature</td>
  </tr>
  <tr>
    <td>clip-B+ siglip-B</td>
    <td><b>200.0</b></td>
    <td>151.7</td>
    <td>140.0</td>
    <td>160.0</td>
    <td>114.6</td>
    <td>113.2</td>
    <td>158.5</td>
    <td>140.3</td>
    <td>111.8</td>
    <td>107.5</td>
    <td>110.0</td>
    <td>47.5</td>
    <td>50.0</td>
    <td>50.0</td>
  </tr>
  <tr>
    <td>clip-L+dinov2-L</td>
    <td>195.0</td>
    <td>148.3</td>
    <td><b>143.3</b></td>
    <td><b>195.0</b></td>
    <td>131.3</td>
    <td>122.6</td>
    <td>158.0</td>
    <td>147.3</td>
    <td>119.3</td>
    <td>140.5</td>
    <td>105.7</td>
    <td>47.5</td>
    <td>50.0</td>
    <td>50.0</td>
  </tr>
  <tr>
    <td>clip-B+siglip-B, dino-B</td>
    <td>195.0</td>
    <td>151.7</td>
    <td>136.7</td>
    <td>175.0</td>
    <td>107.8</td>
    <td>108.8</td>
    <td>158.3</td>
    <td>140.5</td>
    <td>105.3</td>
    <td>125.0</td>
    <td>110.0</td>
    <td>65.0</td>
    <td>50.0</td>
    <td>55.0</td>
  </tr>
  <tr>
    <td colspan="15">(b) combine features by increasing image token number</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L</td>
    <td>190.0</td>
    <td>145.0</td>
    <td>120.0</td>
    <td>185.0</td>
    <td>117.3</td>
    <td>120.6</td>
    <td>159.3</td>
    <td>158.0</td>
    <td>117.5</td>
    <td>132.5</td>
    <td>101.4</td>
    <td>42.5</td>
    <td><b>55.0</b></td>
    <td><b>67.5</b></td>
  </tr>
  <tr>
    <td>clip-L+dinov2-L</td>
    <td><b>200.0</b></td>
    <td>158.3</td>
    <td>141.7</td>
    <td>180.0</td>
    <td>136.1</td>
    <td>117.6</td>
    <td>160.5</td>
    <td>155.3</td>
    <td>122.8</td>
    <td>122.5</td>
    <td>107.1</td>
    <td>57.5</td>
    <td>50.0</td>
    <td>55.0</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L, dinov2-L</td>
    <td>185.0</td>
    <td>160.0</td>
    <td>130.0</td>
    <td>178.3</td>
    <td>174.7</td>
    <td>137.1</td>
    <td>161.3</td>
    <td>165.3</td>
    <td>122.8</td>
    <td>147.5</td>
    <td>109.3</td>
    <td><b>80.0</b></td>
    <td>50.0</td>
    <td>62.5</td>
  </tr>
</table>

Table 12: Detailed results on the MME dataset (Supplementary to Table 6). In the table, CR, NC, TT, and CR correspond to commonsense reasoning, numerical calculation, text translation, and code reasoning, respectively.

<table>
  <tr>
    <th rowspan="2">Image Encoder</th>
    <th colspan="5">MathVista</th>
    <th colspan="3">MathVerse</th>
    <th colspan="3">HC-M3D</th>
  </tr>
  <tr>
    <th>TextQA</th>
    <th>VQA</th>
    <th>geometry</th>
    <th>Math Word</th>
    <th>FigureQA</th>
    <th>Plane Geometry</th>
    <th>Functions</th>
    <th>Solid Geometry</th>
    <th>DI</th>
    <th>BC</th>
    <th>AG</th>
  </tr>
  <tr>
    <td>clip-B</td>
    <td>33.5</td>
    <td>27.4</td>
    <td>62.0</td>
    <td>17.2</td>
    <td>19.0</td>
    <td>29.7</td>
    <td>17.1</td>
    <td>23.0</td>
    <td>39.6</td>
    <td>8.6</td>
    <td>62.0</td>
  </tr>
  <tr>
    <td>clip-L</td>
    <td><b>38.6</b></td>
    <td><b>35.2</b></td>
    <td><b>62.0</b></td>
    <td><b>21.0</b></td>
    <td><b>21.2</b></td>
    <td><b>31.4</b></td>
    <td><b>19.1</b></td>
    <td><b>24.7</b></td>
    <td><b>40.8</b></td>
    <td><b>14.5</b></td>
    <td><b>53.4</b></td>
  </tr>
  <tr>
    <td colspan="13">(a) combine features by concatenating the hidden feature</td>
  </tr>
  <tr>
    <td>clip-B+ siglip-B</td>
    <td>37.3</td>
    <td>24.0</td>
    <td><b>64.4</b></td>
    <td>12.9</td>
    <td>19.0</td>
    <td>27.8</td>
    <td>18.1</td>
    <td>22.2</td>
    <td>37.5</td>
    <td>11.0</td>
    <td>56.6</td>
  </tr>
  <tr>
    <td>clip-L+dinov2-L</td>
    <td>36.1</td>
    <td>25.7</td>
    <td>59.1</td>
    <td>11.3</td>
    <td>19.7</td>
    <td>29.8</td>
    <td>18.1</td>
    <td>23.2</td>
    <td>38.8</td>
    <td>12.1</td>
    <td>60.8</td>
  </tr>
  <tr>
    <td>clip-B+siglip-B, dino-B</td>
    <td>38.6</td>
    <td>26.8</td>
    <td>60.6</td>
    <td>12.4</td>
    <td>19.3</td>
    <td>28.8</td>
    <td>17.5</td>
    <td>22.5</td>
    <td>38.3</td>
    <td>9.8</td>
    <td>58.3</td>
  </tr>
  <tr>
    <td colspan="13">(b) combine features by increasing image token number</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L</td>
    <td>31.7</td>
    <td>31.8</td>
    <td>54.8</td>
    <td>14.0</td>
    <td><b>22.7</b></td>
    <td>29.4</td>
    <td>16.9</td>
    <td>22.6</td>
    <td>37.7</td>
    <td>11.0</td>
    <td>59.4</td>
  </tr>
  <tr>
    <td>clip-L+dinov2-L</td>
    <td>32.3</td>
    <td>24.0</td>
    <td>53.9</td>
    <td>15.1</td>
    <td>19.7</td>
    <td>29.4</td>
    <td>15.7</td>
    <td>22.3</td>
    <td>37.3</td>
    <td>9.8</td>
    <td>55.0</td>
  </tr>
  <tr>
    <td>clip-L+siglip-L, dinov2-L</td>
    <td>29.1</td>
    <td>24.6</td>
    <td>58.7</td>
    <td>11.8</td>
    <td>19.0</td>
    <td>29.4</td>
    <td>16.9</td>
    <td>22.6</td>
    <td>36.8</td>
    <td>9.6</td>
    <td>57.1</td>
  </tr>
</table>

Table 13: Detailed results on MathVista, MathVerse, and HC-M3D (Supplement to Table 6). Except for the AG (agreement) metric (lower is better), all other metrics indicate higher is better.
Instruction for Step 1: Determining image dependency

Instruction: Assess whether solving the following problem necessarily requires images.

Examples:
如图是一株美丽的勾股树，其中所有四边形都是正方形，所有的三角形都是直角三角形，若正方形A、B的面积分别为5、3，则最大正方形C的面积是（）

△ABC的两内角平分线OB、OC相交于点O，若∠A=110°，则∠BOC =（）

Needs Image

△ABC的两内角平分线OB、OC相交于点O，若∠A=110°，则∠BOC =（）

No image is needed, a picture can be drawn from the question. Let's rewrite!

Rewrite Requirements: By reducing textual information and necessitating the extraction of problem-solving information from images, or by concurrently revising both images and text.

△ABC的两内角平分线OB、OC相交于点O，若∠A=110°，则∠BOC =（）

Text is redundant, information can be obtained from images.

Figure 5: Instruction for step 1: determining image dependency.

Instruction for Step 2: Construct highly relevant image-text data

Instruction: Try to alter image to ensure data with different answers for different image. If changing only the image falls short, try modifying both the image and options simultaneously.

Examples:

Find \( m\angle H \)

Choices:
A: 97 ✔️
B: 102
C: 107
D: 122

Before

Find \( m\angle H \)

Choices:
A: 97
B: 102
C: 107 ✔️
D: 122

After

By revising the image, the correct answer is altered

Figure 6: Instruction for step 2: construct highly relevant image-text data.

Reviewer paper summaries: This paper analyzes current multimodal mathematics benchmarks to show their weak visual reliance. The control cases are built by either shuffling or removing the image context. In response, a new math benchmark is proposed to ensure visual dependency with manual modification of existing benchmarks. An architecture design choice study on how to improve math VLMs follows, which state that combining different visual encoders does not help math tasks unlike general VQA tasks. Finally, a low-level hallucination study reveals that more visual encoders enhance hallucination in math tasks only.

This paper emohasis the role of visual modality in multimodal mathmatical reasoning.
The authors first observe that data utilized in mathematical LVLMs fails to enable the model to establish a strong correlation between images and text. And the current benchmarks in this domain also fail to measure LVLMs' ability to capture the visual information.
Then the author propose HC-M3D benchmark. With HC-M3D, consisting the same question but different images. They find that LVLMs fail to accurately identify these differences, and in more than half of the cases, they stick to their original predictions.
Finally, the authors attempt to seek the root cause of such problem, and find that using different visual encoders does not contribute to math reasoning performance.

This paper focuses on benchmarking the role of visual modality in mathematical reasoning for MLLMs. The authors train some existing MLLMs under  three settings: using aligned image and question pair, using random images and questions, training without images. The results show that there’s no big difference in the results, suggesting MLLMs don’t rely on images to solve the problems. Furthermore, the authors construct a new dataset of mathemtical problems consisting of questions with strong relevance between texts and images. They further run experiments on the new benchmark and analyze the results.
Summary of strengths: - A detailed analysis in highlighting the "lack of multimodality" in multimodal math benchmarks, enabling clearer understanding of the multimodal math capacity of VLMs.
- A hand-crafted dataset necessitating visual context to help develop future multimodal math VLMs.
- I found the hallucination study interesting, as it re-affirms common sense that adding self-supervised visual encoder enhances low-level feature understanding.

1. The research problem is important: visual modality is very important in mathmatical reasoning. It is the foundation of many human's mathmatical reasoning ability. 
2. The findings are interesting and insightful. This paper reveals that although many recent mathmatical LVLMs are trained on math visual problems, their ability to capture visual information is very poor.
3. I like the design of Table 1's experiment, which clearly shows that current LVLMs struggle to learn multimodal math datasets' visual-textual reliance, compaing with natrual image instruction tuning.
3. The writing is general clear and easy to follow.

1. The analysis of existing math MLLMs shows insights of the role of visual modality. The conclusion that MLLMs rely very little on images to solve math problems is surprising. 

2. A new testing dataset is proposed and a new benchmark is established based on this dataset. The authors alter some images of some questions while keeping the texts the same. This is an important benchmark to test the capability of MLLMs in leveraging visual cures. 

3. This papers has thorough experimental results as well as analysis on the results to provide more insights. it also investigates the impac tof different visual encoders, which are important for future study.
Summary of weaknesses: - I feel there's a missing experiment here: the visual control experiment of Table 1 should also be conducted using the proposed HC-M3D benchmark to ensure that it truly necessitates the visual context unlike existing benchmarks.
- It seems that the weak correlation between the reasoning performance and hallucination bit counterintuitive to me. The paper will benefit from further analysis on this account.

1. Although Mathverse[1] is mentioned and used in the paper, I think the design scope of Mathverse is similar with HC-M3D, since it also emphasis the image-reliance of multimodal math reasoning. It could be benificial to further explain the difference between HC-M3D and mathverse and the advantage of HC-M3D.
2. The attempts of exploring different visual encoders for enhancing model's visual dependency is hindered by the training dataset that the authors use. As is shown in Table 1, the model learn almost zero visual textual reliance on G-LLaVA's training dataset. I suggest the author could try using multimodal mathmatical dataset with high visual-textual reliance. For example, Euclid[2] uses dataset generation engine to produce large scale geometry image-text pairs and observes significant difference between different visual encoders' performance.

[1] Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., ... & Li, H. (2025). Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?. In European Conference on Computer Vision (pp. 169-186). Springer, Cham.

[2] Zhang, J., Liu, O., Yu, T., Hu, J., & Neiswanger, W. (2024). Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions. arXiv preprint arXiv:2412.08737.

1. After thorough analysis, can the authors give some suggestions for future research in mathmatical reasoning using MLLMs based on the findings of this paper? 

2. I suggest the authors add a teaser image in the introduction to show some failure cases (low BC rate of existing MLLMs). This will help readers to learn the key points more easiily.
Comments, suggestions and typos: - When combining multiple visual encoders, can you also try interleaving the tokens from different encoders as done in [1]?

[1] https://arxiv.org/abs/2401.06209

Please see above

-------Comments after discussion-------
I have read the author's response detailedly and I'm generally stisfied with the rebuttal. 
However, my second question is not fully addressed -- although Math-LLaVA's dataset shows slightly higher visual-textual reliance, 3% gap is still not considered as a strong reliance (e.g. comparing with LLaVA training dataset). Therefore, we can not draw convincing conclusion with such dataset. I understand it is challenging to develop another general geometry dataset with high visual-textual reliance. At very least, the author should mention this as a weakness and include it as a future work.

Please see the weakness section.
Overall assessment scores: 4.5

4

4