MathGLM-Vision: Solving Mathematical Problems with Multi-Modal Large Language Model

## Abstract

Abstract Large language models (LLMs) have demonstrated significant capabilities in mathematical reasoning, particularly with text-based mathematical problems. However, current multi-modal large language models (MLLMs), especially those specialized in mathematics, tend to focus predominantly on solving geometric problems but ignore the diversity of visual information available in other areas of mathematics. Moreover, the geometric information for these specialized mathematical MLLMs is derived from several public datasets, which are typically limited in diversity and complexity. To address these limitations, we aim to construct a fine-tuning dataset named MathVL, and develop a series of specialized mathematical MLLMs termed MathGLM-Vision by conducting Supervised Fine-Tuning (SFT) on MathVL with various parameter-scale backbones. To extensively evaluate the effectiveness of MathGLM-Vision, we conduct experiments on several public benchmarks and our curated MathVL-test consisting of 2,000 problems. Experimental results demonstrate that MathGLM-Vision achieves significant improvements compared with some existing models, including backbone models and open-source mathematical MLLMs. These findings indicate the importance of diversity dataset in enhancing the mathematical reasoning abilities of MLLMs.

## 1 Introduction

Recent advancements in computational linguistics have led to substantial progress in solving mathematical problems using Large Language Models (LLMs) with multi-step reasoning processes []. For example, models like GPT-4 [], Qwen [], GLM-4 [], LLaMA [, ] have demonstrated impressive performance on mathematical datasets such as GSM8K [] and MATH []. Furthermore, the development of specialized mathematical models is expanding the potential of LLMs in this domain. These models, specifically designed for mathematical problem solving, include notable contributions such as WizardMath [], MAmmoTH [], MathCoder [], MetaMath [], DeepSeekMath [], and others [, , , , , ]. These advancements highlight the growing proficiency of LLMs in handling intricate mathematical reasoning and problem-solving tasks.

Despite significant advancements, the majority of models designed for mathematical problem solving still rely predominately on textual representations. This limits their effectiveness in scenarios that require visual information. Notably, approximately 63% of mathematics questions in Chinese K12 education include visual elements, highlighting the critical role of visual information in comprehending and solving mathematical problems.

Figure: Figure 1: Insight experiments demonstrates the significance of visual information in solving mathematical problems. (Left) A performance comparison of different models with and without visual inputs. (Right) The accuracies of MathGLM-Vision on MathVL-test with and without visual inputs.
Refer to caption: x1.png

Therefore, a crucial question arises: Is visual information essential for solving these mathematical problems that include visual elements? To verify this, we conduct a series of insightful experiments comparing the performance of these models such as GPT-4o, Claude-3.5-Sonnet, Qwen-VL-Max, and Gemini-1.5-Pro on MathVL-test, both with and without visual inputs. As shown in Figure 1, the results clearly demonstrate that the inclusion of visual elements significantly enhances the models’ ability to accurately solve complex mathematical problems. Conversely, the exclusion of visual information leads to a pronounced decrease in performance, emphasizing the essential role that visual context plays in solving mathematical problems that incorporate visual elements.

Currently, multi-modal large language models (MLLMs) are at the forefront of efforts to integrate visual and textual information for solving mathematical problems. Close-source models such as GPT-4V [], Gemini [], Claude3 [], Qwen-VL [], along with several open-source MLLMs like CogVLM [], MiniGPT [], LLaVA-1.5 [], SPHINX-MoE [], and LLaVA-NeXT [], demonstrate substantial potential in addressing geometric reasoning challenges. Additionally, specialized geometric MLLMs like G-LLaVA [], GeoGPT4V [] and Math-LLaVA [] are particularly focused on enhancing capabilities in this domain. However, these models still face several challenges and limitations that need to be addressed.

- •
Current MLLMs, particularly those specialized in mathematics, predominantly focus on solving geometric problems and tend to overlook the diversity of visual information in mathematics. This visual information encompasses a broad spectrum of elements, including arithmetic, statistics, algebra and word problems, each integral to different mathematical domains beyond geometry.
- •
Current fine-tuning dataset for specialized mathematical MLLMs, typically sourced from public datasets like GeoQA and Geometry3K, often lack diversity and complexity. This limitation restricts the models’ ability to effectively solve a broader range of mathematical problems.
- •
Current specialized mathematical MLLMs are predominantly designed to process single-image inputs and lack the capability to handle multiple images simultaneously. This limitation hampers their ability to tackle complex problems that necessitate the integration of information from multiple visual sources.

In response to these challenges and limitations, we introduce MathGLM-Vision, a promising specialized mathematical multi-modal large language model designed to seamlessly integrate visual information with textual analysis. MathGLM-Vision is designed to enhance the model’s ability to interpret and solve complex mathematical problems involving visual elements, thereby expanding the range of problems that can be addressed effectively. As demonstrated in Figure 1, we compare the performance of MathGLM-Vision on MathVL-test with and without visual inputs. The results clearly indicate that MathGLM-Vision incorporating visual inputs, significantly outperforms its text-only inputs that ignores visual information. This enhanced capability highlights the importance of integrating multi-modal inputs in advancing the performance of specialized mathematical models.

Figure: Figure 2: Performance comparison of the different multi-modal large language models. (Left) The accuracies of MathGLM-Vision and other MLLMs among three evaluation datasets. (Right) The accuracy of MathGLM-Vision and other MLLMs on MathVL-test across different categories.
Refer to caption: x3.png

In this paper, we construct a fine-tuning dataset named MathVL, which encompasses both open-source data and our specially curated Chinese data collected from K12 education. The MathVL dataset is meticulously designed to incorporate a diverse range of mathematical problems, consisting of textual and visual inputs. For textual information, the MathVL dataset covers a variety of mathematical subjects such as arithmetic, algebra, geometry, statistics, and word problems. It includes various types of questions, including fill-in-the-blank, multiple-choice, and free-form. For visual information, the MathVL dataset involves elements like functions, statistical data, graphs, charts, LaTeX expressions, and geometric figures, providing a comprehensive resource for complex mathematical problem solving.

With our constructed MathVL dataset, we develop a series of specialized mathematical MLLMs, collectively referred to as MathGLM-Vision, with different parameter scales. Specifically, MathGLM-Vision-9B, MathGLM-Vision-19B and MathGLM-Vision-32B are fine-tuned on three backbone models: GLM-4V-9B, CogVLM2, and CogVLM-32B, respectively. Moreover, we establish a benchmark dataset named MathVL-test, which contains 2,000 problems designed to evaluate the ability of MathGLM-Vision and other MLLMs in solving mathematical problems involving visual information. Through extensive evaluation experiments on four public benchmark datasets and one curated MathVL-test, we validate the effectiveness of our MathGLM-Vision. The results in Figure 2 demonstrate that MathGLM-Vision exhibits superior performance in understanding and solving complex mathematical problems with visual elements compared to existing MLLMs. For instance, on the geometry problem solving (GPS) minitest split of MathVista [], MathGLM-Vision-9B achieves a 39.68% relative improvement for GLM-4V-9B, MathGLM-Vision-19B achieves a 65.06% relative improvement for CogVLM2, and MathGLM-Vision-32B achieves a 51.05% relative improvement over CogVLM-32B.

## 2 MathVL Dataset

To enhance the capabilities of MLLMs in solving mathematical problems, previous efforts [, , , ] focus on constructing high-quality datasets. Nevertheless, the majority of these datasets fall into the category of Visual Question Answering (VQA), which generally involves descriptive or identification tasks rather than conventional mathematical problems. Furthermore, the answers in some public datasets like Geometry3K [], GeoGPT4V [], MathV360K [] for standard mathematical questions are often too simplistic, usually providing only the final answer without the intermediate steps necessary for a thorough understanding. It is well-established that including step-by-step solutions can significantly enhance the reasoning capabilities of large language models [, , , ]. Figure 3 demonstrates the distribution of answer lengths in current open-source mathematical datasets.

Figure: Figure 3: Analysis of answer lengths in several open-source mathematical datasets like MathV360K, GeoGPT4V, and Geometry3K.
Refer to caption: x5.png

To address these issues, we construct a fine-tuning dataset MathVL, including both several public datasets and our curated Chinese dataset collected from K12 education levels. This dataset is meticulously crafted to encompass a diverse array of mathematical problems that incorporate visual information. Each problem is presented with detailed step-by-step solutions, aiming to enhance the problem-solving skills of MLLMs by providing them with both the context and the procedural knowledge necessary for effective reasoning and comprehension.

Open-Source Data. We first collect open-source datasets from GeoQA+ [], Geometry3K [], ChartQA [], and UniGEO-Calculation []. These datasets commonly serve as seed data for constructing enhanced datasets. Through observation and statistical analysis, we discover that 57% of the answers within these datasets are comprised of fewer than 50 words, indicating that many questions are answered directly without elaboration or explanation. To enrich these dataset with comprehensive step-by-step solutions, we employ GPT-4o to generate the detailed solutions for each question, thereby enhancing the learning and reasoning potential of these datasets. After generating the detailed answers, we perform a rigorous judgement process to ensure the accuracy of the solutions provided by GPT-4o. Additionally, we adopt a public instruction tuning dataset named Geo170K [], which is constructed using GeoQA+ and Geometry3K as seed data and contains more than 110K geometric question-answer pairs. We also incorporate another public dataset, GeomVerse [], as part of our resources. In the end, the detailed statistics of the open-source datasets used in MathGLM-Vision is provided in Table 1.

**Table 1: The detailed statistics of the open-source datasets used in MathGLM-Vision.**
| Datasets | ChartQA | UniGeo-Calculation | Geometry3K | GeoQA+ | Geo170K | GeomVerse | ALL |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Samples | 7,398 | 3,499 | 2,101 | 6,026 | 117,205 | 9,339 | 145,568 |

Chinese Data Collected from K12 Education. We construct a dataset specifically focused on K12 education, comprising 341,346 mathematical problems with textual and visual inputs. This dataset is meticulously curated to encompass a board range of mathematical topics and difficulty levels tailored to the Chinese educational curriculum. It features various question types such as multiple-choice, fill-in-the-blank, and free-form questions across disciplines including arithmetic, algebra, geometry, statistics, and word problems. Mathematically, this dataset can be represented as $D_{MathVL}^{zh}=\{Q,A,I_{s}\}$, where $Q$ represents the question, $A$ represents the answer, and $I_{s}$ represents one or more images for each corresponding question. To build this dataset, we first process the images by adding a white border around each image and enhancing their resolution to ensure that MLLMs can effectively recognize and interpret these images. This modification is crucial for facilitating the accurate extraction of visual information. Next, we extract 341,346 samples from a raw dataset containing 685,670 samples by implementing a selective filtering process. This selection is based on two specific criteria: (1) filtering out samples where the answer includes images or the question is incomplete, and (2) eliminating samples with answer that are fewer than 50 words in length to ensure the responses are sufficiently detailed for model training. After constructing this dataset, we categorize and analyze it based on mathematical topics associated with each question. Detailed statistics about the distribution of these categories are presented in Table 2. Figure 4 demonstrates some examples sampled from the constructed Chinese dataset, providing a visual representation of the mathematical topics of questions included. More dataset cases are provided in Appendix A.

**Table 2: Detailed statistics regarding the distribution used in MathGLM-Vision.**
| Types | Arithmetic | Geometry | Algebra | Statistics | Word Problems | ALL |
| --- | --- | --- | --- | --- | --- | --- |
| Samples | 7,207 | 291,879 | 20,111 | 18,284 | 3,865 | 341,346 |

Figure: Figure 4: Examples sampled from the constructed Chinese dataset.
Refer to caption: x6.png

## 3 MathGLM-Vision

Model Architecture. We employ CogVLM [] and GLM-4V-9B [] architectures as our backbone models, and conduct Supervised Fine-Tuning (SFT) on our constructed MathVL dataset. Specifically, we utilize three pre-trained multi-modal large language models (MLLMs) for the fine-tuning process: GLM-4V-9B, CogVLM2-19B, and CogVLM-32B. This results in the development of three distinct variants of MathGLM-Vision, designated as MathGLM-Vision-9B, MathGLM-Vision-19B, and MathGLM-Vision-32B, respectively. Table 3 demonstrates an overview of the MathGLM-Vision series, detailing the different model parameters and configurations. Further details about the abovementioned three pre-trained MLLMs are available in Appendix B.

**Table 3: An overview of MathGLM-Vision series along with model parameters and configurations.**
| Model | LLM Size | ToTal Size | Language Model | Image Encoder |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Layers | Hidden Size | Heads | Layers | Hidden Size | Heads |  |  |  |
| MathGLM-Vision-9B | 9B | 20B | 40 | 4096 | 32 | 63 | 1792 | 16 |
| MathGLM-Vision-19B | 8B | 19B | 32 | 4096 | 32 | 63 | 1792 | 16 |
| MathGLM-Vision-32B | 32B | 43B | 58 | 6656 | 52 | 63 | 1792 | 16 |

Model Training. To maintain the general vision-language understanding skills of MathGLM-Vision, we incorporate 19 open-source visual question-answering datasets (VQA datasets) into the MathVL dataset. More details about the task type and visual context of VQA datasets are provided in Appendix C. These datasets are meticulously selected to challenge and enhance the model’s ability to interpret and integrate visual and textual information, ensuring it retains a broad understanding across various contexts. By merging these varied sources, we enhance MathGLM-Vision’s specialized capabilities for mathematical problem-solving and simultaneously preserve its robustness in general vision-language tasks. In the end, we conduct supervised fine-tuning (SFT) across the combined VQA and MathVL datasets. The training process undergoes 35,000 iterations with a learning rate of 1e-5 and a batch size of 128. To ensure the stability of the training, we activate the visual encoder’s parameters and adjust its learning rate to be one-tenth of that used for the remaining training parameters. The details of the SFT procedures are described in Appendix D.

## 4 Experiments

### 4.1 Experimental Setup

Evaluation Datasets. We assess our MathGLM-Vision using three well-established public benchmark datasets(MathVista [], MathVerse [], and Math-Vision [] datasets) alongside our specially curated dataset MathVL-test. The MathVL-test dataset comprises 2,000 sampled cases, distinct from those in the MathVL dataset, ensuring a rigorous and unbiased evaluation of MathGLM-Vision’s capabilities. Besides, we adopt the testing protocol [] and utilize MathVista-GPS dataset to evaluate MathGLM-Vision’s ability to solve geometry problems. Additionally, we evaluate MathGLM-Vision’s general vision-language understanding skills using the MMMU benchmark []. Detailed descriptions for these benchmark datasets are provided in Appendix E.

Compared Models. We compare MathGLM-Vision with other Multi-Modal Large Language Models (MLLMs), including closed-source MLLMs such as Gemini [], GPT-4V [], Claude3 [], and Qwen-VL [], and open-source MLLMs like mPLUG-Owl [], LLaMA-Adapter-V2 [], InstrctBLIP [], LLaVA-1.5 [], ShareGPT4V [], SPHINX [], InternLM-XC2 [], and InternVL []. Additionally, we compare MathGLM-Vision with recent specialized mathematical MLLMs, including G-LLaVA [], LLaVA-1.5-G [], ShareGPT4V-G [], and Math-LLaVA [].

Evaluation Metrics. We adopt top-1 accuracy to evaluate our MathGLM-Vision across MathVista-GPS, MathVista, MathVerse, MathVision, and MathVL-test. Our evaluation process follows the pipeline outlined in the aforementioned benchmark datasets, which involves using LLMs to extract predicted answers from the model’s responses. Accuracy is then calculated by comparing these extracted answers against the ground truths.

### 4.2 Main Results

Results on public benchmark datasets. To comprehensively assess the ability of MathGLM-Vision in solving mathematical problems, we evaluate its performance against other MLLMs across several public benchmark datasets, including MathVista-GPS [], the testmini subset of MathVista [], MathVerse [], and Math-Vision []. Table 4 demonstrates the overall results from these evaluations. The experimental results indicate that our constructed MathVL dataset can significantly improve MathGLM-Vision’s mathematical reasoning capabilities. For example, MathGLM-Vision-9B achieves a 64.42% accuracy on the MathVista-GPS dataset, marking a substantial 39.68% improvement over its backbone model, GLM-4V-9B. Besides, across various parameter scales, MathGLM-Vision consistently surpass all backbone models on different evaluation benchmarks, highlighting the significant enhancements that MathVL brings to the MathGLM-Vision’s problem-solving skills. Notably, MathGLM-Vision outperforms all open-source specialized mathematical MLLMs across various benchmarks. The superior performance suggests that the high-quality and diverse data, complete with detailed step-by-step solutions, are crucial for improving MLLM’s mathematical reasoning capabilities. More importantly, MathGLM-Vision-32B outperforms even the advanced GPT-4V on the more challenging Math-Vision benchmark, demonstrating its superior capacity to tackle complex mathematical problems. Detailed experimental results on public benchmark datasets across different task types can be found in Appendix F.

**Table 4: Results on several public benchmark datasets. Comparison of model performance on the testmini set of MathVista and geometry problem solving (GPS) of MathVista. For MathVerse dataset, results are evaluated on Vision Dominant with CoT-E. For Math-Vision dataset, all 3,040 samples included in the data are evaluated. The highest accuracy among all baseline MLLMs is marked in red, while the highest accuracy among various variants of MathGLM-Vision is marked bold.**
| Model | Input | LLM | MathVista (GPS) | MathVista | MathVerse | MathVision |
| --- | --- | --- | --- | --- | --- | --- |
| Closed Source Models |  |  |  |  |  |  |
| Gemini Pro | Q, I | - | 40.40 | 45.20 | 36.80 | 17.66 |
| Gemini-1.5-Pro | Q, I | - | 53.85 | 63.90 | 51.08 | 19.24 |
| GPT-4V | Q, I | - | 50.50 | 49.90 | 50.80 | 22.76 |
| GPT-4-turbo | Q, I | - | 58.25 | 58.10 | 43.50 | 30.26 |
| GPT-4o | Q, I | - | 64.71 | 63.80 | 56.65 | 30.39 |
| Claude3-Opus | Q, I | - | 52.91 | 50.50 | 31.77 | 27.13 |
| Claude3.5-Sonnet | Q, I | - | 64.42 | 67.70 | 48.98 | 37.99 |
| Qwen-VL-Plus | Q, I | - | 33.01 | 43.30 | 19.10 | 10.72 |
| Qwen-VL-Max | Q, I | - | 46.12 | 51.00 | 35.90 | 15.59 |
| Open Source Models |  |  |  |  |  |  |
| General Multi-modal LLMs |  |  |  |  |  |  |
| mPLUG-Owl | Q, I | LLaMA-7B | 23.60 | 22.20 | 12.47 | 9.84 |
| LLaMA-Adapter-V2 | Q, I | LLaMA-7B | 25.50 | 23.90 | 4.50 | 9.44 |
| InstructBLIP | Q, I | Vicuna-7B | 20.70 | 25.30 | 15.36 | 10.12 |
| LLaVA-1.5 | Q, I | Vicuna-13B | 24.04 | 27.60 | 12.70 | 11.12 |
| ShareGPT4V | Q, I | Vicuna-13B | 38.35 | 29.30 | 16.20 | 11.88 |
| SPHINX-MoE | Q, I | Mixtral 8*7B | 31.20 | 42.30 | 19.60 | 14.18 |
| SPHINX-Plus | Q, I | LLaMA2-13B | 16.40 | 36.70 | 14.70 | 9.70 |
| InternLM-XC2 | Q, I | InternLM2-7B | 63.00 | 57.60 | 24.40 | 14.54 |
| InternVL-1.2-Plus | Q, I | Nous-Hermes-2-Yi-34B | 61.10 | 59.90 | 21.70 | 16.97 |
| Geo-Multi-modal LLMs |  |  |  |  |  |  |
| G-LLaVA | Q, I | LLaMA2-7B | 53.40 | 28.46 | 12.70 | 12.07 |
| G-LLaVA | Q, I | LLaMA2-13B | 56.70 | 35.84 | 14.59 | 13.27 |
| LLaVA-1.5-G | Q, I | Vicuna-7B | 32.69 | 45.22 | 13.96 | 14.13 |
| LLaVA-1.5-G | Q, I | Vicuna-13B | 36.54 | 48.34 | 15.61 | 14.88 |
| ShareGPT4V-G | Q, I | Vicuna-7B | 32.69 | 45.07 | 16.24 | 12.86 |
| ShareGPT4V-G | Q, I | Vicuna-13B | 43.27 | 49.14 | 16.37 | 14.45 |
| Math-LLaVA | Q, I | Vicuna-13B | 57.70 | 46.60 | 19.04 | 15.69 |
| MathGLM-Vision and Backbone Models |  |  |  |  |  |  |
| GLM-4V-9B | Q, I | GLM-4-9B | 46.12 | 46.70 | 35.66 | 15.31 |
| MathGLM-Vision-9B | Q, I | GLM-4-9B | 64.42 | 52.20 | 44.20 | 19.18 |
| CogVLM2 | Q, I | LLaMA-3-8B | 39.61 | 40.85 | 25.76 | 13.20 |
| MathGLM-Vision-19B | Q, I | LLaMA-3-8B | 65.38 | 61.10 | 42.50 | 21.64 |
| CogVLM-32B | Q, I | GLM2-32B | 41.06 | 40.04 | 35.28 | 19.32 |
| MathGLM-Vision-32B | Q, I | GLM2-32B | 62.02 | 62.40 | 49.20 | 26.51 |

Results on MathVL-test. We also evaluate MathGLM-Vision and several close-source MLLMs using our specially constructed MathVL-test dataset. Given that MathVL-test comprises multiple images for each question, we exclusively assess the performance of these close-source MLLMs through API calls. As depicted in Table 5, the results clearly demonstrate that MathGLM-Vision significantly outperforms both its backbone models and other leading closed-source MLLMs across various model sizes. Specifically, our MathGLM-Vision-32B outperforms the advanced GPT-4o with a significant margin, achieving an accuracy of 59.00% compared to GPT-4o’s 51.05%. Compared to the backbone model, GLM-4V-9B, MathGLM-Vision-9B achieves an impressive accuracy of 57.05% with a significant improvement of 86.5%. This superior performance suggests that MathGLM-Vision, when conducting SFT on the MathVL dataset, notably enhances its capability to tackle complex Chinese mathematical problems. Additionally, we report the accuracy across various categories, as illustrated in Figure 2 (See Right). MathGLM-Vision significantly outperforms other advanced MLLMs in the domains of geometry and statistics. In contrast, Claude3.5-Sonnet excels in algebra and arithmetic, demonstrating superior performance. Meanwhile, MathGLM-Vision-19B ranks second in performance in the domain of arithmetic, showing its strong abilities in this area as well. GPT-4o exhibits the highest performance in word problems domain, while MathGLM-Vision also exhibits robust performance, surpassing both Gemini-1.5-Pro and Claude3.5-Sonnet in this category.

**Table 5: Results on MathVL-test. A detailed comparison of the performance of MathGLM-Vision and various other leading close-source MLLMs on the MathVL-test. The highest accuracy among all baseline MLLMs is marked in red, while the highest accuracy among various variants of MathGLM-Vision is marked bold.**
| Model | Input | LLM Size | MathVL-test |
| --- | --- | --- | --- |
| Gemini-1.5-Pro | Q, I | - | 52.03 |
| GPT-4V | Q, I | - | 35.89 |
| GPT-4-turbo | Q, I | - | 42.19 |
| GPT-4o | Q, I | - | 51.05 |
| Claude3.5-Sonet | Q, I | - | 46.84 |
| Claude3-Opus | Q, I | - | 33.77 |
| Qwen-VL-Plus | Q, I | - | 28.50 |
| Qwen-VL-Max | Q, I | - | 35.61 |
| GLM-4V-9B | Q, I | 9B | 30.59 |
| MathGLM-Vision-9B | Q, I | 9B | 57.05 |
| CogVLM2 | Q, I | 8B | 27.47 |
| MathGLM-Vision-19B | Q, I | 8B | 57.30 |
| CogVLM-32B | Q, I | 32B | 30.86 |
| MathGLM-Vision-32B | Q, I | 32B | 59.00 |

### 4.3 Generalizability of MathGLM-Vision

In addition to its proficiency in mathematical reasoning, we further assess MathGLM-Vision’s capabilities in general vision-language understanding by conducting experiments on the MMMU benchmark []. This benchmark is specifically designed to evaluate the ability of models to comprehend and process information across a variety of academic and professional disciplines, providing a comprehensive test of general vision-language understanding. Table 6 shows the performance of MathGLM-Vision, a specific variant fine-tuned exclusively on MathVL without the inclusion of VQA datasets, and backbone models. Compared to CogVLM2, MathGLM-Vision-19B achieves comparable performance in terms of generalizability, underscoring its capacity for simultaneous multi-modal understanding and mathematical reasoning. However, MathGLM-Vision-32B shows a slight reduction in performance across multiple categories on the MMMU benchmark. Besides, MathGLM-Vision, when fine-tuned with VQA datasets, outperforms its variant lacking VQA datasets. This indicates that omitting VQA datasets from the fine-tuning process limits the general vision-language understanding abilities. Thus, the Specialized Fine-Tuning (SFT) process using our MathVL incorporated with VQA datasets not only enhances MathGLM-Vision’s mathematical reasoning abilities but also preserves its generalizability.

**Table 6: Generalizability of MathGLM-Vision on the MMMU benchmark.**
| Model | MMMU | Art & Design | Business | Sci. | Health & Med. | Human. & Social Sci. | Tech. & Eng. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CogVLM2 | 40.2 | 58.3 | 30.0 | 26.7 | 41.3 | 38.6 | 53.3 |
| w/o VQA datasets | 38.1 | 60.8 | 28.7 | 34.0 | 36.7 | 43.3 | 32.9 |
| MathGLM-Vision-19B | 40.2 | 63.3 | 37.3 | 27.3 | 36.0 | 46.7 | 37.6 |
| CogVLM-32B | 42.9 | 63.3 | 31.3 | 32.0 | 43.3 | 62.5 | 36.7 |
| w/o VQA datasets | 38.6 | 62.5 | 26.7 | 28.0 | 34.0 | 56.7 | 33.8 |
| MathGLM-Vision-32B | 40.0 | 60.0 | 28.7 | 34.0 | 38.7 | 52.5 | 34.8 |

### 4.4 Further Analysis

Effect of Chinese Dataset. To validate the effectiveness of the adopted Chinese dataset in MathVL, we conduct an extend experiment that involves fine-tuning GLM-4V-9B with open-source datasets, deliberately excluding Chinese data collected from K12 education. The purpose of this experiment is to assess the specific contributions of the Chinese dataset to the capabilities of MathGLM-Vision. Table 8 shows a comparison of performance results. Compared to the backbone model GLM-4V-9B, a variant MathGLM-Vision-9B that undergoes Specialized Fine-Tuning (SFT) exclusively with open-source data exhibits significant improvement on the minitest of MathVista, particularly in geometry problem solving (GPS) and geometry reasoning (GEO). This indicates that fine-tuning on diverse open-source data can markedly enhance model performance in specific mathematical areas. MathGLM-Vision, incorporating both open-source data and Chinese data, outperforms the variant tuned only with open-source data on the minitest of MathVista, highlighting the significant value added by integrating the Chinese dataset in the training process. Notably, compared to the variant without Chinese data, MathGLM-Vision achieves a significantly higher accuracy on the MathVL-test. These findings confirm that the inclusion of the Chinese dataset not only enhances the model’s capability in handling complex mathematical problems but also contributes significantly to its overall performance on a diverse set of tasks within MathVista.

Effect of VQA Datasets. To explore the effect of VQA datasets on the performance of MathGLM-Vision, an extended experiment can be designed where Specialized Fine-Tuning (SFT) is applied exclusively to mathematical datasets, deliberately excluding VQA datasets. Table 8 demonstrates the performance comparison achieved by different models on MathVista. Compared to the backbone model GLM-4V-9B, a variant of MathGLM-Vision-9B achieves significant improvements on geometry problem solving (GPS) and geometry Reasoning (GEO). However, it exhibits a decline in the overall accuracy on the minitest of MathVista (ALL). The decline can be attributed to the composition of MathVista, which comprises five tasks, with question-answering types (such as graphical question-answering, textbook question-answering, and visual question-answering) comprising up to 60.6% of the tasks. Omitting VQA training in MathGLM-Vision impacts the model’s ability to effectively process and respond to these multi-modal questions. Notably, within specific subsets of MathVista, such as GPS and GEO, a variant of MathGLM-Vision-9B slightly below the standard MathGLM-Vision-9B. This observation suggests that VQA datasets are crucial for preserving overall multi-modal understanding, their impact may vary depending on different task types. Besides, VQA datasets can indirectly bolster mathematical reasoning skills, which in turn enhances image recognition capabilities.

**Table 7: Effect of the constructed Chinese data.**
| Model | MathVista | MathVL-test |  |  |
| --- | --- | --- | --- | --- |
| GPS | GEO | ALL |  |  |
| GLM-4V-9B | 46.12 | 44.35 | 46.70 | 30.59 |
| + SFT on Open-source Data | 62.98 | 61.51 | 50.40 | 47.55 |
| MathGLM-Vision-9B | 64.42 | 62.34 | 52.20 | 57.25 |

### 4.5 Error Analysis

We meticulously analyze the causes of errors in MathGLM-Vision-32B on the MathVL-test dataset and illustrate the distribution of these errors in Figure 5. We summarize these errors in MathGLM-Vision-32B into five types: reasoning error, knowledge error, vision recognition error, calculation error, and question misunderstood error. The most prevalent type of errors, accounting for 69.1% of the total, is identified as Reasoning Error. This indicates a significant challenge in the MathGLM-Vision-32B’s logical deductions and inferential reasoning. Improving these capabilities can dramatically enhance the MathGLM-Vision-32B’s overall performance. Knowledge Error, which made up 12.7% of the errors, relates to the model’s misapplicaion or lack of specific factual information. Vision Recognition Error accounts for 11.4% of the total errors and involves inaccuracies in interpreting visual data. This type of error can be reduced through the implementation of more advanced vision encoders. Furthermore, the fact that Calculation Error constitutes only 4.3% of the errors suggests that MathGLM-Vision-32B demonstrates considerable robustness in numerical and computational tasks. Lastly, Question Misunderstood Error, which constitutes 2.5% of the total, occurs when the model fails to correctly interpret question. Enhancing natural language processing capabilities and refining context understanding can significantly reduce these types of errors. Addressing these identified error types through targeted enhancements can significantly boost the overall effectiveness of MathGLM-Vision-32B. Figure 6 demonstrate some cases of the Calculation Error category. More detailed examples of these errors can be found in Appendix G.

Figure: Figure 5: Error distribution of MathGLM- Vision-32B.
Refer to caption: extracted/5833108/figs/error_distribution.png

## 5 Related Works

### 5.1 Multi-Modal Language Model

The development of Multi-Modal Language Models (MLLMs) have emerged as a significant area of research, which are designed to integrate information from multiple modalities—typically text and images—to perform tasks that require a holistic understanding of both visual and linguistic inputs. Pioneering efforts such as ViLBERT [] and LXMERT [] have advanced this field by conducting the joint pre-training on image-text datasets. They process text and image inputs separately before fusing them for final task layers, significantly improving performance on tasks like image captioning and visual question answering. The continues evolution of MLLMs has lead to innovations in data fusion techniques. Notable models such as CLIP [], ALIGN [], and BLIP [] have adopted contrastive learning paradigms to align visual and language information from billions of image-text pairs. Concurrently, the success of LLMs [, , , , , , , , , ] facilitates the integration of LLMs into multi-modal tasks by utilizing pre-training alignment and visual instruction tuning, leading to the emergence of multi-modal language models (MLLMs) [, , , , , ].
Despite MLLMs have demonstrated remarkable capabilities on tasks such as image caption and visual question answering, they stall face significant challenges in solving mathematical problems that involve visual information [, , , ].

### 5.2 Mathematical Reasoning

Recently, math-specific LLMs [, , , , , , , ] have demonstrated remarkable abilities in handling mathematical reasoning tasks that only involve textual information. These models have been specifically trained on web-scale instruction mathematical dataset or fine-tuned on specialized mathematical problem sets. For instance, WizardMath [] and MetaMath [] have implemented data augmentation methods to enhance the models’ ability to understand and solve mathematical problems by enriching the MATH [] and GSM8K [] datasets. Recent research has also focused on creating specialized MLLMs for mathematical tasks. UniGeo [] and UniMath [] have demonstrated enhanced datasets and conventional deep learning approaches for geometric problem solving. MLLMs like G-LLaVA [], GeoGPT4V [], and Math-LLaVA [] are tailored for mathematical problem solving, incorporating both geometric understanding and algebraic reasoning. Additionally, several benchmark datasets [, , , ] are proposed to evaluate the multi-modal mathematical reasoning abilities of MLLMs.

## 6 Conclusion

In this paper, we attempt to address the issues in current mathematical MLLMs. We construct a fine-tuning dataset named MathVL, upon which we conduct a Supervised Fine-Tuning (SFT) process. This initiative results in the development of a series of enhanced MLLMs, designated as MathGLM-Vision. Specially, MathGLM-Vision contains three variations: MathGLM-Vision-9B, MathGLM-Vision-19B, and MathGLM-Vision-32B, each fine-tuned on different backbone models: GLM-4-V, CogVLM2, and CogVLM-32B, respectively. These developed MathGLM-Vision significantly improve the capabilities of mathematical reasoning, achieving substantial performance improvements. Relative to their respective backbone models, MathGLM-Vision-9B, MathGLM-Vision-19B, and MathGLM-Vision-32B show improvements of 39%, 65%, and 53.7% on the Geometry Problem Solving (GPS) minitest split of MathVista, demonstrating the effectiveness of MathVL in enhancing the mathematical problem-solving abilities of MLLMs. Additionally, we evaluate the effectiveness of MathGLM-Vision on our curated MathVL-test dataset. Experimental results reveal that MathGLM-Vision not only surpass their backbone models in specialized mathematical tests but also preserve the generalizability capabilities in general vision-language understanding domains.

## Appendix A Dataset Cases

In this section, we provide a detailed overview of specific cases from our constructed MathVL dataset. These cases demonstrate the variety of mathematical disciplines covered by MathVL, including arithmetic, geometry, algebra, statistics, and word problems. Figure 7, Figure 8, Figure 9, Figure 10, and Figure 11 depict the types of problems that MathGLM-Vision is designed to tackle in each respective category. Each of these categories is critical for assessing the comprehensive mathematical capabilities of MathGLM-Vision. By tackling a wide range of problems, MathGLM-Vision demonstrates its versatility and robustness in addressing diverse mathematical tasks.

Figure: Figure 7: Cases of arithmetical problems in our MathVL dataset.
Refer to caption: x8.png

Figure: Figure 8: Cases of geometric problems in our MathVL dataset.
Refer to caption: x9.png

Figure: Figure 9: Cases of algebraic problems in our MathVL dataset.
Refer to caption: x10.png

Figure: Figure 10: Cases of statistical problems in our MathVL dataset.
Refer to caption: x11.png

Figure: Figure 11: Cases of word problems in our MathVL dataset.
Refer to caption: x12.png

## Appendix B Backbone Models

We utilize the following multi-modal large language models as our backbone models for conducting Specialized Fine-Tuning (SFT) on the constructed MathVL. The detailed description of each backbone model can be represented as follows:

- •
GLM-4V-9B is a bilingual (Chinese and English) multi-modal large language model, developed collaboratively by Zhipu.AI and Tsinghua University. It is built upon the foundational architecture of GLM-4-9B, enhancing its capabilities to handle complex multi-modal interactions. GLM-4V-9B takes a high resolution of 1120 * 1120 images as visual inputs. In comprehensive evaluations that test various capabilities including combined language skills, perceptual reasoning, text recognition, and chart understanding, GLM-4V-9B consistently outperforms competitors such as GPT-4-turbo-2024-04-09, Gemini 1.0 Pro, Qwen-VL-Max, and Claude 3 Opus, demonstrating its superior performance across multiple modalities.
- •
CogVLM2 is a series of open-source multi-modal large language models derived from Meta-Llama-3-8B-Instruct, developed by Zhipu.AI and Tsinghua University. This series contains two models: cogvlm-llama3-chat-19B and cogvlm2-llama3-chinese-chat-19B. The former is a monolingual language model focused on English and the latter is a bilingual model supporting both English and Chinese. CogVLM2 is designed to handle extended content lengths up to 8K and accepts high-resolution images up to 1344 * 1344. Here, we choose cogvlm2-llama3-chinese-chat-19B as our backbone to pre-train our MathGLM-Vision-19B.
- •
CogVLM-32B is a close-source multi-modal large language model, developed by Zhipu.AI and Tsinghua University. It is based on the GLM-32B architecture and is optimized for handling complex multi-modal tasks. CogVLM-32B is engineered to process visual inputs at a high resolution of 1120 * 1120, enabling detailed image analysis and enhanced interaction with visual data.

## Appendix C Descriptions of VQA Datasets

Here, we provide a detailed description of collected visual question answering datasets (VQA) datasets. Table 9 demonstrates details 19 different VQA datasets, including task types and visual context.

**Table 9: Summary of VQA datasets.**
| Dataset | Task | Visual Context |
| --- | --- | --- |
| DocVQA | Figure Question Answering (FQA) | Document Image |
| DVQA | Figure Question Answering (FQA) | Bar Chart |
| FigureQA | Figure Question Answering (FQA) | Charts and Plots |
| PlotQA | Figure Question Answering (FQA) | Bar, Line, Scatter |
| MapQA | Figure Question Answering (FQA) | Map Chart |
| IconQA | Math Word Problem (MWP) | Abstract Scene |
| TabMWP | Math Word Problem (MWP) | Table |
| CLEVR-Math | Math Word Problem (MWP) | Synthetic Scene |
| TQA | Textbook Question Answering (TQA) | Scientific Figure |
| AI2D | Textbook Question Answering (TQA) | Scientific Figure |
| ScienceQA | Textbook Question Answering (TQA) | Scientific Figure |
| A-OKVQA | Visual Question Answering (VQA) | Natural Image |
| VQA2.0 | Visual Question Answering (VQA) | Natural Image |
| PMC-VQA | Visual Question Answering (VQA) | Medical Image |
| VizWiz | Visual Question Answering (VQA) | Natural Image |
| Super-CLEVR | Visual Question Answering (VQA) | Synthetic Scene |
| VQA-AS | Visual Question Answering (VQA) | Abstract Scene |
| VQA-RAD | Visual Question Answering (VQA) | Medical Image |
| TextVQA | Visual Question Answering (VQA) | Natural Image |

## Appendix D Implementation Details

We provide a detailed overview of the Specialized Fine-Tuning (SFT) process applied to our MathGLM-Vision. The specific hyperparameters used during this process are outlined in Table 10.

**Table 10: The detailed setup of the SFT procedures.**
| parameters | MathGLM-Vision-9B | MathGLM-Vision-19B | MathGLM-Vision-32B |
| --- | --- | --- | --- |
| Total steps | 35,000 | 35,000 | 35,000 |
| Global Batch Size | 128 | 128 | 128 |
| Learning Rate | $1e^{-5}$ | $1e^{-5}$ | $1e^{-5}$ |
| Learning Rate Schedule | cosine decay | cosine decay | cosine decay |
| Warmup Ratio | 0.01 | 0.01 | 0.01 |
| Weight Decay | $5e^{-2}$ | $5e^{-2}$ | $5e^{-2}$ |
| Optimizer | AdamW | AdamW | AdamW |
| Input Resolution | 1120 * 1120 | 1344 * 1344 | 1120 * 1120 |
| Image Length | 1600 | 2304 | 1600 |

## Appendix E The detailed description of Benchmark Datasets

In this section, we provide an in-depth description of the benchmark datasets used to evaluate the performance of MathGLM-Vision. These benchmark datasets have been carefully curated to test the MLLMs’ capabilities. The detailed description of benchmark datasets is provides as follows.

- •
MathVista
MathVista is a comprehensive benchmark dataset designed to rigorously evaluate the mathmetical reasoning capabilities of language models (LMs), especially in varied visual contexts.
This dataset offers a comprehensive evaluation benchmark designed to integrate mathematical reasoning with visual understanding, focusing on five primary tasks: figure question answering (FQA); geometry problem solving (GPS); math word problem (MWP); textbook question answering (TQA); and visual question answering (VQA).
- •
MathVista-GPS
MathVista-GPS, a subset of the MathVista Dataset, specifically focuses on the domain of geometry problem solving. The questions in this subset range from basic shape recognition to more advanced problems involving theorems, calculations and reasoning.
- •
MathVerse
MathVerse is designed to provide a fair and comprehensive assessment of MLLMs’ capabilities in visual mathematics. The benchmark comprises 2,612 high-quality, multi-subject math problems, each featuring diagrams and converted into six different versions by human annotators. These versions offer varying levels of multi-modal information, allowing for a thorough evaluation of MLLMs’ understanding of visual diagrams.
- •
MATH-Vision
The MATH-Vision (MATH-V) dataset comprises 3,040 high-quality mathematical problems, each featuring a visual context and sourced from 19 real math competitions. This extensive and diverse collection allows for a comprehensive evaluation of LMMs’ ability to interpret and reason with visual information in mathematical contexts.
- •
MMMU
The Massive Multi-discipline Multi-modal Understanding and Reasoning (MMMU) benchmark encompasses 11.5K questions across six disciplines, including Art, Business, Health & Medicine, Science, Humanities & Social Science, and Tech & Engineering. The tasks in MMMU challenge models to perform sophisticated multi-modal analysis and apply domain-specific knowledge, demanding a higher level capability in comprehension and integration.

## Appendix F Detailed Experimental Results on Public Benchmark Datasets

Results on the testmini subset of MathVista. To comprehensively evaluate the performance of MathGLM-Vision across various task types featured in the MathVista dataset, we systematically evaluate it on the testmini subset. This subset has been carefully selected to represent a diverse range of mathematical problem types, ensuring a robust assessment of our model’s capabilities. Table 11 shows the evaluation results on the testmini subset of MathVista across various task types. Notably, MathGLM-Vision-19B and MathGLM-Vision-32B surpass human performance in overall accuracy, highlighting the advanced capabilities of these models in handling complex mathematical problems. In particular, MathGLM-Vision excels significantly in geometry problem solving (GPS) and geometry reasoning (GEO), demonstrating its superior proficiency in mathematical reasoning.

Table: Table 11: Accuracy scores on the testmini subset of MathVista. Input: $Q$: question, $I$: image, $I_{c}$: image caption, $I_{t}$: OCR texts detected from the image. ALL: overall accuracy. Task types: FQA: figure question answering, GPS: geometry problem solving, MWP: math word problem, TQA: textbook question answering, VQA: visual question answering. Mathematical reasoning types: ALG: algebraic reasoning, ARI: arithmetic reasoning, GEO: geometry reasoning, LOG: logical reasoning, NUM: numeric common sense, SCI: scientific reasoning, STA: statistical reasoning. The highest accuracy among all baseline MLLMs is marked in red, while the highest accuracy among various variants of MathGLM-Vision is marked bold.

Results on the testmini set of MathVerse. To thoroughly evaluate the performance of MathGLM-Vision across 12 detailed subjects within the MathVerse dataset, we conduct comprehensive experiments and report the results in Table 12. This analysis delves into the model’s ability to address a broad spectrum of mathematical challenges, ranging from geometry to functions. As shown in Table 12, MathGLM-Vision surpasses all open-source MLLMs and most close-source MLLMs. However, it still falls short by 14% compared to the performance of GPT-4V. In some subjects such as Angle, Analytic, and Property, MathGLM-Vision achieves better performance compared the advanced GPT-4V. For example, MathGLM-Vision-32B shows remarkable performance in plane geometry, particularly in handling angle-related problems, where it achieves a 60.1% accuracy, showcasing its strong geometric reasoning capabilities.

Table: Table 12: Mathematical Evaluation on Different Subjects and Subfields in MathVerse’s testmini Set. Len: Length; Anal: Analytic; Apply: Applied; Vol: Volume; Coord: Coordinate; Prop: Property; Exp: Expressio. The highest accuracy among all baseline MLLMs is marked in red, while the highest accuracy among various variants of MathGLM-Vision is marked bold.

Results on Math-Vision datasets. To effectively assess MathGLM-Vision’s ability across diverse subjects and difficulty levels within the Math-Vision dataset, we conduct a series of detailed evaluation experiments and report results in Table 13. Specifically, GPT-4V leads the close-source models with an overall accuracy of 22.76%, yet it remains significantly below the human performance benchmark of 75.66%. MathGLM-Vision shows competitive performance across a variety of mathematical disciplines compared to most of close-source MLLMs, with MathGLM-Vision-32B achieving the overall accuracy of 26.5%, closely approaching that of GPT-4V. Notably, MathGLM-Vision-32B excels in solid geometry with a accuracy of 29.1%, significantly outperforming the accuracy of 23.8% on GPT-4V. This superior performance in solid geometry highlights MathGLM-Vision-32B’s advanced spatial reasoning and geometric processing capabilities, which are essential for tackling complex three-dimensional problems.

**Table 13: Comparison of model performances across various mathematical subjects. Subjects: Alg: algebra, AnaG: analytic geometry, Ari: arithmetic, CombG: combinatorial geometry, Comb: combinatorics, Cnt: counting, DescG: descriptive geometry, GrphT: graph theory, Log: logic, Angle: metric geometry - angle, Area: metric geometry - area, Len: metric geometry - length, SolG: solid geometry, Stat: statistics, Topo: topology, TransG: transformation geometry. The highest accuracy among all baseline MLLMs is marked in red, while the highest accuracy among various variants of MathGLM-Vision is marked bold.**
| Human Performance |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Overall | Alg | AnaG | Ari | CombG | Comb | Cnt | DescG | GrphT | Log | Angle | Area | Len | SolG | Stat | Topo | TransG |
| Human (testmini) | 75.66 | 57.9 | 79.0 | 100.0 | 100.0 | 47.4 | 94.7 | 89.5 | 63.2 | 63.2 | 36.8 | 52.6 | 73.7 | 89.5 | 89.5 | 100.0 | 73.7 |
| Open-source MLLMs |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LLaVA-v1.5-7B | 8.52 | 7.0 | 7.1 | 10.7 | 7.1 | 4.8 | 10.5 | 7.7 | 10.0 | 9.2 | 15.6 | 10.2 | 9.8 | 5.3 | 8.6 | 4.4 | 4.8 |
| SPHINX (V2) | 9.70 | 6.7 | 7.1 | 12.9 | 7.5 | 7.7 | 6.0 | 9.6 | 16.7 | 10.1 | 11.0 | 11.8 | 12.5 | 8.2 | 8.6 | 8.7 | 6.0 |
| ShareGPT4V-7B | 10.53 | 5.5 | 3.6 | 12.9 | 10.1 | 4.8 | 7.5 | 11.5 | 14.4 | 10.9 | 16.2 | 11.8 | 12.3 | 9.8 | 15.5 | 17.4 | 11.3 |
| LLaVA-v1.5-13B | 11.12 | 7.0 | 14.3 | 14.3 | 9.1 | 6.6 | 6.0 | 13.5 | 5.6 | 13.5 | 10.4 | 12.6 | 14.7 | 11.5 | 13.8 | 13.0 | 10.7 |
| ShareGPT4V-13B | 11.88 | 7.5 | 15.5 | 16.4 | 10.7 | 8.9 | 9.0 | 11.5 | 8.9 | 7.6 | 11.6 | 13.0 | 17.4 | 10.3 | 8.6 | 8.7 | 12.5 |
| SPHINX-MoE | 14.18 | 7.8 | 17.9 | 14.3 | 15.6 | 9.5 | 11.9 | 12.5 | 15.6 | 12.6 | 16.2 | 15.6 | 17.8 | 13.5 | 12.1 | 8.7 | 16.1 |
| InternLM-XComposer2-VL | 14.54 | 9.3 | 15.5 | 12.1 | 15.3 | 11.3 | 10.5 | 14.4 | 22.2 | 19.3 | 19.7 | 15.6 | 15.0 | 11.9 | 15.5 | 26.1 | 15.5 |
| Closed-source MLLMs |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen-VL-Plus | 10.72 | 11.3 | 17.9 | 14.3 | 12.7 | 4.8 | 10.5 | 15.4 | 8.9 | 14.3 | 11.6 | 6.4 | 10.0 | 14.3 | 6.9 | 8.7 | 11.31 |
| Qwen-VL-Max | 15.59 | 10.7 | 19.1 | 20.0 | 16.9 | 12.5 | 17.9 | 16.4 | 12.2 | 21.0 | 13.3 | 14.2 | 19.8 | 11.5 | 20.7 | 13.0 | 17.3 |
| Gemini Pro | 17.66 | 15.1 | 10.7 | 20.7 | 20.1 | 11.9 | 7.5 | 20.2 | 21.1 | 16.8 | 19.1 | 19.0 | 20.0 | 14.3 | 13.8 | 17.4 | 20.8 |
| GPT4V | 22.76 | 27.3 | 32.1 | 35.7 | 21.1 | 16.7 | 13.4 | 22.1 | 14.4 | 16.8 | 22.0 | 22.2 | 20.9 | 23.8 | 24.1 | 21.7 | 25.6 |
| MathGLM-Vision |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MathGLM-Vision-9B | 19.2 | 18.6 | 20.2 | 19.3 | 15.3 | 18.5 | 20.9 | 26.0 | 18.9 | 15.1 | 23.1 | 20.4 | 18.3 | 23.8 | 19.0 | 17.4 | 14.3 |
| MathGLM-Vision-19B | 21.6 | 22.0 | 29.8 | 23.6 | 22.4 | 18.5 | 25.4 | 25.0 | 17.8 | 16.0 | 20.2 | 22.0 | 20.3 | 21.3 | 20.7 | 30.4 | 23.2 |
| MathGLM-Vision-32B | 26.5 | 22.9 | 20.2 | 24.3 | 23.1 | 28.0 | 20.9 | 34.6 | 27.8 | 23.5 | 31.2 | 26.8 | 30.1 | 29.1 | 22.4 | 17.4 | 26.2 |

## Appendix G Error Cases

Figure  12, Figure 13, Figure 14, and Figure 15 show examples of errors made by MathGLM-Vision-32B on the MathVL-test dataset. Each figure highlights a specific type of error, providing valuable insights into the model’s limitations and areas for improvement.

Figure: Figure 12: An example of reasoning error. MathGLM-Vision-32B mistakenly believed that the height of the water would initially increase rapidly and then slowly, indicating a reasoning error.
Refer to caption: x13.png

Figure: Figure 13: An example of vision recognition error. MathGLM-Vision-32B incorrectly interpreted the geometric properties of the diagram, leading to a vision recognition error.
Refer to caption: x14.png

Figure: Figure 14: An example of calculation error. MathGLM-Vision-32B made an error in calculating the lengths and areas in the isosceles trapezoid, leading to an incorrect solution.
Refer to caption: x15.png

Figure: Figure 15: An example of premature conclusion error. MathGLM-Vision-32B prematurely concluded that AB is perpendicular to CD without proper reasoning, leading to a premature conclusion error.
Refer to caption: x16.png

## Appendix H Case Study

Figure  16, Figure 17, Figure 18, and Figure 19 present several case studies from MathGLM-Vision-32B. These figures showcase the model’s performance in various scenarios, highlighting its strengths in providing concise and clear answers with logical mathematical reasoning. Compared to other close-source MLLMs, MathGLM-Vision-32B stands out for its ability to deliver precise and understandable solutions.

Figure: Figure 16: An example of solid geometry problem. MathGLM-Vision-32B correctly calculated the shortest path with shorter steps, while GPT-4o made a reasoning error, leading to a different conclusion.
Refer to caption: x17.png

Figure: Figure 17: An example of a planar geometry problem. MathGLM-Vision-32B correctly utilized axial symmetry to determine the angle sum, while Claude-3.5-Sonnet arrived at the correct answer through an erroneous calculation process. Despite the correct final answer, the calculation process was incorrect and overly complex.
Refer to caption: x18.png

Figure: Figure 18: An example of a planar geometry problem. MathGLM-Vision-32B correctly used the properties of similar triangles to find the area ratio, while GPT-4o misinterpreted the geometric relationships and misapplied the formula for the area ratio, leading to an erroneous conclusion.
Refer to caption: x19.png

Figure: Figure 19: An example of a geometric transformation problem. MathGLM-Vision-32B correctly used the properties of rotation to determine the angle, while QWen-VL-Max misunderstood the problem requirements and incorrectly calculated the rotation angle as 90°.
Refer to caption: x20.png

## Appendix I Model Evaluation

Evaluation on public benchmarks
The existing public benchmarks for evaluating a wide array of open-source and close-source models are neither timely nor comprehensive enough. To compare our MathGLM-Vision with the state-of-the-art open-source and close-source LLMs, we have supplemented the evaluations for some models missing from the public benchmark leaderboard.

We generate LLMs’ responses through API access (for closed-source models) and local inference (for open-source models). The evaluation was then conducted following the official evaluation code from each benchmark’s GitHub repository. The source of the models used in the evaluation can be found in Table  14.

**Table 14: The source of the models used in the evaluation.**
| Model | Input | LLM Size | Source |  |
| --- | --- | --- | --- | --- |
| Closed Source Models |  |  |  |  |
| Multi-modal LLMs |  |  |  |  |
| Gemini Pro | Q, I | - | [gemini-pro](https://ai.google.dev/) |  |
| Gemini 1.5 Pro | Q, I | - | [gemini-1.5-pro](https://ai.google.dev/) |  |
| GPT-4V | Q, I | - | [gpt-4-vision-preview](https://platform.openai.com/) |  |
| GPT-4-turbo | Q, I | - | [gpt-4-turbo](https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4) |  |
| GPT-4o | Q, I | - | [gpt-4o](https://platform.openai.com/docs/models/gpt-4o) |  |
| Claude-3-Opus | Q, I | - | [claude-3-opus-20240229](https://www.anthropic.com/api) |  |
| Claude-3.5-Sonnet | Q, I | - | [claude-3-5-sonnet-2024620](https://www.anthropic.com/api) |  |
| Qwen-VL-Plus | Q, I | - | [qwen-vl-plus](https://help.aliyun.com/zh/dashscope/developer-reference/vl-plus-quick-start) |  |
| Qwen-VL-Max | Q, I | - | [qwen-vl-max](https://help.aliyun.com/zh/dashscope/developer-reference/vl-plus-quick-start) |  |
| Open Source Models |  |  |  |  |
| General Multi-modal LLMs |  |  |  |  |
| mPLUG-Owl | Q, I | 7B | [mPLUG-Owl](https://github.com/X-PLUG/mPLUG-Owl) |  |
| LLaMA-Adapter-V2 | Q, I | 7B | [LLaMA-Adapter V2](https://github.com/ml-lab/LLaMA-Adapter-2) |  |
| InstructBLIP | Q, I | 7B | [InstructBLIP](https://huggingface.co/docs/transformers/main/en/model_doc/instructblip) |  |
| LLaVA-1.5 | Q, I | 13B | [LLaVA-v1.5-13B](https://github.com/haotian-liu/LLaVA) |  |
| ShareGPT-4V | Q, I | 13B | [ShareGPT4V-13B](https://huggingface.co/Lin-Chen/ShareGPT4V-13B) |  |
| SPHINX-MoE | Q, I | 8*7B | [SPHINX-MoE](https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/main/SPHINX/README.md) |  |
| SPHINX-Plus | Q, I | 13B | [SPHINX-Plus](https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/main/SPHINX/README.md) |  |
| InternLM-XC2 | Q, I | 7B | [InternLM-XComposer2-VL-7B](https://huggingface.co/internlm/internlm-xcomposer2-vl-7b) |  |
| InternVL-1.2-Plus | Q, I | 34B | [InternVL-Chat-V1-2-Plus](https://huggingface.co/OpenGVLab/InternVL-Chat-V1-2-Plus) |  |
| Geo-Multi-modal LLMs |  |  |  |  |
| G-LLaVA | Q, I | 7B | [G-LLaVA-7B](https://huggingface.co/renjiepi/G-LLaVA-7B) |  |
| G-LLaVA | Q, I | 13B | [G-LLaVA-13B](https://huggingface.co/renjiepi/G-LLaVA-13B) |  |
| LLaVA-1.5-G | Q, I | 7B | [LLaVA-1.5-7B-GeoGPT4V](https://huggingface.co/caishihao/GeoGPT4V-LLaVA-1.5-7B-v1) |  |
| LLaVA-1.5-G | Q, I | 13B | [LLaVA-1.5-13B-GeoGPT4V](https://huggingface.co/caishihao/GeoGPT4V-LLaVA-1.5-13B-v1) |  |
| ShareGPT4V-G | Q, I | 7B | [ShareGPT4V-7B-GeoGPT4V](https://huggingface.co/caishihao/GeoGPT4V-ShareGPT4V-7B-v1) |  |
| ShareGPT4V-G | Q, I | 13B | [ShareGPT4V-1.5-13B-GeoGPT4V](https://huggingface.co/caishihao/GeoGPT4V-ShareGPT4V-13B-v1) |  |
| Math-LLaVA | Q, I | 13B | [Math-LLaVA-13B](https://github.com/HZQ950419/Math-LLaVA) |  |

Evaluation on MathVL-test
We evaluate MathGLM-Vision and several close-source
MLLMs using our specially constructed MathVL-test. The evaluation process of our MathVL-test is conducted through 3 key-step: generation, extraction, and scoring.

For the generation step, the model responses are generated by providing the model with queries which incorporate the Chain of Thought (CoT) template, questions, and diagram information. The reponses of close-source MLLMs is generated through API access. For the extraction step, we use GPT-3.5-turbo to extract the model’s answer based on the reponses of first step. Finally, in the scoring step, the score for each question is determined by GLM-4 based on the comparison between the extracted answer and the standard answer.

The prompts used to guide the LLM in response generation, answer extraction and scoring can be found in Table  15.

**Table 15: Prompts for response generation, answer extraction and scoring.**
| Task | Prompt |
| --- | --- |
| Response Generation | You are a very skilled math teacher. Please provide a detailed, step-by-step solution to the question, following a step-by-step format. Be sure to conclude with a summary that states "The answer to this question is" followed by the final result. |
| Answer Extraction | Please read the following example. Then extract the answer from the model response and type it at the end of the prompt. Hint: Please answer the question requiring an integer answer and provide the final value, e.g., 1, 2, 3, at the end. Question: Which number is missing? Model response: The number missing in the sequence is 14. Extracted answer: 14 Hint: Please answer the question requiring a floating-point number with one decimal place and provide the final value, e.g., 1.2, 1.3, 1.4, at the end. Question: What is the fraction of females facing the camera? Model response: The fraction of females facing the camera is 0.6, which means that six out of ten females in the group are facing the camera. Extracted answer: 0.6 |
| Scoring | Please determine if the extracted_answer correctly answers the question. The correct answer needs to be extracted from the answer without recalculating it, and the answer in the answer should be considered the final answer. Also, do not judge whether the answer is correct. The question may contain multiple sub-questions, and correctly answering the question includes correctly answering every sub-question and every result within each sub-question. A relative error divided by the absolute value of the original answer of less than 0.01 is allowed. If the prediction does not contain an answer, it is considered wrong. If the answer is not numerical, determine the equivalence of the expression, not just the value. If there is one mistake, the answer is wrong. Only if all results given in the prediction are correct is it considered correct. There is no need to consider whether the solution process of the prediction is complete. Please first extract the answers given by the prediction, determine the relative error, check if each sub-question is answered correctly, and finally give the judgment in a single line (output only "yes" or "no" in a single line). |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.	In Section 2, the authors mention that MathVL includes “step-by-step solutions.” Were these solutions uniformly generated for all problem types, or was there variation based on the problem’s complexity? Could examples of different solution formats be provided?
2.	When curating MathVL, which specific properties were prioritized in selecting public datasets? Was the primary focus on dataset size, diversity of problem types, or specific visual elements?
3.	There is limited explanation on how the model handles equations or complex symbolic mathematics within visual inputs. For instance, were any specialized tokenization or embedding techniques employed for this type of input?
4.	Were any strategies, like curriculum learning, employed to ensure the model progressed from simpler to more complex mathematical problems? Adding this detail could be useful for readers interested in the training dynamics.
5.	MathGLM-Vision is designed to handle multiple images per problem, yet most benchmarks (e.g., MathVista) are single-image-based. How did the model perform on tasks involving single versus multiple images? Could the authors provide specific examples from MathVL-test where multiple images significantly contributed to?
soundness: 3
strengths: 1.	MathVL is a contribution as it diversifies the visual components beyond traditional geometry, covering topics like arithmetic, algebra, and statistics. This expansion enhances the MLLM’s utility across a broader spectrum of math problems.
2.	The experimental results show that MathGLM-Vision, with visual input, consistently outperforms its text-only counterparts, validating the value of multi-modal inputs.
3.	The paper uses a comprehensive set of benchmarks, including MathVL-test and public datasets, to evaluate performance. This approach provides a well-rounded assessment of MathGLM-Vision's mathematical reasoning capabilities and general vision-language understanding.
4.	The detailed analysis of error types—reasoning, vision recognition, knowledge, calculation, and question misunderstanding—provides useful insights into the model’s weaknesses and areas for improvement.
summary: The paper presents MathGLM-Vision, a specialized multi-modal large language model designed to solve mathematical problems. Specifically, the authors introduce MathVL, a fine-tuning dataset with a diverse range of mathematical problems involving visual elements. MathGLM-Vision models are fine-tuned on MathVL and are evaluated across public benchmarks. Results demonstrate significant performance improvements in mathematical problem-solving.
weaknesses: 1.	While the dataset is innovative, the model’s architecture and training process primarily involve data engineering and fine-tuning rather than a new modeling approach. This diminishes the paper's novelty as it focuses on enhancing existing models with improved data rather than proposing a novel methodological framework.
2.	The paper does not provide in-depth information about adjustments made to the base model architectures (GLM-4V, CogVLM) to optimize them for mathematical reasoning tasks. Adding technical details on modifications could clarify how MathGLM-Vision addresses the unique challenges of mathematical problem-solving.
3.	Although the paper mentions experiments on general vision-language benchmarks, the analysis is brief, and performance drops in non-mathematical tasks. This may suggest a trade-off between specialization in mathematical tasks and general multi-modal understanding, a limitation for applications beyond mathematics.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The VQA dataset's size (i.e., the number of data points it contains) would be helpful to clarify; 
2. For the error analysis, could you confirm which tool or method was used to identify error types? Was it done through a specific automated tool, manual annotation, or perhaps a LLM?
soundness: 3
strengths: 1. This article provides the MLLM community with high-quality Math VL data. The introduced dataset, MathVL, is very useful and significantly enhances the mathematical abilities of some MLLMs.
2. The experiments in this article are very detailed, and the writing is excellent.
summary: Current MLLMs specialized in math focus too much on geometry and miss other kinds of visual math information. To improve this, this paper created a new dataset called MathVL and developed MathGLM-Vision, which performed better on math problems than other models.
weaknesses: The paper claims three contributions: dataset, model, and benchmark. However, the dataset was only validated on weaker models, and the differences from existing datasets are not clearly described (details below). The developed model is essentially a fine-tuning version of the proposed dataset without any architectural innovation (while SFT is a common practice, it does not constitute a contribution since the model is merely a product of the dataset, especially as the final model is not state-of-the-art). Therefore, the claimed contribution are not justified well.

The difference between this article and Math-LLaVA is mentioned in lines 81-88, where it states that Math-LLaVA focuses on solving geometric problems. However, after reading Math-LLaVA, I found that it also includes various tasks, such as Math Word Problems and Textbook Question Answering. 


1. For Table 4: 1) I am particularly interested in the effectiveness of MathVL on other stronger models, such as the 7B InternLM-XC2, since the base model used in the experiment seems to have low mathematical ability according to the results in the table (about 40% accuracy on MathVista). 2) Why is the accuracy for GLM-4V-9B at 46.7 on MathVista, with its fine-tuned MathGLM-Vision-9B version reaching 52.2, while CogVLM2, which has an accuracy of only 40.85, achieves 61.1 after fine-tuning as MathGLM-Vision-19B? Why does a weaker model show better performance after fine-tuning?
2. Table 5, results for GLM-4V-9B, GLM-4V-9B w/o VQA datasets, and MathGLM-Vision-9B are missing.
3. Table 6, it seems to lack the exact composition method of MathVL-test in the paper. I am curious whether it is sampled from Open-source Data or Open-source Data + Chinese K12 Data. If it is the latter, then the performance drop with only SFT on Open-source Data alone would not necessarily indicate the importance of Chinese K12 Data, given that a significant performance drop is only observed on MathVL-test without Chinese K12 Data.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: My primary concern revolves around the paper's contribution to the community. While I acknowledge that the paper introduces a model for addressing mathematical problems and offers a useful benchmark for the field, I believe that the most impactful contribution to the advancement of Math MLLMs would be the provision of the dataset for others to use in training their own models. Making the data publicly available would significantly enhance the reproducibility and accessibility of the research, fostering broader progress in the development of Math MLLMs.
soundness: 3
strengths: 1. The proposed model, MathGLM-Vision, achieves good results on several benchmarks. The performance on MathVL-test is promising.
2. This paper presents multiple cases to better understand the collected dataset.
summary: This paper introduces MathVL,  a diverse and comprehensive multi-modal mathematical dataset for SFT. MLLMs of different scales trained on MathVL have demonstrated excellent performance, proving the dataset's effectiveness for solving mathematical problems. Beside, the paper also establishes a benchmark dataset called MathVL-test to evaluate the mathematical reasoning abilities of MLLMs using a multi-image format.
weaknesses: 1. The writing of the paper requires improvement. Several citations are not formatted correctly (e.g., Lines 75-78).
2. The paper appears to be more engineering-focused, with the primary contribution being the introduction of the MathVL dataset. While this is a valuable resource, the abstract does not mention any plans to make the dataset open-source or publicly available. Without access to the dataset, the practical impact of the work is significantly diminished, as other researchers cannot replicate or build upon the proposed model. Therefore, making the dataset publicly available would greatly strengthen the paper’s contribution and its value to the broader research community.
3. Maybe more figure examples can help to understand better the challenges presented in Line82-85. For example, the paper claims that current MLLMs tend to overlook the diversity of visual information in mathematics. Some visualizations or analyses would better support this conclusion.

Meta-review: This paper introduces a diverse and comprehensive multi-modal mathematical dataset MathVL, a benchmark dataset MathVL-test in a multi-image format, and several MLLMs of different scales fine-tuned on MathVL which show promising performance. While the reviewers appreciate these efforts, they pose a few concerns:  1. while the primary contribution is the MathVL dataset, there is no open-source plan to make it publicly available, so that others can replicate or build upon the proposed model; 2. the developed model is not state-of-the-art, and is essentially a fine-tuning version of the proposed dataset without any architectural innovation or modeling/training methodology innovation. No rebuttal is provided. AC suggests the authors address the above concerns and other questions posed by the reviewers.

Decision: Reject