MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine

## Abstract

Abstract Multi-modal Large Language Models (MLLMs) have recently showcased superior proficiency in general visual scenarios. However, we identify their mathematical capabilities remain under-explored with three areas to be improved: visual encoding of math diagrams , diagram-language alignment , and chain-of-thought (CoT) reasoning . This draws forth an urgent demand for an effective training paradigm and a large-scale, comprehensive dataset with detailed CoT rationales, which is challenging to collect and costly to annotate manually. To tackle this issue, we propose MAVIS , a MA thematical VIS ual instruction tuning pipeline for MLLMs, featuring an automatic data engine to efficiently create mathematical visual datasets.
We design the data generation process to be entirely independent of human intervention or GPT API usage, while ensuring the diagram-caption correspondence, question-answer correctness, and CoT reasoning quality. With this approach, we curate two datasets, MAVIS-Caption (558K diagram-caption pairs) and MAVIS-Instruct (834K visual math problems with CoT rationales), and propose four progressive stages for training MLLMs from scratch.
First, we utilize MAVIS-Caption to fine-tune a math-specific vision encoder (CLIP-Math) through contrastive learning, tailored for improved diagram visual encoding. Second, we also leverage MAVIS-Caption to align the CLIP-Math with a large language model (LLM) by a projection layer, enhancing vision-language alignment in mathematical domains. Third, we adopt MAVIS-Instruct to perform the instruction tuning for robust problem-solving skills, and term the resulting model as MAVIS-7B.
Fourth, we apply Direct Preference Optimization (DPO) to enhance the CoT capabilities of our model, further refining its step-wise reasoning performance.
On various mathematical benchmarks, our MAVIS-7B achieves leading results among open-source MLLMs, e.g., surpassing other 7B models by +9.3% and the second-best LLaVA-NeXT (110B) by +6.9%, demonstrating the effectiveness of our method. Code and data will be released https://github.com/ZrrSkywalker/MAVIS .

## 1 Introduction

The pursuit of artificial general intelligence necessitates models to seamlessly interpret and generate multi-modal data. In recent years, the advent of Large-language Models (LLMs) (Brown et al., ; Touvron et al., ; ; Chiang et al., ) and their Multi-modal extension (MLLMs) (Zhang et al., ; Gao et al., ; Su et al., ; Ye et al., ) have significantly facilitated this process across various fields, such as healthcare (Singhal et al., ; Shu et al., ), autonomous driving (Yang et al., ; Jin et al., ), and robotics (Li et al., ; Liu et al., ). Although MLLMs exhibit remarkable performance in diverse tasks and benchmarks, one arena where they have yet to fully demonstrate their potential is mathematical problem-solving in visual contexts.

Figure: Figure 1: (a) We compare the attention map of class tokens from CLIP ViT-L (Radford et al., ) and our CLIP-Math. Our vision encoder can better capture significant mathematical information within diagrams. (b) We compare the diagram captioning capabilities between GPT-4V (OpenAI, ) and our MAVIS-7B, where GPT-4V fall short of accurately recognizing mathematical elements. (c) We compare the chain-of-thought (CoT) reasoning between different models, showcasing that GPT-4V and Gemini-Pro (Gemini Team, ) suffer from low-quality reasoning process.
Refer to caption: x1.png

Existing efforts (OpenAI, ; ; Zhou et al., ) for text-only mathematics have attained considerable progress, largely attributed to the availability of sufficient and easily accessible training data.
In contrast, solving visual mathematical problems remains a significant challenge for MLLMs, primarily due to the absence of a fully validated, effective training pipeline and the acute shortage of large-scale, high-quality datasets. Visual mathematical data is not only more costly to collect from publicly available sources compared to text-only data, but also requires expensive manual annotation to produce accurate step-by-step chain-of-thought (CoT) rationales integrating diagram information.
In light of these challenges, we identify three critical issues that impede the visual mathematical capabilities of MLLMs.

- i.
Unsatisfactory math diagram embeddings by vision encoders. Most MLLMs adopt a frozen CLIP (Radford et al., ) as the vision encoder, which is pre-trained by natural images capturing real-world scenes with rich colors and textures. In contrast, math diagrams are composed of abstract curves, shapes, and symbols with a monochromatic color scheme, exhibiting large semantic gaps to general scenarios. As visualized in Figure 1 (a), the attention map of CLIP struggles to capture important information within math diagrams, which cannot provide satisfactory visual embeddings for LLMs to understand.
- ii.
Diagram-language misalignment between vision encoders and LLMs. Likewise, the vision-language pre-training stage of MLLMs also adopts natural image-caption pairs for cross-modal alignment. Due to the domain gap, while they can generate accurate captions for real-world images, but fall short of recognizing basic mathematical elements and narrating their relations. As compared in Figure 1 (b), even GPT-4V (OpenAI, ) produces low-quality descriptions for simple geometric figures and functions, indicating LLMs are not well aligned with the visual embedding space of math diagrams.
- iii.
Inaccurate CoT reasoning capabilities with visual elements by MLLMs. Referring to the CoT evaluation in MathVerse (Zhang et al., ), incorporating the diagram input would adversely affect the reasoning quality of MLLMs compared to using only the text-only question. As visualized in Figure 1 (c), we observe the problem-solving process of GPT-4V and Gemini-Pro (Gemini Team, ) both suffer from low-quality CoT reasoning accuracy. This demonstrates the incapability of MLLMs to leverage visual cues for precise step-by-step mathematical problem-solving.

Therefore, to mitigate these issues, it is essential to develop an extensive dataset and effective training approach tailored to visual mathematics. In this paper, we propose MAVIS, a MAthematical VISual instruction tuning paradigm and an automatic data generation engine for MLLMs, which aims to fully unleash their potential for diagram visual encoding and reasoning capabilities. We introduce two meticulously curated datasets, a progressive four-stage training pipeline, and a visual mathematical specialist, MAVIS-7B. We summarize the contributions of our work as follows.

- •
Automatic Mathematical Visual Data Engine.
To eliminate the need for labor-intensive annotation and expensive GPT API (OpenAI, ; ) usage, we designed our data engine to be entirely rule-based and fully automated. This engine handles every aspect of mathematical data creation, including diagram drawing, caption generation, question-answer synthesis, and CoT rationale production. With this approach, we curate two large-scale, high-quality mathematical visual datasets, MAVIS-Caption and MAVIS-Instruct, widely covering plane geometry, analytic geometry, and function.
MAVIS-Caption consists of 558K diagram-caption pairs automatically created by our data engine with accurate vision-language correspondence.
MAVIS-Instruct includes 834K visual math problems, which includes 582K data constructed by our data engine and additional 252K data augmented by GPT-4V from manual collection and existing datasets (Chen et al., ; Lu et al., ). Each problem is annotated with a CoT rationale, and modified to contain minimized textual redundancy that enforces MLLMs to pay more attention on visual diagrams.
- •
Four-stage Training Pipeline. Our training framework involves four progressive stages designed to sequentially address the aforementioned identified deficiencies in MLLMs. Firstly, we utilize MAVIS-Caption to fine-tune a math-specific vision encoder by contrastive learning, termed CLIP-Math, to enable better visual representations of math diagrams. Subsequently, we align this encoder with the LLM to ensure effective diagram-language integration also by MAVIS-Caption. After that, our MAVIS-Instruct is adopted to instruction-tune the MLLM, which provides sufficient step-wise problem-solving supervision. Finally, we employ Direct Preference Optimization (DPO) (Rafailov et al., ) with annotated CoT rationales in MAVIS-Instruct to further enhance the reasoning capabilities of our model.
- •
Mathematical Visual Specialist.
After the four-stage training, we develop MAVIS-7B, an MLLM specifically optimized for visual mathematical problem-solving. On various evaluation benchmarks, our model achieves leading performance compared to existing open-source MLLMs, e.g., surpassing other 7B models by +9.3% and the second-best LLaVA-NeXT (110B) (Li et al., ) by +6.9% on MathVerse (Zhang et al., ). The quantitative results and qualitative analysis both validate the significance of our approach.

## 2 Automatic Data Engine

To cope with the substantial data requirements of MLLMs, it is essential to have access to extensive training instances. However, for visual mathematics, the paucity of publicly available datasets poses a challenge, and creating such data manually is also not feasible due to the high cost involved. Therefore, we develop an automatic data engine to efficiently generate high-quality math diagrams (Section 2.1), detailed captions (Section 2.2), and question-answer pairs with CoT rationales (Section 2.3).

### 2.1 Diagram Generation

Covering most mathematical scenarios, we adopt three diagram types: plane geometry, analytic geometry, and function. Note that all the logic of the data engine is implemented in Python, and we employ Matplotlib for the graphical rendering of the diagrams.

#### Plane Geometry Diagram.

As such diagrams typically consist of spatial combinations of various basic shapes, we utilize principles from multi-hop data curation to develop customized generation rules. These rules allow for the iterative integration of new shapes into existing configurations. Initially, we establish a core set of shapes, including squares, rectangles, triangles, sectors, etc, for diagram generation. Starting with a randomly selected shape, we extend another shape from the set along one of its straight sides. By iterating this process, we can construct diverse plane geometry diagrams featuring different combinations of shapes. Additionally, we randomly label the vertices with letters (e.g., A, B, C) and annotate numerical values relevant to geometric properties (e.g., side lengths and angles), simulating realistic plane geometry problems.

#### Analytic Geometry Diagram.

Likewise, our approach begins by defining a basic figure set that differs slightly from that used in plane geometry; for example, we include additional elements such as points and line segments. We then construct a Cartesian coordinate system, complete with grid lines and scaled axes. The range of the coordinate system is randomly determined within a predefined scope. Subsequently, we select a number from 1 to 3 to indicate the number of figures to be drawn on the graph, and randomly choose coordinates for the top-left vertices to plot these figures at varied sizes (using these points as centers for circles). Unlike plane geometry, we ensure that the figures do not overlap, except for points and segments, and maintain the figure areas within a suitable scale.

#### Function Diagram.

We focus on seven fundamental function types: polynomial, sine, cosine, tangent, logarithmic, absolute value, and piece-wise polynomial functions. For each function type, we parameterize the equations with random variables, such as coefficients and constants within a predefined range (e.g., $a$ and $b$ in $y=ax+b$), which facilitates the generation of diverse function graphs. We also adopt the same Cartesian coordinate system employed for analytic geometry. Additionally, for specific caption or question-answering samples, we also plot key features like extreme points and zero points of the functions, providing additional visual information that aids in the understanding and reasoning of these mathematical functions.

Figure: Table 1: Statistics of MAVIS-Caption.

### 2.2 MAVIS-Caption

With our mathematical visual data engine, we first curate a diagram-caption dataset, MAVIS-Caption, as shown in Figure 2, aiming to benefit the diagram visual representations and cross-modal alignment.

#### Data Overview.

As presented in Table 2.1, the MAVIS-Caption dataset comprises 588K diagram-caption pairs. This includes 299K for plane geometry, 77K for analytic geometry, and 212K for function. The average word length of the captions is 61.48 words, reflecting their detailed descriptive nature. The overall vocabulary size is 149, indicating the diversity in language expression. We adopt different strategies to generate captions for three types of diagrams. It is important to note that GPT-4 (OpenAI, ) is only utilized during the template creation stage; it is not used at any point during the automatic caption generation process.

#### Plane Geometry Caption.

We follow the iterative geometric generation process to develop regulations for an accurate and detailed caption. We first prompt GPT-4 to create three sets of language templates: the descriptive content for fundamental shapes (e.g., “A Triangle {} with two congruent sides {} and {}”), the phrases to denote specific attributes (e.g., “Angle {} measures {} degrees”), and the conjunction to link two adjacent shapes (e.g., “Attached to edge {} of shape {}, there is a {}”). Then, based on various generation scenarios, we fill and merge these templates to acquire a coherent description of the geometric figure.

#### Function Caption.

As function diagrams typically showcase a single curve, we directly utilize GPT-4 to generate templates describing various properties of functions, including expressions, domains, ranges, extreme points, and zero points. Each template is then filled based on specific cases, such as “The expression of the function is $y=-3x^{3}-2x^{2}-2x-2$. Within the range of x values $[-3.0,4.0]$, zero points occur at $-0.83$ …”.

#### Analytic Geometry Caption.

We also employ GPT-4 to obtain two sets of language templates: the description of coordinates and attribute information for basic figures (e.g., “The square with its base left corner at {} features sides of {} in length”) and the spatial relation for nearby figures (e.g., “On the bottom right of {}, there is a {}”). The captions are then formulated by filling in the coordinates and selecting appropriate spatial relationship templates through coordinate comparison.

Figure: Figure 2: MAVIS-Caption Dataset. We showcase three diagram-caption pairs of plane geometry, function, and analytic geometry in MAVIS-Caption, generated by our developed data engine.
Refer to caption: extracted/5971112/figs/paperfig2.png

### 2.3 MAVIS-Instruct

Besides the diagram-caption data, we curate MAVIS-Instruct of extensive problem-solving data, which endows MLLMs with visual mathematical reasoning capabilities and serve as the basis for Direct Preference Optimization (DPO) (Rafailov et al., ), as shown in Figure 3.

#### Data Overview.

As illustrated in Table 2.1, the MAVIS-Instruct dataset consists of a total of 834K visual math problems. Given that the proportion of analytic geometry problems is relatively small, we classify them with function problems for simplicity. Each problem in MAVIS-Instruct includes a CoT rationale providing step-by-step solutions, with an average answer length of 150 words. We have minimized textual redundancy in the questions, eliminating unnecessary contextual information, distracting conditions, and attributes readily observable from the diagrams. This reduction in text forces MLLMs to enhance their capability to extract essential content from visual inputs. MAVIS-Instruct is assembled from four distinct sources to ensure broad coverage.

#### Data Engine Generated Problems.

Within our data engine, we manually craft rigorous regulations to produce visual math problems with accurate CoT annotations. Similar to caption generation, GPT API is not involved in the automatic synthesis process of questions, answers, and CoT rationales.

- •
Plane Geometry Problems. We initially prompt GPT-4 to compile a comprehensive set of mathematical formulas applicable to each basic shape (e.g., Pythagorean theorem for right triangles and area formula for circles). Then, for a geometric diagram, we randomly select a known condition within a shape as the final solution target, and systematically deduce backward to another condition, either within the same shape or an adjacent one, using a randomly selected mathematical formula. This deduced condition is then set as unknown, and we continue iterative backward deductions as necessary. The final condition, along with any conditions in the last step, are presented as initial attributes in the question. The rationales can be simply obtained by reversing this backward deduction process.
- •
Function Problems. As the properties of functions are predetermined, we utilize GPT-4 to generate diverse reasoning templates. These templates facilitate the solving of one function property based on other provided properties, thereby ensuring the generation of high-quality function rationales. The related function properties include analytical expression, function values, zeros, extremum points, monotonicity, derivatives, and integrals.
To accurately reason these properties, the CoT annotation incorporates understanding of function types, solving the analytical expressions of equations, and interpreting function graphs.

Figure: Figure 3: MAVIS-Instruct Dataset. We showcase the generated visual math problems from four sources within MAVIS-Instruct, which contain detailed rationales and minimized textual redundancy.
Refer to caption: extracted/5971112/figs/paperfig3.png

#### Data Engine Captions Annotated by GPT-4.

Given the detailed captions and diagrams generated by our data engine, we can prompt GPT-4V with these sufficient conditions to synthesis question-answering data and ensure its correctness. We first generate a new set of 17K diagram-caption pairs that do not overlap with the previous MAVIS-Caption, which avoids answer leakage within the detailed caption. Then, we prompt GPT-4V to generate 3 new problems with rationales, obtaining 51K data in total from the diagram-caption pairs.

#### Manual Collection Augmented by GPT-4.

To incorporate high-quality problems found in real-world contexts, we manually collect 4K math problems with diagrams from publicly available resources. Recognizing that these sources often lack detailed rationales and may contain redundant text, we initially utilize GPT-4V to annotate a detailed solving process and streamline the question text to reduce redundancy. Subsequently, for each collected instance, we input the question, rationale, and diagram into GPT-4 and employ customized few-shot prompts to generate 20 new problems per original, comprising 15 multiple-choice questions and 5 free-form questions. This process contributes a total of 83K problems to the dataset.

#### Existing Datasets Augmented by GPT-4.

Given existing well-organized geometric datasets, we can also leverage them to expand MAVIS-Instruct. Referring to previous prompt designs, we augment the 8K training set from two dataset, Geometry-3K (Lu et al., ) and GeoQA+ (Chen et al., ), into 80K visual problems with accompanying rationales, mapping each original problem to 10 new ones. Due to the scarcity of publicly available function data, we do not include function problems from this source.

## 3 Mathematical Visual Training

With the curated datasets, we devise a four-stage training pipeline for endowing MLLMs with mathematical visual capabilities as shown in Figure 4. They respectively aim to mitigate the three deficiencies within existing MLLMs, i.e., diagram visual encoding, diagram-language alignment, and mathematical reasoning skills in visual contexts.

Figure: Figure 4: Four-stage Training Pipeline of MAVIS. With our curated MAVIS-Caption and MAVIS-Instruct, we adopt four progressive stages for training a mathematical visual specialist from scratch.
Refer to caption: x2.png

### 3.1 Stage 1: Training CLIP-Math

To enhance CLIP’s (Radford et al., ) inadequate visual encoding of math diagrams, we utilize MAVIS-Caption to train a specialized CLIP-Math encoder. Specifically, we fine-tune a pre-trained CLIP-Base model following the conservative learning scheme. The math diagrams are fed into the learnable vision encoder, while the corresponding captions are processed by the text encoder, which remains frozen to provide reliable supervision. Via contrastive training, the model learns to adapt from its original natural image domain to mathematical contexts, increasing its focus on essential visual elements within diagrams, as demonstrated in Figure 1 (a). The optimized CLIP-Math encoder now delivers more precise and robust representations of math diagrams, establishing a solid foundation for the subsequent visual interpretation of LLMs.

### 3.2 Stage 2: Aligning Diagram-language

After acquiring the CLIP-Math encoder, we further integrate it with LLMs using MAVIS-Caption to boost cross-modal alignment between math diagrams and language embedding space. Using a simple two-layer MLP as the projection layer, we transform the visual encodings from CLIP-Math, and prepend them as a prefix to the LLM input.
This process, guided by the diagram captioning task, enables the LLM to accurately recognize mathematical components and spatial arrangements. With the diagram-language alignment, LLMs are equipped with the interpretation capability in math diagrams, serving as an initial step toward deeper mathematical reasoning.
In this stage, we freeze the CLIP-Math, and train the projection layer along with the LoRA-based (Hu et al., ) LLM.

Table: Table 3: Evaluation on MathVerse’s testmini Set with Six Problem Versions. ‘CoT-E’ and ‘Acc’ denote the scores of CoT evaluation strategy and the scores of direct ‘true or false’ accuracy, respectively. ‘^∗’ denotes previous mathematical visual specialists. The highest scores for closed-source and open-source MLLMs are marked in red and blue respectively.

### 3.3 Stage 3: Instruction Tuning

On top of that, we leverage MAVIS-Instruct to endow MLLMs with CoT reasoning and problem-solving capabilities in visual mathematics. The detailed rationales within each problem’s solution provide high-quality reasoning guidance for MLLMs, significantly enhancing their step-by-step CoT process. Furthermore, as we have minimized the redundancy within question texts during the construction process, such text-lite problem formats, referring to MathVerse (Zhang et al., ), facilitate MLLMs to capture more essential information from the visual embeddings for problem-solving, rather than relying on shortcuts to only process the textual content. In this stage, we unfreeze both the projection layer and apply LoRA (Hu et al., ) for the LLM for a thorough tuning.

### 3.4 Stage 4: Preference Alignment with DPO

After the instruction tuning phase, the resulting model gains the capability for CoT reasoning on visual math problems. However, it may still produce inaccurate intermediate steps due to insufficient supervision for generating the best reasoning path. To address this, we further apply CoT preference alignment using the DPO (Rafailov et al., ) algorithm to further enhance the model’s reasoning performance. Specifically, we adopt the instruction-tuned model to first infer CoT reasoning process on the 582K problems generated by data engine within MAVIS-Instruct. Then, we filter out the incorrect outputs (88K data) based on the final answer as the negative reasoning samples in DPO, and directly utilize the annotated CoT process as the positive samples. We only unfreeze the LoRA parameters for DPO training, and finally obtain our mathematical specialist, MAVIS-7B.

## 4 Experiment

We first detail our experimental settings in Section 4.1, and then discuss the quantitative on different benchmarks and qualitative examples in Sections 4.2 and 4.3, respectively. Please refer to the Appendix for more data details and ablation studies.

### 4.1 Experimental Settings

#### Implementation Details.

We adopt a CLIP ViT-L (Radford et al., ) as the pre-trained model to fine-tune our CLIP-Math, and utilize Mammoth2-7B (Yue et al., ) as the base LLM to construct MAVIS-7B. In the first stage, we fine-tune the CLIP for 10 epochs with a batch size 16 and an initial learning rate $2e^{-6}$. In the second stage, we train the diagram-language alignment for 1 epoch with a batch size 32 and an initial learning rate $2e^{-6}$, and adopt LoRA (Hu et al., ) with a rank 128. In the third and fourth stages, we adopt the same training settings as the second one.

#### Evaluation Schemes.

We evaluate our model MAVIS-7B on several popular mathematical benchmarks, MathVerse (Zhang et al., ), GeoQA (Chen et al., ), FunctionQA (function problems in MathVista (Lu et al., )), MMMU-Math (the math problems in MMMU (Yue et al., )), MathVision (Wang et al., ), three mathematical categories in MathVista, and We-Math (Qiao et al., ). We compare a variety of existing MLLMs, including two mathematical visual specialist (Gao et al., ; Shi et al., ), two LLMs (OpenAI, ; ), and other general MLLMs (Bai et al., ; Gao et al., ; Ye et al., ; Liu et al., ; Chen et al., ; Gao et al., ; Dong et al., ; Liu et al., ; Chen et al., ; Gao et al., ).

Table: Table 4: Evaluation on Six Mathematical Benchmarks. ‘MMMU-Math’ denotes the math problems within the test set of MMMU. ‘GPS’, ‘ALG’, and ‘GEO’ denote geometry problem solving, algebraic, and geometry in MathVista’s testmini set. ‘S1’, ‘S2’, and ‘S3’ denote different problem steps in We-Math’s testmini set. ‘^∗’ denotes previous mathematical visual specialists. The highest scores for closed-source and open-source MLLMs are marked in red and blue respectively.

### 4.2 Quantitative Performance

As shown in Table 3 for the MathVerse benchmark, MAVIS-7B achieves the best overall scores in both CoT evaluation and accuracy among open-source MLLMs with only a 7B model size, and consistently surpasses the second-best method on different problem versions. Specifically, our model surpasses the powerful InternLM-XComposer2 (7B) (Dong et al., ) by +9.3% and ShareGPT4V (13B) (Chen et al., ) by +17.8% CoT evaluation scores. Compared to other mathematical visual specialist, i.e., G-LLaVA (7B) (Gao et al., ) and the concurrent Math-LLaVA (13B) (Shi et al., ), MAVIS-7B exhibits superior problem-solving capabilities with higher CoT evaluation scores of +19.5% and +11.1%, respectively. In addition, our model is also advantageous to the most powerful open-source MLLM series, LLaVA-NeXT (Li et al., ), from 8B to 110B model sizes, demonstrating the math-specific proficiency of MAVIS-7B. Note that, the improvement brought by DPO (our fourth-stage training) is more apparent in CoT evaluation compared to the accuracy scores, indicating that the preference alignment learning can effectively boost the CoT reasoning capabilities.

Table 4 showcases the performance comparison on six other mathematical benchmarks, where our model still attains remarkable performance among other MLLMs. In detail, MAVIS-7B outperforms the closed-source Qwen-VL-Max (Bai et al., ) by +6.1% in MMMU-Math, +3.6% in MathVision, and around +10% in three subsets of We-Math. Our model even exceeds GPT-4V (OpenAI, ) in the three mathematical categories of MathVista, indicating our problem-solving and reasoning proficiency. We also observe that, the enhancement from DPO increases from ‘S1’ to ‘S3’ of We-Math, which well demonstrates its benefit on math problems with more intricate reasoning steps.

### 4.3 Qualitative Analysis

In Figure 5, we compare the mathematical problem-solving examples between MAVIS-7B and GPT-4V (OpenAI, ). As presented, our model not only showcases better accuracy in understanding the geometric elements, function curves, and coordinate axes in mathematical diagrams, but also performs higher-quality step-by-step reasoning process for formula substitution and numerical calculation. This demonstrates the effectiveness of our four-stage training pipeline and automatic data engine for enhanced diagram understanding and CoT reasoning.

Figure: Figure 5: Problem-solving Comparison of MAVIS-7B and GPT-4V.
Refer to caption: x3.png

## 5 Conclusion

In this paper, we propose MAVIS, the first mathematical visual instruction tuning paradigm for MLLMs. We first introduce two high-quality datasets by a delicate data engine, MAVIS-Caption and MAVIS-Instruct, containing large-scale diagram-language and problem-solving data. Then, we customize a three-stage training framework to progressively train the math-specific vision encoder, the diagram-language alignment, and the mathematical reasoning capabilities of MLLMs. The obtained specialist model, MAVIS-7B, achieves superior performance across different mathematical visual benchmarks, demonstrating the potential to serve as a new standard for future research.

## Appendix A Appendix

### A.1 Related Work

#### Visual Instruction Tuning.

The advancement of large language models (LLMs) (Brown et al., ; Jiang et al., ; Touvron et al., ; Chiang et al., ) with instruction tuning has significantly enhanced zero-shot capabilities across a range of tasks. Drawing inspiration from this, LLaMA-Adapter series (Zhang et al., ; Gao et al., ; Han et al., ) propose a zero-initialized attention mechanism to align frozen vision encoders (Radford et al., ) with LLaMA (Touvron et al., ) for multi-modal learning.
LLaVA series (Liu et al., ; ) employ a linear projector for vision-language alignment, establishing visual instruction tuning as a standard training approach in the multi-modal field. Flamingo (Alayrac et al., ) and OpenFlamingo (Awadalla et al., ) have honed visual representation by integrating a cross-attention resampler with vision encoders. SPHINX series (Gao et al., ; Lin et al., ) utilize a blend of visual encoders to make the LLM cognizant of various image aspects. InternVL series (Chen et al., ; Dong et al., ; Team, ) employ a large vision encoder and QFormer (Li et al., ) to incorporate high-quality visual information through a multi-stage training methodology.
LLaVA-NexT (Liu et al., ; Li et al., ; ) further introduces the ‘AnyRes’ technique to manage images at any given resolution, and LLaVA-NexT-Interleave (Li et al., ) extends the scope widely to interleave multi-image settings. There are also recent efforts to apply visual instruction tuning to 3D (Guo et al., ; Xu et al., ) and video (Li et al., ; Fu et al., ) scenarios.
Despite the impressive strides made in both model capability and training efficiency by multi-modal large language models (MLLMs) through visual instruction tuning, there is currently no MLLM specifically designed for mathematical problem-solving, nor a substantial dataset available for such purposes in the open-source community. In this paper, we mitigate the issue by proposing MAVIS with high-quality mathematical visual datasets and training paradigms.

#### Mathematics in Large Models.

Recent research has predominantly concentrated on text-only mathematical problem-solving using LLMs. MAmmoTH (Yue et al., ; ) have compiled extensive collections of mathematical problems, training LLMs using the reasoning processes described in solutions. MetaMATH (Yu et al., ) has expanded upon this by rewriting existing problems to create a larger dataset. MathCoder (Wang et al., ) and ToRA (Gou et al., ) introduced a tools agent approach, employing Python code and symbolic resolvers during the training phase, significantly outperforming traditional models that rely on text-only mathematical reasoning.
However, in the multi-modal field, despite the introduction of several datasets such as Geometry3K (Lu et al., ), GeoQA (Chen et al., ), UniGeo (Chen et al., ), UniMath (Liang et al., ), and GeomVerse (Kazemi et al., ), aiming at enhancing the performance of MLLMs in solving graphical mathematical problems, these datasets are quite limited in scale and domain. Based on these datasets, G-LLaVA (Gao et al., ) has developed superior capabilities for understanding graphical geometries but struggles with mathematical problems in other domains. The comprehensive benchmark MathVerse (Zhang et al., ) has also highlighted the existing MLLMs’ unsatisfactory capacity for encoding visual diagrams in diverse mathematical domains.
Therefore, there is a pressing need for the development of more robust encoders for mathematical images and the tuning of MLLMs with mathematical visual instructions, for which we propose MAVIS to address the challenges.

### A.2 Human Evaluation of MAVIS-Instruct

To assess the dataset’s coverage, validity, and quality, human verification is employed. The creation process of our MAVIS-Instruct dataset can be broadly categorized into two approaches:

- $\bullet$
GPT-generated: This method leverages GPT-4 to generate new problems (including questions, rationales, and answers) based on existing problems with diagrams. While this approach produces fluent, human-like sentences, it may be influenced by the inherent capabilities and occasional instability of GPT-4V.
- $\bullet$
Data Engine: As the main source of our mathematical visual data, this method utilizes the custom automatic data engine to generate new problems (including diagrams, questions, rationales, and answers), without relying on GPT models. It guarantees 100% correctness due to the use of rigorous templates, though it may occasionally exhibit rigid expressions.

Specifically, we evaluate four aspects(Diagram, Question, Rationale and Answer) of each problem using seven metrics. Each metric is scored on a scale of 1 to 3, where 1 denotes poor, 2 denotes moderate, and 3 denotes good. The human evaluation results are shown in Figure 7 and score statistics are shown in Figure 7. In addition, we also showcase some specific examples in Figure 8 and Figure 9. We analyze each aspect as follows:

- $\bullet$
Diagram: The diagrams in GPT-generated problems are directly collected from existing sources with rigorous human filtering, ensuring high quality, resulting in scores close to 3. In contrast, for rule-based problems, the diagrams are drawn accurately using Python code driven by our data engine, which guarantees correctness. However, these diagrams may lack alignment with human aesthetic preferences, as indicated by 3% of them receiving an appearance score of 1.
- $\bullet$
Question: Regarding the questions, both GPT-generated and rule-based problems display a high degree of accuracy in aligning with the diagram elements. This is attributed to the well-crafted prompts used with GPT-4 and the meticulous template design of the data engine. Nevertheless, rule-based questions may occasionally exhibit minor fluency issues, as they lack human refinement.
- $\bullet$
Rationale: In terms of the rationales, most instances feature a precise and detailed chain-of-thought (CoT) reasoning process. However, in a few cases (3% receiving an accuracy score of 1), some GPT-generated rationales contain minor reasoning or calculation errors, which are inherent to GPT-4’s limitations in problem-solving. These errors usually affect only one or two steps and do not compromise the overall logic. Conversely, the rule-based rationales are highly accurate due to the carefully designed data engine, although there is still room for improvement in language fluency.
- $\bullet$
Answer: The answers in both methods achieve high correctness scores. For GPT-generated problems, we prompt GPT-4 to identify a known condition from the original problems as the answer. Similarly, for rule-based problems, we randomly select a known attribute from the generated diagrams to serve as the answer.

Overall, the randomly sampled instances show that our dataset exhibits good question quality and answer accuracy.

Figure: Figure 6: Human Evaluation Results on 200 randomly sampled problems in MAVIS-Instruct, 100 GPT-generated and 100 Data Engine. We set three levels (1, 2, and 3) for each metric, and report average scores.
Refer to caption: x4.png

Figure: Figure 8: Diagram Examples in MAVIS-Instruct. The first three diagrams showcase superior correctness and appearance, while a small portion of Data Engine generated diagrams (3%) are not aligned with human preference, e.g., the fourth diagram.
Refer to caption: x6.png

Figure: Figure 9: Accurate Rationale Examples in MAVIS-Instruct. Most GPT-generated and Data Engine-generated rationales ensure correctness.
Refer to caption: extracted/5971112/figs/R1/correct.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: mentioned in the weakness
soundness: 3
strengths: - The dataset creation pipeline is quite rigorous ranging for the three categories – including caption and instruct data generation.
- The size of the captioning and instruct dataset is quite large, which is a plus too. The method for creating diagram-caption data using GPT-4 as template generators instead of caption generators is interesting.
- The paper performs 4-stage training: CLIP-Math, Aligning diagram-language alignment, instruction tuning and preference alignment. 
- The final numbers showcase the benefits of their data and training on the evaluations.
summary: The paper argues that the existing models lack in mathematical reasoning in the visual contexts due to the lack of math diagram understanding by the vision encoders, limited diagram-language alignment data, and inaccurate capability of large models (e.g.,GPT-4V, Gemini-Pro) to perform chain-of-thought reasoning. To address this, the paper proposes MAVIS, a rule-based and automatic data engine used for visual instruction tuning of the 7B large multimodal models. Specifically, the paper focuses on plane geometry diagrams, analytic geometric diagrams, and function diagrams. Subsequently, the paper performs 4-stage training on MAVIS-caption and instructs to achieve good performance on the MathVerse dataset. While the data collection, and training pipelines look reasonable, the paper severely lacks solid experiments to establish the usefulness of the generated data.
weaknesses: - Mammoth-2 is supposedly a strong model for math reasoning for text. It remains unclear how much benefit does the MAVIS data give over finetuning Mammoth-2 with existing visual instruction tuning dataset (e.g., LLaVA data). This is crucial to control for the choice of the base model itself. 
- The main result is on the MathVerse dataset which consists of author-created solid geometry and function problems and plane geometry problems sourced from GeoQA and Geometry3K. The authors augment their dataset with the problems from the same dataset in L338. It is crucial to see if most of the improvements in Table 2 are in the plane geometry, while solid geometry and functions still suffer. 
- Related to above: there is a synthetic component to the dataset and a real-data (e.g., Geometry datasets and sourcing problems from internet) component to the instruct data. How much of the performance comes from the synthetic data and how much of it is contributed by adding the real data is unclear. 
- There are no experiments to study data scaling — does training with all the data better than training with 25%, 50% and 75% of the entire dataset. If the data quality is high, then the model performance should scale with data.
- Table 4 has too many dashes for open-source MLLMs which makes it hard to put the model improvements in the complete context. The table makes it look as if some numbers have been skipped selectively. 
- The public sources (or protocol) for selecting the math problems from public sources in L322 is unclear. There is no data contamination study to understand the overlap between the generated data and evaluation data. 
- Minor: Table 3 and 4 adds a * on the models that are consider math specialist but there is no * on the MAVIS models themselves. I would consider them math specialists too.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - In Section 3.2, why only freeze the CLIP visual encoder and train the projection layer along with the LoRA-based LLM? In LLaVA [1], both the visual encoder and LLM are frozen, with only the projection layer trained to align the visual encoder and LLM. If training the LoRA-based LLM is better, please provide a rationale and ablation experiments to support this approach. 

- The core idea behind the rule-based caption and instruction data collection is akin to template filling, and the concept for constructing CoT data seems similar to deductive reasoning?

[1] Liu H, Li C, Wu Q, et al. Visual instruction tuning[J]. Advances in neural information processing systems, 2024, 36.
soundness: 3
strengths: - The motivation behind this work is clear. For example, it identifies the semantic gap between natural images with rich colors and textures and mathematical diagrams with grayscale and abstract elements. Therefore, constructing math diagram-caption pairs is necessary to enhance visual embeddings.
- The data collection pipeline does not require extensive manual annotation or the use of OpenAI API, it is rule-based and automated, making it relatively low-cost. 
- This work involves a substantial amount of engineering effort, collecting 1.4 M data in the mathematical domain. The model also achieved high performance across multiple benchmarks.
summary: This work primarily aims to enhance the visual mathematical reasoning capabilities of MLLMs. The authors observe that existing visual encoders struggle to capture abstract information in math diagram, and MLLMs lack strong visual mathematical reasoning abilities. To address this, they propose a data collection pipeline and collected approximately 1.4 million math diagram-caption pairs and VQA with CoT data. By fine-tuning the CLIP visual encoder, aligning CLIP with the LLM, and applying Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) alignment techniques, they train a 7B math MLLM.
weaknesses: - The main contribution of this paper is the proposal of an automated data collection pipeline that doesn’t require extensive manual annotation or the use of the OpenAI API. However, the paper lacks an overview figure in the main text, which may make it challenging for readers to grasp the key idea of this data collection pipeline.
- This work lacks a certain degree of novelty. For instance, it does not propose a new model architecture, and the training approach is a combination of the popular methods, such as CLIP’s contrastive training[1], LLaVA’s visual instruction tuning, and Direct Preference Optimization[3] (DPO). The model and training method are not specifically optimized for the mathematical domain; instead, they rely on a general model and training approach, with only a mathematical dataset provided for fine-tuning. (Fortunately, the engineering effort is substantial.)

[1] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PMLR, 2021: 8748-8763.

[2] Liu H, Li C, Wu Q, et al. Visual instruction tuning[J]. Advances in neural information processing systems, 2024, 36.

[3] Rafailov R, Sharma A, Mitchell E, et al. Direct preference optimization: Your language model is secretly a reward model[J]. Advances in Neural Information Processing Systems, 2024, 36.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - I'm wondering the result of Table 6, Mathverse but on vision only version. Since CLIP-math helps the model improve its visual representation, a performance improvement on vision-only version can be more convincing.
soundness: 3
strengths: - The writing is clear and easy to follow.
- The idea of using dataset generation engine to enhance MLLM math visual reaoning ability is straightforward and effective.
- The training stretagy, including the staged training, and the dataset design are well-established.
summary: This paper introduce MAVIS, a multimodal large language model trained with mathmatical dataset generated by data engine.
Specifically, the authors constructed MAVIS-caption to train CLIP-Math with better mathmatical visual representation ability and MLP multimodal connector. Then the authors constructed MAVIS-Instruct to endow MLLM instruct follow and CoT reasoning ability.
The evaluation result shows that MAVIS achieve good performance on multiple math-domain benchmarks.
weaknesses: - Although MAVIS is a mathematical visual specialist, the performance is still similar with some generalist MLLMs. Especially on vision-only version of Mathverse, where MAVIS is only 0.9% better than LLaVA-NEXT-110B. Suggesting that the MAVIS's visual representation learning is not very effective.
- The generation engine can only cover a limited set of mathmatical diagrams. For example, circumcircle of a triangle, which is a very common geometry shape, can not be generated by the engine. Since the engine is manually designed, it will be challenging to cover all kinds of math visual diagram types. If the goal is to train a generalize math MLLM, discussion on the generalize ability of MAVIS is needed, since the training shape is limited.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. This paper comprehensively investigates different aspects of training multimodal math models, providing good insights in this field. The scope includes data curation, visual encoding, training pipelines, and learning methodologies. The exploration of better math-specific visual encoding, Math-CLIP, is interesting. This is a good reference for future work.

2. The proposed data engine is novel and of great value for obtaining large-scale training data. As discussed in the paper, the collection and annotation are expensive for multimodal math, while this data engine effectively alleviates this point. The author also shows detailed human study and comparison to ensure the data quality.

3. The MAVIS-7B model shows competitive performance in different benchmarks. The results validate the effectiveness and generalizability of the proposed data and training method.
summary: This paper proposes MAVIS, a mathematical instruction tuning framework, including an automatic data generation engine, two multimodal mathematical datasets, and a specialized model MAVIS-7B. The mathematical instruction tuning of MAVIS consists of four progressive steps from visual pretraining to preference alignment, respectively targeting different issues in this field. The rule-based data engine can generate numerous mathematical data without costly human and GPT engagement, which contributes to sufficient data resources for training. The model MAVIS-7B shows good results on different benchmarks.
weaknesses: 1. The data engine is a rule-based system with fixed language templates. How to ensure the diversity of captions and questions in training data? It is very important to use wide and diverse data for robust model training.

2. The author only provides the results of the 7B model. What if the model size is larger, for example, 13B?

3. The author adopts introduced data and methods to train MAVIS from scratch. However, it is unknown whether these approaches are generalized enough to help a pre-trained large model, for example, LLaVA, to improve the mathematical performance.

Meta-review: This paper introduces MAVIS, a multimodal instruction tuning framework designed to enhance mathematical reasoning capabilities in large multimodal language models (MLLMs). It features an automatic data engine that generates mathematical datasets, including MAVIS-Caption and MAVIS-Instruct, without requiring manual annotations. The proposed training pipeline addresses visual encoding, alignment, and chain-of-thought (CoT) reasoning. Experiments demonstrate MAVIS-7B's significant improvements across multiple benchmarks, including MathVerse and GeoQA. While reviewers appreciated the dataset's scale, training strategy, and competitive results, concerns were raised about limited generalization, constrained mathematical diagram diversity, and the computational limitations of the rule-based engine. These issues were addressed through additional experiments and clarifications during the rebuttal phase. Overall, the work is recommended for Accept (poster) due to its substantial contributions to mathematical visual reasoning in MLLMs.

Decision: Accept (Poster)