MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine
Authors: Renrui Zhang
Xinyu Wei
Dongzhi Jiang
Ziyu Guo
Yichi Zhang
Chengzhuo Tong
Jiaming Liu
Aojun Zhou
Shanghang Zhang
Peng Gao
Hongsheng Li
Venue: ICLR 2025 Poster (2025)
Keywords: Large language model
multimodal learning
mathematics
Primary area: foundation or frontier models, including LLMs

Abstract: Multi-modal Large Language Models (MLLMs) have recently showcased superior proficiency in general visual scenarios. However, we identify their mathematical capabilities remain under-explored with three areas to be improved: visual encoding of math diagrams, diagram-language alignment, and chain-of-thought (CoT) reasoning. This draws forth an urgent demand for an effective training paradigm and a large-scale, comprehensive dataset with detailed CoT rationales, which is challenging to collect and costly to annotate manually. To tackle this issue, we propose MAVIS, a MAthematical VISual instruction tuning pipeline for MLLMs, featuring an automatic data engine to efficiently create mathematical visual datasets.
We design the data generation process to be entirely independent of human intervention or GPT API usage, while ensuring the diagram-caption correspondence, question-answer correctness, and CoT reasoning quality. With this approach, we curate two datasets, MAVIS-Caption (558K diagram-caption pairs) and MAVIS-Instruct (834K visual math problems with CoT rationales), and propose four progressive stages for training MLLMs from scratch.
First, we utilize MAVIS-Caption to fine-tune a math-specific vision encoder (CLIP-Math) through contrastive learning, tailored for improved diagram visual encoding. Second, we also leverage MAVIS-Caption to align the CLIP-Math with a large language model (LLM) by a projection layer, enhancing vision-language alignment in mathematical domains. Third, we adopt MAVIS-Instruct to perform the instruction tuning for robust problem-solving skills, and term the resulting model as MAVIS-7B. Fourth, we apply Direct Preference Optimization (DPO) to enhance the CoT capabilities of our model, further refining its step-wise reasoning performance.
On various mathematical benchmarks, our MAVIS-7B achieves leading results among open-source MLLMs, e.g., surpassing other 7B models by +9.3% and the second-best LLaVA-NeXT (110B) by +6.9%, demonstrating the effectiveness of our method.

Published as a conference paper at ICLR 2025
MAVIS:
MATHEMATICAL VISUAL INSTRUCTION TUNING WITH AN AUTOMATIC DATA ENGINE
Renrui Zhang- Xinyu Wei"*, Dongzhi Jiang', Ziyu Guo?, Yichi Zhang', Chengzhuo Tong' Jiaming Liu Aojun Zhou', Shanghang Zhang² Peng Gao", Hongsheng Li
'CUHK MMLab & MiuLLrr Lab
Peking University
'Shanghai AI Laboratory
CCPII under InnoHK
renruizhang@linn cukk. euh..edu...u........ hk, allen_wei@stu. pku. edu. cn
Equal contribution
Project lead
Corresponding author
ABSTRACT
Multi-modal Large Language Models (MLLMs) have recently showcased superior proficiency in general visual scenarios. However, we identify their mathemati- cal capabilities remain under-explored with three areas be improved: visual encoding of math diagrams, diagram-language alignment, and chain-of-thoug!! (CoT) reasoning. This draws forth urgent demand for effective training paradigm and a large-scale, comprehensive dataset with detailed CoT rationales, which is challenging to collect and costly annotate manually. To tackle this issue, we propose MAVIS, MAthematical VISual instruction tuning pipeline for MLLMs, featuring an automatic data engine efficiently create mathematical visual datasets. We design the data generation process to be entirely independent human intervention or GPT API usage, while ensuring the diagram-caption correspondence, question-answer correctness, and CoT reasoning quality. With this approach, we curate two datasets, MAVIS- Cappion (558K diagram-caption pairs) and propose four progressive stages for training MLLMs from scratch. First, we utilize MAVIS-Caption to fine-tune math-specift vision encoder (CLIP-Math) through contrastive learning, tailored for improved diagram visual encoding. Second, we also leverage MAViS-Caption to align the CLIP-Math with large language model (LLM) by a projection layer, enhancing vision-language alignment in mathematical domains. Third, we adopt MAVIS-Instruch robust problem-solving skills, and term the resulting model as MAVIS- -BB. Fourth, we apply Direct Preference Optimization (DPO) to enhance the CoT capabilities of our model, further refining its dep-wise reasoning performance. On various mathe- matical benchmarks, our MAVIS-7B achieves leading results among open-source MLLMs, e.g., surpassing other 7B models by +9.3% and the second-best LLaVA, NeXT (1 10B) by +6.9%, demonstrating the effectiveness of our method. Data and models are released at https: /ttttp::/githteee httptggtttt/t///////////ht com
(834K visual math problems with CoT rationales), and
perform the instruction tuning for
INTRODUCTION
The pursuit of artificial general intelligence necessitates models to seamlessly interpret and generate multi-modal data. In recent years, the advent of Large-language Models (LLMs)) (Brown et al., 2020; Touvron et al., 2023a;b; Chiang et al., 2023) and their Multi-modal extension (MLLMs) (Zhang et al., 2024a; Gao et al., 2023b; Su et al., 2023; Ye et al., 2023a) have significantly facilitated this process across various fields, such as healthcare (Singhal et al., 2023; Shu et al., 2023), autonomous driving (Yang et al., 2023; Jin et al., 2024), and roboties (Li et al., 2023b; Liu et al., 2024b). Although MLLMs exhibit remarkable performance in diverse tasks and benchmarks, one arena where they have yet to fully demonstrate their potential is mathematical problem-solving in visual contexts.

---

Published as a conference paper at ICLR 2025
Math Diagram
CITL
CLI VIT-L.
Our CLIP-Math
MAVES-78:
ABC is drows with orgle CBA mecsuring 30 degress, square cADE to edge with angle degrees mecsuinng and a is 10.0 unins, the desorce behanon points and B units, squere. CBDE has insorbeed circk..
GPE4N
mage shons gometiic figure points labeled wrich Paint line extending at 5 with redia of 10 units, reaching point C, The figure includes en inseribed arde centened with the ouadriateed formed points C, D,
of
sogmeet
90 foreed There
arclee centered
DE ane paralie...
(a) Attention Comparison between CLIP and our CLIP-Math
(b) Captioning Comparison between GPT-4V and our MAVIS-7B
Question:
GPT-4V:
Gemini-Pro:
MAVIS-7B
Find the domain and nege of the fanction using interaal nottioon
which goes through (-2.4), (-1-2).
X
from the graph. function goes
The darein of the function is the / axt all x-schee for which the
V
the groph, ae liww that the ands srl.
(9.0) (1,4).
There hellow point xx-3.
funciio is for
Answers do,3in(-3,]] range[-4.0]
detined heree
V Reeides hellow
but ceciudin
x
The range is the of possible cupputt (yredee))
[-4, -330(-5.33
X
noterion [-3.3]
demsin (-9.3)
We
the
narge
groph,
Fron the graph The lowest yolal (st 2 1e13
function can take er.
V funchion mastman bunnd reashes miviss diat x--12 and
The hishert yoeles O, ben since poit (-300)
function npens yraskuee banwees -4 and G. incldding -4 and
v The fanction attsinl all vilues
fuection neach
X Thus, the nange (-4,
We write this interval note asi [44.0]
V Thas, the range in (-4.0)
(c) CoT Reasoning Comparison between Different MLLMs
Figure I: (a) We compare the attention map of class tokens from CLIP ViT-L (Radford et al., 2021) and our CLIP-Math. Our vision encoder can better capture significant mathematical information within diagrams. (b) We compare the diagram captioning capabilities between GPT-4V (OpenAI, 2023c) and our MAVIS-7B, where GPT-4V fall short of accurately recognizing mathematical elements. (c) We compare the chain-of-thought (CoT) reasoning between different models, showcasing that GPT-4V and Gemini-Pro (Gemini Team, 2023) suffer from low-quality reasoning process.
Existing efforts (OpenAI, 2023b;a; Zhou et al., 2023) for text-only mathematics have attained considerable progress, largely attributed the availability of sufficient and easily accessible training data. In contrast, solving visual mathematical problems remains a significant challenge for MILMMs primarily due to the absence of fully validated, effective training pipeline and the acute shortage of large-scale, high-quality datasets. Visual mathematical data is not only more costly collect from publicly available sources compared to text-only data, but also requires expensive manual annotation to produce accurate step-by-step chain-of-though (CoT) rationales integrating diagram information. In light of these challenges, we identify three critical issues that impede the visual mathematical capabilities of MLLMs.
Unsatisfactory math diagram embeddings by vision encoders. Most MLLMs adopt frozen CLIP (Radford et al., 2021) as the vision encoder, which is pre-trained by natural images capturing real-world scenes with rich colors and textures. In contrast, math diagrams are composed of abstract curves, shapes, and symbols with a monochromatic color scheme, exhibiting large semantic gaps to general scenarios. As visualized in Figure I (a), the attention map of CLIP struggles to capture important information within math diagrams, which cannot provide satisfactory visual embeddings for LLMs to understand.
ii Diagram-langee Iision-langure pre- pre-training stage of MLLMs also adopts natural image-caption pairs for cross-modal alignment. Due to the domain gap, while they can generate accurate captions for
misalignment between vision encoders and LLMs. Likewise, the
2

---

Published as a conference paper at ICLR 2025
real-world images, but fall short of recognizing basic mathematical elements and narrating their relations. As compared in Figure 1 (b), even GPT-4V (OpenAI, 2023c) produces low-quality descriptions for simple geometric figures and functions, indicating LLMs are not well aligned with the visual embedding space of math diagrams.
ii. Inaccurate CoT reasoning capabilities with visual elements by MLLMs. Referring to the CoT evaluation in Math Verse (Zhang et al., 2024b), incorporating the diagram input would adversely affect the reasoning quality of MI.LMs compared to using only the text- only question. As visualized in Figure (c), we observe the problem-solving process of GPT-4V and Gemini-Pro (Gemini Team, 2023) both suffer from low-quality CoT reasoning accuracy. This demonstrates the incapability of MLLMs to leverage visual cues for precise step-by-step mathematical problem-solving.
Therefore, mitigate these issues, it is essential to develop extensive dataset and effective training approach tailored to visual mathematics. In this paper, we propose MAVIS, a MAthematical VISual instruction tuning paradigm and an automatic data generation engine for MLLMs, which aims fully unleash their potential for diagram visual encoding and reasoning capabilities. We introduce two meticulously curated datasets, a progressive four-stage training pipeline, and a visual mathematical specialist, MAVIS-74 7B. We summarize the contributions of our work as follows.
Automatic Mathematical Visual Data Engine. To eliminate the need for labor-intensiee annotation and expensive GPT API (OpenAI, 2023c;b) usage, we designed our data engine to be entirely rule-based and fully automated. This engine handles every aspect of math- ematical data creation, including diagram drawing, caption generation, question-answer synthesis, and CoT rationale production. With this approach, we curate two large-scale, high-quality mathematical visual datasets, MAVIS-Caption and MAVIS-Instruch widely covering plane geometry, analytic geometry, and function. MAVIS-Captinn consists of 558K clagram-caption pairs automatically created by our data engine with accurate vision- language correspondence. MAVIS-Instrun includes 834K visual math problems, which includes 582K data constructed by our data engine and additional 252K data augmented by GPT-4V from manual collection and existing datasets (Chen et al., 2021c; Lu et al.. 2021). Each problem is annotated with a CoT rationale, and modified to contain minimized textual redundancy that enforces MILMM to pay more attention on visual diagrams.
Four-stage Training Pipeline. Our training framework involves four progressive stages designed to sequentially address the aforementioned identified deficiencies in MLLMs. Firstly, we utilize MAVIS- Caption to fine-tune a math-specific vision encoder by contrastive learning, termed Math,-Mtth, enable better visual representations of math diagrams. Subsequently, we align this encoder with the L.M to ensure effective dligram-Ilnnguge integration also by MAVIS-Caption. After that, our MAVIS-Instruee is adopted instruction- tune the MLLM, which provides sufficient step-wise problem-solving supervision. Finally, we employ Direct Preference Optimization (DPO) (Rafailov et al., 2024) with annotated CoT rationales in MAVIS- Instruct to further enhance the reasoning capabilities of our model.
Mathematical Visual Specialist. After the four-stage training, we develop MAVIS-7B, an MLLM specifically optimized for visual mathematical problem-solving. On various evaluation benchmarks, our model achieves leading performance compared to existing open-source MLLMs, e.g., surpassing other 7B models by +9.3% and the second-best LLaVA-NeXT (110B) (Li et al., 2024a) by +6.9% on Math (Zhang et al., 2024b). The quantitative results and qualitative analysis both validate the significance of our approach.
AUTOMATIC DATA ENGINE
To cope with the substantial data requirements of MLLMs, it is essential to have access to extensive training instances. However, for visual mathematics, the paucity of publicly available datasets poses challenge, and creating such data manually also involves high cost. Therefore, illustrated in Figure 2, we develop an automatic data engine to efficiently generate high-quality math diagrams (Section 2), captions (Section 2.2), and question-answer with rationales (Section 2.3).
3

---

Published as a conference paper at ICLR 2025
Geometry Problem Generation:
Fundomertaal
nakery cage
2
K
Betondd
Rchiceole
ltepitta
Rationale
32"
Caption Viter
tepplii
Caph
D C
A
cs* 0000-114
IC, 6CD
What the arcs of sucier C302
Function Problem Generation:
Recionale
Rut lerionele
Exp oil
VA
Expression
vv
x yr 22iagl0k
Frome the 20% 1ihurtt
Atl The sere of the
Caption
foart
Roox Pitgiter111
Overview of Automatic Data Engine. We present the generation pipelines of geometry (Top) and function (Bottom) problems within the proposed automatic data engine, including diagrams, questions, captions, and Chain-of- ofaiinof-Thhuuht (CoT) rationales.
Figure 2:
2.1 DIAGRAM GENERATION
Covering most mathematical scenarios, we adopt three diagram types: plane geometry, analytic geometry, and function. Note that all the logic of the data engine is implemented in Python, and we employ Matplotlib for the graphical rendering of the diagrams.
Plane Geometry Diagram. As such diagrams typically consist of spatial combinations of various basic shapes, we utilize principles from multi-hop data curation to develop customized generation rules. These rules allow for the iterative integration of new shapes into existing configurations. Initially, we establish a core set of shapes, including squares, rectangles, triangles, sectors, etc, for diagram generation. Starting with randomly selected shape, extend another shape from the set along one of its straight sides. By iterating this process, we can construct diverse plane geometry diagrams featuring different combinations of shapes. Additionally, randomly label the vertices with letters (e.g., A, B, C) and annotate numerical values relevant to geometric properties (e.g., side lengths and angles), simulating realistic plane geometry problems.
Analytic Geometry Diagram. Likewise, our approach begins by defining basic figure set that differs slightly from that used in plane geometry; for example, we include additional elements such as points and line segments. We then construct Cartesian coordinate system, complete with grid lines and scaled axes. The range of the coordinate system is randomly determined within predefined scope. Subsequently, we select a number from to 3 to indicate the number of figures to be drawn on the graph, and randomly choose coordinates for the top-left vertices to plot these figures varied sizes (using these points as centers for circles). Unlike plane geometry, we ensure that the figures do not overlap, except for points and segments, and maintain the figure areas within suitable scale.
Function Diagram. We focus on seven fundamental function types: polynomial, sine, cosine, tangent, logarithmic, absolute value, and piece-wise polynomial functions. For each function type, we parameterize the equations with random variables, such as coefficients and constants within predefined range (e.g., and in y ar + b), which facilitates the generation of diverse function graphs. We also adopt the same Cartesian coordinate system employed for analytic geometry. Additionally, for specific caption extreme points and zero points of the functions, providing additional visual information that aids in the understanding and reasoning of these mathematical functions.
question-answering
samples, also plot key features like
4

---

Analytic Geometry Caption. We also employ GPT-4 to obtain two sets of language templates: the description of coordinates and attribute information for basic figures (e.g.. "The square with its base left corner at (1 features sides of [] in length") and the spatial relation for nearby figures (e.g., "On Published as a conference paper at ICLR 2025
Diagram:
Caption:
diameter, edga renoved. ractongle CBEF... Conrected adge restongle with semicircle outside using HI the diometer, and side HI eliminated. The lengths of AB, CB, and EB are each 7.0 units.
rectangle ABCD has semicircle inserbbe using
the coordinate system. logarithmic function passing through origin Additionally, there 4*)e)(5),
the outset, circle with center 3) and radius Consequently. rectongle originates left corner height 7), ... At (-4, there is ellipse dimensions: height and width
16"x/5
tongent to the logarthhmic curve at the paint (1. (1.4leg(5)). The zero of the logeritionic function at x-d, and the intersection point of the line with the y-exis is 10651665-4 45)g(()))
Figure 3: MAVIS-Caption Dataset. We showcase three diagram-captinn pairs of plane geometry, function, and analytic geometry in MAVIS-Caption Caption, generated by our developed data engine.
2.2 MAVIS-CAPTION
With our mathematical visual data engine, we first curate diagram-caption dataset, MAVIS-Caption as shown in Figure 3, aiming to benefit the diagram visual representations and cross crosl-modal alignment.
Data Overview. As presented in Table the Appendix, the MAVIS- SWVII-Cattion dataset comprises 588K diagram-caption pairs. This includes 299K for plane geometry, 77K for analytic geometry, and 212K for function. The average word length of the captions is 61 .48 words, reflecting their detailed descriptive nature. The overall vocabulary size is 149, indicating the diversity language expression. We adopt different strategies to generate captions for three types of diagrams. It is important to note that GPT-4 (OpenAI, 2023b) only utilized during the template creation stage; is not used at any point during the automatic caption generation process.
Plane Geometry Caption. We follow the iterative geometric generation process to develop regula- tions for an accurate and detailed caption. We first prompt GPT-4 to create three sets of language templates: the descriptive content for fundamental shapes (e.g., "A Triangle (] with two congruent sides /] and (/"), the phrases to denote specific attributes (e.g.. "Angle |] measures [] degrees") "), and the conjunction to link two adjacent shapes (e.g., "Attached to edge of shape (}. there is a /?"). Then, based on various generation scenarios, we fill and merge these templates to acquire a coherent description of the geometric figure.
Function Caption. As function diagrams typically showcase a single curve, we directly utilize GPT- 4 to generate templates describing various properties of functions, including expressions, domains, ranges, extreme points, and zero points. Each template then filled based on specific cases, such as "The expression of the function is --==3rr 3x.3 2x.² 2x 22 Within the range of x values [-3.0, 4.0), zero points occur at -0.83...
the bottom right of (}, there is [J"). The captions are then formulated by filling in the coordinates and selecting appropriate spatial relationship templates through coordinate comparison.
2.3 MAVIS-INSTRUCT
data, curate LAVIS-IIntruse of extensive problem-solvine data, which endows MLLMs with visual mathematical reasoning capabilities and serve as the basis for Direct Preference Optimization (DPO) (Rafailov et al., 2024), shown in Figure 5.
Besides the
5

---

Published as a conference paper at ICLR 2025
Disgrum:
1x/0
Question & Rationale:
Given the analytic Abato"x + b) and fallowing condmiass: cureve creas 6). What one
ABC is an isscceies triongle, with A equaliig acc CADE takes the form of parulielagra, AB Angle cBo spen 30 ct (39) Calculate the area clDE.
A tey compony wants te design toy that out along porabolic curve then furction stroight to have the form Find volues and c that make ((K) bath contiruous and differenticble.
In the given if ongle: hos measure of 35.0 degrees, what the messurs angle 27
track
A: 55 135 145 b. 155
The Figure shows that is pasitive. Y ot xcO is 6. thus (bs(b) : From the groph we aleo know when xtd2, yermiutte
Given triongle
on
D2 cROE, with ongle cBo 50* cE sin(301) = 42 * (33) x 1/2 (3)) equating to about 654.2.
thus Abst2"o
The
Since ongle orgle
b) 2 Upon aslving. equational give
10we1// (4)yright) 2
145 Tharafere, option selected.
((4)
Abol2"s
-3.
Data Engine
Data Engine
Manual Collection Augmented by GPT-4
Existing Datesets
Generated Problems
Generated Problems
Augmented by GPT-4
Dataset. We showcase the generated visual math problems from four sources within MAVIS-Instruce which contain detailed rationales and minimized textual redundancy.
Figure 4:
Data Overview. As illustrated in Table 4 of the Appendix, the MAVIS-Instruct dataset consists of a total of 834K visual math problems. Given that the proportion of analytic geometry problems is relatively small, we classify them with function problems for simplicity. Each problem in MAVIS- Instruct includes a CoT rationale providing step-by-step solutions, with an average answer length of 150 words. We have minimized textual redundancy in the questions, eliminating unnecessary contextual information, distracting conditions, and attributes readily observable from the diagrams. This reduction in text forces MILMs enhance their capability extract essential content from visual inputs. MAVIS-Instruch is assembled from four distinct sources to ensure broad coverage.
Data Engine Generated Problems. Within our data engine, manually craft rigorous regulations to produce visual math problems with accurate CoT annotations. Similar to caption generation, GPT API is not involved in the automatic synthesis process of questions, answers, and CoT rationales.
Plane Geometry Problems. We initially prompt GPT-4 to compile a comprehensive set of mathematical formulas applicable to each basic shape (c.g., Pythagorean theorem for right triangles and area formula for circles). Then, for a geometric diagram, we randomly select a known condition within shape as the final solution target, and systematically deduce backward to another condition, either within the same shape or an adjacent one, using randomly selected mathematical formula. This deduced condition then set as unknown, and we continue iterative backward deductions necessary. The final condition, along with any conditions in the last step, are presented as initial attributes in the question. The rationales be simply obtained by reversing this backward deduction process.
Function Problems. As the properties of functions are predetermined, we utilize GPT-4 to generate diverse reasoning templates. These templates facilitate the solving of one function property based on other provided properties, thereby ensuring the generation of high-qualing function rationales. The related function properties include analytical expression, function values, zeros, extremum points, monotonicity, derivatives, and integrals. To accurately reason these properties, the CoT annotation incorporates understanding of function types, solving the analytical expressions of equations, and interpreting function graphs
Data Engine Captions Annotated by GPT-4. Given the detailed captions and diagrams generated by our data engine, we prompt GPT-4V with these sufficient conditions to synthesis question- answering data and ensure its correctness. We first generate a new set of 17K that do not overlap with the previous MAVIS-Caption which avoids answer leakage within the detailed caption. Then, we prompt GPT-4V to generate 3 new problems with rationales, obtaining 51 K data in total from the diagram-caption pairs.
pairs

---

Published as a conference paper at ICLR 2025
Stage-1
Stage-2
Stage-3
Stage-4
Training CLIP-MATH
Aligning Diogram Langugge
Instruction Tuning
Preference Alignment with DPO
Figure 5: Four-stage Training Pipeline of MAVIS. With our curated MAVIS-Captin and MAVIS- Instruct, adopt four progressive stages for training mathematical visual specialist from seratch.
Manual Collection Augmented by GPT-4. To incorporate high-quality problems found in real- world contexts, we manually collect 4K math problems with diagrams from publicly available resources. Recognizing that these sources often lack detailed rationales and may contain redundant text, we initially utilize GPT-4V to annotate a detailed solving process and streamline the question text reduce redundancy. Subsequently, for each collected instance, we input the question, rationale, and diagram into GPT-4 and employ customized few-shot prompts to generate 20 new problems per original, comprising 15 multiple-choice questions and 5 free-form questions. This process contributes a total of 83K problems to the dataset.
Existing Datasets Augmented by GPT-4. Given existing well-organized geometric datasets, we can also leverage them to expand MAVIS- Referring to previous prompt designs, we augment the 8K training set from two dataset, Geometry-3K (Lu et al., 2021) and GeoQA+ (Chen et al., 2021b), into 80K visual problems with accompanying rationales, mapping each original problem to 10 new ones. Due to the scarcity of publicly available function data, we do not include function problems from this source.
MATHEMATICAL VISUAL TRAINING
With the curated datasets, we devise four-stage training pipeline for endowing MLLMs with mathematical visual capabilities. They respectively aim to mitigate the three deficiencies within existing MILMs, i.e., diagram visual encoding, diagram-language alignment, and mathematical reasoning skills in visual contexts.
3.1 STAGE 1: TRAINING CLIP-MATH
To enhance CLIP's (Radford et al., 2021) inadequate visual encoding of math diagrams, we utilize MAVIS-Caption to train specialized CLIP-Math encoder. Specifically, we fine-tune pre-trained CLIP-Base model following the conservative learning scheme. The math diagrams are fed into learnable vision encoder, while the corresponding captions are processed by the text encoder, which remains frozen to provide reliable supervision. Via contrastive training, the model learns to adapt from its original natural image domain to mathematical contexts, increasing its focus on essential visual elements within diagrams, demonstrated in Figure I (a). The optimized CLIP-Matee encoder now delivers more precise and robust representations of math diagrams, establishing a solid foundation for the subsequent visual interpretation of LLMs.
3.2 STAGE ALIGNING DIAGRAM-LANGUASE
After acquiring the CLIP-Math encoder, we further integrate it with LLMs using MAVIS- to boost cross-modal alignment between math diagrams and language embedding space. Using a simple two-layer MLP as the projection layer, transform the visual encodings from CLIP-Math and prepend them as a prefix to the LLM input. This process, guided by the diagram captioning task, enables the LLM accurately recognize mathematical components and spatial arrangements. With the diagram-Languag, alignment, LLMs are equipped with the interpretation capability in math diagrams, serving initial step toward deeper mathematical reasoning. In this stage, freeze the CLIP-Math and train the projection layer along with the LoRA-bai -bssd (Hu et al., 2021) LLM.
3.3 STAGE 3: INSTRUCTION TUNING
On top of that, we leverage MAVIS-Instruct solving capabilities in visual mathematics. The detailed rationales within each problem's 's solution
Instruct to endow MLLMs with CoT reasoning and problem-
7

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation on Verse's testmini Set with Six Problem Versions. 'CoT-E" and 'Ace' denote the scores of CoT evaluation strategy and the scores of direct 'true or false' accuracy, respec- tively. denotes previous mathematical visual specialists. The highest scores for closed-source and open-source MLLMs are marked red and blue, respectively.
Text
Vision Intensive
Visine
Model
Dominant
Lite
Daminaar
Only
Size
CoTEE
Ace
CoTEE
Aec
CoI-E
Aec
CoT-E
Acc
CoTE
Ace
CoT-B
Ace
Roselww
Random Chance
Human
64.9
71.2
70.9
61.4
68.3
66.7
11MG
CharGPT
$1.3
33.3
Closed sunrrce MLLMy
Owen- VL-Plas
21.3
11.8
26.0
15.7
18.5
9.0
13.0
21.8
10.0
Gemini-Pra
33.3
Qwen
33.6
24.1
GPT-4V
54.4
39.4
63.1
54.7
56.6
41,4
51.4
34,9
30.8
34.4
50.3
31.6
Open nource
ALaMA-Adaptes-22
78
5.8
7.8
6.2
6.3
6.2
6.1
4.2
4.4
6.1
Imagelind LLM
7B
10.0
9.2
13.2
114
11.5
11.3
9.8
8.9
11.8
11.2
3.5
3.4
mPLUGG Oef2
76
10.3
11.6
6.6
11.1
6.3
8.0
4.9
MinGPT-v2
11.1
11aVA-155
17.1
88
12.6
7.4
7.4
90
SPHINX-POO Plus
13B
14.0
12:2
16.3
13.9
12.8
11.5
12.9
11.6
14.7
13.5
132
104
G-LLaVA"
76
15.7
16.6
22.2
20,9
20.4
20.7
16.5
17.2
12.7
14.6
6.6
9A
LLVA--NoXT
8B
17.2
13.6
21.6
19.4
19.7
15.2
17.6
16.8
14.9
15.2
12.1
11.3
ShareGPP4Y
SPHINX-MoE
Math-LLaVA"
13B
24.1
19.0
34.2
21.2
22.7
21.1
20.2
20.3
17.6
22.3
16.4
Interal M.XC2.
7R
25.9
16.5
36.9
22.3
28.3
17.0
20.1
15.7
24.4
16.4
19.8
11.0
HLAVA-NeXT
22.6
MAVIS-7B w/o DPO"
76
33.7
27.5
42.5
41.4
36.3
29.1
33.3
27.4
29.3
24.9
27.1
14.6
MAVIS-BB
41.6
27.9
provide high-quality reasoning guidance for MLLMs, significantly enhancing their step-by-step CoT process. Furthermore, as we have minimized the redundancy within question texts during the construction process, such text-lite problem formats, referring to Math Verse (Zhang et al., 2024b), facilitate MLLMs to capture more essential information from the visual embeddings for problem- solving, rather than relying shortcuts to only process the textual content. During this stage, unfreeze both the projection layer and the LoRA-based (Hu et al., 2021) L.LM to perform thorough dnstruction-folloiin tuning.
3.4 STAGE 4: PREFERENCE ALIGNMENT WITH DPO
After the instruction tuning phase, the resulting model gains the capability for CoT reasoning on visual math problems. However, it may still produce inaccurate intermediate steps due to insufficient supervision for generating the best reasoning path. To address this, we further apply CoT preference alignment using the DPO (Rafailov et al., 2024) algorithm further enhance the model's reasoning performance. Specifically, we adopt the instruction-tune on the 582K problems generated by data engine within MAVIS-Instruet. Then, we filter the incorrect outputs (88K data) based on the final answer as the negative reasoning samples in DPO, and directly utilize the annotated CoT process as the positive samples. We only unfreeze the LoRA parameters for DPO training, and finally obtain our mathematical specialist, MAVIS-7B.
model to first infer CoT reasoning process
EXPERIMENT
We first detail our experimental settings in Section 4.1, and then discuss the quantitative on different benchmarks and qualitative examples in Sections 4.2 and 4.3, respectively. Please refer to the Appendix for more data details and ablation studies.
8

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation on Six Mathematical Benchmarks. 'MMMU-Math' denotes the math problems within the test set of MMMU. 'GPS', 'ALG', and 'GEO' denote geometry problem solving, algebraic, and geometry in Math Vista's testmini set. 'SI', 'S2', and 'S3' denote different problem steps in We-Math's 's tesimini set. denotes previous mathematical visual specialists. The highest scores for closed and open-source MLLMs marked in red and blue, respectively.
Model
LLM Size
GooQA
FunctioOAA
MMMU Math
Vistin Vision
MathVist ALo
Math
cBs
32
Dovelincs
Random Chance
17.1
21.6
7.2
34.1
25.8
22.7
Fummnn
92.3
84.2
68.6
48.4
90.9
51.4
CharGPT GPE4
13.1
31.7
32.2
Clone-snwree MLLMs
Qwen VL-Plis
10.7
38.5
39.1
39.3
Ques- Ques-VE-Max VL-Max
36,3
15.6
40,8
30.3
20.5
GPT-VV
22.8
50.5
51.0
65.5
38.2
Open source MLLMs
TLaMA-Adiper V2
7B
18.1
10.5
23.0
8.2
25.5
26
24.5
mPLUG Owl2
7B
15.7
bluc29.0
18.8
8,6
blue
blue27.7
blue14.2
UniMath
50.0
ULAVA-15
13B
20.3
blx3339
24.0
11.2
b6.e
bluc38.5
6lue16.7
ShareGPT4
11,9
SPHINX-MoE
14.2
GILLAVA"
138
67.0
blue24
blue27
13
bls36.11
32.4
30.1
32.7
Mub--1aaVA
ble623
blu33.77
blue 30.1
Muuel 14.5
57.7
56
blue37.5
bluc10.5
Mue33.4
InsemL XC2
7B
blae56.4
blue8.77
63.0
56.6
62.3
47.0
33.1
33.0
LLaVA-NeXT
1108
53.7
36.9
31.5
MAVIS-TB w/s DPO
7B
66.7
40,3
29.2
18.6
63.2
58.3
63.0
56.9
37.1
33.2
MAVIS-7B"
7B
50.01
19.2
63.2
57.2
37.9
34.6
4.1 EXPERIMENTAL SETTINGS
Implementation Details. We adopt CLIP ViT-L (Radford et al., 2021) as the pre-trained model to fine-tune our CLIP Math, and utilize Mammoth2-7B (Yue et al., 2024) the base LLM to construct MAVIS-7B. In the first stage, we fine-tune the CLIP for 10 epochs with a batch size 16 and an initial learning rate 2e 26-6, In the second stage, train the diagram- languagg alignment for epoch with batch size 32 and an initial learning rate 2e - and adopt LoRA (Hu et al., 2021) with a rank 128. In the third and fourth stages, we adopt the same training settings as the second one.
Evaluation Schemes. We evaluate our model MAVIS- on several popular mathematical bench- marks, MathVerse Verse (Zhang et al., 2024b), GeoQA (Chen al., 2021c), FunctionQA (function problems in Math Vista (Lu et al., 2023)), MMMU-Math (the math problems in MMMU (Yue et al., 2023a)), Vision (Wang et al., 2024b), three mathematical categories in Math Math and We-Math (Qiao et al., 2024). We compare variety of existing MLLMs, including two mathematical visual special- ist (Gao al., 2023a; Shi et al., 2024), two L.MSs (OpenAI, 2023a;b), and other general MILMs (Bai et al., 2023b; Gao et al., 2023b; Ye et al., 2023b; Liu et al., 2023a; Chen et al., 2023b; Gao et al., 2024; Dong et al., 2024; Liu et al., 2024a; Chen et al., 2023a; Gao al., 2024).
4.2 QUANTITATIVE PERFORMANCE
As shown in Table for the Math Verse benchmark, MAVIS-7B achieves the best overall scores in both CoT evaluation and accuracy among open-source MILMS with only a 7B model size, and consistently surpasses the recond-best method on different problem versions. Specifically, our model surpasses the powerful Intern_M-XComposee XComposer2 (7B) (Dong et al., 2024) by +9.3% and ShareGPT4V (13B) (Chen et al., 2023b) by +17.8% CoT evaluation scores. Compared to other mathematical visual specialist, i.e., VA (7B) (Gao et al., 2023a) and the concurrent Math-LLaVA (13B) (Shi et al., 2024), MAVIS-7B exhibits superior problem-solving capabilities with higher CoT evaluation scores of +19. 5%% and +11. 1.1 1.1% respectively. In addition, our model is also ad advantageous to the most powerful open- source MLLM series, LLaVA-NeXT (Li et al., 2024a), from 8B to 110B model sizes, demonstrating the math math-specific proficiency of MAVIS-7B. Note that, the improvement brought by DPO (our fourth-stage training) is more apparent in CoT evaluation compared to the accuracy scores, indicating that the preference alignment learning can effectively boost the CoT reasoning capabilities.

---

Published as a conference paper at ICLR 2025
Recegaize tringgle inaege sagan smallest possible loneth hopare when point P the line segaecat ABB The smallest possibl lergth of AP the length (-A, which is 13 enios, Therefire, the corrot axswrr is: C:
To determine the y-value aed based the dingees, observe the point where the lise carve interseets the yeaxts, From the graph, of smi, the yoralte Ini B.
Questien:
is the diagram iiturrating the secnesio. point situcted onsil of Lite segnect tangent to cirds potrt sovea clong circle
Question:
point Since Ofl 4 12. sqrid2 122) Dherefore, OP and AP -3-5 Therefore, the answer Is.
Whar toe youtue at based on the diagram?
From the dlinguun, it clear caat at - 0, the yoralne is approsims auallyy 2.5, So the unover A..
of
Choleos
smalloi posilic
A.25
D.0
Choices: A:S B.3 C.13 D:
To caleatate the lotal area of the slape ACDEF, we cen break it Bewn into simpler doppes and calesiate their areas seperatcly d using
Sirce is AlS is right triangle, and Bs the topstenass BE ABBhaco45y shagnty - 12.73, BF squiB112 nang 22100000 foae(SS)) So p a p- A-BBB-BBBB 12731
height, 112e yees
m
1:201355,00 53 37 93, Tocal Aroo - -2111.00
AB 8.Dadd Czgle ia coleatated oppenik eide
Qucetive:
AB a o.0 Becasee ABCD is di ades the same lesgth Henes, CB - Al - Cl mosems 5.0. CBC seecclo inaglle BE is also 9.0, Angle F-52 BP EBiaaeS3l
How would calouiate the sold ane of the stops ACD!
110t111 0) neaty equal beight DR3 bil heightr2 4835..
Ancle F- depros All
Classess
What the inderr EBF9 Plaxie unseal queilic direcly.
B: D:
trinngle
7.0 Hypotenuse I1A, E8-B3+
Figure 6: Problem-solving Comparison of MAVIS-7B and GPT-4V.
Table 2 showcases the performance comparison on six other mathematical benchmarks, where our model still attains remarkable performance among other MIL.Ms. detail, MAVIS-7B outperforms the closed-source Qwen-V VL-Max (Bai et al., 2023a) by +6.1 % in MMMU-Math, +3.6% in Math Vi- sion, and around + +10% in three subsets of We-Math. Our model even exceeds GPT-4V (OpenAI, 2023b) in the three mathematical categories of MathVista indicating our problem-solving and rea- soning proficiency. We also observe that, the enhancement from DPO increases from 'S1' to 'S3' of We-Math, which well demonstrates its benefit math problems with more intricate reasoning steps.
4.3 QUALITATIVE ANALYSIS
In Figure 6, we compare the mathematical problem-solving examples between MAVIS-7B and GPT- 4V (OpenAI, 2023c). As presented, our model not only showcases better accuracy in understanding the geometric elements, function curves, and coordinate axes in mathematical diagrams, but also performs higher-quality step-by-step reasoning process for formula substitution and numerical calculation. This demonstrates the effectiveness of our four-stage training pipeline and automatic data engine for enhanced diagram understanding and CoT reasoning.
CONCLUSION
In this paper, we propose MAVIS, the first mathematical visual instruction tuning paradigm for MLLMs. We first introduce two high-quality datasets by a delicate data engine, MAVIS-Capie and
containing large-scale diagram-language and problem-solving data. Then, we customize a three-stage training framework to progressively train the math-specific vision encoder, the diagram-language alignment, and the mathematical reasoning capabilities of MLLMs. The obtained specialist model, MAVIS-7B achieves superior performance across different mathematical visual benchmarks, demonstrating the potential to serve new standard for future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This project is funded part by National Key R&D Program of China Project 20222D0161 100, by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s 's InnoHK, by NSFC-RGC Project N_CUHK498/24, Hongsheng Li is a PI of CPII under the InnoHK.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2338.. 01390, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv, abs/2308. 12966. 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. /2966, 2023b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in neural information processing systems, pp. 1877-1901, 2020.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. GeoQA: A geometric question answering benchmark towards multimodal numerical reason- ing. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 513-523, Online, August 2021a. Association for Computational Linguistics. doi: 10. 18.13/53/ 1/2021 findings-acl.466 URL https://aclantti
wwwpor///00/ f findings- acl. 4..
Jiaqi Chen, Jianheng Tang, Jinghui Qin. Xiaodan Liang, Lingbo Liu, Eric P. Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. ArXiv, abs/2105. 14517, 2021b. URL /ppi. semant Lescholar. org/ CorpusID: 235253782.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2105. 2tt.:::2115..5511.. 14517, 2021c.
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Uni- geo: Unifying geometry logical reasoning via reformulating mathematical expression. ArXiv, abs/2212.02746, 2022.
Jun Chen, Deyao Zhul Xiaoqian Shenl Xiang Li, Zechun Liu2 Pengchuan Zhang, Raghuraman Krishnamoorthi2 Vikas Chandra2 Yunyang Xiong, and Mohamed Elhoseiny, Minigpt-v2: Large language model as unified interface for lision-language multi-task learning. arXiv preprint 10.0472330.09448, 2023a.
Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. ArXiv, abs/2311.12795 2023b. URL https: /api......////.../ wesemnntiscchllle org/ Corpus ID: 265308687.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404 1682/, 2024.
11

---

Published as a conference paper at ICLR 2025
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. blog/2023-00 00----0033--000 -0-vi cura/, March 2023.
httsss//mmmsss.mmm org/
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang. Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm- xcomposer2: Mastering free-form text-image composition and comprehension in vision- asion-language large model. arXiv preprint (rXiv:2401.1642) 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal iims in video analysis. arXiv preprint arXhv:2405.21075. 2024.
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal 2023a.
large language model. arXiv preprint 2rXiv:2312.113
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient 2023b.
visual instruction model. arXiv preprint arXiv:2304.1500
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang. Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05933 2024.
Google Gemini Team. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:23/2.11805, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, ct al. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.1 0..3309.74555 17452, 2023.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with molti-modality for 3d understanding, generation, and instruction following, arXiv preprint 27X.v22309.00615 2023.
Ziyu Guo Renrui Zhang*#, Xiangyang Zhu Chengzhuo Tong, Peng Gao, Chunyuan Li, and Pheng-An Annn Heng. Sam2point: Segment any 3d as videos in zero-shot and promptable manners. arXiv preprint arXiv:2408. 16766668,,,,8 2024.
Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng- Ann Heng. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint arXiv:2501. 13926, 2025.
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03905 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1rXiv:2106.09683 2021.
Yueru Jia, Jiaming Liu, Sixiang Chen, Chenyang Gu, Zhilue Wang, Longzan Luo, Lily Lee, Pengwei Wang, Zhongyuan Wang, Renrui Zhang, et al. Lift3d foundation policy: Lifting 2d large-scale pretrained models for robust 3d robotic manipulation. arXiv preprint arXiv:2411.1862) 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel. Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril. Thomas Wang, Timothée Lacroix, and William El Sayed. 2024.
Mixtral of experts. Arxiv
12

---

Published as a conference paper at ICLR 2025
Dongzii Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang. Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXhv:2502.09621 2025.
Bu Jin, Yupeng Zheng, Pengfei Li, Weize Li, Yuhang Zheng, Sujie Hu, Xinyu Liu, Jinwei Zhu, Zhijie Yan, Haiyang Sun, al. Tod3cap: Towards 3d dense captioning in outdoor scenes. arXiv preprint arXiv:2403. 19589, 2024.
Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint 1rX1v:2312.12241, 2023.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger 1lms supercharge multimodal ca- pabilities in the wild, May 2024a. URL https://llava-aaaaaaaa
wtliavvavl.githu.... 1o/blog/
Feng Li. Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next: Tackling multi-image, video, and 3d in large multimodal models, June 2024b. URL
ito.b.oobbll...... ho/blog/
2024-06-15 11116-111aaeeeee ineettinttereavee.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. llatt-eext-nnterllave:: Tackling multi-image, video, and 3d large multimodal models, 2024c. URL https: org/ abs/2407. 07895.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-
training for unified vision-languan understanding and generation. In International Conference Machine Learning, pp. 12888-12900. PMLR, 2022.
KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06335 2023a.
Xiaoqi Li, Mingxu Zhang, Yiran Geng, Haoran Geng, Yuxing Long, Yan Shen, Renrui Zhang, Jiaming Liu, and Hao Dong. Manipllm: Embodied multimodal large language model for object-centric robotic manipulation. arXiv preprint arXiv:2312.16211. 2023b.
Zhenwen Liang, Tianyu Yang, Jipeng Zhang, and Xiangliang Zhang. Unimath: A foundational and multimodal mathematical reasoner. In EMNLP, 2023.
Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shang- hang Zhang, and Hongsheng Li. arwwaan-uudderstand: Leveraging visual prompts to enable milms to comprehend what you want, 2025. URL https: /ttp///aaavvvvvvv//////eeeeaaav. org/ /2/b////40... 2027 1.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Kegin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint 1rXiv:2311.07575. 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Leava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https: //1lava-vl, git.tithtbb..... io.bb.o//bbb to/blll///////ooo 2200002/22222222
30- llava-nexte
Jaaming Liu, Chenxuan Li, Guanqun Wang, Lily Lee, Kaichen Zhou, Sixiang Chen, Chuyan Xiong, Jiaxin Ge, Renrui Zhang, and Shanghang Zhang. Self-corrected multimodal large language model for end-to-end robot manipulation. arXiv preprint arXiv:2405.1
1154/055774111 /7418, 2024b.
13

---

Published as a conference paper at ICLR 2025
Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation. NeurlPS 2024, 2024c.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chur Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprini trXiv:2105.04/66 2021.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun yue Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. ArXiv, abs/2310.02255, 2023.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021.
OpenAl. Chatgpt. https: /ttps://chat..e openai com, 2023a.
OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023b.
OpenAl. GPT-4V(ision) system card, 2023c. URL http://ppeee..ccmeeeeeeerchaaaca
gpt-4v-system- card.
Tianshuo Peng, Mingsheng Li, Hongbin Zhou, Renqiu Xia, Renrui Zhang, Lei Bai, Song Mao, Bin Wang, Conghui He, Aojun Zhou, al. Chimera: Improving generalist model with domain-specific experts. arXiv preprint arXiv:2412.0598, 2024.
Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue. Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407. 200.:2007.0/244 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021. URL https: / /ppi. semant Lescholar. porg/orrpssD:
231591445.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024. URL
alss/ 18290.
Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng. Lidong Bing, and Roy Ka-Wei Lee, Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.1777 17294, 2024.
Chang Shu, Baian Chen, Fangyu Liu, Zihao Fu, Ehsan Shareghi, and Nigel Collier. Visual med-alpaca: A crrameter-efficien biomedical IIm with visual capabilities, 2023.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617 2023.
Yixuan Su, Tian Lan, Huayang Li. Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to cestruction-follow them all. arXiv preprint arXiv:2305. 2023.
Yiwen Tang, Zoey Guo, Zhuhao Wang, Ray Zhang, Qizhi Chen, Junli Liu, Delin Qu, Zhigang Wang, Dong Wang. Xuelong Li, et al. Exploring the potential of encoder-free architectures in 3d Imms. arXiv preprint 2502.000000 2025.
InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397. 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09288, 2023b.
Guanqun Wang, Xinyu Wei, Jiaming Liu, Ray Zhang, Yichi Zhang, Kevin Zhang, Maurice Chong, and Shanghang Zhang. Mr-mllm: Mutual reinforcement of multimodal comprehension and vision perception, 2024a. URL https: / /arxiviivv..///aa//aaaa/a/aaavva org/ abs///////2// 15768.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint 4rXiv:2402.14804, 2024b.
Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang. Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration LLMs for enhanced mathematical reasoning. In The Twelfth International Conference on Learning Representations, 2024c. URL https:tp////. hwtp///ppeeeeeeeeeeeeeeeeeeeeeep n forum?id 2
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Cfain-of-thouuht prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models understand point clouds. arXiv preprint arXiv:2308. 16911, 2023.
Senqiao Yang. Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312. 140.4 14074, 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hchong Chen, Junfeng Tian, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023a.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou, mplug-ow12: Revolutionizing multi- large language model with modality collaboration, 2023b.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang. James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12288 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16500 2023a.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023b.
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint Xrxiv:2405. 03548, 2024.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. LLaMA-adapter Efficient fine-tuning of large language models with zero-initio irroiinitialized attention. In The Tweifth International Conference on Learning Representations, 2024a. URL https: //pppe/eevvveeeeeeeeeeeeeeeeeeee ew net forum? ditp:mmmmmdddddiieeaaooo XAHN2
15

---

Published as a conference paper at ICLR 2025
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi- Ilm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 14624, 2024b.
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verificattion. arXiv preprint arXiv:2308.07921, 2023.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. RELATED WORK
Visual Instruction Tuning. The advancement of large language models (LLMs) (Brown et al., 2020; Jiang et al., 2024; Touvron et al., 2023b; Chiang et al., 2023) with instruction tuning has significantly enhanced zero-shot capabilities across a range of tasks. Drawing inspiration from this, LLaMA- Adapter series (Zhang et al., 2024a; Gao et al., 2023b; Han et al., 2023) propose a zero-initialized attention mechanism to align frozen vision encoders (Radford et al., 2021) with LLaMA (Touvron et al., 2023a) for multi-modal learning. VL series (Liu et al., 2023b;a) employ linear projector for vision-langual
alignment, establishing visual instruction tuning as a standard training approach in the multi-modal field. Flamingo (Alayrac et al., 2022) and OpenFlamingo (Awadalla al., 2023) have honed visual representation by integrating a cross-attention resampler with vision encoders. SPHINX series (Gao et al., 2024; Lin et al., 2023; 2025) and MR-MLLM (Wang et al., 2024a) utilize a blend of visual encoders to make the LLM cognizant of various image aspects. InternVL. series (Chen et al., 2024; Dong et al.. 2024; Team, 2023) employ a large vision encoder and Q- Former (Li et al., 2022) to incorporate high-quality visual information through a multi-stage training methodology. LLaVA-NexT (Liu et al., 2024a; Li et al., 2024a;b) further introduces the 'AnyRes' technique to manage images at any given resolution, and LLaVA-Nex'S NexT-Interleave (Li et al., 2024c) extends the scope widely to interleave multi-image settings. There are also recent efforts to apply visual instruction tuning to 3D (Guo et al., 2023; Xu et al., 2023; Guo et al., 2024; Tang et al., 2025), video (Li et al., 2023a; Fu et al., 2024), reasoning Guo et al. (2025); Jiang et al. (2025); Peng et al. (2024), and robotics Jia et al. (2024); Liu et al. (2024c) scenarios. Despite the impressive strides made in both model capability and training efficiency by multi-modal large language models (MLLMs) through visual instruction tuning, there is currently no MLLM specifically designed for mathematical woblemm-solving nor a substantial dataset available for such purposes in the open-source community, In this paper, we mitigate the issue by proposing MAVIS with high-quality mathematical visual datasets and training paradigms.
Mathematics in Large Models. Recent research has predominantly concentrated on text-only mathematical groblem-solving using LLMs. MAmmo'TH (Yue et al., 2023b; 2024) have compiled ex- tensive collections of mathematical problems, training LLMs using the reasoning processes described in solutions. MetaMATH (Yu et al., 2023) has expanded upon this by rewriting existing problems to create a larger dataset. MathCoder (Wang et al., 2024c) and ToRA (Gou et al., 2023) introduced tools agent approach, employing Python code and symbolic resolvers during the training phase, significantly outperforming traditional models that rely on text-only mathematical reasoning. How- ever, in the multi-modal field, despite the introduction of several datasets such as Geometry3K (Lu et al., 2021). GeoQA (Chen et al., 2021b), UniGeo (Chen et al., 2022), UniMath (Liang et al., 2023), and Geom Verse (Kazemi et al., 2023), aiming at enhancing the performance of MILMs in solving graphical mathematical problems, these datasets are quite limited in scale and domain. Based on these datasets, G-LLaVA (Gao al., 2023a) has developed superior capabilities for understanding graphical geometries but struggles with mathematical problems in other domains. The comprehensive benchmark Vathe (Zhang et al., 2024b) has also highlighted the existing MLLMs' unsatisfactory capacity for encoding visual diagrams in diverse mathematical domains. Therefore, there is pressing need for the development of more robust encoders for mathematical images and the tuning of MIL.Ms with mathematical visual instructions, for which we propose MAVIS to address the challenges.
A.2 HUMAN EVALUATION OF MAVIS-INSTRUE
To assess the dataset's coverage, validity, and quality, human verification is employed. The creation process of our MAVIS-Inntruet dataset can be broadly categorized into two approaches:
This method leverages GPT-4 to generate new problems (including ques- tions, rationales, and answers) based on existing problems with diagrams. While this approach produces fluent, human-like sentences, it may be influenced by the inherent capabilities and occasional instability of GPT-4V.
GPT-generated:
Data Engine: As the main source of our mathematical visual data, this method utilizes the custom automatic data engine to generate new problems (including diagrams, questions,
17

---

Published as a conference paper at ICLR 2025
Duta Eugies
Scae
Queetioo
Rattonale
Anowa
EST
103
250
Comenton
Apprssedc
Castecterts
Plurney
Cuanertens
Flutacy
Cormudees
Appuarenna
Correcten
Flamary
Correttiis
Hlvetryy
Corrobin
Figure 7: Human Evaluation Results on 200 randomly sampled problems in MAVIS- Instruct, 100 GPT-generated and 100 Data Engine. We set three levels (1, 2, and 3) for each metric, and report average scores.
Figure 8: Human Evaluation Statistics 200 randomly sampled problems MAVIS- Instruct, 100 GPT-generated and 100 Data Engine. We count the numbers of three score levels (1, 2, and 3) for each metric.
rationales, and answers), without relying on GPT models. It guarantees 100% correctness due to the use of rigorous templates, though it may occasionally exhibit rigid expressions.
Specifically, we evaluate four aspects(Diagram, Question, Rationale and Answer) of each problem using seven metrics. Each metric scored on a scale of 3, where 1 denotes poor. denotes moderate, and denotes good. The human evaluation results are shown in Figure 7 and score statistics are shown in Figure In addition, we also showcase some specific examples in Figure and Figure 10. We analyze each aspect follows:
The diagrams in GPT-generated problems are directly collected from existing sources with rigorous human filtering, ensuring high quality, resulting in scores close In contrast, for rule- based problems, the diagrams are drawn accurately using Python code driven by our data engine, which guarantees correctness. However, these diagrams may lack alignment with human aesthetic preferences, as indicated by 3% of them receiving an appearance score
Diagram:
Question: Regarding the questions, both GPT-generated and rule-based problems display a high degree of accuracy in aligning with the diagram elements. This is attributed to the well-crafted prompts used with GPT-4 and the meticulous template design of the data engine. Nevertheless, rule-based questions may occasionally exhibit minor fluency issues, as they lack human refinement.
Rationale: In terms of the rationales, instances feature a precise and detailed chain-of- thought (CoT) reasoning process. However, in a few cases (3% receiving an accuracy score of 1), some GPT-generated rationales contain minor reasoning or calculation errors, which are inherent to GPT-4's limitations in problem-solving. These eπrоrs usually affect only one two steps and do not compromise the overall logic. Conversely, the rule-based rationales are highly accurate due to the carefully designed data engine, although there is still room for improvement in language fluency.
The answers in both methods achieve high correctness scores. For GPT-generated problems, we prompt GPT-4 to identify known condition from the original problems as the answer. Similarly, for rule-based problems, we randomly select known attribute from the generated diagrams to serve as the answer.
Answer:
Overall, the randomly sampled instances show that our dataset exhibits good question quality and answer accuracy.
18

---

Published as a conference paper at ICLR 2025
GPT-generated
Date Engine
Date Engine
Date Engine
Appearance Score: 3
Appearance Score: 3 A Apperracee Score: 3
A Apeurance Score: 1
Figure Diagram Examples in MAVIS- The first three diagrams showcase superior correctness and appearance, while small portion of Data Engine generated diagrams (3%) are not aligned with human preference, e.g., the fourth diagram.
Q: If AB:3, BC:4, and DE:5, then which of the following equals DF?
Q:
relation the tuble. What the rarge of the relation? Enter the values seporated by commas.
Q: ABC is an iassceles triangle CBBE is a parallelogram. Can calculate the area the entire CBDE?
Q: ABCD square. EDF is isosceles triangle which ED DF equal. measurement shape EDF?
Ri.. triangle ABE are equal corresponding ongles tnlongle ACF, making
triangles similar.. we can the fallowing proportion: AB/AC AE/AP: Given that
R: find range the relation, identify all the y- values the table.
R: Given that AB is 42, ABC being isosceles triangle that BC also 42, The
length Since ABCD is square with all sides equal, Ab also 1. Given that angle AED 60
range 3,2,711
parallelogram with angle cBb 30 CB
have DE equals /(3) (33/3, Becouse EDF is an isosceles triangle, DP is also /33/3.
AC=AB-BC=7,
As mapping, the onswer is Range 2, 1.
integer
CE smn 42 56*((2) 1/2 1176*/(2), equating about 1663.12 12.
becomes: 3/7 DE/DF: Given that DE 5, we solve for DF: DF 35/3.
GPT Generated
GPT Generated
Data Engine
Data Engine
Figure 10: Accurate Rationale Examples in MAVIS-Instruct. Most GPT-generated and Data
Engine-generan rationales ensure correctness.
19

---

Published as a conference paper at ICLR 2025
Table Statisties of MAVIS-Caption.
Table Subject Distribution of Inntsnn
Statistic
Number
Statistic
Number
Total Captions
Total questions
834K
Total number
Multiplc-choice questions
615K (62.4%)
Average length (words)
Free-form questions
218K (37.6%)
Average length (characters)
339.68
Data Engine Generated Problems
582K
Vocahulary size
418
Geometry questions
466K (80.0%)
Function questions
116K (20.0%)
Plane Geometry
Total number
299K (50.9%)
Data Engine Captions Aunotated by GPT-4
51K
Average length (words)
69.77
Geometry questions
30K (58.8%)
Average length (characters)
385.85
Function questions
21K (41.2%)
Vocahulary size
195
Manual Collection Augmented by GPT-4
83K
Analytic Geometry
-Geometry questions
72K (86.5%)
Total number
77K (13.1%)
Function questions
IKK (13.5%)
Average length (words)
39.64
Existing Dutasets Augmented by GPT-4
118K
Average length (characters)
210.10
Geometry questions
H18K (100.0%)
Vocabulary size
158
Function questions
(0%)
Function
Number of unique images
611K (73,3%)
Total number
212K (36.0%)
Number of unique questions
804K (96.5%)
Average length (words)
61.48
Number unique answers
675K (81. .0%)
Average length (characters)
321.46
Average question length
44.60
VVocauulayy size
149
Average answer length
62.82
A.3 ABLATION STUDY
A.3.1 MAVIS-CAPTION
To validate the enhancement of Math-CLIP' diagram perception capability, sampled 100 validation diagram-caption pairs and computed their cosine similarity using both CLIP and Math- CLIP. The results, shown in Table 5, indicate that Math-CLIP encodes features. Additionally, the attention visualization Figure ((a) of the main paper further demonstrates that Math-CLIP captures mathematical visual elements within diagrams more effectively, highlighting the efficacy of MAVIS-Caption.
discriminative diagram
To validate the role of MAVIS-Caption in second-stage training, we present both quantitative and qualitative results for diagram captioning on the same 100 validation pairs in the first column of Table The use of MLAVS--Cattio significantly enhances the diagram understanding capability. This shows that MAVIS-Caption helps the LLM generate accurate captions from diagrams, improving its ability comprehend each visual token from Math-CL.II and align visual elements with textual descriptions. We also evaluated MAVIS's performance on Math Verse without second-stage training, as shown in the second column of Table Without MAVIS- quality of MAVIS-7B is somewhat compromised. This suggests that training the model in diagram captioning improves its mathematical expression capability, enabling it to produce language expressions that align with mathematical concepts. This foundational skill supports the generation of subsequent CoT reasoning steps.
training, the CoT reasoning
Table 5: Diagram Perception Enhancement by Math-CLIP. using MAVIS-Caption in the first stage. We calculate the average cosine similarity among 100 validation diagram-caption pairs.
Vision Encoder
Matched Pair
Unmatched Pair $
CLIP
0.22
0.24
Math-CLIP
0.83
0.17
A.3.2 MAVIS-INSTRUCT
Redundant Text When curating questions for HAAVIS-Instrue we minimize the redundant content within the question texts, which refers to the directly observable content in the diagram, e.g., the
20

---

Published as a conference paper at ICLR 2025
Table 6: Diagram Understanding Enhancement and Mathematical Expression Enhancement in LLM using MAVIS-Caption in the second Stage. We compare the METEOR and CIDEr scores for diagram captioning on 100 validation samples, as well the accuracy and CoT evaluation results on Math Verse, both with and without the MAVIS-Caption training.
Dlagram-Captinn Pairs
MathVerss
Training Data
METEOR
CIDEr
Acc (%)
CoT-E (%)
MAAIIS-Caption w/o MAVIS- Cappion
23.7
161.3
28.4
35.2
14.0
69.4
25.6
32.8
presence of shapes or intersection points of functions. Such information is repetitive to visual components, and may assist MILMS in bypassing the process of diagram interpretation, thereby our data enforces
harming their related skills. By mostly avoiding redundant texts in MAVIS- MLLMs to learn stronger diagram interpretation capabilities. In Table 7, we add redundant texts (diagram captions) to the Data Engine Generated Problems for training, leading to expected performance drop.
For each instance in MAVIS-Instruct incorporate detailed rationales for problem-solving either generated by GPT-4 or our rule-based data engine. In Table 8, we remove all intermediate rationales of each problem in MAVIS-Instruct, and train the model to directly output the final answer. As shown, both the CoT evaluation and accuracy scores are degraded. This demonstrates the significance of our rationale annotations, which effectively improves the CoT reasoning capabilities of MLLMs.
CoT Rationales
Table Reasoning Capability Enhancement for MLLM, using the third stage.
Table 7: Diagram Interpretation Enhancement for MILM, using MAVIS-Instruct the third stage. We compare the results by adding redundant texts (diagram captions) the Data Engine Generated Problems within MAVIS-Instruct.
Math Verse
Training Data
MAVIS-Instruct
Math Verse
GcoQA
FunctionQA
Acc
CoT-E
w/o Redundant Texts
28.4
68.3
50.0
Rationales
28.4
35.2
w Redundant Texts
26.5
66.5
48.4
w/o Rationales
25.2
26.6
A.3.3 COMPARED TO GENERAL VISUAL INSTRUCTION DATA
Since Mammoth-2 is highly capable LLM for mathematical tasks, one possible question is whether simply integrating vision encoder into Mammoth-2 and training it with conventional visual in- struction tuning data would suffice for effectively solving visual-based mathematical problems. To compare MAVIS data with other visual instruction tuning datasets and investigate the specific ben- efits of MAVIS data in Mammoth-2 (7B), we conduct an ablation study. We utilize the data from LLaVA-NeXT (558K for pre-training and 760K for fine-tuning) and compare with our MAVIS data (558K MAVIS- for pre-training and 834K MAVIS-Instruct for fine-tuning). Performance is evaluated using the accuracy metric on Math Verse, excluding the DPO training stage for fairness.
Table 9: Ablation study results for comparison between MAVIS Data and other visual instruction tuning data. The first row in the table represents the original LLaVA-NeXT-SB.
Visual Encoder
LLM
Pre-training
Fine-tuning
Math Verse Ace (%)
CLIP
LLaMA-3 (8B)
LLaVA data
LLaVA data
15.6
CLIP
Mammoth-2 (7B)
LLaVA data
LLaVA data
18.3
CLIP
Mammoth- (7B)
LLaVA data
MAVIS-Instruet -AAIS-Instruct
25.7
CLIP
Mammoth- 2 (7B)
MAVIS-Caption
26.4
Math-CLIP
Mammoth-2 (7B)
MAVIS-Caption
MAVIS-Instruct
27.5
21

---

Published as a conference paper at ICLR 2025
Based the results presented in Table make the following observations:
1. Mammoth-2 vs. LLaMA-3: Mammoth-2 achieves +2.7 improvement in accuracy com- pared to LLaMA-3, highlighting its prior knowledge and inherent capability mathematical problem solving.
2. Impact of MAVIS- Fine-tuning with MAViS-Instruct significantly enhances performance by +7.4, underscoring the substantial advantage our dataset for mathematical reasoning tasks compared to general visual instruction datasets.
3. MAVIS-Caption and Math-CLIP: Using MAVIS-Caption for pre-training and employing the Math-CLIP encoder further boosts performance, leading to enhanced mathematical visual perception and reasoning capabilities. Overall, MAVIS data contributes +9.2 improvement in accuracy over Mammoth-2 trained with LLaVA data.
A.3.4 PERFORMANCE ACROSS DIFFERENT SUBJECTS
Although MAVIS-Instruee contains a substantial number of high-quality solid geometry problems that were manually curated, our data engine only generates plane geometry and function problems. Therefore, we aim to evaluate the performance of the MAVIS model across different mathematical domains, specifically plane geometry, functions, and solid geometry. We provide the detailed subject scores of MAVIS-7B on Math Verse, comparing the CoT evaluation score (note that the subject-level accuracy scores are not publicly released) with other models on the official leaderboard.
Table 10: Performance comparison across different models on Plane Geometry, Solid Geometry, and Functions of Math Verse evaluation tasks.
Model
All (CoT-Eval)
Plane Geometry
Solid Geometry
Functions
LLaVA-NeXT
17.2
15.9
19.6
23.1
ShareGPT4V
17.4
16.9
15.0
20.2
SPHINX
22.8
24.5
15,8
19.5
InternLM-XC2
25.9
26.2
20.1
23.7
MAVIS-7B
35.2
37.1
28.9
31.0
The results shown in Table 10 demonstrate that our model achieves leading performance across all three subjects. Notably, its proficiency in plane geometry and functions can be attributed to the training with our meticulously curated MAVIS dataset. Additionally, for solid geometry, which shares similarities with plane geometry in both visual appearance and reasoning process, we helieve that our model effectively generalizes its learned knowledge and reasoning capabilities, leading to enhanced performance in this domain as well.
A.3.5 SYNTHETIC DATA VS REAL DATA
integrate both synthetic problems generated by the data engine (633K, 76%) and real-world problems augmented with GPT (201K, 24%). The synthetic data is composed of both geometry and function problems, while the real-world data primarily focuses on geometry. We conduct an ablation study to assess the contributions of these data components, excluding the DPO training stage to ensure fairness.
In
Table 11: Ablation study of synthetic and real data contributions to MAVIS-7B's performance.
Synthetic Data
Real-world Data
Math Verse Ace (%)
GeoQA
FunctionQA
MMMU-Math
22.6
44.2
37.1
34.6
24.3
66.4
25.8
29.8
27.5
66.7
40.3
39.2
The results shown in Table 11 indicate that the two data sources exhibit complementary characteristics, both playing a crucial role in achieving the final performance. Specifically, synthetic data significantly enhances the results FunctionQA and MMMU-Math, these benchmarks include substantial
22

---

Published as a conference paper at ICLR 2025
proportion of function-relateds problems. Conversely, real-world data has greater impact GeoQA, given its stronger alignment with the crometry-focuseee nature of this benchmark.
A.3.6 DATA SCALING
good instruction tuning dataset should exhibit the characteristic data scaling: as the dataset size increases, the model trained on it should demonstrate progressively better performance. To verify that MAVIS-Instruct possesses this property, we conduct an ablation study on the 834K HAVIS-Instrun dataset by randomly sampling 25%, 50%, and 75% of the data for instruction tuning, excluding the DPO stage. We then evaluate the models using the accuracy metric on Math Verse. The results, as shown in Table 12, indicate that the performance of MAVIS-7B consistently improves as the data scale increases. This demonstrates the promising potential of our dataset to further enhance mathematical reasoning capabiliities with larger-scale utilization.
Table 13: Comparison of different training set- tings.
Table 12: Performance of MAVIS-7B at different data proportions.
LLMs
Caption CIDEr
MathVerse Acc (%)
25 %
50%
75%
100%
Frozen
79.6
26.2
23.3
25.7
26.9
27.5
Unfrozen
146.2
28.
LoRA-based
161.3
28.4
A.3.7 GENERALIZATION ABILITY
Although Data Engine considers many problem types possible. it is inherently challenging for a manually designed engine to cover all possible types of mathematical visual diagrams. To alleviate this, incorporate real-world data by manual collection and GPT augmentation, which well enhances the generalization capablitity of MAVIS-7B. In Figure 11, we selected examples involving inscribed triangles (they cannot be generated via our data engine) from publicly available math benchmarks, and tested MAVIS-7B BBB on these examples, where our model performs accurate reasoning.
Since AB tangent circle B, have OB Therefore, angle ABO 90; Angle 36", givnng
therefore angle equal angic
degrees.
ongle Since
Alio, equal to angle equal to angle Aco which 50 degrees, Since ongle CAB is half of angle BOC, it 25 degrees. Therefore, ongle BAO is equal to orgle OAC minus angle CAS which is degrees.
B
equal angle ongle argle
equal
And equal angle c plu angle ORC, have ongle equal to half of angle AdB, which is half of 54", giving us 27".
Q: AB tangent to circle O at point B. and the extended line of AO intersects circke O at point Cornect BC, ongle A 360, then angle C equal to
circle O, chard AC paraliel redial O8, ongle, BOC 50.0, the degrue ongle OAB is
Figure 11: Examples for MAVIS-7B solving math problems with circumcircles of triangles.
A.3.8 ENHANCING VISION-ONLY PERFORMANCE
To evaluate the impact of OCR datasets MAVIS- 7B's performance, we conducted an ablation study focusing on vision-only and vision-dominant problems Math Verse. These problems require the model to interpret question texts rendered directly in diagrams, thus relying heavily on OCR capabilities. MAVIS-7B, however, was initially not trained with OCR-specific datasets, limiting its performance in these tasks.
In contrast, generalist models like LLaVA-NeXT include extensive OCR datasets such as OCRVQA. Doc and SynDog-EN which significantly enhance their OCR capabilities. To bridge this gap, we incorporated OCR datasets (OCRVQA and DocVQA) in our third-stage instruction tuning to improve MAVIS-75 - OCR performance.
The results, as shown in Table 14, indicate a notable improvement in vision-dominant and vision-only problems for MAVIS-7B after the inclusion of OCR datasets, highlighting the potential of better OCR
23

---

Published as a conference paper at ICLR 2025
integration for further boosting its performance. In Figure 12, we also showcase some failure cases of our MAVIS-7E 7B with OCR training on vision-only problems. Although the vision-only results are improved via the OCR instruction dataset, the model still suffers from limited perception capabilities of questions and visual elements within the diagram. This indicates that the OCR capability still the bottleneck of vision-only performance. We leave this future work further enhance the OCR capabilities of MAVIS for mathematical visual elements.
Question Diagram:
Question & Diagram:
Question & Diagram:
amnns ciceed
Answer:
Answer:
C
MAIS-78
MAIS-78 with
MAIS-7B
Since angle ABC.
gives diggrmm tangent circle point parpendioskr tongest aOBG tangency. 0B-0C radil the drdie). <AB0:36. 90-366-54. Since 208C-180
angles CBD
To solve 10-((7))9gg()) we
congrvont
firat detteriin 1(7) and (() from the grapha From the blx graph (y=f(0), at xet, f0/--6 From the pink groph (yng(())
meosuring 40
(70 = Becouse parallel
g03-2. substinutngg these
acts transversel, ongle CBB also mecoures 35 degrees. Thanefore, the onswer is b
10x(-0)9122-4000 42.
valsed the expression gives
ABBC is isoseeies, cCx208c:54. Thus, the correct onswer B: 54".
Figure 12: Failure cases of MAVIS-7B with OCR training on vision-only problems.
Table 14: Impact of OCR data on solving vision-only problems.
Model
LLM Size
All
Text Dominant
Text Lite
Vision Intensive
Vision Dominant
Vision Only
LLaVA NeXT
8B
15.6
19.4
15.2
16.8
15.2
11.3
LLaVA-NeXT
110B
24.5
24.1
22.1
20.7
MAVIS 7B
7B
28.4
41.6
29.5
27.9
24.7
18.3
MAVIS-7B w/ OCR
7B
28.9
40.8
29.2
27.4
26.2
21.1
A.3.9 BASE LLM
We investigate different LLMs for the MAVIS model. As shown in Table 15, MAVIS is not very sensitive to LLM choices, and still surpasses previous models with the same LLM.
Table 15: Performance Comparison using Different LLMs. We compare the accuracy and CoT evaluation results on Math Verse.
Marhe
Method
Base LLM
Acc
CoT-E
SPHINX-PIE -lus
LLaMA2-13B
12.2
14.0
ShaveGPT4V
Vicuna-1 13
13.1
17.4
M-XC2.
IntemL.M2-7B
16.5
25.9
I.I.MA2-13R
24.5
30.7
Vicuna-13B
24.8
30.6
MAVIS
InterL.M2-7B
28.0
33,8
MAmmoTH1
28.4
35,2
A. DIFFERENT TRAINING SETTINGS
Our training strategy is similar to LLaVA, but with key differences in the pre-training stage: we train both the projection layer and the LoRA-based LLM,, whereas LLaVA only trains the projection
24

---

Published as a conference paper at ICLR 2025
layer. This design choice stems from the fundamental differences between general visual tasks and mathematical tasks:
1. For general visual tasks (e.g., LLaVA), training MILMss typically requires the L.M to generate daily natural language responses, such as descriptive captions instruction-follwwiii outputs. These outputs often rely on pre-existing knowledge within the pre-trained LLM. As result, in LLaVA, there is no need to unfreeze the LLM learn new types of outputs.
In contrast, for mathematical domains, LLMs need generate math-specific responses, such as geometric descriptions, functional explanations, formulas, and theorems. These outputs often involve domain-speeifee knowledge not inherent in pre-trained LLMs. Given this, we incorporate learnable LoRA layers to infuse new knowledge into the LLM, enhancing its capability to produce high-quality mathematical expressions. Concurrently, we aim to prevent the L.LM from overfitting to diagram captioning tasks during alignment. Therefore, using LoRA A-based tuning allows us to preserve the I.LM's generalizable pre-trained language knowledge while injecting specialized math-specific capabilities.
To further investigate the impact of different training settings model performance, conduct an ablation study comparing various LLM training settings during the alignment stage. We evaluate two tasks: the CIDEr score for diagram captioning on 100 validation samples (following the same setting as in Table 6 of the Appendix) and the accuracy score on Math Verse. The results, as shown in Table 13, indicate that the LoRA-base Abased approach performs best, enabling MLLMs to generate high- quality mathematical captions while preserving pre-trained knowledge for improved problem-solving capabilities.
A.3.11 ENHANCING PRE-TRAINED MLLM
To investigate whether our curated data and training techniques can improve the mathematical perfor- mance of a pre-trained large model (LLa Ve-XTT,, we conducted an ablation study. Specifically, we progressively employed
for instruction tuning, followed by DPO alignment on top of LLaVA-NeXT-8B. with both training stages performed for one epoch using a learning rate x 10 The results, as shown in Table 16, demonstrate that these two continual training stages significantly enhance LLaVA-NeXT's ability to solve mathematical problems, with notable improvements across all evaluation categories.
Table 16: Performance improvement of LLaVA-NeXT-R with
and DPO ali alignment.
Model
LLM Size
All
Text Dominant
Text Lite
Vision Intensive
Vision Dominant
Vision Only
LLaVA-NeXT
8B
15.6
19.4
15.2
16.8
15.2
11.3
MAVIS-Instruct
8B
22.8
32.3
25.3
24.6
DPO
8B
24.0
33.7
26.9
25.4
19.1
15.1
A.4 DETAILS OF AUTOMATIC DATA ENGINE
A.4.1 DIAGRAM GENERATION
In this section, we detail the implementation specifics of the process for generating diagrams related plane geometry, analytic geometry, and function domains.
Plane Geometry Diagram. Inspired by previous multi-hop reasoning methods (Kazemi et al., 2023; Wei et al., 2022; Nye et al., 2021), we employ an iterative generation method over logical theories to generate plane geometric images along with corresponding captions and question-answerin pairs, whose complexity can be controlled across multiple axes. Specifically, we first define a set of fundamental geometrie shapes in Figure 13.
Within each shape, new basic shapes can generated by extending a particular edge. For each basic shape, initially define meta reasoning process:
Emn-1 On, E [1,2],
On-1, C
(I)
25

---

Published as a conference paper at ICLR 2025
ADDE
Figure 13: The set of fundamental shapes in plane geometry diagrams, whose straight edges can be extended into other basic shapes.
where O represents the initial side length of the shape, C. denotes the additional conditions required to complete meta reasoning. and Em provides detailed explanation of the meta reasoning process. For example, when considering an isosceles triangle as the (n - 1)th shape in a sequence, the vertex angle is still required as Cm to reason about base side length, and then expand to the n²th shape, with F. serving as the explanation of this process. The variable = indicates that there are sets of possible meta reasoning for the shape, indicates the length of the generating sequence, which is also the number of hops of reasoning required to answer the question. The initial side, extend side, and additional conditions for meta-reasoning of each basic shape can be referred to in Figure 13. In the final shape, question-answen pairs pertinent to this shape can be generated as
On, C 5n. Li An, A E [1,
(2)
where C, represents the additional conditions required to solve the problem, while Q and A denote the question and answer, respectively. Eg refers to the detailed explanation of the solving process. The variable m indicates that there are m pairs of ceestion-answering and corresponding detailed explanations within the shape. By applying meta reasoning to the n 1th shape, the initial side length of the nth shape can be deduced. Therefore, for a complex composite figure consisting of shapes, the overall question- gusseinn-answering pair can be defined as follows:
a-1 Σ Cmp: Cg,, Q% E/ A=1
O₁.
A
Each shape defines sufficient number of conditions, explanations, and answers to ensure the diversity of the generated question question-answerie answering pairs. Based on the aforementioned rules, controlling the length of the generation sequence regulate the number of reasoning steps, and controlling the type of questions can manage the knowledge required for solving the problems. Thus, we can generate questions of varying difficulty levels, which can also be illustrated in Figure 14a.
Analytic Geometry Diagram. The image generation method for analytic geometry is relatively straightforward. First, we randomly select range within the coordinate system: the minimum value of x is chosen as an integer between 12, -8], and the maximum value of x is chosen as an integer between [8, 12]: the range for y is the same as for x. Then, we define the following basic shapes: point, line segment, line, circle, ellipse, rectangle, square, polygon, and sector. During the generation process, we select number between and as the number of shapes to generate. The generation rule is that nonlinear shapes other than points, line segments, and lines must not overlap.
Function Diagram. The generation of function graphs is also straightforward as shown in Fig- ure 14b. We define the following basic functions, each with set of parameters that can be randomly selected:
Sine Function
= A sin(J x Φ). where the amplitude A is random integer between 1 and 3, the frequency f is either I or 2, and the phase 6 is random integer between and 2n.
Cosine Function
y A cos(f x + Φ), where the amplitude A is random integer between 1 and 3, the frequency f is either I or 2, and the phase o is a random integer between and 2T..
26

---

Published as a conference paper at ICLR 2025
Extend into another basic shape.
30
30
2
2 Meta Reasoning: Since AB=2 and A 30, calculate angle isosceles triangle Thus, 1.04
2
B
Initial Condition: AB 2
Randomly select edge
Initial Condition: CB 1.04
baisc shope
Condition for
External
Reasoning:
Meta Reasoning: A 30
(Why @: Since sector only has radius extended, length can be derived from the known length of BC without additional conditions.)
Caption:
Randomly select template.
ABC on isosceles triangle where AB AC A 30
Caption:
condition Question:
an isosceles triangle
AC, / /AA 30
Question: AB=2, 7 30, /CBb what the of sector CBD? Analysis:
Extended from side BC, BCD conforms sector.
Step 1: Since AB-2 and Z = 30, we can calculate the angle C of Sin(
isosceles triangle c (180 30)/2 Using
of Sines,
0,97 1 004;
Step 3: Thus, crea of sector CBD o sr*
0.85.
(a) single process generating plane geometry diagrams and corresponding question-answeriigg pairs as well as image captions. In this example, the generation sequence length is specified as 2. Initial side length painted in pink, C.. is painted in green, while C. is painted in ycllow. Whenever a new basic shape is generated, its caption is appended to the previous caption.
Initiliaze
Derive
Zeros: x=0
Randemly select templste.
Q: What are the zero points the function?
Asymptote: xi-1/4
Randomly select function type: y o"log(="x
None
Minimm: Nonz
Derivative: x11)/(+*)+)
Randomly initialize
The function has one zero point,
a--2. be4 col
the porameters:
Gernate
caption
The figure graph of f(x)= lagge 4*x).
Caption:
100
350
2log(('*x
(b) A single process is used for generating function diagrams along with the corresponding question-answer pairs and image captions. Once the functional expression determined, all its properties be directly computed, function plot can generated accordingly. caption for function diagram simply states functional expression.
The pipeline of data engine, consisting of (a) the generation of plane geometry diagrams and (b) the generation of function diagrams.
Figure 14:
Tangent Function
// A. lan(S x Φ), where the amplitude A is random integer between and 3, the frequency is either 1 or 2, and phase & is random integer between and 2T.
Polynomial Function
P(z) a,, "L a,-1-= ..... a + an. where the degree is random integer between and 4. The coefficients o; are randomly selected integers ranging from -3
piece-wise Function
piece-wise polynomial functions are divided into or 3 segments, with each segment's parameters identical those of a polynomial function.
Logarithmic Function
= a loga(c + d), where the coefficient a is randomly cho- sen from (-3, -2, 1, 1,3,), the base b randomly chosen from {2, 10, [e]}, the coefficient is a random integer between 1 and 3, and the coefficient d random integer between and 6, ensuring that e x + d is positive.
Absolute Function
y = la b). where and are random integer between -5 and 5.
27

---

Published as a conference paper at ICLR 2025
-75
130
53 23 -12
At outset, Located -3), circle encompasses rodius of I. Later, sector originates from center (-5, 0) and extends radius Following this, pentagon with points of (-1055. 0,94), (4.55, -2.76). (8.0, 2.0). (4.55, 6.76). (-1.05, 4.94) established.
To start with, geometric figure, ellipse, (0, -7), measures 4 in height and in width. There exists point at (7. 5). Also. With left bottom corner (4, -2), rectongle extends a height width of Moreaver. line segment is establishe between (-5, (-5.5).
At the There ellipse with a height of and width of centered (-8, -2). Beyond this, Crossing at (0, the y-axis and (8.0, the x-axis, a straight exists.
Figure 15: Examples of analytical geometry diagram caption.
We first determine the domain range to be displayed on the function graph. For trigonometric functions, the domain is set [-,, π|. For piece-wise polynomial functions, the minimum value of x is random integer between 1-1,, -8]. and the maximum value of is a random integer between [8, 12). For other functions, the minimum and maximum values of x are random integers within the ranges of [-6, -3] and [3, 6]. respectively. During the plotting process, we calculate the local maxima, minima, and zeros of the function by iterating through the domain. then render the x-coordinates of these extrema and zeros the x-axis of the function graph.
A.4.2 MAVIS-CAPTION
In this section, we detail how the captions corresponding to images in the MAVIS AAVIS-Caption Dataset are generated with our automatic data engine.
Plane Geometry Caption. Based the generation process described in Section 1.4.1, when generating each shape, a caption randomly selected from set captions for that shape and some connecting words randomly added. We also randomly select some edges angles and state their measurements the caption. After generating the raw caption, we use GPT-3.5 to refine it, enhancing its linguistic structure and semantic diversity. An example is shown in Figure ??.
Function Caption. According to the function graph generation process described in Section A.4.1, we record the function's 's zeros and extrema. Additionally, we also record the function's 's expression and asymptotes. These attributes are incorporated into randomly selected caption template to form the function graph's caption. Some examples are provided Figure 16.
Analytic Geometry Caption. For each shape, maintain set of caption templates that describe the shape's 's type, coordinate position, and other attributes. the generation process described in Section 1, select template and randomly add some diverse connecting words to form complete caption. Examples of some captions are shown Figure 15.
A.4.3 MAVIS-INSTRUCT
Manual Collection Augmented by GPT-4. To complement the dataset with real-world problem- solving scenarios, hire 8 human experts manually collect visual math problems from various public 1.2. spanning plane geometry, analytic geometry, and function. For problems, we try to obtain their content complete as possible, including questions, diagrams, answers, and rationales available. The collection process consists of the following steps:
1. Problem Collection: We gathered problems from three public sources as comprchensively as possible, including questions, diagrams, answers, category information, and rationales where
28

---

Published as a conference paper at ICLR 2025
3'5in12
y 3'log2,
325
121
320
-020
The equction of the function
The function is given 3*lag(2, 2*x 6). With an range of [-3.0, 5.0], find the zero
The expression for the function is = 12 1l. The zeros for the ronge [-3.0, found to be -05. The function spons ronge volues from at -0.5 to 13.0 at 6.0 across the interval [-3.0, 6. No
y= 1*sin(1*x 3). Observing the range of [=\u03c0. \u03c0], the zero points determined are -3.0, 0.14. The function spans range of values from -10 1.71 10 -1.43 ocress the interval (-T,, J. The functions lack asymptotes.
points at -2.5. Within the given
range [-3.0, 5.0]. the maximum value the function 12.0 5.0, and there
minimum value point this ronge. An asymptate of the function is observed at: -3.00.
asymptotes are present in the functions.
Figure 16: Function diagram captions.
available. The problems are primarily at the high-school level, covering plane geometry and functions (including analytic geometry).
Based on their initial categories (subject, subfield, and difficulty level), the problems were organized into distinct groups. Six expert annotators were tasked with meticulously verifying the correctness and completeness each problem. They refined the detailed chain-of- thought (CoT) rationales and ensured that there inspecting the diagrams. This rigorous verification process resulted in a total of 4K verified problems.
Data Verification:
overlap with evaluation data by visually
3. Text-lite Construction: To optimize the problems for training mathematical visual capabilities, the 4K problems were processed using GPT-4V with a customized prompt (as shown in Figure 15). This step involved removing redundant information from the question text to create concise, text-lite problems, specifically tailored to our training objectives.
Then, we first feed all the related information into GPT-4V to eliminate the redundant information within text questions, constructing the text-lite version of problems by the prompt Figure 17. Then, we design three types of prompts for GPT-4 to augment 15 multiple-choice questions (including 10 multiple-choice and 5 binary-choice, i.e., "True" or 'False') and 5 free-form questions, respectively, as shown in Figure 18. We do not adopt GPT-4V here, since GPT-4V itself would misunderstand diagrams for low-quality data augmentation. The newly generated problems contain detailed CoT rationales and diverse question forms.
Existing Datasets Augmented by GPT-4. Previous efforts have been made to provide some small- scale, plane geometry datasets, e.g., GeoQA (Chen al., 2021c), GeoQA+ (Chen et al., 2021a), and Geometry3K (Lu et al., 2021). Although they are limited in data scale for tuning MILMs and include no rationales, we can also regard them as seed dataset and adopt GPT-4 to augment larger-scale training data. We do not utilize GPT-4V here for the same reason aforementioned. In detail, we design types of question generation approaches using different prompts, as shown in Figure 19. For Geometry3K, as the question texts are normally brief and contain marginal descriptive information, posing challenges for GPT-4 understand the diagram, we only augment them to generate binary-choice questions, i.e., 'Ture' or 'False'. For GeoQA+, we can leverage the sufficient redundant information within their texts to generate more diverse and accurate multi-choice and free-form questions. Likewise, GPT-4 can produce CoT rationales for each problem.
'nttss:/hommewrr.stuuyy hhtps://wwwwixl wwwm///t...ommhhhhhccccccccc "htps://mattspace.coii
29

---

Published as a conference paper at ICLR 2025
Iaagram
Pronpe
Original Quastion (Aloeve) and Refined Qusstion (Balow)
Find the lengtt the mussing side of the right triangle Figure
Husic complets the salk accerding to the information prenied and the read wnnont
will aise meth preb en than includes correspending image. Flease nemaxe the information that ciready present the problesin [foussssion)
Find the length of the missing side of the triongle in Figuns.
quedtion)
cny
inforeation slrendy prasect is image thene infermaiion the [quedtien] that can be retuased lncted an the itaage, then [fox puucttten on] should nonzie archanged. Do not ofter the onig nol conteer of the (sauctin))
long shown. Form un exprassioo fory in terms x
the shrpe pantogon
on long Form expression
terms of
You read the quection boted keaye [puethie]] Flease
the relutes speeric
recastont aformerie ranoved DurpuP your autver with "tix_quathion" and "neasort 260N foreat ONLY.
Ir the given figure, angle BAC-19 and angle OdD=16.
Salve for 1.gking
Solve 1. givity
Figure 17: Manually collect visual math problems text-lite version.
Diagram
Original Questioo
GPT generated QA pairs (multi-cholee)
In the figure shown, The perimeter
Qusee loni Booed on the inage. how mony sides dees the pelygon hove? Cheios: (3.4.5666)
of ADF is Zo feet.
What is perimeter
Analyaisi From the problee contect, the pelygen discuraed is ADF which is hexagen, idestifying the triongles that fit sirulituausly in the shope, one to the numbar of sides of the langer shape.
hexagon feet?
Analysis
We are given that on equilateral triangle. equilcterdl triongle, all three sides are equal and therefore the perimeter
GPT generuted QA pairs (judge)
Queee Shaise: Dles, Ne]
image. triongle?
triangle equels (number sides) x (side length) =3 AF given that ABCDEF regular hexagon, In reguler haxagon, six are equal therefore the panimeter of hexagon equals (nutiber of sides) (side length) =5 AP (where AP is also side of hexagen). substituting AF=2a formula yields 6 A Fx6(2 a a The answer is 4a.
Anolysici Bosed image, It triongle
Aneaer: No
GPT generated QA pairs (fre-QA)
Qusetioni Besed on the image, what the relst relitiondiip between the side heseger?
Analysial Since ADF is component the regular havegon, the side of the triongle and the heregon identical, This reletionship cracial for the triangte's 's's properties.
into this
calculating henagon's inss properties Aneaer: The zide lengths ane equel
Figure 18: We design different types of prompts for GPT-4 to augment 15 multiple-choice questions and 5 free-form questions, respectively.
Data Engine Captions Annotated by GPT-4. Given the delicately designed data engine for automatic diagram-caption creation, can utilize the generated large-scale pairs to annotate question- answering data using GPT-4V. Different from the previous two sources that augment questions based on questions, we utilize the GPT-4V model here for caution data with reasons: first, the detailed caption from our data engine can well guide GPT-4V for relatively higher-quality visual embedding; second, the visual input serves guidance question forms. As shown in Figure 27 and Figure 28, we adopt different prompts for function and plane geometry problems, ensuring that the generated question-answe answering data is of high quality for instruction tuning.
provide additional spatial information for broad
Data Engine Generated Problems:
PLANE GEOMETRY. Based on the generation process described in Section A.4.1, we pose questions about the final shape in the generation sequence. We designed 6 types of questions: finding the perimeter, finding the area, finding the base length, finding the angle, finding the are length, and finding the extended edge length. Each type of question has set of templates that can be randomly
30

---

Published as a conference paper at ICLR 2025
Prompt
generated enhenee data
Please modify and omplity the provided information according to the following principles
Quisetioni 6 Poien P (-4, -8), Point (-3:3)) it (-4-1), and Paint 5 (-5, and paraliel Cheise) (Mes, Ne]
you output follow following principles:
Anclysia For parallel they Answor: Yes
gradient.
"Question": question that needs be modified based the information provided, you con Modify the numerical values, Reask questions based on "Answer" and 'Analysis", Represent the question using different representation
Qusetiori For Line LJ, with (o, 0) and (t. 2), and Line KL. with x (0, () and colculated separctely?
How
"Choice': Besed the modified accordingly. If the given question does not require choice, then choice field null. Choices should es diverse as passible, and options provided should include example with null Choice
provided
Question. Choice
Cheise
Equal slopes imply paralled lines. The colculation alope cornelation between cecrdiotes xcand
you can previde
you "Analysis" Detailed onelysis and onswers to your "Question"
Answar: Slope 17 (y² (o2-xx), Siope Rt 1y4
Qusetioni Given Point 6-1, -1,, Paint (0,0) Point R (-1,6 and Poiet s type shape PIRR
"Anwwer: The answer to this question. Choice is not rull, it needs to correspond content Choice. Choice null, only output "Analyais" obtain the final
Choise A. Parallelegran & Rentongle C, Sector Tricngle
Anolysiai Ospaltt edçe panollel, S is paralidograms Answer: A. Panalleiegnom
We design 3 types of question generation approaches using different prompts to augment existing visual mathematical dataset.
Figure 19:
Text Dominant
Text Lite
Vision Dominant
Vision Only
AB spon 8.0. DF spon 5.0, ABC is decron CBO sector, D&EF a rectangle but its side EF is extenddd to seni-circle inside the nectonalee What is the overall penimeter of DEEF?
DF spon 6.0. What is the everall perimater of OBEF?
What the overall perimeter of DEEP?
Figure 20: The Text Dominant, Text Lite, Vision Dominant, and Vision Only versions of the same question. Text Dominant and Text Lite use the same image. In the text, the necessary conditions for solving the problem are highlighted in red, while redundant descriptive conditions are highlighted in blue. In the Vision Only version, the question is rendered in the image, with no textual format.
selected, shown in Figure 21-26. As for the answer and analysis, each shape has a set of templates for different types of questions to choose from, as shown in Section 1.4.1.
To further enhance the model's s understanding of different forms of questions and better utilize the diverse modal information in the text and images, we divided the plain geometry questions generated by the Data Engine into four versions referring Math Verse (Zhang et al., 2024b): Text Dominant, Text Lite, Vision Dominant, and Vision Only.
Text Dominant
We marked all the conditions required for solving the problem in the diagram and also described these conditions in the text, along with some redundant descriptive text.
Text Lite
All the conditions required for solving the problem are randomly divided into two parts: one part is marked the diagram, and the other part is described in the text. In other words, the conditions in the diagram and the conditions in the text do not overlap.
Vision Dominant
All the conditions required for solving the problem are marked in the diagram, while the text only contains the question without any conditions.
Vision Only
Not only are the conditions required for solving problem marked in the diagram, but the question is also rendered in the diagram, leaving the text portion empty.
The differences among the four versions of the same question are illustrated in Figure 20. Each basic shape will retain a set of redundant conditions. During the shape generation process, there a 50% probability of including these redundant conditions.
31

---

Published as a conference paper at ICLR 2025
perimeter
M area
What is the perlinetor of (shape_ic]
What area of entire shape (shope_id)?
Hew long is the perimater of (shape_id]?
How much area of the entire shope (shope_id}?
What is the total length of the perimeter of (shope_idj7
What the total area of the shape ishape_idy?
Can you calculare the penimeter of [shape_id]?
Can you calculate the area the entire shape (shape_idj?
How do you find the penimeter of [shope__dd}
What is the overall area of the shope (shope__ld) idj?
What is the measurement of the perimeter of (shape id)?
How do you find the area of the entire shope [shope_id]?
What is the total perimeter measurement of [shape_id)?
What is the measurement of the total area of (shape_idf?
Could you detanmine the perimeter of [shope_idf)
What is the total area measurement of the shape (shape_id]?
What would the penineter of [shape_id] [i]__ddjeaa be?
Could you determine the area of the entire shape (shope_id} idj?
What is the penineter length of (shape_idd}
What would the area of the entine shape (shope_idd be?
Hew would you calculate the perimeter of [chape_id]?
What the area size of the entire shope (shope_idd)
What' s the perimeter of (shope_id]?
How would you calculate the total area. of the shope (shope__dd) idj?
Hew much is the penineter of [shape_id]}
What" the area of the entine shope (shape_idß?
Can you tell me the perimeter of (shape_id]?
How much orea does the shope (shope_id) _ippe_id cover?
What is the tatal boundary langth of (shape_id])
Can you tell the area of the whole shape (shape_id}?
What is the entire perimeter length of shhppe_ic]?
What the overall
measurement the shape (shape_idj?
Hew do you measure the perimeter of [shape_id]?
What full area of the shope (shape__
Hew would you determine the perimeter of (shape_id)?
How you calculate the of the shape (shape_idjp
What is the full perimeter of sshope_id)?
What total surface
the shape (shape_ide
What is the overall penimeter of [shape_id]?
How determine
shope (shape_id}p
Hew can we colculate the perimeter of [shape_id}?
total of shope (shape_idj?
Figure 21: Perimeter problem templates.
Figure 22: Area problem templates.
base length
What is the length of the base @base_side__de in isosceles triangle (shape_id}?
Con you tell me the length of the base [base_side_id]] in the isosceles triongle (shape_id}?
What the measurement of the base (base_side__te the isosceles triangle (shope_id))
How long the base (base_side__ in the isosceles triangle (shape_id}?
What the base length (base_side_idf
isosceles triangle (shape_id]?
Could you provide the length
base (base_side_id) the isosceles triangle (shape_id}?
Con you specify the length
base (bose_side_ide in the isosceles triangle (shape_idj}
to know the length the base (base_side_id) the isosceles triangle (shape_id),
Please tell me the length the base (bose_side_id]) in the isosceles triangle (shape_id),
What the length of the side (base_ssie__dd that forms the base of the isosceles triangle (shape_id}>
Figure 23: Base length problem templates.
FUNCTION. All functions will be examined with two types of questions: finding the derivative and finding the extrema. After obtaining the derivative, we calculate whether the derivative has zeros within the given domain. The presence of zeros determines the method for caleulating the extrema.
32

---

of (oro_id) the Published as a conference paper at ICLR 2025
L angle
In isosceles triangle (shape_id), what the measure of angle (bottom_angle_D) and angle trattom__ngle__1
In isosceles triangle (shape_id), what is the measure of angle (bottom_angle_o) and angle ahottom_angle__e
What are the measures of angle @bottom_angle_00 and angle (bottom_angle__. the isosceles triangle (shope_id}?
Can tell me the measures of angle (bottom_angle_0) and angle (bettom_angle_]) in the isosceles triangle (shape_id}?
In isosceles triangle {shape_id), what are the measures angle (bottom_angle and angle (bottom_ange _aarmm__n___??_____
What is the measurement of angle (bottom_angle_0) and angle (bottom_angle__) in the isasceles triangle (shope_id{)
How large angle (bottom_angle.. and angle (bottom_angle in the isosceles triangle (shape_id} Please provide the measures of angle (bottom_angle_O) and ongle (bottom_angle_]) in the isosceles triangle (shape_id).
What the measures of ongles (battom_angle__. and (bottam_ongle, the isosceles triangle [shape_id}?
Can you specify the measures of angle (bottom_angle_0) and angle (battom_angle__ in the isosceles triongle (shape_id{?
In the isosceles triangle (shape_id). what are the measurements of angle (bottom_angle__ 0) and angle @bottom_angle__])
Figure 24: Angle problem templates.
extend side length
b arc length
shape [shape_id],
length (extend_side,
length (extend_side siddd__dd___________ shope (shope_idf? (shape_idf?
In sector (shope_id); _i]_ what is the length of are (arc_id] What is the longth of are (arc_id) idf sactor (shape_id] id]? Can you tell me the length of arc (onc_id) in sector (shope_id}} id)? In the sector (shape_id), what the measurement of are (arc_idj?
Con you tell length pextend_side_id in shope
In the shape (shope_id), what is the measurement (extren__side_dd]
How long sixtenn__sde____e in shape [shape_idjp Please provide the length of [extend_sidd__te in shape. (shape_id).
How long are (arc_id) in sector (shape_id)? Please provide the length are (are_id) sector (shape_id),
What the length (arc_idd in the sector (shape_ id}? Could you specify the length of (are_id) id] sector [shope_idjp
What is the side length (extend_side_id) in the shape (shape_idj?
Could you specify the length of (extend_side__dd) in shape (shape_ie]]
What the [shope_idj?
What is the measurement of leetted__idd__ide in the shope (shape_idf?
(shape_id), long (d]?_id]?
In shope (shope_id), d)) how long (extend_side_idy)
Figure 26: Extend side length problem tem- plates.
Figure 25: Are length problem templates.
33

---

Published as a conference paper at ICLR 2025
Pronept
by
Hore e same suppeemrtooy if formotion for the degren
Questras
Plose first watch
saah that
Rot coal
then hould cammally snabas the qustten the neaer presen
Acconding captior
aldrer
serech
Nortt you rhould other faión meoine clerg with the
Goptton
The hypa cuartins in math chaic
Bur and!
Coptiot:
Houchaad strictly follew this formur and arder (end every probiem
New oic Auai acr) 00.0 1.0y 790 dur under
caachave yn Phaaghh 3) With on rarce
grice
QuES LON% What the
QUES
voive the functicn the
be mi vigter vass
171. Eqplrin the fuectee with the renge
fanch on dbles mox mur volae, which i
ISES: the
thit 200,
Cuortias
The caymphotes fanchan.
D. HoD
scyett und 120 The 10- 1i
Figure 27: The function prompt for GPT-4V and the generated questions and rationals.
Dagran:
Prongi Gremenyy
by
is some sup aperrrr inforstiin the diggne gime
G Quit
Plose por wonch syme and eok euest on that can
Then ahaald correct cedycis ii the ppeshan and the onswer.
Fe doccel DET'
dipurs couts
semething no
"Accarding orpion'
Naxt, you theald give other poles olong with tie
Quemtie
Caption:
The typc ouc our th tiretaace
Batioea
tecole
You dwaid mntetly liw mis foreet ond ender (ead every rollen
atuoted our
OA Es TIE Wien the bight the trpposio ABC
Radeiiient
mmaadun ABCO The haight shoud
Quettio
wade Scarr DaTG, The piee aner of
Ad viees telleegl (B4) sed6d) 10.1, orower
Angle 925 sassmm siot mmance
ANLSEER 717
bational
nou spon
34 31/3 oown 16
10-1000-11000000000 20-0
Figure 28: The geometry prompt for GPT-4V and the generated questions and rationals.
34

---

Published as a conference paper at ICLR 2025
Additional Condition for Meta Reasoning
Initial Side
Extend Side
AB
BC
AB
0
AB
0
CE
CDE equilateral triangle
AB
AB
0
AD
AD
AB
CD
BC
AB
CD
AC
AB
C x degree
BC
AC
AB
LB = degree
Figure 29: The initial side, extend side, and additional conditions for meta- neatoine of each basic shape. Some special shapes are not extended and only appear the last position of the generation sequence, thus their extend side is Ø.
35


Reviews:
[{"review_id": "hhwygNEYZA", "reviewer": "Reviewer_dHAm", "summary": "The paper argues that the existing models lack in mathematical reasoning in the visual contexts due to the lack of math diagram understanding by the vision encoders, limited diagram-language alignment data, and inaccurate capability of large models (e.g.,GPT-4V, Gemini-Pro) to perform chain-of-thought reasoning. To address this, the paper proposes MAVIS, a rule-based and automatic data engine used for visual instruction tuning of the 7B large multimodal models. Specifically, the paper focuses on plane geometry diagrams, analytic geometric diagrams, and function diagrams. Subsequently, the paper performs 4-stage training on MAVIS-caption and instructs to achieve good performance on the MathVerse dataset. While the data collection, and training pipelines look reasonable, the paper severely lacks solid experiments to establish the usefulness of the generated data.", "questions": "mentioned in the weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The dataset creation pipeline is quite rigorous ranging for the three categories – including caption and instruct data generation.\n- The size of the captioning and instruct dataset is quite large, which is a plus too. The method for creating diagram-caption data using GPT-4 as template generators instead of caption generators is interesting.\n- The paper performs 4-stage training: CLIP-Math, Aligning diagram-language alignment, instruction tuning and preference alignment. \n- The final numbers showcase the benefits of their data and training on the evaluations.", "weaknesses": "- Mammoth-2 is supposedly a strong model for math reasoning for text. It remains unclear how much benefit does the MAVIS data give over finetuning Mammoth-2 with existing visual instruction tuning dataset (e.g., LLaVA data). This is crucial to control for the choice of the base model itself. \n- The main result is on the MathVerse dataset which consists of author-created solid geometry and function problems and plane geometry problems sourced from GeoQA and Geometry3K. The authors augment their dataset with the problems from the same dataset in L338. It is crucial to see if most of the improvements in Table 2 are in the plane geometry, while solid geometry and functions still suffer. \n- Related to above: there is a synthetic component to the dataset and a real-data (e.g., Geometry datasets and sourcing problems from internet) component to the instruct data. How much of the performance comes from the synthetic data and how much of it is contributed by adding the real data is unclear. \n- There are no experiments to study data scaling — does training with all the data better than training with 25%, 50% and 75% of the entire dataset. If the data quality is high, then the model performance should scale with data.\n- Table 4 has too many dashes for open-source MLLMs which makes it hard to put the model improvements in the complete context. The table makes it look as if some numbers have been skipped selectively. \n- The public sources (or protocol) for selecting the math problems from public sources in L322 is unclear. There is no data contamination study to understand the overlap between the generated data and evaluation data. \n- Minor: Table 3 and 4 adds a * on the models that are consider math specialist but there is no * on the MAVIS models themselves. I would consider them math specialists too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0GfURLe8SF", "reviewer": "Reviewer_uZLE", "summary": "This work primarily aims to enhance the visual mathematical reasoning capabilities of MLLMs. The authors observe that existing visual encoders struggle to capture abstract information in math diagram, and MLLMs lack strong visual mathematical reasoning abilities. To address this, they propose a data collection pipeline and collected approximately 1.4 million math diagram-caption pairs and VQA with CoT data. By fine-tuning the CLIP visual encoder, aligning CLIP with the LLM, and applying Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) alignment techniques, they train a 7B math MLLM.", "questions": "- In Section 3.2, why only freeze the CLIP visual encoder and train the projection layer along with the LoRA-based LLM? In LLaVA [1], both the visual encoder and LLM are frozen, with only the projection layer trained to align the visual encoder and LLM. If training the LoRA-based LLM is better, please provide a rationale and ablation experiments to support this approach. \n\n- The core idea behind the rule-based caption and instruction data collection is akin to template filling, and the concept for constructing CoT data seems similar to deductive reasoning?\n\n[1] Liu H, Li C, Wu Q, et al. Visual instruction tuning[J]. Advances in neural information processing systems, 2024, 36.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation behind this work is clear. For example, it identifies the semantic gap between natural images with rich colors and textures and mathematical diagrams with grayscale and abstract elements. Therefore, constructing math diagram-caption pairs is necessary to enhance visual embeddings.\n- The data collection pipeline does not require extensive manual annotation or the use of OpenAI API, it is rule-based and automated, making it relatively low-cost. \n- This work involves a substantial amount of engineering effort, collecting 1.4 M data in the mathematical domain. The model also achieved high performance across multiple benchmarks.", "weaknesses": "- The main contribution of this paper is the proposal of an automated data collection pipeline that doesn’t require extensive manual annotation or the use of the OpenAI API. However, the paper lacks an overview figure in the main text, which may make it challenging for readers to grasp the key idea of this data collection pipeline.\n- This work lacks a certain degree of novelty. For instance, it does not propose a new model architecture, and the training approach is a combination of the popular methods, such as CLIP’s contrastive training[1], LLaVA’s visual instruction tuning, and Direct Preference Optimization[3] (DPO). The model and training method are not specifically optimized for the mathematical domain; instead, they rely on a general model and training approach, with only a mathematical dataset provided for fine-tuning. (Fortunately, the engineering effort is substantial.)\n\n[1] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PMLR, 2021: 8748-8763.\n\n[2] Liu H, Li C, Wu Q, et al. Visual instruction tuning[J]. Advances in neural information processing systems, 2024, 36.\n\n[3] Rafailov R, Sharma A, Mitchell E, et al. Direct preference optimization: Your language model is secretly a reward model[J]. Advances in Neural Information Processing Systems, 2024, 36.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j3g4TyEFIm", "reviewer": "Reviewer_zbps", "summary": "This paper introduce MAVIS, a multimodal large language model trained with mathmatical dataset generated by data engine.\nSpecifically, the authors constructed MAVIS-caption to train CLIP-Math with better mathmatical visual representation ability and MLP multimodal connector. Then the authors constructed MAVIS-Instruct to endow MLLM instruct follow and CoT reasoning ability.\nThe evaluation result shows that MAVIS achieve good performance on multiple math-domain benchmarks.", "questions": "- I'm wondering the result of Table 6, Mathverse but on vision only version. Since CLIP-math helps the model improve its visual representation, a performance improvement on vision-only version can be more convincing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The writing is clear and easy to follow.\n- The idea of using dataset generation engine to enhance MLLM math visual reaoning ability is straightforward and effective.\n- The training stretagy, including the staged training, and the dataset design are well-established.", "weaknesses": "- Although MAVIS is a mathematical visual specialist, the performance is still similar with some generalist MLLMs. Especially on vision-only version of Mathverse, where MAVIS is only 0.9% better than LLaVA-NEXT-110B. Suggesting that the MAVIS's visual representation learning is not very effective.\n- The generation engine can only cover a limited set of mathmatical diagrams. For example, circumcircle of a triangle, which is a very common geometry shape, can not be generated by the engine. Since the engine is manually designed, it will be challenging to cover all kinds of math visual diagram types. If the goal is to train a generalize math MLLM, discussion on the generalize ability of MAVIS is needed, since the training shape is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MnXXZjJAnX", "reviewer": "Reviewer_3ZYS", "summary": "This paper proposes MAVIS, a mathematical instruction tuning framework, including an automatic data generation engine, two multimodal mathematical datasets, and a specialized model MAVIS-7B. The mathematical instruction tuning of MAVIS consists of four progressive steps from visual pretraining to preference alignment, respectively targeting different issues in this field. The rule-based data engine can generate numerous mathematical data without costly human and GPT engagement, which contributes to sufficient data resources for training. The model MAVIS-7B shows good results on different benchmarks.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper comprehensively investigates different aspects of training multimodal math models, providing good insights in this field. The scope includes data curation, visual encoding, training pipelines, and learning methodologies. The exploration of better math-specific visual encoding, Math-CLIP, is interesting. This is a good reference for future work.\n\n2. The proposed data engine is novel and of great value for obtaining large-scale training data. As discussed in the paper, the collection and annotation are expensive for multimodal math, while this data engine effectively alleviates this point. The author also shows detailed human study and comparison to ensure the data quality.\n\n3. The MAVIS-7B model shows competitive performance in different benchmarks. The results validate the effectiveness and generalizability of the proposed data and training method.", "weaknesses": "1. The data engine is a rule-based system with fixed language templates. How to ensure the diversity of captions and questions in training data? It is very important to use wide and diverse data for robust model training.\n\n2. The author only provides the results of the 7B model. What if the model size is larger, for example, 13B?\n\n3. The author adopts introduced data and methods to train MAVIS from scratch. However, it is unknown whether these approaches are generalized enough to help a pre-trained large model, for example, LLaVA, to improve the mathematical performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)