DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models
Authors: Chengke Zou
Xingang Guo
Rui Yang
Junyu Zhang
Bin Hu
Huan Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Visual Mathematical Benchmark
Vision Language Models
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce DYNAMATH, a dynamic visual math benchmark designed for an in-depth assessment of the reasoning robustness of VLMs.

Abstract: The rapid advancements in Vision-Language Models (VLMs) have shown great potential in tackling mathematical reasoning tasks that involve visual context. Unlike humans who can reliably apply solution steps to similar problems with minor modifications, we found that state-of-the-art VLMs like GPT-4o can consistently fail in these scenarios, revealing limitations in their mathematical reasoning capabilities. In this paper, we investigate the **mathematical reasoning robustness** in VLMs and evaluate how well these models perform under different variants of the same question, such as changes in visual numerical values or function graphs.
While several vision-based math benchmarks have been developed to assess VLMs' problem-solving capabilities, these benchmarks contain only static sets of problems and cannot easily evaluate mathematical reasoning robustness.
To fill this gap, we introduce **DynaMath**, a dynamic visual math benchmark designed for in-depth assessment of VLMs. **DynaMath** includes 501 high-quality, multi-topic *seed* questions, *each represented as a Python program*. Those programs are carefully designed and annotated to enable the automatic generation of a much larger set of *concrete* questions, including many different types of visual and textual variations. 
**DynaMath** allows us to evaluate the generalization ability of VLMs, by assessing their performance under varying input conditions of a seed question. We evaluated 14 state-of-the-art VLMs with 5,010 generated concrete questions (10 per seed question). Our results show that the worst-case model accuracy, defined as the percentage of correctly answered seed questions in all 10 variants, is significantly lower than the average-case accuracy. In addition, many models show high consistency in answering these questions -- the incorrectness of a certain variant of a seed question is not only due to inherent randomness. Our analysis emphasizes the need to study the robustness of VLMs' reasoning abilities, and **DynaMath** provides valuable insights to guide the development of more reliable models for mathematical reasoning.

Published as a conference paper at ICLR 2025
DYNAMATH: A DYNAMIC VISUAL BENCHMARK FOR EVALUATING MATHEMATICAL REASONING ROBUSTNESS OF VISION LANGUAGE MODELS
Chengke Zou1.2*t, Xingang Guo Rui Yang'", Junyu Zhang', Bin Hu', Huan Zhang' 'University of Illinois at Urbana- Shaana-Caampain, Universitty of California, Berkeley chengke_ chengke_zou@bere zou@berkeles
edu, (xingang2, ry21, junyuz (onyu26)@11111 eiiliiniss.eed edu Aarhuu@@iling malliinois... edu, huan@huan saaer-hhanggcc.. com Project page: https://dynmmath.gihubbb
ABSTRACT
The rapid advancements in Vision-Language Models (VLMs) have shown great potential in tackling mathematical reasoning tasks that involve visual context. Un- like humans who can reliably apply solution steps to similar problems with minor modifications, we found that tate-of-the-ar VLMs like GPT-4o can consistently fail in these scenarios, revealing limitations in their mathematical reasoning ca- pabilities. In this paper, we investigate the mathematical reasoning robustness in VLMs and evaluate how well these models perform under different variants the same question, such as changes in visual numerical values or function graphs. While several vision-based math benchmarks have been developed assess VLMs' problem-solving capabilities, these benchmarks contain only static sets of problems and cannot easily evaluate mathematical reasoning robustness. To fill this gap, we introduce DYNAMATH, dynamic visual math benchmark de- signed for in-depth assessment of VLMs. DYNAMATH includes 501 high-quality, multi-topic seed questions, each represented as Python program. Those pro- grams are carefully designed and annotated to enable the automatic generation of much larger set of concrete questions, including many different types of vi- sual and textual variations. DYNAMATII allows us to evaluate the generalization ability of VLMs, by assessing their performance under varying input conditions of seed question. We evaluated 14 state-of-the of-the--rtt art VLMs with 5,010 generated concrete questions (10 per seed question). Our results show that the worst-case model accuracy, defined as the percentage of correctly answered seed questions all 10 variants, is significantly lower than the average-case accuracy. In addition, many models show high consistency in answering these questions - the incorrect- ness of certain variant of a seed question not only due inherent randomness. Our analysis emphasizes the need to study the robustness of VLMs' reasoning abilities, and DYNAMATH provides valuable insights to guide the development of more reliable models for mathematical reasoning.
INTRODUCTION
Leveraging pretraining on vast Internet-scale datasets, Large Language Models (LLMs) (Brown, 2020; Ouyang et al., 2022; Touvron et al., 2023; Achiam et al., 2023) and Multi-modal Large Lan- guage Models (MILMs) (Team et al., 2023; Bai et al., 2023; Liu et al., 2024c;a) have achieved remarkable performance across a wide range of tasks. Among them, Vision-Language Models (VLMs) (Zhu et al., 2023; Zhang et al., 2024b) stand out, showing exceptional promise as versatile assistants capable of integrating vision and language for problem-solving.
Among their visual comprehension abilities across different domains, mathematical reasoning (Lightman et al., 2023; Zhang et al., 2024f) stands out as crucial measure of human-like intelli- gence, requiring both math knowledge and logical thinking. Recent work has proposed many bench- marks for evaluating the mathematical reasoning ability of VLMs. MATHVISTA (Lu et al., 2023)
Equaa contribution.
Work done during internship UTUC.

---

Published as a conference paper at ICLR 2025
Seed Question 78, Variant 9 To the function diffecenciable ac
Seed Question 78, Variant Is fination differentiable 0?
Choice: (A) Tes
(8) Mo
44
Chatoes (4) Yes Mo
4
Response from GPT-4o:
Response from GPT-40:
The graph vo provided seeas to dezict V-shaped functica, Nhich indicative of absolute value functica or similer type. Fenction has sharp corner at x = 8, which suggests that it 1s not smooch at that poist.
The graph shown the absolune value funesion, 7 [x], vhich forms a "V" shape. To detarmine if Function is differentiable at point, the Function contimous, and the derivative that point.
The fonction |x] is continuous derivative does nxise that point...
but the
Since the graph shoss sharp comner correct ansver
the
Therefore, the correct choice is: (B)
(0) No, the function is not differentiable at
Wrong Answer and Explanation to Q78
(b) Correct Answer and Explanation to Q78
Figure An example of consistent failures in GPT-4o. Seed question 78 in our DYNAMATH benchmark generates a graph of a shifted absolute value function. GPT-4o consistently provides incorrect answers for variant 9 (left) with 90% repetition consistency, while it can successfully answer variant 7 (right) with 100% repetition consistency. We tested for other 8 variants involving non-zero shifts of the absolute value function, GPT-4o insists that the "sharp comner" at x 0 and produces incorrect answer for 7 variants. More failure examples are in Appendix F.
was the first benchmark specifically designed to evaluate visual mathematical reasoning. Recent closed-source models, such as Claude 3.5 Sonnet and GPT-40, along with open-source models like LLaVA-One Vision (Li et al., 2024). have demonstrated average performance surpassing that of hu- mans. Benchmarks such as MATH-V (Wang et al., 2024a) and MATHVERSE (Zhang et al., 2024c) demonstrate the current limitations of VLMs in handling challenging mathematical problems and understanding mathematical diagrams. Following typical evaluation pipelines, these benchmarks contain a static set testing questions on which a VLM will be scored.
Our work inspired by recent studies (Nezhurina et al., 2024; Zheng et al., 2023; Zong al., 2023; Mirzadeh et al., 2024), which found that even powerful LLMs struggle to reliably solve simple text reasoning problems under different input values or conditions. We found that this issue is even more pronounced in VLMs due to the added complexity of visual context. In the setting of math problems, we identified consistent failure cases on variations of simple questions. As illustrated Figure 1, we identify a simple question asking whether shifted absolute value function f(x) = |æc al is differentiable at x = O. Despite the shift, this question is still quite simple and poses no challenges to humans. While GPT-4o can give correct answers for some values of it consistently gives wrong answer for many different values of a * O. Drawing inspiration from human reasoning, where the same steps be applied to solve similar problems with varying conditions, robust rea- soning model should exhibit the same ability. This raises important questions about the robustness of VLMs' reasoning abilities: are the reasoning procedures in VLMs robust problem variations that pose no challenge to humans?
To address this question, we comprehensively study the robustness of mathematical reasoning in VLMs by introducing new benchmark, DYNAMATH. DYNAMATH is dynamic visual math benchmark designed for an in-depth assessment of VLMs' reasoning robustness. Unlike existing benchmarks, which contain static dataset of benchmarking questions, DYNAMATH contains 501 high-quality seed questions covering multiple mathematical topics: Plane Geometry, Solid Geom- etry. Analytic Geometry, Algebra, Puzzle Tests, Graph Theory, Statistics, Scientific Figures, and Arithmetic. Each seed question is represented as a carefully designed Python program; upon run- ning, a program generates diverse concrete instances of one seed question with random variations in its conditions. The program is individually written for each seed question and considers multiple possible types of variations each question, such as variations of numerical values, function types, graph structure, geometry, mathematical operations, etc. The questions also span varying difficulty levels, from elementary school to high school and undergraduate, with the latter two dominating- The process of dynamic benchmark generation and evaluation is presented in Figure 2. During eval- uation, many concrete questions are created from a single seed question, and thus the actual number of questions evaluated can be much greater (e.g., 10x more) than the number of seed questions.
We conducted extensive experiments on DYNAMATH to evaluate the reasoning robustness of cur- rent -tate-of-the-art (SOTA) closed-source models, including GPT-4o, Gemini Pro, and Claude-3.5
2

---

Published as a conference paper at ICLR 2025
Seed Question 169
Code for Question Variant Generation
The purple and orange curves are f(x) and g(x). f(xgg()) even odd?
Choice: (A) odd (8) even (4) neither
Concrete Questions Answers
_..._______ (eelf))
Sandunly select
Integer
2/2, 5. 1/813 vote. ct.ieiiettt 5, 323. 1/2, 3, 3/2, 37533
actr wt.chieeef-- 3, -23. -2 1/22. -2
X
selfic 1..rondist(-1, 0
sxtr.d: 76. 1e. 6)
acte quary = "The purple and orange curves..."
a Detormin the asseer beasd on thx parisy of and celr.d S ss and sXl o self sreece
for Variant
Answer for
star aetr.d 8.2 setf, sraser
axtric
rum):
information.
Answer for Veriant
Answer for Veriant 4:
Vaniant 5, 7..
Answer Matching
Vision-Language Models
Vaniant (B)
Variant 3
(B) even
Gemini
Qwen
Variant
(A)
odd
Variant 4
(A) odd
The dynamic benchmark generation procedure in DYNAMATH. A seed question is repre- sented as a program that can generate many concrete questions with different variations. The plots for concrete questions are randomly generated along with the corresponding ground-truth answers. During evaluation, all concrete variants of the seed questions are considered, allowing us to evaluate the worst-case model performance and robustness.
Figure 2:
Sonnet, as well as open-source VLMs such as the Intern series (Chen et al., 2024), LLaVA- v1.6 series (Liu et al., 2024b), DwL (Wang et al., 2024c), DeepSeek-VL VL. (Lu et al., 2024), and Llama 3.2 (Dubey et al., 2024). For each seed problem, we randomly generated 10 variants, resulting in evaluation dataset of 5,010 concrete problems. On these problems, evaluate both average-case accuracy and worst-case accuracy. The worst-case accuracy is defined as the percent- age of correctly answered seed problems in all 10 variants. We observe that all considered VLMs have a worst-case accuracy that is close to or less than 50% of the average-case accuracy, signify- ing their unreliability in handling question variations. In addition, consistency on these VI.Ms, which characterizes the model randomness to ensure that a low worst- case accuracy is not solely caused by occasional random errors but also consistent errors on certain variants of seed problem. Our main contributions and findings can be summarized as:
also evaluate the repetition
We are the first to study the mathematical reasoning robustness of VLMs and identified new weakness in VLMs: they may consistently fail on certain variants of simple math questions that pose no challenges to humans. Such a weakness is prevalent in many state-of-the-art VLMs.
We introduce DYNAMATII, a dynamic benchmark comprising 501 individually designed pro- grams capable of generating large number of question variants across different types. Our work is the first dynamically generated benchmark for evaluating the math capability of VLMs.
Based on 5,010 concrete questions generated by DYNAMATI, we conduct an extensive evaluation of both SOTA closed-source and open-source VLMs. We find noticeable gap between the average- case accuracy and worst-case accuracy among all models, indicating that many VLMs do not have robust reasoning capabilities even on relatively simple mathematical questions.
RELATED WORK
Mathematical Reasoning Benchmarks. Reasoning ability is key indicator of intelligence, prompting researchers to develop various benchmark datasets to assess the mathematical reason- ing capabilities of L.LMs and VLMs. Numerous benchmarks have been proposed for evaluating this ability in the text-only domain, including (Amini et al., 2019; Hendrycks et al., 2020; 2021; Cobbe et al., 2021; Mishra et al., 2022: Frieder et al., 2024; Yu et al., 2023; Zhang et al., 2024a). Addi- tionally, recent research has begun to shift its focus towards the evaluation of robustness and the creation of dynamic benchmarks for language models. Several studies (Stolfo et al., 2022; Wu et al., 2023; Srivastava et al., 2024; Nezhurina et al., 2024; Qian al., 2024; Kurtic et al., 2024; Mirzadeh et al., 2024) assess the language models' robustness to the changing of item names or value con- ditions in the text-based question. However, many real-world problems, such as those involving
3

---

Published as a conference paper at ICLR 2025
statistical charts and geometry, rely on visual context. To assess visual mathematical reasoning, sev- eral benchmarks have been designed around geometry tasks (Lu et al., 2021; Chen et al., 202 1) or multiple-choice questions (Liu et al., 2023; Yue et al., 2024). Among these, Liu et al. (2023) studied the robustness of VLMS when faced with changes in the order of multiple-choice questions. Recent efforts have expanded these benchmarks to cover broader array of topics and question formats, such as MATHVISTA (Lu et al., 2023), MATHVERSE (Zhang et al., 2024e), and MATH-V (Wang al., 2024a). Despite the diverse range of questions and visual contexts in these benchmarks, they share a common limitation: both the visual components and text remain static. This allows mod- els to potentially achieve high scores by memorizing patterns from the training data, rather than applying true reasoning skills. In contrast, this paper introduces DYNAMATH, dynamic visual math benchmark that provides a more rigorous assessment of VLMS' reasoning capabilitiess through dynamically generating math questions with visual content.
Vision-Language Models (VLMs) With the success of LLMs, numerous closed-source VLMs, such as GPT-40, Gemini, and Claude 3.5, have excelled across a variety of visual- under- standing and conversational tasks, highlighting the potential of multimodal Al assistants. In the open-source domain, several efforts are actively advancing the field. Approaches like LLaMA- Adapter (Zhang et al., 2024d; Gao et al., 2023) and MiniGPT-4 (Zhu et al., 2023) leverage frozen language models with a limited number of trainable parameters, demonstrating promising results. Furthermore, a range of VLMs trained on larger multimodal datasets has been open-sourced, push- ing the frontier of visual comprehension and generalization abilityy. Notable examples include the Intern VL..5 and Intern VL2 series (Chen et al., 2024), InterLLM-XCompoer (Zhang et al., 2023; Dong et al., 2024), LLaVA-v1.6 series (Liu et al., 2024b), VLa Vision (Li et al., 2024), Qwen-VL (Bai et al., 2023; Wang et al., 2024c), and DeepSeek-Vi VL (Lu et al., 2024). These models contribute significantly to advancing the capabilities of VLMs in prior visual benchmarks.
BENCHMARK DESIGN
We present DYNAMATH, curated evaluation dataset aimed at assessing the robustness of visual language models (VLMs) in multimodal mathematical reasoning across wide variety of mathe- matical tasks with dynamic visual and textual contexts.
3.1 DATASET COLLECTION
Our benchmark collection comprises two phases: seed question collection and program-based ques- tion generation. In the initial phase, we selectively curate set of high-quality mathematics problems that necessitate reasoning based on visual information. The subsequent phase involves transform- ing each seed question into hode- prototypes, allowing for the generation of diverse concrete questions under randomly sampled conditions.
Seed question Collection. The seed questions are sourced from existing visual math datasets and publicly available online resources. We identify 107 questions from Vitta (Lu et al., 2023), covering fundamental concepts in analytic geometry, planar geometry, and statistics. Additionally, we source 27 questions from MATH-V (Wang et al., 2024a), which serve as prototypes for topics related to arithmetic, puzzle tests, and solid geometry. To augment the dataset's 's breadth and depth, we included 45 questions based on scientific figures and 48 undergraduate--eww questions focused on graph theory, drawn from the MMMU dataset (Yue et al., 2024) and various accessible educational materials. Furthermore, we incorporated 236 questions requiring advanced reasoning on topics such as functions, geometry, and statistics, all gathered from publicly available resources the internet. To diversify the question types represented in our collection, we also developed 38 new problems by ourselves covering linear algebra, set theory, and algorithmic flow.
Following the collection of seed questions, we conducted a comprehensive review to eliminate any questions that included excessively complex images, as these would pose challenges for program- matic generation. Ultimately, as shown in Figure 4(b), our benchmark consists of 501 seed ques- tions, with 227 (45.3%) sourced from established visual math datasets, while 274 (54.7%) are newly collected or developed from public resources.
Note that our goal is not to create the most challenging, competition-leye benchmark as in (Wang et al., 2024a), but rather to provide relatively easy benchmarks with diverse variants to evaluate robustness. Nonetheless, we ensure that the difficulty of our questions is comparable to the levels of datasets such MATHVERSE (Zhang et al., 2024e) and MATHVISTA (Lu et al., 2023).
4

---

Published as a conference paper at ICLR 2025
9.5
Am
-3.3
04
-7.4
6.2
077
60
0.4
(a) Numerical Value Variants
(b) Geometric Transformations
(c) Gruph Structure Variants
(d) Function Type Variants
Figure 3: Examples of variation types in DYNAMATH. More examples are in Appendix B and D.
mogram--asee Question Generation. After establishing our seed questions, we recruited group of college STEM students to annotate each question with the common strategies they em- ployed in solving them. These annotations served as prototypes for developing corresponding pro- grams tailored to each question. As illustrated in Figure 2, each question is represented as carefully crafted Python program, which encompasses defined range of conditions for sampling and algo- rithmic calculations to derive the solution. Additionally, we implemented a drawing function in each program, utilizing libraries such Matplotlib and Pyglet generate corresponding images based on varying conditions. Specifically, 470 of the question programs incorporate a plotting function that leverages the randomly sampled conditions to create the visual context of the question, while the remaining 31 question programs utilize fixed images, randomizing only the textual elements. This programmatic approach allows the generation tions by executing the generation program multiple times, facilitating the efficient creation of new problems and enabling the evaluation of the reasoning robustness of VLMs.
large number of concrete benchmark ques-
As shown in Figure 3, we integrate various types of variants enrich the diversity of question generation for DYNAMATH:
1. Numerical Value Variants: Modifying numerical quantities evaluate the VLM''s proficiency in handling different numerical values and performing arithmetic operations.
Geometric Transformations: Altering shapes, angles, dimensions, and relative positions amine the spatial and geometric understanding of VLMs.
Function Type Variants: Varying different types of mathematical functions (e.g.. linear, quadratic) to evaluate how well models generalize across functional representations.
Color Variants: Changing object or curve colors randomly to test the model's recognition of visual patterns and its robustmess to superficial alterations.
Symbolic Substitutions: Modifying symbolic elements such as mathematical operations to de- termine the model's adaptability to various symbolic representations.
Graph Structure Variants: Modifying graph layouts, networks, or other structural representa- tions to assess the model's comprehension of relationships and topological features.
7. Real-life Contexts Variants: Adjusting the contents of real-world scenarios (c.g., calendars, time-related problems, piker-like questions) to test the model's contextual understanding and application to practical situations.
Each variant category targets specific facet of mathematical reasoning, making DYNAMATH comprehensive benchmark for evaluating the flexibility,, robustness, and accuracy of VLMs in solv- ing mathematical problems. Detailed diagrams of each variation are provided in Appendix B.
3.2 DATASET STATISTICS
Detailed statistics on the data composition of DYNAMATH are presented in Table 1. DYNAMATH encompasses nine mathematical topics: Solid Geometry (SG, 3.0%), Puzzle Tests (PT, 3.4%), Arith- metic (AR, 5.2%). Scientific Figures (SF, 9.0%), Graph Theory (GT, 9.6%), Algebra (AL, 10.2%), ), Plane Geometry (PG, 15.4%), Analytic Geometry (AG, 19.4%), and Statistics (ST, 25.0%). Exam- ples for each topic are provided in Appendix D. Each topic necessitates a nuanced understanding of image context, foundational mathematical knowledge, practical reasoning abilities. and logical deduction skills. Importantly, the dataset is designed to cater to varying levels of difficulty, rang- ing from elementary to undergraduate education, with a notable focus on high school (55.3%) and undergraduate (32. 1%) levels. In terms of question types, the dataset consists of 59.1% % numerical questions, 34.7% multiple-choice questions, and 6.2% free-form questions. While VLMs might occasionally answer multiple-choice questions correctly by chance, free-form questions provide more precise evaluation of the model's 's capabilities. Consequently, our dataset emphasizes free-form questions, distinguishing it from previous visual math benchmarks such as MATHVISTA (Lu al., 2023), MATHVERSE (Zhang et al., 2024c), and MATH-V (Wang et al., 2024a), which predomi- nantly include more than 50% multiple-choice questions.
5

---

Published as a conference paper at ICLR 2025
Variants Distribution
Statistic
Number
140
Total seed questions (programs)
501
100
42
Created from existing datasct
227 (45.3%)
80
Newly designed questions
274 (54.7%)
Topics
46-
Solid geometry (SG)
15 (3.0%)
6-101
101-00
10"1101
10
101100
10
10t
Puzzle test (PT)
17 (3.4%)
Arithmetic (AR)
26 (5.2%)
Variant Numbers
Scientific figure (SF)
45 (9.0%)
(a)
Graph theory (GT)
48 (9.6%)
Algebra (AL)
51 (10.2%)
Plane geometry (PG)
77 (15.4%)
Analytic geometry (AG)
97 (19.4%)
New problem 54.7%
Statistics (ST)
125 (25.0%)
Levels
Elementary school (EL)
63 (12.6%)
MMMU 18.6%
High school (HI)
277 (55.3%)
Ma Vista 21.4%
Undergraduate (UN)
161 (32. %6)
Question Types
5.4%
Numerical questions
296 (59.1%)
Math-V
Multiple-choice questions
174 (34.7%)
Free-form questions
31 (6.2%)
(b)
Figure 4: (a) Variant number distribution and (b) source composition of DYNAMATH.
Table 1: Statistics of DYNAMATH.
In Figure 4(a), we illustrate the distribution of variant numbers among the 501 seed questions. No- tably, approximately 30.5% of the seed questions have possible variant number ranging from 10 to 10º. Nearly 93% the seed questions contain more than 10 variants, and 17.4% of them have more than 10th potential variants, demonstrating the diversity of variations in our dataset.
3.3
EVALUATION PROTOCOLS
Our evaluation process consists of two stages: answer extraction and score calculation. Follow- ing the methodology of prior work (Lu et al., 2022), we utilize prompt engineering and template matching extract answers. Prompts guide the model to generate responses in both full and short answer formats. After generation, the short answer is extracted for comparison with the ground truth. Detailed prompts used in experiments can be found in Appendix C.
Our dataset contains N 501 seed questions in total. For each seed question in the dataset, we generate M = 10 variants, resulting in a total of 5, 010 concrete questions. We evaluate two met- rics: average-case accuracy (Aavg) and worst-case accuracy (Awst) over these variants. The two metrics are different from prior benchmarks that evaluate only a single instance of question. The metries are defined as follows:
M
N
1 N
Aang
[)ns( = GT(i,))), Amst
Σ
min [[Ans(i,j) GT JEIIMM
M
where Ans(i. j) and GT represent the generated answer and the ground truth answer for variant j of question i. We also define Reasoning Robustness (RR) the ratio between the average-case performance and the worst-case performance:
Awst
RR
Aavg
The model's response uncertainty reflects both the impact of input changes and inherent uncertainty. the latter of which can be represented by the concept of repetition consistency (RC), similar to self-consistency (Wang al., 2022). We define repetition consistency as:
1 Σ [[Ans=(i,) = Ans(i,))) RC(i,j = K
(3)
where K is number repetitions and Ans (i.j) is the k-th repetition for j-th variant of i-th seed question. The repetition consistency represents the model's confidence in the answer Ans(i,j).

---

Published as a conference paper at ICLR 2025
Table 2: Average-case accuracy Aung on DYNAMATH with 5,010 generated questions. "ALL" represents overall accuracy. Question topics and difficulty levels (PG, EL. etc) are defined in Table
Model
ALL
PG
SG
AG
AL
PT
GT
ST
SF
AR
EL
HI
UN
Closed warreed Large Multimedal Models (LMMs)
Zero-shot GITT-4o
63.7
56.8
$2.0
61.0
76.9
51.8
58.1
69.3
62.4
61.5
68.6
61.8
36.8
7em-shot Claude-3.5
64.8
49.9
49.3
55.3
81.0
44.1
69.4
78.2
62.2
61.2
66.7
62.6
33.3
Zero-shot Gemini Pro 1.5
60.5
52.7
42.7
61.6
70.8
20.6
65.2
69.8
50.2
54.2
62.9
59.2
37.1
3-shot CoT GPT-4o
64.9
58.1
59.3
57.7
84.1
51.2
61.9
71.0
60.9
57.7
66.2
62.5
34.8
3-shot CoT 3,5
49.1
80.2
37.1
78.2
64.9
55.0
63.0
61.5
30.5
3-shot CoT Gemini Pro 1.5
58.7
52.6
45.3
56.7
72.9
21.8
57.9
66.0
54,9
48.1
59.0
58.3
34.2
Open-source Vision Langwage Models (VLMs)
Qwen2-Vrr --n22-VL-72B
55.1
48.1
48.7
50.9
57.6
28.2
45.0
68.9
56.4
54.2
61.3
57.4
30.7
Owen2-VL--72 (3-shot CoT)
52.4
45.1
44.7
47.5
59.4
19.4
44.2
67.1
52.9
53.1
61.0
53.6
28.6
Qwen2-VL-77
42.1
40.3
38.7
39.9
37.1
8.2
44.8
52.1
41.1
39.2
47.6
42.2
24.4
In2emVLL2-76
54.0
44,5
34.7
43.8
67.6
35.3
$1.0
66.7
55.1
51.5
60.3
52.9
26.4
I12em
31.3
21.3
38.8
42.9
15.3
58.1
43.1
38.1
51.0
41.5
23.4
IntemVLL--26B
41.0
35.8
26.0
37.3
38.8
13.5
46.9
$1.9
39.6
40.4
52.1
38.5
22.5
InternV1.2 8B
39.7
33.9
37.3
32.5
46.9
15.9
42.1
47.8
39.1
37.3
51.1
37.4
19.6
Llama-3.2-908
44.0
47.5
37.3
36.8
46.5
12.4
44.8
56.8
39.8
30.0
45,4
43.8
22.2
Deepseek VL chat
21.5
16.0
13.3
26.5
12.9
4.7
32.7
24.3
24.2
15.0
28.3
19.0
16.0
Llava-vl 6-34B
21.4
25.3
27.6
14.9
7.6
36.8
27.8
23.1
23.8
Llava-vl 6-vicuna 13B
19.8
14.7
10.0
23.4
8.2
10.0
21.5
28.2
19.6
10.0
27.1
16.5
14.1
Alva-vl1.577
16.6
10.5
7.3
19.5
6.5
8.2
32.3
17.5
20.2
10.8
18.9
13.3
11.7
Haman
Human performance
77.3
79.9
66.7
80.4
77.5
73.5
69.8
78.0
78.9
75.0
78.6
79.8
72.7
EXPERIMENT
In this section, we conduct thorough experiments assess the performance and reasoning robustness of various closed-source and open- open-source models on the DYNAMATH dataset. Subsequently, we present detailed quantitative results and qualitative analyses in Sections 4.2 and 4.3, respectively.
4.1 EXPERIMENTAL SETUPS
We evaluate the performance of two sets of models the DYNAMATH benchmark, which involves 10 variations for each seed question, resulting in total of 5010 questions. The first group com- prised SOTA closed-source VLMs, such GPT-4o, Gemini Pro 1.5, and Claude-3.5 Sonnet, with zero-shot and 3-shots with of SOTA open-source VLMs, including Qwen2-VL (7B, 72B) (Wang et al., 2024c), Intern (8B, 26B, 40B, 76B) (Chen et al., 2024), Llava-v1.5 (7B) (Liu et al., 2024a), Llava-v1.6 (13B, 34B) (Liu al., 2024b), Deepseek-VE VL (7B) (Lu et al., 2024), and Llama 3.2 (90B) (Dubey al., 2024). We specifically explored open-source models with varying parameter sizes to analyze the impact of model size on reasoning robustness. The deployment of open-source models relied on the Imdeploy package (Contributors, 2023). We set the temperature to 0.0 for all models to reduce inherent randomness. Details regarding the prompts and hyperparameters used in this experiment are outlined in Appendix C.
Thoughh (CoT) configurations. The second group consisted
To assess human performance, generated new variant dataset consisting of 1002 concrete ques- tions (2 variants per seed question). These questions were divided into 20 questionnaires, each containing 50 51 questions. We then recruited 20 undergraduates or graduates from STEM to help establish the baseline for human performance based on their average scores.
For the few-shot setup, we follow the standard approach by including three demonstration examples, each accompanied by the associated visual elements. Given the diverse range of topies covered in DYNAMATH, provide topic-specific demonstration examples to ensure its relevance for each problem in DYNAMATH. Specifically, we curated five demonstration examples from Math Vista (Lu al., 2023) and Math Vision (Wang al., 2024a) for each topic, and then randomly select three examples when evaluating DYNAMATH problems within the corresponding topic. In addition, we incorporate detailed reasoning steps in the demonstration examples, following typical Chain-of- Thought (CoT) setup (Wei et al., 2022). Detailed demonstrations and prompts in Appendix C.3,
4.2 EXPERIMENTAL RESULTS
In this section, we present a detailed comparison of the top-performing VLMs on DYNAMATIL, as shown in Table and Table 3.
Overall Results on Average Accuracy. Table illustrates the average-case performance of vari- ety of closed-source and open-source models. Within the closed-source category, GPT-4o, Claude-
7

---

Published as a conference paper at ICLR 2025
Table 3: Worst-case accuracy Awst on DYNAMATH with 5,010 generated questions. "ALL" repre- sents overall accuracy. Question topies and difficulty levels (PG, EL, etc) are defined in Table 1.
Model
ALL
PG
SG
AG
AL
PT
GT
ST
SF
AR
EL
HI
UN
Closed sourredd Lorge Multimodal Models (LMMs)
Zero-shot GPT-4o
34.7
37,7
33.3
25.8
54,9
11.8
18.8
38.4
35.6
46.2
46.0
34,3
31.1
7em-shot Clande-3.5
35.3
22.1
26.7
18.6
62.7
23.5
27.1
53.6
24.4
42.3
49.2
33.2
33.5
Zero-shot Gemini Pro 1.5
26.9
28.6
20.0
19.6
39.2
5.9
22.9
35.2
13.6
30.8
41.3
26.7
21.7
3-shot CoT GPT-4o
32.3
31.2
40.0
21.6
54.9
17.6
20.8
36.8
26.7
46.2
47.6
30.7
29.2
3-shot CoT Claude-3.5
32.1
27.3
26.7
11.3
5499
10.4
56.0
31.1
30.8
39.7
32.9
28
3-shot CoT Gemini Pro 1.5
23.6
27.3
26.7
14.4
39.2
5,9
18.8
27.2
17.8
26.9
33.3
23.1
20.5
Open sourced Vision Language Models (VLMs)
Qwen2-VL-722
28.3
27.3
33.3
15.5
31.4
0.0
16.7
43.2
26.7
42.3
41.3
30.3
19.9
Qwen2 VL.72B (3-shot COT)
22.8
8.2
35.3
32.8
23.55
Qwen2-VLL-7B
13.8
22.1
6.7
7.2
13.7
0.0
12.5
16.8
11.1
19.2
25.4
12.3
11.8
InternVL22776.
24.6
24.7
20.0
15.5
37.3
5.9
12.5
32.8
20.0
38.5
39.7
23.1
21.1
Intemn
14.2
14.3
6.7
9.3
13.7
0.0
10.4
21.6
13.3
19.2
28.6
14.1
8.7
Intern 26B
14.4
19.5
0.0
6.2
9.8
0.0
18.3
20.0
11.1
26.9
34,9
12.3
9.9
InternV1.22-0
10.4
20.0
5.2
15.7
10.4
9.6
11.1
23.8
9.4
6.8
1lama-3.2-900
13.0
22.1
20.0
7.2
7.8
0.0
12.5
16.8
13.3
3.8
15.9
14.1
9.9
Deepseck VB. chat
4.2
7.8
0.0
3.1
0.0
0.0
10.4
4.0
2.2
3.8
7.9
2.9
5.0
Llava-v1.6338
2.0
6.7
15.9
3.7
Llava v1 vicuna vicuna 13B Llavavl1.5-7
2.8
7.8
0.0
4.1
0.0
0.0
2.1
2.4
0.0
0.0
6.3
2.9
1.2
1.8
3.9
2.1
0.0
0.8
0.0
3.2
1.8
3.5, and Gemini Pro 1.5 exhibit average accuracies higher than 60%, with Claude-3.5 achieving the highest zero-shot average accuracy at 64.8%. However, there remains an 12.5% disparity when compared to human performance, which stands 77.3%. This highlights the need for further de- velopment in the reasoning ability of VLMs. Regarding the 3-shot CoT performance, it is intriguing to note that there no consistent improvement across different dlosed-sourcesd models, confirm- ing findings from previous research (Wang et al., 2024a). For instance, while 3-shot CoT GPT-4o manages to enhance zero-shot performance from 63.7% to 64.9%, both 3-shot CoT Claude-3.5 and 3-shot CoT Gemini Pro 1.5 experience decline in performance (64.8% - 62.5% and 60.5% - 58.7% respectively). Moving on to the open-sourced models, although they generally underperform when compared to closed-sourced models, the gap has been narrowed by recent models such as Qwen2 and V1.2, which have more than 70B parameters. This noteworthy progress is evi- dent when comparing them to previous benchmark results like MATHVISTA (Amini et al., 2019), MATHVERSE (Zhang et al., 2024e), and MATH-V (Wang et al., 2024a). It highlights the promis- ing potential of open-source models in the visual math reasoning domain. Moreover, there is clear scaling trend observed in open-source models, indicating higher performance as model sizes increase. For example, Qwen2 VL boosts its score from 42. 1 1% to 55. % when scaling its parameter size from 7B to 72B, while Intern V1.2 sees an increase from 39.7% to 54.0%.
Overall Results on Worst-case Accuracy. Table 3 presents the worst-case accuracy of different models across 10 problem variants, revealing a significant decline in scores for all models. Notably, the perbest-performing model, Claude-3.5, achieves a zero-shot score of only 35.3%, indicating current VLMs are not sufficiently robust to handle variations in context and images. The situa- tion is even more concerning for open-source models: the best-performing model, Owen2-VLL-22) achieves a score of 28.3%, while smaller models like Llava-v1. 11..vv..vvvieee 1-vvv--naaal- 13B score only 2.8%. Our evaluation results highlight the limited reasoning robustness of both open-source and closed-source models, underscoring the necessity for the community to address these limitations future research.
Fine-grained Results. In Table 2 and Table 3, we present detailed results categorized by different question topics and difficulty levels. From topical perspective, we observe that the Puzzle Test (PT) topic challenges both open-source and flosedsoource models. The top-performing closed-source model. GPT-4o, and the leading open-source model, Intern VL2-76B, achieve average-case accura- cies of 51.8% and 35.3%, respectively, while humans score 73.5%. Notably, all open-source models demonstrate poor performance (0.0%) on the worst-case accuracy metric, except Intern (5.9%). Despite this gap, Table 2 shows that closed-source models such as Claude-3.5 can surpass human scores on specific topics like Algebra (AL) and Statistics (ST), which is promising. When considering difficulty levels, all models demonstrate a trend of decreasing average accuracy as the difficulty increases, illustrated in Table 2. In contrast, human performance remains consistent across difficulty levels, indicating that current VLMs are still not adept at handling more difficult visual math problems compared with human capabilities.
8

---

Published as a conference paper at ICLR 2025
S5.82
51.74
50
.45-9
Repseiing
40
30
32.70
29.55
26.2
20
10
19.53
14.14
10,84
clesde
C3P740
Qwen2-229
Hntem/i2.-0g 789
gemsee
InerrVi 220
InternFL 4440
Llera 3 2-002
Intend 2:09
Lieaa Desponnk
Uivaa d-
tlevad
320
Reanoning
(64
20
472
40
22.0
X-4
Sguie
(b)
Figure 5: Comparing reasoning robustness across different (a) models and (b) topics.
Model name
GPT-4o
Gemini
Qwen2-VI.-72B
InternVL.2-766
Repetition Consistency (%)
94.1
92.5
98.9
99.0
Table 4: The Repetition Consistency (RC) for different models over 5 repetitions.
Reasoning Robustness. We use the reasoning robustness (R.R) metric, defined in Eq 2. to measure the robustness of VLMs by evaluating the relative performance consistency across question variants. We defer the detailed reasoning robustness results Appendix H.3. Figure 5 (a) compares the RR of all VLMs used in our experiments. Notably, Claude-3.5 and GPT-4o achieve the highest robustness among all tested models. Moreover, consistent with previous findings, closed-source models demonstrate greater robustness than open-source models, with reasoning robustness scaling with model size. However, Qwen2-72B and InternVL2-76B outperform Geminí, highlighting the robustness limitations of even large models like Gemini. In Figure 5 (b), compare the reasoning robustness across different question topics for GPT-4o and Qwen2-VL-72 VL-72B. The results show that the VLMS particularly robust in Arithmetic and Algebra question types, indicating their strong arithmetic calculation abilities, which are less affected by changes in visual conditions. However, GPT-4o still exhibits weaknesses the Puzzle Test. Similarly, Qwen2-VL-72B shows shortcomings in both Puzzle Test and Analytic Geometry topics, achieving nearly 0% RR and 30% RR, respectively. These weaknesses suggest directions for future improvement of these models.
Repetition Consistency. To ensure robust analysis and account for the inherent randomness in model outputs, we calculate repetition consistency (RC) as defined in Eq 3. This metric evaluates the model's output confidence across multiple generations for the same question. Specifically, we produce five responses for 501 questions and then compute their consistency relative to the first response. The results, detailed in Table reveal the consistent outputs of four closed-source and open-source models, with RC' values ranging from 92% to 99%. Compared with the low reason- ing robustness scores, VLMs have much smaller consistency on different question variants. These findings reinforce our arguments that VLMs lack robustness in varying question conditions.
Consistent Failure Cases. An interesting phenomenon we observed is that some seed questions are solvable in certain variants but result in consistent failures in others (repetition consistency RC =1 for 5 or 10 repetitions). The example in Figure 1 is representative case: the question easily solvable when the absolute value function the origin, but any shifts tend to lead con- sistent failures on GPT-4o. We extensively examined our dataset and counted the number of such instances. Specifically, GPT-4o, Gemini Pro 1.5, VL--2B, and VnternVL22-761 exhibited 21.8%. 18.4%, 29.9%. and 28.3% of these types of questions, respectively, out of our 501 seed questions. These examples highlight the unreliability of VLMs on mathematical reasoning tasks.
4.3 QUALITY STUDY
Qualitative Examples of GPT-4o. this section and Appendix G, we provide a few qualitative examples of leading VLMs answers. Our analysis reveals that current VLMs can consistently pro- duce incorrect responses to specific question variants while generating accurate answers to others. As illustrated in Figure 1. GPT-4o demonstrates the ability provide correct responses in variant 7, showcasing accurate perception, question understanding, and reasoning ability. However, in variant
9, where the underlying required capabilities remain the same with only slight shift in the image,

---

Published as a conference paper at ICLR 2025
Seed Question 12: What is the period of this function? Answer the question with floatiing-point number.
Variant
Vaniant
X
Variant
X
Variont 4
Vaniant
6.283
Answer: 6,283
Answer: 2.283
Answer:
8.283
Answer: 6,283
Figure/ 7: Example of the Memorization Phenomenon: the generated variants of seed Question 12 and the corresponding responses from Claude 3.5 Sonnet. The model's response remains 2 with high probability, regardless of changes in the conditions depicted in the diagram.
GPT-4o fails to accurately interpret the function's position with high degree of confidence and consistency. This discrepancy raises concerns about the reasoning robustness of current VLMs. For additional examples of GPT-4o and other models, please refer to the Appendix G.
Memorization Phenomenon. In our experiments, we observe a phenomenon where current VLMs tend provide the same answer regardless of changing conditions, indicating memorization rather than reasoning based on generalized underlying principles. When we test variant questions that have the same structure but different parameters and images, the model frequently offers the same answer with high probability, ignoring the specific variations we introduced. Among the 171 questions incorrectly answered by Claude 3.5 Sonnet, this issue accounts for 4.1% of instances. A representative case is illustrated in Figure 7, where altering the period of a sinusoidal function (e.g., from 2T to π or 4x) does not affect the model's response, which consistently remains 2T. The exis- tence of this phenomenon highlights the models' lack of comprehensive problem analysis and their limited ability to generalize across different scenarios.
Error Analysis. We conducted error analysis Claude 3.5 Sonnet to identify potential failure modes on DYNAMATI. Specif- ically, analyzed the 169 questions where Claude 3.5 Sonnet failed, examining the reasoning paths and final answers in detail. The statistical distribution of various eтrоr types presented in Fig- ure 6. We considered five types of errors: figure reading errors, reasoning errors, knowledge errors, calculation errors, and halluci- nation errors. Figure reading errors account for 33. 33.1% % of the to- tal errors, despite Claude 3.5 Sonnet having specially reinforced perception capabilities. This indicates that there is still consid- erable way to go for VLMs to accurately read and interpret data from images. Reasoning errors account for 26.6%, making them the second-largest cause of errors. This suggests that the model's reasoning processes still delicate and be easily disrupted by minor changes in conditions and image input. Calculation errors, which constitute 18.9% of the errors, likely result from the sig- nificant computational challenge imposed by our randomly generated conditions without specially designed parameters, as opposed to simpler questions in prior work that are easier to compute. In ad- dition, Hallucination errors make up 17.5 17.8%, showing that the model tends fabricate non-existent information. More failure examples can be found in Appendix F.
33.1% figure-reading
luciintaiio
calculation error 18. 9%
reasoning errar 26 6%
3.6%
knowledge error
Figure 6: Error Analysis of Claude-3.5 Sonnet.
CONCLUSION
In this work, we introduce DYNAMATH, dynamic visual math benchmark designed to system- atically analyze the robustness of mathematical reasoning capabilities in current leading vision- language models (VLMs). By employing program-based problem generation, we can create diverse variants by altering visual conditions in the seed problems. Our evaluation reveals that leading closed-source and open-source VLMs are sensitive condition changes in question variants, de- spite their required underlying capabilities remaining the same. This raises significant concerns within the VLM community on mathematical reasoning tasks. Our detailed results and analysis not only identify the weak points of current VLMs but also shed light the causes of their errors, thereby facilitating the development and evaluation more robust VLMs in the future.
10

---

Published as a conference paper at ICLR 2025
Reproducibility Statement. We have implemented several measures ensure the reproducibility of our results. This includes providing detailed examples from our dataset, comprehensive descrip- tions of the prompts, and the hyperparameters used in our experiments. Additionally, dataset is open-sourced for reproducibility.
Acknowledgment Huan Zhang supported in part by the Al 2050 program Schmidt Sciences (AI 2050 Early Career Fellowship). The authors thank OpenAl's researcher access program for providing part of the API credits used in our work.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 23XX::3303308774,, 2023.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation-based for- malisms. arXiv preprint arXiv: 2019.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. 2023.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005..4116: 2020.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:21055 2.Ki::2005.15177 14517, 2021.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 2xvv:2110014166 2021.
LMDeploy Contributors. Lmdeploy: toolkit for compressing, deploying, and serving Ilm. https:/ //gttub.. Com/InternIIe Imdeploy, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internin-xcomposer2: Mastering free- form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401.16420, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2XXv:2447.2178.. 2024.
Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, and Julius Berner. Mathematical capabilities of chatgpt. Advances in neural information processing systems, 36, 2024.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, al. Llama-ade Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint 2304v:2304. 15010, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint 00::20009.03000. 2020.
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874 2021.
11

---

Published as a conference paper at ICLR 2025
Eldar Kurtic, Amir Moeini, and Dan Alistarh. Mathador-Im: A dynamic benchmark for mathemati- cal reasoning on large language models. arXiv preprint arXiv:2406. 12572, 2024.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0m 03326, 2024.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 26296-26666 26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https:// 1lava-vl.grt gightu............ to/blog/ 201//22224--1222 1-----111----------------- 1laaaanevvaaa..a.a
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: your multi-modal model all-around player? arXiv preprint arXiv:2307.06281, 2023.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et Deepseek-vl: towards real-world vision-language understanding. arXiv preprint 2405:55033, 2024.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint 205.:::005.04/::: 2021.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507-2521, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. 2023.
Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, Jiao Sun, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.0655 06592, 2024.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning large language models. arXiv preprint arXiv:2410.05222, 2024.
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord. Ashish Sabharwal, Peter Clark, et al. Lila: A unified benchmark for mathematical reasoning. arXiv preprint 2rXiv:2210.175111 2022.
Marianna Nezhurina, Lucia Cipolina-Kuul Mehdi Cherti, and Jenia Jitsev. Alice in wonderland: Simple tasks showing complete reasoning breakdown in state-of-the-an large language models. arXiv preprint Xrxiv:2406. X4v22400.000666,, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730 2022.
12

---

Published as a conference paper at ICLR 2025
Kun Qian, Shunji Wan, Claudia Tang, Youzhi Wang, Xuanming Zhang, Maximillian Chen, and Zhou Yu. Varbench: Robust language model benchmarking through dynamic variable perturbation. arXiv preprint (1XXx::44061768// 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust clip: Un- supervised adversarial fine-tume fine-tuning vision embeddings for robust large vision-language models. arXiv preprint 2e.::2402.2338 2024.
Samuel Henrique Silva and Peyman Najafirad. Opportunities and challenges in deep learning ad- versarial robustness: A survey. arXiv preprint arXiv:2007.00753 2020.
Saurabh Srivastava, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402. 19450, 2024.
Alessandro Stolfo. Zhijing Jin, Kumar Shridhar, Bernhard Schölkopf, and Mrinmaya Sachan. A causal framework to quantify the robustness of mathematical reasoning with language models. arXiv preprint 2x.iv:22/0./2000 /2023, 2022.
Hao Sun. Reinforcement learning in the era of Ilms: What is essential? what is needed? an perspective rlhf, prompting, and beyond. arXiv preprint arXiv:2310.06144 2023.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wen- han Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint ar arXiv:2401.05500 3, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- -aptist Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. /805, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint 11Xv:2221..14275.. 2022.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Mcasuring multimodal mathematical reasoning with math- vision dataset. arXiv preprint arXiv:2402.1445 14804, 2024a.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce Ilms step-by-step without human annotations. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lonng Papers), pp. 9426-9439, 2024b.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing rision-languee model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024c.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1 / 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of thhthuugttt prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-24888 2022.
13

---

Published as a conference paper at ICLR 2025
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.024777 2023.
Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang. and Lei Han. Rorl: Ro- bust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 15:23851-23886 2022.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406. 10216, 2024a.
Rui Yang. Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards- in-context: oultiiobeeetive alignment of foundation models with dynamic preference adjustment. arXiv preprint ar X1v:24402.1027.. 2024b.
Longhui Yu, Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9561 2024.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsich. Robust deep reinforcement learning against adversarial perturbations on state observations. Ad- vances in Neural Information Processing Systems, 33:210224
2020.
Hugh Zhang. Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.003322 2024a.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024c.
Pan Zhang, Xiaoyi Dong Bin Wang. Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuan- grui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al. Internlm-xcomposee A vision- language large model for advanced text-image comprehension and composition. arXiv preprint arXiv:2309. 12, 2023.
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama adapter: Efficient fine-tuning of large language models with -notiiiiil attention. In The Twelfth International Conference on Learning Representations, 2024d.
Llama-adapter:
Renrui Zhang. Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal 11 truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 16244, 2024e.
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, al. Mavis: Mathematical visual instruction tuning. arXiv preprint 247::4007000779 2024f.
Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations, 2023.
Wanqi Zhou, Shuanghao Bai, Qibin Zhao, and Badong Chen. Revisiting the adversarial robustness of vision language models: multimodal perspective. arXiv preprint arXiv:2404. 200e:244441987777 /9287, 2024.
14

---

Published as a conference paper at ICLR 2025
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing Asiio-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Yongshuo Zong, Tingyang Yu, Ruchika Chavhan, Bingchen Zhao, and Timothy Hospedales. Fool your (vision and) language model with embarrassingly simple permutations. arXiv preprint arXiv:2310.01651, 2023.
15

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE WORK
Although our benchmark matches the difficulty levels of MATHVERSE and MATHVISTA, one limitation of our work is that the difficulty level is relatively limited compared to MATH-V (Wang et al., 2024a), due to the dynamic nature of the questions. Adapting very challenging questions into our program structures requires substantial human effort, which currently prevents from curating a large number of complex visual math reasoning questions. In the future, we hope to leverage strong foundational models aid in designing an automatic pipeline for dynamic math question design and generation.
Furthermore, the selection of seed questions can introduce unintended bias in DYNAMATH dataset. For instance, the most challenging question topic for VLMs, the Puzzle test, only dominates 3.4% of the whole dataset. It remains an open problem to study the bias in open-source datasets and requires further efforts. Regarding the variation generation process, we have identified limitation: we cur- rently consider only individual types of variants, such as Numerical Value Variants or Function Type Variants, for each seed question. However, in many cases, it is possible combine different types of variants, such as Color Variants and Numerical Value Variants. We will explore the integration of different variant types to further investigate the reasoning robustness of VI.Ms.
Scalability of DYNAMATH The current design of DYNAMATH relies heavily human effort to curate high-quality seed questions. However, it is important to scale up the design process of DynaMATH for constructing more comprehensive and challenging benchmarks. Below, we outline the primary challenges and discuss potential solutions:
key challenge in scaling DYN is incorporating dynamic visual elements for each question. Unlike text-only benchmarks, our dataset includes image for every problem with different variants (c.g., graphs, geometric shapes, function plots, real-life content). This requires careful design of the drawing program, adding significant manual effort, especially in quality control and verification, which complicates full automation.
promising solution is to leverage LLMs to automate the generation of dynamic benchmarks. LLMs have shown proficiency in generating text-based problems and writing code (Mirzadeh al., 2024). It is possible to break down benchmark topics and subtopics, prompting the LLM to generate diverse problem sets and corresponding Python programs for visual elements. However, the gen- erated problems should be dynamic, with parameterizable Python code to produce multiple image variants. To this end, DYNAMATH is valuable benchmark since seed questions can serve as high-quality human demonstrations to guide the L.MS for this task. This LLM--assssted approach could significantly reduce manual effort. However, some human intervention will still be necessary to ensure the selection of correct and high-quality samples from LLMs.
While we have to leave the LLM-assisted dynamic benchmark generation as a future work, DYNA- MATH can serve as good baseline that is completely crafted by human beings, and future work on automated dynamic benchmark generation may compare to DYNAMATH in terms of diversity and quality.
Future Work Moving forward, an intriguing approach to enhance VLM robustness involves lever- aging adversarial training (Zhou et al., 2024; Schlarmann al., 2024) on DYNAMATH, or utilizing reinforcement learning from human feedback (Ouyang et al., 2022; Sun, 2023; Rafailov et al., 2024; Yang et al., 2024b) with fine-grained process rewards (Uesato et al., 2022: Wang et al., 2024b; Luo et al., 2024), more robust rewards (Yang et al., 2024a; Zhang et al., 2024c). While prior successes in robust machine learning (Silva & Najafirad, 2020; Zhang et al., 2020; Yang et al., 2022) and trustworthy LLMs (Sun al., 2024) offer valuable insights, adapting these methods to VLMs in cost-effective way remains an open challenge and an area of potential exploration.
B VARIATION TYPES OF DYNAMATH
MYNAMATH introduces several types of variations based on the seed questions. In Figure 8, we illustrate six distinct types of variations. This diversity allows our dataset to effectively evaluate the visual robustness of VLMs.
16

---

Published as a conference paper at ICLR 2025
4.0
4,6
9.7
1.9
9.5
7.0
A
-10
8.3
-3.3
==
7.2
-5.3
4.2
0.3
7.4
6.2
0.7
6.0
0.4
0.2
-02
6.9
5.1
1.1
1.0
8.2
-5.
A=
5.2
7.8
2.6
-0.4
8.7
-8.5
3.5
7.2
0.0
(a) Numerical Value Variants
(b) Geometric Transformations
f1 = O:1)
0(M"2)
fl OlleN)
O(N)
O(N)
22oghi
r3 OCN"
ONN ZlegNi
O(NlggN)
OtNlogNi
OIN22)
14 2(N-2
(c) Function Type Variants
(d) Symbolic Substitution
|0: 08
19:45
02: 4 |
16:28
(e) Real-life Contexts Variants
Graph Structure Variants
Figure 8: Variantion types considered in DYNAMATH benchmark
C DETAILED EXPERIMENT SETUP
In this section, we provide more details about our experiment designs.
C.I PROMPTS FOR RESPONSE GENERATION
In our experiments, we prompt the VLMs to generate responses to different types of questions, such multiple choice, float, and text types. The prompts used for these question types shown in Table 5.
C.2 PROMPTS FOR ANSWER EXTRACTION
To simplify the answer extraction process, we use the following prompts to extract the answer and reload it in JSON format, which can be easily used for template matching with ground truth answers:
17

---

Published as a conference paper at ICLR 2025
Answer type
prompt
multiple choice
the problem a multiple choice problem, just provide the correspon- ing choice option, such A', 'B", 'C', "D'.
float
the answer is a numerical value, format it three-digit floating- point number.
text
Please answer the question in the following form: (specific requirement question).
Table 5: The prompt for different questions and answer types in answer generation.
Response Instruction
Please provide an answer to the question outlined above. Your response should adhere to the following JSON format, which includes keys: 'solution' and 'short answer'. The 'solution' key can contain reasoning steps needed to solve the question. The 'short answer' key should only provide concise response. Example of expected ISON response format:
"solution";
"[Detailed step-by-step explanation]" ,
"short answer": [Conciss Answer]
Closed-source models can consistently extract answers and regenerate them in JSON format. con- trast, some open-source models, particularly those with relatively low parameter sizes, struggle to output JSON format reliably. We have identified patterns in their inconsistent responses, which de- viate from the expected code scripts. By supplementing these scripts, efficiently check, annotate, and extract informal answers.
C.3 PROMPTS FOR FEW-SHOT CHAIN OF THOUGHT
Below we provide the few-shot CoT prompt for plane geometry topic. More demonstration examples for each topic can be found in Appendix E
18

---

Published as a conference paper at ICLR 2025
Few-shot CoT Prompt
Please answer the question based on the last image. To clearly explain the task, we provide the following example:
For image I: Please answer the question and provide the correct option letter, e.g., A, B, C. D, at the end. Question: As shown in the figure, AB the diameter of the semicircle, point O is the center of the circle, point C is point the extended line of AB, and CD is tangent to the semicircle at point D. If AB = 6.0, CD = 4.0, then the value of sin is
3 (C)
2
Choices: (A)
(B)
D
4
5 3
Answer: (B)
Let's think this step by step:
To solve this problem, Here's step-by-step breakdown of the approach:
make use of the properties of the circle and the tangent line.
1. Recognize the Geometry: - AB the diameter of the semicircle, so the length of AB 6. This makes the radius of the circle AO BO 3. C. tangent the semicirele at point D. Tangents to a circle are perpendicular to the radius at the point of tangency, meaning OD 1 CD.
Set Up the Triangle: We are asked to find sin( The triangle we should focus on is triangle ODC. By the properties of tangent, ZODC = 90
3. Apply Pythagoras' Theorem: In the right triangle ODC, we have the lengths OD = (since it's the radius the circle), and CD = (given). find OC, use Pythagorean theorem:
OC = OD² + CD=
0C 32 4
16 =
OC =5
4. Find sin(ZC): From triangle ODC, sin(LC) OD
00
sin(<C)) 3
5
Thus, the correct option is B.
For image 2:
For image 3: ...
Now please answer the following question based on the last image: Find the perimeter of the orange triangle. Please answer in floating-point number.
19

---

Published as a conference paper at ICLR 2025
C.4 MODEL HYPERPARAMETERS
We all parameters except temperature their default values. We temperature 0 for closed- source models and open-source models to reduce the randomness in the model generation. Table displays the parameters we used for generation in VLMs.
Table Hyperparameters various VLMs.
Model
Hyperparameters
GPT-4o
model gpt- remperatre 0.0, max_oke 4096
3,5
model
c -0000-00-----5000m---4444422 lemperature = 0.0, max, Jokens 1024
Gemini Pro 1.5
model gww.ni--..-ppr..eennn..... wenperauuee 0.0, max lokens 8192
Owen2-VLL-2BB
model C htwp://tttntttuttttttt tomperature 0.0, max_tokens 2048
QWen2-VIL-7B
model Coen:/0we22--111188777ttrrr,, temperature cemppee
0.0, max_tokens 2048
IntemV1.227661
model app.nsttaa///tteeerm////aaaaaaa ama3-766, lemperature 0.0, max tokens 1024
Intemv VL2440
model Opens/7aab//nneeeni222 103, temperature
1024
Intem IntemVL2-26B VL2226B
model 0pens/71ab/IInteeni222 263, temperature = 0.0. tokens 1024
12tem
model Open:G9tabbIItteennt22----- 8B, temperature 0.0, max tokens 1024
Deepseek-VE VL-Teeek-LL
temperature - U.U,
1024
Llama-3.2
model m wtt.s--eei--iiie--eeee-ee Instruss. lemperature 0.0. tokens 1024
Llava-v1.6-38
model = 110haahhhhtii///eeeeaaaaa v1.6 34b, temperature = 0.0, moxens = 1024
Llava
13B
lemperature max
Llava-v1.5-BB
model = muldd===llhhoriaa//ooveee..... lemperature 0.0, Jakens 1024
20

---

Published as a conference paper at ICLR 2025
D VARIANT EXAMPLES FOR DIFFERENT TOPICS IN DYNAMATH
In this section, we show sample problems DYNAMATH for different topics including multiple variants, including Solid Geometry (SG), Puzzle Tests (PT), Arithmetic (AR), Scientific Figures (SF), Graph Theory (GT), Algebra (AL), Plane Geometry (PG), Analytic Geometry (AG), and Statistics (ST).
Topic: Solid Geometry (SG)
Q129 from DYNAMATH: What is the volume of this azure right square pyramid?
h=13!
h= 19
h=9
18
18
8
Variant
Variant 2
Variant 3
Q188 from DYNAMATH: Are two planes parallel? choice: (A) Yes (B) No
Variant
Variant
Variant
Q320 from MANA Which line is longer, the pink or the red line? choice: (A) pink
(B) red (C) Their lengths are the same.
19, 0,
13. 7.
(3.0.4) 19.3.0)
10, 4, 1)
16. 7.
6
10
10
Variant
Variant
Variant 3
21

---

Published as a conference paper at ICLR 2025
Topie: Puzzle test (PT)
Q115 from DYNAMATH: The sum of the three numbers on each of the two lines of the cross is 76. Find the number in the center.
1
19
31
1
89
51
68
71
Variant 1
Variant
Variant
Q282 from DYNAMATH: Fill in the white spaces make the equations work. choice: (A) 13, 25, 5, and 12 (B) 25, 5, 12, and 12 (C) 13, 4, 25, 13.
19
+
29
5
+
6
8
+
=
20
+
+
+
+
+
+
+
+
1
+
20
15
38
+
2
36
+
32
45
Variant
Variant
Variant 3
Q284 from DYNAMATH: Find the missing value.
44
6
42
25
21
20
21
10
11
59
35
45
40
10
40
12
12
28
2
30
13
Variant
Variant
Variant
22

---

Published as a conference paper at ICLR 2025
Topie: Arithmetic (AR)
Q7 from DYNAMATH: In the addition sum to the right, three digits have been replaced with star. What is the value of star?
1 8 + 1 * 0
1 7
1 3
+ 1 x 2
+ 1
+ 1 * 9
+ 1 7
+ + x1
4 6 7
526
5 9
Variant
Variant 2
Variant
Q25 from DYNAMATH: What is the missing computed symbol? Choices: (A) + (B) (B) (C)
/D)/
4 8
4 12 8 + 3 -3
-0
3 9- 6 4
Variant
Veriant 2
Variant
Q316 from DYNAMATH: According to the boarding pass, how long is the flight time of this airplane? Answer the question using the total number of minutes.
BOINDING PASS
FROM JFK NEW YORK
JFK
LHR
LHR LONDON
Pestager
Tigg
Variant
Jon 2025
02 2025
FROM
10
JFK
LHR
JFK NEW YORK
LHR LONDON
Variant
07, 2025
Jon 07 2005
04:89AM
FROM:
TO
JFK -C
LHR
JFK NEW YORK
LHR LONDON
Teuesh
Variant
08.21
OF 2005
Jen 07. 2025
23

---

Published as a conference paper at ICLR 2025
Topie: Scientific figure (SF)
Q323 from DYNAMATH: Two containers of the same gas (ideal) have these masses and temperatures Which box has atoms with the largest average thermal energy? choice: (A) A (B) (C) Their average thermal energy is the same.
A 13
Box
Box o
A
p
21 TC
41
Variant
Variant
Q325 from DYNAMATH: Three equally spaced identical long straight wires carry different currents. In which direction will the middle wire try to move when the currents are switched on? choice: (A) to the left (B) to the right (C) stay the same
92A
1A
55A
29A
82A
95A
3A
Variant
Variant
Variant
Q331 from DYNAMATH: The graph shows the force on an object of mass as a function of time. For the time interval 0 to 10 s, what is the total change in the momentum of the object?
Variant
Variant 2
Variant
24

---

Published as a conference paper at ICLR 2025
Topie: Graph theory (GT)
Q42 from DYNAMATH: Is the graph shown connected? choice: (A) Yes (B) No
Variant
Variant
Variant 3
Q137 from DYNAMATH: What is the first edge added to the MST when running Kruskal's Algorithm? In the case tie, choose the edge which comes first in alphabetical order i.c. if you had to choose between AS and AE, then you would choose AE first.
19
24
Variant
Variant 2
Variant
Q259 from DYNAMATH: The tree shown in image reserves expression. Calculate this expression and output the result.
Variant
Variant
Variant
25

---

Published as a conference paper at ICLR 2025
Topic: Algebra (AL.
Q305 from DYNAMATH: The store has combinations of candies. Each candy type has the same price. Find the price of the fourth combination.
1. * *
*
S15
1.
1.
2. *
$44
2. *
* *
510
3.
S71
3.
$77
3.
409
4. * $7
4. *
4.
Variant
Variant
Variant 3
Q351 from DYNAMATH: Which function has the highest order or growth? choice: (A) fl
(B) 12 (C) B (4) 14
f1 0(Ng2)ogg)
f2 = 011)
f1 0(2"N)
f2 O(N-3)
OlNg2)ogN)
f2 O(N)
O(NI)
O12
f3 orlagN)
= o(1)
002 NN
OllogN)
Variant
Variant
Veriant
Q465 from DYNAMATH: 210 customers were surveyed about their product preferences. The results are displayed in the Venn diagram below. How many more customers prefer only Non-Organic products than only Organic ones?
40
49
83
45
79
74
50
23
79
Organic
Organic
Non-Orgenic
Organic
Neither:
Non Non-Organic
Variant
Variant
Variant
26

---

Published as a conference paper at ICLR 2025
Topic: Plane geometry (PG)
Q28 from DYNAMATH: The two rectangles shown the picture have the same area. what the ratio
Variant
Variant
Variant 3
Q43 from DYNAMATH: What fraction of the shape is azure?
Variant 1
Variant 2
Variant
Q53 from DYNAMATH: What the area of blue ring?
R=7 5=5
R=6 r=4
R-5 r-3
Variant
Variant
Variant 3
27

---

Published as a conference paper at ICLR 2025
Topie: Analytic geometry (AG)
Q68 from DYNAMATH: What the green curve? choice: (A) parabola (B) line (C) logarithmic function (D) trigonometric function.
Variant
Variant
Q87 from DYNAMATH: What is the limit of the function as approaches I from the left side?
2x. 3,85-
Variant
Variant
Variant 3
Q111 from DYNAMATH: The image shows the derivative of f(x). Where is the local max
of f(x at?
Variant 1
Variant
Variant 3
28

---

Published as a conference paper at ICLR 2025
Topie: Statisties (ST)
Q72 from DYNAMATH: According the markov chain shown in the image, what is the probability of the event 'A to B"?
Al
02
67
Variant
Variant
Variant
Q161 from DYNAMATH: On which number is the spinner more likely to land?
5
6
7
7
1
3
8
8
Variant
Variant
Variant
Q447 from DYNAMATH: The Gantt chart below represents different tasks. Which task
starts the earliest? Choices: (A) Task A (B) Task B (C) Task C (D) Task D (E) Task E
10
12
14
14
10
14
Variant
Variant
Variant 3
29

---

Published as a conference paper at ICLR 2025
E
MORE DEMONSTRATION EXAMPLES FOR FEW-SHOT EXPERIMENT
Demonstration Examples for Topic Algebra
The two markers with question mark have the same number. Which number do you have to put instead of the question mark that the caleulation is correct?
10
+
2
18
Answer: 3
Let's think this step by step: To solve the problem: The equation is: 10+?+? + 18 Where both question marks represent the same number. Let's denote number as x. The equation becomes: 10 +
+ 18
Combine like terms: 12 + 2x 18.
Now, solve for 2r 18 12, hence 2x 6, and therefore 3. So, each question mark should be replaced with to satisfy the equation.
Demonstration Examples for Topic Arithmetic
How many plums (see the picture) weigh as much as an apple?
Answer: 3
Let's think this step by step:
Since the scale is balanced, the total weight on the left side equals the total weight on the right side. Let the weight one apple A, and the weight of one plum be P.
2. The total weight the left side of the scale: 3.4 + 2P.
The total weight on the right side of the scale: + 8P.
Since the scale is balanced, A + SP..
set the two expressions equal to each other: 3.4 + 2P
### Solving for A in terms of P.
1. Subtract A from both sides: 2A + 2P- 8P.
2. Subtract 2P from both sides: 2/4 6P.
3. Divide both sides by 2: A = 3P.
### Final Answer: One apple weighs much as 3 plums.
30

---

Published as a conference paper at ICLR 2025
Demonstration Examples for Scientific Figure
If the population of the red squirrels increased, what would happen to the amount of pine available?
(A) The amount pine available would remain the same.
(B) The amount of pine available would decrease.
(C) The amount of pine available would double.
(D) The amount of pine available would increase.
Tegghs
Pllnay
plant
Ene lhe foo fart Die Brysttenst
Answer: (B)
Let's think this step by step: In the food web Isle Royale National Park, the red squirrels are primary consumers that feed on pine, which primary producer. If the population of red squirrels increases, the consumption of pine would increase because more squirrels would be feeding decrease.
As result, the amount of pine available would
Conclusion: If the population of red squirrels increased, the amount of pine available would decrease. Thus, the correct option is B.
31

---

Published as a conference paper at ICLR 2025
Demonstration Examples for Topic Puzzle Test
Please determine the total weight in the fourth image.
10kg
20kg
24kg
Auul
Answer: 27
To solve the puzzle and determine the total weight the fourth image, let's break down based on the given weights:
1. First image: A frog and sheep together weigh 10 kg.
2. Second image: A horse and frog together weigh 20 kg.
3. 3.Trrrd image: A horse and sheep together weigh 24 kg.
We
set up three equations to represent these situations:
Frog + Sheep 10 kg (Equation 1)
Horse Frog 20 kg (Equation 2)
-
Horse + Sheep = 24 kg (Equation 3)
Solving the system of equations:
1. From Equation 2, we can express the weight of the horse:
Horse = 20 kg Frog (from Equation 2).
Substitute Horse in Equation 3:
(20 kg Frog)* Sheep 24 kg.
20 kg Frog + Sheep 24 kg.
-Sheep Frog 4 kg (Equation 4).
3. Now use Equation (Frog Sheep = 10 kg) and Equation 4 (Sheep Frog 4 kg): Adding these two equations: (Frog + Sheep) (Sheep Frog) 10 kg + kg. gives:
2 x Sheep= = 14 kg, so Sheep = kg.
Substitute the value of Sheep kg) into Equation
Frog + 7 kg 10 kg, so Frog
kg.
Finally, substitute the value of Frog (3 kg) into Equation
Horse 3 kg 20 kg, so Horse 17 kg.
Total weight in the fourth image:
The fourth image has the horse, the sheep, and the frog together.
The total weight is:
Horse (17 kg) + Sheep (7 kg) Frog (3 kg) 27 kg.
Thus, the total weight in the fourth image is 27 kg.
F ERROR TYPE EXAMPLES
In this section, we present various examples of error types encountered Claude 3.5, including figure reading error, calculation error, reasoning emror, hallucination error, and knowledge error.
32

---

Published as a conference paper at ICLR 2025
Figure Reading Error
Q96 from DYNAMATH: What is the distance between the x-intercept and y-intercept of blue line?
Claude-3.5 Answer:
1. Identify the z-intercept and y-intercept: The x-intercept where the line crosses the
r-axis (y = 0). From the graph, this appears to be at (3, 0). The y-intercept is where the line crosses the y-axis (z = 0). From the graph, this appears be (0. 4).
2. Calculate the distance using the distance formula: distance = V x1)² 2 (y2 y1)²
where (x /1) is (3, 0) and (x2, y2) is (0, 4). 3. distance /(0 3)² + 0)2 5. Therefore, the distance between the x-iintecept and y-intercept is units.
Ground Truth: Distance = v Rz 2.828.
Error Analysis: The model misreading the coordinate of the points and gives the wrong answer.
Figure Reading Error
Q185 from DYNAMATH: Based on the measurement results shown the diagram, this blue angle is ().
Claude-3.5 Answer: To determine the angle represented by the blue lines in the protractor
image:
1. Observe that the protractor is a 180-degree semicircle.
Note that one blue line is at the O-degree mark.
3. Locate where the other blue line intersects the protractor scale.
This intersection occurs approximately 25 degrees.
5. Since the protractor measures angles from 0 to 180 degrees, the angle formed by the blue
lines directly read as 25 degrees. Therefore, the measurement of the blue angle is approximately 25 degrees.
Ground Truth: 1550 degrees.
The model fail to distinguish between set of the numbers for degree reading. In addition, even for outer degree numbers, the model gives an inaccurate number reading.
Error Analysis:
33

---

Published as a conference paper at ICLR 2025
Calculation Error
Q108 from DYNAMATH: The function f(x) is cubic function. What is the value of f(x) when x x=
f(x)
3
2
63
3
159
Claude-3.5 Answer:
1. First, let's recognize that for a cubic function f(x)) ar + buz + cx d. we need to
find the values a, b, c, and
2. We can the given points to set up system of equations: f(0) d = 3,
f(1) = a + + + 19,
f(2) Sa + 4b 2c
63.
f(3) 27a + 9b 3e 3 159.
3. From the first equation, we know d
4. Subtracting equations:
f(1) f(0) a b c
f(2) f(1) 7a + 3b c=
f(3)-f(2) 19a + 5b c= 96
5. S.oovving this system, we get: a = 7,
So, our function (() 4x 7x +r +
7. Now, we can calculate f44) Therefore, f4) 391.
=
391
Ground Truth: Let f(x) = or + bx. + cr + d, using the number given in the table, one can get: 4,
10, d = 3. So, function is f(x) 4x 2x 10x 3 and
f(4) = 331.
Error Analysis: The model fails to solve the system of equations for value b and c.
34

---

Published as a conference paper at ICLR 2025
Reasoning Error
Q146 from DYNAMATH: Find the missing number.
17
72
Claude-3.5 Answer:
The diagram shows central number 4 with six numbers radiating from it.
2. The visible numbers are 7, 17, 28, 68, and 71.
Let's check if there's consistent mathematical relationship between 4 and these numbers: x = 28 (one of the given numbers) 4 x 17 = 68 (another given number)
This suggests that each number is either a factor or multiple of 4.
5. 71 doesn't 't fit this pattern directly, but 71 + 4 = 75, which is 3 25. Following this pattern, the missing number should also be related by multiplication addition.
7. number that fits this pattern and completes the sequence is 13, because: 4 13 52, and 52 + = 56, which is 4 14.
8. Therefore, the missing number likely 13.
Ground Truth: 71 x 4 = 284.
Error Analysis:
The first 4 steps from the model are correct, and the model can observe that each number is either factor multiple of correctly. However, step 5-8 is not correct and the reasoning to get its final answer 13 is not correct.
35

---

Published as a conference paper at ICLR 2025
Reasoning Error
Q34 from DYNAMATH: What the area of this parallelogram?
19ft
8ft
3ft
Claude-3.5 Answer: To find the area of parallelogram, we use the formula: Area = base height. In this case:
1. The base is given as 19 ft.
The height the perpendicular distance between the base and the opposite side, which is 8 ft.
Area 19 ft 8 ft 152 sq ft.
Ground Truth: To find the area of parallelogram, we first need to compute the height
8 3 7.416. Then the area of the parallelogram is 7.416 140.904 sq ft.
Error Analysis: The model fail to reason that we need to compute the height using the given numbers in the plot.
Hallucination Error
Q146 from DYNAMATH: Find the area of the shaded part.
r=2
Claude-3.5 Answer:
1. The circle has radius
2. The shaded area is sector of the circle minus right triangle.
The sector angle is 1/8 of the full circle (45%).
4. Area of the sector m1/ = 1.5708. Area of the triangle (1/2)r2 (1/2)2² 2
Shaded area Area sector Area triangle = 1.5708 -0.42927.
5.
Taking the absolute value: 0.42927 0.4292.
Ground Truth: Area of the shaded part (1/4) (1/2)/2 1.1416.
The model hallucinates the sector angle be 45 which should be 90% In addition, instead of realizing that the area should be a positive number, the model take the absolute value of negative number and return the answer.
Error Analysis:
36

---

Published as a conference paper at ICLR 2025
Hallucination Error Q120 from DYNAMATH: Is this graph a cyclic graph? choice: (A) Yes (B) No
Claude-3.5 Answer:
To determine this graph is cyclic, we need check if there's path that starts and ends at the same node without repeating any edges. Looking at the graph:
1. It has nodes connected
star-like pattern.
2. The central node is connected to all other nodes.
3. There are no connections between the outer nodes.
4. To form cycle, would need be able to start at node, traverse through others, and return to the starting node without repeating edges.
5. In this graph, any path that tries to return starting node would have go through the central node twice, which means repeating an edge.
Therefore, there no cycle in this graph. Since there is no cycle, this graph is not cyclic. The final answer is (B) No.
Ground Truth:
Since there is path that starts and ends at the same node without repeating any edges, this a cyclic graph. Hence the correct solution (A) Yes.
The model hallucinates the path in the graph and concludes to the wrong answer.
Error Analysis:
37

---

Published as a conference paper at ICLR 2025
Knowledge Error
Q289 from DYNAMATH: The diagram shows Exam score of students. What is the average score of this exam?
Exam Score
Stem
Leaf
2
3
3
4
2
3
7
4
8
2
9
2 4 4
Claude-3.5 Answer:
1. Sum up the scores: + + 2-
2 6
2 + 4 + =2
2. Count the number of students: There are 9 stem numbers, representing students.
3. Calculate the average: Average Total Score / Number of Students = - 5.777777...
4. Round three decimal places: 5.778.
Ground Truth: 54.6.
Error Analysis: The model fails read stem- stem-and- and-leaf plot.
G QUESTION VARIANT EXAMPLES
We present several examples of generated variants of seed questions from dataset along with corresponding responses from GPT-4o (Figure 9), Gemini (Figure 10), and Qwen2-VL-72B (Figure 11). These examples illustrate that the leading VLMs exhibit lack of robustness to variations in image input, demonstrating unreliable perception and reasoning abilities.
38

---

Published as a conference paper at ICLR 2025
H ADDITIONAL EXPERIMENT RESULTS
In this section, we present additional experiments.
H.1 REASONING ROBUSTNESS ON DIFFERENT VARIATION TYPES
In terms of different variant types DYNAMATH, as shown in Figure 12, we find that both GPT-4o and Swen2-VL-722) are sensitive to variations graph structure, geometric transformation, and function type. Additionally, Qwen2-VL-72B is vulnerable to symbolic substitution variants. These weaknesses suggest directions for future improvement of these models.
100
graph strueture veriants peometiic transfo maaton Tunction type variants
color variants nureercc value variants
real life contect va verlants symbolic substitution
Reasoning Robusiness (%)
80
67.7
68.2
60.8
61.4
55.1
40
41.3
37.0
34.1
33.3
GPT-4o
Qwen2-VL-722
Figure 12: Comparing reasoning robustness (RR) across different variation types.
H.2 ADDITIONAL FAILURE CASE ANALYSIS
In this section, we present more results on the failure case analysis.
Failure v.s. Difficulty Levels We conducted in-depth failure analysis based on problem diffi- culty, categorized into elementary (63 questions), high school (277 questions), and undergraduate (161 questions) levels. The detailed results presented in Figure 13.
elementary
High school
undergraduste
Failure Number
20
16
12
4-
Calculation Error
Figure Reading
Hallucination Emrrr Knowledge Error
Reasoning Error
Figure 13: Failure cases across different difficulty levels.
The results indicate that high school and undergraduate problems account for the majority of failure cases. Among the error types, knowledge errors are the least frequent, implying that VLMs have solid grasp of mathematical concepts and facts. However, reasoning, hallucination, figure reading, and calculation errors are more prevalent, highlighting that VLMs may struggle with interpreting visual data and performing accurate calculations and reasoning.
Failure v.s. Problem Topics
We performed an in-depth analysis of failure cases based on problem types. The detailed results can be found in Figure 14.
39

---

Published as a conference paper at ICLR 2025
Table Reasoning Robustness RR models on DYNAMATH with 5,010 generated questions, testing with 0 temperature. "ALL" represents overall accuracy. Question topics (PG, SG, EL, etc) are defined Table
Model
ALI.
PG
SG
AG
A1.
PT
GT
ST
SF
AR
EL.
HI
UN
Closed sourced Large Muliimodal Models (LMMs)
Zero-shot GPT-4o
$4.8
66.4
64.1
42.2
71.4
22.7
32.3
55.4
56.9
75.0
61.1
55.5
84.5
Zero Claude
44.3
Zero-shot Gemini Pro 1.5
44.5
54.2
46.9
31.8
55.4
28.6
35.1
50.5
31.0
56.7
65.7
45.1
58.5
3-shot CoT GPT-4o
49.8
53.7
67.4
37.5
65.3
34.5
33.7
51.9
43.8
80.0
71.9
49.1
83.9
3-shot CoT Claudo-3.5 3,5
51.7
55.6
22.4
68.5
0.0
17.9
71.6
47.9
55.9
63.0
53.4
88.7
3-shot CoT Gemini Pro 1.5
40.1
51,9
58.8
25.5
53.8
27.0
32.4
41.2
32.4
56.0
56.5
39.6
60.0
Open-sourced
Mutrimodol Models (LMMs)
Owen2-V1.-720 V2-
51.8
56.8
68.5
30.4
54.4
0.0
37.0
62.7
47.2
78.0
67.4
52.8
64.8
Qwen2-V1L-221 (3-shot CoT)
43.4
54.8
59.7
17.4
59.4
0.0
18.9
48.9
42.0
72.5
67.7
43.8
49.9
QWen2: V1-7B
54.8
27.0
V
45.8
55.4
57.7
35.3
55.1
16.7
24.5
49.2
36.3
74.6
65.8
43.7
80.0
Intern Intern V1.2-112-408
33.9
45.6
31.3
23.9
32.0
0.0
27.2
37.2
30.9
50.5
56.1
33.9
37.2
V1.emmV1.2-26
35.0
54.3
0.0
16.6
25.3
0.0
40.0
38.5
28.1
66.7
67.1
31.9
44.2
VL2-
26.1
38.3
53.6
15.9
33.5
0.0
24.8
20.1
28.4
41.2
46.6
25.1
34.9
1.ama.3.2-908
16.9
33.5
32.2
Deepeek-VL--B---e
19.5
48.8
0.0
11.7
0.0
0.0
31.8
16.4
9.2
25.6
28.1
15.2
31.1
Jlav-vl1.-333
22.1
48.5
52.6
14.9
13.2
0.0
12.7
17.4
24.0
33.3
44.2
21.3
22.4
sovvvvl.--vicunaa
53.1
17.6
0.0
0.0
17.5
Llava 1ava-x1.5-777
10.8
37.0
0.0
10.6
0.0
0.0
12.9
4.6
0.0
35.7
16.8
13.6
10.6
20
18
tes: griph theory analytic gearetry
SUIkEEs adennticc fgure
gouretty pare goomatry
antomedi
16
16
Failure Number
12
4-
3
g
Calculation
Figure Reading
Hallucination Error
Knowledge Error
Reasoning
Figure 14: Failure cases across different problem topics.
From Figure 14, we have the following observations based the failure reasons and problem types:
The Puzzle test shows concentration of reasoning errors, with no other error types present, suggesting that VLMs may struggle with the logical and abstract reasoning required for puzzles.
Graph theory, analytic geometry, arithmetic, and statistics problems exhibit more errors related to figure reading, indicating difficulties in interpreting visual data.
Solid geometry and algebra problems are prone to calculation errors, highlighting potential issues with numerical operations in handling such questions.
Plane geometry has high incidences of hallucination and reasoning errors, suggesting chal- lenges in both generating relevant information and applying logical reasoning.
H.3 DETAILED REASONING ROBUSTNESS RESULTS OF ZERO TEMPERATURE
As shown Table 7, we present the full results reasoning robustness (RR) defined in Eq 2. We can better understand how the reasoning robustness correlates with question types and difficulty levels.
40

---

Published as a conference paper at ICLR 2025
H.4 RESULTS OF DIFFERENT PROMPT TEMPLATE
To investigate other prompt templates, we designed the following prompt aims to improve the rea- soning and reduce memorization issues for VLMs:
Prompt Template for improving reasoning and reduce memorization
You are solving advanced visual math problems that require logical reasoning and detailed analysis of the provided image and question. Carefully examine the image and break the problem into smaller steps to ensure accurate and thoughtful reasoning. Avoid relying memorized answers, patterns, shortcuts. Instead, justify each step of your solution explicitly based on the information the image.
Task: Please answer the following question: [new question), ensuring your explanation according the provided image and question. Focus on reasoning rather than recalling.
We evaluated the performance of GPT-4o and Qwen2-VL-72b on 10 variants with temperature using this newly designed prompt, and the average accuracy rate, worst-case aceuracy, and reasoning robustness can be found in Table 8. The results show that both average accuracy and worst-case accuracy have improved with the use of the designed prompt. This suggests that carefully crafted prompt can enhance the performance of VLMs. However, there is no significant improvement in reasoning robustness, highlighting the ongoing limitations in the robustness of current VLMs.
Table 8: Performance comparison between Zero-shot and Zero-shot with New Prompt for GPT-4o and Qwen2-VL-72b.
Model
Zero-shot
Zero-shot w New Prompt
Aavg
Awst
RR
Aavg
Awst
RR
GPT-4o
63.7%
34.7%
54.8%
65.6%
36. %
55.0%
Swen2-VL-726
55.1%
28.3%
51.8%
57.8%
29.5%
51.0%
H.5 MORE ON MEMORIZATION PHENOMENON
We also tested the newly designed prompt with problems where memorization was evident. Unfor- tunately, the model still tends to provide the same answers, regardless of changing conditions:
For seed question 78 in DYNAMATH, GPT-4o consistently argues that shifted absolute function not differentiable at
For seed question 12 in DYNAMATH, Claude-3.5-Sood 5-Snnntnett repeatedly reads the period of sinusoidal function as 2n, regardless of the actual period shown the image. We believe more systematic study is necessary to effectively address this issue.
A screenshot of the web version of GPT-4o and Claude-3.5 for these two examples be found in Figure 15 and Figure 16. More systematic studies are necessary to effectively address this issue.
H.6 EVALUATING THE ROBUSTNESS OF DYNAMATH
An important question to ask whether dynamic benchmarks are robust enough. In other words, if we provide synthetic data generated by DYNAMATH, of DYNAMATH? The best way to investigate this to perform thorough experiments, including pre-training and fine-tuning VLMS using DynaMATH. However, due limited resources, unable to perform full-scale pre-training or fine-tuning of VL.Ms to thoroughly investigate potential data leakage involving DYNAMATH. As a proxy investigation, we conducted in-context learning experiment.
models perform well other variants
Specifically, we used variants I to 3 of DYNAMATHI few-shot demonstration examples and tested the VLM's response to a question from variant 4. As a controlled experiment, we directly used a question from variant 4 both as demonstration example and test question (i.e., asking the model the same question was shown). This setup provides a preliminary indication of potential data leakage, well the expected performance if the model had memorized the data. We performed
41

---

Published as a conference paper at ICLR 2025
Table 9: In-context evaluation of DYNAMATH
Model
Original Performance
Few-shot
Controlled Experiment
GPT-4o
64.5%
65.3%
73.1% 1%
Qwen2-72b
53.7%
57.4%
77.0%
Table 10: The Variance of Average Accuracy for different models participating 5 repetitions tests with 0) temperature
Model name
GPT-4o
Gemini
Qwen2-72B
Intern VL2-76B
Variance of Average Accuracy (%)
1.86
1.26
0.89
2.12
these experiments closed-source model, GPT-4o, and one open-source model, Qwen2-72b. The results can be found in Table 9.
These results indicate that even with few variants provided as context, the performance improve- ment is marginal compared to the original performance and baseline results. Nevertheless, whether pre-training or fine-tuning can "hack" dynamic benchmarks needs more systematic studies, which is important for future work.
H.7 VARIANCE OF AVERAGE ACCURACY
In our main paper, we have reported repetition consistency as measure of randomness of model output. Here, we also calculate the variance of the average accuracy over five repetitions Table 10. Specifically, for set of 501 questions, we conducted five separate evaluations and determined the variance of their average accuracies. The resulting variance for GPT-4o, Gemini, Qwen2-VL, and Intern VI.2 is minimal, ranging from approximately to 2 percentage points. This small variance enhances the reliability our results.
H.8 MORE RESULTS ON CIRCULAR CONSISTENCY
In DynaMath, our primary focus is on image-based variants, such as Numerical Value (in the image) Variants and Geometric Transformations, so we initially did not test for circular consistency. Cir- cular consistency applies only multiple choice questions (MCQ) and the contents of the question still static; only the order of the choices changed. To address your concern, evaluated the circular consistency (Liu et al., 2023) of two representative models, GPT-4o and Qwen2- VL 76B, specifically using MCQ questions from DynaMath. Interestingly, both models exhibited high repeti- tion consistency under circular shifts, achieving scores of 90.2% % and 92.2 %, respectively, other words, the model's output is consistent in cases regardless of the order of the choices. The current models seem to be robust to the circular shifts MCQ problems.
42

---

Published as a conference paper at ICLR 2025
Question 169
Answer type: "muitiple choice"
The purple and orange curves are f(0) and g(x), Is (())(()) even or odd? Choios: (A) odd (8) even (C) neitierr
Subject: analytic geametry'
Level: "high school"
Response from GPT--
Variant 1
Vaniant 2
To detersine if prodhct the furctioons f(x) and a(x) is even or odd, ve need analyze the
Io deternine if the function ((0)g(x)) is even or odd, we need to analyse the properties of the functions f(x) and R(K).
properties of each funotion.
Cleck if T(x) is even or odd: A Functios ((0) is even ir ((-x) 100) all
even if h(-x) h(x) for all 0ld Finasions Fanction b(8) is if b(-x)
Punction: function
odd r(-x) -f(x)
- -l(4) for all x,
The purple curve seees to be symeetric about the origin, suggesting that f(x) is likely odd.
Froe the gosph: The purple caree, ((0). is syanstric about the yraxis, indicating that 200) Is 110). The orange curve, 2(x). 1s syametric about or origin, indicating that g(x) is odd: et X) ((x).
((-a)
Exarining the
it sppears to be line passes through origin with consistect sTope; indicating that ((*) ix 1ikely add.
Kow, Set is consider the product (())g()); r(-0)((x) f(0((2(0)) -f00g())-
Determine the product
f(x)g((): If both f(x) and g(x) are odd, their product f(x)g(x) is since (odd)
shovs that 1002t)) #(x)g(x), which means the product funotion is odd.
(odd)
Tserafore, ((x)g(+))
Thus, f(x)g()))
(3) even.
(A) add.
x
Variant 3
Variant 4
To determine if the product f(x) and p(x) is even or odd, se observa each function:
deterniae whether the product [(x)g(s) is od4, asalyse the g(x) individuslly.
funesions f(x)
The orange curve, g(*), is Tineer finettion
passing through origin, suggesting becsuse R(-x) -R(4).
odd
& function is even if f(-x) a f(x) for all in its dosain. Au even Cunction is syanstri about the raaii. Feom the grape, the around the y-axis, indicating it is an even function.
1(X) is syszetzic
The yerple aurve, f(x), is horizontal line, suggesting it is constant. If constant, I(x) is function because f(-x) f(x).
A function is odd if f(-x) -f(x) for all in its domain. an edd. function as synnetric about the crigin. The curve suggesting
The product
even function and odd Function is always odd because f(-x)g(-x) = f(x)(= =
to have origin syenetry, add function.
F(x)g(+)) which satiefies the definition of an add function. Therefore, ((x)g(x) is
The product of an even fanction and odd function always because:
even.
[( [(-))(( x) - r(0) ((()) = -0(0)(())
Therefore, the Function #(0)(()) is (A) odd.
Variant 5
Variant 6
To determine if the product f(x)g(x) is odd, check whether f(x) p(x) have specific symmetry properties.
deternine the product ((00(()) is odd, neither,
analyse the properties of Functious ((x) and g(x).
à functioo is even if f(-x) = f00 for all x in its dorain, and it is add if F(
à function h(x) is even if h(-x) h(x) and odd if h(-x) -h(x). Pron the graph, appears:
f(x). Pron the graph:
represeating ((x) appears syaseteie about origin, siggesting function. The orange curve reprasmniing ing g(s) coustast. across the axis, suggesting it is an even function (g(x) p(-x) since its value doesn't 't change).
The purple
odd
purple curve r(x) ayaactric about the origin, suggesttngg it night be an odd function.
The orange curve R00) is syaaetric about the yeaxis, function.
indicaring it night
For product r(x)a()) be aven, r(02(0) f(-x)e(=x); for it. to be odd, #00gg00 -f(xx)=(xx).
The product of even function function is alvays odd.
Since no have one even funcsion (g(x)) and odd funosion (f(x)), their
Thacefore, ((s)g(s) is (A) odd.
product, f(0g(s)) will be odd, Tharefore, ((x)g(s) is
(A) odd.
Figure 9: Example of the generated variants of Question 169 and the corresponding responses from GPT-4o.
43

---

Published as a conference paper at ICLR 2025
Answer type: "multiple choice"
Question 75
Are the red line the blue Line parallet?
Subject: 'analytic geometry"
Choice: (4) Yes (B) No
Level: "high school"
Response from Gemini:
Variant 1
Variant 2
Parallel lines have the same slope. The red 1ine passes through points (8,-2) (1,1). Its alope is (I
Parallel lines have the same slope. bas slope of -2 and the bime T5ne has slope approxinately 4, Since the stopes are different, the lines not parallel.
The passes through points (1.0) and (2,3)- Its slose (3-01/12-1) the slopes same, the lines are parallel
2))/(1
3/1
Therefore, the (B) No.
answer
(A) Yes.
X
Variant 3
Variant
Parallel Lines have the slope. The red line decressing, indicating a negative slope, while the blue 1ine horizantal, meaning it has slope of 0. Since the limes have different slopes, they not parallel.
Parallel lines have the slope. has positive slope and blue line has negative slope. Therefore, they are not parallel.
Therefore, the answer is No.
Therefore, ansver (B) No.
Vaniant 5
Variant
Parallel lines have the stope, gees down units for I unit it right, so slope is -6, The blue line also down units for every unit noves to the right, so it also has slope of
Parallel lines have the slape. The red line is horizontal and has slops of 8, The blue Hine is horisontal nonzero slope.
line
Thersfore, the rod line 1ine parallel.
Since both lines have the same slope, they are parallel.
Therefore, ansver (B) No.
Therefore, Yes.
Figure 10: Example of the generated variants of Question 75 and the corresponding responses from Gemini.
44

---

Published as a conference paper at ICLR 2025
Answer type: "float"
Question
Subject: onalytic geometry'
Khat is the global einiaux of this function?
Level: "high school"
Response Qwen2-VL.-7BB
Variant
Vaniant
The function show in the is simeoidal winimus
The glebel ninimum of the Function appears the point where function resches i2a on. examining the graph, the losest point seems be approxinataly -2/2. walne approxinately -8.
MA
simssoidal
the lovest point on the saon, Prom the graph, it
can be observed that the
plobal nimimum value apprexinately
the
Vaniant 3
Variant 4
The appears periodic function with a period of n. Observing the graph, the function nesches its rinimun volue amz winimun veloe the function 15 approxinately -7.
The global ainimus of furction point the grapa. Observing the graph, the Towest paint Ls at the coordliate
/2.
VW
on.
Variant
Variant
The global minimum of the
function appeans to be coslue Fanetion, The global sinimms of cosine Funation 1. Bovever, since graph vertically yy stratched, the global minimus is -6.
function observed graph. examining graph. it appears that the
wWW
Loveett point in at the bottoo of the trough. which
The approximately valise
this point is -8.
Variant
The global einimon the Functiio can be found by obsarving the graph. The fanction reaches its the dips the most. by looking the graph, me can see that
The global minimum of function appears aporodiaately -6.6. observed at the loweat: point of the graph.
points
M W
the Function reaches itx
X
value approxintelly
Figure 11: Examples of the generated variants of Question 346 and the corresponding responses from Qwen2-VL--7B are provided. Notably, variant 3 derives the correct answer but has an erro- neous perception.
45

---

Published as a conference paper at ICLR 2025
You are solving advanced visual math problems that require logical reasoning and detailed analysis of the provided imaae data. Carefully examine the image and break the problem into smaller steps to ensure accurate and thoughtful reasoning Avoid relying on memorized answers, patterns, or shortcuts. Instead, justify each step of your solution explicilly based on the information in the image.
Taskc Please answer the follow ng question is the function cifferentiable atx=0? chcices (A) Yes, (B) No, ensuring your explanation references specific emmnnt the visual provided Focus on reasoning rather than recalling
determine whether function differentiable at 0, let's analyze the graph detail:
Shape of the Graph: The graph clearly has two distinct linear segments meeting at 0.
Continuity: For a function be differentable at point must first continuous. From the there is jump gap this point.
graph, the function continuous
Slope of Singments:
The slope the segment
negative decreasing).
The slope of the segment for æ >> posi
positive tive (line increasing).
Sharp
transition
segment
right segment occurs abruptly
at 0, form sharp corner. At this point, the slope of the left segment does not equal the slope right segmant Thus, the derivative does not exist at
Condusion:
The function not differentiable at because the graph exhibitss a sharp corner at this point,
resuling undefined fineedee derivative
Correct answer: (8) No
Figure 15: Memorization phenomenon on GPT-4o with seed question 78 in MANAMATH.
46

---

Published as a conference paper at ICLR 2025
require logical reesoning detailed anelysis the image Carefully image and problem steps
solving
thoughtfull reesoning. relying explicitly the information the image. period function y acos/bx), question floating- point number", ensuring your explanation according provided image
enswers, s, patterns, shortouts. Insteed, justify step your solution
the following integers?
reasoning
recalling.
Let me solve this step by step by analyzing the given image and function y acos((x).
First, let's identify and from the graph:
determines the amplitude (half the distance from minimum maximum)
determines the pertod (attects the borizontal strecch/oomprrssionn)
2. Finding a (amplitude):
Looking graph, the maximmm points reach y
The miniuon points reach y -2
Therefore, the amplitude 2
Since amplitude jal, we conclude a 2
Finding (frequency):
The perfod of standard costx) Zit
For cos(bo)), the period (2n/bbl)
Looking graph::
Oue comuplete cycle goes from to
Therefore, the perlod 2nt
Using the formula
Solving for b: -1
Therefore, b-1
perfod of the fumetion:
Since we found 1
The period - 2n//b - 2n/1 22
2n 6:28118307799566
Therefore, the period the function approxdmately 6.283185307179866
This matches length 2n.
graph, complete cycle
Copy
Figure 16: Memorization phenomenon on Claude-3.5 with seed question 12 in DYNAMATH.
47


Reviews:
[{"review_id": "0Oj0eegeBA", "reviewer": "Reviewer_bDGf", "summary": "This paper introduces DYNAMATH, a dynamic visual benchmark designed to evaluate the mathematical reasoning robustness of Vision-Language Models (VLMs). DYNAMATH contains 501 seed questions implemented as Python programs that can generate diverse variants by altering visual conditions while maintaining the same underlying mathematical concepts. The benchmark spans multiple mathematical topics and difficulty levels. The authors evaluated 14 state-of-the-art VLMs (both closed and open-source) and found significant gaps between average-case and worst-case performance, indicating limitations in VLMs' mathematical reasoning robustness.", "questions": "In addition to the problems in weaknesses. There are some extra questions and suggestions.\n1. Can we convert the existing problem into a dynamic/programmatic version?\n2. Including analysis of the correlation between different types of variations and model performance will be better.\n3. Investigate potential relationships between training data/training techniques and performance on different variation types if possible.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The dynamic nature of the benchmark is innovative and addresses a critical gap in existing evaluation methods for VLMs' mathematical reasoning capabilities.\n2. The evaluation is comprehensive. The author tests multiple aspects of mathematical reasoning across 9 different topics and they evaluate both closed and open-source models. The detailed performance includes various types of variations (numerical, geometric, functional, etc.).\n3. The ablation analysis is strong: (a) detailed performance breakdown by topic and difficulty level;(b) Introduction of meaningful metrics (worst-case accuracy, reasoning robustness);(3)Thorough error analysis with clear categorization\n4. The methodology is well-documented. The author provides a clear explanation of benchmark creation process.", "weaknesses": "1. The authors acknowledge that the difficulty level is relatively limited compared to some existing benchmarks like MATH-V. The requirement for programmatic generation may restrict the inclusion of more complex problems\n2. The selection of seed questions and their variants might introduce unintended biases. The paper doesn't deeply discuss potential limitations in the variation generation process.\n3. The human evaluation seems limited (only 10 participants). It is unclear if the human evaluators' backgrounds are representative, which may limit the validation of human performance.\n4. Generalization of the conclusion on realistic questions. The questions (especially the vision content) are created by the program, which is quite different from the figures in the application (like the exam paper; hand-written question, or photo of the book). If the model fits well on these tasks, does it mean the model performs well on the realistic scenario?\n5.Missing in-depth analysis of the performance gap between the worst-case and the standard accuracy. If we use a large amount of synthetic data in a similar manner, can the model work well also on these problems? If the performance gap can be resolved by introducing more synthetic data, the conclusion will be trivial and have no interesting news.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vOGX6nywfk", "reviewer": "Reviewer_GLTU", "summary": "This paper introduces a dynamic visual benchmark designed to evaluate the mathematical reasoning robustness of VLMs. Unlike static benchmarks, DynaMath generates question variations programmatically to test models under diverse conditions. This allows for a nuanced assessment of VLMs’ robustness and generalization across mathematical tasks.", "questions": "1. It is uncertain if the benchmark could be easily hacked by just adopting the same program-based question generation to produce training data and then finetune the models.\n2. Is the reasoning robustness relevant to the difficulty level and type of questions and variants?\n3. Is there any preliminary study on how to scale the dynamic generation approach for more comprehensive and difficult benchmarks?\n4. It would be insightful to know if the authors have tested any interventions (e.g., prompt adjustments) to increase the robustness and reduce the memorization phenomenon.\n5. Is there any interpretability of the observations on the mathematical reasoning robustness?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed dynamic benchmark for evaluating the mathematical reasoning robustness of VLMs are complementary to existing benchmarks, and it aligns well with the challenges in real-world mathematical applications, where variations in problem conditions often arise.\n- Extensive evaluations on reasoning robustness are conducted and some observations are beneficial to the community.", "weaknesses": "- Although the paper presents some error analysis, the model robustness is still underexplored. Quite a few factors are not investigated, such as the difficulty level and type of questions and variants.\n- Besides the question itself, the prompt template can also result in different responses. The robustness regarding to prompt variants is neither studied nor excluded from the research scope with clear explanation.\n- Deeper insights into the failure modes and interpretability would be great.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LSVXow8kyK", "reviewer": "Reviewer_Fgu5", "summary": "DynaMATH, a new benchmark for evaluating visual reasoning in vision language models, offers greater controllability than previous tools like MathVistas. Using 501 programmable seed questions with multiple variants, it tests both closed and open-source VLMs, revealing insights about worst-case accuracy and memorization patterns.", "questions": "Can the authors provide a rough time estimate for designing/redesigning seed questions? This is important for future benchmark design. Also, how can we scale up the design flow for a larger dynamic math benchmark?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper effectively presents comprehensive and interesting findings.\n2. The benchmark's controllability should be highlighted as a key strength.", "weaknesses": "1. Move Figure 7 to the main paper to better illustrate the benchmark's appearance.\n2. The paper lacks details about the seed question curation process and should address the benchmark's scalability limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eeI4s1y9Ty", "reviewer": "Reviewer_Nqb6", "summary": "The paper introduces DYNAMATH, a dynamic visual math benchmark aimed at evaluating the mathematical reasoning robustness of Vision-Language Models (VLMs). \nThe benchmark reveals that state-of-the-art VLMs, like GPT-4o, struggle with mathematical reasoning when visual numerical values or function graphs change, despite these tasks being simple for humans. \nDYNAMATH includes 501 seed questions, each represented as a Python program, capable of generating varied concrete questions with different visual and textual alterations. \nThe benchmark was used to assess 14 VLMs (including proprietary ones and open-source ones) with 5,010 generated questions, showing a significant gap between average-case and worst-case model accuracy. \nThe paper's contributions include identifying the weakness in VLMs' reasoning abilities, introducing DYNAMATH for robustness evaluation, and providing insights to guide the development of more reliable VLMs for mathematical reasoning.", "questions": "1. What about the zero-shot / few-shot COT results for open-source VLMs? It would be beneficial to present those results and do some analysis. \n2. Evaluating VLMs with temperature not set to 0 will make the results less reproducible. Will the evaluation result vary much when you run the evaluation multiple times? Can you provide an error bar for it? \n3. Would you please provide the detailed distribution of problems in DYNAMATH (multiple choice, answer is float, answer is text)? \n4. In MMBench [1], the authors have defined another consistency named circular consistency? Will the Repetition Consistency still be that high (>80%) when circular shift is applied to choices for MCQ problems in DYNAMATH?\n\n[1] MMBench: Is Your Multi-modal Model an All-around Player?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. DYNAMATH is the first dynamic visual math benchmark designed to evaluate the robustness of mathematical reasoning in VLMs. \nThe dynamic generation of varied problem sets more accurately simulates real-world problem-solving, providing a more genuine assessment of model capabilities.\n2. With 501 high-quality seed questions that can generate a multitude of variant problems across various mathematical topics and difficulty levels, DYNAMATH allows for an in-depth evaluation of VLMs' generalization abilities under different input conditions, revealing the limitations of current models when dealing with problem variations.", "weaknesses": "1. Lacks analysis of open-source VLMs performance under the zero-shot / few-shot setting.\n2. Lacks analysis of the reproducibility of evaluation results (since temperature is not 0).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)