DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models

## Abstract

Abstract The rapid advancements in Vision-Language Models (VLMs) have shown great potential in tackling mathematical reasoning tasks that involve visual context.
Unlike humans who can reliably apply solution steps to similar problems with minor modifications, we found that state-of-the-art VLMs like GPT-4o can consistently fail in these scenarios, revealing limitations in their mathematical reasoning capabilities.
In this paper, we investigate the mathematical reasoning robustness in VLMs and evaluate how well these models perform under different variants of the same question, such as changes in visual numerical values or function graphs.
While several vision-based math benchmarks have been developed to assess VLMs’ problem-solving capabilities, these benchmarks contain only static sets of problems and cannot easily evaluate mathematical reasoning robustness.
To fill this gap, we introduce DynaMath , a dynamic visual math benchmark designed for in-depth assessment of VLMs. DynaMath includes 501 high-quality, multi-topic seed questions, each represented as a Python program . Those programs are carefully designed and annotated to enable the automatic generation of a much larger set of concrete questions, including many different types of visual and textual variations. DynaMath allows us to evaluate the generalization ability of VLMs, by assessing their performance under varying input conditions of a seed question. We evaluated 14 state-of-the-art VLMs with 5,010 generated concrete questions (10 per seed question). Our results show that the worst-case model accuracy, defined as the percentage of correctly answered seed questions in all 10 variants, is significantly lower than the average-case accuracy.
In addition, many models show high consistency in answering these questions – the incorrectness of a certain variant of a seed question is not due to inherent randomness. Our analysis emphasizes the need to study the robustness of VLMs’ reasoning abilities, and DynaMath provides valuable insights to guide the development of more reliable models for mathematical reasoning.

## 1 Introduction

Leveraging pretraining on vast Internet-scale datasets, Large Language Models (LLMs) (Brown, ; Ouyang et al., ; Touvron et al., ; Achiam et al., ) and Multi-modal Large Language Models (MLLMs) (Team et al., ; Bai et al., ; Liu et al., ; ) have achieved remarkable performance across a wide range of tasks. Among them, Vision-Language Models (VLMs) (Zhu et al., ; Zhang et al., ) stand out, showing exceptional promise as versatile assistants capable of integrating vision and language for problem-solving.

Among their visual comprehension abilities across different domains, mathematical reasoning (Lightman et al., ; Zhang et al., ) stands out as a crucial measure of human-like intelligence, requiring both math knowledge and logical thinking.
Recent work has proposed many benchmarks for evaluating the mathematical reasoning ability of VLMs. MATHVISTA (Lu et al., ) was the first benchmark specifically designed to evaluate visual mathematical reasoning. Recent closed-source models, such as Claude 3.5 Sonnet and GPT-4o, along with open-source models like LLaVA-OneVision (Li et al., ), have demonstrated average performance surpassing that of humans. Benchmarks such as MATH-V (Wang et al., ) and MATHVERSE (Zhang et al., ) demonstrate the current limitations of VLMs in handling challenging mathematical problems and understanding mathematical diagrams. Following typical evaluation pipelines, these benchmarks contain a *static* set of testing questions on which a VLM will be scored.

Figure: Figure 1: An example of consistent failures in GPT-4o. Seed question 78 in our DynaMath benchmark generates a graph of a shifted absolute value function. GPT-4o consistently provides incorrect answers for variant 9 (left) with 90% repetition consistency, while it can successfully answer variant 7 (right) with 100% repetition consistency. We tested 7 other variants involving non-zero shifts of the absolute value function, and in each case, GPT-4o insists incorrectly that the “sharp corner” is at $x=0$, leading to incorrect answers for all 7 variants. More failure examples are in Appendix F.
Refer to caption: x1.png

Our work is inspired by recent studies (Nezhurina et al., ; Zheng et al., ; Zong et al., ; Mirzadeh et al., ), which revealed that even powerful LLMs struggle to reliably solve simple text reasoning problems under different input values or conditions.
We found that this issue is even more pronounced in VLMs due to the added complexity of visual context. In the setting of math problems, we identified consistent failure cases on *variations of simple questions*. As illustrated in Figure 1, we identify a simple question asking whether a shifted absolute value function $f(x)=|x-a|$ is differentiable at $x=0$. Despite the shift, this question is still quite simple and poses no challenges to humans. While GPT-4o can give correct answers for some values of $a$, it consistently gives a wrong answer for many different values of $a\neq 0$.
Drawing inspiration from human reasoning, where the same steps can be applied to solve similar problems with varying conditions, a robust reasoning model should exhibit the same ability. This raises important questions about the robustness of VLMs’ reasoning abilities:
are the reasoning procedures in VLMs robust to problem variations that pose no challenge to humans?

To address this question, we comprehensively study the robustness of mathematical reasoning in VLMs by introducing a new benchmark, DynaMath. DynaMath is a *dynamic* visual math benchmark designed for an in-depth assessment of VLMs’ reasoning robustness.
Unlike existing benchmarks, which contain a static dataset of benchmarking questions, DynaMath contains 501 high-quality *seed* questions covering multiple mathematical topics: Plane Geometry, Solid Geometry, Analytic Geometry, Algebra, Puzzle Tests, Graph Theory, Statistics, Scientific Figures, and Arithmetic. Each seed question is represented as a carefully designed Python *program*; upon running, a program generates diverse *concrete* instances of one seed question with random variations in its conditions. The program is individually written for each seed question and considers multiple possible types of variations in each question, such as variations of numerical values, function types, graph structure, geometry, mathematical operations, etc.
The questions also span varying difficulty levels, from elementary school to high school and undergraduate, with the latter two dominating.
The process of dynamic benchmark generation and evaluation is presented in Figure 2. During generation, many concrete questions are created from a single seed question, and thus the actual number of questions evaluated can be much greater (e.g., 10$\times$ more) than the number of seed questions.

We conducted extensive experiments on DynaMath to evaluate the reasoning robustness of current state-of-the-art (SOTA) closed-source models, including GPT-4o, Gemini Pro, and Claude-3.5 Sonnet, as well as open-source VLMs such as the InternVL2 series (Chen et al., ), LLaVA-v1.6 series (Liu et al., ), Qwen2-VL (Wang et al., ), DeepSeek-VL (Lu et al., ), and Llama 3.2 (Dubey et al., ). For each seed question, we randomly generated 10 variants, resulting in an evaluation dataset of 5,010 concrete problems. On these problems, we evaluate both average-case accuracy and worst-case accuracy. The *worst-case accuracy* is defined as the percentage of correctly answered seed problems in *all* 10 variants. We observe that all considered VLMs have a worst-case accuracy that is close to or less than 50% of their average-case accuracy, signifying their unreliability in handling question variations. In addition, we also evaluate the *repetition consistency* on these VLMs, which characterizes the models’ inherent randomness to ensure that a low worst-case accuracy is not caused by occasional random errors but consistent errors on certain variants of a seed problem. Our main contributions and findings can be summarized as:

Figure: Figure 2: The dynamic benchmark generation procedure in DynaMath. A seed question is represented as a program that can generate many concrete questions with different variations. The plots for concrete questions are randomly generated along with the corresponding ground-truth answers. During evaluation, all concrete variants of the seed questions are considered, allowing us to evaluate the worst-case model performance and robustness.
Refer to caption: x2.png

- •
We are the first to study the mathematical reasoning robustness of VLMs and identified a new weakness in VLMs: they may consistently fail on certain variants of simple math questions that pose no challenges to humans. Such a weakness is prevalent in many state-of-the-art VLMs.
- •
We introduce DynaMath, a dynamic benchmark comprising 501 individually designed programs capable of generating a large number of question variants across different types. Our work is the first dynamically generated benchmark for evaluating the math capability of VLMs.
- •
Based on 5,010 concrete questions generated by DynaMath, we conduct an extensive evaluation of both SOTA closed-source and open-source VLMs. We find a noticeable gap between the average-case accuracy and worst-case accuracy among all models, indicating that many VLMs do not have robust reasoning capabilities even on relatively simple mathematical questions.

## 2 Related Work

#### Mathematical Reasoning Benchmarks.

Reasoning ability is a key indicator of intelligence, prompting researchers to develop various benchmark datasets to assess the mathematical reasoning capabilities of LLMs and VLMs. Numerous benchmarks have been proposed for evaluating this ability in the text-only domain, including (Amini et al., ; Hendrycks et al., ; ; Cobbe et al., ; Mishra et al., ; Frieder et al., ; Yu et al., ; Zhang et al., ). Additionally, recent research has begun to shift its focus towards the evaluation of robustness and the creation of dynamic benchmarks for language models. Several studies (Stolfo et al., ; Wu et al., ; Srivastava et al., ; Nezhurina et al., ; Qian et al., ; Kurtic et al., ; Mirzadeh et al., ) assess the language models’ robustness to the changing of item names or value conditions in the text-based question. However, many real-world problems, such as those involving statistical charts and geometry, rely on visual context. To assess visual mathematical reasoning, several benchmarks have been designed around geometry tasks (Lu et al., ; Chen et al., ) or multiple-choice questions (Yue et al., ). Recent efforts have expanded these benchmarks to cover a broader array of topics and question formats, such as MATHVISTA (Lu et al., ), MATHVERSE (Zhang et al., ), and MATH-V (Wang et al., ). Despite the diverse range of questions and visual contexts in these benchmarks, they share a common limitation: both the visual components and text remain static. This allows models to potentially achieve high scores by memorizing patterns from the training data, rather than applying true reasoning skills. In contrast, this paper introduces a dynamic visual math benchmark that provides a more rigorous assessment of VLMs’ reasoning capabilities through dynamically generating math questions with visual content.

#### Vision-Language Models (VLMs)

With the success of LLMs, numerous closed-source VLMs, such as GPT-4o, Gemini, and Claude 3.5, have excelled across a variety of visual-based understanding and conversational tasks, highlighting their potential as multimodal AI assistants. In the open-source domain, several efforts are actively advancing the field. Approaches like LLaMA-Adapter (Zhang et al., ; Gao et al., ) and MiniGPT-4 (Zhu et al., ) leverage frozen language models with a limited number of trainable parameters, demonstrating promising results. Furthermore, a range of VLMs trained on larger multimodal datasets has been open-sourced, pushing the frontier of visual comprehension and generalization ability. Notable examples include the InternVL1.5 and InternVL2 series (Chen et al., ), InternLM-XComposer (Zhang et al., ; Dong et al., ), LLaVA-v1.6 series (Liu et al., ), LLaVA-OneVision (Li et al., ), Qwen-VL (Bai et al., ; Wang et al., ), and DeepSeek-VL (Lu et al., ). These models contribute significantly to advancing the capabilities of VLMs in prior visual benchmarks.

## 3 Benchmark Design

We present DynaMath, a curated evaluation dataset aimed at assessing the robustness of visual language models (VLMs) in multimodal mathematical reasoning across a wide variety of mathematical tasks with dynamic visual and textual contexts.

### 3.1 Dataset Collection

Our benchmark collection comprises two phases: seed question collection and program-based question generation. In the initial phase, we selectively curate a set of high-quality mathematics problems that necessitate reasoning based on visual information. The subsequent phase involves transforming each seed question into code-based prototypes, allowing for the generation of diverse concrete questions under randomly sampled conditions.

#### Seed question Collection.

The seed questions are sourced from existing visual math datasets and publicly available online resources. We identify 107 questions from MathVista (Lu et al., ), covering fundamental concepts in analytic geometry, planar geometry, and statistics. Additionally, we source 27 questions from MATH-V (Wang et al., ), which serve as prototypes for topics related to arithmetic, puzzle tests, and solid geometry. To augment the dataset’s breadth and depth, we included 45 questions based on scientific figures and 48 undergraduate-level questions focused on graph theory, drawn from the MMMU dataset (Yue et al., ) and various accessible educational materials. Furthermore, we incorporated 236 questions requiring advanced reasoning on topics such as functions, geometry, and statistics, all gathered from publicly available resources on the Internet. To diversify the question types represented in our collection, we also developed 38 new problems by ourselves covering linear algebra, set theory, and algorithmic flow.

Following the collection of seed questions, we conducted a comprehensive review to eliminate any questions that included excessively complex images, as these would pose challenges for programmatic generation. Ultimately, as shown in Figure LABEL:fig:source_composition, our benchmark consists of 501 seed questions, with 227 (45.3%) sourced from established visual math datasets, while 274 (54.7%) are newly collected or developed from public resources.

Note that our goal is not to create the most challenging, competition-level benchmark as in (Wang et al., ), but rather to provide relatively easy benchmarks with diverse variants to evaluate robustness. Nonetheless, we ensure that the difficulty of our questions is comparable to the levels of datasets such as MATHVERSE (Zhang et al., ) and MATHVISTA (Lu et al., ).

#### Program-based Question Generation.

After establishing our seed questions, we recruited a group of college STEM students to annotate each question with the common strategies they employed in solving them. These annotations served as prototypes for developing corresponding programs tailored to each question. As illustrated in Figure 2, each question is represented as a carefully crafted Python program, which encompasses a defined range of conditions for sampling and algorithmic calculations to derive the solution. Additionally, we implemented a drawing function in each program, utilizing libraries such as Matplotlib and Pyglet to generate corresponding images based on varying conditions. Specifically, 470 of the question programs incorporate a plotting function that leverages the randomly sampled conditions to create the visual context of the question, while the remaining 31 question programs utilize fixed images, randomizing only the textual elements. This programmatic approach allows the generation of a large number of concrete benchmark questions by executing the generation program multiple times, facilitating the efficient creation of new problems and enabling the evaluation of the reasoning robustness of VLMs.

In DynaMath, we integrate various types of variants to enrich the diversity of question generation:

- 1.
Numerical Value Variants: Modifying numerical quantities to evaluate the VLM’s proficiency in handling different numerical values and performing arithmetic operations.
- 2.
Geometric Transformations: Altering shapes, angles, dimensions, and relative positions to examine the spatial and geometric understanding of VLMs.
- 3.
Function Type Variants: Varying different types of mathematical functions (e.g., linear, quadratic) to evaluate how well models generalize across functional representations.
- 4.
Color Variants: Changing object or curve colors randomly to test the model’s recognition of visual patterns and its robustness to superficial alterations.
- 5.
Symbolic Substitutions: Modifying symbolic elements such as mathematical operations to determine the model’s adaptability to various symbolic representations.
- 6.
Graph Structure Variants: Modifying graph layouts, networks, or other structural representations to assess the model’s comprehension of relationships and topological features.
- 7.
Real-life Contexts Variants: Adjusting the contents of real-world scenarios (e.g., calendars, time-related problems, or poker-like questions) to test the model’s contextual understanding and application to practical situations.

Each variant category targets a specific facet of mathematical reasoning, making DynaMath a comprehensive benchmark for evaluating the flexibility, robustness, and accuracy of VLMs in solving mathematical problems. Detailed diagrams of each variation are provided in Appendix A.

### 3.2 Dataset Statistics

Detailed statistics on the data composition of DynaMath are presented in Table 1. DynaMath encompasses nine mathematical topics: Solid Geometry (SG, 3.0%), Puzzle Tests (PT, 3.4%), Arithmetic (AR, 5.2%), Scientific Figures (SF, 9.0%), Graph Theory (GT, 9.6%), Algebra (AL, 10.2%), Plane Geometry (PG, 15.4%), Analytic Geometry (AG, 19.4%), and Statistics (ST, 25.0%). Examples for each topic are provided in Appendix C. Each topic necessitates a nuanced understanding of image context, foundational mathematical knowledge, practical reasoning abilities, and logical deduction skills. Importantly, the dataset is designed to cater to varying levels of difficulty, ranging from elementary to undergraduate education, with a notable focus on high school (55.3%) and undergraduate (32.1%) levels. In terms of question types, the dataset consists of 35.5% multiple-choice questions and 64.7% free-form questions. While VLMs might occasionally answer multiple-choice questions correctly by chance, free-form questions provide a more precise evaluation of the model’s capabilities. Consequently, our dataset emphasizes free-form questions, distinguishing it from previous visual math benchmarks such as MATHVISTA (Lu et al., ), MATHVERSE (Zhang et al., ), and MATH-V (Wang et al., ), which predominantly include more than 50% multiple-choice questions.

In Figure LABEL:variant_distribution, we depict the distribution of variant numbers among the 501 seed questions. Notably, approximately 30.5% of the seed questions have a possible variant number ranging from 10 to $10^{2}$. Nearly 93% of the seed questions contain more than 10 variants, and 17.4% of seed questions have more than $10^{6}$ potential variants, demonstrating the diversity of variations in our dataset.

Figure: Table 1: Statistics of DynaMath.

### 3.3 Evaluation Protocols

Our evaluation process consists of two stages: answer extraction and score calculation. Following the methodology of prior work (Lu et al., ), we utilize prompt engineering and template matching to extract answers. Prompts guide the model to generate responses in both full and short-answer formats. After generation, the short answer is extracted for comparison with the ground truth. Detailed prompts used in our experiments can be found in Appendix B.

Our dataset contains $N=501$ seed questions in total. For each seed question in the dataset, we generate $M=10$ variants, resulting in a total of $5,010$ concrete questions. Inspired by prior research in LLMs (Srivastava et al., ), we evaluate two metrics: average-case accuracy ($\mathcal{A}_{avg}$) and worst-case accuracy ($\mathcal{A}_{wst}$) over these variants. The two metrics are different from prior benchmarks that evaluate only a single instance of a question. The metrics are defined as follows:

$$ $\displaystyle\mathcal{A}_{avg}=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{M}\sum_{j=1}^ {M}\mathbb{I}[\text{Ans}(i,j)=\text{GT}(i,j)],$ (1) $\displaystyle\mathcal{A}_{wst}=\frac{1}{N}\sum_{i=1}^{N}\min_{j\in[1,M]} \mathbb{I}[\text{Ans}(i,j)=\text{GT}(i,j)],$ $$

where $\text{Ans}(i,j)$ and $\text{GT}(i,j)$ represent the generated answer and the ground truth answer for variant $j$ of question $i$. We also define reasoning robustness ($RR$) as the ratio between the average-case performance and the worst-case performance:

$$ $RR=\frac{\mathcal{A}_{wst}}{\mathcal{A}_{avg}},$ (2) $$

The model’s response uncertainty reflects both the impact of input changes and inherent uncertainty, the latter of which can be represented by the concept of repetition consistency ($RC$), similar to self-consistency (Wang et al., ). We define repetition consistency as:

$$ $RC(i,j)=\frac{1}{K}\sum_{k=1}^{K}\mathbb{I}[\text{Ans}_{k}(i,j)=\text{Ans}(i,j )],$ (3) $$

where $K$ is number of repetitions and $\text{Ans}_{k}(i,j)$ is the $k$-th repetition for $j$-th variant of $i$-th seed question. The repetition consistency represents the model’s confidence in the answer $\text{Ans}(i,j)$.

## 4 Experiment

In this section, we conduct thorough experiments to assess the performance and reasoning robustness of various closed-source and open-source models on the DynaMath dataset. Subsequently, we present detailed quantitative results and qualitative analyses in Sections 4.2 and 4.3, respectively.

### 4.1 Experimental Setups

We evaluate the performance of two sets of models on the DynaMath benchmark, which involves 10 variations for each seed question, resulting in a total of 5010 questions. The first group comprises SOTA closed-source VLMs, such as GPT-4o, Gemini Pro 1.5, and Claude-3.5 Sonnet, with zero-shot and 3-shots with Chain-of-Thought (CoT) configurations. The second group consists of SOTA open-source VLMs, including Qwen2-VL (7B, 72B) (Wang et al., ), InternVL2 (8B, 26B, 40B, 76B) (Chen et al., ), Llava-v1.5 (7B) (Liu et al., ), Llava-v1.6 (13B, 34B) (Liu et al., ), Deepseek-VL (7B) (Lu et al., ), and Llama 3.2 (90B) (Dubey et al., ). We specifically explored open-source models with varying parameter sizes to analyze the impact of model size on reasoning robustness. The deployment of open-source models is based on the lmdeploy package (Contributors, ). Details regarding the prompts and hyperparameters used in this experiment are outlined in Appendix B.

To assess human performance, we generated a new variant dataset consisting of 501 concrete questions (1 variant per seed question). These questions were divided into 10 questionnaires, each containing 50 to 51 questions. We then recruited 10 undergraduates or graduates from STEM to help establish the baseline for human performance based on their average scores.

For the few-shot setup, we follow the standard approach by including three demonstration examples, each accompanied by the associated visual elements. Given the diverse range of topics covered in DynaMath, we provide topic-specific demonstration examples to ensure its relevance for each problem in DynaMath. Specifically, we curated five demonstration examples from MathVista (Lu et al., ) and MathVision (Wang et al., ) for each topic, and then randomly selected three examples when evaluating DynaMath problems within the corresponding topic. In addition, we incorporate detailed reasoning steps in the demonstration examples, following a typical Chain-of-Thought (CoT) setup (Wei et al., ). Detailed demonstrations and prompts are listed in Appendix B.4.

**Table 2: Average-case accuracy $\mathcal{A}_{avg}$ on DynaMath with 5,010 generated questions. “ALL” represents overall accuracy. Question topics and difficulty levels (PG, EL, etc) are defined in Table 1.**
| Model | ALL | PG | SG | AG | AL | PT | GT | ST | SF | AR | EL | HI | UN |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Closed-sourced Large Multimodal Models (LMMs) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Zero-shot GPT-4o | 63.7 | 56.8 | 52.0 | 61.0 | 76.9 | 51.8 | 58.1 | 69.3 | 62.4 | 61.5 | 68.6 | 61.8 | 36.8 |
| Zero-shot Claude-3.5 | 64.8 | 49.9 | 49.3 | 55.3 | 81.0 | 44.1 | 69.4 | 78.2 | 62.2 | 61.2 | 66.7 | 62.6 | 33.3 |
| Zero-shot Gemini Pro 1.5 | 60.5 | 52.7 | 42.7 | 61.6 | 70.8 | 20.6 | 65.2 | 69.8 | 50.2 | 54.2 | 62.9 | 59.2 | 37.1 |
| 3-shot CoT GPT-4o | 64.9 | 58.1 | 59.3 | 57.7 | 84.1 | 51.2 | 61.9 | 71.0 | 60.9 | 57.7 | 66.2 | 62.5 | 34.8 |
| 3-shot CoT Claude-3.5 | 62.5 | 49.1 | 48.0 | 50.6 | 80.2 | 37.1 | 58.1 | 78.2 | 64.9 | 55.0 | 63.0 | 61.5 | 30.5 |
| 3-shot CoT Gemini Pro 1.5 | 58.7 | 52.6 | 45.3 | 56.7 | 72.9 | 21.8 | 57.9 | 66.0 | 54.9 | 48.1 | 59.0 | 58.3 | 34.2 |
| Open-sourced Vision Language Models (VLMs) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2-VL-72B | 55.1 | 48.1 | 48.7 | 50.9 | 57.6 | 28.2 | 45.0 | 68.9 | 56.4 | 54.2 | 61.3 | 57.4 | 30.7 |
| Qwen2-VL-7B | 42.1 | 40.3 | 38.7 | 39.9 | 37.1 | 8.2 | 44.8 | 52.1 | 41.1 | 39.2 | 47.6 | 42.2 | 24.4 |
| InternVL2-76B | 54.0 | 44.5 | 34.7 | 43.8 | 67.6 | 35.3 | 51.0 | 66.7 | 55.1 | 51.5 | 60.3 | 52.9 | 26.4 |
| InternVL2-40B | 41.8 | 31.3 | 21.3 | 38.8 | 42.9 | 15.3 | 38.3 | 58.1 | 43.1 | 38.1 | 51.0 | 41.5 | 23.4 |
| InternVL2-26B | 41.0 | 35.8 | 26.0 | 37.3 | 38.8 | 13.5 | 46.9 | 51.9 | 39.6 | 40.4 | 52.1 | 38.5 | 22.5 |
| InternVL2-8B | 39.7 | 33.9 | 37.3 | 32.5 | 46.9 | 15.9 | 42.1 | 47.8 | 39.1 | 37.3 | 51.1 | 37.4 | 19.6 |
| Llama-3.2-90B | 44.0 | 47.5 | 37.3 | 36.8 | 46.5 | 12.4 | 44.8 | 56.8 | 39.8 | 30.0 | 45.4 | 43.8 | 22.2 |
| Deepseek-VL-7B-chat | 21.5 | 16.0 | 13.3 | 26.5 | 12.9 | 4.7 | 32.7 | 24.3 | 24.2 | 15.0 | 28.3 | 19.0 | 16.0 |
| Llava-v1.6-34B | 27.1 | 21.4 | 25.3 | 27.6 | 14.9 | 7.6 | 32.7 | 36.8 | 27.8 | 23.1 | 35.9 | 23.8 | 16.6 |
| Llava-v1.6-vicuna-13B | 19.8 | 14.7 | 10.0 | 23.4 | 8.2 | 10.0 | 21.5 | 28.2 | 19.6 | 10.0 | 27.1 | 16.5 | 14.1 |
| Llava-v1.5-7B | 16.6 | 10.5 | 7.3 | 19.5 | 6.5 | 8.2 | 32.3 | 17.5 | 20.2 | 10.8 | 18.9 | 13.3 | 11.7 |
| Human |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Human performance | 75.8 | 80.5 | 60.0 | 83.5 | 78.4 | 76.5 | 64.6 | 74.4 | 77.8 | 61.5 | 74.6 | 78.3 | 72.0 |

### 4.2 Experimental Results

In this section, we present a detailed comparison of the top-performing VLMs on DynaMath, as shown in Table 2 and Table 3.

#### Overall Results on Average Accuracy.

Table 2 illustrates the average-case performance of a variety of closed-source and open-source models. We respectively highlight the highest scores in each category for closed-source and open-source models. Within the closed-source models, GPT-4o, Claude-3.5, and Gemini Pro 1.5 exhibit average accuracies higher than 60%, with Claude-3.5 achieving the highest zero-shot average accuracy at 64.8%. However, there remains an 11.0% disparity when compared to human performance, which stands at 75.8% average accuracy. This highlights the need for further development in the reasoning ability of VLMs. Regarding the 3-shot CoT performance, it is intriguing to note that there is no consistent improvement across different closed-sourced models, confirming findings from previous research (Wang et al., ). For instance, while 3-shot CoT GPT-4o manages to enhance zero-shot performance from 63.7% to 64.9%, both 3-shot CoT Claude-3.5 and 3-shot CoT Gemini Pro 1.5 experience a decline in performance (64.8% $\to$ 62.5% and 60.5% $\to$ 58.7% respectively).
Moving on to the open-sourced models, although they generally underperform when compared to closed-sourced models, the gap has been narrowed by recent models such as Qwen2 and InternVL2, which have more than 70B parameters. This noteworthy progress is evident when comparing them to previous benchmark results like MATHVISTA (Amini et al., ), MATHVERSE (Zhang et al., ), and MATH-V (Wang et al., ). It highlights the promising potential of open-source models in the visual math reasoning domain. Moreover, there is a clear scaling trend observed in open-source models, indicating higher performance as model sizes increase. For example, Qwen2-VL boosts its score from 42.1% to 55.1% when scaling its parameter size from 7B to 72B, while InternVL2 sees an increase from 39.7% to 54.0%.

#### Overall Results on Worst-case Accuracy.

Table 3 presents the worst-case accuracy of different models across 10 problem variants, revealing a significant decline in scores for all models. Notably, the highest-performing model, Claude-3.5, achieves a zero-shot score of only 35.3%, indicating current VLMs are not sufficiently robust to handle variations in context and images. The situation is even more concerning for open-source models: the best-performing model, Qwen2-VL-72B, achieves a score of 28.3%, while smaller models like Llava-v1.6-vicuna-13B score only 2.8%. Our evaluation results highlight the limited reasoning robustness of both open-source and closed-source models, underscoring the necessity for the community to address these limitations in future research.

**Table 3: Worst-case accuracy $\mathcal{A}_{wst}$ on DynaMath with 5,010 generated questions. “ALL” represents overall accuracy. Question topics and difficulty levels (PG, EL, etc) are defined in Table 1.**
| Model | ALL | PG | SG | AG | AL | PT | GT | ST | SF | AR | EL | HI | UN |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Closed-sourced Large Multimodal Models (LMMs) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Zero-shot GPT-4o | 34.7 | 37.7 | 33.3 | 25.8 | 54.9 | 11.8 | 18.8 | 38.4 | 35.6 | 46.2 | 46.0 | 34.3 | 31.1 |
| Zero-shot Claude-3.5 | 35.3 | 22.1 | 26.7 | 18.6 | 62.7 | 23.5 | 27.1 | 53.6 | 24.4 | 42.3 | 49.2 | 33.2 | 33.5 |
| Zero-shot Gemini Pro 1.5 | 26.9 | 28.6 | 20.0 | 19.6 | 39.2 | 5.9 | 22.9 | 35.2 | 15.6 | 30.8 | 41.3 | 26.7 | 21.7 |
| 3-shot CoT GPT-4o | 32.3 | 31.2 | 40.0 | 21.6 | 54.9 | 17.6 | 20.8 | 36.8 | 26.7 | 46.2 | 47.6 | 30.7 | 29.2 |
| 3-shot CoT Claude-3.5 | 32.1 | 27.3 | 26.7 | 11.3 | 54.9 | 0.0 | 10.4 | 56.0 | 31.1 | 30.8 | 39.7 | 32.9 | 28.0 |
| 3-shot CoT Gemini Pro 1.5 | 23.6 | 27.3 | 26.7 | 14.4 | 39.2 | 5.9 | 18.8 | 27.2 | 17.8 | 26.9 | 33.3 | 23.1 | 20.5 |
| Open-sourced Vision Language Models (VLMs) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2-VL-72B | 28.3 | 27.3 | 33.3 | 15.5 | 31.4 | 0.0 | 16.7 | 43.2 | 26.7 | 42.3 | 41.3 | 30.3 | 19.9 |
| Qwen2-VL-7B | 13.8 | 22.1 | 6.7 | 7.2 | 13.7 | 0.0 | 12.5 | 16.8 | 11.1 | 19.2 | 25.4 | 12.3 | 11.8 |
| InternVL2-76B | 24.6 | 24.7 | 20.0 | 15.5 | 37.3 | 5.9 | 12.5 | 32.8 | 20.0 | 38.5 | 39.7 | 23.1 | 21.1 |
| InternVL2-40B | 14.2 | 14.3 | 6.7 | 9.3 | 13.7 | 0.0 | 10.4 | 21.6 | 13.3 | 19.2 | 28.6 | 14.1 | 8.7 |
| InternVL2-26B | 14.4 | 19.5 | 0.0 | 6.2 | 9.8 | 0.0 | 18.8 | 20.0 | 11.1 | 26.9 | 34.9 | 12.3 | 9.9 |
| InternVL2-8B | 10.4 | 13.0 | 20.0 | 5.2 | 15.7 | 0.0 | 10.4 | 9.6 | 11.1 | 15.4 | 23.8 | 9.4 | 6.8 |
| Llama-3.2-90B | 13.0 | 22.1 | 20.0 | 7.2 | 7.8 | 0.0 | 12.5 | 16.8 | 13.3 | 3.8 | 15.9 | 14.1 | 9.9 |
| Deepseek-VL-7B-chat | 4.2 | 7.8 | 0.0 | 3.1 | 0.0 | 0.0 | 10.4 | 4.0 | 2.2 | 3.8 | 7.9 | 2.9 | 5.0 |
| Llava-v1.6-34B | 6.0 | 10.4 | 13.3 | 4.1 | 2.0 | 0.0 | 4.2 | 6.4 | 6.7 | 7.7 | 15.9 | 5.1 | 3.7 |
| Llava-v1.6-vicuna-13B | 2.8 | 7.8 | 0.0 | 4.1 | 0.0 | 0.0 | 2.1 | 2.4 | 0.0 | 0.0 | 6.3 | 2.9 | 1.2 |
| Llava-v1.5-7B | 1.8 | 3.9 | 0.0 | 2.1 | 0.0 | 0.0 | 4.2 | 0.8 | 0.0 | 3.8 | 3.2 | 1.8 | 1.2 |

#### Fine-grained Results.

In Table 2 and Table 3, we present detailed results categorized by different question topics and difficulty levels. From a topical perspective, we observe that the Puzzle Test (PT) topic challenges both open-source and closed-source models. The top-performing closed-source model, GPT-4o, and the leading open-source model, Qwen2-VL-72B, achieve average-case accuracies of 51.8% and 28.2%, respectively, while humans score 76.5%. Notably, all open-source models demonstrate poor performance (0.0%) on the worst-case accuracy metric, except Llama-3.2-90B (5.9%). Despite this gap, Table 2 shows that closed-source models such as Claude-3.5 can surpass human scores on specific topics like Algebra (AL), Graph Theory (GT), and Statistics (ST), which is promising. When considering difficulty levels, all models demonstrate a trend of decreasing average accuracy as the difficulty increases, as illustrated in Table 2. In contrast, human performance remains consistent across difficulty levels, indicating that current VLMs are still not adept at handling more difficult visual math problems compared with human capabilities.

Figure: Figure 5: Comparing reasoning robustness across different models (top), topics (middle), and variant types (bottom).
Refer to caption: x3.png

#### Reasoning Robustness.

We use the reasoning robustness ($RR$) metric, defined in Eq 2, to measure the robustness of VLMs by evaluating the relative performance robustness across question variants. Figure 5 (top) compares the $RR$ of all VLMs in our experiments. Notably, GPT-4o and Claude-3.5 exhibit the highest robustness among all tested models. Additionally, aligning with previous findings, closed-source models typically demonstrate greater robustness than open-source ones. However, Qwen2-72B and InternVL2-76B outperform Gemini, highlighting the robustness limitations of even large models like Gemini. In Figure 5 (middle), we compare the reasoning robustness across different question topics for GPT-4o and Qwen2-VL-72B. The results show that the two VLMs are particularly robust in Arithmetic and Algebra question types, indicating their strong arithmetic calculation abilities, which are less affected by changes in visual conditions. However, GPT-4o still exhibits weaknesses in the Puzzle Test. Similarly, Qwen2-VL-72B shows shortcomings in both Puzzle Test and Analytic Geometry topics, achieving nearly 0% $RR$ and 30% $RR$, respectively. In terms of different variant types in DynaMath, as shown in Figure 5 (bottom), we find that both GPT-4o and Qwen2-VL-72B are sensitive to variations in graph structure, geometric transformation, and function type. Additionally, Qwen2-VL-72B is vulnerable to symbolic substitution variants. These weaknesses suggest directions for future improvement of these models.

**Table 4: The Repetition Consistency ($RC$) for different models over 5 repetitions.**
| Model name | GPT-4o | Gemini | Qwen2-72B | InternVL2-76B |
| --- | --- | --- | --- | --- |
| Repetition Consistency (%) | 94.1 | 92.5 | 98.9 | 99.0 |

#### Repetition Consistency.

To ensure a robust analysis and account for the inherent randomness in model outputs, we calculate repetition consistency ($RC$) as defined in Eq 3. This metric evaluates the model’s output confidence across multiple generations for the same question. Specifically, we produce five responses for 501 questions and then compute their consistency relative to the first response. The results, detailed in Table 4, reveal the consistent outputs of four closed-source and open-source models, with $RC$ values ranging from 92% to 99%. Compared with the low reasoning robustness scores, VLMs have much higher inherent consistency. These findings reinforce our arguments that the low reasoning robustness of VLMs primarily stems from the model’s sensitivity to question variants rather than inherent randomness.

### 4.3 Quality Study

#### Consistent Failure Cases.

An interesting phenomenon we observed is that some seed questions are solvable in certain variants but result in consistent failures in others (repetition consistency $RC=1$ for 5 or 10 repetitions). The example in Figure 1 is a representative case: the question is easily solvable when the absolute value function is at the origin, but any shifts tend to lead to consistent failures on GPT-4o. We extensively examined our dataset and counted the number of such instances. Specifically, GPT-4o, Gemini Pro 1.5, Qwen2-VL-72B, and InternVL2-76B exhibited 21.8%, 18.4%, 29.9%, and 28.3% of these types of questions, respectively, out of our 501 seed questions. These examples highlight the unreliability of VLMs on mathematical reasoning tasks.

#### Qualitative Examples of GPT-4o.

In this section and Appendix F, we provide a few qualitative examples of leading VLMs’ answers. Our analysis reveals that current VLMs can consistently produce incorrect responses to specific question variants while generating accurate answers to others. As illustrated in Figure 1, GPT-4o demonstrates the ability to provide correct responses in variant 7, showcasing accurate perception, question understanding, and reasoning ability. However, in variant 9, where the underlying required capabilities remain the same with only a slight shift in the image, GPT-4o fails to accurately interpret the function’s position with a high degree of confidence and consistency. This discrepancy raises concerns about the reasoning robustness of current VLMs. For additional examples of GPT-4o and other models, please refer to the Appendix F.

Figure: Figure 6: Example of the Memorization Phenomenon: the generated variants of seed Question 12 and the corresponding responses from Claude 3.5 Sonnet. The model’s response remains $2\pi$ with high probability, regardless of changes in the conditions depicted in the diagram.
Refer to caption: x6.png

#### Memorization Phenomenon.

In our experiments, we observe a phenomenon where current VLMs tend to provide the same answer regardless of changing conditions, indicating memorization rather than reasoning based on generalized underlying principles. When we test variant questions that have the same structure but different parameters and images, the model frequently offers the same answer with high probability, ignoring the specific variations we introduced. Among the 171 questions incorrectly answered by Claude 3.5 Sonnet, this issue accounts for 4.1% of instances. A representative case is illustrated in Figure 6, where altering the period of a sinusoidal function (e.g., from $2\pi$ to $\pi$ or $4\pi$) does not affect the model’s response, which consistently remains $2\pi$. The existence of this phenomenon highlights the models’ lack of comprehensive problem analysis and their limited ability to generalize across different scenarios.

Figure: Figure 7: Error Analysis of Claude-3.5 Sonnet.
Refer to caption: x7.png

#### Error Analysis.

We conducted an error analysis on Claude 3.5 Sonnet to identify potential failure modes on DynaMath. Specifically, we analyzed the 169 questions where Claude 3.5 Sonnet failed, examining the reasoning paths and final answers in detail. The statistical distribution of various error types is presented in Figure 7. We considered five types of errors: figure reading errors, reasoning errors, knowledge errors, calculation errors, and hallucination errors. Figure reading errors account for 32.2% of the total errors, despite Claude 3.5 Sonnet having specially reinforced perception capabilities. This indicates that there is still a considerable way to go for VLMs to accurately read and interpret data from images. Reasoning errors account for 26.9%, making them the second-largest cause of errors. This suggests that the model’s reasoning processes are still delicate and can be easily disrupted by minor changes in conditions and image input. Calculation errors, which constitute 20.5% of the errors, likely result from the significant computational challenge imposed by our randomly generated conditions without specially designed parameters, as opposed to simpler questions in prior work that are easier to compute. In addition, Hallucination errors make up 17.0%, showing that the model tends to fabricate non-existent information. Examples are provided in Appendix D.

## 5 Conclusion

In this work, we introduce DynaMath, a dynamic visual math benchmark designed to systematically analyze the robustness of mathematical reasoning capabilities in current leading vision-language models (VLMs). By employing program-based problem generation, we can create diverse variants by altering visual and textual conditions in the seed problems. Our evaluation reveals that leading closed-source and open-source VLMs are sensitive to condition changes in question variants, despite their required underlying capabilities remaining the same. This raises significant concerns within the VLM community on mathematical reasoning tasks. Our detailed results and analysis not only identify the weak points of current VLMs but also shed light on the causes of their errors, thereby facilitating the evaluation and development of more robust VLMs. Moving forward, an intriguing approach to enhance VLM robustness involves leveraging adversarial training (Zhou et al., ; Schlarmann et al., ) on DynaMath, or utilizing reinforcement learning from human feedback (Ouyang et al., ; Rafailov et al., ; Yang et al., ) with fine-grained process rewards (Uesato et al., ; Wang et al., ; Luo et al., ), or more robust rewards (Yang et al., ; Zhang et al., ).

#### Limitation.

Although our benchmark matches the difficulty levels of MATHVERSE and MATHVISTA, one limitation of our work is that the difficulty level is relatively limited compared to MATH-V (Wang et al., ), due to the dynamic nature of the questions. Adapting very challenging questions into our program structures requires substantial human effort, which currently prevents us from curating a large number of complex visual math reasoning questions. In the future, we hope to leverage strong foundational models to aid in designing an automatic pipeline for dynamic math question design and generation.

#### Acknowledgment

Huan Zhang is supported in part by the AI2050 program at Schmidt Sciences (AI 2050 Early Career Fellowship).

## Appendix A Variation Types of DynaMath

DynaMath introduces several types of variations based on the seed questions. In Figure 8, we illustrate six distinct types of variations. This diversity allows our dataset to effectively evaluate the visual robustness of VLMs from different aspects.

Figure: Figure 8: Variation types considered in our DynaMath benchmark
Refer to caption: x8.png

## Appendix B Detailed Experiment Setup

In this section, we provide more details about our experiment designs.

### B.1 Prompts for response generation

In our experiments, we prompt the VLMs to generate responses to different types of questions, such as multiple choice, float, and text types. The prompts used for these question types are shown in Table 5.

**Table 5: The prompt for different questions and answer types in answer generation.**
| Answer type | prompt |
| --- | --- |
| multiple choice | If the problem is a multiple choice problem, just provide the corresponing choice option, such as ’A’, ’B’, ’C’, or ’D’. |
| float | If the answer is a numerical value, format it as a three-digit floating-point number. |
| text | Please answer the question in the following form: (specific requirement in question). |

### B.2 Prompts for answer extraction

To simplify the answer extraction process, we use the following prompts to extract the answer and reload it in JSON format, which can be easily used for template matching with ground truth answers:

Closed-source models can consistently extract answers and regenerate them in JSON format. In contrast, some open-source models, particularly those with relatively low parameter sizes, struggle to output JSON format reliably. We have identified patterns in their inconsistent responses, which deviate from the expected code scripts. By supplementing these scripts, we efficiently check, annotate, and extract informal answers.

### B.3 Model Hyperparameters

We set all parameters except temperature to their default values. We set temperature = 0.0 for both closed-source models and open-source models to reduce the randomness in model generation and make our evaluation of robustness more accurate. Table 6 displays the parameters used for VLM generation.

**Table 6: Hyperparameters for various VLMs.**
| Model | Hyperparameters |
| --- | --- |
| GPT-4o | model = gpt-4o-0806, temperature = 0.0, max_tokens = 4096 |
| Claude-3.5 | model = claude-3-5-sonnet-20240620, temperature = 0.0, max_tokens = 8192 |
| Gemini Pro 1.5 | model = gemini-1.5-pro, temperature = 0.0, max_tokens = 8192 |
| Qwen2-VL-72B | model = Qwen/Qwen2-VL-72B-Instruct, temperature = 0.0, max_tokens = 2048 |
| QWen2-VL-7B | model = Qwen/Qwen2-VL-7B-Instruct, temperature = 0.0, max_tokens = 2048 |
| InternVL2-76B | model = OpenGVLab/InternVL2-Llama3-76B, temperature = 0.0, max_tokens = 2048 |
| InternVL2-40B | model = OpenGVLab/InternVL2-40B, temperature = 0.0, max_tokens = 2048 |
| InternVL2-26B | model = OpenGVLab/InternVL2-26B, temperature = 0.0, max_tokens = 2048 |
| InternVL2-8B | model = OpenGVLab/InternVL2-8B, temperature = 0.0, max_tokens = 2048 |
| Deepseek-VL-7B-chat | model = deepseek-ai/deepseek-vl-7b-chat, temperature = 0.0, max_tokens = 2048 |
| Llama-3.2-90B | model = meta-llama/Llama-3.2-90B-Vision-Instruct, temperature = 0.0, max_tokens = 2048 |
| Llava-v1.6-34B | model = liuhaotian/llava-v1.6-34b, temperature = 0.0, max_tokens = 2048 |
| Llava-v1.6-vicuna-13B | model = liuhaotian/llava-v1.6-vicuna-13b, temperature = 0.0, max_tokens = 2048 |
| Llava-v1.5-7B | model = liuhaotian/llava-v1.5-7b, temperature = 0.0, max_tokens = 2048 |

### B.4 Prompts for few-shot Chain of Thought

Below we provide the few-shot CoT prompt for plane geometry topic. More demonstration examples for each topic can be found in Appendix E

## Appendix C Variant Examples for Different Topics in DynaMath

In this section, we show sample problems in DynaMath for different topics, including Solid Geometry (SG), Puzzle Tests (PT), Arithmetic (AR), Scientific Figures (SF), Graph Theory (GT), Algebra (AL), Plane Geometry (PG), Analytic Geometry (AG), and Statistics (ST).

## Appendix D Error Type Examples

In this section, we present various examples of error types encountered in Claude 3.5, including figure reading error, calculation error, reasoning error, hallucination error, and knowledge error.

## Appendix E More Demonstration Examples for Few-shot Experiment

## Appendix F Question Variant Examples

We present several examples of generated variants of seed questions from our dataset along with corresponding responses from GPT-4o (Figure 9), Gemini (Figure 10), and Qwen2-VL-72B (Figure 11). These examples illustrate that the leading VLMs exhibit a lack of robustness to variations in image input, demonstrating unreliable perception and reasoning abilities.

Figure: Figure 9: Example of the generated variants of Question 169 and the corresponding responses from GPT-4o.
Refer to caption: x36.png

Figure: Figure 10: Example of the generated variants of Question 75 and the corresponding responses from Gemini.
Refer to caption: x37.png

Figure: Figure 11: Examples of the generated variants of Question 346 and the corresponding responses from Qwen2-VL-72B are provided. Notably, variant 3 derives the correct answer but has an erroneous perception.
Refer to caption: x38.png

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: In addition to the problems in weaknesses. There are some extra questions and suggestions.
1. Can we convert the existing problem into a dynamic/programmatic version?
2. Including analysis of the correlation between different types of variations and model performance will be better.
3. Investigate potential relationships between training data/training techniques and performance on different variation types if possible.
soundness: 3
strengths: 1. The dynamic nature of the benchmark is innovative and addresses a critical gap in existing evaluation methods for VLMs' mathematical reasoning capabilities.
2. The evaluation is comprehensive. The author tests multiple aspects of mathematical reasoning across 9 different topics and they evaluate both closed and open-source models. The detailed performance includes various types of variations (numerical, geometric, functional, etc.).
3. The ablation analysis is strong: (a) detailed performance breakdown by topic and difficulty level;(b) Introduction of meaningful metrics (worst-case accuracy, reasoning robustness);(3)Thorough error analysis with clear categorization
4. The methodology is well-documented. The author provides a clear explanation of benchmark creation process.
summary: This paper introduces DYNAMATH, a dynamic visual benchmark designed to evaluate the mathematical reasoning robustness of Vision-Language Models (VLMs). DYNAMATH contains 501 seed questions implemented as Python programs that can generate diverse variants by altering visual conditions while maintaining the same underlying mathematical concepts. The benchmark spans multiple mathematical topics and difficulty levels. The authors evaluated 14 state-of-the-art VLMs (both closed and open-source) and found significant gaps between average-case and worst-case performance, indicating limitations in VLMs' mathematical reasoning robustness.
weaknesses: 1. The authors acknowledge that the difficulty level is relatively limited compared to some existing benchmarks like MATH-V. The requirement for programmatic generation may restrict the inclusion of more complex problems
2. The selection of seed questions and their variants might introduce unintended biases. The paper doesn't deeply discuss potential limitations in the variation generation process.
3. The human evaluation seems limited (only 10 participants). It is unclear if the human evaluators' backgrounds are representative, which may limit the validation of human performance.
4. Generalization of the conclusion on realistic questions. The questions (especially the vision content) are created by the program, which is quite different from the figures in the application (like the exam paper; hand-written question, or photo of the book). If the model fits well on these tasks, does it mean the model performs well on the realistic scenario?
5.Missing in-depth analysis of the performance gap between the worst-case and the standard accuracy. If we use a large amount of synthetic data in a similar manner, can the model work well also on these problems? If the performance gap can be resolved by introducing more synthetic data, the conclusion will be trivial and have no interesting news.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. It is uncertain if the benchmark could be easily hacked by just adopting the same program-based question generation to produce training data and then finetune the models.
2. Is the reasoning robustness relevant to the difficulty level and type of questions and variants?
3. Is there any preliminary study on how to scale the dynamic generation approach for more comprehensive and difficult benchmarks?
4. It would be insightful to know if the authors have tested any interventions (e.g., prompt adjustments) to increase the robustness and reduce the memorization phenomenon.
5. Is there any interpretability of the observations on the mathematical reasoning robustness?
soundness: 3
strengths: - The proposed dynamic benchmark for evaluating the mathematical reasoning robustness of VLMs are complementary to existing benchmarks, and it aligns well with the challenges in real-world mathematical applications, where variations in problem conditions often arise.
- Extensive evaluations on reasoning robustness are conducted and some observations are beneficial to the community.
summary: This paper introduces a dynamic visual benchmark designed to evaluate the mathematical reasoning robustness of VLMs. Unlike static benchmarks, DynaMath generates question variations programmatically to test models under diverse conditions. This allows for a nuanced assessment of VLMs’ robustness and generalization across mathematical tasks.
weaknesses: - Although the paper presents some error analysis, the model robustness is still underexplored. Quite a few factors are not investigated, such as the difficulty level and type of questions and variants.
- Besides the question itself, the prompt template can also result in different responses. The robustness regarding to prompt variants is neither studied nor excluded from the research scope with clear explanation.
- Deeper insights into the failure modes and interpretability would be great.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: Can the authors provide a rough time estimate for designing/redesigning seed questions? This is important for future benchmark design. Also, how can we scale up the design flow for a larger dynamic math benchmark?
soundness: 4
strengths: 1. The paper effectively presents comprehensive and interesting findings.
2. The benchmark's controllability should be highlighted as a key strength.
summary: DynaMATH, a new benchmark for evaluating visual reasoning in vision language models, offers greater controllability than previous tools like MathVistas. Using 501 programmable seed questions with multiple variants, it tests both closed and open-source VLMs, revealing insights about worst-case accuracy and memorization patterns.
weaknesses: 1. Move Figure 7 to the main paper to better illustrate the benchmark's appearance.
2. The paper lacks details about the seed question curation process and should address the benchmark's scalability limitations.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What about the zero-shot / few-shot COT results for open-source VLMs? It would be beneficial to present those results and do some analysis. 
2. Evaluating VLMs with temperature not set to 0 will make the results less reproducible. Will the evaluation result vary much when you run the evaluation multiple times? Can you provide an error bar for it? 
3. Would you please provide the detailed distribution of problems in DYNAMATH (multiple choice, answer is float, answer is text)? 
4. In MMBench [1], the authors have defined another consistency named circular consistency? Will the Repetition Consistency still be that high (>80%) when circular shift is applied to choices for MCQ problems in DYNAMATH?

[1] MMBench: Is Your Multi-modal Model an All-around Player?
soundness: 3
strengths: 1. DYNAMATH is the first dynamic visual math benchmark designed to evaluate the robustness of mathematical reasoning in VLMs. 
The dynamic generation of varied problem sets more accurately simulates real-world problem-solving, providing a more genuine assessment of model capabilities.
2. With 501 high-quality seed questions that can generate a multitude of variant problems across various mathematical topics and difficulty levels, DYNAMATH allows for an in-depth evaluation of VLMs' generalization abilities under different input conditions, revealing the limitations of current models when dealing with problem variations.
summary: The paper introduces DYNAMATH, a dynamic visual math benchmark aimed at evaluating the mathematical reasoning robustness of Vision-Language Models (VLMs). 
The benchmark reveals that state-of-the-art VLMs, like GPT-4o, struggle with mathematical reasoning when visual numerical values or function graphs change, despite these tasks being simple for humans. 
DYNAMATH includes 501 seed questions, each represented as a Python program, capable of generating varied concrete questions with different visual and textual alterations. 
The benchmark was used to assess 14 VLMs (including proprietary ones and open-source ones) with 5,010 generated questions, showing a significant gap between average-case and worst-case model accuracy. 
The paper's contributions include identifying the weakness in VLMs' reasoning abilities, introducing DYNAMATH for robustness evaluation, and providing insights to guide the development of more reliable VLMs for mathematical reasoning.
weaknesses: 1. Lacks analysis of open-source VLMs performance under the zero-shot / few-shot setting.
2. Lacks analysis of the reproducibility of evaluation results (since temperature is not 0).

Meta-review: This work proposed a new benchmark called DynaMath, to examine the math reasoning capability and roboustness for VLMs. Drawing the inspiration from how human conduct reasoning on the math problems, the authors proposed a programmatic way of generating a set of visual reasoning question variations based on 501 seed questions spanning in different domains. These variations play an important role to study how the sutle changes to the base question would affect the final performance of VLMs. Through extensive evaluations, the authors showed that even the state-of-the-art model like GPT-4o is still not robust enough to handle the math reasoning problems. These findinds provide a new perspective about the visual reasoning capability for VLMs and shed lights to the degisn of more robust visual reasoning models.

The main contributiion of this work is the proposal of a new math reasoning benchmark and the associated extensive evaluations on a wide range of VLMs. As pointed out by the reviewers, the proposed benchmark differs from previous one in that it provides a controllable setting to make sure the image and question varients are sourced from the same type, and thus make it possible to examine the robustness of VLMs. The ACs believe this new benchmark will bring a lot benefit to the community while developing more robust visual reasoning models for math problems. There are a few weaknesses for this work, while most of them were addressed. In particular, reviewers had concerns about the generality of the proposed method to generate evaluation sample varients, and some other reviewers questioned the unintended bias for the dataset as it originates from 501 seed questions. The ACs think the response from the reviewers are mostly persuasive. However, one point to note is that the proposed benchmark is still a static one depsite the full set was generated by altering the seed one dynamically (as pointed out by Reviewer bDGf). Ideally, a true dynamic benchmarks should work like a moving target for which the VLMs cannot easily tackle by memorization. In this sense, the ACs suggest the authors to define the 'dynamic' clearly to avoid misunderstanding in the final version.

This work got four positive ratings (6,6,8,8), which clearly demonstrate its strengths and contributions to the community. As such, the ACs recommend an acceptance. Given that the proposed benchmark is the first that can support evaluating the robustness for VLMs in a controllable way, the ACs think it worth to highlight this work in the conference.

Decision: Accept (Poster)