DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models

Authors: Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, Huan Zhang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Visual Mathematical Benchmark, Vision Language Models

Abstract:
The rapid advancements in Vision-Language Models (VLMs) have shown great potential in tackling mathematical reasoning tasks that involve visual context. Unlike humans who can reliably apply solution steps to similar problems with minor modifications, we found that state-of-the-art VLMs like GPT-4o can consistently fail in these scenarios, revealing limitations in their mathematical reasoning capabilities. In this paper, we investigate the **mathematical reasoning robustness** in VLMs and evaluate how well these models perform under different variants of the same question, such as changes in visual numerical values or function graphs.
While several vision-based math benchmarks have been developed to assess VLMs' problem-solving capabilities, these benchmarks contain only static sets of problems and cannot easily evaluate mathematical reasoning robustness.
To fill this gap, we introduce **DynaMath**, a dynamic visual math benchmark designed for in-depth assessment of VLMs. **DynaMath** includes 501 high-quality, multi-topic *seed* questions, *each represented as a Python program*. Those programs are carefully designed and annotated to enable the automatic generation of a much larger set of *concrete* questions, including many different types of visual and textual variations. 
**DynaMath** allows us to evaluate the generalization ability of VLMs, by assessing their performance under varying input conditions of a seed question. We evaluated 14 state-of-the-art VLMs with 5,010 generated concrete questions (10 per seed question). Our results show that the worst-case model accuracy, defined as the percentage of correctly answered seed questions in all 10 variants, is significantly lower than the average-case accuracy. In addition, many models show high consistency in answering these questions -- the incorrectness of a certain variant of a seed question is not only due to inherent randomness. Our analysis emphasizes the need to study the robustness of VLMs' reasoning abilities, and **DynaMath** provides valuable insights to guide the development of more reliable models for mathematical reasoning.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730028975490

thread_type:
official_review

<thread_object>
creator: reviewer_nqb6

<content>
summary:
The paper introduces DYNAMATH, a dynamic visual math benchmark aimed at evaluating the mathematical reasoning robustness of Vision-Language Models (VLMs). 
The benchmark reveals that state-of-the-art VLMs, like GPT-4o, struggle with mathematical reasoning when visual numerical values or function graphs change, despite these tasks being simple for humans. 
DYNAMATH includes 501 seed questions, each represented as a Python program, capable of generating varied concrete questions with different visual and textual alterations. 
The benchmark was used to assess 14 VLMs (including proprietary ones and open-source ones) with 5,010 generated questions, showing a significant gap between average-case and worst-case model accuracy. 
The paper's contributions include identifying the weakness in VLMs' reasoning abilities, introducing DYNAMATH for robustness evaluation, and providing insights to guide the development of more reliable VLMs for mathematical reasoning.

strengths:
1. DYNAMATH is the first dynamic visual math benchmark designed to evaluate the robustness of mathematical reasoning in VLMs. 
The dynamic generation of varied problem sets more accurately simulates real-world problem-solving, providing a more genuine assessment of model capabilities.
2. With 501 high-quality seed questions that can generate a multitude of variant problems across various mathematical topics and difficulty levels, DYNAMATH allows for an in-depth evaluation of VLMs' generalization abilities under different input conditions, revealing the limitations of current models when dealing with problem variations.

weaknesses:
1. Lacks analysis of open-source VLMs performance under the zero-shot / few-shot setting.
2. Lacks analysis of the reproducibility of evaluation results (since temperature is not 0).

questions:
1. What about the zero-shot / few-shot COT results for open-source VLMs? It would be beneficial to present those results and do some analysis. 
2. Evaluating VLMs with temperature not set to 0 will make the results less reproducible. Will the evaluation result vary much when you run the evaluation multiple times? Can you provide an error bar for it? 
3. Would you please provide the detailed distribution of problems in DYNAMATH (multiple choice, answer is float, answer is text)? 
4. In MMBench [1], the authors have defined another consistency named circular consistency? Will the Repetition Consistency still be that high (>80%) when circular shift is applied to choices for MCQ problems in DYNAMATH?

[1] MMBench: Is Your Multi-modal Model an All-around Player?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730316928022

thread_type:
official_review

<thread_object>
creator: reviewer_fgu5

<content>
summary:
DynaMATH, a new benchmark for evaluating visual reasoning in vision language models, offers greater controllability than previous tools like MathVistas. Using 501 programmable seed questions with multiple variants, it tests both closed and open-source VLMs, revealing insights about worst-case accuracy and memorization patterns.

strengths:
1. The paper effectively presents comprehensive and interesting findings.
2. The benchmark's controllability should be highlighted as a key strength.

weaknesses:
1. Move Figure 7 to the main paper to better illustrate the benchmark's appearance.
2. The paper lacks details about the seed question curation process and should address the benchmark's scalability limitations.

questions:
Can the authors provide a rough time estimate for designing/redesigning seed questions? This is important for future benchmark design. Also, how can we scale up the design flow for a larger dynamic math benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730799506340

thread_type:
official_review

<thread_object>
creator: reviewer_gltu

<content>
summary:
This paper introduces a dynamic visual benchmark designed to evaluate the mathematical reasoning robustness of VLMs. Unlike static benchmarks, DynaMath generates question variations programmatically to test models under diverse conditions. This allows for a nuanced assessment of VLMs’ robustness and generalization across mathematical tasks.

strengths:
- The proposed dynamic benchmark for evaluating the mathematical reasoning robustness of VLMs are complementary to existing benchmarks, and it aligns well with the challenges in real-world mathematical applications, where variations in problem conditions often arise.
- Extensive evaluations on reasoning robustness are conducted and some observations are beneficial to the community.

weaknesses:
- Although the paper presents some error analysis, the model robustness is still underexplored. Quite a few factors are not investigated, such as the difficulty level and type of questions and variants.
- Besides the question itself, the prompt template can also result in different responses. The robustness regarding to prompt variants is neither studied nor excluded from the research scope with clear explanation.
- Deeper insights into the failure modes and interpretability would be great.

questions:
1. It is uncertain if the benchmark could be easily hacked by just adopting the same program-based question generation to produce training data and then finetune the models.
2. Is the reasoning robustness relevant to the difficulty level and type of questions and variants?
3. Is there any preliminary study on how to scale the dynamic generation approach for more comprehensive and difficult benchmarks?
4. It would be insightful to know if the authors have tested any interventions (e.g., prompt adjustments) to increase the robustness and reduce the memorization phenomenon.
5. Is there any interpretability of the observations on the mathematical reasoning robustness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731155689259

thread_type:
official_review

<thread_object>
creator: reviewer_bdgf

<content>
summary:
This paper introduces DYNAMATH, a dynamic visual benchmark designed to evaluate the mathematical reasoning robustness of Vision-Language Models (VLMs). DYNAMATH contains 501 seed questions implemented as Python programs that can generate diverse variants by altering visual conditions while maintaining the same underlying mathematical concepts. The benchmark spans multiple mathematical topics and difficulty levels. The authors evaluated 14 state-of-the-art VLMs (both closed and open-source) and found significant gaps between average-case and worst-case performance, indicating limitations in VLMs' mathematical reasoning robustness.

strengths:
1. The dynamic nature of the benchmark is innovative and addresses a critical gap in existing evaluation methods for VLMs' mathematical reasoning capabilities.
2. The evaluation is comprehensive. The author tests multiple aspects of mathematical reasoning across 9 different topics and they evaluate both closed and open-source models. The detailed performance includes various types of variations (numerical, geometric, functional, etc.).
3. The ablation analysis is strong: (a) detailed performance breakdown by topic and difficulty level;(b) Introduction of meaningful metrics (worst-case accuracy, reasoning robustness);(3)Thorough error analysis with clear categorization
4. The methodology is well-documented. The author provides a clear explanation of benchmark creation process.

weaknesses:
1. The authors acknowledge that the difficulty level is relatively limited compared to some existing benchmarks like MATH-V. The requirement for programmatic generation may restrict the inclusion of more complex problems
2. The selection of seed questions and their variants might introduce unintended biases. The paper doesn't deeply discuss potential limitations in the variation generation process.
3. The human evaluation seems limited (only 10 participants). It is unclear if the human evaluators' backgrounds are representative, which may limit the validation of human performance.
4. Generalization of the conclusion on realistic questions. The questions (especially the vision content) are created by the program, which is quite different from the figures in the application (like the exam paper; hand-written question, or photo of the book). If the model fits well on these tasks, does it mean the model performs well on the realistic scenario?
5.Missing in-depth analysis of the performance gap between the worst-case and the standard accuracy. If we use a large amount of synthetic data in a similar manner, can the model work well also on these problems? If the performance gap can be resolved by introducing more synthetic data, the conclusion will be trivial and have no interesting news.

questions:
In addition to the problems in weaknesses. There are some extra questions and suggestions.
1. Can we convert the existing problem into a dynamic/programmatic version?
2. Including analysis of the correlation between different types of variations and model performance will be better.
3. Investigate potential relationships between training data/training techniques and performance on different variation types if possible.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732239812050

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 1

comment:
We sincerely thank all the reviewers for investing their valuable time to review our work. We are encouraged to see that the dynamic nature of DynaMATH is recognized as **innovative** and **addresses a critical gap** in the existing evaluation methods (bDGf, Nqb6, Fgu5); our benchmark **complements existing ones** and **aligns well with the real-world mathematical applications**, where the problem conditions often vary (GLTU, Nqb6). Reviewers praised our **comprehensive evaluation** across diverse topics and models (bDGf, GLTU, Fgu5, Nqb6), found our **findings interesting** (Fgu5), and appreciated our **clear documentation** (bDGf). 

In response to the reviewers comments, we have uploaded a revised version of our paper, where the new results and major modifications are highlighted in orange. Here are the list of the updates to the original submission:

1. Section 4: we updated the evaluation results with temperature = 0. 

2. Section 4: we included few-shot CoT results on Qwen2-VL-72b (Table 2 and Table 3)

3. Section 4: we updated the human evaluation results to include a total of 20 participants.

4. Appendix A: we added more discussions on the limitations and scalability of DynaMATH.

5. Appendix D: we present more examples and variants on each problem type. 

6. Appendix H: we have added more experimental results including: reasoning robustness across different variants types, problem types, and difficulty levels (Appendix H.1, H.3), in-depth failure case analysis (Appendix H.2), results with other prompt template (Appendix H.4, H.5), evaluation the robustness of DynaMATH (Appendix H.6), average accuracy variance with multiple over repeated evaluations (Appendix H.7), and more results on the circular consistency (Appendix H.8).

Besides these new results and technical discussions added above, we also made minor revisions in many aspects to further improve paper quality overall, including typo and grammar fixes, figure clarity improvements, and the addition of latest references.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732502895137

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you for the valuable feedback and we look forward to discussing with reviewers

comment:
Dear Reviewers,

We would like to thank you again for your encouraging comments and constructive feedback. We are grateful to all reviewers for positively recognizing our contribution on creating a *dynamic* math benchmark for evaluating the *reasoning robustness* of VLMs. Based on your valuable comments, we have added **comprehensive additional experiments and result analysis**, including evaluation with zero temperature, CoT on open-source models, evaluations under different prompt templates, the addition of a new circular consistency metric, further human evaluations, and more in-depth analysis of results. We also clarified and addressed all questions. Your feedback has helped us to make our paper significantly stronger.

Since the discussion period is closing very soon on Nov 26, we hope you can kindly give us additional feedback and reevaluate our paper based on our responses and the revised paper. We thank you again for positively supporting our paper and we look forward to having fruitful discussions with you.

Sincerely,  
DynaMath Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposed a new benchmark called DynaMath, to examine the math reasoning capability and roboustness for VLMs. Drawing the inspiration from how human conduct reasoning on the math problems, the authors proposed a programmatic way of generating a set of visual reasoning question variations based on 501 seed questions spanning in different domains. These variations play an important role to study how the sutle changes to the base question would affect the final performance of VLMs. Through extensive evaluations, the authors showed that even the state-of-the-art model like GPT-4o is still not robust enough to handle the math reasoning problems. These findinds provide a new perspective about the visual reasoning capability for VLMs and shed lights to the degisn of more robust visual reasoning models.

The main contributiion of this work is the proposal of a new math reasoning benchmark and the associated extensive evaluations on a wide range of VLMs. As pointed out by the reviewers, the proposed benchmark differs from previous one in that it provides a controllable setting to make sure the image and question varients are sourced from the same type, and thus make it possible to examine the robustness of VLMs. The ACs believe this new benchmark will bring a lot benefit to the community while developing more robust visual reasoning models for math problems. There are a few weaknesses for this work, while most of them were addressed. In particular, reviewers had concerns about the generality of the proposed method to generate evaluation sample varients, and some other reviewers questioned the unintended bias for the dataset as it originates from 501 seed questions. The ACs think the response from the reviewers are mostly persuasive. However, one point to note is that the proposed benchmark is still a static one depsite the full set was generated by altering the seed one dynamically (as pointed out by Reviewer bDGf). Ideally, a true dynamic benchmarks should work like a moving target for which the VLMs cannot easily tackle by memorization. In this sense, the ACs suggest the authors to define the 'dynamic' clearly to avoid misunderstanding in the final version.

This work got four positive ratings (6,6,8,8), which clearly demonstrate its strengths and contributions to the community. As such, the ACs recommend an acceptance. Given that the proposed benchmark is the first that can support evaluating the robustness for VLMs in a controllable way, the ACs think it worth to highlight this work in the conference.