PolyMATH: A Challenging Multi-Modal Mathematical Reasoning Benchmark

## Abstract

Abstract Multi-modal Large Language Models (MLLMs) exhibit impressive problem-solving abilities in various domains, but their visual comprehension and abstract reasoning skills remain under-evaluated.
To this end, we present PolyMATH , a challenging benchmark aimed at evaluating the general cognitive reasoning abilities of MLLMs. PolyMATH comprises 5,000 manually collected high-quality images of cognitive textual and visual challenges across 10 distinct categories, including pattern recognition, spatial reasoning, and relative reasoning.
We conducted a comprehensive, and quantitative evaluation of 15 MLLMs using four diverse prompting strategies, including Chain-of-Thought and Step-Back.
The best scores achieved on PolyMATH are ∼ similar-to \sim 41%, ∼ similar-to \sim 36%, and ∼ similar-to \sim 27%, obtained by Claude-3.5 Sonnet, GPT-4o and Gemini-1.5 Pro respectively - highlighting the logical and visual complexity of these questions.
A further fine-grained error analysis reveals that these models struggle to understand spatial relations and perform drawn-out, high-level reasoning.
This is further strengthened by our ablation study estimating MLLM performance when given textual descriptions in place of diagrams.
As evidenced by ∼ similar-to \sim 4% improvement over textual descriptions as opposed to actual images, we discover that models do not truly comprehend visual diagrams and the spatial information therein, and are thus prone to logical errors.
Finally, we evaluate the OpenAI o1 models and find that their performance only matches the human baseline, highlighting the difficulty of the benchmark.
The results on PolyMATH highlight the room for improvement in multi-modal reasoning and provide unique insights to guide the development of future MLLMs 1 1 1 Codebase: https://github.com/kevinscaria/PolyMATH Dataset: https://huggingface.co/datasets/him1411/polymath ∗ * Equal Contribution † † \dagger Currently in Google DeepMind ‡ ‡ \ddagger Currently in Amazon (The work was done prior to joining Amazon) ♢ ♢ \diamondsuit Currently in Microsoft .

## 1 Introduction

Large Language Models (LLMs) (Brown et al., ; Jiang et al., ; Touvron et al., ; Achiam et al., ) and Multi-modal Large Language Models (MLLMs) (OpenAI, ; Team et al., ; Su et al., ; Chen et al., ) have rapidly become a pivotal area of research.
MLLMs with robust reasoning capabilities in visual contexts can solve complex educational problems (Seo et al., ; Wang et al., ), support analysts with logical queries on statistical data (Wu et al., ; Yang et al., ), and contribute to advanced research areas such as theorem proving and scientific discovery (Taylor et al., ; Dong et al., ; Trinh et al., ).
Despite their impressive performance in various assessments of human-like intelligence, these models still exhibit notable shortcomings on tasks requiring cognitive and logical reasoning, such as commonsense numerical reasoning, scientific problem-solving, and abstract puzzles (Wang et al., ; Lu et al., ).
Existing evaluation benchmarks (Fu et al., ; Liu et al., ; Li et al., ; Fu et al., ; Sun et al., ) have focused primarily on specific concrete domains.
While general-purpose visual question-answering (VQA) datasets capture some elements of mathematical reasoning, a systematic investigation into abstract and general cognitive reasoning which are essential for tasks like visual puzzles remains an underexplored frontier.

In this paper, we present PolyMATH, a benchmark specifically crafted to evaluate the complex multi-modal cognitive reasoning capabilities of MLLMs.
We propose a task taxonomy to guide the development of PolyMATH:
(1) we identify ten distinct reasoning skills, including spatial reasoning, pattern recognition, and numerical reasoning.
and (2) we cover a diverse array of visual contexts, including images with venn diagrams, spatially-related layouts, as well as geometric figures.
PolyMATH is a meticulously curated dataset of 5000 multimodal reasoning problems newly acquired from a publicly available source (Table [2](#S3.T2)).
The problems of the original source have been crafted and rigorously reviewed by expert annotators, and require diverse fine-grained problem-solving capabilities.
Additionally, we provide detailed textual representations of diagrams of the samples.
As denoted in fig. [1](#S0.F1), these problems are designed to assess the logical reasoning abilities of the average high school student over text and diagrams.
We observe that MLLMs fail to demonstrate the cognitive reasoning skills required to solve these questions.

We conduct extensive experiments on PolyMATH with state-of-the-art (SOTA) closed-source MLLMs like the Claude family (3.5 Sonnet, 3 Sonnet, 3 Haiku), Gemini-1.5 Pro, and GPT-4o, and 9 open-source MLLMs like LLaVA (34B) and ShareGPT4V.
We evaluate them via zero shot, few shot, Chain-of-Thought (Wei et al., ) and step back prompting (Zheng et al., ).
We show that PolyMATH is a challenging benchmark, with human performance (established by qualified human annotators with graduate degrees) reaching only 66.3% accuracy.
The most powerful model we evaluate, Claude-3.5 Sonnet, achieves the best score of 41.90% followed by GPT-4o, which attains 36.50%.
The best open source models like LLaVA-v1.6 Mistral (7B) and ShareGPT4V (13B) achieves the accuracy of 15.20% and 12.80% respectively.
We additionally create a diagram only subset (test-img)
of the benchmark to gauge the gap in visual reasoning abilities between the multi-modal models and average human capability.
We find that the performance of these models drops further to 26.20% for Claude-3.5 Sonnet and 22.50% by Gemini-1.5 Pro when evaluated on test-img only.
In contrast with human cognitive patterns, when given text descriptions in place of the diagram in these questions, model accuracy improves by $\sim$4-7%.
We also conduct an error analysis on Claude-3.5 Sonnet, Gemini-1.5 Pro and GPT-4o, and find that the most common errors stem from misunderstanding diagrams ($\sim 60\%$), misidentifying logical patterns ($\sim 25\%$), and forgetting relational information ($\sim 12\%$).
Finally, we evaluate OpenAI o1 models (OpenAI, ) on without diagram questions of the benchmark and observe 66.72% accuracy (o1-preview), 2% points below than the human baseline.

Figure: (a) Dataset categorization
Refer to caption: /html/2410.14702/assets/figs/dataset_stats_pie_chart_wdwo.png

## 2 Related Work

The development of MLLMs builds on the progress of LLMs (Touvron et al., ; ; OpenAI, ; Jiang et al., ) and large vision models (Kirillov et al., ; Zhang et al., ; ; ).
These models extend LLMs to handle a wider range of tasks across multiple modalities, including 2D images  (Li et al., ; Dai et al., ; Alayrac et al., ; Li et al., ), 3D point clouds  (Guo et al., ; Xu et al., ), audio  (Han et al., ; Su et al., ), and video  (Zhang et al., ; Chen et al., ). Notable examples like OpenAI’s GPT-4V  (OpenAI, ) and Google’s Gemini  (Team et al., ) demonstrate advanced visual reasoning capabilities, setting new benchmarks in the multimodal space.

As MLLMs rapidly advance  (Li et al., ), there is a growing need for benchmarks that evaluate mathematical problem-solving in visual contexts.
Existing benchmarks, such as GeoQA (Chen et al., ), VQA (Goyal et al., ), and UniGeo (Chen et al., ), focus mostly on geometric problems.
Other efforts target skills in abstract scenes, geometry diagrams, charts, and synthetic images (Chen et al., ; Masry et al., ).
Recent datasets also assess external knowledge, commonsense reasoning, and scientific or medical understanding (Zhang et al., ).
MathVista (Lu et al., ) expands multimodal math tasks, while MMMU (Yue et al., ) focuses on college-level problems. Prior work evaluates LLMs across diverse domains like QA, mathematics, and science (Bubeck et al., ; Nori et al., ), while recent research (Zhang et al., ) explores whether models like GPT-4V perform vision and language tasks independently or together.

Existing extensive benchmarks  (Fu et al., ; Liu et al., ; Li et al., ; Xu et al., ) primarily focus on concrete, real-world problems within specific domains.
These benchmarks often include comparatively simple diagram interpretation questions involving plots or mathematical questions related to geometry, which primarily evaluate models’ abilities to parse information from a single image and solve problems using well-established logical principles and formulae.
However, they do not sufficiently test models’ capabilities in abstract visual reasoning, including spatial recognition, visual logic and puzzle solving, and pattern recognition.
This limitation represents a notable gap, as visual puzzle tasks require logical leaps that differ fundamentally from reasoning patterns over textual or linguistic problems.
Moreover, spatial reasoning questions assess models’ abilities to internalize and manipulate configurations in 3D space, as well as reason over spatial information and infer implicit relationships based on positional data.
This category of questions aligns closely with human cognition and reasoning abilities, and evaluating model performance against human baselines on these questions reveals the substantial gap in reasoning abilities that models must bridge to approach human-comparable reasoning capability.
Our proposed dataset aims to address this gap by challenging and comprehensively evaluating previously underexplored model skills in categories where their performance still lags significantly behind human reasoning baselines.
Additionally, we provide a detailed analysis of the strengths and weaknesses of these models across a wide range of categories and skills, shedding light on specific reasoning errors and their frequency of occurrence across categories and in comparison to one another.

Figure: Figure 3: Examples of with diagram and without diagram questions. In addition to the question image, PolyMATH includes the metadata shown above. Question without diagram is not present in test-img while both kinds of questions will be present in testmini.
Refer to caption: /html/2410.14702/assets/figs/wdwo_example_diag_schema.png

## 3 Curating PolyMATH

Table: Table 1: An overview of our question categorization schema. Questions are categorized on the basis of the information provided in the question and the reasoning skills assessed.

PolyMATH is curated mainly from questions directed at students taking the National Talent Search Examination, a nationwide competitive exam held by the National Council of Educational Research and Training of India.
These questions and their solutions are created by experts in their fields and rigorously peer-reviewed, and thus contain minimal errors.
These questions aim to assess Scholastic Aptitude (SAT), or the ability to recall domain-specific scientific and mathematical knowledge, as well as Mental Ability (MAT), or the ability to think logically and apply a range of analytical skills.
We catalog the skills assessed by each sample along the categorization schema defined in Table [1](#S3.T1).

### 3.1 Collection Pipeline

To guarantee high-quality data, we manually collected image snippets and engineered a streamlined, automated framework for curation and annotation.
Continuous human reviews were conducted throughout the process, ensuring quality and preventing error propagation.

- •
Step 1: We generate a universally unique identifier (UUID) for a given question paper to identify all the questions curated from it.
- •
Step 2: Annotators manually collected separate snippets of each question and their associated contextual information (including disconnected pieces) that apply to multiple questions.
- •
Step 3: An image merging script automatically identified and merged question images (in case the question gets split by pages) with their relevant context images.
- •
Step 4: We used an LLM to transcribe the questions and their ground truth answers.
We also generate additional metadata, including category (§[3.2](#S3.SS2)), whether it contains a diagram (Fig [3](#S2.F3)), and image description (§[3.3](#S3.SS3)).
A manual check was performed to ensure the quality of the generated metadata.
- •
Step 5: An annotation file, where each row corresponds to a question, is automatically created and populated.

### 3.2 Dataset categorization

We develop a categorization schema that catalogues questions on basis of the information provided and the type of reasoning assessed by the question.
Based on the continuous human evaluation during collection, we identify 10 distinct question categories.
We enumerate these categories along with their definitions in Table [1](#S3.T1).
We further distinguish between questions with diagram and without diagram.
The with diagram questions are designed around the information presented in the diagrams (Fig [3](#S2.F3)).
We show examples of with diagram and without diagram questions for each category in §[F](#A6).
The overall per-category distribution, along with the with diagram and without diagram split, is visualized in Figure [2](#S1.F2).

### 3.3 Additional metadata

The complexity of collected question images and the heavy presence of diagram-based reasoning tasks makes PolyMATH a challenging multi-modal benchmark.
To make PolyMATH usable for both text and vision model evaluations, we provide transcriptions of questions and answers.
To further facilitate text-based evaluation, we generate detailed, human-vetted text descriptions of attached diagrams such that a human could visualize the image based on this description (Fig [3](#S2.F3)).
Results on text-only characterization of questions in our dataset can be found in §[4.3](#S4.SS3.SSS0.Px1).

### 3.4 Quality Assurance

Following the collection and annotation process, we conduct a comprehensive quality check.
We discard samples that are [1] of low resolution, [2] outside the scope of the categories (Table [1](#S3.T1)), or [3] missing vital information.
We also discard samples with noticeable watermarks and other visual noise that renders the sample illegible.
Our subject-expert annotators rectify incorrectly-extracted ground truth answers.
Concurrently, we verify that the questions belong to their assigned categories, and correct any observed misalignments therein.

Table: Table 2: An overview of the per-category distribution of questions in the test, testmini, and test-img splits of PolyMATH. testmini and test-img are 1000-instance subsets, aimed at faster and image-focused evaluations respectively. We also report the frequency of with diagram and without diagram questions for each category.

### 3.5 Division of the testmini Subset.

The final iteration of PolyMATH comprises 5000 questions.
To enable faster model validation, we extract a 1000-instance subset, testmini, using stratified sampling over all categories.
All quantitative results reported in this work were obtained on this testmini subset of PolyMATH.
We also create a test-img
question set, consisting solely of 1000 with diagram questions, aimed at faster, focused assessment of models’ visual comprehension.
Owing to the imbalance of with diagram questions across categories, we use a random sampling strategy to create test-img.
(^2^22All datasets (test, testmini and test-img) will be publicly released)
For data distribution, see Table [2](#S3.T2).
Further details on data collection and annotation are available in §[C](#A3).

## 4 Experiments

We conduct a systematic evaluation of existing MLLMs on PolyMATH.
We first introduce the experimental setup in this section.
Then we present our findings followed by multiple dataset analysis experiments.
Additional experimental results and qualitative examples are present in §[D](#A4) and [H](#A8).

### 4.1 Experimental Setup

#### Evaluation Models:

We examine the performance of foundation models across two distinct categories on PolyMATH: (a) Closed-source MLLMs, represented by models like GPT-4o (gpt-4o-2024-05-13) (OpenAI, ),OpenAI O1 (o1-preview-2024-09-12, o1-mini-2024-09-12) (OpenAI, ), Gemini-1.5 Pro (gemini-1.5-pro-002) (Team et al., ), Claude-3.5 Sonnet (claude-3-5-sonnet-20240620) (Anthropic, ) and Claude 3 Haiku and Sonnet (claude-3-sonnet-20240229, claude-3-haiku-20240307) (Anthropic, )
(b) Open-source MLLMs, such as
LLaVA (v1.5-13B, v1.6-Mistral-7B, v1.6-Vicuna-13B) (Liu et al., ),
LLaVA-v1.6-34B (Liu et al., ), G-LLaVA (7B, 13B) (Gao et al., ), ShareGPT4V (7B, 13B) (Chen et al., ) & Qwen2-VL-2B-Instruct (Wang et al., ) (c) Text Based LLMs Reka Flash (Ormazabal et al., ), Llama-3 (70B) (AI@Meta, ), Mistral Large (AI, ).
We conduct experiments on all open-source models using six NVIDIA A100 GPUs.
Hyperparameters are available in §[D](#A4).

Table: Table 3: Results of closed-source MLLMs on the testmini split of PolyMATH. We report model results using the following prompting strategies: zero shot inference, few short inference, Chain-of-Thought, and Step Back prompting. For each prompting setting, the highest and lowest scores achieved by a model per category are highlighted. In addition to model accuracy, we report a Random chance baseline (i.e. the accuracy of a model that randomly selects an option without visibility into the question, and a Human eval baseline, where we report the average scores of six human evaluators.)

Table: Table 4: Results of open-source MLLMs on the testmini split of PolyMATH. We report model results using zero shot inference. The highest and lowest scores achieved by a model in each category are highlighted.

#### Implementation Details

All reported results are based on the $testmini$ subset of the dataset.
To establish a baseline for comparison, we simulate random chance by selecting a random option for multiple-choice questions over 1000 trials.
Additionally, the problems in PolyMATH were independently solved by the paper’s authors (four engineering graduates and two PhDs), serving as a human performance baseline.
We evaluate the benchmark using various prompting methods, including zero shot, few shot (2-shot), Chain-of-Thought (Wei et al., ), and Step Back prompting (Zheng et al., ).
For multiple-choice questions, we use exact match for answer comparison.
The model inference prompts are structured to elicit a step-by-step solution, the final answer, and the corresponding option.
Details about these prompts are provided in §[E](#A5).
As part of our analysis, we conducted three additional experiments: (1) analyzing model performance on the test-img split, (2) converting the questions from test-img into text, along with the transformation of diagrams into descriptions, and (3) evaluating OpenAI o1 models on questions without diagrams.

Table: Table 5: Results of visual comprehension ablation study test-img split of PolyMATH. We use MLLMs and conduct multi-modal inference on questions containing diagrams, and then use the same MLLMs to infer on the same questions, but with a detailed text description in place of the diagram. For each inference setting, the highest and lowest scores achieved by a model per category are highlighted. Additionally, we report the performance of text-only LLMs on the textual representation of these questions.

### 4.2 Results

#### Closed Source Models

Across various prompting strategies (Table [3](#S4.T3)),
Claude-3.5 Sonnet performed best with these advanced prompts, achieving up to 41.90% accuracy in Step Back Prompting, compared to 39.70% in zero shot.
GPT-4o followed closely, especially in FC and PS questions, showing strong performance with zero shot and Step Back Prompting.
Gemini-1.5 Pro performed moderately across all categories but lacked dominance in any specific area, while Claude Haiku being the smallest of the closed sourced MLLMs,
consistently underperformed across all prompting strategies.
In terms of prompting strategies, Chain-of-Thought and Step Back Prompting enhanced the performance of top models like Claude-3.5 Sonnet and GPT-4o, allowing them to excel in tasks requiring structured reasoning and re-evaluation.
Both strategies led to marked improvements over zero shot prompting, particularly in categories like SR, PR, and LR.

#### Open Source Models

Table [4](#S4.T4) showcases the results of popular open-source MLLMs.
LLaVA-v1.6-Mistral-7B model achieved the highest overall score of 15.2%, demonstrating remarkable performance across several categories.
Notably, it excelled in OD (22.78%), SR (19.54%), RR (18.1%), and MR (15.18%) indicating its proficiency in generating precise, coherent, and relevant responses, even for out-of-distribution samples.
The ShareGPT4V (13B) model exhibited the second-highest overall score of 12.8%, with outstanding performance in the PR (13.66%), SC (12.5%), RR (15.24%), MR (10.71%), SR (12.64%), and OD (17.72%) categories.
Other models, such as LlaVA-v1.6-Vicuna 13B, LlaVA-1.5 (13B), G-LLaVA (13B), and LlaVA-v1.6 (34B), exhibited varying levels of success across the different categories, highlighting their individual strengths and weaknesses in handling the diverse reasoning aspects tested by the dataset.

#### Human Evaluation

To ascertain the difficulty of the dataset, we asked six graduate students specifically for the evaluation of human performance on PolyMATH.
We assigned questions from a specific problem category to each student.
This strategy aimed to prevent them from gaining additional information from another question from same category.
They were asked to provide only the final answer without detailed reasoning.
Therefore, we do not report the Chain-of-Thought evaluation results for human performance, alongside the ‘Random Chance’ baseline.

### 4.3 Experimental Analysis

#### MLLMs Rely More on Image Descriptions than Image

To evaluate the visual reasoning capabilities of closed-source models, we conducted inference on the test-img subset, which contains questions with diagrams.
Additionally, we generated a text-only version of test-img by replacing all diagrams with detailed textual descriptions.
Both experiments were carried out in a zero shot setting.
Our analysis reveals three key findings. First, we observed a noticeable decline in performance on test-img, particularly for models like GPT-4o and Claude-3.5 Sonnet, compared to their results on the testmini subset. This suggests that both models perform well on questions without diagrams, and their decreased accuracy on test-img is largely due to the presence of diagram-based problems.
Second, when we replaced the diagrams in test-img with text descriptions, the performance of all models improved by approximately $\sim 3-4\%$, indicating that the models struggle with visualizing diagrams and benefit from textual representations.
Finally, we evaluated popular text-only LLMs such as LLaMA-3 (70B), Reka Flash, and Mistral Large on the text-description version of test-img.
Their scores ($\sim 15\%$) were significantly lower than those of the MLLMs ($\sim 27\%$), underscoring the advantage of multi-modal models in handling visually-grounded tasks.

Table: Table 6: The types of errors found in model reasoning patterns. The errors are defined to be mutually distinct and leave very little room for ambiguity. We also report the frequency of these errors for each model (Gemini-1.5 Pro, Claude-3.5 Sonnet, GPT-4o) over the 236 questions analysed.

#### A Closer Look at Model Errors

We analysed total of 236 samples where all three state of the art MLLMs (Claude-3.5 Sonnet, GPT-4o and Gemini-1.5 Pro) gave incorrect answers on testmini.
Based on the manual inspection of the responses, we identified 7 types of errors that MLLMs make (Table [6](#S4.T6)).
The total error distribution of all three models is present in Table [11](#A4.T11).
Qualitative examples for category-wise errors are present in §[H](#A8).
The most common error on this dataset was Logical Flaw (LF), occurring in nearly $\sim 60\%$ of incorrect samples.
Spatial Misunderstanding (SM), which involves a lack of understanding of diagram structure and content, was a close second ($\sim$ 25%).
Figure [4](#S4.F4) shows the category-wise distribution of the two types of error.
These errors were most prevalent in OD, PR, and SC category of questions, as making uncommon logical leaps and fully comprehending visual information (which models fall short of) is integral to solving these questions.
Additionally, in questions involving extrapolation over multiple weakly connected data points, models came to conclusions that contradicted earlier data, pointing to a lack of information retention.
Finally, we find that models fell into the same fallacious reasoning patterns as one another - for example, making the assumption that a pattern holds across each row, when the correct reasoning involves a pattern replicated across columns.
The category with the highest % of shared errors was PR, where we observed that GPT4-o, Gemini-1.5 Pro, and Claude-3.5 Sonnet followed the same incorrect reasoning structure on nearly 80% of the analysed samples.
Thus, despite their differences, in practice we see that MLLMs share the same strengths and shortcomings.
For more details, see §[G](#A7).

Figure: Figure 4: Frequency of Logical Flaw (LF) and Spatial Misunderstanding (SM) errors across different question categories. We report per-model figures to enable a comparison of model abilities. They are most prevalent in the OD, PR, and SC categories of questions, owing to the amount of logical leaps and visual reasoning required by these questions.
Refer to caption: /html/2410.14702/assets/figs/error_analysis.png

#### Evaluation of OpenAI o1 models

To understand the capabilities of OpenAI’s latest text-only reasoning models (o1-preview and o1-mini), we evaluate these models on 595 questions of testmini that do not contain diagrams.
We also present human baseline scores on the without diagram section of testmini.
The results of our study are presented in Table [7](#S4.T7).
o1-preview ($\sim 67\%$) has scores that are competitive with human performance ($\sim 68\%$), while o1-mini ($\sim 58\%$) lags behind the human baseline by 10%.

Table: Table 7: Results of OpenAI o1-mini and o1-preview on the without diagram (text-only) samples from the testmini split. We observe that while overall, human cognitive abilities have a slight edge over o1 models, over certain categories (PR, MR), o1 models outperform human performance.

## 5 Conclusion

In this work, we introduce PolyMATH, a benchmark designed to systematically analyze the mathematical reasoning capabilities of state-of-the-art models in visually complex scenarios.
Our evaluation of 14 prominent foundation models highlights that significant advancements have been made, especially with the GPT-4o and Claude-3.5 Sonnet models.
However, a substantial gap of $\sim 24\%$ still exists between Claude-3.5 Sonnet, the best-performing model, and human performance.
This disparity sets a clear direction for future research, emphasizing the need for models that can seamlessly integrate mathematical reasoning with visual comprehension.
Moreover, our analysis of model reasoning errors and experiments on samples containing diagrams and their textual representations offer valuable insights for future investigations.

## Appendix

## Appendix Overview

- •
Section [A](#A1): Limitation and Future Work
- •
Section [B](#A2): Extended Related Work
- •
Section [C](#A3): Data Collection Pipeline Details
- •
Section [D](#A4): Additional Experimental Details
- •
Section [E](#A5): Prompts for Dataset Curation and Experiments
- •
Section [F](#A6): Dataset Examples
- •
Section [G](#A7): More details on Error Analysis
- •
Section [H](#A8): Qualitative Error Analysis

## Appendix A Limitation and future work

Our benchmark, PolyMATH, makes key contributions by integrating mathematical and visual tasks.
While we have made progress in evaluating model performance, we recognize certain limitations.
One limitation is dataset coverage.
Although PolyMATH covers a wide range of tasks and visual contexts, some mathematical problems and visual types may be underrepresented.
Additionally, focusing on mathematical reasoning within visual contexts, especially in domains like competitive high-school-level questions involving problems in spatial and logical reasoning, requires a more labor-intensive data collection process than text-only or general-purpose datasets.
Consequently, the scalability and generalizability of our benchmark to other areas remain challenging.
Annotations were performed by the authors meticulously, however, due to the diversity of questions and images appearing in these sources, the annotations lack a consistent format.

In future iterations, our benchmark will aim to cover a wider range of problems and visual contexts, with unified and comprehensive annotations.
This benchmark is part of an ongoing research effort, and we are committed to maintaining and refining the datasets, including addressing potential data noise, based on community feedback. Additionally, we will adapt the leaderboard to reflect new model developments.
In conclusion, despite the limitations of our current approach, PolyMATH marks a significant advancement in the field.
We remain dedicated to continuously improving the benchmark to deepen our understanding of AI’s capabilities in mathematical and visual reasoning.

## Appendix B Extended Related Work

High-quality evaluation datasets and benchmarks are crucial for assessing the progress of machine learning models in solving real-world tasks (Liao et al., ). Mathematical reasoning benchmarks have emerged as a significant focus area, posing challenges for large foundational models like Large Language Models (LLMs) and Multi-modal Large Language Models (MLLMs). Initial datasets addressed basic algebraic (Hendrycks et al., ) and arithmetic (Roy & Roth, ) word problems with limited scope. Subsequent efforts, including MATH (Hendrycks et al., ), GSM8K (Cobbe et al., ), MMLU (Hendrycks et al., ), and others (Zhou et al., ; Yue et al., ; Wang et al., ; Gao et al., ; Luo et al., ), expanded the range and quality of textual mathematical problems, establishing robust benchmarks for LLM evaluation.

Despite substantial mathematical reasoning encapsulated in visual modalities, most existing benchmarks (Amini et al., ; Cobbe et al., ; Mishra et al., ; Frieder et al., ; Lu et al., ) are textual only. Moreover, some datasets exhibit performance saturation, with GPT-4 achieving 92.0% accuracy on GSM-8K (Cobbe et al., ), a grade-school mathematics dataset. The rapid advancement of Large Multimodal Models (LMMs) necessitates robust multimodal benchmarks, as current benchmarks (Antol et al., ; Kembhavi et al., ; Kahou et al., ; Mathew et al., ) provide limited coverage of rigorous scientific domains crucial for general-purpose AI assistants.

While these benchmarks assess text-only mathematical reasoning, the rapid progress of MLLMs necessitates high-quality benchmarks for evaluating visual mathematical problem-solving. Prior attempts like GeoQA (Chen et al., ), while MathVista (Lu et al., ) and MMMU (Yue et al., ) incorporated various multimodal tasks and college-level questions, respectively.

MLLMs, building upon LLMs (Touvron et al., ; ; OpenAI, ; Jiang et al., ; Brown et al., ) and large vision models (Radford et al., ; Kirillov et al., ; Zhang et al., ; ; ), have become increasingly prominent. They extend LLMs to diverse tasks and modalities, including 2D images (Li et al., ; Dai et al., ; Alayrac et al., ; Li et al., ), 3D point clouds (Guo et al., ; Xu et al., ; Hong et al., ), audio (Han et al., ; Su et al., ), and video (Zhang et al., ; Chen et al., ). Noteworthy examples like OpenAI’s GPT-4V (OpenAI, ) and Google’s Gemini (Team et al., ) exhibit exceptional visual reasoning capabilities, setting new benchmarks in multi-modal performance.

However, their closed-source nature hinders broader application and development of MLLMs. Concurrently, open-source MLLMs like LLaMA-Adapter (Zhang et al., ; Gao et al., ), LLaVA (Liu et al., ; ; ), MiniGPT-4 (Zhu et al., ; Chen et al., ), mPLUG-Owl (Ye et al., ), Qwen-VL (Bai et al., ), InternLM-XComposer (Dong et al., ), and SPHINX (Lin et al., ; Gao et al., ) have been explored, leveraging CLIP (Radford et al., ) for image encoding and LLaMA (Touvron et al., ) for multi-modal instruction tuning, advancing MLLMs’ visual understanding and generalization.

Despite comprehensive benchmarks (Fu et al., ; Liu et al., ; Li et al., ; Xu et al., ) for general visual instruction-following scenarios, the specific potential of MLLMs for visual mathematical problem-solving remains under-explored. Prior studies like VQA (Antol et al., ; Goyal et al., ), VizWiz (Gurari et al., ), and ParsVQA-Caps (Mobasher et al., ) evaluate LMMs’ general visual question answering abilities on open-ended image queries. Additionally, works have assessed LMMs’ specific skills beyond natural scenes, such as abstract shapes (Antol et al., ; Lu et al., ; Ji et al., ), geometry diagrams (Seo et al., ; Lu et al., ; Chen et al., ; Cao & Xiao, ), charts (Methani et al., ; Masry et al., ; Kahou et al., ; Chang et al., ; Kafle et al., ), documents (Singh et al., ; Mathew et al., ; Liu et al., ), synthetic images (Dahlgren Lindström & Abraham, ; Li et al., ; Bitton-Guetta et al., ), external knowledge (Schwenk et al., ; Shah et al., ), commonsense reasoning (Zellers et al., ; Yin et al., ), scientific knowledge (Lu et al., ; Kembhavi et al., ; ), and medical understanding (Zhang et al., ; Lau et al., ).

Generative foundation models like GPT-3 (Brown et al., ), GPT-4 (OpenAI, ), Claude (Anthropic, ), LLaMA (Touvron et al., ), and LLaMA-Adapter (Zhang et al., ) can solve various downstream tasks (Wei et al., ) without task-specific fine-tuning. Prior work has evaluated their text-based abilities in QA, math, medicine, coding, and science (Bubeck et al., ; Nori et al., ; Chen et al., ; Fu et al., ; Sun et al., ; Wang et al., ; Huang et al., ; ; Liu et al., ; Zhang et al., ). Some work focused on specialized pretraining for improved visual math and chart reasoning, like PixStruct (Lee et al., ), MatCha (Liu et al., ), and UniChart (Masry et al., ). On the vision-language front, models like LLaVA (Liu et al., ), miniGPT4 (Zhu et al., ), InstructBLIP (Dai et al., ), Flamingo (Alayrac et al., ; Awadalla et al., ), LLaMA-Adapter V2 (Gao et al., ), and Multimodal Bard (Google, ) leverage paired (Schuhmann et al., ; Sharma et al., ; Lin et al., ) and interleaved (Zhu et al., ) image-text data. Additionally, specialized versions like LLaVAR (Zhang et al., ; Ye et al., ) emphasize document understanding and math comprehension. Recent works like Visit-Bench (Bitton et al., ), LVLM-eHub (Yu et al., ), MMBench (Liu et al., ; Xu et al., ; Shao et al., ) assess these models’ instruction-following and reasoning capabilities.

Large language models (LLMs) have demonstrated remarkable reasoning abilities, further enhanced by approaches like chain-of-thought (CoT) (Wei et al., ), program-of-thought (PoT) (Chen et al., ), and inductive reasoning (Wang et al., ; Tan & Motani, ). The feasibility of using LLMs to solve the Abstraction and Reasoning Corpus (ARC) challenge has been verified using zero-shot, few-shot, and context-grounded prompting (Tan & Motani, ).

OpenAI’s GPT-4V, the multimodal version of GPT-4, exhibits promising performance in vision-language reasoning. However, a fine-grained study of its strengths and limitations is still lacking. Recent work (Zhang et al., ) explores whether large multimodal models (LMMs) like GPT-4V execute vision and language tasks consistently or independently, contributing pioneering efforts in this field.

## Appendix C Data Collection Pipeline Details

#### Collection Pipeline:

To ensure high-quality samples, all data samples were manually collected as image snippets from publicly available websites.

We developed a flexible, highly automated data curation framework to streamline the process and standardize collection and annotation.
Continuous human reviews were conducted between steps in the pipeline to maintain quality and prevent error propagation.

- •
Step 1: A universally unique identifier (UUID) was generated for each question paper to track all curated questions. This step also updated a shared record containing details of the paper and the annotator’s alias, enabling efficient assignment of questions for peer review.
- •
Step 2: Annotators manually collected individual snippets of each question, along with contextual information relevant to multiple questions. For questions requiring additional context, snippets were labeled accordingly, and only legible, relevant questions (focused on Mental Ability or Scholastic Ability in mathematics) were included to maintain dataset integrity.
- •
Step 3: An image-merging script automatically identified and merged split question images or context snippets (based on the naming convention) using open-source image processing tools(^3^33https://opencv.org/). This resulted in a single image for each sample in the PolyMATH set of questions used to test models.
- •
Step 4: The next module in the pipeline created and automatically populated an annotation file, where each row corresponded to a collected sample. Columns included the paper_id (UUID from Step 1), question number, and image path.
- •
Step 5: Using an answer key or solution set, LLM-powered transcription extracted the ground truth answers for each question. Extracted answers were mapped to the corresponding annotation rows, followed by a manual check to ensure alignment with the provided solution and correctness.

## Appendix D Additional experiment details

#### Hyperparameters:

The following hyperparameters were used in our experiments:

**Table 8: Hyperparameters used in the experiments**
| Model | Hyperparameters |
| --- | --- |
| Gemini-1.5 Pro | temperature: 1, top_p: 0.95, top_k: 64,<br>max_output_tokens: 8192,<br>response_mime_type: text/plain |
| GPT-4o | top_p: 0.1, temperature: 1,<br>max_output_tokens: 4096, stream: False |
| Claude Family | top_p: 0.1, temperature: 1,<br>max_output_tokens: 4096, stream: False |
| Open Source Models | max_new_tokens: 3600, temperature: 0.7,<br>top_p: 0.3, num_beams: 1 |

Further, Table [9](#A4.T9) provides the source repositories and model cards for the various models used in our experiments.
Table [10](#A4.T10) shows the performance of open-source models across categories using two additional prompting strategies: $Chain\text{-}of\text{-}Thought$ and $Step\text{-}back$.
Table [11](#A4.T11) shows the total count of error analysis sample distribution that was conducted.

Table: Table 9: Models used to evaluated PolyMATH, along with their release dates and source repositories. We use both open-source and closed-source models for a comprehensive evaluation.

Table: Table 10: Results of open-source MLLMs on the testmini split of PolyMATH. We report model results using Chain-of-Thought, and Step Back prompting methods.

Table: Table 11: Type of errors made by Gemini-1.5 Pro, GPT4-o, and Claude-3.5 Sonnet over various question categories.

Table: Table 12: Per-category accuracy scores achieved by six human evaluators. The average human accuracy over all categories is 66.62%.

## Appendix E Prompts for dataset curation and experiments

The various prompts are detailed in this section. Table [13](#A5.T13) is the prompt used for the categorization of questions into various problem types. Table [14](#A5.T14) is the prompt used for generating the alternate image description of the question which is present as detailed in the additional metadata section §[3.3](#S3.SS3).
Table [15](#A5.T15), [16](#A5.T16), [17](#A5.T17) show cases the zero shot prompt, Chain of thought and Step back prompt for inference on PolyMATH respectively.
Table [18](#A5.T18) shows the answer extraction prompt from the MLLM response
Table [19](#A5.T19) shows the text based inference for Analysis [5](#S4.T5).

Table: Table 13: Prompt used for categorization of question of image.

Table: Table 14: Prompt used to generated detailed textual description of diagrams.

Table: Table 15: Prompt for zero shot inference

Table: Table 16: Prompt for Chain-of-Thought inference

Table: Table 17: Per-category and meta-prompts for Step Back prompt inference

Table: Table 18: Prompt to transcribe list of answer options from question image

Table: Table 19: Prompt for text-only inference.

## Appendix F Dataset Examples

Figures [5](#A6.F5), [6](#A6.F6), [7](#A6.F7), [8](#A6.F8), [9](#A6.F9), [10](#A6.F10), [11](#A6.F11), [12](#A6.F12), [13](#A6.F13),
[14](#A6.F14) demonstrate examples from each question category defined in Table [1](#S3.T1).

Figure: Figure 5: Questions belonging to the figure_completion (FC) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/1.png

Figure: Figure 6: Questions belonging to the logical_reasoning (LR) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/2.png

Figure: Figure 7: Questions belonging to the mathematical_reasoning (MR) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/3.png

Figure: Figure 8: Questions belonging to the numerical_reasoning (NR) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/4.png

Figure: Figure 9: Questions belonging to the odd_one_out (OD) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/5.png

Figure: Figure 10: Questions belonging to the pattern_recognition (PR) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/6.png

Figure: Figure 11: Questions belonging to the perspective_shift (PS) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/7.png

Figure: Figure 12: Questions belonging to the relative_reasoning (RR) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/8.png

Figure: Figure 13: Questions belonging to the sequence_completion (SC) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/9.png

Figure: Figure 14: Questions belonging to the spatial_reasoning (SR) category
Refer to caption: /html/2410.14702/assets/figs/dataset_categories/10.png

## Appendix G More details on Error Analysis

We leveraged 2 authors of this work to act as error evaluators independently and in parallel.
Each evaluator has a graduate degree in Computer Science and experience in similar puzzle-solving.
Owing to the clear and mutually-exclusive definitions of error types, there is little ambiguity in identifying the error type of the incorrect responses.
Our measure of inter-evaluator agreement is Cohen’s Kappa (K), found to be 0.9 - indicating near-unanimous agreement.
For questions where there was disagreement in evaluations, a consensus was reached after discussion.

## Appendix H Qualitative Error Analysis

This section presents examples of the qualitative error analysis that was carried out. Figures [5](#A6.F5), [6](#A6.F6),
[7](#A6.F7),
[8](#A6.F8),
[9](#A6.F9),
[10](#A6.F10),
[11](#A6.F11),
[12](#A6.F12),
[13](#A6.F13) and
[14](#A6.F14)
contains examples of failures by three proprietary models viz. Gemini-1.5 Pro, GPT-4o, and Claude-3.5 Sonnet across all categories.

Figure: Figure 15: Erroneous model reasoning patterns observed on an FC question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/1.png

Figure: Figure 16: Erroneous model reasoning patterns observed on an LR question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/2.png

Figure: Figure 17: Erroneous model reasoning patterns observed on an MR question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/3.png

Figure: Figure 18: Erroneous model reasoning patterns observed on an NR question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/4.png

Figure: Figure 19: Erroneous model reasoning patterns observed on an OD question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/5.png

Figure: Figure 20: Erroneous model reasoning patterns observed on a PR question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/6.png

Figure: Figure 21: Erroneous model reasoning patterns observed on a PS question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/7.png

Figure: Figure 22: Erroneous model reasoning patterns observed on an RR question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/8.png

Figure: Figure 23: Erroneous model reasoning patterns observed on an SC question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/9.png

Figure: Figure 24: Erroneous model reasoning patterns observed on an SR question
Refer to caption: /html/2410.14702/assets/figs/qualitative_analysis/10.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The paper notes a performance improvement when diagrams were replaced by textual descriptions, indicating that MLLMs may not be effectively processing visual information. Could the authors provide more qualitative examples or specific insights into how the textual descriptions differed from the diagrams and why these led to improved performance?
soundness: 2
strengths: S1: The authors provide a detailed categorization schema, covering ten reasoning types, which ensures a thorough assessment of MLLM capabilities in cognitive tasks.

S2: The authors implemented a rigorous evaluation using multiple inference strategies, including zero-shot, few-shot, Chain-of-Thought, and Step Back prompting.

S3: A fine-grained analysis highlights common pitfalls in model reasoning, such as logical flaws and spatial misunderstandings, providing insights into areas that need improvement.
summary: The paper introduces POLYMATH, a novel benchmark designed to evaluate the mathematical and visual reasoning capabilities of multi-modal large language models (MLLMs). The authors conducted extensive evaluations on 15 state-of-the-art MLLMs, analyzing their performance through multiple prompting strategies, such as zero-shot and Chain-of-Thought prompting. Results reveal significant gaps in MLLM performance compared to human baselines, particularly in spatial reasoning and diagram interpretation.
weaknesses: W1: While the paper presents error analysis for major models like Claude-3.5 Sonnet and GPT-4o, extending this analysis to more open-source models could yield deeper insights into shared weaknesses.

W2: The reported improvements when replacing diagrams with text descriptions raise questions about MLLM comprehension of visual information. A more in-depth discussion or qualitative examples would enhance understanding.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Table 5 and table 3's performance are not consistant, the reviewer understands they are on different subsets, but the big difference is not expected. Could the author explain why?
- For the second weakness above, could the author give hypothesis on why inference scaling doesn't help the model much? And what is expected to improve our model on PolyMath?
- Since the error analysis suggest logical flaw to be the main bottle neck of MLLM's performance (which is irrelavant to image information), and all data can be converted to textual format. What make it necessary for this dataset to be made multimodal?
soundness: 3
strengths: - The writing is clear and easy to follow.
- The proposed dataset is challenging, which is useful for future development of LLMs and MLLMs.
- The ayalysis showing that MLLMs rely more on text descriptions and the detailed error analysis are insightful.
summary: This paper introduce PolyMath, a multimodal math benchmark curated from India National Talent Search Examination.
PloyMath is carefully curated following 5 steps on collection, categorization and quality assurance.
Extensive experiemnts are conducted, showing that the best MLLM only achieve at most 41.9% performance on PloyMath.
Further analysis find that MLLM rely more on textual descriptions than visual information, and the logical flaw to be the most common error.
Study also finds that openai-o1 model performs closely with human baseline on textual only version of the dataset.
weaknesses: - The findings in the paper are not new. Firstly, for 'MLLMs rely more on text than image', MathVerse[1] specifically create 6 versions of their dataset with incremental reliance on visual information, showing the same trend.
- The error analysis suggest logical flaw to be the main bottle neck of MLLM's performance. But Table3 and 4 suggests that different inference stretagy does not help much, comparing with zero-shot. (only about 2%-6%) So it's unclear how we can improve our models on this dataset.
- Insufficient discussion on related works. Since PolyMath is scoped as abstract visual reasoning benchmark, the author should discuss how PloyMath is different from and better than existing benchmarks in this field, like Puzzle-VQA[2], Marvel[3].

[1] Zhang, R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., ... & Li, H. (2025). Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems?. In European Conference on Computer Vision (pp. 169-186). Springer, Cham.

[2] Chia, Y. K., Han, V. T. Y., Ghosal, D., Bing, L., & Poria, S. (2024). PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns. arXiv preprint arXiv:2403.13315.

[3] Jiang, Y., Zhang, J., Sun, K., Sourati, Z., Ahrabian, K., Ma, K., ... & Pujara, J. (2024). MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning. arXiv preprint arXiv:2404.13591.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Till the deadline of the review process, the link to the benchmark is not open-source. I hope the authors would organize the benchmark website properly during the rebuttal so that the benchmark’s composition and evaluation content could be inspected.
2. Experiment analysis is a little superficial. I hope the authors may provide some detailed analysis of the experimental results to yield more insightful conclusions.
3. The paper proposes many problems of current LLM models, but does not provide possible solutions. It will make this paper more meaningful if the authors would do some explorations on it.
soundness: 2
strengths: 1. A new multimodal mathematical reasoning benchmark, POLYMATH, is proposed, covering 10 different categories and contributing to the advancement of mathematics reasoning tasks.
2. The paper conducted a comprehensive and quantitative evaluation of 15 MLLMs using four different prompting strategies.
summary: This paper introduces POLYMATH, a multimodal mathematical reasoning benchmark designed to assess the performance of large multimodal language models (MLLMs) in terms of visual understanding and abstract reasoning abilities. The benchmark covers 10 different categories, including cognitive text and visual challenges such as pattern recognition, spatial reasoning, and relative reasoning. The researchers conducted a comprehensive and quantitative evaluation of 15 MLLMs using four different prompting strategies, including Chain-of-Thought and Step-Back.
weaknesses: 1. The paper does not provide open-source examples from the POLYMATH dataset. Without such examples, the unique challenges and value of the dataset are not clearly conveyed, weakening the paper’s effectiveness.
2. The analysis provided is superficial and covers mostly well-known conclusions without exploring the performance differences across models in different reasoning tasks or the underlying reasons. This shallow analysis fails to offer new insights or robust support for the conclusions, limiting the paper's contribution to the field.
3. Although the paper mentions the issue of large models’ mathematical reasoning capabilities, it lacks further exploration of this topic. The absence of an in-depth discussion or future directions weakens the paper's forward-looking impact and completeness.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Based on the examples in Figures 1 and 3, it appears that the format of the questions and answer choices varies. For instance, in the first question of Figure 1, choices are labeled with letters (A, B, C, D) at the bottom, while in the second question, choices are labeled with numbers (1, 2, 3, 4) on the right side. I believe reorganizing all questions in a consistent format could improve the validity of the evaluation.
2. The dataset is unevenly distributed, with three out of ten patterns accounting for more than 40% of the baseline data, while some patterns represent only 3% of the data. Could the imbalanced data distribution lead to misleading conclusions in Section 4.2?
3. In Lines 462 to 466, simply comparing test-img and testmini makes the argument that "decreased accuracy on test-img is largely due to the presence of diagram-based problems" problematic. It's possible that test-img has a different distribution compared to testmini. For example, the questions in test-img may be more challenging, prompting designers to include diagrams for better understanding.
4. The authors adopt four prompting strategies for evaluating the MLLMs but only mention performance differences in one sentence (Lines 418-421). Could more insights or explanations be provided regarding these four strategies?
5. As the paper presents the complete PolyMATH dataset with 5,000 images, it seems unusual to report results on only 20% of them. Could the authors also provide results for the entire dataset? This would be beneficial for comparing new MLLMs more comprehensively in future research.


Overall, I appreicate the effort author and team paid on construct such a comprehensive datasets. I am willing to adjust my score if questions metioned previously can be solved and clarified.
soundness: 2
strengths: 1. A extensive mannually created benchmark: PolyMATH consists 5k high quality images covering 10 patterns, allowing for a more thorough assessment of MLLMs' abilities across various abstract reasoning concepts. The benchmark can be further splited into mini subset for quick evaluation and diagram-only subset for visual-centric abstract evaluation.
2. A comprehensive evaluation: The author conduct a comprehensive evaluation over 15 MLLMs with four prompting approaches, demonstrating current MLLMs visual abstraction capabilities. Further ablation study provide insight on weakness of MLLMs (diagram understanding)
3. The paper already release the dataset for public use.
summary: The paper present PolyMath, a benchmark evaluating visual abstraction reasoning consists of 5000 mannually collected images acrosss 10 categories. The author also extract testmini (1k) and test-image (with diagram) subset of the benchmark for diverse usage. A comprehensive experiment on testmini evaluating 15 MLLMs with four diverse prompting strategies. The overall results demonstrate a gap between current MLLMs and humans. A fine-grained error analysis reveals model's weak capability on spatial relaition and high-level reasoning. Further ablation study on providing aditional textual description shows that the main difficulty may be comprehensing visual diagrams.  In the end, the author also evaluated the newly introduced GPT-o1 and find that its performance align with human baseline. The author release their benchmark for public usage.
weaknesses: 1. Benchmark Positioning:  The comparison between previous VQA or read-world dataset is not enough. I think there is a bunch of benchmarks related with visual abstraction reasoning (https://arxiv.org/abs/2202.10284). Is PolyMATH different than previous dataset or If it is just a combination of previous one. Can the evaluation on PolyMATH bring new insight compared to previous ones?
2. Pattern Design:  The paper mentions the cognitive reasoning and abstract reasoning. But it seems lack a sufficient elaboration on the why do we need to evaulate these patterns on MLLMs. It will be helpful to show practical application of these pattern in real world scenraio or relavent research from human coginition literature.
3. Value Justification: From the ablation study on diagram-only subset, the result showing the major issue blocking models abstract visual reasoning ability is the diagream comprehension. Can we understand the model have difficulties on percepting diagrams? If so, it may make the evaluation less insightful if the major bottleneck is in model's perception abilities instead of abstract reasoning ability. An experiment showing how much detail and vision information MLLMs can percept from diagream will be helpful.

Meta-review: This paper presents POLYMATH, a benchmark designed to assess the mathematical and visual reasoning capabilities of MLLMs. Reviewers commend the challenging nature of the proposed data and the comprehensive quantitative evaluation conducted on 15 MLLMs with four distinct prompting strategies. The error analysis, revealing that MLLMs rely more on text descriptions than diagrams, is also noted as interesting. However, the main concerns are as follows: 1. The provided analysis lacks novelty, predominantly covering well-established conclusions and thus failing to present novel insights. 2. The experimental analysis is cursory, lacking profound conclusions and omitting in-depth discussions regarding future directions. 3. The paper identifies numerous problems with current LLM models but fails to offer potential solutions. 4. The dataset is considerably imbalanced, which may lead to potentially misleading conclusions. The AC concurs that this benchmarking work could be enhanced by further resolving these issues.

Decision: Reject