ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection

## Abstract

Abstract As the field of Multimodal Large Language Models (MLLMs) continues to evolve, their potential to revolutionize artificial intelligence is particularly promising, especially in addressing mathematical reasoning tasks. Current mathematical benchmarks predominantly focus on evaluating MLLMs’ problem-solving ability, yet there is a crucial gap in addressing more complex scenarios such as error detection, for enhancing reasoning capability in complicated settings. To fill this gap, we formally formulate the new task — multimodal error detection , and introduce ErrorRadar , the first benchmark designed to assess MLLMs’ capabilities in such a task. ErrorRadar evaluates two sub-tasks: error step identification and error categorization , providing a comprehensive framework for evaluating MLLMs’ complex mathematical reasoning ability. It consists of 2,500 high-quality multimodal K-12 mathematical problems, collected from real-world student interactions in an educational organization, with rigorous annotation and rich metadata such as problem type and error category. Through extensive experiments, we evaluated both open-source and closed-source representative MLLMs, benchmarking their performance against educational expert evaluators. Results indicate significant challenges still remain, as GPT-4o with best performance is still around 10% behind human evaluation. The dataset will be available upon acceptance.

## 1 Introduction

On the path to Artificial General Intelligence, Large Language Models (LLMs) such as GPT-4 (OpenAI, ) have emerged as a central focus in both industry and academia (Minaee et al., ; Zhao et al., ; Zhu et al., ). As the real world is inherently multimodal, the evolution of Multimodal Large Language Models (MLLMs) such as the latest GPT-4o (OpenAI, ) and Gemini 1.5 (Reid et al., ), has become a rapidly growing area of interest, demonstrating remarkable effectiveness in diverse applications such as E-commerce (Jin et al., ; Xu et al., ; ), medicine (Xiao et al., ; He et al., ), urban planning (Zhou et al., ; Yan et al., ; Hao et al., ), etc. In particular, multimodal reasoning stands to significantly benefit education scenarios from the robust capabilities of MLLMs (Wang et al., ; Li et al., ), given its reliance on multimodal inputs to comprehensively grasp users’ intentions and needs.

Within the multimodal sphere, mathematical scenarios pose a significant challenge, demanding sophisticated reasoning abilities from MLLMs (Lu et al., ; Ahn et al., ). These scenarios have attracted considerable research aimed at pushing the boundaries of MLLMs’ reasoning capabilities (Hu et al., ; Jia et al., ; Lu et al., ; Shi et al., ; Zhuang et al., ). Besides, various representative benchmarks have been designed to measure MLLMs’ performance in complex mathematical reasoning tasks, which involve multi-step reasoning and quantitative analysis within visual contexts (Lu et al., ; Zhang et al., ; Qiao et al., ; Peng et al., ).

Figure: Figure 1: Comparison of research scope between previous work and our proposed ErrorRadar benchmark on mathematical reasoning tasks.
Refer to caption: extracted/5909434/figures/comparison.png

Scrutinizing the off-the-shelf mathematical reasoning benchmarks, there is a predominant focus on evaluating the problem-solving capabilities of MLLMs, prioritizing the accuracy with which MLLMs can solve mathematical problems (Wang et al., ; Lu et al., ; Zhang et al., ), as depicted in Figure 1 (a). However, in educational contexts, it is even more crucial to consider user-oriented needs, such as error detection. As indicated in Figure 1 (b), this involves not only pinpointing the first incorrect step in a student’s step-by-step solution but also categorizing the types of errors made, which is a multifaceted process that requires a deep understanding of both mathematical concepts and cognitive processes (Davies et al., ; Rabillas et al., ).

Towards this end, addressing the aforementioned research gap, we aim to formulate the new task of evaluating MLLMs in the context of error detection scenarios, and therefore introduce the corresponding benchmark termed ErrorRadar. We have designed two sub-tasks to comprehensively assess the performance: error step identification and error categorization. To construct a rich and reliable dataset, we initially sourced a collection of multimodal K-12 level math problems from an educational organization and subsequently refined the dataset through rigorous manual annotation to ensure quality. In particular, we also collect real students’ answers for each multimodal question for a relatively robust experimental setting, compared to other relevant benchmarks (as shown in Table 1). Furthermore, we categorized the dataset to better align with diverse needs as follows: Problem types: plane geometry, solid geometry, diagram, algebra, and mathematical common sense; and Error categories: visual perception errors, calculation errors, reasoning errors, knowledge errors, and misinterpretation of the problem. In summary, the ErrorRadar comprises 2,500 high-quality instances derived from real-life problem-solving data, providing a foundational dataset to enhance the complex reasoning capabilities of MLLMs for the research community and industry.

For ErrorRadar, we carry out an extensive experimental analysis to determine the proficiency in complex mathematical reasoning of various MLLMs. The evaluation encompasses both the latest open-source MLLMs (e.g., InternVL2 (Chen et al., ), LLaVA-NEXT (Liu et al., ), CogVLM2 (Wang et al., )), and closed-source MLLMs (e.g., GPT4-o (OpenAI, ), Gemini Pro 1.5 (Reid et al., ), Claude 3.5 (Anthropic, )). Our focus was on their error detection capabilities, specifically the identification of the erroneous step and the classification of the error type. To establish a comparative human performance standard, we involved expert human educators who possess a graduate-level degree or higher qualifications.
The results demonstrate that ErrorRadar poses a significant challenge with human evaluation achieving less than 70%.

From in-depth evaluation of representative MLLMs, we obtain the following insightful findings: ❶ Closed-source MLLMs, particularly GPT-4o, consistently outperform open-source MLLMs in both sub-tasks, and show more balanced accuracy across different error categories; ❷ Weaker MLLMs exhibit an over-reliance on simpler categories, while stronger models handle complex tasks better; ❸ Both MLLMs and humans perform better on error step identification compared to error categorization, as localizing specific errors is inherently simpler than categorizing errors.

Our contributions can be summarized as follows:

- ❶
We take the first step to formulate the multimodal error detection task, and introduce a multimodal benchmark termed ErrorRadar for evaluation. This benchmark serves as a standard operator for assessing the complex mathematical reasoning capabilities of the latest MLLMs.
- ❷
We meticulously curate an extensive dataset comprising approximately 2,500 high-quality instances with rigorous annotation and rich metadata derived from real user interactions in an educational organization. To the best of our knowledge, this is the first attempt to use real-world student problem-solving data to evaluate MLLMs, providing a protocol for future research on MLLMs’ complex mathematical reasoning.
- ❸
Our comprehensive experimental evaluation of more than 20 MLLMs, both proprietary and open-source, highlight the substantial room for improvement (i.e., 7%-15% in accuracy) in the complex mathematical reasoning capabilities, underscoring the necessity for further research.

## 2 The ErrorRadar Dataset

### 2.1 Task Formulation

Figure: Figure 2: Example of our well-annotated multimodal mathematical reasoning dataset ErrorRadar, and performance comparison on error categorization and error step localization tasks among representative MLLMs. It is evident that even simple math problems can be mishandled by the currently superior MLLMs in one or both tasks, highlighting the challenging nature of our proposed multimodal error detection setting.
Refer to caption: extracted/5909434/figures/task_definition.png

Basic Setting. In this task, we aim to assess the model’s capability to detect errors in mathematical problem-solving processes across multiple samples. Let $N$ represent the total number of samples in the evaluation set. For each sample indexed by $i\in\{1,2,\dots,N\}$, the input set $\mathcal{I}_{i}$ is represented as:

$$ $\mathcal{I}_{i}=\{T_{\text{text},i},T_{\text{image},i},A_{\text{correct},i},A_ {\text{incorrect},i},\{S_{k,i}\}_{k=1}^{n_{i}}\},$ $$

where:

- •
$T_{\text{text},i}$ ➜ the textual statement of the $i$-th problem.
- •
$T_{\text{image},i}$ ➜ the image representation of the $i$-th problem.
- •
$A_{\text{correct},i}$ ➜ the correct solution for the $i$-th problem.
- •
$A_{\text{incorrect},i}$ ➜ the incorrect student solution for the $i$-th problem.
- •
$\{S_{k,i}\}_{k=1}^{n_{i}}$ ➜ the sequence of $n_{i}$ steps taken by the student in the $i$-th problem, with each $S_{k,i}$ being a distinct step. Here, $k\in\{1,2,\dots,n_{i}\}$ represents the index of the student’s steps.

The task is to define a model $\mathcal{M}$ that maps the input space to an output space for each sample, where the output set $\mathcal{O}_{i}$ for the $i$-th sample is:

$$ $\mathcal{O}_{i}=\{Step_{x,i},C_{\text{error},i}\},$ $$

where:

- •
$Step_{x,i}$ ➜ the index of the first incorrect step $S_{x,i}$ in the sequence $\{S_{k,i}\}_{k=1}^{n_{i}}$ for $i$-th problem.
- •
$C_{\text{error},i}$ ➜ the category of error for the $i$-th problem, where $C_{\text{error},i}\in\{\text{VIS},\text{CAL},\text{REAS},\text{KNOW},\text{MIS}\}$.

Thus, the model’s task is to learn a function $\mathcal{M}$ that maps the input $\mathcal{I}_{i}$ to the output $\mathcal{O}_{i}$ for all samples $i\in\{1,2,\dots,N\}$:

$$ $\mathcal{M}:\mathcal{I}_{i}\rightarrow\mathcal{O}_{i}\quad\forall i\in\{1,2, \dots,N\}.$ $$

More concrete examples can be seen in Figure 2 and Appendix A. The discrepancies within the five error categories are delineated as follows:

- ✯
Visual Perception Errors (VIS): These errors arise when there is a failure to accurately interpret the information contained within images or diagrams presented in the question due to visual issues.
- ✯
Calculation Error (CAL): These errors manifest during the calculation process, which may include arithmetic mistakes such as incorrect addition, subtraction, multiplication, or division, errors in unit conversion, or mistakes in the numerical signs between multiple steps.
- ✯
Reasoning Error (REAS): These errors occur during the problem-solving process when improper reasoning is applied, leading to incorrect application of logical relationships or conclusions.
In
- ✯
Knowledge Error (KNOW): These errors result from incomplete or incorrect understanding of the knowledge base, leading to mistakes when applying relevant knowledge points.
- ✯
Misinterpretation of the Question (MIS): These errors occur when there is a failure to correctly understand the requirements of the question or a misinterpretation of the question’s intent, leading to responses that are irrelevant to the question’s demands. For instance, if the question asks for a letter and a number is provided, or vice versa.

Error Step Identification. The function for identifying the first incorrect step, $f_{\text{step},i}$, is a mapping from the sequence of steps $\{S_{k,i}\}_{k=1}^{n_{i}}$ to the index of the first error $x_{i}$ in the sequence, formulated as:

$$ $f_{\text{step},i}:\{S_{k,i}\}_{k=1}^{n_{i}}\rightarrow Step_{x,i}\quad\text{ where}\quad x_{i}=\arg\min_{k}\{S_{k,i}:S_{k,i}\ \text{is incorrect}\}.$ $$

However, this function can be further decomposed into a stepwise transition function $\psi_{k,i}$, which governs the evolution of the solution process. Let $\Psi_{i}$ represent the sequence of transitions for the $i$-th sample:

$$ $\Psi_{i}=\{\psi_{1,i},\psi_{2,i},\dots,\psi_{n_{i},i}\},$ $$

where each $\psi_{k,i}$ maps the current step $S_{k,i}$ to the next step $S_{k+1,i}$ as $\psi_{k,i}:S_{k,i}\rightarrow S_{k+1,i}$
Thus, the first erroneous step can be defined as the first $k$ where $\psi_{k,i}(S_{k,i})$ leads to an incorrect $S_{k+1,i}$, i.e., the point where $\psi_{k,i}(S_{k,i})\neq\psi_{k,i}^{\text{correct}}(S_{k,i})$.

Error Categorization. The error categorization function $f_{\text{error},i}$ is a mapping from the incorrect answer $A_{\text{incorrect},i}$ to the error category $C_{\text{error},i}$, but this mapping can also depend on intermediate contextual factors such as the specific step where the error occurred. Let $\Theta_{i}$ represent the error-generating context, defined as the tuple of $(S_{x,i},A_{\text{incorrect},i})$. We now define the error categorization function as:

$$ $f_{\text{error},i}:\Theta_{i}\rightarrow C_{\text{error},i}\quad\text{where} \quad C_{\text{error},i}\in\{\text{VIS},\text{CAL},\text{REAS},\text{KNOW}, \text{MIS}\}.$ $$

This function can be expressed as a composite function:

$$ $f_{\text{error},i}=g(A_{\text{incorrect},i},S_{x,i},\mathcal{F}(T_{\text{text} ,i},T_{\text{image},i})),$ $$

where $g$ is a higher-order function that takes into account not only the incorrect answer $A_{\text{incorrect},i}$ and the erroneous step $S_{x,i}$ but also contextual features $\mathcal{F}(T_{\text{text},i},T_{\text{image},i})$, derived from both the textual and image-based problem representations.

Performance Metric. The model’s performance is evaluated using accuracy metrics for both the first erroneous step identification and the error categorization tasks. Let $G_{\text{step},i}$ and $G_{\text{error},i}$ be the ground truth labels for the first erroneous step and the error category for the $i$-th sample, respectively. The accuracy for identifying the first erroneous step is defined as:

$$ $\text{Acc}_{\text{step}}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}(Step_{x,i}=G_{ \text{step},i}),$ $$

where $\mathbb{I}(\cdot)$ is the indicator function, which returns 1 if the predicted step matches the ground truth, and 0 otherwise.
Similarly, the accuracy for error categorization is:

$$ $\text{Acc}_{\text{cate}}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}(C_{\text{error},i }=G_{\text{error},i}).$ $$

### 2.2 Data Source & Annotation

Following the roadmap shown in Figure 4, this section includes how we collect and annotate ErrorRadar dataset to ensure the overall data quality. Different from the conventional benchmarks that rely on public datasets or modified textbook collections (Lu et al., ; Zhou et al., ), ErrorRadar dataset is uniquely sourced from the question bank of a global educational organization. This repository encompasses a vast array of mathematical problems in K-12 levels, totaling over a million entries. Initially, we curated approximately 180,000 math problems that feature a single image-based question stem, aligning with our goal to target a multimodal assessment setup.

Subsequently, we refined our selection by evaluating the universality and articulation of the problem content. For each problem, we identified multiple incorrect answers. To ensure the dataset’s relevance for error detection tasks, we selected the most frequently given incorrect answer as the student’s response. Additionally, we scrutinized cases where the most common incorrect answer was due to system input errors despite the answer being correct. In such instances, we amended the dataset by incorporating the next most frequently incorrect answer.

Furthermore, since error detection tasks necessitate a step-by-step reasoning process, we enriched our dataset with new content through manual annotation. Specifically, we provided professional annotators with the original multimodal QA data, student’s incorrect answers, and the pedagogical team’s analysis of correct answer process. Based on this comprehensive initial data, annotators delineated the erroneous steps leading to the incorrect answers (More details in Appendix B.1).

### 2.3 Dataset Details

As illustrated in Table 2, ErrorRadar dataset comprises a substantial collection of 2,500 multimodal math questions designed for error detection tasks. It predominantly includes plane geometry problems, with solid geometry, diagram, algebra, and math commonsense questions making up the remainder, highlighting its focus on diverse mathematical problems. it also categorizes errors into visual perception, calculation, reasoning, knowledge, and question misinterpretation. Key statistics indicate a diverse dataset with an average reasoning step of 7.6, a variety of question lengths, and a wide range of reasoning steps (up to 38). Detailed distribution of ErrorRadar and definition of problem type can be seen in Figure 4 and Appendix B.2, respectively.

Figure: Table 2: Key statistics of ErrorRadar.
Refer to caption: extracted/5909434/figures/dataset_roadmap.png

## 3 Experiments and Analysis

### 3.1 Evaluation Protocols

In ErrorRadar benchmark, we propose an evaluation strategy using template matching rules. The evaluation process consists of three stages: response generation, answer extraction, and performance calculation. Initially, the MLLMs generate responses given the inputs, which incorporates the multimodal mathematical question, wrong answer, and its step-by-step reasoning, using the template from Appendix C.2. Subsequently, the short answer text can be extracted from the detailed response. Finally, the model performance is based on the detailed score calculation as shown in Section 2.1. The final score will be calculated by averaging the scores from three rounds of assessment.

### 3.2 Experimental Setup

In our experimental setup, we meticulously categorized and evaluated a diverse array of MLLMs into three distinct groups to assess their capabilities across error detection tasks. (i) The Open-Source MLLMs category encompassed models such as InternVL-2 (Chen et al., ), Phi-3-vision (Abdin et al., ), Yi-VL (Young et al., ), DeepSeek-VL (Lu et al., ), LLaVA-v1.6-Vicuna (Liu et al., ), MiniCPM-LLaMA3-V2.5 (Yao et al., ), MiniCPM-V2.6 (Yao et al., ), Qwen-VL (Bai et al., ), GLM-4v (GLM et al., ), and LLaVA-NEXT (Liu et al., ), each demonstrating their unique strengths and capabilities in handling different types of errors. (ii) The Closed-Source MLLMs featured proprietary models like Qwen-VL-Max (Bai et al., ), Claude-3-Haiku (Anthropic, ), Claude-3.5-Sonnet (Anthropic, ), Gemini-Pro-1.5 (Reid et al., ), GPT-4o-mini (OpenAI, ), and GPT-4o (OpenAI, ), providing a comparison point for the performance of models that are not publicly accessible. (iii) Lastly, the Human Performance category served as a benchmark for natural intelligence, allowing us to gauge how closely MLLMs can emulate human cognitive functions across tasks such as visual perception (More details in Appendix C.1). We provide the prompts for MLLMs and sources of MLLMs in Appendix C.2 and C.3, respectively.

### 3.3 Experimental Results

**Table 3: Comparison of open-source and closed-source MLLM performance (accuracy in percentage) across error detection tasks. We denote STEP and CATE for the performance of error step identification task (i.e., $Acc_{step}$) and error categorization task (i.e., $Acc_{cate}$), respectively. We also denote VIS, CAL, REAS, KNOW, and MIS for the performance of visual perception error, calculation error, reasoning error, knowledge error, and misinterpretation of the question. The highest and second highest scores among MLLMs in each column are highlighted in red and blue, respectively. Note that the grey cell in CAL column indicates the exceptional value, as more than 70% categories predicted by the MLLM are CAL (More analysis on Sec 3.3.1 Obs.2).**
| Multimodal Large Language Models | Parameters | LLM | STEP | CATE | VIS | CAL | REAS | KNOW | MIS |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Open-Source MLLMs |  |  |  |  |  |  |  |  |  |
| InternVL2 (Chen et al., ) | 2B | InternLM-2 | 9.8 | 25.1 | 32.2 | 38.8 | 12.2 | 0.0 | 24.4 |
| Phi-3-vision (Abdin et al., ) | 4B | Phi-3 | 37.5 | 40.7 | 9.6 | 99.6 | 6.6 | 3.4 | 4.1 |
| Yi-VL (Young et al., ) | 6B | Yi | 15.7 | 32.1 | 9.1 | 77.1 | 4.9 | 14.3 | 0.0 |
| DeepSeek-VL (Lu et al., ) | 7B | DeepSeek | 16.2 | 35.7 | 4.6 | 90.9 | 0.4 | 28.6 | 6.5 |
| LLaVA-v1.6-Vicuna (Liu et al., ) | 7B | Vicuna-v1.5 | 30.3 | 17.7 | 40.3 | 14.9 | 8.3 | 0.0 | 55.3 |
| InternVL-2 (Chen et al., ) | 8B | InternLM-2.5 | 44.2 | 44.1 | 12.4 | 99.6 | 13.6 | 10.9 | 2.4 |
| MiniCPM-LLaMA3-V2.5 (Yao et al., ) | 8B | LLaMA3 | 37.4 | 38.0 | 4.1 | 100.0 | 2.1 | 2.5 | 0.0 |
| MiniCPM-V2.6 (Yao et al., ) | 8B | Qwen2 | 17.0 | 39.8 | 11.4 | 87.8 | 12.1 | 10.1 | 17.9 |
| Qwen-VL (Bai et al., ) | 9B | Qwen | 23.8 | 38.9 | 8.6 | 99.1 | 3.5 | 0.0 | 0.8 |
| GLM-4v (GLM et al., ) | 13B | GLM-4 | 44.6 | 44.1 | 2.5 | 92.9 | 25.8 | 0.0 | 0.0 |
| LLaVA-v1.6-Vicuna (Liu et al., ) | 13B | Vicuna-v1.5 | 36.9 | 47.8 | 0.0 | 74.5 | 53.7 | 0.8 | 2.4 |
| CogVLM2-LLaMA3 (Wang et al., ) | 19B | LLaMA3 | 15.0 | 20.1 | 43.3 | 33.8 | 0.7 | 13.4 | 0.0 |
| InternVL2 (Chen et al., ) | 26B | InternLM-2 | 50.4 | 51.2 | 39.2 | 84.6 | 35.6 | 0.8 | 10.6 |
| LLaVA-NEXT (Liu et al., ) | 72B | Qwen1.5 | 51.8 | 45.0 | 7.1 | 86.0 | 32.0 | 7.6 | 0.8 |
| InternVL2 (Chen et al., ) | 76B | Hermes-2 Theta | 54.4 | 49.5 | 33.4 | 92.4 | 25.1 | 10.9 | 8.1 |
| Closed-Source MLLMs |  |  |  |  |  |  |  |  |  |
| Qwen-VL-Max (Bai et al., ) | - | - | 48.7 | 52.9 | 15.2 | 78.9 | 50.5 | 14.3 | 36.6 |
| Claude-3-Haiku (Anthropic, ) | - | - | 45.6 | 48.0 | 10.4 | 77.4 | 46.8 | 4.2 | 1.6 |
| Claude-3.5-Sonnet (Anthropic, ) | - | - | 50.2 | 49.5 | 35.7 | 48.4 | 64.8 | 21.0 | 11.4 |
| Gemini-Pro-1.5 (Reid et al., ) | - | - | 55.0 | 52.7 | 43.5 | 55.7 | 63.1 | 18.5 | 13.0 |
| GPT-4o-mini (OpenAI, ) | - | - | 52.0 | 44.5 | 9.1 | 46.8 | 62.7 | 31.9 | 13.0 |
| GPT-4o (OpenAI, ) | - | - | 55.1 | 53.1 | 46.3 | 50.4 | 64.9 | 9.2 | 46.3 |
| Human |  |  |  |  |  |  |  |  |  |
| Human performance | - | - | 69.8 | 60.7 | 66.8 | 75.9 | 47.6 | 35.3 | 53.7 |

#### 3.3.1 Main Results

Finding #1: Closed-source MLLMs generally outperform open-source MLLMs in both error detection tasks, with GPT-4o demonstrating the strongest performance. Table 3 shows that closed-source MLLMs generally outperform open-source MLLMs in both STEP and CATE tasks, and they also exhibit relatively more balanced performance across the five error categories. This superiority can likely be attributed to the proprietary datasets and advanced training resources available to closed-source models, which allow for more robust fine-tuning (Shi et al., ; Yu et al., ; Wang et al., ). Notably, GPT-4o stands out as the best model, achieving the highest scores not only in STEP and CATE tasks, but also in the VIS, REAS, and MIS categories, demonstrating its overall versatility and strength. Gemini-Pro-1.5 ranks second in the STEP task, while Qwen-VL-Max takes second place in the CATE task. Among open-source MLLMs, LLaVA-NEXT-72B and InternVL2-76B achieve the two best results, although their performance still lags behind GPT-4o.

Figure: Figure 5: The distribution of CAL and non-CAL of closed-source and open-source MLLMs with top-3 CAL ACC.
Refer to caption: extracted/5909434/figures/CAL_bias.png

Finding #2: Weak open-source MLLMs tend to predict CAL category, leading to unusually high performance. Table 3 indicates that MLLMs with relatively low performance in the CATE task tend to exhibit unusually high performance in CAL category. Specifically, open-source models like MiniCPM-LLaMA3-v2.5 even achieve a 100% accuracy in CAL, while Phi-3-vision and InternVL-2-8B reach 99.6%. Upon analyzing the category prediction distribution of CAL and non-CAL from Figure 5 (See details of all MLLMs in Appendix C.4), it becomes clear that open-source MLLMs with the top three CAL accuracy predict over 90% of instances as CAL category, suggesting an over-reliance on this category. In contrast, closed-source MLLMs with top three CAL accuracy do not exhibit this extreme trend of prediction bias. This phenomenon likely arises from weaker MLLMs attempting to overfit on CAL category, a relatively simpler classification, to compensate for their inability to handle more complex scenarios (Tirumala et al., ; Xu et al., ).

Finding #3: MLLMs with strong overall performance tend to handle STEP easier than CATE. From Table 3, the best open-source MLLMs, such as InternVL2-76B, and the best closed-source MLLMs, like GPT-4o, exhibit a tendency where their STEP performance surpasses that of CATE. This trend holds even for human performance, where accuracy on STEP is higher (69.8%) compared to CATE (60.7%). The reason for this disparity is likely that identifying the error step is inherently easier, as it involves localizing a specific point of failure. On the other hand, categorizing the error requires more complex reasoning and contextual understanding to classify the nature of the error, which adds difficulty. This mirrors the challenges in object detection, where identifying the object is often simpler than fully understanding its attributes or behavior (Zou et al., ; Jiao et al., ).

Finding #4: CAL is the easiest category for MLLMs, while KNOW is the most difficult. CAL is the category with the highest performance among most MLLMs (excluding those with exceptional values), which could be attributed to the structured and deterministic nature of calculations, where errors often result in clear, quantifiable deviations from expected outcomes, making them more straightforward to detect (Lewkowycz et al., ; Kojima et al., ). Conversely, KNOW stands out as the most challenging category, suggesting that MLLMs struggle significantly with tasks requiring deep factual understanding and contextual reasoning. The complexity of knowledge errors likely stems from the need for comprehensive domain expertise, which current MLLMs may not fully encapsulate yet. Even human performance reflects this trend, with knowledge error scoring notably lower than other categories, albeit with higher accuracy than MLLMs, highlighting the inherent difficulty of this task for both humans and AI (Kandpal et al., ; Feng et al., ).

Finding #5: MLLMs still have a gap to close to reach human-level intelligence in error detection. Human performance significantly outperforms the best MLLMs in both the STEP and CATE tasks, with accuracy scores of 69.8% and 60.7% respectively, compared to the highest MLLM scores of 55.1% and 53.1%. Notably, the detection of VIS by humans is markedly superior to the best MLLMs, with a difference of nearly 20%. This substantial lead may be attributed to the sophisticated pattern recognition inherent to human visual processing (Doerig et al., ), which MLLMs, despite their advancements, have yet to fully emulate. Besides, it is interesting to note that human performance in REAS detection is lower than all closed-source MLLMs but higher than almost all open-source MLLMs. This may suggest that closed-source MLLMs benefit from proprietary datasets and algorithms that better capture the nuances of logical reasoning (Wang et al., ).

#### 3.3.2 Visual Perception Analysis

Figure: Figure 6: The error category distribution of misjudged VIS cases of GPT-4o.
Refer to caption: extracted/5909434/figures/gpt4o_bad_case_distribution.png

Finding #1: Closed-source MLLMs are most likely to misjudge VIS as REAS in error categorization task. Taking the best-performing GPT-4o model as an example, as shown in the Figure 8, 48% of VIS are misclassified as REAS, followed by 30% being misjudged as MIS. In multimodal mathematical scenarios, where the MLLM needs to handle information involving both visual and linguistic elements simultaneously, particularly in problems related to plane and solid geometry, the complexity of the diagrams makes it difficult for the model to accurately extract certain features, leading to the frequent misclassification of VIS as REAS. For instance, if an erroneous response to a mathematical query originates from VIS (e.g., misinterpreting a diagram), MLLM may mistakenly attribute this to a flaw in logical reasoning that occurs subsequent to initial visual misinterpretation.

Finding #2: Open-source MLLMs are more likely to misclassify VIS as CAL. Taking the open-source model CogVLM2-LLaMA3, which performs best in identifying VIS, as an example, CAL accounts for 64% of misclassified category, as illustrated in the Figure 8. When handling complex visual information, especially in geometry problems, the MLLM often struggles to accurately extract key features. Due to the open-source MLLM’s weaker multimodal integration capabilities, it simplifies visual issues into numerical calculation problems. The lack of sufficient training and data for visual-related errors is also a key reason behind this phenomenon (Wichmann & Geirhos, ).

#### 3.3.3 Relation between Error Category and Error Step

Finding #1: There is a close relationship between different error category and their distribution in the reasoning steps. As shown in Figure 8, VIS tends to occur in the earlier to mid-stages, accounting for a median proportion of 0.5 of total steps. In contrast, MIS, REAS, CAL, and KNOW are more likely to arise in the later stages, with their median proportions ranging from 0.7 to 0.9.

Finding #2: VIS occurs in the earlier stages of problem-solving reasoning. This finding could be closely linked to the sequence in which students approach the task (Binz & Schulz, ; Kennedy & Romig, ). Since image content often serves as a key reference early on, any misinterpretation of this visual information directly impacts the subsequent problem-solving steps. Students typically first examine the image, and then integrate the information before proceeding to reasoning or calculation. As a result, visual perception errors arise earlier compared to other types of errors.

Finding #3: Other error categories are primarily in later stages of problem-solving reasoning. This may be linked to the increasing cognitive load students encounter during problem-solving. Cognitive Load Theory posits that information complexity ranges from low to high interactivity (Paas et al., ; Binz & Schulz, ). While low-interactivity information can be understood independently, high-interactivity information requires simultaneous processing of related elements, thus increasing cognitive load (Kennedy & Romig, ; Abbad-Andaloussi et al., ). In later stages, students must integrate complex information from multiple sources. For instance, calculating the distance between two points needs increasing interactivity heightens cognitive load, leading to errors like forgetting to take the square root or miscalculating differences. Consequently, as cognitive load rises, the frequency of errors in later steps also increases.

#### 3.3.4 Scaling Analysis

Figure: Figure 9: The accuracy of STEP and CATE of two representative MLLM series: LLaVA-NEXT and InternVL2. We denote Tiny, Small, Middle, Large as the 2B, 8B, 26B, 76B for InternVL2 and None, 7B, 13B, 72B for LLaVA-NEXT, respectively.
Refer to caption: extracted/5909434/figures/scale_acc.png

Finding #1: The performance of MLLMs on STEP task increases with the scale of parameters. We observe a phenomenon similar to the scaling law (Kaplan et al., ) in our experiments. As shown in Figure 9, when the size of the InternVL2 model increases from Tiny to Huge, the accuracy of STEP task rises from 9.8% to 54.4%, showing an improvement of 44.6%. Similarly, as the size of LLaVA-NEXT increases from Small to Large, its accuracy of STEP also improves from 30.3% to 51.8%, demonstrating that larger MLLMs exhibit greater reasoning ability in localizing erroneous steps.

Finding #2: CATE task is relatively more difficult to improve through scaling. While the accuracy of CATE shows a trend of improvement for both the InternVL2 and LLaVA-NEXT models as their size increases from Tiny (Small) to Middle, a slight decrease is also observed when the model size reaches Large. We presume that this is because CATE is a more challenging task compared to STEP, and merely increasing the model size without fine-tuning is insufficient for sustained improvement and may even introduce bias (Aghajanyan et al., ; Muennighoff et al., ). This phenomenon can also be seen in Table 3, consistent with Section 3.3.1 Obs.2.

#### 3.3.5 Visual Perception Case Study

Visual perception errors are critical in multimodal error detection tasks, as they impact the accurate comprehension of mathematical problems presented with both text and diagrams. As illustrated in Figure 10, the five primary categories of visual errors observed in GPT-4o (the MLLM with best overall and VIS performance) include distance perception, diagram perception, spatial perception, flip/fold perception, and shape perception. These categories differ in their cognitive demands: distance perception focuses on point identification; diagram perception on quantitative estimation; spatial perception on geometric visualization; flip/fold perception on mental rotation; and shape perception on object classification (Lu et al., ; Zhang et al., ). Detecting such errors is challenging because they often require both intricate visual processing and precise interpretation of mathematical relations, which can be difficult to encode in current MLLMs. To overcome these challenges, future MLLMs should incorporate more advanced visual reasoning capabilities, possibly through enhanced alignment between vision and language modalities, enabling better detection and correction of complex perception errors (Song et al., ; Sun et al., ). This could significantly improve the robustness of MLLMs in mathematical and other perception-heavy tasks.

Figure: Figure 10: Category of bad cases where GPT-4o predicts visual perception errors incorrectly.
Refer to caption: extracted/5909434/figures/gpt_visual_bad_case_category.png

## 4 Related Work

Benchmarks for Mathematical Reasoning.
Recent advancements in mathematical reasoning benchmarks have led to the development of both pure text and multimodal assessments (Lu et al., ; Wang et al., ; Zheng et al., ; Huo et al., ). While datasets like GSM8K (Cobbe et al., ), MATH (Hendrycks et al., ), SuperCLUE-Math (Xu et al., ), and MathBench (Liu et al., ) focus on text-based problems, the field has expanded to include multimodal benchmarks that introduce visual elements, pushing the boundaries of AI’s mathematical understanding. For instance, MathVista (Lu et al., ) evaluates AI’s performance on visual math QA tasks; MATH-V (Wang et al., ) focuses on multimodal mathematical understanding with competition-derived questions; MathVerse (Zhang et al., ) assesses visual diagram comprehension using CoT strategies; CMMU (He et al., ) tests multi-disciplinary, multimodal math understanding with a broad range of Chinese-language questions; and MathScape (Zhou et al., ) further advances the field by presenting longer, more complex, and open-ended multimodal problems. The aforementioned benchmarks assess the mathematical reasoning capabilities of MLLMs by evaluating their problem-solving levels, but they overlook tasks based on the student’s perspective, such as error detection, and thus fail to comprehensively evaluate the more complex role of current MLLMs. Therefore, we propose the ErrorRadar benchmark, which is entirely based on real student response data to evaluate the proficiency of MLLMs in error detection tasks.

Multimodal Large Language Models.
Generative foundation models such as GPT-4 (OpenAI, ), Claude (Anthropic, ), and Gemini (Pal & Sankarasubbu, ) have significantly advanced various task solutions without fine-tuning (Cui et al., ; Yan & Lee, ; Zou et al., ; Zhong et al., ).
Similarly, current open-source MLLMs, built on top of powerful LLMs, have also demonstrated promising potential in multimodal tasks such as image captioning (Yang et al., ) and visual question answering (Fan et al., ). For instance, LLaVA-NEXT (Liu et al., ) proposed projecting visual embeddings, extracted by a pretrained vision encoder, into the word space through a single MLP layer, where LLMs like LLaMA, Vicuna, and Mistral are fine-tuned to understand these post-projection tokens. In a similar fashion, Phi3 (Abdin et al., ), DeepSeek-VL (Lu et al., ), MiniCPM-V (Yao et al., ), ChatGLM (GLM et al., ), CogVLM (Wang et al., ), Intern-VL (Chen et al., ), Qwen-VL (Bai et al., ) and Yi-VL (Young et al., ) also utilize a projector (or adapter, shared compression layer, etc.) to align the visual embeddings extracted from a vision encoder with text embeddings, which are then concatenated and fed into LLM.
Therefore, we propose ErrorRadar, a comprehensive benchmark on a fine-grained evaluation of MLLMs’ ability to detect errors based on students’ answers and reasoning steps, thereby advancing the development of complex multimodal system.

## 5 Conclusion

In conclusion, this work introduces ErrorRadar, the first multimodal benchmark designed specifically for evaluating MLLMs’s reasoning in mathematical error detection scenarios. By focusing on both error step identification and error categorization, ErrorRadar bridges a critical research gap in assessing MLLMs’ capabilities in complex mathematical reasoning. The dataset’s construction, based on real-world student interactions, ensures a robust evaluation framework that reflects genuine user needs. Our extensive experimental analysis, comparing leading open-source and proprietary MLLMs, reveals significant challenges in error detection, highlighting the need for continued advancements in this domain. As MLLMs continue to evolve, ErrorRadar serves as an essential benchmark for driving improvements in the effectiveness of multimodal reasoning systems in real-world applications, on the path to Artificial General Intelligence.

## Appendix A More Multimodal Question Examples

Figure: Figure 11: Multimodal mathematical example one (type: counting) from ErrorRadar dataset.
Refer to caption: extracted/5909434/figures/task_definition1.png

Figure: Figure 12: Multimodal mathematical example two (type: plane geometry) from ErrorRadar dataset.
Refer to caption: extracted/5909434/figures/task_definition2.png

Figure: Figure 13: Multimodal mathematical example three (type: plane geometry) from ErrorRadar dataset.
Refer to caption: extracted/5909434/figures/task_definition3.png

Figure: Figure 14: Multimodal mathematical example four (type: counting) from ErrorRadar dataset.
Refer to caption: extracted/5909434/figures/task_definition4.png

Figure: Figure 15: Multimodal mathematical example five (type: plane geometry) from ErrorRadar dataset.
Refer to caption: extracted/5909434/figures/task_definition5.png

## Appendix B Additional Dataset Details

### B.1 Annotation Details

To ensure the quality and relevance of the ErrorRadar dataset for error detection tasks, we employed a rigorous manual annotation process, involving professional educational experts as annotators. This section outlines the details of the annotation procedure, focusing on how the data was enriched with step-by-step reasoning processes, identification of erroneous steps, and error categorization.

Annotator Selection and Training. Given the complexity of the task, we recruited a group of ten annotators with specialized knowledge in educational theory and mathematics, particularly in K-12 pedagogy. These annotators were trained extensively to familiarize themselves with the structure and expectations of the task. The training covered the specifics of multimodal problem-solving in mathematics, typical student error patterns, and the need for precise identification of reasoning steps that led to incorrect answers. The annotators were also briefed on using the provided tools and the quality assurance process.

Annotation Process. Each mathematical problem in the dataset was annotated with a step-by-step reasoning process, capturing both correct and incorrect approaches to problem-solving. Annotators were provided with:

- 1.
The original question stem (comprising both text and image components).
- 2.
The student’s most frequent incorrect answer.
- 3.
The correct answer to the question.
- 4.
The pedagogical analysis of the correct reasoning process, prepared by educational experts.

Based on these inputs, annotators were tasked with:

- 1.
Step-by-Step Reasoning Annotation: For each problem, annotators mapped out the logical steps that students should ideally follow to arrive at the correct answer. This involved identifying key stages in the problem-solving process, such as formula application, arithmetic operations, or logical deductions.
- 2.
Error Step Identification: For problems where students provided incorrect answers, annotators identified the exact steps where the reasoning went wrong. These error steps were explicitly marked and linked to the incorrect responses, ensuring that they could be traced back to specific problem-solving mistakes.
- 3.
Error Categorization: Once the erroneous step was identified, annotators assigned an appropriate error category based on a predefined schema. These categories included common types of errors such as misinterpretation of the question (More details can be seen in Section 2.1). The categorization was designed to align with known student error patterns in mathematical learning.

Quality Control and Cross-Validation. To ensure annotation accuracy and consistency, each problem underwent two rounds of cross-checking:

- 1.
First Round of Cross-Validation: After the initial annotation, another annotator independently reviewed the annotations. Any discrepancies between the first and second annotators were flagged for further analysis.
- 2.
Second Round of Cross-Validation: In the second round, if the errors or discrepancies persisted, the problem was escalated to a senior educational expert who acted as the annotation lead. The annotation lead adjudicated these contentious cases, ensuring that the final decision was both pedagogically sound and aligned with the dataset’s goals.

Dataset Refinement. Throughout the annotation process, we worked closely with the educational organization from which the dataset originated. This collaboration ensured that the annotations were not only reliable but also adhered to the standards of the organization’s question bank. Additionally, ongoing feedback and updates from the organization helped refine the dataset, making it more accurate and relevant for multimodal error detection tasks.

Annotation Duration and Effort. The annotation process for the ErrorRadar dataset spanned over a period of at least two months. During this time, the annotators, comprised of both professional educational experts and domain specialists, worked meticulously through several stages of preparation, annotation, and validation. Each annotator dedicated significant time to understanding the dataset, reviewing the provided pedagogical analyses, and applying their domain knowledge to identify and categorize errors. The first phase, involving step-by-step reasoning annotation, took approximately six weeks, while the subsequent cross-validation and quality control efforts accounted for the remaining two weeks. Given the complexity of the tasks and the necessity for high precision, the team’s sustained efforts ensured that the final dataset was of the highest quality.

By incorporating these comprehensive annotations, ErrorRadar provides a robust foundation for studying student errors in mathematical reasoning and enables the development of advanced models for error detection and correction.

### B.2 Definition of Problem Type Category

The ErrorRadar dataset distinguishes five primary types of multimodal mathematical problems, each characterized by unique features:

- ✯
Plane Geometry Problems: These involve two-dimensional shapes and figures, requiring knowledge of properties such as angles, lines, and polygons. Solving these problems often depends on understanding basic geometric principles and theorems about plane figures.
- ✯
Solid Geometry Problems: In contrast to plane geometry, solid geometry involves three-dimensional objects, such as cubes, cylinders, and spheres. These problems require spatial visualization and understanding of volume, surface area, and the relationships between different three-dimensional shapes.
- ✯
Diagram-Based Problems: These require analysis of provided visual information, such as graphs, charts, or diagrams, to solve mathematical queries. Interpreting visual data correctly is crucial, as these problems test the ability to extract and analyze quantitative information from visual aids.
- ✯
Algebra Problems: Algebra problems focus on abstract symbols and variables to represent numbers and relationships. These include tasks like solving equations, manipulating algebraic expressions, and understanding functions. The problem-solving process typically involves logical reasoning and manipulation of mathematical symbols.
- ✯
Math Commonsense Questions: These encompass a variety of problem types, including time judgment, direction judgment, counting, and pattern recognition. Unlike the other categories, math commonsense challenges rely on everyday mathematical reasoning and problem-solving strategies that do not necessarily require formal mathematical knowledge, testing intuitive understanding rather than procedural skills.

These problem types highlight the ErrorRadar dataset’s diverse nature, with each category presenting distinct challenges and requiring specific reasoning abilities.

## Appendix C Additional Experimental Details

### C.1 Human Performance Evaluation

In the Human Performance section, the evaluation involved three educational expert evaluators, each independently performing the two subtasks — error step identification and error categorization — on a set of multimodal math problems. To ensure the validity of their assessments, a rigorous cross-checking procedure was implemented. After the initial independent evaluations, the results from all three experts were compared for both the identification of error steps and the categorization of those errors. When discrepancies arose, particularly in cases where the experts disagreed on which step contained an error or how an error should be classified, a structured conflict resolution process was followed.

The cross-check process began with identifying areas of disagreement between the evaluators. These conflicts were discussed in a series of consensus meetings, where the evaluators would review the conflicting steps or categorizations in detail. Each expert provided their rationale, referencing the mathematical principles involved as well as the multimodal representations of the problems. Through open dialogue, the evaluators aimed to reach a consensus on the correct interpretation of the error.

In cases where consensus could not be easily achieved, a majority-vote system was employed. However, for particularly complex or ambiguous cases, an additional adjudicator — who did not participate in the initial evaluations but had equivalent expertise — was consulted to provide a final judgment. This adjudicator reviewed the contentious cases along with the evaluators’ justifications, ensuring an unbiased final decision. The outcome of this process was the creation of a refined ground truth dataset that balanced expert knowledge with the goal of consistent and reliable error identification and categorization.

### C.2 Prompt for MLLM Evaluation

Figure: Figure 16: Prompt for error step identification task.

Figure: Figure 17: Prompt for error categorization task.

### C.3 Model Sources

Table 4 details specific sources for the various MLLMs we evaluated. The hyperparameters for the experiments are set to their default values unless specified otherwise.

**Table 4: Sources of our evaluated MLLMs.**
| MLLMs | Source | URL |
| --- | --- | --- |
| InternVL2-2B | local checkpoint | [https://huggingface.co/OpenGVLab/InternVL2-2B](https://huggingface.co/OpenGVLab/InternVL2-2B) |
| InternVL2-8B | local checkpoint | [https://huggingface.co/OpenGVLab/InternVL2-8B](https://huggingface.co/OpenGVLab/InternVL2-8B) |
| InternVL2-26B | local checkpoint | [https://huggingface.co/OpenGVLab/InternVL2-26B](https://huggingface.co/OpenGVLab/InternVL2-26B) |
| InternVL2-76B | local checkpoint | [https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) |
| Phi-3-vision-4B | local checkpoint | [https://huggingface.co/microsoft/Phi-3-vision-128k-instruct](https://huggingface.co/microsoft/Phi-3-vision-128k-instruct) |
| Yi-VL-6B | local checkpoint | [https://huggingface.co/01-ai/Yi-VL-6B](https://huggingface.co/01-ai/Yi-VL-6B) |
| DeepSeek-VL-7B | local checkpoint | [https://huggingface.co/deepseek-ai/deepseek-vl-7b-chat](https://huggingface.co/deepseek-ai/deepseek-vl-7b-chat) |
| LLaVA-v1.6-Vicuna-7B | local checkpoint | [https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-7b-hf) |
| LLaVA-v1.6-Vicuna-13B | local checkpoint | [https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf](https://huggingface.co/llava-hf/llava-v1.6-vicuna-13b-hf) |
| LLaVA-NEXT-72B | local checkpoint | [https://huggingface.co/llava-hf/llava-next-72b-hf](https://huggingface.co/llava-hf/llava-next-72b-hf) |
| MiniCPM-V2.5-8B | local checkpoint | [https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5](https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5) |
| MiniCPM-V2.6-8B | local checkpoint | [https://huggingface.co/openbmb/MiniCPM-V-2_6](https://huggingface.co/openbmb/MiniCPM-V-2_6) |
| Qwen-VL-9B | local checkpoint | [https://huggingface.co/Qwen/Qwen-VL-Chat](https://huggingface.co/Qwen/Qwen-VL-Chat) |
| GLM-4v-13B | local checkpoint | [https://huggingface.co/THUDM/glm-4v-9b](https://huggingface.co/THUDM/glm-4v-9b) |
| CogVLM2-19B | local checkpoint | [https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B](https://huggingface.co/THUDM/cogvlm2-llama3-chat-19B) |
| Qwen-VL-Max | qwen-vl-max-0809 | [https://modelscope.cn/studios/qwen/Qwen-VL-Max](https://modelscope.cn/studios/qwen/Qwen-VL-Max) |
| Claude-3-Haiku | claude-3-haiku | [https://www.anthropic.com/api](https://www.anthropic.com/api) |
| Claude-3.5-Sonnet | claude-3-5-sonnet | [https://www.anthropic.com/api](https://www.anthropic.com/api) |
| Gemini-Pro-1.5 | gemini-1.5-pro-latest | [https://deepmind.google/technologies/gemini/pro/](https://deepmind.google/technologies/gemini/pro/) |
| GPT-4o-mini | gpt-4o-mini-2024-07-18 | [https://platform.openai.com/docs/models/gpt-4o-mini](https://platform.openai.com/docs/models/gpt-4o-mini) |
| GPT-4o | gpt-4o-2024-08-06 | [https://platform.openai.com/docs/models/gpt-4o](https://platform.openai.com/docs/models/gpt-4o) |

### C.4 CAL and non-CAL Distribution of MLLMs

In this section, we indicate the distribution of CAL and non-CAL category predictions of 21 MLLMs we evaluate, as shown in Figure 18. It can be seen that there is a bias towards CAL category among most open-source MLLMs, while closed-source ones except for Claude-3-Haiku and Qwen-VL-Max do not have such a bias for error categorization task.

Figure: Figure 18: Distribution of CAL and non-CAL category predictions of all MLLMs we evaluate.
Refer to caption: extracted/5909434/figures/CAL_appendix.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: To make this paper more solid and have more potential impact on the LLMs society, could you please address the issues in the weakness?
soundness: 2
strengths: Originality: The paper’s introduction of multimodal error detection in MLLMs is novel, as it addresses an important but understudied aspect of AI reasoning.

Quality:
ERRORRADAR’s dataset is substantial, with 2,500 rigorously annotated K-12 problems, and the authors conduct extensive experiments across over 20 language models, ensuring robust evaluation.

Clarity:
The paper is mainly well-organized, with clear explanations of the benchmark’s design, tasks, and experimental results.

Significance:
ERRORRADAR provides a critical tool for advancing MLLM performance in educational contexts, offering a realistic challenge that highlights current model limitations.
summary: This paper presents ERRORRADAR, an innovative benchmark aimed at evaluating the capabilities of Multimodal Large Language Models (MLLMs) in mathematical error detection, a new direction within mathematical reasoning. The authors identify a significant gap in current benchmarks, which tend to focus primarily on problem-solving, and address this by proposing a formalized task that emphasizes complex reasoning and error identification. ERRORRADAR comprises 2,500 high-quality K-12 math problems derived from real-world student interactions, with extensive metadata and rigorous annotation, making it a unique and robust contribution to the field.

The authors' experimental evaluation, involving both open-source and proprietary MLLMs, effectively highlights the challenges in this task, with models like GPT-4 still trailing human evaluators by approximately 10%. This emphasizes the limitations of current MLLMs in error detection and suggests a clear path for future research.

The strengths of this work lie in its originality, its real-world data foundation, and the thorough experimental analysis across a range of MLLMs. However, the benchmark’s long-term impact will depend on whether it can spur advancements in complex reasoning within MLLMs. Overall, this paper makes a meaningful contribution to the field and provides a promising resource for evaluating and improving MLLMs' mathematical reasoning capabilities.
weaknesses: References: Key references are missing, undermining the paper’s scholarly foundation. For example, *MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI*, a relevant multimodal benchmark is omitted and should be cited in the discussion of related work. It includes collage level mathematics questions and has a similar error classification as your paper. Besides, the mention of unrelated fields, such as e-commerce and urban planning (P1 L38-39), distracts from the focus on educational applications and should be removed.

Error Analysis: While the error analysis identifies issues and possible explanations, it would be more valuable if accompanied by actionable suggestions. For instance, in Finding #4 within the main results, the authors could propose adding domain-specific knowledge to the dataset of foundation model as a possible solution. The improvement suggestions are only observed in 3.3.5 Visual Perception Case Study, which makes the analysis feel limited in scope. Without concrete recommendations throughout, the analysis lacks depth and does not contribute substantially to advancing practical solutions in the field.

Writing and Structure: The paper’s structure is suboptimal. Placing the related work after the introduction would provide a clearer flow, allowing readers to understand the motivations and contributions of ERRORRADAR in context with prior work. Specific points also lack consistency, such as the mixed use of "Obs.2" and "Finding #2" in P9 L455 and P7 L349, which should be standardized.

Figure Presentation: The figures are challenging to interpret and require considerable improvement. For instance, P6 Figure 5 lacks clarity, especially in differentiating the small section labeled "other" for miniCPM. The pie chart format does not effectively convey comparisons between models; a diverging bar chart could better illustrate the "CAL" percentage. Additionally, the color scheme is confusing, with the inner pie chart darker than the outer one with no meaning. It is also unnecessary, as the values are reflected in the size of the pie chart. Similarly, the images of questions in P10 Figure 10 suffers from low resolution, making the table at the central and other details difficult to read. Also, the organization of 5 cases can be improved, for example, they can be presented separately. The quality and readability of both figures should be substantially enhanced.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Why does this benchmark have to focus on image-included samples? Does there exist some text-only benchmarks for text-only error detection?

2. How the authors identify and define the steps in the students' solutions?

3. The work only focuses on the first incorrect steps. Cannot we also consider all the incorrect steps? Would it be possible that the models actually identified other incorrect steps after the first one but were regarded as false predictions?
soundness: 2
strengths: 1. The paper introduces a new benchmark for error detection in mathematical reasoning, which is a unique focus compared to existing benchmarks. The dataset is well-constructed from real-world student data, with rigorous annotation and a comprehensive evaluation of multiple models.
2. The authors did a comprehensive evaluation involving 21 MLLMs and provide some detailed analysis.
3. This work has the potential to significantly impact the development of educational AI systems by improving error detection capabilities.
summary: The paper introduces ERRORRADAR, a benchmark for evaluating the complex mathematical reasoning of multimodal large language models (MLLMs) through error detection tasks. The key contributions include:

1. New Task Formulation: Establishes the multimodal error detection task, focusing on error step identification and error categorization.
2. Dataset Creation: Compiles a dataset of 2,500 K-12 multimodal math problems based on real student interactions, with detailed annotations and error categories.
3. Comprehensive Evaluation: Assesses various open-source and closed-source MLLMs, comparing their performance with human evaluators.
4. Insights and Challenges: Highlights the challenges MLLMs face in matching human-level performance, particularly in complex reasoning tasks.

Overall, ERRORRADAR provides a framework for improving the reasoning capabilities of MLLMs in educational settings.
weaknesses: Overall, this work proposes a good benchmark but has some issues, primarily related to the task introduction, the soundness of the experimental results, and the depth of the analysis.

1. The authors claim the benchmark addresses "more complex" mathematical reasoning, but the dataset contains only K-12 level questions. It would be beneficial for the authors to elaborate on how the error detection tasks for K-12 questions require complex reasoning, better with specific examples.

2. Although the authors provide the formulated task representation in section 2.1, it is not clear or correct enough. Specifically:
    1. From equations on line 188 and 199, The input and output for the two subtasks are not clearly defined. It lacks clarity on whether the question text, image, correct answer, and incorrect answer are part of the input for each of the subtask.
    2. The combined description of inputs and outputs for both subtasks (lines 149, 161, and 166) is confusing, suggesting the task requires training a model that complete the two subtasks simultaneously, which is not the case.
    3. The use of complex variables in Section 2.1 might be unnecessary and make the easy thing more complex, and could be simplified with natural language for better understanding.
Overall, I would suggest the author re-organizing the content in section 2.1 and make it clear and simple.

3. The result presentation and metrics for error categorization may have some issues.
    1. The metric used for reason-specific results resembles recall rather than accuracy. For error categorization, using precision, recall, and F1-score would better reflect performance on biased data and biased prediction. Could the authors provide results on these metrics and explain how they might change the interpretation of the results?
    2. The red value for CAL in Table 3 is not the highest, contrary to what is indicated.

4. The conclusion on line 355 cannot be supported. The accuracies of STEP are not consistently better than those of CATE (e.g., InternVL2 actually achieves much better scores on CATE instead), and they are not hugely different even for strong MLLMs in many cases. Additionally, the two subtasks have different numbers of classification types, which may influence the accuracy values. 

5. More experimental settings can be considered, such as evaluating the impact of including or excluding correct/incorrect answers in STEP, or testing a pipeline approach where the output of STEP feeds into CATE.

6. While the analysis is detailed, it lacks deeper insights or actionable findings. For example, does the output of the models explain why models miscategorize errors? Are there some fundamental ability shortage of the models that we need to further improve? What can we expect and improve for the future applications of MLLMs in education? Etc. The authors may consider providing deeper analyses and discussion on these or other issues.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: What are the reasons that human performance is so far from 100% (despite the source being human annotations), given that the annotators and participants seem to be educators?

What is the foundation for the 5 error types $C$? I.e. are they based on the authors’ intuition, heuristics from prior work, a survey of mathematics educators, etc.?

Can finding #1 be simply assumed by deduction from the generally better performance of closed-source models across tasks?

Is the dataset private, in the sense that it’s unlikely to have been seen by current generation MLLMs?
soundness: 2
strengths: This work addresses a concrete gap, which is clearly established in Table 1: establishing a benchmark that is multimodal (image + text) and has error detection support. I also think several aspects of the analysis in this paper are interesting and valuable, e.g. comparison of several diverse models, attempts to have a comprehensive annotation process, and recruitment of examples reflecting various different topics (shown in stats in Table 2).
summary: This paper proposes a new benchmark for evaluating multimodal LLMs (MLLMs) on mathematical reasoning tasks, and in particular introducing the component of error detection, wherein models are evaluated on detecting where and what types of reasoning errors occur in solutions. These results are compared across several models (both closed and open), and with a baseline from human educators.
weaknesses: Overall, I think more work is needed to establish the robustness of the benchmark.

Given the relatively low performance of the human expert baselines, it’s worth analyzing what the human errors are due to. This would help to:
1. Ensure the low human performance is not due to label noise
2. Better calibrate our expectations of MLLMs

For example, I would not have assumed current generation models would beat or come close to strong human baselines on any of these tasks (given that human educators set the baselines across educational contexts, and that LLMs have well documented flaws in mathematical reasoning).

If human expert performance is really this low on such tasks, then there is a chance that it could be due to:
1. A major, unobserved flaw in our current mathematical education system
2. Low agreement between annotators, which might challenge the establishment of this benchmark as having strong ground truth annotations (if educators can plausibly disagree to this extent)

One way of ensuring #2 does not come into play would be to provide a detailed analysis of inter-annotator agreement. However, the consensus-based process described in Appendix C might not support this (unless the initial disagreements were recorded, and some metric of agreement could be computed from these). In any case, I strongly encourage the authors to provide some reasoning as to why the human performance is where it is, and some evidence that this is not due to factors like label noise, natural and plausible disagreement, etc. I acknowledge that model performance can be evaluated against the human noise/agreement ceiling to mitigate this, but such a low ceiling (e.g. 35.3% on KNOW) makes it a bit difficult to estimate the reliability of the model performance findings.

When thinking about the value of the paper’s findings to the community:
- It’s not clear to me that Finding #1 in the main results (i.e. 3.3.1) makes a significant contribution. It’s generally known that leading closed-source models (e.g. GPT-4o) outperform open models, and this is not surprising given the commercial incentives of model providers and the large cost to train them. Is there some nuance in the results that couldn’t be obtained without this contribution? If so, it would be helpful to highlight this here.
- The extremely high performance of relatively weak MLLMs on CAL doesn’t seem adequately explained by the hypothesis in the paper that such models overfit to this category to avoid learning more complex ones. If the category is indeed simple, then stronger models should perform well on it too, unless (intuitively) doing better on CAL tends to causally lower performance on other categories. So, I think a more compelling explanation of this phenomenon is needed to trust that such a result would generalize. For example, an alternative hypothesis is that these models have seen these or similar questions during their pretraining, trivially memorizing the questions or near neighbors, and the models that do not perform well on this category have not. I notice two papers are cited as to the paper’s hypothesis for the result; if there is some more compelling explanation relating to CAL overfitting, a clear explanation and allusion to those papers’ results would be very helpful.
- Likewise, the explanation for Finding #3 doesn’t seem complete. The analogy to object detection isn’t clear; a more direct analogy would be if localization (predicting where an object is) is simpler than classification (predicting what an object is). However, is this the case empirically? Categorization seems simpler than “fully understanding [objects’] attributes or behavior” since solving a classification task doesn’t seem to require this deeper understanding (e.g. we would not claim that an ImageNet-trained classifier understands the attributes or behavior of zebras, for example, but distinguishing them from horses seems possible via pattern recognition.
- The findings in section 3.2.2 generalize across closed-source vs. open MLLMs, but is there an a priori reason to believe that the openness of a model causally affects its capabilities or confusion on different tasks? Or is this more of a commentary on scale/performance in general? I.e. could a line be drawn between low-performing or high-performing models? This seems like it would agree with the scaling law analysis presented in 3.3.4. Otherwise, again it’s not clear that this interpretation of results is robust to e.g. a low-capability closed-source model (such as local models used in things like Apple Intelligence) or a high-capability open model (such as large LLaMa models).
- Additionally, the analysis in 3.2.2 might benefit from plotting a confusion matrix, rather than highlighting only the most common mis-associations in results (this could be in an appendix). Figures 6 and 7 show the erroneously selected categories, but not what they were supposed to be, which makes it challenging to observe what kinds of nuanced reasoning errors might be driving this behavior (and where models could be improved, more specifically).


Some notes on writing and presentation:
- The abstract obscures the core contributions, and also contains several subjective claims (e.g. “comprehensive”, “rigorous”, “rich” without definition of what is meant by these). It would be helpful to simplify it to focus on the factual contributions of this work.
- Similarly, the applicability of multimodal LLMs to mathematical reasoning tasks is cited as “especially” promising, but no reason is given for this (e.g. over and above text-only LLMs, which are largely used for mathematical tasks currently). I recommend making this clearer, since it would help readers quickly orient to the potential contribution of the given work.
- Line 096: What exactly does “featuring advanced research scopes such as complex reasoning” mean?
- Line 099: It is a bit odd for the paper to refer to its own contribution as insightful. This is a determination traditionally left to external reviewers, the community, etc. Such cases occur in other parts of the paper as well (e.g. in the abstract as noted previously, or like "well-annotated” in the Figure 2 caption). I encourage the authors to carefully use neutral language when discussing their results, and focusing on the facts established in the paper.
- Re: notation in 2.1, $T$ is used to denote problem, but this is a bit confusing (given that text is a subset of a problem, and text also begins with T). Wouldn’t it be clearer to use $P$ for problem, or $Q$ for question, or similar? Considering that $A$ is answer, $S$ is step $\mathcal{M}$ is model, etc.
- Similarly the use of $Step$ for an incorrect step, such that $Step \in S$ is a bit confusing, considering that this also seems to be denoted by $S_{x,i}$. Also, shouldn’t the output be simply $x$, i.e. the step index rather than the step content itself? It might be helpful to do a pass on notation to help simplify and make it more intuitive, so the reader can quickly follow.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Question: Did you consider evaluating ERRORRADAR with models beyond mainstream MLLMs, such as hybrid symbolic-neural models or models with dedicated reasoning layers? How well do you believe ERRORRADAR would generalize across different model architectures?
Suggestion: Including diverse model types could reveal whether certain architectures perform better on specific error categories, providing insights into optimal architectural approaches for error detection in complex reasoning tasks.


Question: How do you envision ERRORRADAR performing in real-world educational settings, such as in online tutoring or AI-driven assessment tools? Have you considered testing or simulating the benchmark under real-time conditions?
Suggestion: Running a real-world simulation or discussing any potential deployment challenges would provide readers with valuable context on how ERRORRADAR might function in practical applications, enhancing its relevance.

Question: Did you observe any biases in error detection, such as models favoring certain error categories? For instance, do any models show a tendency to overpredict certain types of errors?
Suggestion: Providing an analysis of category-specific biases could help address fairness concerns in educational applications. Understanding these biases could inform the development of more balanced models and benchmark settings.


Question: Can you provide further details on the annotation process, such as inter-annotator agreement scores or quality control measures? How were discrepancies handled, especially for complex or ambiguous cases?
Suggestion: Including inter-annotator agreement metrics and a description of the quality control process would improve confidence in the dataset’s reliability, especially given the nuanced nature of error categorization.

Question: Did you conduct any ablation studies or baseline evaluations focused on specific error types, such as isolating visual perception errors? How does each modality contribute to the overall performance of ERRORRADAR?
Suggestion: Performing ablation studies or creating error-specific baselines could clarify which modalities (textual vs. visual) or model components contribute most to performance, potentially guiding improvements in multimodal model design.
soundness: 4
strengths: Originality:A primary strength of this paper is its originality, specifically in redefining evaluation for mathematical reasoning tasks. ERRORRADAR goes beyond traditional problem-solving benchmarks by focusing on error detection, introducing two distinct tasks: error step identification and error categorization. These tasks address a critical aspect of educational assessment that current benchmarks overlook, particularly when using multimodal data. By categorizing errors into five types—visual perception, calculation, reasoning, knowledge, and misinterpretation—the benchmark provides a nuanced approach that aligns with real-world educational needs. The use of real student responses adds further originality, as few existing benchmarks leverage authentic student data to assess model capabilities in complex reasoning.

Quality: The methodological quality of this paper is high, marked by rigorous data curation and detailed experimental design. The ERRORRADAR dataset consists of 2,500 well-annotated, multimodal math problems with stepwise incorrect student responses. The dataset is collected from a large educational organization, ensuring diversity and relevance to actual student problem-solving scenarios. The authors evaluate a comprehensive set of open-source and closed-source MLLMs and benchmark them against human evaluators, providing a robust comparison. This broad evaluation captures multiple facets of model performance, highlighting strengths and limitations across both error identification and categorization tasks. The inclusion of accuracy metrics for different models across error types offers valuable insights for improving MLLMs’ reasoning abilities.

Clarity: The paper is generally well-structured and clear in its presentation, with a logical progression from problem formulation to methodology and results. Figures, such as performance comparisons and error distribution visuals, effectively support the narrative, enhancing reader comprehension. The detailed explanation of the error types and tasks aids in understanding the benchmark’s scope and objectives. However, certain sections, particularly those describing the technical setup for error categorization, could benefit from further clarification, which would increase accessibility for a wider audience. Overall, the clarity of the main contributions is strong, though minor improvements could enhance the paper’s readability.

Significance: The significance of this work is substantial, as it opens new directions for evaluating MLLMs in educational contexts, where error detection is as crucial as problem-solving accuracy. By focusing on multimodal mathematical error detection, ERRORRADAR highlights a critical area where current models fall short of human performance, particularly in understanding errors in visual and logical reasoning. The findings emphasize the need for further research to bridge this gap, making the benchmark a valuable tool for both academia and industry. Given the growing use of AI in educational technology, ERRORRADAR’s contributions are timely and likely to stimulate further work in developing models that better support human learning and assessment.
summary: The paper,  introduces ERRORRADAR, a new benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on their ability to detect errors in complex mathematical reasoning. Unlike traditional benchmarks that focus solely on problem-solving accuracy, ERRORRADAR addresses the challenge of identifying and categorizing errors in student solutions. It includes two main tasks: error step identification, which locates the first incorrect step in a student's solution, and error categorization, which classifies the error type.

The dataset comprises 2,500 annotated K-12 math problems, incorporating real student answers from a large educational organization and covering a range of error categories such as visual perception, calculation, reasoning, knowledge, and question misinterpretation. Extensive experiments comparing various open-source and closed-source MLLMs, including GPT-4o and Gemini 1.5, demonstrate that while MLLMs are improving, they still perform below human-level accuracy in error detection. ERRORRADAR provides a valuable resource for the research community, offering a structured approach to assess and enhance MLLMs' reasoning abilities in multimodal educational settings.
weaknesses: Issue: The paper evaluates a broad range of MLLMs, yet the chosen models are not fully representative of the diversity in architectural design or training methodologies. The focus is largely on mainstream MLLMs, which may limit insights into the broader applicability of ERRORRADAR to varied model architectures.
Suggestion: Incorporating models that represent different architectural approaches, such as hybrid symbolic-neural systems or models with explicit reasoning layers, could offer a more comprehensive view of how ERRORRADAR evaluates different reasoning frameworks. This would help determine whether specific architectures consistently perform better on certain error types, which could have implications for future MLLM design.

Issue: While ERRORRADAR’s dataset is derived from real student interactions, the paper does not address how the benchmark might perform in actual deployment scenarios, such as in online tutoring systems or educational applications where students interact with AI in real-time.
Suggestion: The authors could simulate a real-world deployment setting, assessing factors such as response time, user feedback integration, or adaptability in evolving learning environments. This would add practical value, helping readers understand how ERRORRADAR might function outside of controlled benchmarking scenarios and providing insights into potential deployment challenges.


Issue: Different MLLMs may exhibit varying biases toward certain error categories due to their underlying training data and design, but the paper does not delve into these biases in detail. This oversight could affect the fairness and reliability of the models in educational contexts, where bias could disadvantage certain student demographics.
Suggestion: An analysis of category-specific biases in the models would add depth to the findings. For instance, are certain models disproportionately prone to labeling calculation errors over reasoning errors? Investigating these tendencies could inform guidelines for model selection and training adjustments to ensure balanced performance across error types.

Issue: The paper touches on cognitive load associated with different types of errors (e.g., reasoning errors appearing in later problem-solving steps), but it does not provide an in-depth analysis of how cognitive load might impact error detection performance across models.
Suggestion: Including a more detailed cognitive load analysis, possibly with additional metrics or models designed to handle high cognitive load tasks, would provide richer insights. For example, testing models’ performance on questions with high interactivity and cognitive demand could highlight specific areas where models may benefit from cognitive load-aware training.


Issue: While the dataset annotation process is described, details regarding inter-annotator agreement, consistency checks, and the process for resolving discrepancies are sparse. This information is crucial for understanding the reliability of the dataset, especially given the complexity of categorizing mathematical errors.
Suggestion: Adding inter-annotator agreement metrics, outlining quality control measures, and discussing how ambiguous cases were handled would strengthen the dataset’s credibility. It would also help readers gauge the dataset’s reliability for evaluating complex reasoning tasks.


Issue: The evaluation focuses on overall model performance, but lacks targeted baselines or ablation studies for each error type, especially for the multimodal aspects of error detection, which involve both textual and visual information. This limits the insights into specific error categories and multimodal model performance.
Suggestion: Introducing error-specific baselines, such as comparing models on tasks that isolate visual or reasoning errors, would clarify which aspects of ERRORRADAR are most challenging. Ablation studies that selectively disable visual or language components could also reveal the contribution of each modality to overall performance, guiding future multimodal benchmark development.

Issue: The paper does not address the computational demands of using ERRORRADAR to evaluate large models, which could impact its adoption in academic and industry settings with limited resources.
Suggestion: Discussing the benchmark’s computational requirements, particularly for larger MLLMs, and proposing strategies to improve efficiency (e.g., optimized sampling or reduced evaluation steps for preliminary testing) would make ERRORRADAR more accessible. Additionally, insights into potential trade-offs between accuracy and computational efficiency could benefit readers interested in practical deployment.

Meta-review: This was a borderline paper that I believe was submitted too soon.  The reviewers pointed out extensive issues with the paper and while the authors did significant work in addressing those issues during the rebuttal period, the main result was that one reviewer moved from marginally below to marginally above threshold.  

There has been extensive reviewer analysis and author response, but to summarize, the paper simply needs more work, both isn terms of establishing the robustness of the benchmark and in the writing and presentation.  This paper could possibly be bumped up given all the authors work in the rebuttal period, but I really think it would be better if they re-wrote it and submitted again.  The requested corrections are extensive and have not been properly reviewed by all reviewers.

Decision: Reject