Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
Authors: Yunzhuo Hao
Jiawei Gu
Huichen Will Wang
Linjie Li
Zhengyuan Yang
Lijuan Wang
Yu Cheng
Venue: ICML 2025 oral (2025)
Keywords: Benchmark
Multimodal
Reasoning
Primary area: deep_learning->large_language_models
TL;DR: We contribute a challenging multimodal reasoning benchmark.

Abstract: The ability to organically reason over and with both text and images is a pillar of human intelligence, yet the ability of Multimodal Large Language Models (MLLMs) to perform such multimodal reasoning remains under-explored. Existing benchmarks often emphasize text-dominant reasoning or rely on shallow visual cues, failing to adequately assess integrated visual and textual reasoning. We introduce EMMA (Enhanced MultiModal reAsoning), a benchmark targeting organic multimodal reasoning across mathematics, physics, chemistry, and coding. EMMA tasks demand advanced cross-modal reasoning that cannot be addressed by reasoning independently in each modality, offering an enhanced test suite for MLLMs' reasoning capabilities. Our evaluation of state-of-the-art MLLMs on EMMA reveals significant limitations in handling complex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of-Thought prompting and test-time compute scaling underperforming. These findings underscore the need for improved multimodal architectures and training paradigms to close the gap between human and model reasoning in multimodality.

Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
Yunzhuo Hao I1 Jiawei Gu 2 Huichen Will Wang * Linjie Li
Zhengyuan Yang*
Lijuan Wang*
Yu Cheng*
Abstract
Multimodal reasoning question EMMA
The ability to organically reason over and with both text and images is a pillar of human intelli- gence, yet the ability of Multimodal Large Lan- guage Models (MLLMs) to perform such multi- modal reasoning remains under-explored. Exist- ing benchmarks often emphasize text-dominant reasoning rely on shallow visual cues, fail- ing to adequately assess integrated visual and textual reasoning. We introduce EMMA (En- hanced MultiModal reAsoning), benchmark tar- geting organic multimodal reasoning across math- ematics, physics, chemistry, and coding. EMMA tasks demand advanced cross-modal reasoning that cannot be addressed by reasoning inddpen- dently in each modality, offering enhanced test suite for MILMs reasoning capabilities. Our evaluation of state-of-the-at MLLMs on EMMA reveals significant limitations in handling com- plex multimodal and multi-step reasoning tasks, even with advanced techniques like Chain-of- Thought prompting and test-time compute scaling underperforming. These findings underscore the need for improved multimodal architectures and training paradigms to close the gap between hu- man and model reasoning in multimodality. The project homepage can be accessed at https: / //emma-bennhee github. io/.
.22
-30
Question: Three point charges, of +Q. = and -3Q are placed equidistent as shown Which vector best describes the net direction of the electric force acting on the iQ charge?
Human reasoning
The direction of the electric forre due ro-2 and caQ can he vistalized with a quick free-hody
sketch of the electiic forces.
Combining the focces.
repel
attract
v |+/Q|=|-201 repel attrect
Model reasoning (GFT40)
Forzes Oue to the chargex:
Magnitudes (Coulomb's Law).
-2Q
+Q
Force to -2 F. s0
(Farce points cowmoard-He letts
Forca due to I /3Q: Fl sec
The + -0 charge mepa the +Q charge.
Error:
While the model understands that like charges repel, incornectly identifiee the force downward right'. when should upward left
Figure 1: A sample multimodal reasoning question in EMMA.
Humans engage in graphical reasoning: guided by the principles of electric force. they draw force vectors with appropriate directions and visually compute their sum. While GPT-4o understands that like charges repel, mistakes the direction of the repulsive force, highlighting its limitations multimodal reasoning.
visual reasoning enables us to draw insights from complex visual information. Combining these skills provides a robust framework for solving technical and creative challenges.
Recent advancements in Large Language Models (LLMs) have significantly enhanced their reasoning abilities (Ope- nAI, b; Qwen, 2024; Zhao et al., 2024; DeepSeek, 2024), enabling strong performance on tasks such as formal logic reasoning (Hendrycks et al., 2020b), graduate-level aca- demic question answering (Rein et al., 2023), and competi- tive programming (codeforces; Bai et al., 2023; Hendrycks et al., 2020a). Despite these successes, these models primar- ily focus text-only reasoning, leaving open question: can Multimodal LLMs (ML.LMs) effectively reason across both language and visual inputs?
1. Introduction
Multimodal reasoning is fundamental to human intelligence. For example, interior designers combine textual descriptions with mental imagery to optimize room layouts. Text-based reasoning allows us to analyze abstract concepts, while
Equual contribution Universsty of Electronic Science and Tech- nology of China Sun Yat-sen University 'University of Washing- "Microsoft "The Chinese University Hong Kong. Corre- spondence to: Yu Cheng <chengyu05 wawpggyyyus@@gaal.coommm gmail.com>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
A major bottleneck in addressing this question is the lack of appropriate benchmarks. Existing multimodal benchmarks

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Math
Physics
Chemistry
Coding
Qf monker has smm off gloce of Capasis Jxdi map, Wast dos the piace die monkee hes son off emaggon
lig
the intalioo slane studuue slown the imgg cxlestor
make
5.
longih. The mas sabsd atgle of to reaeeve to veriail as shown, and ndloeed. sped the masi ut the botoon of swing
rechding engle doable and tight boodd but exchaling those innlving hydnsgmn Note: Ulnnggrd armee Consder component presest in the mandionn anaduer
a. 60 miss h 7.7 Fimes
imagr
2.14
10
Knowledge- baseed Counting
Q. Ki sadens, andagan Sixp caciio
Q Which code stigpet will genmute
Transformation
Visuel Decomposition Simulation
O besahta mode conmmation on gidt, mxnty aime ligttd colorci obbo und othen darke. wil
long magnet placel
Hore
>
A.4
1.
loop oeseted shown. which deectass will the Nort pode har sssanst srece
Advanced Chart Type
P CP-R-Q-5
BP
Structure Recognition
Thebar fedl Sance cureet loop.
time-ppabiing digpass type of cheriial irage saed illaane
O
Simulation
OR
clector
3D Spatial Simulation
Ut A biland bat roltnn smoce sote Jirecion at metess per socoad his odge cabte perperdicait anglt bounces accandn the opposit directins. the graaton nagritude of accemnation for the billand hull was sime:
Hieme provide the SMILES ocmouass mickerale atterh clectos has boen adecatd, deyiceed Irage
orgger
Compliae dee matris.
the visalioaiin doow??
A.4b
4:4b
0.4p
A racccmppacCCCCo mescrpp-oroco MCXCIP--OCcI
PXC PIFENSCIOO
3D
Reaction Simulation
1e-22 10
G
CLL ig
J. T 10
6x 18"
Pattern Inference
Graph Reasoning
reariangement
U The Langarso wante war the insia, On way ii not allowed janp throuht i square chows juny langanoo
alle pasling daungh foam F Wiuch of te tetlewing diggrme shows the reliccedd my onmeety?
incilar, splatical
Qr-An lagone pualina" digaran of chanica image and to theimas elerton flow moch weitt orgs raleesle yadergros change
allowed take?
Nore: The smoot side part nims,
Wask
splino
O Whin venitattion will fillownng mole sripet <b
medecttle the change??
bock
Alignment, Orientation, & Position
A
Reaction Simulation-Pro
(1 freet
Path Tracing
Or The Pf Crromonii Ruccion of the resadie Noit (3111)
Q the cicuut shownn bedon, capecior C- 4F.
wiih 29 aires bolnw wat Minntbl uassaene
oo
inductor L-3H miiste duped contigaaaioo
Path TracingiChange view simulation
The ciera is fed with
O: Suttran reber quras. Sommeral Ha stiny cránkss How many decccs uc left!
unkuton fiequnacy wilh pask vellge iú - V.
Dxtmmmte the magestnde of the macmum instnntrnanu outpet valtage showrs Jispan
ac finator or time
reproserzed by:
8. 0.46
thee changy code below
which gesutoee the fire tmage so ther gascntes the socnal anage insge shown? ap>
c.4/>
4/>
Multi-hop Visual Object Counting
Multi- -hot Visual Reasoning
Graph Reasoning
Polar coordinates
Overview of EMMA. EMMA covers four subjects: math, physics, chemistry, and coding. Questions are categorized based on the specific skills they measure, enabling fine-grained analysis.
Figure 2:
largely test surface-level visual understanding (Antol al., 2015; Goyal et al., 2017; Yu al., 2023; Akter et al., 2024) or textual knowledge (Yue et al., 2024b;a; Marino et al., 2019) recall with multimodal inputs. While some bench- marks (Lu et al., 2024b; Wang et al., 2024a) include math questions with images, Zhang et al. (2024a) have shown that many of these tasks reduce to language-only reasoning, as the visual content is often fully described in text.
visual- and language-bard problem-solving EMMA fea- tures questions that are difficult solve by relying solely on text-based reasoning single visual pass. Instead, solv- ing these problems necessitates back-and-forth process between interpreting visual inputs and applying multimodal reasoning steps, where visual aids are often integral or more efficient for arriving at the solution. For instance, Figure illustrates a sample physies problem that asks for the direc- tion of the net electric force. While GPT-4o understands that like charges repel, it mistakes the direction of the repulsive force, highlighting its limitations in multimodal reasoning.
To address this gap, we introduce EMMA: Enhanced ModtiModal reAsoning benchmark, specifically designed to evaluate the ability to solve problems that require both
2

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Unlike recent benchmarks (Ramakrishnan et al., 2024; Chol- let, 2019), which focus on spatial cognition, or visual puz- zles that can be perfectly represented in text, EMMA in- troduces domain-specific challenges where reasoning can often be strengthened by visual aids. These include tasks like 3D spatial transformations, chemical structure recogni- tion, multi-step physical simulations, and program output visualization (Figure 2). EMMA consists of 992 multimodal reasoning questions gathered from existing benchmarks through a rigorous filtering pipeline, and 1,796 newly con- structed questions created manually in collaboration with do- main experts. Our evaluation of ten sate-offtheearr (SoTA) MLLMs on EMMA reveals three key findings:
2. Related Work
Multimodal Large Language Models Recent years have witnessed rapid progress in MLLM development. Building upon early techniques in lision-language modcling (Tan & Bansal, 2019; Lu et al., 2019; Chen et al., 2020; Radford et al., 2021; Li et al., 2020; Zhang et al., 2021; Yu et al., 2022), modern MLLMs (Li et al., 2024a; Lu et al., 2024a; Team, c; Liu et al., 2024a; Yang et al., 2024a; Achiam et al., 2023; Team, a; Li et al., 2023) leverage the success of LLMs and various visual instruction tuning techniques (Liu et al., 2024b;a; Zhu et al., 2023), achieving impressive perfor- mance in many multimodal tasks.
LLM and MILM Reasoning Sf models (Ope- nAI, b; Qwen, 2024; Zhao et al., 2024; DeepSeek, 2024) now achieve strong performance on tasks such as formal logic reasoning (Hendrycks et al., 2020b), graduate-level academic question answering (Rein et al., 2023), and com- petitive programming (codeforces). These advancements in text-based reasoning have spurred growing interest in multi- modal reasoning, exemplified by visual CoT models (Shao et al., 2024; Zhang et al., 2023) and visual CoT prompting techniques (Zhou et al., 2024). Although visual CoT prompt- ing techniques have shown promise, their focus is primarily on enhancing perception through methods like cropping im- ages to simulate attention. Hence, these approaches offer limited support for tasks that demand more advanced visual reasoning skills, such as visual manipulation or imagination.
MLLLMs struggle with multimodal reasoning: All models perform suboptimally on EMMA, regardless of the usage of Chain-of-TTn Chain-of-Thought (CoT) prompting (Wei et al., 2022). On the balanced subset of EMMA, the best-performing model, Gemini 2.0 Flash Thinking, scores only 48.00% which is 10.75% higher than the best non-reasoning MLLM, Qwen2-Ve VL (Wang et al., 2024b), but still trails human ex- perts by 29.75%. These results suggest a limitation of cur- rent MLLMs in performing in-depth multimodal reasoning.
Test-time compute scaling methods with textual CoTs are insufficient: We explore test-time compute scaling of SoTA MLLMs with different methods (e.g., majority voting, best-of-N, and tournament) up to 16 times, yet they still fail to address the multimodal reasoning challenges in EMMA. Simply increasing the number of candidate responses with textual CoTs does little to compensate for the models' inabil- ity to produce valid visual reasoning steps, particularly for tasks requiring fine-grained spatial understanding multi- step reasoning. In addition, current MLLMs and specialized reward models struggle with complex multimodal reasoning themselves, which can make their reward signals unreliable and limit the utility of test-time compute scaling.
Multimodal Reasoning Benchmarks Most existing rea- soning benchmarks are text-based (e.g.. (Cobbe et al., 2021; Hendrycks et al., 2021: Srivastava et al., 2022; Jin et al., 2023; Suzgun et al., 2022)), but the rising demand for multi- modal evaluation has led to the development of benchmarks across diverse domains (e.g., (Lu et al., 2024b; Wang et al., 2024a; Li et al., 2024b; Yang et al., 2024c; Ying et al., 2024; Chen et al., 2024a; Li et al., 2024c; Cheng et al., 2024)). Re- cent efforts focus on spatial and relational reasoning (Akter et al., 2024; Ramakrishnan et al., 2024) and college-level reasoning requiring domain knowledge (Yue et al., 2024a). However, many multimodal benchmarks contain redundant text-image information, allowing models to bypass visual reasoning (Zhang et al., 2024a). To address this, MMMU- Pro (Yue et al., 2024b) incorporates filtering pipeline to enhance multimodal evaluation. In this work, we further refine such approaches by curating a benchmark that fo- cuses explicitly on tasks requiring strong visual reasoning. Unlike existing benchmarks, our test suite emphasizes mul- timodal reasoning challenges that are difficult to solve with baxt- reasoning and single visual pass.
Through error anal- ysis, we find that SOTA MLLMs frequently struggle with tasks requiring precise spatial simulations, multi-hop visual reasoning, and integration of visual and textual information. These shortcomings are particularly pronounced in prob- lems where visual aids offer a simpler path to the solution. Further, textual CoT negatively impacts model performance on visual-reasoning-heee
Visual reasoning is the bottleneck:
tasks, highlighting the need for new paradigms to improve visual reasoning.
These insights suggest that the performance gap between text-based and multimodal reasoning arises from MLLMs' limited ability to perform fine-gradd -ne-ggained visual reasoning. EMMA highlights the need for new architectures and train- ing paradigms that can better integrate and reason over di- verse modalities, enabling models to leverage both visual and linguistic information more effectively.
3

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Table Key statisties of EMMA.
Statistic
Number
Rea Smaslattoo res de
perermann Sen ABSO
Knowledge--a Based
Total questions
2,788
Counting
Multiple-choice questions
2,002 (72%)
3D Simulation 10%
16%
Free-form questions
786 (28%)
Questions with answers
2,788 (100%)
Quusstions newly added
1,796 (64%)
2D Transformation 10%
Chemistry 42%
Structure Itecognition
Image in the question
2,599 (93%)
Math 32%
Image in the option(s)
195 (7%)
17%
Prohlems with multiple images
298 (10%)
Path Simulation 5%
Physics
Coding 20%
Maltt-Hop Obj
Count 4%
Pestern Interene
3. The EMMA Benchmark
Madiy wil Onl ons
6%
Puth fresing 6.47%
4%
Vi choose Code
shi
3M
of ind
Craph flx 25,
3.1. Overview of EMMA
Strudatioon
We introduce EMMA, an Enhanced MultiModal ReAsoning Benchmark. EMMA composed of 2,788 problems, of which 1,796 are newly constructed, across four domains: math, physics. chemistry, and coding. All questions in FMMA are in either multiple-choice or open-ended formats with short, easily checkable ground truth answers. The key statistics of EMMA are summarized in Table 1, and its composition is presented Figure 3.
Visal Decomansi
Figure 3: Composition of EMMA.
Filtering Mechanisms To filter for questions that require multimodal reasoning, Yue et al. (2024b) provide only the text from multimodal reasoning questions to LLMs and discard questions that can be correctly answered this way. Nonetheless, some of the remaining questions may still not truly measure visual reasoning, as a single pass of visual per- ception and language understanding may suffice to answer them. We extend this one step further (illustrated in Fig- ure 4): we first caption the images multimodal reasoning questions using GPT-4o and then pass both the original text and our generated captions MLLMs, filtering out ques- tions that can be answered under this condition. For each candidate question, we query Llama-3- 7-7---I-Issee (Dubey et al., 2024), GPT-4o, and Qwen2-72B-Innse 2024a) ten times; if any model answers a question correctly at least five times, we discard it, following the 5/10 threshold in MMMU (Yue al., 2024a). This more stringent filtering ensures that the remaining questions require models to en- gage deeply with visual information. We introduce the data collection process for each project in detail below.
To provide fine-grained insights into how MLLMs might fail in multimodal reasoning, assign labels each problem in our benchmark. These labels are either created by domain experts or assigned by GPT-4o and subsequently verified by experts. As shown in Figure 2. questions in EMMA assess a wide array of multimodal reasoning skills. For example, the pattern inference problem in math challenges models to identify and generalize visual patterns; the visual decom- position simulation problem in physics requires graphically decomposing forces to determine resultant effects; the re- action simulation problem in chemistry demands precise interpretation and simulation of electron movement; the 3D visualization problem in coding' evaluates spatial imagina- tion by requiring models to associate function calls with their corresponding 3D representations.
(Yang et al.,
Math We first apply the filtering pipeline to Math- Vision (Wang et al., 2024a) and Math (Lu et al., 2024b), and then manually inspect the remaining set and craft tax- onomy consisting of five categories with a strong focus on multimodal reasoning, including 3D Simulation, 2D Trans- formation, Path Tracing, Multi-hop Object Counting, and Pattern Inference. Next, we use GPT-4o to categorize all questions based on this taxonomy, followed by a manual ver- ification. In addition, we supplement our benchmark with additional pattern inference questions from RAVEN (Zhang et al., 2019), which inherently require multi-hop visual rea- soning. This process results in total of 892 math questions.
3.2. Data Curation
As discussed in Section 2, most existing multimodal reason- ing benchmarks likely contain many problems that primarily measure text-based reasoning. To address this, we employ two-step approach to constructing EMMA (Figure 4). First, we source problems from existing multimodal reasoning benchmarks and apply rigorous filtering to exclude those solvable through text-based reasoning and a single visual pass. Next, we categorize the remaining problems for each subject into fine-graine multimodal reasoning skill tax- onomies and manually collect more samples aligned with these taxonomies to expand our dataset.
Physics We apply the filtering pipeline to multimodal physics problems in OlympiadBench (He et al., 2024), EXAMS-V (Das et al., 2024), and MMMU (Yue et al., 2024a), which yields only 80 problems. In addition, we manually collect more problems online from Learn AP
Different from the other subjects in EMMA, coding questions be assigned more than one category since our visualizations tend to employ multiple advanced techniques.
4

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Image
Image
spine configuration, line style, and axis scaling) either man- ually through prompting MLLMs, with post-hoc manual verification. We provide these design variations labels for each problem. Finally, we construct different types of questions using these visualization sets. Our curation results in 564 multiple-choice coding questions.
Question: fuaciio!!
Question: Mis gndy
Caption: The funmtinn o5
Ception:: A grat-hood pizzle feeneneeeec
3.3. Comparison with Existing Benchmarks
QQuestion Only)
X
(Question Only)
Question Caption]
Question Caption]
Our enhanced data filtering pipeline ensures that EMMA focuses on questions requiring in-depth multimodal reason- ing, i.e., those that cannot be solved solely using text-based reasoning or single visual pass. While MMMU-Pro ( (Yue et al., 2024b) removes questions solvable through their text portion alone, it may still retain problems for which vi- sual reasoning is inessential. In contrast, EMMA applies a stricter filtering criterion, discarding questions solvable with text and image captions. For instance, the left example in Figure 4 (adapted from Vista) asks whether a depicted function is even or odd. Although unsolvable without the image, the problem can be shorteut by extracting the func- tion's text expression embedded in the image. In this case, the role of vision is more aligned with visual perception than with visual reasoning. By eliminating such problems, which MMMU-Pro 's approach would retain, EMMA better evaluates the multimodal reasoning capabilities of models.
Filtored out
Retained
Sourced from existing benchmarks
EMMA Further anriched manually
H
Mamaally
Dorated froe strwtch
collected
Filttered out
Figure 4: Data curation process for EMMA (Sec. 3.2).
Physics (Physics) and Khan Academy (Academy) and filter them, resulting in 76 more new problems. Through manual labeling, we verify that these problems span a wide range of topics, including 3D Field Simulation, Graph Reasoning, and Path Tracing. We note that despite our best efforts, mul- timodal physics problems meeting our criteria are difficult to source and construct.
We also contribute 1,796 novel multimodal reasoning prob- lems across physies, chemistry, and coding. After filtering physics and chemistry problems from all relevant bench- marks to our knowledge (e.g., (He et al., 2024; Das et al., 2024; Yue et al., 2024a)), only 100 remain. We expand this to 1,332 in EMMA by mamually sourcing additional data and hiring domain experts. For coding, EMMA is the first benchmark to systematically evaluate data visual- ization skills using a multiple-choice format, enabling a standardized assessment and obviating the need for MLLMs as judges. Moreover, through meticulous manual labeling or verification, we provide fine-grained labels for each ques- tion (Figure 2), categorizing them based on the specific skills they assess. These labels enable a detailed analysis of MLLM performance, as we demonstrate in Section 5.
Chemistry After filtering the chemistry portion of MMMU (Yue et al., 2024a) and EXAMS-V (Das et al., 2024), we are only left with 42 questions. Since these ques- tions mostly involve reasoning about molecular formulas, we construct more problems on organic chemistry. We ana- lyze the chemical properties of molecules in SMiCRM (Le- ung et al., 2024) with RDKit (lan, 2013), computational chemistry toolkit, and develop 904 novel questions over chemical structure recognition and bond counting. In ad- dition, we draw from the collection of chemical reactions in Li (2009) and collaborate with PhD students in chemistry to annotate reaction outcomes, contributing another 210 questions on reaction simulation.
Coding We design four coding tasks to assess MLLMs in real-world visualization creation scenarios. For instance, to evaluate the ability to reproduce a visualization, we con- struct "Vis Choose Code" questions where models select the code that generates a target chart. Since previous visual- ization design benchmarks all use MLLM judges, we do not source from them, but manually construct all coding ques- tions from scratch. We first identify "seed visualizations" us- ing advanced visualization techniques from CharXiv (Wang et al., 2024c), the matplotlib example gallery (Team, b) fol- lowing (Wu et al., 2024), and our prior experience. Next, we generate four variations for each seed visualization to form a "set" by introducing design variations (e.g., changes in
4. Experiments
4.1. Evaluation Settings
Data Split To create more balanced subset of EMMA, we randomly sample 400 questions (100 per subject) from the benchmark, hereafter referred to as EMMA-mini. Within each subject, we aim for equal representation across cate- gories to the extent possible.
Human Performance To estimate expert-level performance EMMA-mini we hire two human experts per subject and report their average score. This score serves as a baseline
5

---

Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
Table 2: Evaluation results of state-of-the-art MLLMs, which are outperformed by human experts with wide margins. The highest and second- second-highest highest scores are highlighted in green and blue. respectively. 1 indicates CoT improvements, while L denotes reductions.
EMMA
EMMA-mini
CoT
Math
Phys
Chem.
Coding (564)
Overall
Math
Phys.
Chem.
Coding
Overall
(892)
(156)
(1.176)
(2.788)
(100)
(100)
(100)
(100)
(400)
Random choice Human Expert
14.01
25.64
16.50
25.71
18.08
13.00
23.00
27.00
28,00
22.75
75,00
64.50
86.00
85,50
77.75
Claude 3.5 Sonnet
x
25.34
33,97
40,90
38.65
35,08
23.00
34,00
44,00
35.00
34,00
Gemini 2.0 Flasb
X
23.88
38.46
36.31
42.02
33.61
20.00
40.00
36,00
41.00
34,25
GPT-4o
27.24
38.46
31.89
40.07
32.42
30.00
38.00
33.00
40.00
35.25
Qwen2 VL-
X
13.07
42.31
32.06
34.57
33.46
38.00
40.00
34,00
37.00
37.25
LLaVA Onevision 72B
x
27.69
35.90
25.26
28.72
27.33
25,00
32.00
24.00
28.00
27.25
25
24.06
27.84
25.07
31.00
22.00
21.00
28,00
25.50
VL2.57L2.5778B
31.39
38.46
35,20
31.91
33,50
30.00
40.00
38.00
33,00
35.25
Claude 3.5 Sonnet
29.37
41,03
41,07
40.60
37.23
2.15)
30.00
38.00
41.00
39.00
37.00 ( 3.00)
Gemini 2.0 Flasb
25.90
38.46
24.66
40.96
29.12 (L 4.48)
24.00
41.00
36.00
44.00
36.25 () 2.00)
GPT-4o
25.56
43.59
33.67
39.01
32.71 <1 0.29)
27.00
44.00
35.00
38.00
36,00 ( 0.75)
Bwve22-VL-728-Inse
27.69
34.62
24.57
29.43
27.12 (1 6.35)
35,00
34,00
32,00
23,00
31. 11.00 6.25)
LLaVA. Onevision 72B Intem-V.2--llmm25776
22.42
1.538
22,70
30.67
23.82 (1 3.52)
23,00
26.00
23.00
29.00
25. 25 () 2.00)
22.20
32.05
19.73
30.32
23.35
1.72)
27.00
33.00
21.00
32,00
28.25 (1 2.75)
VLtem\L2..5788B
25.56
39.74
27,47
25.18
27.08
6.42)
31.00
36.00
24.00
19.00
27,50 () 7.75)
Gemini Flash Thinking. 1219
31.61
$6.41
"37,93
43.44
38.06
35,00
$7.00
41.00
41.00
43.30
Gemini 2.0 Flash Thinking-0121
37.11
60.26
41.58
48.05
42,50
34.00
63.00
47.00
48.00
48.00
QVQ-72B-Preview
34.00
39.00
24.00
31,00
32.00
o1
41.00
49.00
40.00
53,00
45.75
contextualizing model performance.
leading human experts to achieve a score of 64.5%. For other subjects, however, the best-performing models lag significantly behind human experts, with gaps of 34 34%, 39%, and 32.5% % in math, chemistry, and coding, respec- tively. These results underscore the limitations of current MLLMs in addressing complex multimodal reasoning tasks.
Models We evaluate ten state-of-the-art MILMs under the zero-shot setting, including five open-source mod- els (Qwen2-VL. (72B) (Wang et al., 2024b), QVQ-72B- Preview (Alibaba), 2024a), AntemVL22 (76B) (Team, c), and VL255 (78B) (Chen et al., 2024b)) and five proprietary ones (GPT- 4o (OpenAI, a), Claude 3.5 Sonnet (Anthropic), Gem- ini 2.0 Flash (Deepmind, a), Gemini 2.0 Flash Thinking (Deepmind, b). and ol (OpenAI, b)). Due to rate limits, we report 0l and QVQ performance on EMMA-mini only. All other models are evaluated on the entire benchmark.
(72B) (Li et al.,
On the full EMMA benchmark, closed-source models gen- erally outperform open-source ones, particularly with CoT prompting. Across all subjects, Qwen2-VL-72B--Ne -----L--LLB---rree is the only open- model to place in the top two for any subject. Gemini 2.0 Flash Thinking-012 scores best on ev- ery subject (note that ol is not evaluated on the full set due to rate limits). In particular, it performs exceptionally well in physics, lcading by 16.67% over the second-best model (excluding Gemini 2.0 Flash Thinking- (221),,g--119),, GPT-4o. Cod- ing is another area where it excels, leading the second-best model (excluding Gemini 2.0 Flash Thinking- Gemini 2.0 Flash, by 6.03%. Notably, on EMMA-mini, ol outper- forms Gemini 2.0 Flash Thinking 0112 by 5% in coding, suggesting even stronger coding capabilities. In sum, these results highlight the advantages of optimizing models for reasoning by training them to generate thought processes over traditional MLLM paradigms.
Prompting Strategies For all models except ol, QVQ, and Gemini 2.0 Flash Thinking, we test two prompting strate- gies: (1) Direct prompting, which instructs models to out- put the answers without reasoning steps; and (2) Chain- of-Thought (CoT) prompting (Wei et al., 2022), where we prompt models to "think step-by-step".
4.2. Main Results
Are MILMS Multimodal Reasoners? Table 2 demon- strates that all models perform suboptimally across the subjects in EMMA. On EMMA-mini, the best-perg model, Gemini 2.0 Flash Thinking-0121, achieves an ac- curacy of 48.0K 33.000%, trailing human experts by 29.75%. At the lower end, LLaVA-One Vaee---neiiiion-22 scores only 25.25%, barely surpassing random choice by 2.5%. Drilling down into subjects, the best models show the smallest gap with hu- man performance on physics, with Gemini 2.0 Flash Think- ing scoring 1.5% lower than human experts. This smaller gap may reflect the inherent difficulty of physics problems,
Does CoT help? We observe divergent tendencies in the effectiveness of CoT prompting across closed- and open- source models. We exclude ol, QVQ, and Gemini 2.0 Flash Thinking from this analysis, as they inherently generate CoT as part of their responses. Under direct prompting, accura- cies achieved by the best open-source models are well within 2% of Claude 3.5 Sonnet (the best closed-source model). However, the gap widens significantly under CoT prompt- ing, with the best open-source model underperforming by al-
6

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Table 3: Results of different test-time scaling strategies on EMMA-mini, including (1) majority voting. (2) best-of-N, (3) tournament- style best-of-N, Pass @N N accuracies are included as upper bounds of the scaling performance.
Model
Method
Reward Model
N=1
N=2
N
N=8
N=16
Majority Voting
37.25
36.25
38.25
BoN
GPT-4o (Self)
35.50
35.75
36.75
GPT-40
BoN
Gemini Flash Thinking
36,00
40.75
36.25
36.5
Tournament
Gemini Flash Thinking
40.75
39.25
41.25
35.25
Pass@N
45,00
53,25
65.75
74.00
Majority Voting
37.75
39.25
39.75
BoN
Gemini Flash (Self)
38.25 25
36.50
36.00
Gemini 2.0 Flash
BoN
Gemini Flash Thinking
36,25
75
37.00
40,25
Tournament
Gemini Flash Thinking
36,75
37.25
40.75
38.75
Pass N
45.25
56,25
64,50
75.00
Majority Voting
48.00
49.00
50.75
Gemini 2.0 Flash Thinking
Tournament
Gemini Flash Thinking (Self)
43.50
45,50
47.25
47.25
48,00
Pass@N
53.75
64.50
71:50
81.50
ol
45.75
most 10%. Comparing each model's performance with and without CoT EMMA and EMMA--ini, CoT prompt- ing generally improves performance for closed-source models and reduces performance for open-source mod- els. Notably, Qwen2- WL--VLL-I-In-tu and InternVL.2.5. 78B, the top two open-source models overall under direct prompting, suffer decreases of over 6% in accuracies on both EMMA and EMMA-mini. While some tasks might not benefit significantly from textual CoT, we hypothesize that this divergence arises because open-source models fail to fully leverage the potential of language to assist in mul- timodal reasoning tasks where language could be helpful. We elaborate on this hypothesis in detail in Section 5.
use the best-performing reward model identified in the Best- of-N experiments, which is Gemini 2.0 Flash Thinking.
Does test-time compute scaling help? Table 3 presents results of test-time compute scaling methods on top of GPT- 4o, Gemini 2.0 Flash, and Gemini 2.0 Flash Thinking- 1219. Overall, test-time compute scaling improves model per- formance, but it fails to close the gap to human expert performance. The highest accuracy improvements 5.25% for GPT-4o, 4.5% for Gemini 2.0 Flash, and 7.5% for Gemini 2.0 Flash Thinking. Notably, without test-time scal- ing (N=1), Gemini 2.0 Flash Mhinking-12197 accuracy is 2.25% lower than that of ol, but overtakes ol by 5% with majority voting N=16. Nonetheless, its best performance still lags human performance by 27%.
4.3. Results on Test-Time Compute Scaling
We also observe distinct patterns in test-time compute scal- ing performance across different models. While scaling beyond N=8 for GPT-4o and Gemini 2.0 Flash leads to performance degradation, Gemini 2.0 Flash Thinking con- tinues to benefit incrementally from additional test-time compute, at least up N=16. In fact, stronger base models also achieve higher Pass@N N accuracy: Gemini 2.0 Flash Thinking's Pass N consistently surpasses those of the other two models by around 7%, suggesting that stronger base reasoner is more likely to cover the correct response when given multiple attempts. In sum, these results suggest that using a stronger model as the base model raises the upper bound for test-time scaling.
We test three test-time compute scaling methods (Snell et al., 2024; Yang et al., 2024b; OpenAI, b) on EMMA-mini: ma- jority voting, Best-of-N selection, and Tournament selection. Both Best-of-N and Tournament selection require a reward model to select the best response among multiple candidates. We use CoT prompting to generate the candidate responses, so that the reward model has enough context to score the re- sponses. For each test-time scaling method, we experiment with N = 1, 2, 4, 8, and 16, as long as the context length of the reward model allows.
Majority voting selects the most frequent response among batches of N candidate responses, breaking ties by randomly choosing from the most frequent answers.
Majority Voting:
Comparing scaling strategies for each model, we find that GPT-4o and Gemini 2.0 Flash achieve their greatest im- provements when Gemini 2.0 Flash Thinking is used the reward model. Additionally, tournament selection consis- tently outperforms Best-of-N (BoN) selection. These results suggest that using a stronger model as the reward model enables weaker models to achieve better results, particularly when the reward model can make fine-grained decisions involving a couple candidate responses each time. This is intuitive, evaluating responses also requires reasoning.
ofst-of-N: Best-of-N selection (Cobbe et al., 2021; Light- man et al., 2023) chooses the fighest-scoring response (Gu et al., 2024) according to reward model. We explore two configurations: using the base model itself or stronger reasoning model (e.g.. Gemini 2.0 Flash Thinking) as the reward model.
Tournament Selection: In Tournament Selection (Son et al., 2024; OpenAl, c), responses compete in matches, with win- ners advancing through rounds until the final selection. We
7

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
7.55%
Error Type
Physics: 3D Field Simulation
Perceptual Error
Visual Reasaning Emrr
Text Reasoning Eiror
Question: long bat magnet is placed above a current loop oriented shown. In which direction the North pole the magnet feel force to the current loop?
Lock Knowledge
30.19%
10
b-x ey d.-y The har magnet will feel no force due to the current loup.
52.6396
+ A current loop behaves like small bar magnet
whose magnetic-dipole moment points according to the Right- Hand Rule.
From the figure, the current goes around the loop in such way that you curl your right-hand (ingers in the direction
9.43%
+2
current, your thumb points up (the- = direction). Consequently, the loop acts like a magnet whose "north" side on top (at yl. Hence the force the north pole -y.
Figure 5: Distribution of error types made by ol on the math and coding portions of EMMA-mini. The majority of errors arise in visual reasoning.
Error Analysis:
The occurred because limited visual spatial imagination, leading the wrong conclusion about the direction of the thumb when applying the Right- Hand Rule. Additionally the spatial nature the magnetic field is not properly simulated, leading to faulty reasoning of the 'north' polc's position.
On the other hand, we find that self-reward modeling often underperforms. Even using Gemini 2.0 Flash Thinking for self-reward modeling yields performance consistently below that of majority voting. We conjecture that self-reward mod- eling may be less effective because the model's evaluation criteria may be disrupted by its own generation patterns, making it less sensitive differences in the reasoning of the generated responses than an independent reward model.
Human Reasoning (Ground Truth):
curren loop will produre a magnetic according to the Right Hand Rule as shown below.
B
I
Then, as shown on the left, B points out from the middle of the current direction at location where the har magnet located. This magnetic field, pointing to the right, will have the effect of repelling the North pole direction. The correct answer is a.
directed in the
5. Error Analysis
magnet, forcing the bar magnet to right, the
Error Distribution We present an analysis of the errors made by ol on the math and coding portions of EMMA--mni. In total, ol incorrectly answers 59 math questions and 47 coding questions. Figure 5 categorizes these emrors into four types. Perceptual errors, such as misinterpreting visual information, account for 30. 19% of all errors. Lack of knowledge errors, including mistakes related to API usage, contribute 7.55%. Visual reasoning errors, such as failures to simulate 3D processes, constitute the largest category at 52.83%. Finally, textual reasoning errors, including calculation mistakes logical missteps, represent 9.43%. The predominance of visual reasoning errors underscores the limitations of current models in addressing complex visual reasoning tasks. Figure 6 illustrates a representative case: while ol correctly identifies that the problem calls for the application of the Right-Hau Hand Rule, it fails to simulate where the thumb would point when the right-hand fingers curled in the direction of the current. We provide more error cases in Appendix D.
Figure 6: A representative example of visual reasoning error from ol. ol misapplies the Right-Hand Rule due to limited visual spatial simulation skills.
models, it makes all but one open-source model perform worse. In fact, we notice that CoT prompting introduces more hallucinations for open-source models.
2D Transformation questions primarily test for visual sim- ulation and spatial imagination, which are difficult ver- balize. The question in Figure 8, for example. calls for spatial imagination beyond the power of language. In con- trast, Multi- Hop Object Counting can leverage language to describe the relative positions of objects. Hence, we con- jecture that visual-centric tasks, such as 2D Transformation, are poorly suited for textual CoT, whereas tasks that benefit from text- bastd reasoning, such as Multi- Multi-Hop Object Count- ing, allow models to achieve greater performance gains with textual CoT, as evidenced by the closed-source models.
The Effects of Textual CoT Table 2 reveals notable discrepancy: while CoT prompting improves reasoning in closed-source MLLMs, tends to hurt performance open- source models. Although the cause remains unclear without access training data or methods, we analyze error rate per skill category propose conjecture. Figure 7 compares accuracy differences between CoT and Direct prompting for six models on two math tasks: 2D Transformation and Multi- Hop Object Counting. On 2D Transformation, all but one model shows reduced performance with CoT; on Multi-Hop HHop Object Counting, while CoT generally helps closed-source
6. Conclusion
We contribute EMMA, an Enhanced MultiModal reAson- ing benchmark. EMMA features multimodal questions that cannot be solved by independently reasoning within each modality. Evaluation of ten MLLMs reveals sub- stantial performance gap compared to human experts on EMMA, with techniques such as Chain-of- -hhouuhht prompt- ing and test-time compute scaling offering only marginal gains. EMMA highlights the need for new architectures and training paradigms that can better integrate and rea-
8

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
20 Tranaformation
Muld Hop Counding
Impact Statement
Acc. Difference (CaT- Direct) (40
Closed Source Open Source
This work addresses the challenge of evaluating multimodal reasoning in Al systems by introducing EMMA, bench- mark designed to assess models' ability to integrate textual and visual information. By curating diverse set of difficult problems, EMMA provides a rigorous evaluation framework for understanding model performance in multimodal tasks.
5-
Caunte Simees
Gumin Plaat
CPP744
fammitz
Learia Chrvetee
dieute 2.3 Somen
Geman 2.0 Flash
Irammit..
Lavt Oveninn
The societal implications of this work are significant. Im- proved multimodal reasoning could enhance Al applications in science, engineering. and education, enabling more effec- tive problem-solving and decision-making, However, our findings highlight persistent gaps in current models' abil- ity to perform fine-grained visual reasoning, limiting their reliability in high-stakes applications.
Figure 7: Performance differences w/ and w/o CoT for two
math tasks: 2D Transformation and Mup Object Counting.
Math: 2D Transformation
While our curation process effectively filters out shallow multimodal reasoning, manual selection may introduce unin- tended biases in question formulation and domain represen- tation. Additionally, as Al systems improve multimodal reasoning, there is potential for misuse in generating decep- tive or misleading visual content, raising ethical concerns. Addressing these challenges requires continued research into more diverse and automated benchmark construction methods, as well as responsible Al deployment.
(A)
(B)
(C)
(D)
(E)
Question: Rebecca folds a square piece of paper twice. Then she cuts off one the diagram. hen she unfolds the paper. What could the
as you
paper look like now?
GPT-qo with Coll: To solve this problem, we need to carefully analyze the folding and cutting unfolded. v/ Step Understand the folding .... Step a: Understand the cutting process
and determine the resulting pattern when the paper
By exposing critical weaknesses in state-of oftt-----heearrrr models, EMMA sets new standard for evaluating multimodal rea- soning, driving future research toward more advanced AI systems.
Rebecca cuts
folded paper. v/ Since the paper is folded into a smaller square, this ('comer'' is actually a corner the folded square, which corresponds multiple corners the original square.
corner
Final (boeed:
Acknowledgements
Error Analysis: After folding into 1/4, the 'corners' of the folded paper are not the original corners. The solution fails to properly simulate the unfolding process
We sincerely thank the human experts from Tsinghua Uni- versity, Peking University, Fudan University, Nanjing Uni- versity, University of Electronic Science and Technology of China, Sun Yat-sen University. Northwestern Polytechnical University, and University of Science and Technology of China for their valuable support and professional feedback throughout our work.
Direct GPT-40 B.
Human Reasoning (Ground Truth):
Figure 8: An example where textual CoT is unhelpful for solving the problem. While GPT-4o correctly solves the problem without CoT, it answers incorrectly with COT. The thought process demonstrates a superficial association with spatial manipulations and relations rather than genuine visual reasoning.
References
Rdkit: A software suite for cheminformatics, computa- tional chemistry, and predictive modeling. Greg Lan-
drwwa2024plo2coodem, 8331,10):5211, 2013.
Academy, K.
https:/ h/op:///ww.. khanacademy wwwpaaccddenn.oog//eeeeoee wrg/science/..
Khan academy.
son over diverse modalities. Like any benchmark, EMMA has its limitations, which can be improved in future works. Future iterations could also enrich the currently underrepre- sented physics section or expand the chemistry section to incorporate a broader range of chemistry topics. Nonethe- less, EMMA sets a new standard for assessing MLLMs on multimodal reasoning.
Achiam, J., Adler, S., Agarwal, S., Ahmad, I.,, Akkaya, I.,
Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Akter, S. N., Lee, S., Chang, Y., Bisk, Y., and Nyberg,

---

Can MLLMS Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Visreas: Complex visual reasoning with unanswerable questions. arXiv preprint arXiv:2403. 2024.
google--emmniin hagple-gee------------eedeee-bee 2024/
/,
Alibaba.
To see the world with wis- dom. https: /twp::///ww........ http///ww......ht..... wwt.iiggittt....e.eeeee io/blog/ qvq 72b preview/.
Qvq:
Deepmind, G. Gemini 2.0 flash thinking mode.
ottps://all httss://aa..oogle..o. detpleeevvvg/eeiiieeeeee docs/thinnknq--mdee b.
Anthropic.
Claude 3.5 sonnet. https:/ /www. anthropic. www/ news/ "laude-3-5-somee.
DeepSeek, T. Deepseck-rl-ltt-prevvie is now live:
unleashing supercharged reasoning power! https: //api-docs deepseek. ccmmnewss http://www//wwwwlee1 nnwws122 2024.
Antol, S., Agrawal, A., Lu, J. Mitchell, M., Batra, D., Zitmick, C. L.. and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425 2433,, 2015.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The llama herd of models. arXiv preprint arXiv:2407.22 27787 2024.
Bai, J., Bai, S., Chu, Y., Cui, 2., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Fu, D., Guo, R., Khalighinejad, G., Liu, O., Dhingra, B., Yogatama, D., Jia, R., and Neiswanger, W. Isobench: Benchmarking multimodal foundation models on isomor- phic representations. arXiv preprint #rXiv:2404.01266 2024.
Chen, N., Zhang, Y., Xu, J., Ren, K., and Yang, Y. Viseval: A benchmark for data visualization in the era of large language models. IEEE Transactions on Visualization and Computer Graphics, 2024a.
Goyal, Y., Khot, T., Summers-Stay. D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
Chen, Y.C Yu, L., El Kholy, A., Ahmed, F., Gan, Z.,, Cheng, Y., and Liu, J. Uniter: Universal image-text repre- sentation learning. In European conference on computer vision, pp. 104-120. Springer, 2020.
2017.
Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, F.,, Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271,, 2024b.
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. A survey on Ilm-as-a- judge. arXiv preprint 2rXiv:2411.1533
2024.
Han, Y., Zhang, C., Chen, X., Yang, X., Wang, Z., Yu, G., Fu, B., and Zhang, H. Chartllama: A multimodal Ilm for chart understanding and generation. arXiv preprint arXiv:2311 NttT:2311.1648.. 164483 2023.
Cheng, 7.,, Chen, Q., Zhang, J., Fei, H., Feng, X., Che, W., Li, M., and Qin, L. Comt: novel benchmark for chain of multi-modal thought on large vision- vision-language models. arXiv preprint arXiv:2412.12932 2024.
He, C., Luo, R., Bai, Y., Hu, S., Thai, Z. L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems. arXiv preprint 24x::2402.140000 2024.
Chollet, F. On the measure of intelligence. arXiv preprint 1XXiv:911.001477 2019.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint 10. 14168. 2021.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03380 2020a.
codeforces. com/.
Codeforces.
https: /ttp:////ooo....c////////o/oceee
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.00 03000, 2020b.
Das, R. J., Hristov, S. E., Li, H., Dimitrov, D. I., Koychev, I., and Nakov, P. Exams-v: A multi distiiiisiipiie multilingual multimodal exam benchmark for evaluating vision lan- guage models. arXiv preprint 1.XX::2403.10376 2024.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv 2021.
Deepmind, G. Introducing gemini 2.0: our new ai model for the agentic era. https: https://blog, www.eechhwoooggogggogeeeeeee
preprint
10

---

Can MLLMs Reason in Multimodality? EMMA; An Enhanced MultiModal ReAsoning Benchmark
Hu, L., Wang, D., Pan, Y., Yu, J., Shao, Y., Feng, C., and Nie, L.. Novachart: A large-scale dataset towards chart derstanding and generation of multimodal large language models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 3917-3925, 2024.
Liu, E,, Eisenschlos, M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., and Altun, Y. Deplot: One-shot visual language reasoning by plot- to-table translation. arXiv preprint arXiv:2212.10505, 2022.
Jin, Z., Chen, Y., Leeb, F., Gresele, I.., Kamal, O., Zhiheng, L., Blin, K., Adauto, F. G., Bleiman-Weineed M., Sachan, M., al. Cladder: Assessing causal reasoning in lan- guage models. In Thirty-eevnnth conference on neural information processing systems, 2023.
Liu, H.,, Li, Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tun- ing. Advances neural information processing systems, 36, 2024b.
Kwon, W. Li, Z., Zhuang, S., Sheng. Y., Zheng, L., Yu, C. H., Gonzalez, E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z.., Yang, H., et al. Deepseek-vl: towards real-world vision-language understanding, arXiv preprint arXiv: 2403052003. 2024a.
Leung, C. T., Chen, Y., and Gao, H. Smicrm: A benchmark dataset of mechanistic molecular images. arXiv preprint arXiv:2407. 18338, 2024.
Lu, J., Batra, D., Parikh, D., and Lec, S. Vilbert: Pre- training task-agnostic visiolinguistic representations for rision-and-language tasks. Advances in neural informa- tion processing systems, 32, 2019.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang. H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03336, 2024a.
Lu, P., Mishra, S., Xia, T., Qiu, L.., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Infor- mation Processing Systems, 35:2507-2521, 2022.
Li, J. Name reactions. collection of detailed mechanisms and synthetic applications 4th edition, 2009.
Li, Li, D., Savarese, S., and Hoi, S. 21ip-2: Bootstrapping language-imann image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19788 19742. PMLR, 2023.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng, H., Chang, K.- Galley, M., and Gao, J. Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024b.
Li, K., Tian, Y., Hu, Q., Luo, Z., and Ma, Mmcode: Evaluating multi-modal code large language models with visually rich programming problems. arXiv preprint 2XXv:2404.094866 2024b.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requir- ing external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Li, X., Yin, X., Li, C.,, Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. Oscar: Object- semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 121-137. Springer, 2020.
Masry, A., Long, D. X., Tan, Q., Joty, S., and Hoque, E. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 20X:::2003.0024,, 2022.
Methani, N., Ganguly, P., Khapra, M. M., and Kumar, P Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1527-1536, 2020.
Z., Yang, X., Choi, K., Zhu, W., Hsieh, R., Kim, H., Lim, H., Ji, S., Lee, B., Yan, X., et al. Mmsci: multimodal multi-discipline dataset for ped-level scientific compre- hension. In for Accelerated Materials Design-Vienna Vienna 2024, 2024c.
OpenAI. Hello gpt-4o.
index/hello-gpt- https: / http//ppena.ooommmmme
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20055 2023.
OpenAL. 1lms.
Learning to reason with
https://opent wwtppenai.coomiiieoo com/index//

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
OpenAl. Introducing chatgpt pro. https://openai. http://ndee///tcrooooonoooohhhe
Tan, H. and Bansal, M. Lxmert: Learning cross-modality
encoder representations from transformers. arXiv preprint arXiv:1906 1908.07499, 2019.
Physics, L. A. Learn ap physics. https: /www. Learnaphhysics.com
Team, G. G. Gemini 1.5: Unlocking multimodal under-
standing across millions of tokens of context (2024). URL https://oo,. gle/GeminiVI-5,
Qwen, T. Qwq: Reflect deeply on the boundaries of the unknown. https:/ /tw.:///w/nlllle m. github. oo/blog/ qwq- See-33bbprreiie.. 2024.
Team, M. Examples. https: https://matplotlitt http://aappltttlbboooo stable/gallerry/ddeeee hhml1,b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Team, O. Intern112: Better than the best-expanding performance boundaries of open-source multi- modal models with the progressive scaling strat- egy. https://internv/ githuu....g............... hwtpituu...bbbl......ee io/blog/ 2024-07-02-InternvV--
Ramakrishnan, S. K., Wijmans, E., Kraehenbuehl, P., and Koltun, V. Does spatial cognition emerge in frontier models? arXiv preprint arXiv:2410.06-466 2024.
Wang, K., Pan, J., Shi, Wu, Zhan, M., and Li, H. Mea- suring multimodal mathematical reasoning with math- vision dataset, 2024a.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022 2023.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: vl: Enhancing Dision-languas model's perception of the world at any resolution. arXiv preprint 2rX9v:2409.122 /2191, 2024b.
Shao, H., Qian, S., Xiao, H., Song, G., Zong. Z., Wang, L., Liu, Y., and Li, H. Visual cot: Unleashing chain- of-thought reasoning in multi-modal language models. arXiv preprint arXiv:2403. S00.:::4003669999 /6999, 2024.
Wang. Z., Xia, M., Hc, L., Chen, H., Liu, Y., Zhu, R., Liang, K., Wu, X., Liu, H., Malladi, S., et al. Charxiv: Charting gaps in realistic chart understanding in multimodal Ilms. arXiv preprint arXiv:2406. 18521, 2024c.
Shi, C., Yang, C., Liu, Y., Shui, B., Wang, J., Jing, M., Xu, L., Zhu, X., Li, S., Zhang, Y., et al. Chartmimic: Evalu- ating Imm's cross-modal reasoning capability via chart- to-code generation. arXiv preprint wxXiv:2406.09961, 2024.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-though prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling Ilm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.03314, 2024.
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer. S., von Platen, P., Ma, C., Jer- nite, Y., Plu, J., Xu, C., Scao, T. L.., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Compu- tational Linguistics. URL attps://www.aclweb. aclweb. org/ hnt.oootyot//////oo/o /tcpigyy/222// 2020 emnlp-demos. 6.
Son, S., Oh, J.-M., Jin, H., Jang, C., Jeong, J.,. and Kim, K. Varco arena: A tournament approach reference-free benchmarking large language models. arXiv preprint arXiv:2411.01281, 2024.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonse A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0466 2022.
Wu, C., Ge, Y., Guo, Q., Wang, J., Liang, Z., Lu, Z., Shan, Y., and Luo, P. Plot2code: A comprehensive bench- mark for evaluating multi-modal large language models in code generation from scientific plots. arXiv preprint arXiv:2405.0799 2024.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Lc, Q. V., Chi, E. H., Zhou, D., et al. Challenging big-bench tasks and whether chain- of thought can solve them. arXiv preprint 2XXiv2220000966 2022.
Xia, R., Zhang, B., Ye, H., Yan, X., Liu, Q., Zhou, H., Chen, Z., Dou, M., Shi, B., Yan, J., et al. Chartx & chartylm:
12

---

Can MLLMS Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
A versatile benchmark and foundation model for compli- cated chart reasoning. arXiv preprint 1rX 2024.
IEEE/CVF conference on computer vision and pattern recognition, pp- 1p-5557-5555 2021.
Zhang. R., Jiang, D., Zhang, Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.-W., Gao, P., al. Mathverse: Does your multi-modal IIm truly see the diagrams in visual math problems? arXiv preprint arXiv:2403. 144224403.14644, 14624, 2024a.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint 2mXhv:2407.16677 2024a.
Yang, A., Zhang. B., Hui, B., Gao, B., Yu, B., Li, C.,, Liu, D., Tu, J., Zhou, J., Lin, J., al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement, arXiv preprint ar arXiv:2409.122 12122, 2024b.
Zhang, Z.., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. Multimodal chain-of-thouge reasoning in lan- guage models. arXiv preprint arXiv:2302.00923. 2023.
Yang, J., Jimenez, C. E., Zhang, A. L., Lieret, K., Yang, J.. Wu, X., Press, O., Muennighoff, N., Synnaeve, G., Narasimhan, K. R., Yang, D., Wang, S. I,, and Press, O. Swe-bench multimodal: Do ai systems generalize to visual software domains?, 2024c. URL https:// arxiv. org/ abs/ /21///211../00 03859.
Zhang, Z., Ma, W., and Vosoughi, S. Is gpt-4v (ision) all you need for automating academic data visualization? explor- ing vision- vision-language models' capability in reproducing academic charts. In Findings of the Association for Com- putational Linguistics: EMNLP 2024, pp. 2024b.
Ying, K., Meng, F., Wang. J., Li, Z., Lin, H., Yang, Y., Zhang, H.. Zhang, W.. Lin, Y.. Liu, S., et al. Mmt- bench: A comprehensive multimodal benchmark for eval- uating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.10066, 2024.
Zhao, Y., Yin, H.,, Zeng, B., Wang, H., Shi, T., Lyu, C., Wang, L., Luo, W., and Zhang, K. Marco-ol: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:2411 14405, 2024.
Zhou, Q.. Zhou, R., Hu. Z., Lu, P., Gao, S., and Zhang, Y. image-of-thougee prompting for visual reasoning re- finement in multimodal large language models. arXiv preprint arXiv:2405. 1387. 2024.
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models. arXiv preprint arXiv:2205.019/7, 2022.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision- laiionl understanding with advanced large language models. arXiv preprint arXiv: arXiv:2304.1099, 2023.
Yu, W., Yang, Z., Li, L.,, Wang, J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities. arXiv preprint arXiv:2308. 02490, 2023.
Yuc, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang. W. Sun, H., Su, Y., and Chen, W. Mmmu: A massive multi-disciplin multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024a.
Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong. S., Sun, Y., Yu, B., Zhang, G., Sun, H., Su. Y., Chen, W., and Neubig, G. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. arXiv preprint AXii:2449.00281:. 2024b.
Zhang, C., Gao, F., Jia, B., Zhu, Y., and Zhu, S.-C. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5317-5321 1317-5327,22 2019.
Zhang, P., Li, X., Hu, X., Yang, J., Zhang, L., Wang. L., Choi, Y., and Gao, J. Vinvl: Revisiting visual representa- tions in vasion-language models. In Proceedings of the
13

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
A. Overview of the Appendix
identifying how shapes colors evolve across series of diagrams and predicting the next pattern in the sequence. Solving such problems draws on the ability to recognize visual regularities, which are challenging to describe accu- rately using text alone, necessitating strong visual reasoning. Typical examples are provided in Figure 12.
This Appendix organized as follows:
Section B contains details about the composition of EMMA, the data curation process, and comparison with existing benchmarks;
Section contains experimental details, including the prompts used, models tested, hyperparameter settings, and the breakdown results on different categories;
Rotation
D
Translation
D>
Flip
Image:
Image:
Image:
Section D contains additional case studies for each subject.
Question: Which of the rectangles E be covered by patten right-lund side such a way that the result a totally black rectungle?
Question Ma Holer has down
Question: Which of the following pictures you NOT
pietten of flowars the incie window What
B. EMMA Details
the pieces beside?
look like look
B.1. Composition of EMMA
pistrr
Figure Three main types of questions belonging to the 2D Transformation category in math: Rotation, Translation, and Flip- ping.
EMMA comprises 2,788 questions across four subjects: math, physics, chemistry, and coding. We now provide detailed breakdown of EMMA by subject:
Path Tracing
D>
Change of View Simulation
Math The math portion of EMMA consists of 892 ques- tions, of which 562 are multiple-choie questions and 330 are free-form questions. These questions can be categorized into five areas: 2D Transformation (266 questions); 3D Spatial Simulation (275 questions); Path Tracing/Change of View simulation (127 questions); Multi-Hop Visual Object Counting (124 questions); and Pattern Inference (100 questions).
Image:
Image:
<Imagelin
Question: The sided mirros reflect the lnser beams shows <image 1> At which letter does the laser boun leave the picture: <image2>>
Question: how many places the picture are childion holding each other with their left honds?
small picture:
Two typical examples of the Path Tracing/ Change of View Simulation category in math.
Figure 10:
Tasks in the 2D Transformation category often involve operations such as rotating, translating, or flipping shapes. Examples are provided in Figure 9. Humans typically solve these problems by leveraging their ability to "see" and men- tally manipulate objects, simulating spatial transformations to arrive at solution. During data filtering, we observe that models also rely heavily on visual information to solve these problems, as they often fail when provided only with textual descriptions the accompanying images. Similarly, prob- lems the 3D Spatial Simulation category require simi- lar visual reasoning approach, but with the key difference that the simulation must be performed in three-dimensional space. The Path
Question: Subtract sell large rubber spheres. Subtract shiny cylinders. How many objects are left!
Question: Subtecl all red things, Subtract all mame bans. How many objects are left?
of View Simulation cat-
egory involves solving problems akin to maze navigation, where the task requires tracing a path from a starting point to an endpoint while considering changes in perspective. We present two typical examples Figure 10. Problems in the Multi-Hop Visual Object Counting category are sampled from Math- Vista (Lu et al., 2024b), with some examples shown in Figure 11. Unlike straightforward object counting, which might ask, "How many objects are present?", these questions require models to identify objects based on their attributes and perform subtraction operations grounded in visual properties. The Pattern Inference category involves
Question: Subtract all brown Blocks. Subtruct all large blue rubbee things. many objects are left?
Subaract all rubber balls. Subtract shiny things. How many objects are lcd?
Question:
Figure 11: Object Counting category math.
Four typical problems from the Multi-Hop Visual
The physics portion of EMMA includes 156 ques- tions divided into five categories: 3D Field Simulation (37 questions), Graph Reasoning (26 questions). Multi-Hop Visual Reasoning (33 questions), Path Tracing (13 ques- tions), and Visual Decomposition Simulation (47 ques-
Physics
14

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
istry because found that this type of problem makes up the majority of the remaining data set filtered. In these newly developed sections, we focus on "arrow-pushing agrams, common representation used to illustrate electron flow in mechanistic steps. The dataset includes structural molecular identifiers of molecular images in chemical reac- tion mechanisms. The questions are categorized into three types in increasing reasoning difficulty: Knowledge-basee Counting, Structure Recognition, and Reaction Simula- tion.
Madh
Source: RAVEN
Image:
Imager:
di-
17
2
Question: Choose the answer for the missing picture.
Question: Chouse the answer for missing pacture.
Figure 12: Questions in the Pattern Inference category for math sourced from two datasets: Math-Vista (Lu et al., 2024b) and RAVEN (Zhang et al., 2019). Two examples from these sources are presented for illustration.
The Knowledge-based Counting category (456 questions) involves counting the number of chemical bonds of a chem- ical structure. The task requires somain--pecificc knowledge and multiple inference steps to accurately count different bonds. Structure Recognition (474 questions) presents more difficult task. It requires correctly identifying the number and type of atoms and chemical bonds, recogniz- ing the molecular structure, and deriving the corresponding chemical expression or SMILES expression. Notably, each molecule in the diagram corresponds to a unique SMILES expression, encapsulating both compositional and structural information.
tions). All physics questions in EMMA are multiple-choice questions.
The Visual Decomposition Simulation category (47 ques- tions) is often related to topics in physics such as Dynamics, Circular Motion, & Gravitation. It typically involves the analysis of forces acting on static or dynamic objects, which requires visual decomposition of forces and visual simula- tion of current and future states. The 3D Field Simulation category (37 questions) addresses topics like Electric Force, Field, and Potential. These tasks emphasize visual simula- tions of properties and phenomena in three dimee-diimniiooal elec- tric and magnetic fields. The Graph Reasoning category (26 questions) involves interpreting and reasoning about physics-related graphs, such as velocity-time (v-() graphs, displacement-timme (s-t) graphs, and trajectory graphs. The Multi-Hop Visual Reasoning category (33 questions) is a mixed category, which includes some circuit analysis di- agrams, as well as various types of problems that require multi-hop thinking. In the Path Tracing category (13 ques- tions), most problems involve light refraction. These tasks require analyzing the paths of particles by tracing their tra- jectories based on the perspectives depicted in the images.
Reaction Simulation (132 questions) is the task requiring the most advanced visual reasoning. It involves inferring the post-reaction
expression based on the molecular composition and structural information available before the reaction, guided by the direction of electron flow indicated by arrows. Due to its complexity and the poor per- formance of models on the simpler open-ended tasks, this task is presented in the form of multiple-choice questions. Reaction Simulation Pro (105 questions) assess reaction simulation skills and visual reasoning with a higher level of expertise. Each question presents options in the form of images, requiring participants to not only simulate reac- tions but also engage in complex multi-step reasoning by comparing different choices. Graph Reasoning (9 ques- tions) involves graph-based reasoning problems related to chemistry knowledge, such as reaction rate changes.
Chemistry The chemistry portion of EMMA includes 1,176 questions. Only 20 of them come from existing bench- marks, the vast majority of 1,156 are created by us. These questions can be categorized into five areas: Knowledge- based Counting (456 questions); Graph Reasoning (9 questions); Structure Recognition (474 questions); Reac- tion Simulation (132 questions); and Reaction Simulation Pro (105 questions; Based on the test Reaction Simulation skill, the requirements for visual reasoning are more profes- sional, where all options in multiple-choice questions are images). Among the chemistry questions, only the answer corresponding to the category Knowledge-baid Counting is free-form, and all the remaining questions are multiple- choice.
Coding Implementing user interfaces (UIs) is fundamen- tal task in software engineering. In this work, we focus critical aspect of UI development: data visualization. Creat- ing data visualizations not only requires working knowledge of the charting toolkits, but also demands reasoning over how various visual elements coordinate to achieve the de- sired results. To stress-test MLLMs' visualization skills, we design four tasks: Visualization Choose Code (188 ques- tions; given an image of visualization, choosing which visualization program generates it), Code Choose Visu- alization (188 questions; given a visualization program, choosing which image it generates), Modify without the Original Image (94 questions; given a target visualiza- tion image and a visualization program that does not yet
EMMA prioritizes chemistry problems that require rich multimodal reasoning over simple fact recall or direct ap- plications. To fill the gaps in current datasets, we manually developed a new test suite concentrating on organic chem-
15

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
generate the target image, choosing what change should be applied to the program create the target image), and Modify with the Original Image (94 questions; given target visualization image, a visualization program that does not yet generate the target image, and the image that the current program generates, choosing what change should be applied to the program to create the target image). The four examples in Figure 15 illustrate each task. All coding questions in EMMA are multiple-choice questions created by us from scratch. Notably, these tasks simulate various real-world applications of MI.Mss requiring skills essen- tial for replicating a target visualization or redesigning an existing one. To ensure familiarity, all visualization code in our benchmark is generated in Python using matplotlib or seaborn.
Based on this observation, we construct the chemistry sec- tion in EMMA from scratch, which features three tasks of increasing difficulty. The original data is sourced from SMi- CRM (Leung et al., 2024). Notably, the correct answers for the Reaction Simulation task-the most challenging and the one that best reflects vision's role in the reasoning process- are constructed and verified by a PhD candidate in chemical molecules. The questions and answers for the Reaction Simulation task are sourced from Li et al.'s collection of chemical reactions (Li).
Coding We manually curate all questions for coding. Our curation process consists of three stages. In the first stage, we identify "seed visualizations" that employ advanced vi- sualization techniques or present rich space for design variations. We source these seed visualizations through three channels: CharXiv (Wang et al., 2024c) (a benchmark consisting of diverse charts extracted from arXiv papers), the official matplotlib example gallery 2, and our prior ex- perience. For each source, we attempt to reproduce visual- izations demonstrating advanced techniques (e.g., 3D bar charts) in Python using GPT-4o or Claude 3.5 Sonnet, retain- ing only those that MLLMs cannot reasonably replicate after multiple iterations of prompting. The first stage ultimately results in 47 seed visualizations.
Further, similar to the other subjects, we provide fine- grained categorizations for each coding question based on the skills it measures. Since visualizations involve multi- ple design choices and each problem includes at least four visualizations (or visualization programs), each question may be assigned to multiple categories. On average, each question is associated with 2.11 categories. Through man- ual coding, we identify total of ten categories: 3D (108 questions; reasoning about visualizations in 3D), Color & Texture (156 questions; reasoning about the colors and tex- tures of marks), Data Reasoning (108 questions; reasoning about the data in visualizations); Advanced Chart Type (276 questions; involving advanced chart types, such as fish- bone diagrams), Alignment, Orientation, & Position (180 questions; reasoning about how visual elements should be arranged), Gridline (60 questions; reasoning about the use of gridlines), Polar Coordinates (48 questions; reasoning about charts in polar coordinates), Axis & Scale (108 ques- tions; reasoning about the use of axes and scales), Legend (96 questions; reasoning about the appearance, content, and position of legends). Marker & Line (48 questions; reason- ing about the style of markers and lines). Section D contains sample questions for some of the categories.
In the second stage, we generate four variations for each seed visualization. When we prompt MLLMs to reconstruct visualizations during the first stage, MLLM- generated vi- sualizations are often ill-formed, nonsensical, or otherwise fail to achieve the desired effects. However, since these visualizations are generated by MLLMs, they may be indis- tinguishable from the correct visualizations to the models. Yet, it is crucial for MILMS recognize such flaws, as early identification of errors is essential for efficient human- Al collaboration. As such, we include such "buggy" code snippets as variations of seed visualizations. In addition, we further enrich the set by introducing design variations (e.g., changes in spine configuration, line style, or axis scaling) ei- ther manually or through prompting MILMMs, with post-hoc manual verification. After the second stage, we are left with 188 visualizations, organized into 47 sets, each containing four visualizations.
B.2. Additional Data Curation Details
We now provide additional details on data curation for chem- istry and coding.
In the third stage, we construct questions using these 188 vi- sualizations. For Vis Choose Code, we iterate through each visualization within a set and construct questions asking models to select the code snippet used to generate the chart. For Code Choose Vis, we iteratively choose a code snippet from each set and ask models to identify the correspond- ing generated image. For Modification, we first introduce another design variation in each set, and then select two
Chemistry We generate ten responses with state-of-the-art LLMs for image-captioned versions of chemistry questions in existing multimodal reasoning datasets (Yue et al., 2024a; Das et al., 2024; Lu et al., 2022). Questions that are an- swered correctly in at least five out of ten rounds are filtered out. Analysis of the remaining questions shows that most in- volve molecular formulas, indicating that molecular-related tasks often require additional visual information for effec- tive reasoning.
Ptttps: / //tp///appoottt/////eee/ottoottt wel.bbborgg............. www./rr//aaaao.. htt//taallee. heabbe/gallle index. html. This approach is inspired by Wu al. (Wu al., 2024).
16

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
pairs of visualizations from the set, where each set contains relatively well-formed chart and another random chart. We construct questions by comparing the code of the ran- domly selected snippet with others, asking what changes are needed to produce the target visualization, While the target visualization image is requisite, we vary whether the origi- nal visualization image is provided. In sum, this procedure generates four Code Choose Vis questions, four Vis Choose Code questions, and four Modification questions (two with the original image and two without) per set, resulting in total of 564 questions evenly divided among the tasks.
math problems in our dataset but are also adaptable to other dataset, such as MMMU (Yue al., 2024a). In figure 14, we present two examples from MMMU that fall under our defined categories. We hope these categories will inspire further exploration of the visual reasoning capability of MLLMs.
Science (Physics & Chemistry) The latest multimodal rea- soning benchmarks in science, such as (Yue et al., 2024a; Das et al., 2024; Lu et al., 2022), do not provide many mul- timodal physics and chemistry problems. In addition, they often focus on superficial visual cues or heavily rely on text- based reasoning. As pointed out by (Fu et al., 2024), text representation can address 90% of physics and chemistry questions in ScienceQA (Lu et al., 2022). As result, our filtering pipeline leaves only 100 problems in total for these subjects from relevant benchmarks, which we expand to 1,332 with our newly constructed problems.
B.3. Comparison with Other Benchmarks
EMMA stands out from existing multimodal benchmarks by emphasizing questions that truly demand multimodal reasoning capabilities. Through meticulous manual label- ing or verification, we provide fine-graing greiigd labels for each question, categorizing them based on the specific skills they assess. This approach enables more detailed analysis of the limitations of MILMs.
Our benchmark places greater emphasis on the role of vi- sion in multimodal reasoning. Beyond filtering out problems solvable through text alone, manually review and anno- tate the remaining questions ensure strong reliance on visual information.
Math Various benchmark datasets (Lu et al., 2024b; Wang et al., 2024a; Yue et al., 2024a;b) have been proposed to evaluate the mathematical reasoning capabilities of MI.LMss. However, existing math benchmarks often emphasize shal- low perceptual cues or rely heavily on text-dominant rea- soning. In contrast, our dataset mainly focuses on assessing the performance of MLLMs on tasks that require integrated reasoning, particularly those that are highly dependent on visual information. Specifically, we employ an enhanced data filtering pipeline to separate questions that could be answered correctly using only the caption of images. Rep- resentative examples of such problems are illustrated in Figure 13. In some cases, the images provide no additional information required to solve the question, and the answer can be derived entirely from the text of the question alone, as shown in the middle example in Figure 13. In other instances, questions can be solved using image captions, leveraging the text reasoning capabilities of MILMs. These images either consist solely of textual information, as il- lustrated in the rightmost example in Figure 13, can be fully described textually without necessitating further rea- soning involving image transformations. For instance, in the leftmost example in Figure 13, as long as the key textual information in the image, such as y = 0.5", is identified, the question can be solved without employing a graphical approach.
In particular, we focus on two specific types of problems. First, some physics problems require visual imagination and simulation of physical processes. Auxiliary images can significantly enhance both the accuracy and efficiency of problem-solvingg Second, in chemistry, tasks such as molecule counting, structure recognition, and reaction sim- ulation demand effective utilization of visual information. These two types of problems and their related data have been largely overlooked in current science datasets. To ad- dress this gap, we emphasize these under-explored aspects in the science portion of EMMA by manually constructing test questions and carefully sourcing them from existing datasets.
Coding Most current visualization-relate benchmarks as- sess visualization understanding (Masry et al., 2022; Xia et al., 2024; Liu et al., 2022; Methani et al., 2020). In this work, we focus evaluating how well MLLMs can reason in multimodality when generating visualizations. To this end, past work (Han et al., 2023; Wu et al., 2024; Shi et al., 2024; Zhang et al., 2024b; Hu et al., 2024) has proposed the task of visualization reproduction- -nadtuctiiongggeratiiing code to reproduce a target visualization. To evaluate the quality of generations, researchers have developed heuristic measures and employed MLLMs as judges.
In addition to the above, provide a fine-grained taxon- omy for math problems. By first categorizing the questions using GPT-4o and conducting expert-level manual verifica- tion of the classifications, we identify categories that are highly likely to require graphic transformation and spatial simulation. These categories are not only applicable to all
EMMA enhances past work by introducing new task types. Our four tasks enable targeted assessments of Vis2Code, Code2Vis, and Visualization Modification. In particular, while many users rely on MLLMs for visualization debug- ging, this task has not been addressed by existing bench-
17

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Q: Is this an odd function? Yes or No
Q:
A trip of the pupils to the took 135 minutes. How many hours and minutes does it make?
Chase wants to buy 4 kilograms of oval beads and 5 kilograms of star- shaped beads. How much will he spend?
oval beads
$2/kllogram
rectangular beads
$3kilogram
(io) - 05
sier ster-shaped beads
$2/kilogram
spherical beads
$2/kllogram
heart-shaped beads
$3/ilogram
square
$2/kilogram
flower-shaped beads
$2/kllogram
C:
The function is = 0.5*...
C: The image is contains a bus...
C:
The image is a price list
C
Q:
No
C+Q:
2 h 15 min
C+
$18
I+Q:
No
I+ Q:
2 h 15 min
I+ Q:
$18
Figure 13: Three typical examples from other math datasets that not really require the images or can be correctly answered using only the captions of images. The leftmost and rightmost examples are from Math- Vista (Lu et al., 2024b) and the middle example is from Math- Vision (Wang et al., 2024a). Q represents the question, C represents the caption of the image, and represents the image.
For math, after filtering the data, conduct detailed observation and analysis of the remaining problems and develop taxonomy consisting of five categories. We then utilize GPT-4o to assist with the categorization and the prompts used during this process are shown in Table 4.
20 mmentmmmttinn
10 Spatil Strmabiton
Image:
Image:
Qeedion The Eager kanns line wil complee infinte mocnting pettom squams and th repnent. Usw many of the Siltoria foxe kinds rapal insius tandortations winch tis figimit rinus otherie trn theideetry bonifomation kill
plaa
into
+imapti
<:magge2::
Quesioe: liow zazz stepstrcc the selaian bead on Gissge disuage
trauditioo dicctio paralll lia the reliciion meree line some reflorbos armss ane perpendruaat line
C.2. Prompts for Response Generation
Two examples from MMMU (Yue et al., 2024a), which belong our defined categories of 2D Transformation and 3D Spatial Simulation respectively. Our classification is equally appli- cable other datasets.
Figure 14:
We evaluate several state-of-the-art MLLMs on EMMA, considering two different prompting strategies: Direct prompting and Shain-of-Thhouhh (Co7) prompting. Further- more, our benchmark features two question types: Open- ended and Multiple Choice. The corresponding prompts vary according to the type of question and the prompting strategy, as shown in Table 5
marks. We further provide fine-grained, expert-generated categorizations for each question based on skills mea- sures (see Section B.I). Finally, all of our coding questions posed as multiple-choice questions, which removes the need for using MLLMs as judges, which can be unreliable.
C.3. Models and Settings
During math data curation, we filter out questions that mod- els can answer when provided only with the image caption and the question. In this process, each model generates ten candidate responses to ensure reliable and effective filtering. expedite response generation, we use the vL.LM (K won et al., 2023) library, an open-source tool for fast L.M in- ference and serving. For all other cases, load models directly using the Transformers (Wolf et al., 2020) library. All model sources are official and listed in Table 6. When evaluating different models, we use default hyperparameter values unless otherwise specified, with detailed parameter settings provided in Table 6.
C. Experimental Details
C.1. Prompts for Data Curation
As previously discussed, we conduct much filtering to cull questions from existing datasets that genuinely require visual reasoning, which involves using GPT-4o to generate captions for images and passing the captions along with textual questions to MILMs to generate responses. The prompts used to generate captions and responses are shown in Table 4. Notably, when generating image captions, we also provide the corresponding questions to models to make sure the captions are as accurate as possible.
18

---

Can MLLMs Reason in Multimodalit? EMMA; An Enhanced MultiModal ReAsoning Benchmark
Table 4: The prompts used to caption images and generate responses during data curation.
Setting
Prompt
Generate Captions
There is question about the image or figure. Please describe the fine- fine-grained content of the image figure based this question, including scenes, objects, relationships, and any present. Please note that you do not need to answer this question directly, just describe the information of this picture.
Generate Responses
Please first solve the problem step by step, then put your final answer or single letter (if it is a multiple choice question) one "[boxed(]" Here is the natural description of the figure, please solve the following problem based on the description.
Categorize
math problems combining images and text, and existing large models cannot correctly reason through these problems. We have analyzed the reasons why large models fail to classify these problems and have categorized them based on the challenges present in the problems. The categories
There
follows: A: To solve the problem, 2D transformation is required, such as translation, rolation, scaling, shearing, reflection, etc. B: To solve the problem, 3D spatial imagination needed. C: To solve the problem, path tracing/change of view simulation is needed, such as math problem about maze. D: None of the above. It belongs another category. Here is a math problem, please give the math category that you think this problem belongs to and explain why. If you choose D, please additionally include the type of math problem you believe to be.
Table The prompts used for evaluation across different question types and prompting strategics.
Type
Strategy
Prompt
{context) (queexttt/xusutoono
CoT
Answer the question using a single word or phrase and put the answer one "i (booedd))"
Please solve the problem step by step.
Open-ended
{context) {question}
Answer the question using a single word or phrase and put the answer in one " boxed Please ensure that your output only contains the final answer without any additional content (such as intermediate reasoning steps).
Direct
{context) {question} {options) Answer with the option's letter from the given choices and put the letter in one "* (boxeed))".
CoT
Please solve the problem step by step.
Multiple
[context) } {question] {options)
Choice
Answer with the option's letter from the given choices and put the letter in one " "bxeed Direct
Please ensure that your output only contains the final answer without any additional content (such as intermediate reasoning steps).
19

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Table 6:
The sources of models used in the experiments and the hyperparameters configuration. Pass@ I refers to scenarios where evaluation is performed only once, while pass refers that require generating multiple candidate responses.
Model
Parameter Setting
Source
URL
GPT-40 pass@1
temperature 0.0
chatgpt-4o-lates
https///plalCoonn.pppaa...co
GPT-4o p
temperature = 0.7
chatgpt-4o-latest
httpss//plattoomm.pppaa...o
Claude 3.5 Sonnet
temperature = 0.0
claude-3-5-sonnet
https://ww.antbzoppccccoo
Gemini 2.0 Flash
temperature = 0.0
gemini-2.0-ffash-exp
https://aa.ggoogl..des
pass@1
Gemini 2.0 Flash
temperature = 0.7
gemmni-2.0-ftsh-exp
https://ai.google..ee
p
Gemini 2.0 Flash
gemini-2.0-flash-
temperature = 0.0
http:://al.google.dem
Whinking-1219
thinking-exp-1219
Gemini 2.0 Flash Thinking-0121
gemini-2.0-flash-
temperature = 0.0
https://ai.google..de/
thinking-exp-0121
OpenAI ol
interface
https://chatggpt.com/
Qwen2-VLL72B--Instuee
temperature 0.7
local checkpoint
htLps///bugginngcacccco/eeee
Owen2-V4-778-Insttuue
QVQ-72B-Preview
temperature 0.7
local checkpoint
https://uuggingfacc..o co/Omenl
QVQ-72B-Preview
do sample=True, Lemperature = 0.7
LLaVA-Onevision-772
local checkpoint
http:://hugghnnffccc.oo//aa
do sample=True, temperature = 0.7
IntemmL22-Llmma3-
local checkpoint
https:///bogiingfccc.../
(pensvVabb/Intee:nV221llaaa
do sample=True, temperature = 0.7
Intern VL2.5-78B
local checkpoint
https://buug.nnggacce.c.// OpenGVlab//LtteenV2222 5-1112255--88
Table Test-time scaling results on math portion of EMMA-mini using Qwen2.55MathhRRM Ma-2--MM-22B. specialized reward model for math, and some generalist models as reward models. Since Owen2..5MMtth-RM-77 docs not take images as input, we provide GPT-4o-generated captions of images in the problems to it. Overall, generalist reward models provide better reward signals than Swen2.5-Math-RM-255
Model
Method
N=1
N=2
N=4
N=8
N=16
BoN w. Self-RM
29.00
27.00
25.00
GPT-4o
BoN w. Gemini 2.0 Flash Thinking as RM
27.00
30.00
28.00
31.00
BoN -ween2.5-Mat-RRM-25
RM
26.00
24.00
25.00
29.00
BoN w. Self-RM
33.00
24.00
27.00
Gemini 2.0 Flash
BoN Gemini 2.0 Flash Thinking as RM BoN -aatnn25--aah-MM----- RM
24.00
33.00
24.00
25.00
27.00
28.00
23,00
23.00
Gemini 2.0 Flash Thinking
35.00
ol
41.00
20

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
A.5
Reoreate this chart in Python.
8.00
C.0
prewert
Code Cetegory: Advanced Chart Type
Category: Alignment,
Position
PAB
Looks good to me!
MLLM Tosted
@ ca yn clange diooa
Heuristics
can cove!
4
Task: Modily without Original Image Category: 30
Task: Modify with Original Image Category: Poler coordinates
EMMA (coding):
Traditional visualization benchmarks that involve coding: Mostly only the visualization reproduction 2) Entail using MLLMs as judges, which may be unreliable
Diverse tasks different abilities
V.S.
2) Multiple-cholce format enables easy assessment Availability categories allows fine-grained analysis
target visualization, with evaluations conducted using a combination of MLLMs as judges and heuristic methods. In contrast, EMMA introduces four visualization relaaed coding tasks designed to assess multimodal coding abilities across multiple dimensions. By employing multiple-choice format, EMMA eliminates the reliance on potentially unreliable MLLM-based judgment. Additionally, our fine-grained categories facilitate detailed analysis of the limitations of multimodal coding skills.
Figure 15: Traditional visualization benchmarks for coding often require MLLMs to
C.4. Best-of-N With a Specialized Math Reward Model
D. Case Study
While we want to evaluate specialized reward models addition to generalist reward models, they are currently only available for math. Owen2.5-MathhRM-222. (Yang et al., 2024b) is one such specialized LLM for evaluating the quality of responses to math problems. For the math portion of EMMA-mini, we first generate 16 responses using each of GPT-4o and Gemini 2.0 Flash. We then score each response using Qwen2.5-Mi 1.5.5--Mthh--BRe Math-RRM-72B and select the highest-scoring answer. test with N values of 4, and 16. Since Qwen2.5-Math- RM-72B is not multimodal, supply GeT-4o-generated captions for all images in the questions to enable evaluation.
We now present additional case studies, showcasing both correct and incorrect responses by MILMs, organized by subject. We provide the original questions, MLLM re- sponses, ground truth solutions, and our errоr analyses. Some questions also feature ol responses to the captioned version of the original questions. Overall, we find that recurring pattern in error cases is that MLLMs fail to fully engage their multimodal reasoning skills. When questions require multiple visual passes or extended visual simulation or manipulation, MLLMs often skip over these critical steps in their thought processes or produce completely incorrect solutions.
previous experiments, we
Table compares scaling results on the math portion of EMMA-mini using Sowe22.5-Math-RMM. reward model for math, against other generalist reward mod- els. Overall, generalist reward models generate better re- wards than Owen2.5-MMat-RRMT7 Owen2.55MMth-RRM-T is not text-only model, which likely affects its performance on multimodal benchmark like EMMA.
a specialized
We note, however, that
C.5. Breakdown of Experiment Results by Category
In this section, present detailed breakdown of the sults for each category across different subjects. Specifically, the results for math are shown in Table 8, for physics in Ta- ble 9, for chemistry in Table 10, and for coding in Table 11.
21

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Table 8: Performance of state-of-the-art MILMs Math. Column abbreviations: 2D Transformation, 3D 3D Spatial Simulation,
Path Path Tracing, Pattern Inference, MH Multi-Hop Visual Object Counting.
EMMA-mini
EMMA
Overall
2D
3D
Path
MH
Pat (100)
Overall
(100)
(266)
(275)
(127)
(124)
(892)
Random choice
13.00
15.04
12.73
10.24
22.58
9.00
14.01
Human Expert
75.00
Direct Claude 3.5 Sonnet
23.00
26.69
18.18
21.26
49.19
17.00
25.34
Direct Gemini 2.0 Flash
20.00
25.19
20.73
19.69
37.90
17.00
23.88
Direct GPT-4o
30.00
27.44
19.64
17.32
58.87
21.00
27.24
Direct Qwen2-VL.-72B-Ime
VL--7BBIInstrict
38.00
24.81
20.00
18.90
78.23
53.00
33.07
Direct I.laVA-Onevision:
25.00
24.81
22.18
20.47
69.35
8.00
27.69
Direct VnternVT.22-LJammm
31.00
22.18
14.55
22.83
65.32
15.00
25.11
Direct Lntern
30.00
28.95
21.82
18.90
80.65
19.00
31.39
CoT Claude 3.5 Sonnet
30.00
26.69
22.18
22.83
60.48
26.00
29.37
CoT Gemini 2.0 Flash
24.00
23.31
26.55
15.75
37.90
29.00
25.90
CoT GPT-4o
27.00
23.68
17.82
14.17
60.48
23.00
25.56
CoT Gemini 2.0 Flash Thinking- 1219
35.00
30.83
27.64
20.47
60.48
23.00
31.61
CoT Gemini 2.0 Flash Thinking-012)
34.00
32.33
37.82
26.77
64.52
27.00
37.11
CoT OpenAl ol
41.00
CoT -a--n---V-777--mmnaaae
32.00
18.80
16.00
14.96
78.23
37.00
27.69
CoT -LaVVA--nviiiionn728
23.00
19.17
13.45
16.54
64.52
11.00
22.42
CoT
27.00
16.17
14.55
15.75
64.52
15.00
22.20
CoT InternVL2..578BB
31.00
22.18
13.09
16.54
75.81
18.00
25.56
Table 9; Performance of state-of-the-art MLLMs Physics. Column abbreviations: Path Path Tracing, 3D 3D Field Simulation, MII Multi-Hop Visual Reasoning, VD Visual Decomposition Simulation, GR Graph Reasoning.
EMMA-mini
EMMA
Overall
Path
3D
MH
VD
GR
Overall
(100)
(13)
(37)
(33)
(47)
(26)
(156)
Random choice
23.00
38.46
21.62
27.27
31.91
11.54
25.64
Human Expert
64.50
Direct Claude 3.5 Sonnet
34,00
30.77
37.84
36.36
31.91
30.77
33.97
Direct Gemini 2.0 Flash
40.00
38.46
29.73
42.42
38.30
46.15
38.46
Direct GPT-4o
38.00
30.77
40.54
33.33
36.17
50.00
38.46
Direct Qwen2-VL-22
40.00
30.77
48.65
45.45
42.55
34.62
42.31
Direct LLLaVA--nvvision-222
32.00
23.08
27.03
39.39
48.94
26.92
35.90
Direct Intern AnternVL22-Llama3-700
22.00
15.38
32.43
21.21
6.45
23.08
22.44
Direct InternVL.2.5-755
40.00
38.46
43.24
42.42
38.30
26.92
38.46
CoT Claude 3.5 Sonnet
38.00
53.85
37.84
36.36
42.55
42.31
41.03
CoT Gemini 2.0 Flash
41.00
30.77
29.73
36.36
44.68
46.15
38.46
COT GPT-40
44.00
69.23
35.14
39.39
44.68
46.15
43.59
CoT Gemini 2.0 Flash Thinking- 1219
$7.00
61.54
43.24
57.58
61.70
61.54
56.41
CoT Gemini 2.0 Flash Thinking-0122
63.00
61.54
54.05
60.61
$7.45
73.08
60.26
CoT OpenAT o1
49.00
CoT -i--n--VV--77----nieeee
34.00
15.38
35. 14.14
33.33
34.04
46.15
34.62
CoT Onee----niiss
26.00
15.38
10.81
18.18
6.38
15.38
12.18
CoT Intern 7.22
33.00
53.85
27.03
21.21
12,90
38.46
32.05
CoT VLtermVL2.557777
36.00
46.15
35.14
30.30
46.81
42.31
39.74
22

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Table 10: Performance of state-of-the-art MLLMs Chemistry. Column abbreviations: SR Structure Recognition, GR Graph
Reasoning, RS Reaction Simulation, --o-pro- Reaction Simulation Pro, Knowledge based Counting.
EMMA-mini
EMMA
Overall (100)
SR
GR
RS
RS-pro (105)
KS
Overall
(474)
(9)
(132)
(456)
(1176)
Random choice
27.00
24.47
33.33
27.27
35.24
0.44
16.50
Human Expert
86.00
Direct Claude 3.5 Sonnet
44.00
66.88
22.22
55.30
55.24
6.80
40,90
Direct Gemini Flash
36.00
54.01
11.11
53.79
58.10
8.33
36.31
Direct GPT-4o
33.00
47.05
11.11
51.52
42,86
8.33
31.89
Direct Diven2-VL-72BB-mme
34.00
45.99
33.33
48.48
45.71
9.65
32.06
Direct -
24.00
38.19
33.33
39.39
26.67
7.24
25.26
Direct Intern InternV1.2-4 V1..21.
21.00
37.34
22.22
31.45
24.76
8.55
24.06
Direct LntemmL2..5-7B
38.00
55.06
33.33
47.73
43.81
8.99
35.20
CoT Claude 3.5 Sonnet
41.00
57.17
33.33
58.33
58.10
15.57
41.07
CoT Gemini 2.0 Flash
36.00
22.15
33.33
50.00
59.05
11.84
24.66
CoT GPT-4o
35.00
42.41
33.33
51.52
45.71
16.67
33.67
CoT Gemini 2.0 Flash Thinking- 1219
41.00
48.31
33.33
45.45
69.52
17.76
37.93
CoT Gemini 2.0 Flash Thinking-011 0121
47.00
53.16
66.67
48.48
58.10
23.25
41.58
CoT OpenAl ol
40.00
CoT
32.00
33.33
11.11
37.12
42.86
7.89
24.57
CoT O
23.00
33.76
0.00
37.88
20.95
7.24
22.53
CoT IntemVL.2-Llama3-766
21.00
29.11
22.22
30.65
22.86
6.58
19.73
CoT InternVL2.55758)
24.00
37.13
33.33
37.12
33.33
13.16
27.47
Table 11: Performance of state-of-the-art MLLMs Coding. Column abbreviations: CCV Code Choose Vis, VCC Vis Choose
Code, Mwol Modify without Original Image, Mwl Modify with Original Image.
EMMA-mini
EMMA
Overall (100)
CCV
VCC
Mwol (94)
Mwl (94)
Overall (564)
(188)
(188)
Random choice
28.00
22.87
23.94
29.79
30.85
25.71
Human Expert
85.50
Direct Claude 3.5 Sonnet Direct Gemini 2.0 Flash
35.00
32.98
41.49
40.43
42.55
38,65
41.00
39.36
42.02
43.62
45.74
42.02
Direct GPT-4o
40.00
43.09
35.11
40.43
43.62
40.07
Direct Qwen2-Vine Dwen2-VL-772--11
IL-72B-Instrrcct
37.00
35.11
30.85
36.17
39.36
34.57
Direct O
28.00
22.34
28.19
38.30
32.98
28.72
Direct Antern-12-Llamm33776
28.00
27.66
39.52
30.85
28.72
27.84
Direct IntemVLL2..57759
33.00
30.85
31.38
35.11
31.91
31.91
CoT Claude 3.5 Sonnet
39.00
39.36
38.83
43.62
43.62
40.60
CoT Gemini 2.0 Flash
44.00
38.30
46.28
37.23
39.36
40.96
CoT GPT-4o
38.00
40.43
35.11
44.68
38.30
39,01
CoT Gemini 2.0 Flash Thinking- 1219
41.00
43.62
46.81
39.36
40.43
43.44
CoT Gemini 2.0 Flash Thinking-
48.00
47.87
52.13
45.74
42.55
38.06
CoT OpenAl o1
53.00
CoT Doven2-VLL722--Innaie
23.00
31.38
28.72
25.53
30.85
29.43
CoT ILLaVA-Onnevssion7722
29.00
24.47
32.98
35.11
34.04
30.67
CoT Intern Vntem
32.00
24.47
29.79
38.30
35.11
30.32
CoT InternVL...57788
19.00
25.53
25.00
25.53
24.47
25.18
23

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning
Benchmark
Math: 2D Transformation
B
D
E All routes equal
Question:
Between two points four routes drawn. Which route is the shortest?
GPT-4o:
1. Route A: "Tee route goes up and then down, forming triangle. The total harizontal distance units (3 units the left and units to the right),Thu The vertical distance is 3 units and units down. The total distance the sum of the two slanted lines, but the overall horizontal distance between the two
points is still units.
2. Route B: The forms peaks, but horizontal distance between points is still 6 units. The vertical distance units up and units
down, similar Route A. The total distance the sum slanted lines, but overall horizental distance between the two points is still units.
3. Route C: The route forms three peaks, but the total horizontal distance between the two points still 6 units The vertical distance units up and
units down for each peak. The total distance is the sum of the slanted lines, but the overall horizontal distance between the two points is still units.
4. Route D: The route forms four peaks, but the total horizontal distance between the points is still 6 units. The vertical distance is 1.5 units up and 1.5
units down each peak. The distance the
slanted but overall horizental distance between the points still units.
Conclusion: routes the same horizontal distance between two points, the vertical movements cancel out. Therefore, the total distance
between the two points the same for all routes. Thus, the correct answer is: (boxed(E)
Ground Truth:
Figure 16: A sample correct case in math.
24

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Math: 3D Spatial Simulation
<imagel>:
Question:
Six figures were drawn, one on each side of a cube, as shown beside, different positions. On side that does not appear beside this drawing:<imagel>> What is the figure the face opposite to it
<image2>:
<image3>:
(A)
[C]
JD]
(E)
GPT-4o:
solve problem, we need to determine which figure is on the face opposite to the kangaroo (shown <image 1>) on cube
1. Analyze the cube in <imageZ>; The first cube shows three faces: red square, yellow star, green triangle. The second cube shows three different fices: triangle, blue circle, hrown arrow.
2. Identify the relationships between the faces: The green triangle appears both cubes,
this understand the relative positions of the other
faces. On the first cube, green triangle adjacent the square and yellow the second cube, green triangle adjacent the blue circle brown arrow.
3. Determine opposite ficce:: Since the green triangle is adjacent to the red square, yellow star, blue circle, and brown arrow, it cannot be opposite to any
of these. Therefore, the face opposite the green triangle must be the kangaroo (which is not showa in <ámage2> but given samage I>)..
4. Find the face oppositc the kangaroo: The face opposite the green triangle is the kangaroo. The face opposite the kangaroo must be the green triangle.
5. Conclusion:
The figure the face opposite the kanguroo the green triangle.
Answer:
lboxed
Ground Truth:
Figure 17: A sample correct case in math.
25

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Math: Path Tracing Change of View
<imagel>:
Question:
Three people walked through the snow in their winter boots. In which order did they walk through the snow?
<image2>:
GPT-4o:
problem, we need to determine the order which three people walked through snow on the footprints shown the diagram. 1. Identify the footprints: In <imagel>, we see three distinct types of footprints: (A) One with dotted pattern. (B) One with striped pattern, (C) One with solid sole with two ovals. In <image2>, the footprints are labeled as follows: (A) Dotted, Striped, Solid. (B) Dotted, Solid, Striped. (C) Striped, Dotted, Solid. (D) Striped, Solid, Dotted. (E) Solid, Striped, Dotted.
2. Analyze the overlapping footpeints: <imagel>, we that some footprints overlap, which indicates that one person walked the footprints another. solid sole two ovals footprints are on top other two types footprints in several places, indicating person walked last. The striped pattern foosprints overlap the dotted pattern footprints in some areas, indicating that the person with striped pattemn walked second.
The dotted pattern foolprints are at the bottom, meaning this person walked first.
3. Determine the order: Based on the analysis, the order is: I. Dotted pattern (first). 2. Striped pattern (sccond). 3. Solid sole with two ovals (third). 4.
Match the order with the options <image2>; The correct order Dotted, Striped, Solid, which corresponds option
(bower))
Ground Truth: A
Figure 18: A sample correct case in math.
26

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Math: Multi-hop Visual Object Counting
Question:
Subtract all large rubber spheres. Subtract all big shiny cylinders. How many objects are left?
GPT-4o:
Let's amalyze the image step by
1. Identify the objects: There green large rubber sphere the left. There purple small rubber sphere There yellow large rubber rubber cylinder the back. There yellow large shiny cylinder the right. There metallic cube
There purple large
the back.
2. Subtract all large rubber spheres: The only lange rubber sphere is the green sphere on the left. We subtract this object.
3. Subtract all big shiny cylinders: The only big shiny cylinder is the yellow cylinder on the right. We subtract this object.
4. Count the remaining objects: Purple small rubber sphere (still present). Purple large rubber cylinder (still present). Metallic cube (still present).
Yellow large rubber cube (still present). Thus, after subtracting the specified objects, there objects left.
answer: 4,
Ground Truth:
Figure 19: A sample correct case in math.
27

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Math: Pattern Inference
Brain Teaser for IQ Test
Question:
5
Find the value of the square in the figure.
8
GPT-4o:
solve this step by step by assigning variables to the shapes: Let the blue circle be C.). Let the red square be S ). Let the yellow triangle be (T)
Step I: Analyze the first equation: From the first equation: [C $ = This means the of the circle and the is 5.
Step Analyze the second equation: From second equation: [T 8] This simplifies [2T =8] Dividing both sides by [T 4]. the value
yellow triangle (T) is
Step 3: Analyze the third equation: From the third equation: [C+ T 7}. Substitute ( T 4) into this equation: [C 4 - 7]. Subtract from both sides:
[C= 3). So, the value of the blue circle ( C
Step 4: Solve for the red square: Now, back to the first equation: [C + $ 5]. Substitute ( C- 3) into this equation: [3 S 5]. Subtract from both
sides: [S 2]. So, the value of the red square $) is
Final Answer: The value of the square (2ox
Ground Truth:
Figure 20: A sample correct case in math.
28

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Math: 2D Transformation
Question: Each half of this figure is composed of red triangles, 5 blue triangles and 8 white triangles. When the upper half is folded down over the centerline, 2 pairs of red triangles coincide, do 3 pairs of blue triangles. There are red-white pairs. How many white pairs coincide?
01: Count the total number of triangles per half: 2. When folded: 3. Given pairings: 4. Account for these given pairs:
Error Analysis:
5. Update the counts after using these pairs: 6. Remaining pairs to form: The remaining top and bottom sets are the same:
2B 7W on top and 2B + 7W on bottom. 7. Forming the remaining pairs: With equal numbers of blues and whites both sides, the most natural arrangement is to pair like colors with like colors:- Pair the 2 remaining top B with the remaining bottom B 2 more B-B pairs. Pair the remaining top W with the remaining bottom W - W-W pairs. 8. Final Answer: \(\boxed(7]))
The model reasoning suffers from two key issues. First, it failed to "remember" prior pairings, which humans would track easily using the diagram, revealing its inability manage dynamic states in multi-step tasks without visual feedback. Second, tasks without visual feedback. Second, ignored image-derived constraints, like the "3-blue pairs" rule, and wrongly assumed pairing like colors was natural. This highlights the model's broader limitation in integrating textual and visual information, particularly for tasks requiring spatial reasoning and multi-hop logic, where visual aids simplify the solution.
Human Reasoning (Ground Truth):
Forming the remaining 2 top B and bottom B, the image shows there are already 3 B-B pairs. Thus, the 2 top B and bottom B cannot form more B-B pairs but instead create 4 B-W pairs. After this, the remaining W-W pairs total 5, as counted in the image.
Figure 21: A sample error case in math.
29

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Math: 3D Spatial Simulation
5
Question:
The diagram shows the net of cube whose faces are numbered.
2
3
4
Sascha adds the numbers that are on opposite faces of the cube.
6
Which three results does he get?
GPT-4o:
solve problem, we need to determine which faces cube are opposite each other and then numbers on those opposite faces.
1. Step I: Identify opposite faces: The diagram shows the nct of cube. cube, each face bas an opposite face. From the net, we can deduce the
following pairs of opposite fàces: (1)Face 5 is opposite face 6. (2)Face is opposite face (3)Face opposite face 3. 5,
2. Step 2: Add the numbers opposite faces: Now, add the numbers the opposite faces: 4
3. Step
results: The three results Sascha gets 11, and S. Final Answer: hoxed(11, (11, 5,
GPT-4o--apptio: Totally Correct
image depicts net cuhe, consisting numbered square ficess. layout center square laheled
Three squares adjacent center square, forming an inverted on left, on the right, on the top. A square labeled 4, located to the right of extending horizontally. square laheled 6, located helow 3, extending vertically.
2. Task: Sascha is tasked with folding the cube, determining the opposite faces, and adding the numbers those faces.
01 oprevii:::
To solve this problem, need to determine which faces of the cube opposite each other after folding the given and then calculate the sums of numbers those opposite
1. Step Visualizing Net and Cube Folding. Let's first reconstruct the net as described: Square 2 is the center, Square to the left of Square Square is to the right of Square 2, Square is above Square 2, Square 4 is to the right of Square 3, Square below Square 3.
[5]
[1] [2] (4)
[6]
2. Step 2: Assigning Squares to Cube Faces, We can assign the squares to the faces the cube as follows: Front Face: Square 3, Back Face: Square 4
Left Face: Square Right Face: Square 6, Top Face: Square 5, Bottom Face: Square
3. Step 3: Identifying Opposite Faces. The pains of opposite faces are: (1). Front 3 and Back 4 (2). Left and Right (3), Top 5 and Bottom
4. Step Calculating
(1). Front Back Faces: [3 7] Left and Right Faces: 6 7] (3). Top and Bottom Faces: [S
Final Answer: !boxed (7, 71
Error Reason:
This imago can be accurately captioned by GPT-4o, as the descriptive content lotally correct, However, after extracting the information. model needs to visualize the process folding the 2D shape to 3D cube and determine the relative positions of cach face. Unfortunatlly. both GPT GPT-4o and GPT-ol fail perform the correct spatial reasoning required for this task.
Ground Truth: 4, 6, 11
Figure 22: A sample in math.
30

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Math: 2D Transformation
<imagel>:
Question:
Anna cuts the picture mushroom in two halves. She then arranges the pieces together form new picture. What could this new picture look like?
<image2>:
GPT-4o:
Arns 1. Understending the Problem: the first image (imagel)), Arns cuts the mushmom picture vartically into two helves. The task to desermine how the
halves reartanged form
Anslysinn The mushroom vertiallly dows middle, so
mushroom (including the cap and hall the stem)
is separated from the right half. The two halves can be reamunged in different ways, such fiipiigg or rotating them.
they flipped rottted. looks like the original mushromm just split Opption The halves are placod side by side; but the right ball flipped borizontally This creates a symmetrical image Option The two halves are placed one top of the other, but they are not flipped or rotated. This does not seem to form a logical image. Option D: The two logical image. Optioon The liaives placed one on top of the other, but the botom half is flipped borizontally. This does not form a logical image either.
Anslyzing Oytions: Option
are placed
lalves placed
top
half's flipped horizonallyy. also does
4 Conclusion: The only oprino that fonns synuentical picture is Option where the right hull's flpppdd horioontally and placad next the left half Fizal Answer: (R)
GPT-4o-caption:
simage image shows drswing
symmetrical along vertical axis.
distinct parts: Cap: rounded, brown shape on top. Stem: A white oval base connected to the cap. A dashed vertical line divides the mushroom into two equal ltaives, splitting the and stem down the middle. At the bottom, there is small drewing of seissoes, suggesting that the mushrooms will be physically two
2. Botton Section -inage2>; Below the original mustrroom, there are five cectangular images labeled (A), (8), (C), (D), and (E). Each rectengle depiets possible arrangmeent of the mushrom halves, (A): The halves placed side side their original orientaioon, furning full mushroom cornetly. (B): The
swapped flipped, The two halves are rotated 90 clockwise, creating bocizontal arangement with the on the left and the stem on the right. (D) The two halves are stem beluw, (E): The halves flipped horizontally,
sections point
the cap sections
(C):
stacked vertically, with the cap
mirroned where sections
outward.
3. The task to determine which the five options correctly represents new arrangemen of the mushroom halves after being cut and rearanged.
o1 o1-preview:
problem, analyze possible arrungements
cutting vertically.
1. Given: The mushroen symmetrical aloogg verriest miss A flr cutting it vatticllly we lave a left half and right half, each containing ailFofthee
and half of
2. Poosible manipulations: (a) Flip a half horizontally (mimor image over the vertical axis). (b) Rotate (egg 90" clockwise), (6) Swap the halves.
3. Anslyve options: Option A; Placing the holves tegither in their original orientation forms the same picture, not new one. So, it's not neceptable as
arrangement. Option Swapping tlipping halves honzootally results the cup sections meeting the and the sections pointing outward. This encitos a new symmerrical image differemt from the original. Opticn Rotzring both halves 90" clockwise and placing them side by side can reconstruct the musbroom horizontally, Howeeerr since the halves are mirror images, the and stem won't align properly to foom cohorent image.
Stucking the halves vertically doesa' forn meaningful picture because both balves stacking them wor't align these parts semsibly. Option F: Flipping both halves horisontally ly and placing them together the center and the sections face outward This is possible new uraangemen..
the cap and and
stom soctions mest at
Conclusion:
Both Option and Option are possible new arrangements. Ulowever, since we need to choose one, and Option B directly follows from swapping and flipping the halves described, Option is the best choice.
Error Reason:
When using GPT--4o to cxption the image, it fails to securately determine wheter the image has been flipped, rotated or swapped. Human perform such ressoning by memally simulating the imnsoommatonn of the capability believe current MLLLMs lack. This involves visual reasoming, canset achieved thronga text-based ressoning alone.
Ground Truth:
Figure 23: A sample error case in math.
31

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Math: Pattern Inference
Question:
Choose the answer for the missing picture.
GPT-4o:
1. Rows and Columas The grid a 3x3 matris, and each cirele is divided into 8 aqual sections, with some sactions shudol grozn.
2. Row wise Analysis: (A) Finst Row: The number green sections increases by you from left right. Flrst circle: 2 green sections. Second
circl:: 3 grean sostions. Third circle green sections. Second Row The number of grees vections also incrosses by as you move from to right.
First circle: green sections Second circle: 4 green acctions. Third cirole: S grees soctions (C) Third Row: The number of green sections should follow increasing you from left right. First circle:
the
sections. Second circle: green sections. Third circle (missing):
should green sections.
3. Conclusion: The missing circle should have 6 green sections. Now, let's look at the options: Option has 6 green sections.
Tous, the correct answer
GPT-to-caption:
image pattem based reasoning puzzle with following key elements:
1. Main Puzzle Gid (Top Section): A 3x3 grid where the first right cells are filled with circular diagrams, bottom right marked
question mark ("?") Each circular diagnm divided into eight equal sectons Some seciors are shaded in green, while others renwin white. The shuded
green vary number and position. showing
potterss
columns.
2. Rows the Main Grid: Row Call One green sector the buttom-left position. Call 2: Two adjacett groen sectors the left quadrant Cell
Three adjacent green sectors, starting from the left and moving clockwise. Row 2: Cell I: Two greem secton, one in the top-rght and one in the bottom- right quadrant. Cell 2: Three adjaeen green
Borming right
Cell adjacent green sectors, forming stage
covering the bottom and right quidrants Row 3: Cell Three green sectors, forning a pattern the bottom half Cell Four adjacent green sectors, creating "L" shape the left balf. Cell 3: Missing cirole (marked by "Y",. Answer Optices [Bottom Section): Below the
circular diagmass labeled are presented the same structure (right equal sectors, some shaded green). The options are as follewss Option I: Three green sectors in the bottom-ief and botten-- censer quadranss. Option 2: Four adjacent green sectors covering the top and right quadrzats. Uption 3: Three green sectors in the top halt, synaneric quadrants, Option green forming
Each diagrams Sollows
vertical Option aljacent groen sectors, covering bottom-left
cross, symmetric acmoss both axes. Option 6: Five adjscent green sectors forming a large scction from the bottem-lefi the top-right quadrant.
ol- el-preview:
1. Underttand the Crid and the Ciroles: Each circle is divided ins equal sectors, Some sectoos are staded green, and others are white. The shading pattems vary
gri. Gaid Detail;; Row 1: (2.1): green (bottonslen posioon). Cell (1.2): adjacem green sectors lent quadmast. Cell (1.3), adjacent green sectors, starting from left and moving clockwise. (b) Row 2: Cell (2.1): 2 green sectors in top-rigit and bottom-right positions, Cell (2,29:3 wejacent green sectors forming the right half of the cirele. Cell (2,3); 4 adjaeent green sectors forming "V" shape covering and right quadrants.
formung panem creating "L." shape the left half. Cell (3,3): Missing circle (marked by (?")
(3,11 green
half. Cell (3.2). adjocent green
2. Identify Patterns in the Grid: (o) Number of Green Sectom: In each now, the number of green sectons incresses by from left to right. In each column, by top
(3,3) should green continue the pattem, (c) Shuding Pattem Direction: The shading seems progress a purricnlar direction ench row.
number grean
(b) Logeaa Deduction: The missing
3. Predict the Missing Circle's Stading Putten: Since Call (3.2) has 4 adjecent green sectors fonning "L" shape the left hailt it's logical to prodict 5 adjacent green contasuing where previous shading lett off. The green sectors likely contigaous
Cell (3.3)
poosibly starting from the bottom-lefi sector and covering up to the top-right sectoe:
4. Masch the Prediction with the Answer Uptions: Option shows circle with adjacent green sectors fooming large section from the bottom-left to prediction.
tog-right quadrant, which
5. Conclusion: Based on the logical pattems observed in the grid. Opcion 6 the coneec answer.
Error Reason:
Both GPT-4 1-40 and GPT-ol incorrectl slesslly the pattern: they cosolude that the number of green sections changes a regular pattern. However, it's find the octual rule is that the position the green sections changes in regular pattem, By misidentifyung the color, quanity and position, both perform flawed ressuning and generate wrong answer.
Ground Truth: 4
Figure 24: A sample error case in math.
32

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Physics: Visual Decomposition Simulation
Question:
B
object travels along a path shown above, changing velocity as indicated by vectors and Which vector best represents the net acceleration of object from time 4. to 6?
Ground Truth:
GPT-4o
Identify the direction acceleration, same the direction of the change in velocity
1. Velocity at t (UA):
T points upward (+t) and to the right (+-rr).
fn
2. Velocity at (op):
2
Because:
Ts points upward (+y) and to the left (-z).
= (-A),
B
3. Acceleration:
the direction A of can determined graphically.
AJ
Up
UA
Ug + (-ŪA)
At
A
Here, points downward
3) and to the left (-æ).
4. Net Direction:
Placing the B vector "tip- to-tail" with the vector gives direction for Aú (and therefore, a) to the left.
Horizontal components (+ xt) partially cancel.
3
B
Vestical contliaalccopppnnneeeee 3) combinn,
downward (-y) vector.
5. Conclusion
AV=B-A
The net accelcration points downward (-y), corre- sponding to option b.
Figure 25: A sample eror case in physics.
33

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Physics: 3D Field Simulation
a.
d.
+
b.
-20
+3Q
Three point charges, of charge +Q, -2Q and +3Q are placed equidistant as shown. Which vector best describes the net direction of the electric force acting on the +Q charge?
Question:
GPT-40 with CoT
Forces Due to the Charges:
Vertical:
The 2Q charge attracts the +Q charge.
vakQ
3v/k0² 2r
F 2Q.0
F+3Q.
(Force points downward-left).
The +3Q charge repels the + +QQ charge.
Net Force Components:
(Force points downward-right).
Horizontal: F3
kQa 2r
2. Force Magnitudes (Coulomb's 's Law):
Vertical: F,
5v 5v/AQ2
2rd
Force due to -2Q: F 20 2
Net Force Direction:
36O
Force due to +3Q: F-3Q
The net force is determined by
3. Force Directions:
F F
tan 1
-2Q: Downward-leff
+3Q Downward-right.
F2 > 0 and Fy. 0 indicate the
4. Force Components (at 60%)
direction is downward-right.
Horizontal:
Final Answer:
10t F- = +1 F-3Q,= = + 3kq'
2
Direct GPT-40 b.
Human Reasoning (Ground Truth):
The direction of the electric force due to -2Q and + 3Q can be visualized with a quick free-body sketch of the electric forces:
Misunderstand the net force's direction during vector addition. The issue arose from not correctly identifying the force direction from +3Q to +Q (longer red arrow in vector graphic on the right), which causes the net force to point X not downward ' in
Error Analysis:
+0
b.
-30
Figure 26: A sample emorr case in physics.
34

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Physics: Visual Decomposition Simulation
View looking on pool
a.
b.
c.
table
image
Simage
Question:
A billiard ball hits the side of pool table at an angle 0 as shown in the top view above, and bounces away at the same angle, and with the same speed. Which vector indicates the direction of the net change in momentum of the billiard ball?
GPT-4o:
Error Analysis:
To determine the direction of the net change in momentum of the billiard
ball:
Initaal Momentum:
moving
2. Final Nomentum:
The answer incorrectly assumes that the momentum the direction does not change, thus drawing the wrong conclusion: the net change momentum is in the horizontal direction.
After bouncing the wall, bal moves away the same angle 9, the horizontal
Since the speed of the ball ramaina con stant, the magrittde of momentum does change,
only the direction doss.
Analyzing Components Momentum:
The horizontal component of the momentum changes direction (e.g. from left right..
The compoment unchanged.
This
horizontally
Conclusion:
vector representing change The correct anower (b).
right.
Ground Truth:
The correct answer is d The billiard ball has velocity vectors in both the and directions, shown here. The x-component of the velocity remains unchanged, but the coomponent has changed from the positive to the negative direction, a change in velocity-and thus, momentum in the negative direction.
Vx-initial
Vx-final
V Vy-initial
Vy-final
Figure 27: A sample error case in physics.
35

---

Can MLLMs Reason in Multimodalit? EMMA: An Enhanced MultiModal ReAsoning Benchmark
Physics: Graph Reasoning
Question:
An object is dropped and accelerates downwards. As it falls it is affected by air friction, but never reaches terminal vclocity during the course of its fall. The graph that could indicate the magnitude of the object's acceleration as a function of time is
a.
b.
d.
C.
g g
g
g
g
0
0
0
0
GPT-4o:
When object is dropped, it initially accelerates at the accelcration due to gravity, As it falls, air resistance (friction) increases, which reduces the net acceleration. However, since the object never reaches terminal velocity, the accelcration decreases but does not reach zero. let's analyze the options: a. The graph shows constant acceleration which would be the case if there air resistance. This is not correct because the object is affected by air friction.
b. The graph shows acceleration starting at and gradually decreasing, which is consistent with the object being affected by air resistance. The acceleration decreases over time but does not reach zero, which matches the scenario where the object does not reach terminal velocity.
The graph shows acceleration increasing time, which incorrect because air resistance would cause the acceleration decrease, not increase.
d. The graph shows acceleration starting at and decreasing rapidly, which could represent situation where the object reaches terminal velocity. However, the problem states that the object does not reach terminal velocity, this is not the correct ption. e. The graph shows acceleration starting below g and increasing, which incorrect because the object with acceleration of g and air resistance causes it decrease. The answer is
GPT-4o Caption:
The image shows five graphs labeled through c, each representing accelcration versus time:
a. The graph is horizontal line a constant value of g.
time. b. The graph starts at g and curves downward, decreasing
The graph starts below and curves upward, increasing over time.
The graph starts at g and curves downward, decreasing over time.
The graph starts below g and curves upward, increasing over time.
Each graph has a dashed line indicating g reference point.
Figure 28: sample error case physics.
36

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
X
Physics: Graph Reasoning
ol --preview:
ol ol-preview:
Choosing the Correct Graph
sfrcicn tep
lenation time
Understanding the Physical Scanario
is
Oijject Fres Fall with Air Recittnce
Decrease Exponentialyy alyy rocele atton decr woonn exporendally due to increanthg effect
rine stbect drcceed, ocleeestts evred due
accelersion
Aa pair speed ortranca type nickoon ccposen the modion
hacessss reducing acodiaation.
aezaderston appreahhe zamo but never curs mechee within the time trmm consaterdd
Graph Descriptions:
Option tnl: horizartal iconstant accele sttonn incerrect because dosen account
Muthomatical Ansiysiis
decreasing accoleration dua resistance.
has Forte ofdecen
sectoce You artinn poweeereete
popamentia decay accateration.
Foc -n my
Option tej: Starts beloow and curves upwand, increasing time. Incorrect because scenaria.
ia the mass ol otject.
Option bd): Starts it ourvas downword, docraasing ower time. opicos and MI
l scoseeation due gravity.
desoription,
lnner
exparentia, 1d) may represent a liner decrease, which incerreet.
velddty ohe dime
Optiin tel: Stans below and curves upwend increasing over time, incorrect for some
Newton'ss Second
Apoliing Newton's second 3" met
The graph that correctty rspreserts the magniuude the object accrleration as function of time
Option 05::
This showe thet scceleos sccelearton i decrassss cylocitty iy incre incresons
Starts at and decroases exponentially over time.
Vilecity Punction Time:
Ground Truth: (d)
The solution Ic:
x(t)
(1-e-5)
Error Analysis:
where diject drooped
Misunderstanding the characteristics of exponential decay, believed that the declining trend of option B reasonable, but overlooks that the decline of D is more gradual and better fits real physical phenomena. Moreover, the description in the question that "the accelcration never reach cannot correctly matched image of any option.
Substiiut ng back into off):
o(f)
((t) ge
soceleradinn
Figure 29: sample error case physics.
37

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Physics: 3D Field Simulation
+z
<image
Question:
Two long parallel wires, oriented along the y-axis shown, contain and
flowing in opposire directions. Which of the following statements is true?
a. The magnetic field due to circles that wire in a counterclockwise direction.
The magnetic force from wize 21.
strong as
The magnetic force between the two wines is proportional to the inverse square the distance between them.
The force wire in the direction.
The force wire 2f is in the direction.
GPT-4o:
Ground Truth:
Anelysis:
magnetic direetion, as determincel wsing the Right Hand Rule. This field down the direction) the location applying another Right can determine that the magnetic force on 2/ due to magnetic
answer
clockvise
Magnetic Direction
Current (into the pagal): magnatic filld circles clockorie.
we
21
Option is incornect becouse It' fiald ciockwiso, countercleckwtsa.
the diecction.
Magnetic
Wire has force oqual in magnitude wine The magnetic detemmind as follows
Ampine's is lass force per urit langth propertional to J; 4z/dd
wires
equal magnitudia opposite dinection (Nawton's shird lawd)
Magnitude magnetic field due /:
Option incornect.
Md where the distance between the wires Force on wine 2/ due this field:
3. Dependence on Distance:
proportional not
Option is incorrect
xxx
F, (2 Rd
Opposite currents repet. The force on 2/ points in -ar-direction Optioon is correct; Option incomect.
The force per unit lengh of the wises, then,
F.
0,21
Conclusion:
Figure 30: sample correct in physics.
38

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Physics: Graph Reasoning
Question:
(m/s)
billiard ball rolling across table in the +x direction at m's hits the edge the table at aperpendicular angle, and bounces back in the -x direction, now traveling at m/s in theopposite direction, 'The greatest magnitude of acceleration for the billiard ball was time
+2
+1
(x10
a. t = x 10033 S b. = x 10
t = 5 d. = 7 e. X
GPT-4o:
The graph shows velocity of the billiard a function time. Initially, the hall moving the +x direction m/s, and then it down, reverses direction, and moves in -x direction at m/s.
Thee greatest magnitude of acceleration occurs when velocity changes most rapidly, which corresponds to the steepest part velocity-time graph. From the graph, the steepest slope occurs around t 10-3 s, where the is reversing direction.
The answer c.
Ground Truth:
The answer is Acceleration is defined the change in velocity over time:
Av
finci
Vnitial
At
Jinal
miital
Av rise
Ar m, the slope of the graph. The slope of a velocity time graph represents the accelcration of the object, so the greatest acceleration occured where the slope greatest. Here, that would be any time between / = and / = milliseconds.
Because e here is plotted on the y-axis and /is on the x-axis,
Figure 31: sample correct in physics.
39

---

Can MLLMs Reason in Multimodalit? EMMA; An Enhanced MultiModal ReAsoning Benchmark
Chemistry: Reaction Simulation
Question:
An 'arrow-pushing' diagram is common lype of chemical image used to illustrate electron flow in mechanistic steps. Choose the molecular image that represenis the SMILES expression for the molecule after the electron has been relocated, depicted in the image.
R
R
A. C=C/C[O-JCCCC=O
H
B. C=C(C(CC-ONO-
B:
C. C===CO)CCCCC--
D. *1C-0000000000000
GPT-4o:
Let's analyze the arrow-pushing mechanism step by step:
The image shows carbonyl group (C=O) with an arrow indicating electron movement from the double bond the oxygen
atom. This suggests that the oxygen will gain negative charge, becoming an alkoxide ion (0 ).
The second arrow shows the bond between the hydrogen (H) and the adjacent carbon breaking, with the electrons moving towards the carbon-carbon bond. This suggests the formation of double bond between the carbon atoms.
The result electron movement is formation of new C-C double bond the oxygen becoming negatively
charged (0)
Now, let's examine the options:
Option A: This structure shows C-C double bond, negatively charged oxygen (O attached to carbon, and carbonyl
group (C=O). This does not match expected structure because the negative charge on wrong oxygen.
Option B: This structure shows C=C double bond, negatively charged oxygen (O ), and carbonyl group (C=O) attached the same carbon. This matches the expected structure after the clectron movement.
Option C: This structure shows a C=C double bond, but the negative charge is on a different oxygen, which does not match the expected structure.
Option D: This structure is incomplete and does not match the expected structure.
The answer is (B).
Ground Truth: B
Figure 32; sample correct case chemistry.
40

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Chemistry: Knowledge-based Counting
CI
In the transition-state structure shown in the image, calculate the total number of bonds in the structure, including single, double, and triple bonds but excluding those involving hydrogen. Note: Disregard arrows. Consider all components present the transition-state structure shown in the image.
Question:
<image_
1>
ol: A clear way to see why there are 17 total bonds (excluding those to hydrogen) is to break the structure into three parts and count them: 1. Cyclohexyl moicty: Six C single bonds within the cyclohcxane ring. Tetrahedral region around the acyl center: One single bond between the oxygen and the carbonyl carbon. One single bond between the same oxygen and the eyclohexyl carbon. One double bond the acyl group (C=0). One single bond between the carbonyl carbon and the aromatic ring. Aromatic ring with chloro
substituent: In the Kekulé representation, the benzene ring has three single bonds and three double bonds (six ring bonds total). One single bond between ring carbon and the chlorine atom. Adding these together, and ignoring any bonds to hydrogen: Single bonds: 6 (cyclohexane ring) + (O-C in tetrahedral region) + (0- cyclohexyl carbon) (carbonyl C aromatic ring) 3 (single bonds in the benzene ring) + (benzene CI) 13 single bonds. Double bonds: (C=O in the acyl group) (double bonds in the benzene ring) = 4 double bonds. Thus, the total is 13 4 17 bonds (excluding any to hydrogen).
Ground Truth: By directly counting the bonds marked
Error Analysis:
the diagram, can there total 21 bonds.
The error lies in undercounting the total number bonds. The original answer missed four key single bonds: two C-O bonds within the ester group, one O-C bond connecting the ester oxygen the cyclohexyl ring, and one C-O bond in the acyl center. These omissions led to total of 17 bonds instead of the correct 21 bonds.
CI
17
16
H
9
1S
19
0
/2
14
21
20
2
5
123
O
3
Figure 33: A sample error case in chemistry.

---

Can MLLMs Reason in Multimodalit? EMMA: An Enhanced MultiModal ReAsoning Benchmark
Coding: 3D, Color Texture, and Axis & Scale
Ieart sedurn
somee
Lin derietty drape
de Lnble Netther Suaree Miheratt de 00N-00000 100 1010 Racter upta cittree
2011
Question:
Which vinulizarion this code possibly generste? We are using Python version 3.1 mapplotlb version 3.6.3, and (if applicaille). stop step. Then, append new after your thought process to provide your final answer. This new line should contain EXACTLY one letter (onz of"A", "B", 1C7, or "D"), No other content
GPT-4o:
The code imports libraries such litplotlib, the seabor "dark"".
seeburn, and pardas..
dictionary
corresponding percentages different
agreement levels.
The data converted into pendes DataFrame and the 'Task' column set the index.
Several veriables are calcuisted to position the bars correctly for a diverging stacked bar
chart. These varisbles include neut_ne leftt disagree, leeft_arrial Lly disagres.
and Teft, [eft_ffully disegree.
A horizontal bar chart is created using barhil.) where each bar represents different
level agreement (e.g.. disagree, Partially disagree, etc.). honizonallly, negative
disagreement positive
agreement.
to range from -100 to 100, the labels adjusted
7. A vertical line drawn at x== separate the negative and positive sides.
A legend added to the plot, and the layout adjusted for better display.
The code generates diverging stacked horizontal bar chart, where the left agreement.
disagreement right
Figure 34: A sample correct case in coding.
42

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Coding: Legend and Color & Texture
isport netplotliD.pyplet as plt
isport nuepy np
np.randon.secl442
rand(58)
np.randoon rand 100
sizes random,, (5ad 1008
size, Legendeedddddd [188, 388, 608, 908]
plt. fit.figureffissiee0.., 61)
scatter plt.scatterix, y, sesizes, aloha=0.6, edgecolors='W", (inewidth=e:55 for size in size_legend:
plt.scatter((1), [], s=size, edgecolors=""", alpha=8.6, label=f'Size (size)")
plt. let.eentssatteeroonnsst,,, Traneon=True, labelspecing=t, title="Point Sizes", Toc="center right") plt.show(]
A
C
Question:
Which visualization can this code possibly generate? We are using Python version 3.11.0, matplotiib version 3.6.3, and scaborn version 12.2 (if applicable). Think step by step. Then, append new line after your thought process to provide your final answer. This new line should contain EXACTLY one letter (one of A"," "B", "C", "D"), No other content allowwe last
GPT-4o:
Error Analysis:
y, which are scoled 100, This means the scatter plut will have points with and vilues
GPT-4o assumes that, since the code does not explicitly specify color for the legend markers, they will inherit the default color of the scutter plot, However, it fails to the function called multiple markers in legend. result, each call assigns distinct color to the markers, meaning the final logend markers will not all be blue.
2. The sizts arra generated using rpurondoon rand(5e) 1988, moaring ring the size of each
create plat it and sture The Rlphert.E E makes the soints sami traneparent, and reprostarentgt adds white edges the cointe.
pll. acsttert)))
plt.scatter(
4. The size, liae_llgeerd Jegend ett contains speat e diee [106, 200, 590, 9b8) for the legend The
legend slaced at the "owneer
lapand v, plat,
Stowe
The key afferance betwoon the optionsl the color ohe legend markers, Since the code does specify which blue.
detault
plot,
Fiall
Ground Truth:
The code specifies that the legend placed the right, which eliminates Since the pit scatter)) fumetion called multiple times (in loop) render the markers the legend, each call assigns distinet color to the markers. Therefore, the markers will assume different colors, Carrect Answer:
Figure 35: A sample error case in coding.
43

---

Can MILMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Coding: Legend and Position
Isport satpastii.cso.om pit
daces torppaaaa, 16,
trermai Lincacer0,, 3,3, 1. 188
crepy Sorguur,
"Date", Beane", "Alratte "Pillet")
fie, owss ett.sopllanc Fiasioeett,, 171)
for (eee,, (be, crip) in exmerste(r1p aves, flarteidl, cri00))
get cenall
teaetirse"" colom
on plant setes, therell deta, labele Themeo Lisesrpto colere
xlate.""bete")
accest_alabe (1te
ne. lupeeeeeeeelleerrr laft' Te.._tt. www.aniniiiii -6.21
A
B
D
MAN
Question:
visualization this possibly generste? using Python scabora vasion 0.12.2 applliccblle Think step step. appenad new afler your thought proceso to provide your final anowe: This new line should contain EXACTLY one letter fone of""""""" A", "B", "C", or "D"), ofler contert is allowed on this last line.
matpletthh
GPT-4o:
Error Analysis:
oode by importing recemsrr ltrarie:: aary Lartisk, poppol fost platting Iuapy for numerical atlons
GPT-4o notices that the fourth subplot
has
perform
A random seed to encum cialit the randerr ders
vieral reasoning identify
Almes is greeented se invorty spoond aryy from to with 900 points.
relsive position the subples, instead simply refaning saspeiilie location".
yel: AVta in tambaos unction thet genarete 120 menkon valiae between 0J7
Shavenal a tneers spaced army from di so with 100 polnts.
A lnt crop momes derined
dod giid sutppits is ding Sit situsaallion loms, with rigune dine 12v12 inchee.
Ground Truth:
The sight, Jayous tunstion in used adi paidding between subepict..
The conditional the loop tells us that only the founh sohplott (indes 3 low legend The legend is placed with -0.2),
topp tur stea ovar the sous and creck, plorting two mes for each crep:
One line get trandom in orange.
One lime Livta tinee daaa in brown with das starnectiine style.
10. The tii,, wiebel and y-labul are for each subples.
bbox, anchor it lower right Bouth suhplor Since we have rows and columns, the fourth subplot is Row Columo 2. Henoe. is the right unswer.
11. A legand addiect to the fourdh sutplet Index 21 with apeoiic location.
12. Firilyy the plac siaplayed saing prt.uttou
The coce gunestes dud gridio liw clots, such shrwwng heo inno (GO0 Tharnal kr placed outade tourt mubole:
d#teret
Figure 36: A sample error case in coding.
44

---

Can MLLMs Reason in Multimodality EMMA: An Enhanced MultiModal ReAsoning Benchmark
Coding: 3D, Color & Texture, and Axis & Scale
Ieart niqpeetii cxetet att
Free
ra3oda sapert
iesplecta
70, detecoet Prectse wil > 6.3 m00--.3
actecte, www.:/eetttieeto.ooooom Flateet)
"1g.a wasgpinniss. s7jjectie 323
www.il//neetoooo Eig.rt 10r cr. rapp reriiti regestass/wer
(idett Setechal Practee))
alt.art Lla (20060")
fos-etarkeraert away oapictas sireren, Io sesereeenntti ticteleerts, Meriee. (icecid)
A
2013
2.5
40
C
E10
8.00
Question:
Whish siusalication code
taing Py vension vaniun 0.122 applicable) Think step sep, Thei, append your thought process provide your unswer. new should EXACTLY W., 7C", other
Claude 3.5 Somnet:
Error Analysis:
maker
Chasic fails the diffennmeee
crenas masplettors sarae funetin comtaiedd surbice plee wing 3 tww...
coler
Futheenere. linitation understarding, fail
I-tett Log. Heparbite holk from to
spos, spos, apos, dly
Poeen Detentee Pran yrakualatii using ieperestial lunclonn))
emag vindie) actually
Colornae wincil cotormap
without parameter being explicitly variablle
Ircaeeer fut docas kund ara
Ground Truth:
Mtdinss valas should aroud 3.3 thom the foreulai
und have Mix hare while C and have grdien colored bass and C have binned ooler has while B and have continnous ooler lune In the fimation call
ahoe salnngad baan lice pellow to parple
the vitend afteent ecorter velre frurdes to ous, 0/1, mii
horbddrgooo, ypus, spus do, dy, shade True, "viids',, 'aidds'
bar3d
Oiven these covamationa opise masches the code sccennty anoving the connct
gracient aftec prc sn stterr valuee thanwos was from glar mathematinn fommas
color
using cokomap sommiiued valres. asighing colors
to single waich climinats und
colors Boundary Namelovela. cmap spocifies the color har binnod, which leeves A as the cornect anower.
Figure 37: A sample error case in coding.
45

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Coding: Axis & Scale
6X2
144
625
30B
033
Inport rpurandon. scc2ee)
satolatisb-ppplo
Lime Lonapecsio,
mand 528) 0.001
signal 1296:228, 258:208)
9.2
E190 468:4201 9.85
23
00. att_sssssssssci 1, tsight_rsttomii0.
hipace-8.31
23 bi_ a22, htt_assplttiiee.. 013
Tig- (1._epppeeelg
Instsa/ttoonll aggertatacto' ectests /ctests/rteel))
100
22
330
26 cher calerbor cox, caneax <b, orlentatien "rorioonaal))
Freguency (8), Frequescel-111 (-1ll. origan leven" deap "tst")
ay.boaaxss Lick, posttias "top")
sa1. ylasel (Freuancy (Miz2")
24 aet ciasei (Thee (s1')
23 222 ploptott,,,, secensityy oftan=bbacck)
yla-set 1s1
<original code>
Starget images
pit.showl
Replace Liee 14 with:
(g shermemec1)
time no. Lincpacete, 0.06, 200)
hoplace Lines 4-23 with:
trecurccy up. Limspeceeeben 1000, 1580, 500)
sugrat up, randon, randisce, 500) 0.00:
sigral 12091228, 259:2601
9.1
sirectionn's
sigral [180:128, 402:429) 0.05
aditicc doreecioon inn;
intensity 46008 rg..randen. rand(ces) 102
ax2, (directtto Bnn
tig, taxi, =lttsuustotstt,
gridspee_ hwwp:eeeeeeetttreeteteeee
11), rigessen(0,
Replace lines 11-18 with:
itg.aet__rrdappeel beright_ratianiie (6.25, hspaco-6.851
eeterta [R), riae--),, frequency(R), fregencyl-133,
origin-/oo lover". cmag-lnat")
ab fig- wwff_ssupplottgs..e (81)
cbar fiq.colorbaricas, 2anaxs, "uliettation-forizemaa"
axl. viabell -at_viaalll-Fooeee (Miz)")
axl Fin.
ax2 Tip, aolp. sutplltt fgs (2:)
cax axl, inchows sigrat, aspect= auto',
dbar tag.calorrarr asman_ca, ersennatioonmborrzaataa
ectents [time
origi lloar", onage hat')
(-1), Trequency(e) (8). (requmtr(--1),,
ect.vt..cueelll Time (x)1)
tight beyout
aet.pleifcies, intensity, colore (black))
ect, set. wabelllTTee (x)1)
ad (labe11 Intensity"1
ak_cb.axis(
Question:
Our goal is to reproduce the visualization the image shown. The code snippet above currently does not accurately
genenate the target visualizarion.
L.O, matplotlib version 3.6.3, and version (if appliiiic.ll. What change should apply the original code in order gencrate the target visualization? Think by Then, append new line after your thought process to provide your final answer. This new line should contain EXACTLY one letter (one of A", "B", 'C", or "D"). No other content is allowed on this last line.
using Python
GPT-4o:
Error Analysis:
analyze target
provicied oode:
GPT-4o over-complicates the problem. It invents misaligrment between the tanget image and the code to be modified. The only disagreement two
1. Colorbar: the colorber the terget image very this and does not have ticks or lbbels. In the current code, the colorber placed i subplot ix_ cb and has ticks and labels. To mutchh the targer visual cation, we need to remove the ticks and labels from the colorber and make it
2. Subplot Layout: The space between the colorbar and the main plat is minimal in the target image. The ourrent code uses a nestgtt_retiin of [M.,, 3, al., which gives the colorbar
chart
for intensity the code has paddings on both ends due default settings, wherens the target image does not have
space than nacassary, need reduoe teight
subpoot.
3. Ticks Direction: The ticles the
pointing outward the code does not specify the tick direction. This a minor deta I, but it can be adjusted
Adjustment: colorbar avia be tumed match the tanget visuaiization.
these. GPT-4o fails point this out, but ballucinates about other issuss, highlighting limitations in multimodal reasoning.
Based this
adjust the flcutioo
colorber atis.
The cornect antwwet it:
Ground Truth:
The only misalignment herween the code and the target image is that the line chart for intensity in the code has paddings on both ands due to dafault settings, whereas the larget image does not have these, To remove the paddings, one simply acods to share the x-axis between the subplots. Correct Answer: A
Figure 38: A sample error case in coding.
46

---

Can MLLMs Reason in Multimodality EMMA; An Enhanced MultiModal ReAsoning Benchmark
Coding: Gridline and Data Reasoning
A
L appii3 soucl
B
re. 11. 8.5 1:81
Lal.taeeee Calent otbee'. "worte" "tat")
aloweea rerraenie, desspii
-10
00
Eit.
Maptit
00
1.0
-10
3.dircl k.shhihers)
acplatto cur.e a.crcle Locrt S1av68P-01
Ioart meteletihh mnstot
Upert
C
Aiaart snleon
D
Anart sedinn
x-xx
Tip pirleeree
pepptatit, prjartiee/55)
ex.pooort,, Ee A.M. lemtheil. rereiiooeess oslend stove, "partte', "bal/11
as Daatan.arrtenn w.a.trasrfaatall. 3h file Ally calen atataa
Ludente contttens
maaliataauutte culteily Kalosit, srker shar Limeaittett
Locitte M.e.s
11
unt.ntenn V11
Question:
Which code snippet possibly create the chart in the image? We are using Python version 3.11 ..O, matplotlib version 3.6.3, and seabor version 0.12.2. 2. Think step by step. Then, append line after thought process provide your final answer. This line should contain EXACTLY letler (one "f"A", """""""" "B", "C", "D"). other content allowed this last line.
GPT-4o:
Error Analysis:
Let's down the cook smocits who shor
The first mistake analysis is that blue circle between the top two surfaces, not on the middle surfsee. Nometheless, model correctly eliminates
1. 30 Plet The raga shous 30 plet win thrse surtaces quiver piet tarowig Thes
that sode Aese30
optocie anlcutivr
on the grounds that their placements of the circle are on the bottom surface. The more fatal mistake lies in the misinterpretation of the effect of rstride and esrride. Matplotlib 3D surface plots by default carry gridlines. Yet, in the larget chart we do not observe gridlines on the three surfaces, implying that the code must have overtidden this selting. However, GPT-4o concludes that such seltings are "unnecessary" in B, which leads the wrong answer.
visitt and hoc lecipute Recqulaed colin aode crigpets, whore the calor orgunent is torthsse vs aac.
Surfhose: aaartte
surfhcee
these color alphe value for transpawecy.
pird
polve,
Clicke There is live cldce the ridle fece. Thés created be ploding cindein the which netche invatool out aucaritte tne lecaat
Label:: The nos tabnlee re't which mutcheen an.ret staaett
Differences Code Siippetss:
tmage,
Ground Truth:
retride and iutriae may affect surbace resciation,
Valb in Image.
The placement of the bloe circle (between the surfaces) eliminates
The cinde plutted plutted at initl, wiichis mocomett
do not any gridlines on the three surfaces, which are part of default appearance m'mppoltib surface plots, overridden does by setting rstride estride to values large values, causing gridlines to sparse he rendered on the surfaces, Correct Answer:
ciide papttts 75. winch matchee imagu. The suriicas siotied
whehoe rntridde and catride, which since the image does not show vebble cecerfeee rescku
Conclusion:
Buth i and D the dirile cornect beigh, poet (2)) but stllynifernnnd withost
The cornect
Figure 39: A sample error case in coding.
47


Reviews:
[{"review_id": "fbVIqv5GD0", "reviewer": "Reviewer_QCsL", "summary": "This paper proposed a benchmark EMMA (Enhanced MultiModal reAsoning) to feature questions that are difficult to solve by relying solely on text-based reasoning or a single visual pass, covering math, physics, chemistry, and coding domains with 2,788 questions. Ten state-of-the-art MLLMs are further evaluated on the benchmark, revealing (1) a substantial performance gap compared to human experts and (2)  techniques such as Chain-of-Thought prompting and test-time compute scaling offering only marginal gains.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful and careful reading!\n\nAs you have pointed out, our benchmark provides a test suite that reveals significant limitations of even the most advanced MLLMs in handling complex multimodal reasoning tasks. Although state-of-the-art MLLMs have recently achieved strong results on many challenging benchmarks, their poor performance on our benchmark highlights key gaps in their reasoning capabilities. **We attribute this, in part, to our enhanced filtering pipeline, which allowed us to surface questions that genuinely require multimodal reasoning.** Importantly, this pipeline is reusable and could serve as a framework for constructing future benchmarks aimed at other aspects of multimodal understanding.\n\nIn addition, we utilized our benchmark as a testbed to provide technical evaluation and insights into advanced techniques for enhancing MLLM reasoning. By comparing model performance with and without CoT, and evaluating different scaling approaches—such as majority voting, best-of-N, and tournament selection—we highlight the limitations of current prompting and inference-time methods for complex visual reasoning. These technical insights also would not have been possible without a carefully curated benchmark.\n\nBy exposing these limitations, our work points to broader issues, such as potential shortcomings in current model architectures or a mismatch between existing training paradigms and the demands of complex multimodal reasoning. We hope this will encourage further research into more robust multimodal architectures and training strategies.\n\nWe deeply appreciate your constructive suggestions. Following your advice, we plan to explore new methods to tackle these challenges in future work, and we remain committed to advancing the evaluation and development of MLLMs’ visual reasoning capabilities.\n\nThank you again for your insightful and encouraging feedback!", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The paper does not include theoretical proofs.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "No", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n* The proposed benchmark reveals the gap between SOTA MLLMs with human experts, and will serve as a strong benchmark on multimodal reasoning evaluation.\n\n* The experiments reveal that test-time scaling strategies cannot achieve a strong performance yet, calling for new reasoning strategies in the future.\n\n* The experiments are comprehensive, covering many SOTA MLLMs. The curated data quality is also high.\n\nWeaknesses:\n\n* The paper is a pure benchmark without a new method. This is not a reason for rejection, but is a weakness or limitation on technical contributions.", "other_comments_or_suggestions": "Please see weaknesses", "questions_for_authors": "Please see weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nxgFyZsPmm", "reviewer": "Reviewer_PLqU", "summary": "This paper introduces EMMA, a novel benchmark designed to evaluate the vision-language reasoning capabilities of MLLMs.\nUnlike existing benchmarks that focus on shallow visual understanding or text-dominated problem-solving, EMMA emphasizes tasks where solutions inherently require iterative interaction between visual and textual reasoning.\nThe benchmark covers four domains — mathematics, physics, chemistry, and coding — and presents challenges such as 3D spatial transformations, chemical structure analysis, and multi-step simulation. It consists of 992 filtered questions from existing datasets and 1,796 newly curated questions developed in collaboration with domain experts.\nEvaluation of SOTA MLLMs reveals critical limitations: techniques like chain-of-thought prompting and test-time computation scaling (e.g., majority voting) provide only marginal improvements. Moreover, the models struggle with tasks requiring precise spatial simulation or the ability to leverage visual aids for enhanced efficiency, exposing difficulties in fine-grained spatial reasoning, multi-hop visual-text integration, and generating effective visual reasoning steps.\nThe authors argue that current architectures and training paradigms are inadequate for supporting deep multimodal reasoning, calling for innovations to better integrate different modalities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging review and insightful questions. We provide our responses below.\n\n**Q1: What does \"organically reason\" mean?**\n\nBy \"organically reason over and with both text and images\", we refer to the integrated way humans seamlessly blend visual and textual information during reasoning processes. To measure advanced multimodal reasoning capabilities, we target questions that require both modalities working together, where neither text nor images alone are sufficient. Models must dynamically engage with both reasoning channels and effectively fuse textual and visual information to succeed on our benchmark.\n\n**Q2: The filtering pipeline seems to only retain problems that are difficult to describe in words, which can include questions that are more difficult in visual perception than in visual reasoning.**\n\n\nWe agree that our first-step filtering, which removes questions solvable by models given only question text and image captions, may retain examples that are difficult to describe in words, but do not necessarily require strong visual reasoning. For instance, identifying a symbol in a musical staff might be retained simply because the symbol is visually subtle or hard to express textually, even though solving the task does not demand deep visual reasoning. \n\nTo address this potential bias, we installed a second filtering step. Specifically, we manually reviewed the remaining set and constructed a taxonomy that emphasizes multimodal reasoning skills. For example, for math, we identified categories such as 3D spatial simulation and pattern inference. **We then used GPT-4o to categorize the questions according to this taxonomy, followed by a final round of manual verification to ensure quality and relevance.** This process helped ensure that the retained questions require visual reasoning.\n\nHere we provide an [example](https://huggingface.co/datasets/MathLLMs/MathVision/viewer/default/test?views%5B%5D=test&row=3) from the MathVision dataset that is retained after the first-stage filtering but removed during manual verification. Solving the problem requires correctly perceiving all the digits in the image, which MLLMs struggle with. For instance, GPT-4o recognizes the digits at the bottom as \"2\" instead of \"3\", resulting in an incorrect answer. Although the question cleared the first-stage filter, we excluded it as it primarily tests visual perception rather than visual reasoning.\n\nTo further support our claim, we provide the following statistics from the MathVision dataset: out of 3,040 total questions, 2,195 were retained after the initial model-based filtering. **However, only 668 questions (30.43\\%) from that set were ultimately included in EMMA, based on our taxonomy and human verification.** This highlights that our filtering method applies a stricter standard to ensure a focus on multimodal reasoning. As noted in our error analysis (see Figure 5), MLLMs sometimes still fail on EMMA questions due to perceptual errors. However, we do not view these errors as contradictory to the multimodal-reasoning-based nature of the tasks.  Rather, they may reflect perceptual limitations that are prerequisites for successful multimodal reasoning—an important challenge in its own right. In other words, while the immediate failure may stem from perception, the questions still require multimodal reasoning to be solved, highlighting a compound challenge that current models have yet to overcome.\n\nIn summary, we have taken careful steps to ensure that the problems included in EMMA emphasize visual reasoning over low-level visual perception. We appreciate your thoughtful comment, as it raises an important distinction and gives us the opportunity to clarify our data curation process more thoroughly.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "This work is a benchmark study and does not involve theoretical derivations.", "experimental_designs_or_analyses": "Yes, the soundness and validity of the experimental designs and analyses were carefully reviewed. No significant issues were identified.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Yes. the related works are sufficient.", "other_strengths_and_weaknesses": "[Strengths]\n\n- 1) This paper is rich and well-grounded, providing detailed definitions of fine-grained labels, a clear data construction process, and comprehensive experimental results on the key issues currently being explored by the research community.\n\n- 2) The authors have contributed a large number of novel multimodal disciplinary questions through manual collection and expert annotation.\n\n[Weaknesses]\n\n- 1) The term \"organically reason\" mentioned in the abstract is somewhat confusing. How should the term \"organic\" be interpreted in this context?\n\n- 2) Using the criterion of \"whether an image caption can replace the visual input\" to filter the data intuitively seems to only exclude questions that are \"difficult to describe in words,\" but it may not necessarily ensure that the image content is \"more aligned with visual perception than with visual reasoning.\" For example, identifying a symbol in a musical staff or counting the zeros of a function graph whose expression cannot be explicitly determined might be incorrectly sampled. I wonder if this type of bias exists, providing some error analysis could make the work more robust.", "other_comments_or_suggestions": "No", "questions_for_authors": "Pleaser refer to Part of [Other Strengths And Weaknesses]", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XbNm85oHeF", "reviewer": "Reviewer_8w8m", "summary": "This paper introduces EMMA, a visual question answering benchmark requiring multimodal reasoning. EMMA includes questions in four domains: math, physics, chemistry, and coding. The questions in EMMA are filtered so that they are not answerable based on only the image captions and questions. The experiments show that both open-source and closed-source MLLMs fall significantly short of human expert performance. Besides, the effect of chain-of-thoughts prompting and test-time compute scaling on the evaluated models are discussed.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and great questions! We hope our response below helps address them.\n\n**Concern 1: Only 3 models are used in filtering. Other models might be able to solve the retained questions in the text-only setting**\n\n(1) The 3 models used were among the strongest available at the time. If none could consistently solve a question with text and captions, other models are unlikely to succeed under the same conditions.\n\n(2) We also wish to emphasize that MLLM-based filtering is **only the first step in our data pipeline** (Section 3.2). This was followed by manual filtering to ensure retained questions genuinely required multimodal reasoning. In fact, 16.4% of questions were removed at this stage because they did not involve substantial multimodal reasoning.\n\n(3) Following your suggestion, we tested more recent MLLMs in the \"caption and question\" setting. We focused on math to test whether our filter separates visual-reasoning-heavy questions from light ones (hereafter \"heavy\" and \"light\" questions).\nWe selected:\n- **100 \"heavy\"** math questions from EMMA-mini\n- **100 \"light\"** math questions randomly sampled from the filtered-out pool\n\nWe evaluated 5 SOTA MLLMs using only the question and caption, and also report a *full-input* setting (text + image) on heavy questions:\n|Model|Visual Reasoning Light (Text + Caption)|Visual Reasoning Heavy (Text + Caption)|Visual Reasoning Heavy (Text + Image)|\n|---------------------------|------------------------------------|--------------------------------------|-------------------------------------|\n|Claude 3.7 Sonnet|74|41|45|\n|Gemini 2.0 Flash Thinking|72|31|34|\n|GPT-4o|76|31|27|\n|Qwen2.5-VL-72B|74|27|39|\n|InternVL2.5-78B|56|23|31|\n\n- In the caption-only setting, all models perform significantly better on light questions than on heavy ones, indicating that the filtered-out questions can largely be solved via text-based reasoning and that our filtering pipeline effectively identifies visual-reasoning-heavy questions;\n- On heavy questions, providing the image offers only marginal gains over captions. Even advanced MLLMs still struggle to utilize visual information for reasoning, reinforcing the motivation for EMMA.\n\n**Concern 2: Lack of domain knowledge can cause weak performance** \n\nWe agree that a lack of domain knowledge can lead to weak performance on EMMA. However, we believe that the primary factor behind the underperformance of SOTA models on EMMA is not the lack of domain knowledge,  but poor multimodal reasoning skills. Recent benchmarks suggest that SOTA models are equipped with strong domain knowledge. For example, models have achieved 87.7\\% accuracy on GPQA Diamond (which tests graduate-level science knowledge), 78.2\\% on MMMU (which spans many subjects), and 86.5\\% on AIME (a high school-level math competition). **This suggests that the knowledge required for EMMA is largely present in current models.** Hence, we believe that poor multimodal reasoning skills are the bottleneck.\n\n**Concern 3: EMMA is limited to four domains and does not include natural images**\n\nIn scoping EMMA, we explored many disciplines for questions meeting our strict criteria. We found that in many domains the primary challenge lies in knowledge rather than in multimodal reasoning. For example, many biology questions involve tasks such as labeling parts of a complex diagram. These tasks often hinge more on knowledge than on reasoning over multimodal information. Constructing/curating multimodal reasoning questions in these domains is also more difficult. That said, we appreciate your suggestion and plan to involve more disciplines in future work, such as law and medicine. \n\nWe also agree that including natural images would be valuable. However, sourcing high-quality, reasoning-focused problems using natural images has proven to be challenging. Benchmarks like MMMU also include very few natural images. In math, physics, and chemistry, most problems are accompanied by simplified diagrams or sketches designed to reduce ambiguity and clarify assumptions. We will explore ways to design or curate problems involving natural imagery in future work.\n\n> Q1: Meaning of \"a single visual pass\"\n\nWe operationalize \"a single visual pass\" as generating image captions with GPT-4o. Multimodal reasoning questions often require looking at an image multiple times for multi-step/multi-hop reasoning. If a question can be solved with text and captions, its visual content can likely be compressed into text with shallow visual perception/reasoning, and the rest can be handled by text reasoning. Whereas previous work, BLINK [1], assesses perception beyond recognition, we target images that are necessary for reasoning beyond perception.\n\n[1] https://arxiv.org/abs/2404.12390\n\n> Q2: Meaning of Pass@N\n\nPass@N refers to the accuracy when generating N responses per question and checking if **any** of them is correct. Thus, it is an upper bound for accuracy from test-time scaling.", "claims_and_evidence": "1. The paper claims that the state-of-the-art MLLMs struggle with multimodal reasoning. This is supported by the experimental results that the MLLMs achieve much lower accuracy than human experts on the proposed EMMA benchmark.\n2. The paper claims that the EMMA benchmark cannot be addressed by reasoning on solely one modality. The data curation process uses this condition to filter questions with GPT-4o as the reasoning model. However, other evaluted models are not tested in the single-modality setting to support this claim.", "methods_and_evaluation_criteria": "In the experiments, the MLLMs are not only evaluated in direct prompting, but also with chain-of-thoughts prompting and test-time compute scaling. A group of commonly used approach to boost reasoning performance are used to demonstrate the weak performance on the proposed benchmark.", "theoretical_claims": "This paper does not involve theoretical claims.", "experimental_designs_or_analyses": "I checked the soundness/validity of the experimental designs and analysis and did not find any issue.", "relation_to_broader_scientific_literature": "The previous multimodal reasoning benchmarks have shortcuts in the questions so that they may fail to evaluate the multimodal reasoning capability. The newly proposed EMMA benchmark filters the questions in previous benchmarks and also collects new questions. The questions in EMMA are ensured that they are not answerable based off the questions and image captions, making is a challenging benchmark for multimodal reasoning evaluation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other Strengths:\n1. The proposed benchmark is well presented. The paper shows detailed statistics and example questions in the benchmark.\n2. 1,796 questions are newly collected by domain experts, which are valuable to the community.\n3. A few advanced techniques to improve MLLM reasoning are evaluated and analyzed in the benchmark. This demonstrates the advantages and limitations of these techniques.\n4. The paper is well-written and easy to follow. The proposed benchmark is well presented.\n\nOther Weaknesses:\n1. Although the data curation filters out the questions that can be answered based on questions and image captions, only GPT-4o, Llama-3, and Qwen2 are used in this process. So, the weak questions filtered out are restricted to the capability of these models. It would be more convincing if experiments can be conducted to evaluate other MLLMs in the \"question only\" and \"caption and question\" settings.\n2. While EMMA includes challenging multimodal reasoning questions, they are restricted to the domains of math, physics, chemistry, and coding. This would lead to two issues:\n\n+ Answering the questions in EMMA requires strong domain knowledge. Therefore, the accuracy on the benchmark cannot directly reflect the reasoning capability of a model. The lack of domain knowledge can also cause weak performance.\n\n+ The domain of the benchmark is limited. It does not include more diverse domains, e.g., geography and biology. More importantly, none of the images involved in the benchmark are realistic images. While it is acceptable to propose a benchmark in specific domains, the paper should discuss it as a limitation. Besides, the name \"Enhanced Multimodal Reasoning Benchmark\" exaggerates the contribution of the benchmark because it is domain-specific instead of a general-domain benchmark.", "other_comments_or_suggestions": "While the domain of the benchmark is limited, I highly appreciate the level of challenge of the benchmark and the thorough evaluation of MLLMs and reasoning techniques. So, I give a score of 3 as my initial recommendation.", "questions_for_authors": "1. In L 193-194, what does \"a single visual pass\" mean?\n2. What does \"Pass@N\" mean in Table 3?\n\nThese should be clarified in paper writing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofIMGdIkJ4", "reviewer": "Reviewer_Z6QV", "summary": "This paper introduces the EMMA benchmark to evaluate the reasoning capabilities of multimodal LLMs that require the integration of both text and visual cues. The benchmark is curated from existing datasets and supplemented with 1796 newly created questions covering math, chemistry, physics, and coding. A filtering process ensures that questions cannot be answered using text alone. Additionally, the paper evaluates SOTA models, providing a comprehensive analysis of both direct prompting and CoT prompting. The study also explores test-time compute scaling. Results indicate that current SOTA models exhibit a significant performance gap compared to human experts across these evaluations.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the encouraging and thoughtful review! Below is our response to your questions and suggestions.\n\n**Q1: Why did we filter out questions that can be answered using the text and generated image captions?**\n\nOur enhanced filtering pipeline targets questions requiring deep multimodal reasoning that are difficult to solve through text reasoning or one visual pass. If a question can be solved with text and image captions, it means the necessary visual information can be compressed into text. The question is then solvable with one round of visual perception or shallow visual reasoning, with the rest handled by text reasoning. Instead, we focus on problems requiring back-and-forth textual and visual reasoning, which typically demand repeatedly referring back to the image for multi-hop reasoning or mental simulation. This filtering pipeline allows us to curate problems that assess advanced multimodal reasoning.\n\n**Q2: Counterintuitive task of \"Code Choose Vis\"**\n\nWhile \"Code Choose Vis\" may seem counterintuitive at first, it is tied to how humans typically write visualization code, whereby they envision their desired chart, code it, and mentally check if the code achieves their intention. The critical step of asking oneself, \"is the code doing what I want it to do?\" is precisely what \"Code Choose Vis\" problems target. This ability is also prerequisite to debugging visualizations, where one must understand what the current code produces before making corrections.\n\n**Q3: Plans to expand EMMA to more general domains**\n\nWhile most questions in EMMA require domain knowledge, many math questions do not. For example, simulating how a 2D shape will look after transformations or finding patterns in a sequence of shapes relies primarily on logic and general knowledge rather than math expertise.\n\nIn scoping for EMMA, we explored various disciplines for multimodal reasoning questions meeting our strict criteria. We found that most such problems are typically considered \"logic\" or \"spatial reasoning\" tasks, which we ultimately categorized under math. In other domains, such problems are much harder to source or create. Nonetheless, we are very interested in incorporating more general domain multimodal reasoning questions and are actively exploring strategies for expansion. Thank you for this great suggestion!\n\n**Q4: Why does CoT prompting not help with open-source models?**\n\nWe observed that CoT prompting generally improves performance for closed-source models, but tends to hurt performance for open-source models. One possible reason is that open-source models do not effectively leverage language to assist in multimodal reasoning tasks where language could be beneficial. For example, language can often help grounding in multi-hop counting, so it should be theoretically possible to improve performance on this task with CoT, but open-source models fail to capitalize on this.\n\nPrevious work has shown that the quality of training data is crucial for CoT effectiveness. For example, [1] has found that filtering out anomalous samples (e.g., repetitive patterns) can improve CoT performance. Your point regarding the lack of CoT supervision during training aligns with this perspective: incorporating high-quality CoT data can indeed be seen as one way to improve the quality of training data for MLLMs. \n\nWithout access to the training data and pipelines of the tested models, it is difficult to pinpoint the reason behind this divergence. Nonetheless, we believe that multiple factors might contribute to this phenomenon, and a more systematic understanding of CoT prompting in multimodal settings remains an open and important direction for future research.\n\n[1] https://arxiv.org/pdf/2412.05271\n\n**Q5: Does high Pass@16 indicate latent capability and can fine-tuning on similar data boost performance?**\n\nThat is a keen observation. We agree that strong Pass@16 performance may suggest latent reasoning capabilities. However, as you noted, even if models possess relevant knowledge, they have a low probability of applying it correctly. Moreover, correct answers can sometimes stem from flawed reasoning, especially for multiple-choice questions. \n\nThe idea of finetuning on similar questions can indeed be promising. Prior work [2] has highlighted the nuanced interplay between memorization and generalization in finetuning LLMs on logical reasoning, and it would be interesting to investigate this further in the multimodal setting—an interesting direction for future work.\n\n[2] https://arxiv.org/pdf/2410.23123\n\n\n> Suggestion 1: Discussing additional relevant work\n\nThank you for these highly valuable and relevant references. We will include and discuss them in an updated version. \n\n> Suggestion 2: Highlighting top scores in tables and including human performance breakdown on EMMA-mini\n\nThank you for the thoughtful suggestion. We will make sure to more clearly highlight results and include a category-level breakdown of human performance.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes, only two questions \n1) Regarding the filtering mechanisms in L206–L209, what is the motivation for removing questions that could be answered if the original text and generated captions were provided? Captioning an image still relies on visual cues, and if additional reasoning is required, wouldn’t such a question still assess the model’s multimodal reasoning capability?\n2) The ‘Code Chose Vis’ setting seems somewhat counterintuitive. It requires directly reading code and mentally visualizing the output, which is quite challenging even for humans, especially for very long and complex code.", "theoretical_claims": "The paper does not have theoretical claims.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "There are many multimodal benchmarks, such as SeedBench, MMMU, RealWorldQA, MuirBench, and VideoMME. Some benchmarks specifically focus on reasoning, such as MATH-V, Visual CoT, and SpatialRGPT. However, some questions in these benchmarks can be answered using only textual cues, or models have already reached saturation on them. This paper introduces several mechanisms to ensure that both text and images are required for reasoning. Additionally, it demonstrates a significant performance gap compared to human experts.", "essential_references_not_discussed": "There are few works related to MLLMs' spatial reasoning, and all of them provide benchmarks in their papers.\n\n[1] Cheng, An-Chieh, et al. \"SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models.\" arXiv preprint arXiv:2406.01584 (2024).\n\n[2] Liu, Fangyu, Guy Emerson, and Nigel Collier. \"Visual spatial reasoning.\" Transactions of the Association for Computational Linguistics 11 (2023): 635-651.\n\n[3] Nie, Jiahao, et al. \"Mmrel: A relation understanding dataset and benchmark in the mllm era.\" arXiv preprint arXiv:2406.09121 (2024).", "other_strengths_and_weaknesses": "The paper is well-written and provides a thorough approach to benchmark creation, comprehensive experiments, and detailed analysis. Reading this paper gave me valuable insights into the current limitations of multimodal reasoning capabilities.", "other_comments_or_suggestions": "1) It would be helpful to highlight the best and second-best scores in the performance table in the supplementary material, similar to the table in the main paper.\n2) Including a breakdown of human performance for EMMA-mini would be valuable as a reference when analyzing the results.", "questions_for_authors": "1) The benchmarks cover Math, Chemistry, Coding, and Physics, requiring not only multimodal reasoning but also expert knowledge to solve problems. Do you have any plans to expand into more common scenarios that primarily rely on logical reasoning and general knowledge?\n2) Regarding the gap between closed-source and open-source models, do you think this is partly due to most open-source MLLMs not incorporating CoT during training? Would this also lead to hallucinations when CoT is enforced during inference?\n3) For N=16 Pass@N, the performance is quite high for all three models, with some models even outperforming human experts. Does this suggest that the model already possesses the necessary knowledge and reasoning capability but with low probability to generate? If so, would fine-tuning on similar questions from the benchmark provide a shortcut to boosting performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces EMMA, a high-quality benchmark for evaluating the multimodal reasoning abilities of MLLMs across domains such as math, physics, chemistry, and coding. Reviewers unanimously appreciated the benchmark’s novelty, rigorous filtering pipeline, and insightful evaluations, which expose current models’ limitations in visual-textual integration. While some initial concerns were raised about the domain scope and methodology, the authors provided thorough and convincing responses that addressed reviewer queries, leading to rating increase. Given the paper’s strong motivation, comprehensive evaluation, and community value, all reviewers now recommend acceptance. AC agrees that this is a solid contribution and should be accepted.
Decision: Accept (oral)